0:51 Introduction and Housekeeping

0:51 Hello and welcome to today's episode of Rawkode live. I am your host Rawkode. Today we are gonna be taking a look at Tremor. It metrics, logs, collection, pipeline, transformations, loads of really cool things for it to look at today. And it's written in Rust, which makes the project even killer in my book. I'm always excited to see new Rust projects taken the limelight. So before we do that, I'm gonna remember to do my housekeeping for today and that means I'm gonna ask you all very kindly. Please subscribe to the YouTube channel, click that bell and get updates whenever new episodes

1:23 are coming. They are regular weekly and loads of cool projects for us to explore. Also, have a discord channel where loads of people come and chat about the technologies that we cover as well as general Kubernetes and cloud native topics. And we have a monitor observability channel, which would be a good fit for today's episode. Also, I need to thank my employer and want to thank them of course. They're awesome and they allow me to spend my time playing with random technologies that I find exciting and show them to the world. So thank you Equinix metal. Feel free to check out

1:54 the products. We have bare metal clouds. You can get $50 in credit. That is roughly one hundred hours of compute depending on which instance you you choose. So enjoy. Now we have a bit of a treat today. I am not joined by one nor two nor three but four guests from the Tremor team. Hello everyone. Just wave in all directions. So could we start with a quick round of introductions then? I guess we'll start in the top left and work our way around anti clockwise finishing back on me. Please introduce yourself. Tell us who you are and a little bit more

2:04 Guest Introductions (Tremor Team)

2:29 if you wish. Well, hello, everyone. I'm Heinz. Like the catch up, no relations there. Just the same name. I've I've I've done open source for, well, two decades nearly. Quite enjoy it. Really happy that we could do this with Tremor too. And yes, I'm working on Tremor. The this thing we are talking about today, you'll hopefully hear a bit more about it unless we get sidetracked and spend time talking about cats. It has happened before. Yeah. I I really am glad that I can have a blurry background today, which is my my achievement of the week.

3:08 Otherwise, yeah, let's pass it on. Anoop. Okay. Hello, everyone. My name is Anu. I'm a software engineer at Wayfair.com, currently based in Boston. So this was my first real job out of college. I've been working with them for over seven years now, and I've been in, like, various teams within Wayfair. So Tremor is the latest one among them. So, yeah, I I I like working for Tremor with everyone here. I guess it's over to me. I guess I'm the cut harder on Tremor. Heinz and I started about two and a half years ago, and it went to open

3:49 source about a year ago. But, generally, we just sit down, write code, have fun, and try and break it so that it doesn't break up for anyone else in production. And now over to the youngest and newest and freshest member of the team, perhaps the funniest, Mathias. Yeah. Hi. Welcome. Mathias here. I'm I'm the newest addition. I'm I'm the one who's doing all the silly mistakes and bugs, and I'm trying to introduce them into our code base. But those guys always stop me. It's it's annoying. So yeah. I have I am actually also on the

4:26 language and the core team of the Pony language, and this is also, like, how I got to know those people and how I got this job actually. And now I'm writing Rust, although I want to write pony all the time. But it's it's great fun. Yes. Thank you. Alright. Thank you everyone. Lots of different accents as well. We we're all in different locations. I guess you're a fully remote distributed super team. Is that the best way to describe you? We're all distributed across Berlin except to Luke who is He's distributed to Boston. For some strange reason.

5:01 Are not Ah, has an import. Yeah. We stole him from The UK. Ah. He stole me from Ireland. Yes. Okay. So much theft. Yep. Awesome. I I I I I'm gonna have to get a bell or something to keep you all in check, but we'll we'll we'll get through this. Good good luck. Good luck. I know. I'm getting that feeling already. Okay. So we're taking a look at Tremor today. It's it's Rett and Rust even though one or a quarter of your team wants to write Pony all the time. How does Pony and Rust connect? What's what's

5:30 What is Tremor? (High Level Overview)

5:39 the connection there? Or is it Yeah. It's like the focus on safe on the safe runtime, like memory safety, database freedom, all this stuff, like but the approaches are totally different. And Rust has a much bigger community and yeah. Yeah. I am a fan of the pony language. I have tried many times to write more of it, but that's maybe a conversation for another day. We definitely should do a stream on pony. I'm gonna end the next That means we have half of the pony user base here, don't we? That's harsh. No. That's not true. That's not

6:15 fair. We also have half of the Erlang user base here to be fair. Right, brother? That's us too. Okay. That is fair. That is fair. So I think what we've got is a collection of people that like esoteric programming languages and solving Yes. Cool and hard problems. And I think that is a fantastic mix of people to be doing a stream today. Awesome. I only have tears in my eyes. This is going well. Alright. Tremor. Someone tell me what it is, please. I I I said at the start, it's metrics, it's logs, it's transformations, it's event processing. Let let's let's dive into

6:40 Tremor Overview Slides Begin

6:46 that. Much more and so much less. Yeah. Believe we've some slides. Was that right? Yes. We have some custom slides, which were prepared over the last decade with a lot of care and fifteen minutes over lunch. So Tremor. Let let's go a bit into it. Let's start with, first, the big bang. Tremor today is doing about 10,000,000,000 metrics or events every minute here at Wayfair. That's, by the way, where all of us work. So they they effectively sponsor Tremor, which is a nice number, isn't it? 10,000,000,000? Might be a bit more by now, so who knows? Those numbers keep going up. Quick

7:32 three things we are going to talk about in the slides. What the heck is Tremor? Why the hell is Tremor? And where are we hiding if you ever want to use Tremor? And I'm pretty sure everyone will want to use Tremor after today. At least everyone that wants to use Tremor will want to use Tremor after this day. So quick, what the heck is Tremor? And Tremor is very simple because Tremor is a lot of things. We talked already about metrics, about logs, about events. But basically, Tremor is two things, three things, two arrows and one cookie monster.

7:50 Tremor Components: Sources (Onramps), Pipelines, Sinks (Offramps)

8:03 Tremor is sources, so something where data comes from. Tremor is a cookie monster, so something that does something with cookies or events. And Tremor is syncs, which is something where data goes to. And that is pretty configurable. Quick idea about when we are saying source, what do we mean? We mean certain connectors, HTTP, Kafka, TCP, UDP, and a host of other ones and some encodings. Some we support right now is JSON, YAML, message pack, just strings, just binary data, open telemetry since, well, since tomorrow. Since tomorrow? Is that a word? Can I say that? I don't know. I'll say it.

8:46 Yes. You you can say it. Good. Since tomorrow. Worries. Wonderful. So open telemetry since tomorrow. I like since tomorrow. I'll keep using that. Pipelines, which is once the data is in Tremor, what happens to it? We we have a whole programming language called TremorScript, which you all have to suffer through later when poor Rawkode has to write it. And and you can filter the data. You can transform the data. You can sanitize the data to give it a common theme and common look. And you can pretty much build all kind of logic in it that you

9:23 want, Like, data in, data out, do do stuff with it. Usually shown as a cookie monster in our presentations where we don't get sued for using that picture. I hope we don't get sued. I hope you don't get in trouble for this picture, by the way. If you do, I'm sorry. It's not our fault. It's your stream. It's your responsibility. Goodbye. Syncs is the other part, and this is kind of a mirror of the sources. We have, again, the connectors, HTTP, Kafka, TCP, UDP, and so on. There's a whole host of them, And they are on the website with

9:53 documentation. I'm pretty sure we will go there at one point in the future today. And same as the codecs going in, we have them going out. So it could be JSON, YAML, message pack strings, binary, or make up your own, really. So why all that stuff? Why all the work? First thing is that Wayfair, there are thousands of engineers doing doing their things. So there are a lot of metrics and stuff. So there are a lot of people, so there needs to be a good way to deal with all the small things. So that leads to a lot of data.

10:10 Why Tremor Exists (Scale and Motivation)

10:34 We we had last time, we looked at the numbers about 10,000,000,000. I suspect it's a bit more today. Events per millet or 3,000,000,000 a day, 3,000,000,000,000 a day, which for everyone counting to trillion a day is significantly much. To to give it other shape than the silly numbers, it's about 10 terabyte or one hard disk if you buy a really big one or 10 small ones. Two years out of date, I think. So They are two years out of double million. Yeah. So, yes, you can you can multiply that with x, and you choose x to get a

11:13 correct number where x is somewhere between zero and infinity. But it's a ballpark. It's probably not an order of magnitude of. Last but not least, there really wasn't something out there that solved the issue completely without having a lot of trade offs. Because realistically, not everyone is at the scale, and most people are really happy with simple solutions that work for them, which makes perfect sense. And if you're not operating at tens of terabytes a day on data, there is no reason to invest the time, money, or effort into building something that can handle that.

11:51 So, yeah, that's the why. Last but not least, at as promised, we won't take too much time with slides. Where are we hiding? I hope it's if I pluck our stuff here. There's TremorS, the website. There is GitHub repository, which is also linked on the website. There's a doc site, which is also linked on the website. And we have a Discord server. You're all welcome to come by and say hi. There is a link for it. It's probably also worth going in where Tremor started or why it started in detail. It wasn't really built with the the current

12:20 Origin Story: Load Shedding Use Case

12:27 scripting or event processing or, ETL in mind. It was built for a very simple problem. When a network operation center, is under capacity for volume when we hit peak load, Bad things happen. Like, our network operating center folk are seeing, data that's delayed a minute or two minutes. And as you know, in a large production environment, if you're two minutes out of date, well, you're you're putting out the wrong fire. So Tremor was basically introduced to solve that problem and introduce, a form of load shedding and capacity planning and management that worked in real time. So

13:04 that went to production in six weeks, and we spent the past two years avoiding that problem of just having fun. Yes. It worked well. Alright. Awesome. Thank you for the the introduction there. So I guess I've I've got a couple of questions. I I don't know if we covered this during previous conversations, but I know I my previous life, I did a lot of things in the monitoring and observability space. In fact, I worked for Inflox Data, if you're familiar with that company. Yes. This feels a lot like Telegraph. Is that a fair comparison? It it is a subset.

13:20 Tremor vs. Telegraph & Fluent Bit

13:39 So Tremor can pretty much do what Telegraph does to the most part, but it is a generalized solution. You can do it on metrics, like aggregate stuff, sum them together, filter them, rewrite them, but you also can do it on logs. You also can do it on, well, pretty much everything else you want. You have a structured data in, structured data out approach with a language to modify the data in the middle. So, for example, we're we're currently putting a system in production at work that is in the search domain. As Tremor doesn't really care about the underlying

14:15 symbology or the underlying domain of Endeavor, pretty much anything that's semi structured or self structured, in nature can be processed through the system. And we're now starting to get into binary data processing and workloads as well. So that keeps extending. So although logging and metrics are the observability domain was where we started building the engine, it's always generalizing and becoming kinda more useful. And that tends to of pull us into other domains and directions over time. Like, I think the search solution is is very orchestration. It's workflow oriented. So if you imagine Yeah. You have a messaging

14:55 system inbound and outbound, you're trying to index and elementize documents style information, pump it into a bunch of downstream systems that are indexing for search, like to power recommendations and engines and things like this. There is no messaging system that gives you fully validated, authenticated, and cleansed data as a transactional messaging feed, because it doesn't do those complex orchestrations of multiple external systems. So Tremor is now being used for that use case, and that will likely move us down into the orchestration direction. So, very much logging and metrics was the start. Who knows where the rabbit hole goes, looking

15:40 forward? We do support in the influx line protocol as one of our codecs. So you can encode and decode influx data in Tremor. Okay. Cool. Very cool. Alright. Kim, what are the typical deployment architectures for Tremor? Do I run one of them inside of a data center? Do I run them on every node? Do I get a little bit more creative with the paired container? Like, how do you use it yourself? It depends a lot on your use case, as always. Right? That is that is the blanket answer to every question. It depends on your use case.

15:50 Deployment Architectures

16:15 We I'll I'll I'll leave out the clustering part for a moment because Anoop can tell us a bit more about that in a second, I think. He he's working on that at the moment. But, basically, today, Tremor is either run as a sidecar in Kubernetes or as a dedicated server for high load massive pipeline situations where you have a number of servers and load balance over them. Yeah. I'm a big fan of of that approach, and it's something I I try to encourage people to do with with Telegraph and other tools as well is that you really

16:45 get to dictate their own quality of service when you run the collection right next to your application. And then you can have very specific transformations and aggregations that are happening at that layer. I think that's a really, really cool approach to doing that. So I'm excited for what Tremor can do. Something that we've not mentioned, but I I noticed on the website is that Tremor is also a a cloud data foundation project. You wanna Yes. Is that recent, or is I just did I miss the announcement? September, I think. September. We were invited to join

17:00 Tremor as a CNCF Project

17:15 we're enjoined. We're outjoined, I think, is the term. And joint, yes, as a CNCF sandbox project, which is super exciting for us. Yeah. And definitely going against the grain there, not writing your cloud native application in Go. Oof. Yeah. There's only we'll change the CNCF. Go is going out. It's literally going out. Yeah. Go going gone. I think Tremor and maybe Linkerd I did not mean that. I think Tremor and Linkerd are probably the only two that I'm familiar with. They're they're leveraging Rust. Oh, yeah. T I k v as well. That's right. Definitely. That's cool.

17:53 Okay. One more question then. Let's just get hands on. I I asked this about every project that's written in Rust yet, but have you compiled it for VASM and run it in a browser? The answer is definitely not because we use CMD accelerated JSON passing. Probably, that quite a bit of work when you're doing SIMD accelerated JSON processing. We don't even get it to run on ARM yet. So And is that the cause of UMD? Is that the same issue? Yes. It's the same so Rust on ARM has not yet caught up with Rust on x 68. Most normal stuff you

17:55 Technical Deep Dive: Rust, WASM, and SIMD

18:32 can run perfectly fine. I want to prefix it. It's not like Rust doesn't do ARM. But if you go into really low level SIMD stuff, Rust doesn't yet fully support that. For example, you'd use you need to use Nightly for compiling it, that means that every other week, everything breaks. It's just not worth the effort. Yeah. I don't know if that makes me a good Rust developer or a terrible one, but I've never installed the nightly tool chain. And I just tried to avoid it as much as possible. Both. It makes you a terrible good Rust developer.

19:04 I'll take it. It makes you a happy Rust developer with a stable environment to work in and take pleasure out of your trouble. Cool. Can we I mean, who feels comfortable describing CMD? I can give my very naive understanding of it. But I believe the idea is that you have a single value and a register and you can perform multiple operations on it at a much efficient or effective process. Is that correct? Matthias just gave a talk on that. Literally. Literally on Bobconf, what, two weeks ago? Yes. Of me. I don't, like, I don't count

19:37 time anymore. It's Yep. Time is a complicated. Just just a very quick yeah. It's single instruction multiple data. That's what single stands for. So you have basically, you have fat registers in your CPU, and you can put, like, a number of integers, clothes, whatever inside there, or bytes or, like, whatever primitive CPU data type you have. And then you can just run a fat instruction in the same time usually as you do on a single one. And, like, as a curious fact, like, most like, all the Intel CPUs, they want they run their floating point operations on those registers.

20:18 So you have you're already using them, but you're underusing them. Yeah. So most compilers optimize very heavily and and transform your normal code into, like, unload loops and stuff. And so you get so you you get to use SIMD, but in in the SIMD adjacent project, we explicitly use SIMD instruction. It's it's a it's a, like, a whole new channel of CPU instructions in those systems. It's it's it's fun, but it's, again, a repertoire. Yeah. Okay. I mean, so I go, like, completely backwards then. Basically, my entire yeah. That's cool because, you know, what the best

21:02 thing I always say about doing this is that I get to have wicked smart people like you all come on and just teach me all this stuff. So that it's Well, that's the same for us. Like, the Cindy stuff credit goes to Le Meyer et al. He invented this. We've just ported it. We just we we basically, we're in the business of stealing great ideas. But we give credit. We try. Yeah. It's criminal but in a good way. Yeah. It's like Robin Hood. Okay. We have a question from the audience and then let's get hands on with this

21:30 Normalization Use Case & Operator Friendliness

21:34 and start playing with So you're gonna ask him, could this be compared to fluent bet with regards to logs and events? And would it be a good match for integration with connecting with Fluent Bit? And then there's a quick follow-up is like, or would it be a competing system? Does anyone wanna touch on the comparisons to to Fluent Bit? That's a great question. Unfortunately, we don't know a lot about fluent bit, but we can probably go into the question if we ignore the fluent bit bit. So is it good for normalization? That's literally exactly what it's used for in

22:09 production. So the bread and butter use case in our logging and metrics environments, and it's starting to appear in other domains in production at work, is, for the normalization for dimensionalizing real time streaming data. And then based on the classification of that data and the dimensions to within which it was classified, to make decisions on whether, to send that data downstream. So 99.999999999% of the time, Tremor forwards all of the data it receives downstream in production. Every now and again, when our systems go over capacity or there's an issue when one of our data centers are in the cloud,

22:51 we don't have the capacity that we normally have or the systems are flooded. Whatever reason, Tremor needs to throw away some information. And we try and throw away the least important data to the business at any given point in time. And that's a fairly simple algorithm to build and define inside Tremor. So what it typically does in a in the logging system, which is a, fairly simple three tier architecture, on all of the source systems, so when our data center is that's in the in on the hosts that are running application logic, It might be picking up syslog information. It

23:22 might be picking up, data from, journal d or or syscontrol or something like that or syslog, as a sidecar in the Kubernetes case, sends it to a transformation system where it normalizes it. For example, try and get 3,500 developers to agree on what they call their environment. Is it gonna be env or environment in in their data? Are they go even going to agree on on the data structures? No. So we have a team of 20 very dedicated folk in Boston. They just normalize all of these datasets, in production and have a fine grained understanding of our

23:57 capacity requirements from the operational infrastructure perspective, but also from the production business need perspective. In an ecommerce environment, in a production environment, that changes by hour of the day, day of the week, and the season, that we're in in the year. So it's really, really good at this particular task. It's essentially what it was designed for. And it was also designed in an environment where it was designed to be transparent to the existing system and the surrounding architecture. So Tremor is an interposer. It's like a spy. Most of the developers of Wayfair don't even know it's there.

24:37 Yet, all of their messages, all of their logs, and all of their metrics pass through it. But they never need to see it. They don't need to touch it. But they go through Elasticsearch. Elasticsearch, as a front end to a developer is being served in the back end through multiple layers of Tremor. Similar to our influx environments, Tremor is actually feeding our Influx environments, inside work rather than Telegraph. So so Tremor takes on that role and effectively becomes Telegraph for the metrics environment or becomes Logstash for the logging environment. So it would probably work similarly to,

25:16 Fluent Bit, except it's really designed for that large scale high throughput environment. We don't know the specifics of Fluent Bit. There are likely features in Fluent Bit that Tremor does not have. It's likely a good fit for whatever it's currently being used. So if you don't need Tremor, just take a few minutes. Yeah. There there's one thing which is a big difference between Tremor and most systems, including Fluent Bit, is that we design Tremor as less a configurable system than a programmable sys programmable system. We have a deployment language, and we have a scripting language. Well, with systems

25:51 like Fluentd or Logstash, you usually have a very configuration and plugin driven system where you chain up your plugins. Well, in Tremor, you have a query language which lets you describe a graph and a programming language which lets you describe a logic, and they fold into each other. So you're not hopping between different languages, different systems, different logics. You don't write a plugin in git a git in Go and then write a script in Lua. You have this one system that fits together. And operator friendliness was always a very, very important thing. So you will you will well,

26:26 hopefully, you won't see it because if you see it, you make a typo, and the system will tell you. But we have hygienic errors and all those kind of things, which we really are proud of. I think that is one of the biggest differences to other systems that do event processing. Alright. Perfect. Okay. We we spoke a lot. I think we need to we need to do a Well, speaking is always fun, isn't it? Yeah. It can be, especially when I'm talking to you for it. That's for sure. So we've worked not to be bored.

26:45 Transition to Live Demo

26:58 We got a comment saying a sidecar on every pod we require to retrieve logs from. Yeah. That's my understanding. I think that's a really cool pattern. So I would recommend you check it out. And then I don't know if you know this person but Rawkode has said, hey awesome to see the whole team. Hey. Okay. Let's play with Tremor. Yes. And we have a treat. So we we looked at some of your stuff and they're all really interesting. And but we figured we are doing something completely Everyone is expecting we are doing lots today, but we are not.

27:17 Discord Bot Demo Setup

27:33 So so here's the idea. We happen to boss or bot? Did you say bot? Can you read minds? So Tremor is event processing, so it can do very much, including Discord bots. And we happen to have a Discord bot written in Tremor. So the idea is, and you depends on how you're comfortable you are with it. Either you invite the dev of the spot into your server, and you you play around with it there and can add some stuff. Perhaps let some of the users there figure out a feature that is worth adding, or we invite you

28:08 to our development environment for that bot, you can go bananas there. The bot's not going to run on our systems. It's going to be run on yours. So whatever is more comfortable for you. I like the idea of going bananas. I guess maybe we'll use your one, and I'll run it on my side. Is that the preferred approach? What whatever you like. You you the the we just didn't want it to, like, steal people from your server and be like, hey. If you want to play with a bot, come to the Tremor server. We also don't want to be, hey. Install

28:41 the spot on your server. So we let you pick which one you want. Well, let's try it on yours. And if I like it, then maybe I'll install on it. Okay. Let's let's let's you're on our server. So I'll I'll invite the Tremor dev bot to our server so people can play around there. And I will give you first the link to the code where you have the starting point. I'll I'll send you to this no. Not to Matthias. Here you go. That is a starting point. I will send you the secret key as well.

29:15 Please don't share this key on YouTube. That's that's the one part that probably shouldn't be shared. I'm going to invite But please share the URL. We might have multiple commits during the live Yeah. Would be hilarious and awesome. Okay. And let's throw the nice little Tremor bot. So the Tremor bot's the limit on this is going to be it can only read and reply to messages. Messages. We don't give this one admin privileges because it's a playground system. Read, rev. Yeah. Send. There we go. And, yes, I'm a human. Silly thing. Authorized. We have a Tremor bot. We have

30:05 two Tremor bots now. There's Tremor bot and Tremor dev bot. We we have a few ideas what kind of stuff you could add to that, but we want you also to just come up with your own, what you think would be a cool feature in a bot. Okay. So I just wanna make sure I understand where we are right now because I'm I'm already confused. So you have added a new bot to the Tremor Discord channel. Yes. I have the code for it in front of me now, which I'm now sharing my screen. I believe you're going

30:35 to send me something secret, but I don't think you've done that I believe I did. The the there's a secret s s h hash y s s h. Right. Right. Got it. Okay. Actually, that's like, if you if you look at the setup part here, there is put your secret key in the file because the bot needs to authenticate with the Discord API. And this is the secret key that authenticates it. For safety reasons, I will recycle this after today, obviously. But for today, you can play with that as much as you want. Okay. Let's get started then. So do I need

31:15 to clone this repository? Yes. Yes. Okay. That I am confident doing mostly. So, get clone Tremor bot. And we have here now do I need to read the instructions or you're just gonna give me? You can you can read the instructions. Alternatively, I can read the instructions to you or because he has this nice nice Irish accent. I guess you just want me to copy this example to dot YAML and then make a little bit of a Also make in it works by now. Because we are old, we like make files. I mean, I like just files but I

31:57 guess I'll settle for a make file. So let's see what the source looks like. Yeah. This is where I need to do the secret thing. Right? Yes. It's like big big letters. Please please enter the secret stuff here. No? Alright. I don't know them, but I'm horrible with names. Someone else? They're they're just a a punk rock band, but on one of their earlier albums, they had they actually had an intermission halfway through the album. Was just a little tune. And I keep wanting to use it on the stream, but then I know YouTube were gonna ban me.

32:44 I actually tend to buy license rates to it. But apparently that's a major pain in the ass. Okay. I have edit this file. So what I'm not gonna do is open it. What I will do is open it here so we can just kinda see what this configuration looks like. So we just need to insert this token which I've done. We've got some intents. This is just permissions against Discord. Yep. And we just have some sort of internal reference to tell which type of configuration this is. Is nice and simple. Nothing magic. We can move on. Is that right? We

32:50 Exploring Tremor YAML Configuration

33:16 want to kill this eventually so the YAML will go away. That's the one part that isn't fully that is configuration of language. That's on our to do list, but for now, yes, it's YAML. So is that a Tremor configuration? Yes. Right. Okay. That is part of it. That sets up the you remember this source arrow and the Cookie Monster? This is the source arrow. It tells it where data comes from. It's also the sync arrow because we need to talk back to disco. But that is like, here is where data is happening. This is the connector

33:45 to the Two years ago, Tremor had a lot of YAML and not a lot of programming language. Now it has a lot of programming languages. There's not not a lot left of the YAML. The YAML will continue to disappear because it it it provides a horrible developer and operational experience. We started to move towards much more structured, programming because you get a better class of error back to the user. For example, who likes an error from, say, a logging system implemented on the JVM that gives you 99 lines of stock trace for one, oh, you you forgot a comma.

34:22 You've also just, oh, you forgot a comma. Did you mean to add a comma? A comma would be nice here. And we call that hygienic errors. I think that's something about the Rust compiler and and development experience that rubbed off on us. Yep. And we robin hooded into the project, so to speak. I'm pretty sure there are rules when you're a CNCF project that you're not allowed to say bad things about YAML. I'm just I'm just saying you gotta be careful there. Like, you you you Oh, we love YAML and other people's products. Right. Okay. So let's make fail. We have

34:55 run it, and then we can see kind of what it's doing. Did I just run did I just run make run now? Is that is that all I understand? That will start. Yes. Then you can Is it gonna print anything to my terminal that we don't want to see? Yeah. It will well, nothing important. It will just print a few things and be like, hey. Oh. Oh, yes. I am a local driver. Yeah. Yes. Yeah. Because, you know, this is you're sending me secrets on Discord. I I unzipped a binary from an a random person on the Internet

35:21 and installed it to machine. This is not a good example of how to computer, by the way. Just saying. That should Well, could have used an environment. It's nice in the world to see such trust in an individual. Okay. I think it's running. Yep. Oh. No. No. No. No. Ah. We need to set the module path. And we need to give you the modules. This is this is the wonderful thing about live demos. We have no idea what we are doing. So, that would or someone wants to explain it while I copy the files. So Tremor has is now building a a

35:30 Running the Bot & Initial Troubleshooting

36:00 growing library of reusable code, and it also has a a set of links into built in functions in the Tremor runtime and Rust, that we provide as intrinsics that so they can be referenced in the interpreted code. Binary that Hunt sent you, we omitted all of that good stuff. I believe he's probably sending you one now. I'm I literally send it right now. So so this is not a problem on Linux where we have packages or on the docker image because that's included. We are not quite there on linen on OSX. K. So I I need to unzip this

36:41 here. Do I need to unzip it somewhere else? Here. Just here. And it's all Tremor files, so there's the viruses in there are all written in Tremor script. So we want to now set a Tremor pass in the environment. So Tremor on the environment. Makefile takes care of that. But yeah. Generally, yes. Okay. Awesome. There you go. Tremor's running. Invalid authentication. Please check the token. Did I break it? Shouldn't have. I I I would say let's look at the token, but also let's not look at the token. Let me Let's think about the token and

37:24 guess what state I might be in. Oh, yes. Because I'm the no. This is this is me being an idiot. Okay. I don't discard often enough. I gave you the public key, not the secret key. Well, nobody wants the public key. Yes. Well, it it was easier to copy. There's a big big public key button and I copy It was shorter was the logic at the time. It was actually longer, but I couldn't see the secret one. It was hidden because it's a secret. Oh, it failed again. That shouldn't. What does it say? Invalid authentication.

38:00 Was it a good one? Let me just Double check. Yes. That's the one. Yeah. I think I copy and pasted that. Okay. My copy and pasting is is not flawless, but it's decent. Yep. That perhaps my copy and pasting is off. Oh, yes. No. It's not. They they do there are three different things. Uh-huh. Is the secret thing, then there's a token, then there is the other secret. Yeah. Let's try the secret secret thing. Let's let's try the next secret thing. There there are a lot of stuffs to copy. I probably should have read the read me we wrote about

38:41 how to get this data. Sometimes it helps. Can you send me your date of birth, your mother's maiden name, and your first school please as well while we're at Actually, I'll I'll just send you the configuration I tested Okay. It looks like it's working though anyway. Yes. Perfect. We've injected the third secret. We've got what appears to be a working Tremor. Secret. Yes. A working secret. Okay. So on our Discord server, I already created something called bot fun, so we don't spam general. The way the way the Tremor bot works is if you want to look at the

39:18 there's a file called plugins dot yaml, I believe. No. It's called discord dot yaml because obviously. And in there, have different plugins. Hold on. Because I used the temporary directory, I need to get my split into it. I'm I'm glad I'm glad I'm not the only one making his own life miserable. Yes. Okay. What do we wanna see here? There's a if you look into config, there's a zero one Discord YAML. Presenting? So this is separated from the from the config because there are no secrets in here. And you remember the cookie monster in the

39:40 Discord Bot Pipelines & Plugins (YAML)

40:02 image? Yes. The pipeline. There are multiple of those since you are not limited to a single one. And what we have is we have made each plug in a pipeline that are all connected here. So if you look, for example, at line 13 and following, which doesn't mean that I say 13 because you have line numbers turned off. But I'm cheating. I have the file open with line numbers. Yeah. You see we have the pipeline no bot in there Mhmm. And and the output of that, and we connect it to the input of the setup pipeline, the roles pipeline,

40:41 the code of conduct pipeline, the inclusivity pipeline, the welcome pipeline, the jokes pipeline. Yep. Got it. And so those are all the plug ins we have right now. There is the most interesting one to look at is probably the inclusivity one. That is something we Robinhooded from the CNCF discord. Well, we we are we are just stealing stuff from all around. Isn't that the CNCF Slack? Yeah. The Slack. Yes. Yes. The CNCF Slack. So so something we really like is if if someone said, hey, guys, it was like, well, do you mean something else? Perhaps.

41:23 And we wanted that on the Discord server. So we have a a plugin for that that does exactly that. Okay. So how do we do something with this Discord bot? Right now, I'm assuming this Tremor bot is using Tremor and it is it listening to Discord right now? So if we start to type things, is it gonna come into our pipeline and allow us to to transform? Would you believe there are literally people trying to interact with your bot right now you make changes? The demo guys are horrible too. Why why the hell, I guess

41:57 that's not swearing in the bad way, is it not working in the bot function? Oh, no. I am. Nobody knows. Is it? I am Oh, maybe we can fix it during the live session. Yes. We can fix it during the live session. Also, can just use general and there, at least Tremor bot replies, which is a good start. Okay. So if we type here guys, Tremor bot is gonna learn how to type. I bet it doesn't work with it. Oh, we we we should teach Tremor bots, dovish, and maybe Derek as well. Just why not?

42:33 Alright. Alright. Yep. Trips. Okay. Let's let's throw this guy in. There is this. There is the Tremor death bot. Oh, is he in bot fun now? Yeah. Yeah. Okay. There we go. That's you. Yo. This is this is your bot. Okay. Cool. That makes sense. I get it. So this is all powered with Tremor. Yeah. All of that is and you can look at the code for that to get an idea how to do something like that. Is that what's inside of the lib directory? So that is inside the config and the lib directory. There's config plugins, which has the

43:16 pipelines, and the lib directory is additional helper functions, basically. Okay. So let's look at this. What what is it about trickle? Yeah. Well, our naming sort of has a joke running through us. So because it's Tremor Everything is so good. All the the languages that we have will begin with Steve. So Trickle is called Trickle from the Days of Thunder movie because he feels the need for speed. The query language is all about processing speed. So it's sort of a meta joke beginning with his team. So is the language called Trekle? Yes. Yeah. Right. Got it. Perfect. So this

43:25 Introduction to the Trickle Language

44:00 If you think about it, it makes sense. Most query languages in this space for real time are filtering. So they're dropping a lot of data on the floor. So you're taking a fire hose and you're sending a trickle downstream. Okay. Got it. So let's see if we can work this out. Let's see how intuitive this language is. Yep. I'll I'll let you go over this before we explain it. Thanks. Okay. So we've got a select. So right away I'm thinking this is the stream processing function. I'm assuming this is a pattern match on a field called message.

44:34 We're trying to extract channel ID and then I'm not sure what's going on here. Oh, this is the response. Okay. So Yeah. Yeah. We've got a select on a message with a channel ID and then we send and a response to the original ID, the contents in, hey, and and I've got some sort of interpolation syntax here to inject things in and then we got the from. So the from is, okay, this is where the actual match is happening and we're looking for where the content contains the word guys. Interesting use of stars here, I guess.

45:09 It's a block. Yeah. And then the default is to nothing. Easy. I think I understood that relatively well. Tell me what I got wrong. Yeah. You you got it. So the it's it's sort of like a SQL language. So Tremor is the inverse of a database. In a database, you store the data and you run ad hoc queries. In an event processing system, you optimize the query and you throw ad hoc data through it. So this is a is SQL like, but it's not ANSI compliant SQL, because, like, who it's it's twenty twenty one. Who uses columnar,

45:46 data in modern systems? Most people are using structured data like JSON, so we just threw away the columns and tables and we're getting with the the old structured program. So we have nested data structures and it it maps the business domain events much more closely. So it makes it more intuitive. Nice. Okay. Let's take a look at one more of these, and then we'll maybe see if we can modify the behavior in some way. Which one of these is your favorite? Welcome is fairly easy. Jokes is absolutely stupid, and roles is really complex. Okay. So I guess we'll just look at

46:24 them all then. We've got we've got a select here, same again. We've got a send message. This syntax here, are we trying to alright. Okay. So we're looking for some sort of event type where the member edition is is true. Okay. Got it. Okay. I'm I'm curious. This is just running straight up Tremor. Right? Is there a flag of the I could see the events coming in. Can I print them to standard out so I could see what we're You absolutely can? That is a pipeline. If you look at the config where at the zero one Discord one

46:40 Debugging: Viewing Incoming Events

47:02 Yep. Yeah. We'll just modify that, and we'll add a one of the built in pipelines will allow you to redirect the standard output or one of the built in connectors. We'll just configure that, and you'll see the events streaming. Because I imagine if I wanna start adopting Tremor in my infrastructure for, you know think a simple use case up front is gonna be logs and metrics for most people. I probably want to know what the message is coming in from the sources look like before I started to do the transformation. So Yes. We actually have thought about that.

47:31 Line nine. You see there's this debugging Yep. In the pipeline pass through, which is a standard pipeline that always exists, which just passes through data. And that means everything that comes in, we just pass on there. And in line eleven and twelve, we pass this data to standard out. So while you're on commenting, you've just implemented that. Yeah. Does it gonna do that content re It It doesn't auto reload. No. You got to restart it. Alright. And not a noise feature. Please raise an announcement request. Yeah. If I can send a sec up to I'm quite a happy person. So

48:13 let's get some blank lines in here. Let's someone's already typing. I'll just wait. Oh, there we go. Cool. So we can see coming in here, we've got the ID, we've got attachments, we've got the offer avatar. There's that channel ID that we were plucking out and the trickle script. We've got the content. Guess that's We're not gonna be able to see a member edition. That's probably gonna be a little bit more difficult to trigger on the livestream. But Yes. Someone will have to join. Well, we just got to need to make people join the Tremor server if

48:46 they want to see it in here. Okay. I'm getting it. It's it's starting to make sense to me now. It seems like Oh, look. I got not bad. You're onto a live Yes. On the on the stream. Yes. So is it fair to say that this zero one config, is this the is this the most important part of this configuration? This is where I have It's plumbing effectively. It's and this is where we haven't yet gotten rid of the YAML. And this makes it a bit hard to read. There is a RFC on our RFC

49:23 sites about implementing Troy. Anoop? That was the name, wasn't it? Yes. Yeah. Troy. Yeah. Yes. Troy. Which which hopefully will replace the lovely YAML and with something not as readable and helpful for developers, but closer to the other language. I I I have actually, I'm not familiar with Troy. I have no idea what that Neither option really is doesn't exist yet. Tremor deploy. So it's the Tremor deployment language. One of the limitations with YAML, is that we don't have the kind of tooling we would like to give the kind of hygienic errors that we would like,

50:00 Future: Troy (Deployment Language)

50:03 through it as an interface. It becomes cumbersome to describe. With the language that understands a lot of the primitives in the, scripting language, we could reuse a lot of that logic and make it a lot more flexible. So for someone who already knows the scripting and query language, it would be more intuitive not to have to jump into a, a format like YAML and to just use the tools they have and build on top of them. Okay. Got it. I'm really glad I never lied and just said, yeah. Try and put over that. It's a really cool project. And then

50:37 That would be cool then then you can help us implement if you know it already. You're still ahead of us. Awesome. Exactly. I'll I'll come and comment on an RSE for sure. Was someone else gonna talk there? Sorry. I I'm sure I heard another voice. Oh, yes. I was just gonna comment on the Tremor CLI actually has a facility to run your little scripts, like, on its own too. So you you don't really have to start the whole thing like how you did for the Discord case. So you can actually test, like, individual logic by itself, and that's

50:52 Tremor CLI & Testing Capabilities

51:08 actually a really killer feature. So that allows us to, like, test logic in isolation, and you can also actually do, like, kind of, like, unit testing around it or the Tremor CLI has a test feature. You can also write, like, full blown, like, integration test too. So the production logic that you have running in your, like, servers, you can actually, like, validate beforehand, and you can iterate on it really fast. So that's like a that was a huge event. That's a huge advantage compared to the solutions that Tremor was replacing it with things like Logstash really don't allow you to

51:39 do that that way. So that's why, like, Tremor really soon for us, and it's, invaluable. But yeah. Okay. There's lots of tools in that Tremor command line. You can Yeah. It's got five benchmarking or five testing systems including the benchmark environment we use. So you can literally write a directory, fill it full of integration tests, unit tests, benchmarks, and share them with us, and we can run them. So that makes it easy to communicate failures and communicate odd behaviors. Everything through to if you want to debug what the interpreter inside Tremor sees, there's a whole heap of debugging tools that give you

52:20 syntax highlighted, lexical forms, the the tree the the syntax tree that Tremor generates, all syntax highlighted, including syntax highlighted source on the command line. So you don't even have to spin up an IDE if you don't want to. Okay. So I just wanna make sure I understood that correctly. What we're seeing with the Tremor CLI is actually I can have maybe, you know, a day, a week, a month, whatever of JSON or CSV data or something that then I can run against the test that my transcripts are doing, what I actually expect them to do. Yeah. Is that

52:54 something where Tremor provides some sort of assertion framework, or would I bring in something like conf test on top of that to do it? So, yes, if you have a look at the integration tests in the Tremor runtime repo, in Tremor Cloud, you'll see a test folder. In the test folder, you'll see a number of different, frameworks. We do everything from verifying our API works with curl through to running integration tests where Tremor is running Docker, which is running Elastic or Tremor is running Tremor. So you can have multiple Tremor's running in integration tests, and it does

53:30 expect some expect style assertions on that. So, you're in, tell you wanna go to TremorCly tests rather than the, the built in tests. They're more like unit tests. They are Rust tests. So TremorCry tests. Pick. Pick one doesn't matter. So we have a unit testing framework in the scripting language that, is like JUnit or any of the other unit style testing frameworks. So that works in TremorScript. So with the standard library, you can look at the standard library tests. There's only about 900 of them, so it's a long file. Well, this is quite short. Unless I'm in

54:12 the wrong file. You've got one with the other file. It's also a good file. Quite long. Where is the long easier. It's easier to use the short file. Okay. So what I can see here is we define some sort of test suite. We give it a name. The suite itself has multiple tests. That makes sense. I like the fact there's tagging here that tells me that I could probably run a subset of my tests across multiple files. Yes. Yes. We can. Which I'm always excited by. It's the simple things that keep me happy. And then this individual test itself, which is

54:48 also got a name and then we have a test assertion that the host name is my system's host name. Okay. I think that makes sense. Is Tremor doing any sort of enrichment in those messages as they come into my system? The which yes, but only slightly. The I think it does well, adds a few things. It adds a timestamp when the message arrived. We also use the timestamp for sneaky stuff like random number generators to guarantee determinism. So if you would replay the same data with the same timestamps, you get the same randomness, which is really powerful if you want to

55:00 Event Enrichment & Metadata

55:29 test your system and figure out why the hell did this message not arrive if I configure 50% drop rate. Because, well, if you comp replay it with the same same time stamps, off you go, you get the same results. Then it adds some metadata depending on where the data comes from. For example, if it's a HTTP request, you get the headers. You get the HTTP, the other side, the client information, all that. So there is some enrichment happening, and that is usually happening in the metadata. So if you look at an event, there's body, the body, the event data, and the

56:04 metadata, which is then the data that tells you something about the event but isn't the event, if that makes sense. A little bit. So in the in the scripting language, we didn't see it in the scripts that we were using. But if you use the dollar sign in the scripting or query language, that means you're referencing a meta information. Okay. That can be generated by the connectors into the system. So, like, Discord might generate some meta information. And some of the plug ins to the query language, might actually also generate or consume metadata to perform their particular functions.

56:43 So it's, Tremor also has an other form of metadata that isn't exposed in the languages that it uses for things like acknowledgments, guaranteed delivery, circuit breakers, that are running invisible to the user unless you're using an operator designed for the user to interact with those subsystems. So Okay. So let's assuming I'm running Tremor on a bunch of Linux machines on Equinix Metal. Equinix Metal has a metadata service where I can fetch some information from. Is that something that I would do before I run Tremor and populate a config file with key value pairs or something? Is that something I would

57:05 Interacting with External Metadata

57:21 write a plugin for Tremor for to scrape or like how how would I interact with this in that? That's a great question. You would write that in Tremor, deploy it with Tremor, and you could do that live. I you could literally have a stream of changes coming through or pull the metadata system periodically using the metronome or we have a a Chrome operator called Chrono. So once per day, you could download new metadata or every ten seconds. I mean, you can you can literally configure this or code this depending on how you view the scripting language as you like.

57:55 TremorScript vs. Trickle (Language Layers)

57:57 Okay. Can we clarify the difference? Config file and load it. That's that's okay too. So can we clarify the difference? Is there a Tremor language and a Trickle language? Are those different things? Yes and no. So the the Trickle language is is superset of the Tremor language. So you there is a scripting language, and you can embed the scripts inside the query language. So you and then you will have the Troy language which you can embed the query language. It's it's turtles all the way down just with silly t based names. It's yes. Turtles. They start with t. Damn it.

58:32 We need a higher level language called turtle now. It's It's turtles all the way down. So Now the lowest is an old and turtle. Yeah. The lowest level is TremorScript, the scripting language. That gets wrapped by the query language, which lets you describe graphs. Eventually, this gets wrapped. This makes a lot of sense. There are a lot of programming languages that have attempts attempted to build low code or no code flow based languages. And the problem is when the logic gets very, very complicated, it's very difficult to write imperative or procedural logic in flow languages.

59:11 Just like it's almost impossible to write flow based logic in imperative languages, like, look at callback hell in anything from Node. Js to, the modern JVM. Callback hell is indicative of this is a flow oriented environment or system. You're trying to describe a stream of connected things in something not designed around those structural concerns that are flow oriented. And you get exactly the same with business logic or mathematical logic. It doesn't want to be represented as a graph. It wants to be represented in something textual and something more familiar. So we've just made the choice of let's

59:50 use the query language that's flow structured for flow oriented things, and let's plug in something a lot more relevant, to the business logic for what people want to do with that data as it's flowing through. And today, that is a very ETL oriented language, designed around slicing and dicing and processing business like nest data, you know, JSON, which is 99% of JSONS. Okay. I'm gonna throw something out there. I don't know if that deviates too far from your plan of what we were gonna Go ahead throw stuff out. But we're in a position right now where

1:00:20 Demo: Adding a New Source (UDP/TCP)

1:00:30 we have an applicator we did until I think. I killed Until you stop. Yeah. So we can run Tremor, but we can add new sources to this, right? Yeah. And those can interact with the other parts of the pipeline that already exist right now, which is the ability to write to your Discord channel. Yeah. I can imagine a world where I wanna be able to be you know, maybe I want to monitor the CPU and memory on my machine, and if it reaches a certain threshold, send a message to a Discord channel. Is that something we can throw together now or

1:00:50 Why Not Tail Files? (Recommending Vector)

1:00:59 is that something that yes. As long as you have. Is Telegraph the collector? I believe, yes. Isn't it? It is the collector. Yeah. And so you can hook up Telegraph to Tremor to Discord. No problems. Well, what about if we keep it really simple? Can we can we monitor a file with Tremor and if I just echo something into the end of that file, can it just relay onto Tremor does not tail files. You could use UDP, WebSockets or TCP for that. Okay. Netcat. So I can run Telegraph tomorrow to that file and write it to UDP socket and

1:01:33 then we can Or vector. Well, or you can just use Or vector and Netcat. Yes. Oh, yeah. We could just use Netcat. Let's try and keep it simple. Let's not bring in too many tools. So how okay. What's well, how do we do that? How do we how do we expose Tremor to something on my machine that I can send arbitrary messages to your Discord channel? Well, the first thing would be adding a source. Let's add a source. Okay. Let's add a source. Why do we why do we Hans, why do we add Blaster and just

1:01:45 Configuring UDP/TCP Source & Binding (Live Coding)

1:02:00 configure it to send slash e t c passwords through to this one? How about not? Okay. So let how do I do this? Well, let's look at the documentation. So we we'll flag our documentation. Docs. Come on. I've got I've got Well, I can't ever remember what I'm doing. Literally, I can't remember what I am doing. I'm always looking at the documentation. And if you go there into artifacts and then on ramps, you can see what data what sources Tremor can get data from. Okay. So we've identified some vocabulary there that I wasn't particularly familiar with.

1:02:41 I don't see it in the config, but I just ignored it for now. So on ramps are our sources and off ramps are outputs. Yes. Our syncs, which is our fault that we haven't gone through renaming them all to source and syncs. But basically, yes. On, it goes there. Off, it goes out. Okay. So I see a file on ramp here. Can we not just use that? Well, we'll only read a file once. Got it. Okay. So the the story behind this is tailing a file is actually a really complicated thing. Right? It is it sounds easy,

1:03:16 but it isn't. And there are the great folks over at well, now it is where are they now? David. David. Right. Yes. We have project called Vector. It's really good at log tailing, and we don't like spending our time doing log tailing. So we just recommend, hey. Folk use Vector. It's awesome. It connects to Tremor. Okay. So We use it as a sidecar whenever we need to do log tailing. We just use that other project. It's really good at it. So you know? Okay. Well, I mean, I I'm happy just to click control c and run Tremor again

1:03:53 after modifying this file because we just want I just want to show a demo of of of this. I was like, actually, adding a new source and then having it work with one of the syncs. I think the easiest way is do you ever use web Socat? I've used Socat before. Yes. Well, the the yeah. UDP. The simple UDP on ramp which just uses this as a message. Okay. So let's click on UDP. I'm gonna just grab this config here. Yep. And we're gonna drop that in to our config o one discord. Is that right?

1:04:30 Yes. That was said with some degree of confidence. Because there is the secret there is the secret file and I wanted to make sure you don't open it. So this is gonna run a UDP server on local host port 9,000 of which I can send JSON to and it'll do something with it. Nearly. Oh, probably want to use the line's preprocessor. You what you did is put it under binding, but it is an on ramp. And we are sadly in YAML and oh, sorry. In the wonderful world of YAML, where you got to put it under

1:05:07 the on ramp top level thing. Just create a section. Yeah. The the section was on ramps. That's why I hesitated. It's in this hidden file, but you can define one here as well. Alright. Okay. So can I just do on ramps? Yep. Yeah. You can. Okay. Singular. It's on them. Got it. Yeah. Okay. So that's You also could have been really lazy and copied it from the example completely because it's already in there. Okay. So I can save this now and I guess as a UDP socket, we can send a message. You haven't looked at hooked

1:05:43 it up to anything yet. That's a good point. Okay. So I I guess this is what the bindings are. Do we have to buy something from our on ramps to our from our sources to our sync? Okay. Yeah. So is that just a case of saying ID and I'll just say Rawkode. I'm assuming that's arbitrary. We wanna create a Yep. I'm just gonna keep making this up until someone tells me I need to do something else. Well, you're now connecting the Discord on the RAM, not the UDP one. Okay. So I need to use the ID

1:06:14 from above, which I called UDP. Okay. Is that arbitrary as well or is that actually connected to the oh, the type is this one? That's connected to the ID that you're Yeah. You could you could call it snot better if you wanted to. Okay. So now that we've got this out link, we tell it which inputs to forward on. Is that right? Which to which input to forward it. Yes. Which the pass through one is the simplest one. Okay. Let me just copy that so I don't mess that up. Exactly. Be be lazy and just copy the debugging one.

1:06:48 Is that it? Nearly. Nearly? What what The last part is the mapping at the very bottom. Since you made a new binding, you have to tell Tremor to load it. Okay. So binding UDP You call it Rawkode. Rawkode. No. Rawkode on the you call the binding Rawkode. When you put ID in the binding, Rawkode slash o one. I wanna see what I did there. Oh, yeah. Okay. So this is this pending here. Yeah. Okay. So binding Rawkode o one, and is that just I I don't know what the first. That is. That works. That'll work.

1:07:28 And now you should see UDP messages going there to your standard out. So I can just do a make run and it's not gonna blow up. Right? Well, we'll see. Okay. So now we have I don't trust live coding anymore. After the start today, it didn't. Okay. So I wonder if we could to an off run. Yeah. I think it just went to the pipeline only, right, so far? It it it went to the pipeline. It's let's connect. That's a different binding. It's a different binding, so it will not Oh, wow. This is this now I'm learning

1:08:10 something. That works? Bash provides a virtual networking stack, and I can't I don't know if this will is that the right port number? It's 9,000. Yep. It's 9,000. Yeah. I think so. So that's not quite right. Definitely something like this. Oh, I'm not running bash. That could could be a reason. Yeah. I used to use this all the time, but there's like a virtual networking stack and I'm sure I'm gonna get it slightly wrong and need to Google it. But it's something like this. Oh, it didn't work. Alright. You could use Netcat. I know, but now that I now now

1:08:46 I feel I need to teach you this thing. Okay. That's fair. That we have learned something? We're gonna have to upgrade the bash from the ZSH. Bash. I'm not going back to bash. I am sorry. I like z s h to bash. But it's an upgrade. It's it's UDP capable with virtual UDP. What's not to like? So I used to use this in my health checks and my Docker containers. Dev TCP, that must be it. Dev TCP I'm I'm 50% of all the Solaris users. I I want no bash. I want all my scripts to

1:09:24 be good old shell. There. I was close. It's not net, it's just dev UDP. That should have sent a lot to that UDP channel. Well, yes. But you're told that it's Chase. Oh, string is That's a JSON string. Oh, it's not like. Yeah. Okay. Let's see if that works. Didn't. Okay. So maybe that's just my crappy bastric. So why don't we just try it your way? No. No. Well, then then then it's my fault suddenly. It would be net cat minus u local host 9,000 and then you can write. Pass n c dash u. That.

1:10:09 Nope. Maybe space. Okay. And now I just need the JSON payload. Nope. It wasn't wasn't your UDP. Okay. So there's something wrong with connect to because it's sent to the pass through end. Yes. Let's see what pass through end is connected to in the Discord YAML. Okay. So let me just drop out of that pop open a Discord YAML. So okay. So we got the binding, we've got on ramp UDP, out system pass through. And what's pass through in connected to on our links? It's in Discord. It's connected to standard out. No. It's it's a different one because it's

1:10:51 a should should should be a different one. No. It should be the same. Gonna be zero one. They're all mapped to zero one for I. Okay. So where is in sorry. Yeah. I've pipeline system pass through, I was just connected to all from standard out in. That's not what we want. We wanted to it's going out of Discord. So we wanna connect it to the Discord off ramp. Well, we wanted to first go out to standard out to just test. Alright. Let's do that then. So that should already print on standard out? It should. Yes. Which is annoying that it

1:11:33 doesn't. Think it's when it doesn't. Mine's prepostor maybe and throw in slash n. Doesn't echo put a slash n at the end of it? Could could be. So use the lines preprocessor. It's probably just buffered in UDP going, yeah. Yeah. I don't know how to send it yet. Is it finished? Okay. So what do we want to try? The lines speed processor. I could be wrong. Could be wrong. On ramp UDP is perfectly fine. Links is on ramp UDP. Yeah. But preprocessor, is it it's a good idea. So in your UDP configuration Yep. At the same level as config

1:12:20 or just below codecs, preprocessors, colon, return list or dash sorry. Dash as in list item and Yep. So give that a whirl. Okay. I'm gonna try my bastric again then. Nope. Let's try the Netcat one. Well, the bastric wasn't JSON. Oh, yeah. We could we could make a codec as string if that doesn't work. Codec string is a safe one. Yeah. Do do we need to specify, like, the the route, the pipeline to the upfront explicitly in in the binding? Because it's part of different binding. Right? I think so. I think so. Yeah. I think that's why it's not working.

1:13:14 If we had made it yeah. If we had reused it on an existing binding, then it should have worked. But since this is a new binding called Rawkode, the the pass through pipeline isn't connected to anything in that binding. So I think we need to, like, explicitly copy the off ramp part from the second phase to the first phase. So maybe that will work worth trying. Yeah. I hope that made sense, Rawkode. But, yeah, I I can write you. And this is why we want to get rid of this YAML. Oh, yes. What do we have to copy into where?

1:13:49 So copy the pass through, out The scroll down. To Discord from the Discord binding into the Rawkode binding so that you have a full copy of the flow that you want in Rawkode. Think of Rawkode as a separate application. Not quite a one to one mapping, but for live coding, that'll do. Okay. So so we want to copy the pasture from here. Yeah. So copy the below one, the out. So the output from the pass through pipeline, we want to pass in To yeah. To the old front. Yeah. For standard out. So this line, we wanna copy

1:14:34 and add to Rawkode. Okay. So those two lines. Right? Yeah. I think it's one line. Two lines. Or is it one or two? Who knows? Could be one. Should be two lines if there's no wrapping setup. Like that? Yes. Yep. Like that. Yep. Still don't know if it's one or two lines, but that works. I should have changed the codec to string as well, but oh, well. Oh, one thing at a time. That's string, son, not Jason. Yes. I don't think that did that for me there. I think I've broken it. Missing a colon between name and value.

1:15:22 You know what? I'm changing this one thing. Right? Okay. So you're telling me I can just string as a codec and it doesn't care what I pass through. Is that right? Yeah. Exactly. Yeah. String will just do everything you want. Let's remove as as much things that can go wrong in my typing as possible. So hello. I'll go. No. That didn't work. That is super So we have a UDP on ramp. And we I mean, although should we set it because if it's UDP, we're not actually gonna know if it fails because there's no response. Should we use TCP instead? Yeah. Give

1:15:59 it a shot. Give give it a shot. And you can keep it named UDP. It won't if you change the ID, you don't need to change the IP. Okay. It'll be confusing later when you've forgotten why you did it. Okay. So there's there's some buffering or something. But then I I need to know if this works. So See, there So now we can change it back to Jason if you like, or we can configure it and send it out to Discord. I think it is configured to go to Discord, does it not? It's configured to go to system out.

1:16:37 You may need to copy that as well. Okay. So let's add our our final mapping here just to Or we could just move the the The standard out one? Our TCP UDP. The second line of links to our discord and get rid of the binding and the mapping. Move it all into the same discord binding. Keep it simple. Get rid of the other lines. Yeah. Because we just need the UDP, and that will union into the password. But can I just do that and remove this? That removes the rest of the block functionality. You can do that.

1:17:23 But that will ring it to Discord. Right? It will Discord will tell you this is not a message I understand unless you give it the correct format. Structural leads. Which is where all the selects are in. But we can just copy one of those and we can make one and just add another pipeline. Okay. I'll let you just gave me. What what what do we do to make this work? Let's create a pipeline and just give it a give it a random name. Just just pick a name you want for your pipeline. Okay. And that's the stuff that's in the

1:17:45 Creating a Custom Trickle Pipeline

1:17:57 lib directory. Is that right? No. That's in the plugins directory. Config. Oh, contact plugin. Yeah. Call it the Rawkode plugin. Okay. So we're gonna go to config plugin. We're gonna copy the simplest one with the welcome, wasn't it? It's the welcome. Yes. Okay. And we'll call this Rawkode dot trickle. Okay. So now we can do a select. Now we're not reading messages from this card, so this is gonna be different. Right? And you know, so this part of the select creates the message that's outgoing. It's like like, okay. Think about select. If you select host name from table,

1:18:35 then what is between select and from is the data that goes out. Yeah. So just get rid of the content value and replace it with event, which represents what you've passed in and what is sent through TCP. So hello is what you typed the last time. So that would be the string. Was the text? For event, and it's hash rather than dollar for interpolation. There you go. A smiley. There. I'm happy. So if you put a small e for event, it's case sensitive with the variables. There we go. Perfect. One one last thing, the channel ID, since that

1:19:15 isn't the welcome channel, and this is Discord specific, that would be that number. Yes. Did you send it over to Yes. I sent it over Discord. You don't want that all to go to the welcome channel. That's where we have the bot spam channel. Yeah. I just have to move that to the other screen so that we can flash the Goddamn your screens. The secret. Okay. So let's get back. So I copied that ID and then like so. And number, not string. Just just the number. Got it. I believe. I believe to believe. We'll find

1:19:58 out. We'll find out in a second. Okay. So now we've got our match here, and I guess the well, what do we have to change care about that. We're just just true. Just yeah. Just remove the whole where. Else. Just remove the whole thing because we don't care. Yep. Bang it on. Send it on. Everything is fine. This will actually not work because event isn't true. Just kill kill the whole line, make it default true. Got it. This is this is the most powerful message statement ever. Yeah. Is that all we need in our trickle?

1:20:36 Yeah. That's it. I believe it is. That's an entire program right there. Okay. So now how do we connect this to Rawkode. Trickle from our config? So if you the the way the bot works, it it loads all the plugins. So you now have a plugin on Rawkode effectively. Yeah. You don't need to put it in there. Rawkode will be your plugin. So you just make a new line and you connect the outputs of Basically, instead of systems, standard out. First link. That's alright. Can send it to standard out and add a new list element that sends it

1:21:16 to slash pipeline slash Rawkode, squiggly eyes, squiggly slash in, as well as standard apps. So you'll see it on the console, and you'll also get it sent. So online number one up from where you are. It goes into the slides. If you go to the end of this line and make a new list entry Yep. It's called quote slash pipeline because it's a it's a pipeline or flow slash Rawkode because it's Rawkode dot trickle that takes the file name base Yep. And it uses that. And slash, because it's this instance, you're gonna go squiggly eye squiggly.

1:21:59 And slash in because it's going to the standard input stream on that file. So we didn't cover a lot of the details. You can configure your own streams and stuff, but this will work. Yeah. So it's all gonna be Well, we also need to put the output somewhere. We need the output. Yeah. It'll almost work. On Now we need to do something know the line. No. This would be the UDP output. So you change the UDP output to go to Discord output. That should go to the system pipeline path throughout. No. Line numbers. Line number plus one.

1:22:10 Connecting Custom Pipeline to Discord Output

1:22:36 Yes. What's the option? Increment the line numbers. I forgot. Did that turn out? N u. N u. That's what it is. There we go. Yeah. And then after line number 18, and now we have the Rawkode pipeline, and we have slash pipeline Rawkode I out. So we connect the outside of that. So that's the the result of your select statement, which is your message event into Discord. Okay. So slash pipeline slash Rawkode. Slash squiggly eyes squiggly slash out. Squiggly eyes squiggly. That'd be a good pirate name. Oh, yes. Squiggly eyes squiggly. Squiggly eyes squiggly. And we connect that to the

1:23:25 and now it gets confusing. In ramp, on ramp, Discord, I squiggly, I out. So we we loop back effectively. Okay. That's actually making much more sense to me now that we've walked through this. So that's good. Just seems to be a list. It's it's easier to type when you're not trying to say it at the same time across the Internet. Yes. Alright. I'm confident. No. I'm not. I'm in the wrong direction. There you go. Okay. Discord send message to LAR. Yeah. Because it it's it's trying to send, I think, this message as because we connected to this standards.

1:24:06 Oh, you could the past yes. It will it Did we make an infinite loop? Really? I think if you UDP to it, you should see it. TCP to it. Sorry. That's okay. Into that port. Let's have some fun. Nice. Tada. Well, we are we are getting some stuff. The debug no. But we are getting well, we are getting the output from the debug. You have the debug still just enabled, so it sends all the debug messages to Discord. Since I'm not sure there's something sensitive in there, I will delete this. Yeah. I'm pleased to. So if I just take off the standard

1:24:15 Testing the End-to-End Flow

1:24:45 oh, this one. Right? If you just go to line plus two plus three, and instead of connecting 17, instead of connecting this to pass through, just connect that to your Rawkode pipeline without the system. System notates a pipeline that is always there. You don't have to define the pass through pipeline because it always exists. Two colon colon slash. Yes. And there you go. Yeah. I didn't even notice when we typed in system colon colon Rawkode. Where did we add that Rawkode sub module to the standard Oh. System. Still some stuff there. You know what? I'm

1:25:31 fine if you just delete that. I I got I we did what I wanted to see and I'm very chuffed with that. But I love that. So we have the ability with Tremor to add our sources and basically build all of these workflows or pipelines up that allows us to move those messages about the filtering on them and what's the other stuff. That I think is very very cool. If you if you would look at the script, the Rawkode pipeline you wrote, this could be if it were, like, influx data, you could say if CPU users

1:25:40 Demo Recap & Alerting Use Case

1:26:04 is over 15%, send the message, oh my god. It's burning. Yeah. So you could use this for alerting into Discord or Slack or whatever. Yeah. And we're just doing that in this this match statement down here. Like, we can have the different cases an alarm if it goes over a certain value. And if you've seen this alarm trigger before, don't alarm continuously. Only alarm once for the given condition. So there is a state mechanism in Tremor, so you could, cash this in the state mechanism. And if the alarm is set, don't retrigger the alarm if it's already going

1:26:40 off. It's annoying. And then when it's recovered at some point later, maybe send it again. But you can Tremor is quite flexible in terms of its mechanism. There's a hindsight of the mechanism recently that provides a a key value storage, as, essentially sources and syncs. So you can you can store persistent key value information now and and just make it available like it's a service into that logic. Makes it really easy to build stateful apps just by programming little select statements. Yep. Nice. Awesome. So we're we're pretty much approaching an hour and a half now, which is our our

1:27:25 allocated time. Is there anything that we haven't shown that you wanted to show before we finish up today? I think it's actually pretty nice what we actually, other than the stuff that didn't work. Everything worked. This is what you're talking about. Yes. Everything worked. It worked perfectly. I think the messages are all gone. Other than Rawkode says help me, that's still there. Okay. So can we just touch on one thing if you if you've all got a few more minutes before we finish? We mentioned that at the start something that really piqued my my interest, and we talked about clustering.

1:27:50 Future: Tremor Clustering (Distributed System)

1:27:58 Can we cover what how that works, the status of that is, and what's coming maybe? I think that is another topic. The stage is yours. Yeah. Sure. So so far, like, as you can see, so you can run, like, Tremor on your individual host, and you can imagine, like, having different Tremor notes running on different hosts. Right? And with something like Kafka, you can kind of try to have, like, a clustered kind of scenario. But, yeah, we have noticed that for operational needs at Wake, it would be really nice if we could deploy these kind of pipelines

1:28:31 in a true, like, distributed system. So imagine, like, nodes Tremor nodes going down, and then your pipeline, like, recovering automatically, like, the the workload is migrated to some other existing node transparently by the cluster, and the operator doesn't have to intervene. So it's not just the the pipeline redundancy. So there are, like, certain logic that you write as part of your, like, clustered application. So think about, like, when you need to throttle, like, incoming logs or, like, metrics over, like, a set of notes, and then you have to really know, like, how many set of notes are involved to evenly throttle

1:29:05 among all the notes. So things like that, we hope to provide by introducing a true clustered system in Tremor. So we are using draft as the consensus mechanism for the backbone of the cluster, and the actual, like, workload for the Tremor nodes, the pipeline stuff that you have been playing around, they'll actually live outside of the draft context. We call them, like, worker nodes, and they will be, like, hosting the the all the processing logic. And whenever a node goes down or off, ultimately, we want to reach to that idle state of the cluster being, like, auto healing. And, ultimately,

1:29:40 those kind of things will also help, like, auto scale the cluster nicely. You you add a new node, and then you really don't have to do much. Right now, there's, like, a lot of operational overhead and a lot of config tweaking when you when you add one. So so that's where, like, that's kind of the motivation behind adding clustering. So imagine, like, Tremor being, like, a true distributed system. And you you will still have the current option of running Tremor without without the distributed context because, honestly, like, the district adding distributed mechanics like that includes, like,

1:30:11 certain workflows, but it it is, like the dis a distributed system can be, like, a lot more painful to manage, right, compared to, like, a totally, like, a stateless and system that we have right now. So there's, like, a trade after, but we hope, like, the for the workflows, it's going to improve. It will be worth it, and the other option will always be available in Tremor anyways. So, yeah, that's kind of the summary for, like, the team we're trying introduce. So it'll be like you'll be able to write a select statement that contains your JSON like

1:30:42 data, but it's distributed now. So you if if you want to start little bits of SQL and little bits of JSON, you can call yourself a distributed systems programmer and get the big bucks, just by running a SQL query. So that's pretty cool. It's cool trickle right there. Yeah. You can you can write, like, the studio applications in Tremor now. So after this is done. Oh, we're gonna have to have another episode. I think that's that's for sure. Until we are there, it will take a while. Yeah. Surprisingly, distributed systems are not implemented overnight. Tell a lot of my So this work

1:31:19 is ongoing. It'll likely be the simple the simple the simpler infrastructure for clustering will get out first in a in a release over the next few months. We are gonna go all the way towards fully automated, fully migrating, full distributed applications on Tremor, probably towards the end of this year before a lot of that comes out because we're we're gonna test it to absolute destruction before we put any of those features into anywhere near a release. I'm assuming Because at the end of the day, we're a production project. So we run very close to production.

1:32:00 And most pretty much every feature in Tremor, even the Discord bot runs in production. So every feature in there is a production feature. So we're we play it safe when it comes to Yep. The logic. Must have been that's coming in the pony rewrite. Right? Tremor to Retina Pony. That's Tremor then. Rawkode, how are you with the pony rewrite of everything? Yes. It The first ninety percent are done. Does Poly do Cindy Jason yet? Yeah. Well, they say I want to know this. Yeah. It it doesn't need to. Doesn't need to. It's slow enough without it.

1:32:20 Conclusion & Thank You

1:32:37 Alright. I'm gonna let you skip back to your day. That was a really interesting. I think we had a lot of fun there. This is definitely the most fun I've had on our stream. So I'm gonna thank you all for for bringing just such a cool demo to play with Discord and make it really visual and interactive for people that are watching and for bringing the cookie monster. I'll never say no to having more cookie monster in the streams. So again, thank you very much. You very much. Really cool technology. I'm looking forward to playing with it more. Have a

1:33:03 great day and I'll speak to you also. Yeah. Thank you very much. Awesome. Thank you for having us. Thank you very much. Bye.