0:03 Introduction and Workshop Overview

0:03 Hello. Welcome to our first workshop. This is the complete guide to InfluxDB two. Today's workshop, we're going to be exploring one of the more challenging parts of InfluxDB two for people that are just not familiar with creating with flux. There are a few challenges and quirks to this brand new equality language that you only learn through guided tuition or plenty of head banging against the wall. So I'm glad you've chosen this approach. Now, this is a workshop. This is the first time that we've done that in this course. And the format will be, there is a

0:48 README file on GitHub where you can read through all the materials and do the exercises. What I'm gonna do today is read through all the materials and do the exercises. Now I'll give you plenty of warning before I move on to the exercises, so I would encourage you to follow along with me as we go through the materials and then don't watch the exercises. If you're watching live, well, you know, yes, please watch the exercises. But if you're not watching live, pause it. Use that as an opportunity to go and test your own logic. Anyone who joins me

1:05 Warnings

1:27 live today, we will try and make this a little bit more interactive than previous sessions via the tutorials and the lectures, but we want this to be fun. So we have our screen shared here. You can find the resources for this course at github dot com slash rockode academy. I have started adding the materials to the courses repository. There is a InfluxDB complete guide, and we are starting on part five today, an introduction to Flux, the workshop. Excuse me. Now, this course is a living course. This course will be continuing to update week after week

2:08 and possibly in the future as new features and things come and change within InfluxDB too. So I can't exactly go back and modify a livestream, so you may see a little bit of variation and change from what is in the readme during today's session, but I will do my best to follow-up with more workshops as required as the material evolves over time. Okay. Let's see. So in order to dive into Flux today, we do need to start collecting some Oh, that's annoying. Well, let's get rid of that. Got to notice that early. This is the watermark. That's what it is.

2:39 Setting up InfluxDB and Data Collection

2:54 Okay. So in order to explore Flux today, we need some data. So we're going to get InfluxDB to running on my machine. We're going to run with this configuration from Telegraph. I'll copy that. We will jump over here and we'll paste it in. As the instructions note, you will need InfluxDB to be running an organization called Rawkode Academy with a bucket called workshop. So I will do that now also, and I'll just make sure that I don't have any preexisting data. So we run InfluxDB, and I'll get another terminal here. And we'll just give that a few seconds

3:46 to run. I haven't worked out why it takes so long on my Mac, but it just seems to be on my Mac. And I'm assuming it's because I'm on the latest beta of Mac OS Monterey. So maybe something weird going on here. Perfect. And nice coffee. Okay. So let's get our InfluxDB cooking. Loco Jose. Close, but no cigar. Loco https. There we go. So we're gonna click get started. My username, I'll pick a password. We want our organization to be Rawkode Academy and our bucket to be workshop. Zooming in on Influx UI is really difficult.

4:50 So maybe we just have to try and make do with this. I'm gonna click configure later. We'll go to data and tokens and we get an admin token created for us, which I will copy. And we will export this as Influx token. We can then run telegraph like so. All right, let's close that. And that's all these instructions here are also saying. So we created our organisation, we named our bucket workshop and we exported our Influx token environment variable, and this config will just work for you. The configuration of Telegraph that we're using here is just configured with a

5:38 high resolution or short two second interval so that we are collecting metrics every two seconds. We wanna see data pretty fast here today. And we've got a flush interval of six seconds. That just means that Telegraph will collect metrics every two seconds. It will hold them in memory, and every six seconds, it will write them to the database. Really, our flush interval would be higher in normal circumstances, and we leverage batching and buffering much more. However, we do want a pretty quick feedback cycle as we query and work with Flux today. Now we are configuring Telegraph with the CPU

6:15 plugging, although I have disabled the total CPU metric just to force us to do that through aggregation functions within Flux. And we'll talk about that as we move through today's workshop. I've then got it configured with all the other system metrics. So we'll be looking at this, this IO, memory, net, processing swap, and system. These are pretty much all the plugins that work across all operating systems don't really require any configuration and get you data pretty fast that is variable and changes. Again, important for what we're working on today with this workshop. Okay. So now we are going to write

6:48 Writing a Flux Query

6:51 our first Flux query. And the markdown here will be great and it is public, anyone can look at it, but only Rawkode Academy members will get to see this course and my wonderful added commentary and flavor. So while there are things on well, there's little snippets of information in this markdown file, I'm gonna be adding on a little bit more and showing you some tips and tricks from my usage of InfluxDB over the years. Okay. So there are two functions that are always required before the flux engine will execute any query against InfluxDB. Excuse me.

7:30 Excuse me. Now, you need a data source. You have to tell it where to read the data from. And secondly, you need a time window. We'll see this as from and range statements. And there are multiple ways to actually execute a query against InfluxDB. So the first one is we can come here to the explorer view where we get this nice user interface, where we select the bucket, we can select the metric that we want, we can select the fields that we care about, and we click submit. And you'll see in the last five minutes,

7:47 Exploring Data with the InfluxDB UI

8:05 we've got some data coming through. This UI is great for exploring when you're not really sure what you want to be querying quite yet, but it's quite restrictive if you wanna do anything a bit more advanced or complicated. Now you can switch from filters to grips and begin to grip by, you know, the values that are available. But really, what we want to do for the majority of the time is to use the query builder. So we click on query builder and we can actually see the flux code itself. So another, the first step here then would be

8:47 use the query builder whenever possible to at least bootstrap and start your query. It's a really great way to get that from range and any filters through a very visual means. And then from here, can start to change all of the query yourself. We're not gonna use this because we're gonna stick to what is available in the workshop. And I'll also show you one more thing in just a moment. So let's copy our from statement. Okay. We'll just copy both of these and then I'll I'll read over the text. There we go. So, what I've done here,

9:05 Basic Flux Functions: From and Range

9:31 and then you'll see that it took a little bit longer, is that we have a from and a range with no predicates, no filters, no nothing, and it does return quite a lot of data. And it's hard to tell, but there is a probably a fair bit. If we click on this view raw data tab here, which is one of the most important tabs, as you start to explore your data, you'll see, yeah, I've got a fair bit of information here. Okay, so what's this data source? The from function is how we select where to load

9:57 From Function

10:02 the data from. When you're creating against InfluxDB two, you can use from with a bucket name and this will fetch data from that bucket. Buckets have retention policies. They're a very good way to determine how long your data should live and use them as a cascading strategy for your data. We're gonna be talking about, done something on Thursday, where we will look at buckets and retention periods in more detail, but not today. The other thing that's worth noting here is that Flux, well as embedded in InfluxDB two has always been built with the intention of creating multiple

10:43 data sources. So there are other functions besides from to read from SQL compatible stores, CSV, HTTP endpoints and others, but we'll only be working on InfluxDB two for this function. Next, we have to use a range statement. An arranged statement has accompanied with this little pipe forward operator, and I've kind of done my best to explain this here as well. So if you've written any Alexa or F sharp, the pipe forward operator should be something you're familiar with. All it means is that we take the output from the first function and we pass it as an input to

10:59 Pipe Forward Operator

11:26 the second function. So this from selects all data from that bucket. We then pass it into this, and this is a range which will actually filter the data to just the bits that we need, which is the last fifteen minutes. Okay. So you always need a from and you always need a range. Alright. Something else that's cool about Flux, and you've already seen that with our first query, is that it has a concept of a duration literal, which just means that durations and times are first class citizens. You'll see here, we do a dash minus

11:49 Duration and Time Literals

12:04 15 ms to see if I need data from now minus 15. We have seconds available to us. We have hours, we have days, that's supposed to be weeks. I'll fix that. So that's a W. We have months, then I see by MO, and we have access to years. So, you know, pretty flexible. And there are sub seconds duration that rolls too, you can do milliseconds and nanoseconds. Time literals work in multiple ways. You can use a no function to get back to the current time. You can use just the year, month and day, or you can use a fill ISO

12:30 Time

12:41 8,601 time like so. Now, something that is implicit in the query that we have used so far, is there was no yield here. And InfluxDB actually adds this yield to your query automatically for you. So you can always imagine that it's there if you omit it. And when you do more powerful queries across multiple datasets and you're yielding multiple results, the yield will become much more useful. This will be something we look at throughout the course, especially as we move into monitoring real production systems. So yield, you can omit, but you can also name it as we do here. So

13:18 Yield Example

13:23 you'll see that we have a couple of queries here. The first one is a from range of fifteen minutes. We have a filter, which we haven't really talked about yet, but will be something that we talk about in your course. And then we yield that as that dataset as CPU. And then we do the same here, yielding as memory of a different filter. And we should be able to What did I get wrong with Oh, I forgot that. Is it V yield? Here's another tip. You can, when you're using the UI, you will get this list of functions on

14:26 the right hand side. This is invaluable, right? You're never gonna remember the syntax for all of them, just as I have forgotten. You can type the name of the function you need and literally just click inject. And you'll see here, I just forgot the name parameter. So that's a really good point. All functions and flux take named parameters. You'll see that with the range has a parameter called start. It also has a parameter called stop. We could say, give me Last, well, give me ten minutes of data starting fifteen minutes ago, like so. The filter

15:04 function takes a function called fn, the property is called fn, then it takes an anonymous function inside. The yield takes a parameter called name. So all all functions in Flux take named parameters. And let's see, we just maybe just look at this as raw data, We can see this is the CPU results. And if we scroll down, oops, still CPU. There'll be, where does this memory start? Oh, there we go. Available up. Where are you? Try and drag back up. It was a bit finicky. There's UC system at the end of CPU. Too much data.

16:08 There we go. And there's our memory results right here. You can see that's an active channel. So, you can yield multiple things. Really cool if you want to graph things and build correlations and be able to look at how your CPU is affected with your memory. Like, is there a correlation that as memory consumption goes up or CPU issues going up, etcetera? Now, that's a contrast example. Your infrastructure, your applications, you'll know what you're looking for and what you're trying to correlate. Okay. So that is our very first Flux query. We've taken a look at from and range

16:35 Exercises

16:44 and yield. Those are just the things that are required for everything you do. Filter is equally important, but we're gonna talk about that in just a moment. I'm gonna run through the exercises now. So if you want to do the exercises on your own time, you want to do them now, or you want to try them before I run through them, now is the time to pause your video and come back later. I'll cut to 10 in my head while staring intensely. I stopped counting, I don't know when it turns out. Right, I'll do it. Okay, so the exercises

16:55 Starting Exercises 1

17:28 are nice and simple. We're gonna layer these up week after week and add a more complex examples, but really we just wanna get you familiar with entering the query into the UI, and then we'll take a look at the Flux CLI REPL as well. So we want to fetch all metrics over the last five minutes. So we can say from bucket and InfluxDB UI does use language server protocol and Monaco with the Versus Code Editor. So you will get relatively good autocomplete. Workshop range star dash five ms. That's exercise one, really simple. In fact, we

18:11 already had a variation of this right at the start as we were copying and pasting. So really, you just wanna be familiar with the pipe operator, know that it's two characters instead of just one that you may be familiar with from, you know, shells, you know, with bash scripting, you know. So we do have the the greater than same. And it's important to remember, and hopefully you get comfortable with the named parameters. A lot of people trip up with that. And then the duration laterals, and what a lot of people do at the start as they're writing their first tax

18:43 queries, would be to forget the minus and just remember that you need to as a duration. Right? So you have to go back five minutes to get the actual time that you want to query. So we'll take a look at one more thing from here. I just opened a new tab. And if we run Influx, we can see there are a bunch of Oh, what is going on with this terminal? I've had problems with iTerm for a while now, to be fair. Okay. So, we can execute flux queries on the command line. Guess all of my influx commands are gonna

19:00 Executing Flux Queries Outside the UI (CLI & VS Code)

19:44 be acting a little bit funny. Alright. Why don't I open a new tab? Let's just take a look at the help on this. So we can pass in a query that we wish to run. So we could say from bucket workshop range start five ms. Oh, and yeah, we will need a few extra parameters as well. So when we do this on the command line, we will need to provide our --org, which is Rawkode Academy. And we need to export our token again. And it should still be in my history. That's a very old token then apparently. Let's

20:54 grab it from, yeah, let's grab it from here again. Export info token. Not the same. Oh, wait. It's invalid expression. Completion field, not authorization. From I can't remember actually if Flux cares about the quote. Let's do this. From bucket, pipe forward, range star, go. Oh, and fix the end one. There we go. So quite a lot of data there. Yeah. An awful lot of data. And have to remove the Rackle. Oh, that's a shame. Oh, well, you could compile Flux from source and there is a REPL that I think has been removed from the InfluxDLA.

22:21 That's unfortunate. We won't be using that today. This one is definitely broken. That one's definitely broken. Those two are happy. Alright, perfect. So we've seen the CLI. We've seen the UI one more, and there is a Versus Code extension. So, if we jump over here, there's your flux. You'll see that there is this Versus code extension from InfluxData. You can add connections, local HTTP local host. I've got my token and my organization name. And you can see we get a success down here when we test that connection. So let's save this. And that means that you can

23:46 run flux scripts. So we can say run query. We need a flux fail. Right. Query Flux from, and we get awesome all complete from the IDE. Again, using the language server, which the team have been working on. Academy range star dash five m. Yeah. Let's just run that. Rawkode Academy. Oh, workshop. Work show the organization as the academy. So we can run this query. I think it's just loading all the data. And there. Really, I should have narrowed down that query before running. Alright. Let's do that. Let's just grab ten seconds. There we go. And now we can query and look at

25:39 this data in our Versus Code. So it's entirely up to you. It's still quite a lot of data for ten seconds. Maybe I've got my enter full too quick. So it's up to you. I like using this for the auto complete and functional complete and help text and other stuff, but the web UI has also equally as good. And then the CLI is there if you wanna be able to automate or build scripts or into any of this stuff as well. Okay. So that was exercise 1.1. Now we want to fetch all metrics for the last hour,

26:13 excluding the last five minutes. So really, we're just coming back here in one hour, only we're adding a stop because we don't want the last five minutes of data. That's gonna be a large query as well, but the UI should be able to handle it. He says, hopefully. And then the last one is we want to fetch all metrics for the last three hours returned as multiple datasets. How are they? Yeah, there we go. And there's just one thing I really want you to take away from this last task is that we want all metrics for the

26:54 last three hours hourly. So what does that mean? Well, we could do three hours dash two hours and just copy and paste that where we moved down to two hours and one hour, and we moved down to one hour without a stop. And we can yield each of these. Name, one hour, and we'll call this two hour and three hour. And that will work fine. And something else you can actually do is assign the output and then reuse it. It's not very useful in this case because it's just a from statement. This time being reused across multiple

27:55 ranges. But as your queries become more sophisticated and you're doing a lot more, you may have a set of from range and filters that you want to be able to reuse across multiple yields within a single Flux script, and that is entirely possible. And we'll take a look at just how that works next, because we're about to take a look at filtering windows and aggregations. Cool. Okay. So hopefully, you know, that first thirty minutes of this workshop just gets you acquainted with right in flux. The different interfaces that you can use for submitting your queries and working with the data.

28:22 Introduction to Data Filtering

28:36 The UI is a great place to start. Everyone should start there first and then move in to the CLI Versus Code, whatever, as you need. The Versus Code option becomes much more appealing as we start to play with Packager, which is a manifest format for applying InfluxDB configurations to InfluxDB. Kinda like KubeControl where, you know, YAML manifests, we can do all of that within InfluxDB to manage everything from a GitHub pattern. So it's actually quite nice to be able to work with that in Versus Code and have all your queries saved on local Flux scripts. So and we'll be doing more in

29:11 there. Okay. So all of our creating has been extremely superficial thus far, and that's because we aren't doing anything useful. We're just pulling out raw data and actually seeing that with the, you know, thirty minutes of data that we're already collecting and ranging over even a few minutes of that can be quite intensive. And it's just because we're collecting relatively a lot of metrics for this little MacBook to handle. So we wanna be able to make sure that we are using filtering and using filtering effectively and efficiently when we query InfluxDB, the leveraging things that well, the actual tensor

29:14 Filtering on Tags vs Fields (Performance)

29:47 did risk itself to help us with the felt string so we don't have to do all memory. And we talk about that in this next this next section. Alrighty. So what's important here would be this, right? So I'll read this verbatim and talk about it. The filter function can filter data on any field and a dataset. This means that you need to be particularly careful when adding filters and assure that you understand when filtering on a tag or a field. If you think back to the lecture we had at the start of this course, lecture

29:55 Filter Function

30:25 one, an introduction to time series, I talked about how tags are indexed within the time series database and fields are not. Now because fields are not indexed, if we used a filter function to filter on a field, what actually happens is all of that is loaded into memory and essentially looked over and filtered as required. However, if we use filters against the tag values, which are indexed in the time series database, we can actually just push most of that down to the TSDB and say, don't give me this data or only give me this data,

30:58 We're substantially more efficient on memory and faster on CPU. So make sure that you're constructing your queries appropriately. The filter function, and we'll do this by here. The filter function takes a parameter called FN. FN is not a keyword. A lot of people make this mistake when they first start adopting Flux. Fn is just the named parameter and it takes an anonymous function and that anonymous function should return a billion or must return a billion value of true or false. If it returns true, the data will be kept in the dataset and if it's returned false,

31:13 Filler Function

31:40 the data will be omitted. Now you've already seen me do this now, but using the show data toggle is the fastest way to really understand your data as you start creating an Influx. So, we can, we don't need that right now. Definitely don't want an error. Well, there isn't an error, but let's just get a little bit of data. We can remove the yield and use the implicit. And I won't be able to choose that to a minute. So you can kind of see this Oh, no, you can't because of my face. There's a little tiny bit of data here.

32:28 So and that's just because we're trying to show, you know, we've got that set to five minutes, but we're only pulling back one minute. There's not a lot of information here. Right? Like, if I wanted to start filtering on my data that lives in the workshop bucket, I don't really know how to do that without having substantial knowledge into how it works. So you can use the show raw data button, and we'll kind of talk about how this how to navigate this big blob of text here. I'll bring my face back in, but if it gets in the way, I'll

33:10 remove it. So this raw data view that we have in the UI is a annotated CSV almost. Now, what we have here is that information on the data types for each of the columns in CSV, and we have some group key information, which we'll be talking about as we talk about groups moving forward. We have a result, which is our data. We have tables. We have value. We have the start and stop period. So this will tell us about a range within the query. We have the time that this point value was stored. Now, all of this is stored in a

33:57 columnar fashion and the TSDB, which means all of the fields that we store, we can actually, we'll need to pivot them to see them as robust data. Again, stuff we'll cover in subsequent videos, don't worry about it if you don't understand what I mean by pivot the data. But because this is columnar, we have a fields code underscore field, which has a value, which tells us which field we're creating against. And we'll take a look at that as we start to build up some of these filter queries as well. Fields belong to measurements. So this is our

34:25 CPU measurement, will also be a memory one, a system one, a processes one, etcetera, down here. And then we have some tags. So this is a CPU zero that tells us, you know, this is a 16 core machines. We're gonna have CPU zero to 15, and then the host of the data was collected. And this is just based on what telegraph rates to our time series database. But using the show raw data toggle is the best way to understand how to build and construct your query. Unless you want to use a query builder, of course, you can do that too.

34:54 But what I would see here is that I could filter on measurement equals CPU and field equals usage guessed to really quickly break down the status of just one or two things that I wanna see. So let's copy this query here. So this is a from bucket workshop, range of fifteen minutes, and we only care about the CPU measurement. That means we're saying goodbye to disk, disk IO, memory, network, processing system and swap. There we go. That was a quick tour. We run this. And I know fifteen minutes seems seems like a short window, but with that two

35:48 second resolution, that has actually quite a lot of points. So we're getting 30 points per minute times 15 across 16 cores. Very quickly adds up. And you can see just by scrolling on this, you know, we're we've got at least a few hundred tables going on there. So, yeah, that's these filters. Now, you can add many of these filters as you like. We don't wanna fill our measurement again. Let's say we just want a single field. We can do usage guest. That should be a pretty quick query. And now we only get a single field

36:37 for each CPU. So we're still got the 16 CPUs, but the field for each. I think what is important to take away from filtering is that the database will work out optimization to make that as fast as possible. As long as you keep all of your filters together wherever possible. Of course, you're always gonna need filters further down the stack and that's okay. But anything that can you wanna reduce as much of your dataset as possible right after the range because that as long as you're again, as long as you're using tags, we'll push it

37:14 all down to the time series database. So you don't need to load all of that into memory. And that's just as good increase your performance substantially. If we were to call another function here, and I know you don't know what window is yet, but that's alright. Those filters being broken up can no longer be optimized because we have some logic that has to happen in between, meaning the only filter that will get pushed down to the time series database is a measurement. We load everything into memory to perform the window. And then even if this is a tag,

37:46 it doesn't matter. We're still filtering it and memory. So really important to understand and try to remember that if you can keep your filters to the start, you're gonna have a much better performing database. Okay. Now, it's really your own personal preference now, whether how you structure your filters. And what a lot of people like to do is use the android and you can say usage. K. And do that as a single filter. You can have them all as individual filters. It does not matter. Personal preference, use whatever you like. Me, I kind of prefer

38:30 separate filter lines as much as possible because I think it's a little bit cleaner. Of course, if we get down to a certain point here where I have others and I have to do a slightly later filter, then in order to avoid looking over the data twice, although there still are performance things that happens with the runtime that we'll try to avoid that. And I will use the and syntax. But as I said, for the first bunch of filters, I do like to to put them together. This is one filter in the runtime, not two.

39:02 In fact, that's also arbitrary. I could have as many filters like this. These would all be filtered at once and pushed down to the time series database as long as their tags. And that is always, always the caveat with querying and FluxDB too. Stick to tags as much as possible, push the field filtering as late in the call or the script as possible. Okay. Let's talk about these results that we've been getting. And to do that, we have to explain tables and grouping. So Flux returns all data from your queries as a set of tables, which are grouped by

39:33 Understanding Tables and Default Grouping

39:42 a group key. This group key by default is each series within your data where a series is a unique combination of tag and values. Definitely make sure you've checked out the first lecture, an introduction to time series. I talk about series, tags and keys. But let's take a visual representation of this. I'm gonna remove this. We will pull out six seconds of CPU. Woah, what did do? Okay. Thirty seconds. Okay. Think that the six seconds work and I just Oh no, because my flush interval is six seconds. There we go. Yeah. Okay. So, view raw data. Okay. So,

40:40 we have got a from a range and a filter, right? These are the basic constructs of all flux queries. You'll see that we have table zero, we have table one, we have table two, we have table three, so forth, all the way down the stack. So a table is a series based on the default group key. The group key that's been used here is our field name of usage guest, the measurement of CPU, the CPU tag, which is, you know, each CPU will become its own series and the host. So what we have here is a set

41:21 of tables for every CPU core on this machine and for each field available in the dataset, which I think is anywhere from eight to nine, not entirely sure, but let's try and make this more visual. So if we add in a new filter, and we do field equals and we drop this down to one field, we should see one series for each variation of the series, which now because of we have constrained the field to a single one will be the CPU's, meaning we should see 16 tables. Okay. Table zero, table 15 for fifteen, sixteen tables.

42:17 Tables are the thing that trip everybody up at the start. Trying to get the data out because it's columnar and and, you know, not a lot of people are familiar with creating against columnar databases. It's really fast on the read path and the write path, but comes with a little bit of a cognitive overload that people just aren't familiar with. So, I hope you understand tables. If not, leave comments on the video, jump into the Discord, happy to spend more time on it. And we'll be looking at it in a little bit more detail as we run through this

42:47 too. But really, hopefully you have a good grasp of it already. Okay. And we'll copy that just so I can make sure that the query Oh, there's a nice little copy button. I'm not gonna do five minutes. Let's just do thirty seconds. There we go. Now, I think I ask a question here. Yeah, I do. So looking at this query, how many tables do you expect this to return? Okay. So from and range, we filter on the measurement and we are requesting three different fields. I'll give you a moment or you could pause it. And the answer is here. In fact, I

42:53 Copy and Paste

43:40 wouldn't even read the answer. Feel free to peruse that at your own time, but hopefully you got it right. Okay, so now we can change the group key to get the data in a format that we need. Again, once you have an understanding of what values you wanna get from your metric data, alright, and your event data, you can start to piece together your queries and you'll have a better understanding of the group key that you need. So let's assume that we want data above the group by metric, ignoring the different CPUs altogether. So, I don't want a table

44:07 Grouping

44:17 for each CPU. What I want to know is can I have an arbitrary number that tells me of whether my CPU is saturated or not? So, we do a from range, we filter on the measurement, and then we filter on the three fields that we want, usage system, usage user, and usage guest. Now we're introducing the group function where we can provide the number of columns to use, or which columns to use as a group key. Here, the measurement is actually superficial because we're grouping on the field. If we had multiple measurements with a field

44:59 name that duplicated across them all, you would want to make sure that you restrict it properly and you don't have like a response time and multiple measurements all get grouped together because it would skew the values. But it's usually best to be as constrained as possible. So measurement field and then the motor bag. And then if we run this, we'll see that we get all of the CPU numbers. And the same table. Perfect. And we'll have three tables. You can see the two on the left here, and that is for each of the fields. So now we have a table for each

45:51 field that we care about, and we can actually do aggregations across the values, across the CPUs to understand whether the system is under load or not. Alright. How many tables do you think we'll get back now? I just answered that. Sorry. Spoiler. Something else I like to share with people because, again, it's one of those first questions I get all the time about Flux is, but what if I don't want a group? What if I I just want a single table? And you can do that through an empty group. I tend to call it a flatten,

46:15 Flattening Data (Empty Group Key)

46:31 flatten the data, single group. I think that's just a term that maybe I use. Haven't heard a lot of other people use it, but it's just basically no group. We want to remove the group key. So if we jump in here, we can call an empty group like so. And now the zeros all the way down. That just gives us one flat table. And the the use case for that is just having a a single series for graphing or pulling out explicit values and from a bunch of aggregations that you're making across your data. All right, let's

47:17 try and get through this last bit within the hour. And I'll be back on the next video will drop in just a couple of days. Okay, so windows and aggregations. There's a little note here, I just want people to be wary of is that MO and Y behave slightly differently when they're used in window operations than regards to the from. So if you use one month for one year and a from, it will go back thirty one days or three sixty five days. However, in a window operation, they are calendar aligned. So if you see window that's data

47:27 Windows and Aggregations

47:53 by month, you will get the 1 the month to the end of the month. And the same for years, you will get from January 1 to December 31. So just keep that in mind. I think that's intuitive, what people would expect. However, it doesn't have to reiterate that, of course. So what window functions allow us to do are to divide the data that we're creating into windows that we wanna build aggregations for. So let's, as an example, we are collecting our CPU data every two seconds. That is quite high resolution for CPU on a standard Linux machine. As you

48:12 The Window Function

48:31 can see by some of the queries have already tickled the box a little bit. So we probably don't want to understand the load at two second intervals because you know, there are gonna be bursts in CPUs as applications are spinning up and down. It send instructions to the CPU that cause that number to artificially Not artificially. It goes up, but it's just not that important because it drops back down very quickly. So we It's very common with time series data to say, okay, I have two second resolution, but I actually wanna build an aggregation, a one minute, five minute, and fifteen minute

49:02 intervals. Hey, you may be familiar with that already, right? Linux load averages. So here's a pop quiz. I'm about to run this query. How many tables will it return? So this does a filter and a filter. So we've got a single measurement, a single field, and we're windowing. And we know that I have 16 CPUs on my machine. So keep that in mind, have a guess, pause it if you wanna work it out for a minute. I am gonna go run this query. So fifteen minutes of CPU data, windowed every one minute. I'm gonna window it. Oh, in fact, yeah,

49:08 Pop Quiz

49:43 I should run it as one because that's what I said. Right? Two fifty six tables. Is that what you got? Okay. If you want that explained, the answer is here. So depends on how many CPU's you have on your system. My machine has 16 CPUs, so it returns two fifty six tables. Why? Well, we have requested a single metric and we have 16 CPUs, so you may have guessed it, 16 tables would be returned. However, because we're modifying the group key with our new window, and that's important to remember, windowing data does modify the group key.

50:28 What we're actually saying is that we want one minute intervals. And because our range is 15, we are actually multiplying the number of tables by 16. The reason why is that we get our fifteen minute one minute intervals, plus the current interval depending on how far into that we are. So we can actually just take a look. So let's try and start at the top. You can see the windows on the start and the stop. So we have fourteen thirty five to fourteen thirty six, and you see that is not a complete minute because we are halfway through

51:11 this interval approaching this. Whereas the other intervals are complete, fourteen forty four to fourteen forty five. So hopefully that makes sense. Window changes the group key. We are building artificial series based because we're seeing chop this data up. Right? So we're getting unique series because we're bringing a time into dimension of the window. Something that's usually quite cool about doing that though is the colors. So let's take off the window because I wanna show you this. This is cool. What we see here is consistent colors for each series within data. So each CPU core here gets its own

51:24 Window and Changes

51:58 color as we can see in the the legend. So CPU zero, it's a very light blue, CPU nine is very orange, and then we've got this cascading color changes through. And that consistent is the color the series color is consistent across the x axis. Now, when we window the data, we are having a unique series for each window within the data. So actually, the colors now change along the x axis. You can actually see that our first series is this time window here. Our second series is this time window here. Our third series is this one here. Our

52:41 fourth series is here and so forth, all the way across there. I always like the colors when you do that. Very cool. Okay. Now we have bundled data, which is great. We have colors, but we actually need to do some more. We wanna be able to aggregate that. Right? We we have 16 values across each one minute interval and we wanna under we want one single value across each of those intervals to understand the CPU total load, not per CPU. So, one of the things we could do is use the aggregate functions. The one we're using here is called the

52:53 Performing Aggregations (Mean)

53:18 mean. So, we want to calculate the mean across this window. I'll remove that. So this is the same query as before. CPU filter, usage user filter, window by one minute. In fact, if I remove this, I think it's just a range change, but it's still the same data. We can I should think, yeah, I'll spend some time? You're probably wondering why all my data shows on the right here. And it's just because I'm using a fixed range here. If we use the variables, time range start, we'll use the whole thing and then we control it through this here.

54:10 I should have explained that earlier. I just tend to use dash minus five ms so that I Because it's easier in the markdown than saying click on this. So there you go. Now we can bring in the mean. And if we view the raw data, we're gonna get a single value for each series, which can't be graphed because a single value for each series is a point. So you'll see the points along the graph. And just to show you how this was working, there was one line that I removed. So let's copy that again. The one line

54:54 I removed was just drop us down to a single CPU, run it again, we get a single point. And if we view the raw data, we just have one point per window. Okay. So how many tables? Well, you can read the explanation for that there if you wish. What's important here is that this combination of how to add one to the group there. Yeah. I do talk about the group here. So you can read that. Just in the interest of time, there's only a few minutes left. But this is a very common operation to do

55:32 Aggregate Window

55:35 a window aggregation and then flatten or regroup, changes the group key. It's one of the most common time series queries that you're gonna do against your data. So I show that there's a helper called aggregate window to do this all for you. So if we come over here, we can just see, I'll just remove this measurement CPU, build usage user, window one minute, calculate mean, click the go button, and there we go. And we'll remove, we'll use our time range start so we can get the full graph. And now we're starting to get nice graphs

55:47 The Aggregate Window Helper Function

56:13 that show the CPU load over time on this machine at one minute intervals. If we wanna flatten this out a bit, we'd probably do it at five. Of course, I'm gonna need more data points, so let's do this. And now that is what we expect, right? As you work with Grafana and InfluxDB, Prometheus, all of these tools, We just wanna be able to understand fluctuations and values over time, our windows that we decide the resolution that is important to us so that we can have a graph that represents normal so that we can alert on not normal.

56:48 And that is fundamental to what we're doing today. So aggregate windows, you can do it manually. We have a window, we have an aggregate function, we have another group, or just use their function. In fact, if you pull up the InfluxDB docs, the implementation of aggregate window is the window function group, like so. So there are exercises here. I was hoping I would have time to do them today, but I won't. What I will do is I will schedule or I will record myself doing each of these exercises and get that uploaded as a tutorial session and the course.

56:50 Conclusion and Next Steps

57:26 Please go in and check out exercises two in your own time. Use the Discord to ask for help. Use the comments to ask for help. I'm here. I wanna be able to help you. So that is workshop one, an introduction to Flux. I hope you found that useful. Let me know how you got on. I'll speak to you all soon. Have a wonderful day. Bye.