0:37 Welcome & Introduction

0:37 Hello. Welcome to today's episode of Rawkode Live. I'm your host, Rawkode. Today, we are taking a look at Promscale. We're gonna learn all about that in just a few minutes, but before we get started, I do have a little bit of housekeeping. So first, please subscribe to this YouTube channel and click the bell. This is gonna get you notifications for all future episodes of Rawkode Live as we explore the vast cloud native landscape together. Also, if you like to chat, we have a very active Discord server where we're talking all things cloud native Kubernetes and everything in

1:08 between. So come and say hello. I look forward to meeting you. And one last thing, I just wanna thank my employer Equinix Medal. They provide the resources and time for me to invest in the show so that we can all learn cloud native together. So thank you very much. Thank you very much Equinix metal. You can check out Equinix metal was a code Rawkode that will get you $200 and compute that as around four hundred hours on a modest instance. So check it out, have some fun and let me know how you get on. Alright. Now to guide me today on our

1:35 Guest Introductions

1:37 journey into Promscale, I am joined by Matt and Vanneef from the Promscale and Timescale teams. Hello there. How are you both? Hey. How are you doing? The the joys like of when you have two guests on the show and then you throw out the how are you question there's always this like oh who's gonna go first? So every time I do it I catch myself but just a little bit too late and I'm like oh, okay. Okay. So Matt, hi. How are you? Welcome to the show. Yeah. Thank you for having me David. I'm doing well. Awesome. Do you wanna just tell

2:13 us who you are? What you do at timescale? Anything else you wanna share with us? Yeah. So I I've been at timescale for a long time, pretty much since timescale was a company. I used to develop the core database product, And about a year ago, I moved on to become the team lead for Promscale, which we will tell you about today. Awesome. Thank you. And, Vineeth, how are you today? Yeah. Hello, everyone. My name is Vinith. So thanks for having us, David. So it's really a pleasure to join Rawkode. And, yeah, I work at timescale in the team

3:01 observability, and my I primarily contribute to projects at tops and from scale. And it's been last seven months I've been working with timescale, and previously, I worked with other cloud native companies. And I also contributed to Cortex and Open Policy Agent and Kubernetes upstream previously. So yeah. You keep yourself busy. Yep. Yep. Awesome. Well, thank you both so much for taking time out of your days joining us today and and hopefully, you know, we're gonna take a look at Promscale today and a little bit of TOBS. So why don't we start with just a little bit of introduction.

3:40 Introduction to Promscale (Slides)

3:40 I believe Matt, you've got your screen ready for us here and you're just gonna present a couple of slides. So please feel free to take it away whenever you are ready. Yep. So hi everyone. Most of this I already covered. These are the ways you can kind of get in touch with me on the Internet. The other thing I was saying is if you're interested in any of the things I like to geek out about, skiing theater, travel, or photography, Just hit me up on one of the Slack channels. I'm always more than happy to chat,

4:19 especially in this pandemic world we live in. And I will let Vinit also introduce himself. Yep. So yeah. Like, hi again, and my name is Vinit, and I work as a software engineer and Promscale team and the observability team and timescale. And feel free to reach out to me to talk anything about cloud native open telemetry or all, like, observability in general. So I'm very active in CNC of Slack and TimescaleDB community Slack. So you can reach out to me at the rate we need. And feel free to mail me to beneaththattimescale.com to discuss anything about tops, Promscale, or timescale

5:02 d b. And on my free time, I just keep cycling, hiking, and reading books. Like, if you want to discuss about anything or even join me in India, just feel free to reach out to me. Great. So let's start with what Promiscale is. Promiscale is an open source long term store for Prometheus data, which is built on top of TimescaleDB. And TimescaleDB itself is an extension to postgres SQL. So this is a relational database that stores Prometheus observability data. And what's unique about us is that we store all of this data in the relational database,

5:18 What is Promscale?

5:52 which allows you to use SQL analytics to get much more insights into your observability data. The basic architecture is as follows. Prometheus uses the remote right and the remote read protocols to speak to the Promscale connector. The connector on ingestible just ingest data from Prometheus and the write it to time scale DB, which can be a single node installation or horizontally scale, but there should be the installation. And then you can install you can use any kind of visualization or analysis tools, speaking either SQL or PromQL to to get data back out of the store. If you are speaking PromQL,

6:04 Promscale Architecture

6:50 you, connect directly to the connector, while if you are speaking SQL, you speak directly to the PromscaleDB database. But at this point, you guys might be looking at me a little bit side eyed because traditionally, relational databases didn't scale for this kind of workload because this workload is very large. Right? Preservability can ingest billions of rows a day or or even tens or hundreds of billions of rows a day. But time scale makes this possible because it allows you to scale your postgres cluster to allow you to ingest that kind of data volume. In addition, it supports

7:03 Why Use TimescaleDB for Metrics?

7:49 column the time series compression so that your data isn't huge on disk and so that your budget doesn't explode. Right? And so we use all of the kind of best in class column compression algorithms including Gorilla and delta delta encoding, etcetera. In addition, traditionally, relational databases can only scale up and not out. But timescale, again, allows you to horizontally scale out your database as it has a multi node version that scales kind of like you would any NoSQL database. So kind of maybe I told you a little bit about why this is possible, but why would you want to store your data

8:44 Key Benefits of Promscale

8:57 in Promscale? Was it or a couple of reasons. First of all, unlike most kind of for me, TS DataSource, we support backfill, which allows you to easily support ingesting old data or kind of migrating all data into the system. We support flexible querying, as I mentioned before, either in the PromQL or SQL, your choice. We support the flexible ingest. You could use the standard Prometheus pull based model or you can push data directly to Promscale itself. It is easy to deploy and use, and hopefully, the demo will kind of illustrate to that. And we have

9:50 a lot of community support on our Slack channel. But and I didn't want to cover a little bit about why we're concentrating on Prometheus because I think some people might not be so familiar. I think it's safe to say that Prometheus is becoming the common data format for metrics in the industry. And that's mainly because of its easy to understand data layout. It's really a time series based layout where you have a timestamp, a and a set of values associated with a tag set of key value pairs describing your data. And PromQL has really become at the standard in data reporting

10:01 Why Prometheus is Important

10:47 queries today. The other major thing is it's cloud friendly. It's cloud native to Kubernetes. And I think an important point to realize is that the pool based model that Prometheus uses is really good when you have microservices that come and go very quickly. Combined with kind of a service discovery, it really allows you to monitor a fast paced cloud native environment. And now I'll look to beneath, say a few words about tabs, which is the observability stack for Kubernetes. Yep. So Right. So, like, have another tool called TOPS, which is the observability stack for Kubernetes. So

11:32 Introducing TOBS: Observability for Kubernetes

11:51 let's start why TOPS and what is it all about. So deploying observability is way, way, way harder in your Kubernetes cluster because you already have your workloads running, and then you want set up Prometheus and to put the service discovery enable, like, what port what services you want to scrape. And if it's Kubernetes cluster, you need to run the cube state metrics and all this. Like, you have so many moving parts and so much of network configuration, observability level configuration, and you need to configure like, if you're using Promscale, you need to as a point,

12:25 the remote read and write in Prometheus to Promscale and from Promscale to Timescale DB. So any observability tool you install in Kubernetes cluster, you need to configure it and make sure that everything is working as expected. So here comes the tops. So tops is all about shipping observability stack in just one command. And tops comes with all these components. Like, one interesting fact is that we just launched tops, like, released tops, like, a couple of hours back by switching the Prometheus and Grafana help charts to keep Promscale. So we just swapped out, and this is the first demo

13:02 we are doing post lot lot release. And you are just looking at the latest and greatest tops as of today. So, Alex, just fingers crossed for that. And if you look at into the components for queue Promscale, so queue Promscale by default out of the box comes with Promscale alert manager, node exporter, and Promscale operator to manage the Prometheus and alert managers as a custom resource definitions. And you have cube state metrics to it it's something like it's an exporter for your Kubernetes cluster. So you get all the metrics out of your API server. And we also install Promscale

13:36 and TimescaleDB for long term storage of metrics coming from Prometheus, and we have PromLens. Like, if you are building PromQL queries, so it'll be just running. Anytime you have any problems or questions with the PromQL queries, just jump into PromLens and build the queries. Yeah. And why did we switch to cube parameters? Because this is one of the biggest and recent changes we have decided in the last one month. And because there's a lot of community and the community acceptance for the cube Prometheus. So any user we talk about Prometheus, they're already either using Prometheus operator or they are using the cube

14:17 Prometheus. And we have seen, like, heavy lifting already being managed by the Prometheus operator and with the Q Prometheus tags. So we wanted to switch to the Q Prometheus because there's a higher a native high high availability support for Prometheus and alert manager. And also, offers Kubernetes mix sense, which is out of the box alerts and dashboards for your Kubernetes cluster. So it's already pre baked for you. And configuring Prometheus and alert manager as custom resources. So every time you need to make changes to your Prometheus cluster, there is no need that you get into the Prometheus config, change

14:53 the alert manager endpoint, add alerting rules or recording rules. You just put it in in the form of custom resources, and Prometheus operator will reconcile it with the Prometheus for you. And you have Prometheus operator for life cycle management of the Prometheus, the config, and the alert manager. So this is the main motivation for us to switch to the cube Prometheus from native Prometheus and Grafana. And this is the top's architecture as of today. So post launch, like, so many components have been changed. So I just put the sketch as of today. And we have

15:20 TOBS Architecture Overview

15:30 the Kubernetes API server, which gets scraped by the cube state metrics, and Prometheus scrapes the cube state metrics. And you see all these components in your Kubernetes cluster. So you see this dotted section where the tops architecture comes. So once you just do tops install, all these components gets installed by default for you. So you get all the observability for your Kubernetes cluster from cube state metrics, and you get node level metrics using the node exporter. And you have alert manager to pop up the alerts for you if by default from the Kubernetes mix and alerts.

16:05 And you have Grafana for to query the Promscale and Timescale DB. So here, if you observe one interesting thing is that the Grafana points to Promscale but not Prometheus Because Promscale is the long term storage for Promscale, and Grafana directly by default adds the data sources from scale for Promscale, and you run all your Promscale queries to Promscale. So Promscale takes care of executing the Promscale queries on top of Timescale DB for you, and you just get all the Promscale results by from the Promscale. And if you are interested to use SQL for analytics or for visualization within Grafana,

16:46 Like, you have timescale DB data source already enabled for you in Grafana by default, and you can run a scale queries from Grafana on top of timescale DB. And apart from this, if you have any other services running in cluster, Prometheus by default has the additional script configs enabled for you. And if you have the annotations enabled for your services, parts, and endpoints, the Prometheus by default scripts all those services for you. So you do not need to make any changes for that. So this is the here top's architecture in high level. And so, like, this this is something I always

17:20 Simplifying with the TOBS CLI

17:23 I'm always excited to share about because so if you talk about tops in general, you have seen all those components. So if you see tops has the two sections, like, primarily, if you look at as a at at a as a project. So one is the helm chart. So all this packaging of timescale d b, Promscale, Q Prometheus, all this happens as a helm chart. So you can directly install TOPS as a Helm chart, and it's already available. And if you are using TOPS CLA, you'd not care about what Helm and what are the changes happening

17:53 in Helm and how to manage the full life cycle management of your TOPS and your observability stack. So top CLA offers you all that commands. So you just run tops install, and all the architecture is installed for you. And you have upgrade upgrade to upgrade the tops from previous versions to the new version, and you have all the set of comments. And if you see look at the sub commands also. So if you have the final running and you want to get the password in Kubernetes cluster in the native way, you need to find what

18:21 secret it is mounted to, and and you need to find the secret base 64 encoded string and decode it and use it. So it's always a long process to get the passwords out of your Grafana or change the password and everything. So tops just offers you the comments like tops Grafana get password. It just shares the password for you from Grafana. Change password. It just updates the password for you. And even if you do port forward, it just load port forwards to your local system so that it's easy for you to manage and reach to the the Grafana.

18:53 Same with TimescaleDB. Like, if you want to run the SQL commands, you just run TimescaleDB connect, and it just gets you to the PSQL prompt of TimescaleDB, and you can run all the SQL queries then and there. And you have metrics. So one interesting fact about the Promscale long term storage for metrics is that you can get the retention policies per metric level. It's not like it's not it's not global retention policies. You can apply the global retention policies also per parametric retention policies. You can say that this metric is important to me and have the retention policy for this particular

19:28 metric for sixty days and have the global retention policy to thirty days. So all this metric level life cycle management also can be handled by docs. And one other problem we keep noticing with u observability stack operators is that volume expansion. So the more and more you start scraping the targets on getting the data into your long term storage, you need to expand your persistent volumes. So you have this persistent volume claims and everything for you. So there's a top volume comment for you to expand, And you have put forward comments for all the important components

20:06 which the operator would like to interact with. So you just do put forward for Promscale, Prominence, Prometheus, anything just just port forward for you. It's as simple as running cubic deal port forward for you, but tops does all that for you. It just finds all the parts, the services, and it does port forward for you. So this is the top CLA what it offers for you in high level. Great. Before I kind of go into the Promscale vision, David, do you want to kind of kick off the install of tabs? Because it might take a few minutes.

20:40 Installing Promscale with TOBS

20:45 So we can come back to this after. Of course. No worries. So we have my screen share up now. I'm looking at the Promscale repository, but here's the Tobs one. Okay. So in order to install Tobs, will I just do the the curl bash? Yep. Yep. And and we also have, you know, direct downloads from our release page on GitHub because I know a lot of people don't like curl bash, But yeah, that's the easiest way to do it. I'm happy to curl bash all day. Don't worry about it. Okay. Yeah. Alright. Yeah. Just move

21:31 it. Yep. Alright. And then it will oh, what keep context will it use? Whatever I've currently got? Yeah. Yep. Yep. Yes. Fingers crossed. Like, you're just trying the release which like, this it was released, like, couple of hours back. And we've already had one hotfix. So Okay. So yours it says that failed to pass. Can you just show me the Helm version you are using? Yeah. Yeah. Sure. So oh, it's a Go app. Right? So three five four. Okay. It should be good. Like, it says that parse helm show values from YAML to JSON mapping are not allowed in this context

22:01 Debugging Installation Issues

22:33 stream. So Have you seen that before? No. Like, it's just passing the values. What can we do for this is Failed to I mean, when and though try again. Right? Yeah. Yeah. Failed to post home for values from YAML to JSON YAML. Let's talk about what tops install is doing. Alright? It is installing things from this chart repository? Yes. This. And then there's this values file that was changed four hours ago. Right. We suspect maybe there's a I bet it's like a space or something in this file. Alright. Let's So. I'm just gonna validate your YAML just in

23:49 case. Okay. Great. Vinit, could you, kind of Google that error? It's like I'm just able to I just went into the code with it. I'm just looking at so help show values. So what we do before doing help ins help install is that we do help show values, and and we try to pass it to Jason to analyze the data in the values. So help show values as not giving the results as expected is my understanding. So can you just try to run help show values for the release tops for the help chat tops so that

24:27 what's happening there. Yeah. Of course. No worries. Let's just take clone as repository. Give that a Or the easiest way we can do is that maybe you can just point to these values and chat explicitly and check does that work. Show values. Oh, that's good. Okay. So let's just run this command, like, helm install sorry, tabs install. Sorry. Yeah. Tabs install and just point to the chart directory you have locally just to understand, like, does it make So yep. I can see I can see and I can see and chart. I can see and point

25:16 it to the current working directory. Yep. And it's the same. So very strangely. But maybe just you can just go with the Promscale vision and after I'll try to set up this environment and try to look at the issue in the meantime. Okay. What do you think? Yeah. And and we do have a backup environment in case in case something like this happened. But yeah. Okay. Yeah. So let me just give a quick overview of of kind of where we are going with the Promscale products in general. We kind of think that the Promscale should

26:16 be the flexible data storage format for the observability data. I should say data storage system. So we want to support, different, architectures in terms of pull and push. We want to support different formats in terms of remote write JSON text, different query languages, different data ages, and by that, I mean, support for backfill. We want to support different data types, not only floats that are traditional for metrics, but shrinkers and symbols that kind of, are used by other systems. You can think of Telegraph or something else and even different data modalities or signals. Right now, we support metrics,

27:20 but we hope to expand with traces, loggers, and events as well. But you might think why why have all this less ability? Why, for example, do you want to put all of your sync signals into one database? Well, databases are hard to run, and it's this thing called state, which is the bane of a lot of our existence. Say this hard. Right? You have to have PVCs. You have to have proper backups, high availability, failover. In addition to just state, there's the regular problems of dealing with the security with scaling and tuning. So and the valid question is why deal with

28:20 this multiple times? If you could have just one database where you throw all your data into, isn't that preferable? In addition to kind of the ease of use story, There's also an advantage in terms of analyzing the data. If you have all of your data in a single database, you could use joins between the different data types to do correlations and more powerful analytics and prediction. So yeah. We have some resources for you here, but I'm sure we'll also have that at the bottom of the video or whatever. Yeah. Yeah. I like those those things to

29:21 the show notes. Don't worry. We'll make sure they're available to people. They can check them out. Yeah. Russell, DMS would be my my first guess as well. But I think with show values, it's probably one of those rare cases where it's not DNS. Well, that's the trouble with the live demo. Right? You never know what to expect. But Yeah. So, Vineeth, do you have any suggestions or should we just share your screen with access to to our Kubernetes environment? Yep. So currently, I I am just removing the existing top setup so that I can start sharing the screen and install the setup

30:19 again and to show it. So just give me a few seconds so it's just getting deleted. Of course. No problem. Okay. I So can I ask a few questions? Sure. Alright. So your your slides have definitely they set a really good scene about where I think all of these things, like connect together. So I just wanted to make sure that my understanding of it was correct. So Promscale is like, I mean, it's really just like a connector, I think you called it, right? It allows you to use timescale DB for the long term storage of Prometheus data.

30:28 Promscale as a Long-Term Store Alternative

31:00 So what you're saying is like, you are aware that Kubernetes exists and people are running Prometheus and we've got open metrics and all of these things that are just automatically deployed. Like, you you highlight Prometheus, which I think is a fantastic example of just the community rallying together to get metrics to be almost commoditized within our Kubernetes. And Promscale just allows you to drop that into the cluster and take advantage and consume those metrics, but while using TimescaleDB on the backend. And I thought that was, is really interesting. Does that mean that Promscale, it just has like

31:37 a limited time to left data? Is it like three days, seven days and then everything falls back to timescale DB? Promiscule itself is completely only a soft state. It doesn't even save data for minutes. It it actually immediately puts the data into time scale to be itself. Okay. So Promscale still runs in the cluster and that's providing my shorter term storage for things to be consumed through the collectors which then oh, yeah. It uses the remote rate and read API to prompt. Yeah. Okay. So is it is it you could think of Promscale as an alternative

32:23 to Cortex or Thanos or Victoria Metrics or any of the other long term stores. Yeah. And the way that Prometheus works is the remote, right, and point actually pushes data out not instantaneously, but pretty close to instantaneously as soon as the scripts it. So, like, the latency between Prometheus having the data and the data being inside of Promscale, TimescaleDB, or however you want to think about it, that latency is usually less than a second. Okay. Yeah. Yeah. So when you mentioned that I can send you know, an SQL query to from Grafana to timescale. That's just how

33:08 How Promscale Translates PromQL to SQL

33:19 that works. Right? As we've covered timescale a little bit in the past, I'm I'm pretty happy with that now. Now is Promscale is it just forwarding the query to the local Prometheus or is it actually doing any transpilation and then send it to timescale? So on the query side, the PromScale does not contact Prometheus at all. It it takes the PromQL query, converts it to a SQL query that it then executes on the database. It gets the data. It does some post processing, namely some of the functions that PromQL provides, we just reuse that in the goal layer and the connector, if

34:10 you will, and then it returns the results. Fancy. So just a heads up here, David. Can you just share screen? I just think I just found the issue. Maybe we'll just give that right. If not, I'll start sharing the screen and write the demo. So can you just get into the tops repository, like, the web one web? So I I just wanted to point you to an issue there. So there is an issue already filed and we closed it. So it's the cube config permission issue. So can you give your cube config all the permissions,

34:40 like, the seven seven seven or something like that? So just change the cube config permissions so that it's available for everyone. So so should I use my cube config or your cube config? Yep. Your config. Yeah. Your cube config. So whatever you have tried so far. So I'm I just want to say that change the file permissions. So there is an issue with the file permissions with helm. So you just need to give the file all root privileges. So it's a it's absolute issue with helm, and I just went through the closed issues and tops and then

35:19 yeah. Just try to run Helm installments. Yes. Just install here. Let's see how does that work. So it says the same? Yeah. Yeah. It is definitely 777 now. I'll just reset it one more time. Dot q config. At host install. Yeah. I think it may be something different. Strange. So so okay. I I haven't okay. I cannot okay. I just thought of sharing you the issue which was opened upstream and they just changed this cube config file and it was just fixed. So I was just thinking maybe that's the issue to for this thing. Mhmm.

36:16 Installation Success & Accessing Grafana

36:18 Yeah. So what's the cube config you sent me? Is this a cluster where it may work? Yeah. So I just want to install it on my end because only the installation is not working, but you can drive the demo forward. So I'll just share the screen for a couple of minutes to install and set the things up. Yeah. Go for it. So I'll just start sharing my screen. Yep. I think you're able to see my screen now. Yep. Let me just pop that up. Alright. We can see it. Yeah. So this is the Kubernetes cluster which I

37:02 have is in DigitalOcean, and it's in New York region. So there will be a slight latency issue. So just bear with me on that. I'm turning problems installed to the cluster. This has the cube system components, nothing else. So it's just running the prompt install. By the way, like, if you want to install Promscale and the TimescaleDB, we are we do have our tops, like, the Promscale help chart dedicatedly for you. You can directly use the Promscale help chart, and then you can just get started with Promscale. So tops is something like you want to get the complete absorbency stack, then you

37:42 can just get into tops. If not, you just want Promscale with Timescale DB, you can directly use the health chart of Promscale and time scale DB. So as it's getting started there, let's just do get parts to understand what it's exactly doing there. So it says it's installing the observability stack. In in the meantime, actually, David, can you try on your machine running the and passing it to yamllint.com? Okay. So what do want me to do? Sorry. You want me to The show values that we have before that command move it into p b copy.

38:52 Yep. Okay. So we wanna go to the tops chart. We run helm show values. We want a p b copy to my buffer. Oh, yeah. I've got a relative path on my cube config. So let's do Promscale. Yep. P b copy. And you want me to drop this into a YAMLlinter? Yeah. YAMLlinter.com. It is valid. Valid? YAML. Interesting. Yeah. I have. I mean, could it be my cluster? I mean I don't think it's your cluster. It might be like I have a feeling it might be like the YAML library on your system, but I don't

39:45 know actually how to Should we upgrade helm just in case? I have check that. Maybe my helm's a little. Let's see. Of course now we have to wait on brew which is not exactly fast. Yeah. Let's see, do we have upgrade? And they always get confused before between brew upgrade and brew update. But yeah. Yeah. I think d five four is the latest. So either the venture just a bug in that version. Yeah. You can't downgrade with brews. I think I'm kinda stuck with this one. We do have a question from Russell if we wanna tackle that just now.

40:00 Promscale's Vision (Slides)

40:44 Russell is asking, am I right in thinking that timescale DB as a back end could live outside of my Kubernetes cluster? Yes. Absolutely. You can actually, you can have both Promscale and TimescaleDB live outside your closet, but Promscale can live either inside or outside. So it's those two components, you have some some flexibility in. Right. Beneath, did you install, Harish? Okay? Yep. Yep. So you're you're can you just share the screen for me, like Yeah. Of course. Yeah. I'll put your screen up. Hold on. Yep. There we go. So yes. So one click, we just ran this tabs install

41:30 The TOBs CLI

41:37 command and, like, after a couple of minute, it just gives you an output saying that happy observing and what all good installed and how you can manage for time scaling and problems. And in the other tab, when I just took the FPL gate ports, you have all the all the stack installed for you. So just to give yep. So if you see here, we have alert manager, Prometheus, Grafana, and the Grafana DB to store all this extra dashboards and configurations metadata used for Grafana and the timescale DB. So there's a job to run it. And we have Prometheus operator

42:11 cube state metrics, and we do have Promscale and timescale DB running here. So this is the complete observability stack that is offered by the tops. They it just installed it. And once you stack has been installed, you can just do Grafana and get password. And you can just see the password for the Grafana UI, and you do put forward and just log in to the Grafana and just start performing the queries. So from here, I'll let David take over the demo and drive it forward for querying using SQL, prompt queue, and everything. So, David, I just shared the queue

42:49 config file for you with you. So I think you should be accessible you should be able to access this cluster now. Three cluster. I like it. Thanks. I'll keep this one. Mhmm. Yep. Yeah. Alright. I've got access to that. That's that's so weird. I really hope it's not my home version or something similar with that, let's keep going. So you're saying I can run tops and you did Grafana get password. Copy this. And then you said I could just do port forward. Right? Like this. Yep. Alright. So I can go to local host setting. Already I love the top CLI as

43:31 like a helper just to get rid of some of that really annoying repetitive get pods, get labels port forward etcetera. Admin password. Alright. What do we wanna look at first? You might want to look at the kind of the default dashboards that are included. The Cube Prometheus dashboards. Right? Yeah. Do you have a favorite? Let's look at it. Yeah. One thing I just wanted to give a try because I do not want to give up on running tops in your Kubernetes cluster. So I just have a one comment. How can I share it with you? Like, I just want

44:00 Dashboards

44:29 you to run that command, and let's just give a final try. So do we have a chat for weekend? Or If you have your Slack open, my CNCS Slack is there, or you can email me. But, yeah, we don't have a Okay. I'm in. Okay. I am on the CNC of Slack. I'm trying to okay. I'm just sending you the message over there. So for a minute, like, maybe you can run this command by pointing your cube config file, and let's just install a new cluster. So that's the last try I just have. So this is one

45:01 fix I suggested in our Slack community over long, long time ago, and he's he replied, it just works. So I just want to give it a try. So Yeah. Sure. Of course. Alright. So let's let's just close this for a second. Jump back over here. Close the port forward. Let's see. It's just spinning but it's coming. Okay. So, shimod go dash r and it's gonna be the cube config which I have here. Yep. So that's removing read access from group and other. Okay. So let's try again. Make sure I'm on the right cluster. It's working.

46:24 So it's it's a Helm bug that requires the file to be six four four as you have to read access from group and other. So if you just can you open another tab to list the parts? Yep. So it's always the system related configurations, David. Like, it's not the top. So I usually run the top in our CI in Linux, Mac, and every possible environment to test our CLI. And it just passes there, and when it comes to our users, like, users just complain once in a while, and I always see, okay. Oh, boy. It's your configurations

47:02 and all. So it's always there. So do you see any new ports yet? I think you see the new top sports coming up. Yeah. I think you are good to go with your own cluster, there's no need of a back end cluster from my end. Well, good effort tracking than what has gotta be a really infuriating bug. I I I'm really if you had told me the problem was gonna be why does Helen care? And why is there any message to do with your Yeah. Doesn't make any sense to me. Hell, it's a great project. We all have

47:40 bugs in our software, but that one is pretty special. Okay. You got kudos there from Russell. Great work in getting the demo back on. Are back on my cluster tops install has worked. Everyone watching has now learned the hard way that you must check the permissions on your cube config otherwise helm will yell at you and we have already kind of checked, so we think the things are running. I'm gonna see if I can get my own Grafana password there. PB copy, Port forward. Maybe the pod change. Just make sure that your pods are in running state. Just make

48:32 sure. Like, it just takes couple of minutes to start up and all that. Might not quite be ready yet. Where's Grafana? Yeah. Okay. Only one container's ready just now. So we'll give that a few seconds. Oh, there we go. No. No. Alright. So we've got access to all of the Prometheus dashboards. You also mentioned that installs PromLens. That's the project from Julius Vulture. Right? The the UI for Prometheus. Yeah. And we find that it's useful for people getting started with people getting started with the PromQL. There's I think PromQL is a simple language, but there are a

49:35 lot of similarities to it, and Promslens does a great job of visually kind of showing what's going on. Yeah. I'm a big fan of Promslens. I think it's really interesting. Yeah. And I like the way that it just kind of introduces you to the language without it getting too overwhelming as well. I'm I'm just gonna grab some logs on this. I don't know what's going on with our graph. So Just do hyphen c Grafana. Yep. It's the container Grafana, so you need to run hyphen c Grafana to see the log. Yep. So I think now it's running. So it

50:08 says you're able to take the logs. Right? It contains Oh, there we go. Okay. So the database just had completely jet, which is just the job to set up. Yeah. The least little things are sent to test us. However, we're gonna port forward now. Look at that. There we go. Make sure I got the right password. So I think it's usually the trailing slashes. This they get dependent sometimes. So it's the common issue that this keeps in here. And so all these dashboards are from the Kubernetes mix. And so we have dedicated dashboards from the Proms Promscale team. So if you just

50:54 scroll down, you have something called Kubernetes hardware monitoring and cluster monitoring that tag as this Kubernetes. So it is it can it can yeah. These are the blue one blue label. So you can just open one of that dashboard. So it's like an aggregated metrics to understand what is the state of your cluster. So rest all Kubernetes makes sense. Dashboards are dedicated to your stateful sets, demon sets, and all that. So you have different dashboards, and we we have plans, and we have efforts to build more dashboard for timescale DB, prompt scale, observability, and all that. So you'll see more and more

51:28 dashboards and observability joining the family of tops to get more insights into your cluster. And all of these dashboards, they are executing PromQL queries against PromScale directly. So this is not going through Prometheus even though some people, do query Prometheus instead, but that that kind of, has more latency because then Prometheus needs to remote read to Promiscale, which then needs to read from TimescaleDB. So we recommend for people to reduce one hop there and just execute queries directly against Promscale. And is that what Promscale is configured to speak to? Yes. Yeah. Promscale is also configured to Promscale, and you

52:25 can also show the data sources here. So if you just go to the data sources symbol, yep, you can see, like, we have the Prometheus data source. It is you see the Promscale endpoint is being added there, and you have TimescaleDB data source as well. So this, by default, all the dashboards are powered up by the Prometheus data source, but Promscale is the endpoint there. So yep. And I think we can get started with getting started with Promscale doc for SQL queries. So where you can see the beauty of Promscale and Timescale DB and analytics you can get

53:00 SQL for Prometheus

53:01 out of our stack. Alright. Perfect. So do you want to work through this? Is that what you were saying there? Yes. I think Matt will Yeah. We kind of we already kind of went through parts one, two, three because we installed. And so yeah. Let's go through part four. Alright. Let's see. Yeah. So before we do this, we just need to update the time scale DB data source with Postgres as the user because we have Grafana DB as the user. So just get into your like, can you open your CLI, David? I just can you just run talks timescale d b

53:56 get password to get the password of timescale d b? Yep. So, yep, you just need to port forward the graph on again and just copy the password. So what like, we just released to us back end. We were just we just found one bug, a small bug, which is that we just created a dedicated user for TimescaleDB in Grafana, which is Grafana as the user, but it doesn't has the admin permissions to to query the DB. So we are just updating the user role in the TimescaleDB data source so that the queries just work fine. So

54:37 you just need to the user should be a post Chris, and in the password section, you need to reset it to the password which we copied. And just test the connection. Save and test. It's green. Yep. Yep. And now we can just create a new New dashboard. New dashboard. Okay. And just add the data source as the timescale DB. And, yeah, usually, like, you have our beautiful UI to just inject the values, but we already copied. So you can also see the all the metrics available from here here. But can you just click on this

55:27 the edit icon you see here? Right? You can just copy paste the complete query, which we we have noticed from we which we have seen from the docs. Yep. Copy and then edit like yeah. And you just need to paste it here. And can you just take the visualization as table in the right, the visualization? In the right, you have at the right section, panel section. In the panel, you have visualization. Yeah. Yeah. Yeah. Okay. Yeah. And this selected table, like, the sixth one. Yep. And you just need to get the data for this and the table, but okay. You need

56:07 to add prom metric dot to the table. So it's in the scheme of prom hyphen metric. So prom Prom underscore underscore metric. Yeah. And then just click somewhere else. Like, the enter doesn't work here, so it's the issue with the Grafana UI. So you just need to click on the outside of the query. Yeah. It's not. So okay. You need to select instead of time series and format it as just select the table. Oh, yeah. Yeah. There we go. So yep. So we have the data here. Matt, I think you can just talk just go ahead with all the other queries

56:53 we have in the doc. Like, each query has a background and interesting story. Yeah. Yeah. So this is kind of allowing you to query the data point. It's a little bit isn't quite that interesting. I will point out that that we allow you to ingest data both via Prometheus and via direct push. And when people ingest data via direct push, sometimes the PromQL semantics don't quite match because PromQL expects data to be regular. And so this is one way where you could just get the raw data for all of your points directly. But this is a simple

57:56 example just to give you a study, but then if you go back to the tutorial, there are more advanced queries you could do. So for this is a good example, actually, of an advanced query. So So this is doing a percentile Yeah. Which the 0.5% of oh, so okay. So it's trying to work out what the 95 ninety fifth percentile is of the duration of the goal runtime garbage collection in seconds over the last five minutes. Yeah. This is kind of hard to see. It's zero dot five. So this is actually the median. Oh, zero oh, yeah. Okay. Okay. Yeah.

58:48 So this is actually getting the median of the duration. And the one thing I I will point out is that this is a query you cannot do in PromQL. Because in PromQL, everything has to be done on a per series basis before you aggregate it, and this is doing kind of a global median. And these kind of queries are actually quite important if you're doing things like if you're doing things like capacity planning or deeper analysis of your system. For capacity planning, you might switch this. David, as you already said from a zero dot five to zero nine five,

59:45 but it's the same basic idea where you want to get your global kind of percentile instead of what the Promscale forces you to do, which is a double aggregation. First, you aggregate on the per series level and then you aggregate globally, which isn't quite the same thing statistically. Nice. Okay. So this is kind of showing already the value of doing some stuff in SQL. Yeah. That that makes a lot of sense to me. Because I think, you using SQL SQL as a language that most people have been familiar with for their entire careers as well.

1:00:43 I know that people at least I don't. I speak for myself only here. I didn't enjoy learning PromQL. In fact, I hated it. I really really hated it. I worked for Influx Data, Influx DB. I hated learning Flux and I still hate writing Flux. You know, I've I've got twenty years of knowledge of SQL in my head that I wanna be able to use for these kind of things. And I I love where Promscale is fitting into this architecture like, you know, like we said at the start, you know, is now the de facto way to get metrics out of a Kubernetes

1:01:15 cluster. I want access to those living dashboards that are available, but I don't particularly wanna use Prometheus for my long term storage and be able to, you know, just augment and then drop in that timescale DB and prompt skill set in the middle. I think it's a really interesting architecture that just opens up more possibilities for people to have better observability longer term as well. And something that I can't remember who mentioned it, sorry now but you you talked about push as well. Like yes, we've got Promscale, we've got Prometheus doing the pool within the cluster and writing using the remote

1:01:50 API to Promscale. But I can push directly event driven or events or traces right into timescale DB and increase that observability within all of my systems as well. And I think that is a superpower. That is this really really cool. So Yeah. What do what what should we take a look at? Let's see or just pick one or two more things that we can kind of we can draw off. I think it's really interesting to me that the Promscale does query translation to to SQL. So maybe we could take a look at that. But you saw the experts.

1:02:24 What what would you like to show? No pressure. Yeah. Let's just just let us see which actually, let me see if there's any other queries in that in that tutorial that we want to highlight. It looks like the third query here, is that because I don't have the labels as JSON. What's your JSONB function do? Yeah. So this takes the label and then the return of them as as JSONB objects. So this is if you want to do, like, queries using using per series analysis. Right? You're getting now back your your time series values directly

1:03:31 along with the labels that you are familiar with. Nice. Yeah. I'm thinking what we can show you in terms of the translation. Oh, so I think going through the the entire transpilation process is is gonna be cumbersome, but I do want to point people to a to our GitHub page. And on our GitHub page let me just make sure that there's a link. So if you go to the GitHub page, there is a There's a link to our design doc Oh, yes. Which is Yeah. Which is actually a live public design doc that we had

1:04:55 hosted online when we first started the project. And this design doc goes into the details of how we designed the SQL schema to actually store this kind of data. And it's it's pretty technical, but the bottom line is that we kind of have a very unique way of storing these label set that allows a very efficient kind of storage mechanism inside a a prompt q l. Sorry. Inside of a relational database. And so I think if people are really interested in how we do this kind of translation, this is the the way to get that

1:05:49 information. Alright. Awesome. I'll make sure that the design doc here is also in the the show notes. So if anyone does want some light reading with their coffee tomorrow morning, this is a good way to understand how it all works. So Yeah. I think I'll enjoy reading myself. Yeah. I was wondering if you had any any other questions. But just to give you a heads up here in this so if you just scroll down, we we have a bonus section. So it's a very complex query. Like, if you're you can just execute, but it just shows

1:06:28 you the power of SQL if you just go deep down into the metrics. So this is the big query, and you can see like what it it has to offer for you. You can just see That's script. Alright. Let's copy. Let's copy it. Let's see if I can guess where it does then. I'll let you correct me. I mean Okay. Gonna select. It's got a meta max. It's looking at container memory limits where there's a real value. Okay. That makes sense. Now it's then being passed into another select, which is where No, I give up. Go

1:07:13 for it. What is it? I can't juggle that much context in my head at the same time there. Assuming you're gonna pull out with the statistical thing on how much container limits I have on my cluster. I don't know. So if you just scroll up, so there's a small description in one line. It just says that so it just gives you the Kubernetes containers that are over provisioned by finding those containers whose 99 percentile memory utilization is low. So this is one interesting way to understand how your containers are over provisioned. So there is a a big post by Dan Liu about

1:07:51 talking about a simple way to get more values from metrics. So we just went through that post and this query was built by the demon. So you're telling me if I had just scrolled up a little bit, I could've read that verbatim and everyone would've thought I was so smart. And you like I think that's true. Looks there's 30 lines of script SQL query madness trying to work out what it was doing. Thanks for me. Thank you. Anyway But but by the way, I will say that for people watching, even if you learn kind of nothing else from this talk,

1:08:29 go and look at that post by Dan Liu. It really, I think, the power that you might want to get from your from your observability data. And he really describes kind of the the real business value that search analysis has in terms of saving people money. Yeah. Okay. I will throw this into the show notes as well. Lots for people to read. We have tomorrow's coffee. And I ran that query. I updated the from statements to sync with the prompt metric and we can see here that we've got all these containers that have some sort of

1:09:14 max memory allowed. And we can see, yeah, the percentile and all that. Yeah. So How would you rate I'm assuming I mean, you couldn't rate that in prompt q l. Right? That just that just wouldn't be possible. No. No. Yeah. Alright. Awesome. So the so I I guess the point here is that if your percentile used is low, you're probably over provisioning those containers. You could do with provisioning them with less data. Yeah. That makes sense to me. Yeah. Alright. We have a question from Russell. Let's get that on the screen. Russell also says, forgive me. I don't know

1:10:05 a lot about timescale, but can I use all of the tools that I'm already familiar with that already speak Postgres? Yeah. This is one of the beauties of the setup. You absolutely can connect any tool that speaks postgres to this, and you can use any of your available postgres features including, like so time is gonna be as an extension on top of Postgres. You can also install other extensions. You can combine this with, for example, geospatial data using PostGIS. All of the regular postgres machinery just works out of the box for you. Awesome. I can imagine that's quite a high value

1:10:53 selling point to people that are looking for a database for time series data. Like, been able to use not just postgres, but you know, all the tooling that they've been again comfortable with. Very interesting. Yep. Tableau, Power BI, of these things do work with time scale. Nice. Yeah. Alright. Last chance. Maybe just another thing to just just another thing. So we are also working on building more analytic functions for SQL. So what are the queries you are seeing so big so they can be optimized if you start using the analytic functions which will be offered by timescale?

1:11:33 So already we are working and there's another dedicated for project as timescale analytics. So, yep, I just wanted to give a heads up on that too. So you'll have more SQL and more insights to your monitoring data with the analytic functions. Awesome. I've got one other kind of random question that's floating around my head. Just, you know, you're saying that I can use all of the postcard stuff. Know, I've been quite lucky in my career. I've always I think exclusively use postcards. I've never touched that terrible one from Oracle. But they're like, you know, I have access to views and triggers

1:12:05 and PLSQL. Is all of that available to me with timescale? Yes. Yes. Alright. I'm In fact, a lot of those features that you mentioned is how we built promise a promise scale to begin with. The promise scale is a collection of, you know, besides all of the go code and go machinery, it's sequence groups, sequence functions, triggers, etcetera. So even we ourselves are using that underneath the hood. Alright. Awesome. Alright. I'll give you your last chance. Is there anything else you wanna show today before we finish up? No. I would just encourage people to to

1:12:51 join our Slack channel and to we're more than happy to answer any questions that people might have. Alright. Awesome. Well, I hope you've enjoyed having us look at Promscale. It is a really cool way to who can do the existing Prometheus ecosystem that already has a whole bunch of value in Kubernetes, but have a Postgres based long term storage for your metrics, which I think we've seen through the sequel and the conversations today offers fantastic benefits to you and your business. Alright. Matt and Vinith, thank you so much for joining me today. I know we had a

1:13:30 couple of problems. I am not I'm I'm probably gonna lose sleep over that helm thing. It's gonna bother me so much. But we got past it. We got everything working and it was really cool to see it. So thank you both for your time today. Thank you, David. Alright. Have a great day. I'll see soon. Thanks for having us. Yep. I'll see you all soon. Thanks. Bye. Bye bye. Bye.