1:48 Introduction & Guest Welcome

1:48 Hello, and welcome back to the Rawkode Academy. I'm your host, David Flanagan. And today, we are back for another episode of Rawkode live where we take a look at technology and the cloud native in Kubernetes space joined by their creators for the creators, founders, maintainers to give us that inside look to what the project does, the problem it solves, and how cool it is. Today, we're taking a look at a project called Sveltos, and I am joined by Jan Luka. Hello, Adam. Welcome. Hello, and thanks for having me here. I'm Jan Luka. I'm a principal

2:22 engineer at Cisco. I mostly work on distributed systems. And on the side, like, I'm working, like, for many months now, over a year, like, on projects Sveltos, which is what we cover today. Thank you. Awesome. Well, I'm looking forward to learning more all about it. So you said you're a principal engineer at Cisco. Have you been working with Kubernetes for for long now? Yes. I worked at I started working Kubernetes when I joined together. I was there for over a year on the enterprise version of project Calico. Ah. And then I went I was at Cisco, went to together, and

3:02 then went back to Cisco because they start, like, an Kubernetes project at Cisco, and I want to work on it. And, yes, so now I'm back there. Yeah. Very cool. Well, maybe you could kinda give us the the TLDR. What what is Project Sveltos? It's a project to dynamically manage add ons on a fleet of Kubernetes clusters. And the final goal will be that it's a pure intent base. So you can express easily what you want, what type of allowance you want, given, like, the state of a cluster and a bunch of other information and Sveltos

3:14 What is Project Sveltos? (TLDR)

3:46 make it happen for you. Okay. So I have a Kubernetes cluster. I wanna stick something onto it. I have more than one Kubernetes cluster as Sveltos can handle that for us. I mean, I'm assuming when you said fleet, right, that word to me says scale. I have dozens, hundreds, potentially thousands of clusters. But does it scale down well for people that want to experiment and play with it and just run against one or or two clusters? It should scale pretty well. Honestly, I haven't tested, like, with thousands clusters. This will be, like I don't know, like, too many clusters,

4:22 but it's the design. It's it's it's done in a way that it should scale. Now what the point where it's you know, will start having problem scaling, I honestly haven't hit that yet. But, yes, the idea is that you have this management cluster. And in this management cluster, you have, Sveltos running. And then you have a fleet of Kubernetes clusters, which are your managed clusters. The only requirement for Sveltos is that there is network connectivity to those managed clusters. And and from around, pretty much, you can say, like, this is those are the ones that I want to deploy in those clusters,

4:59 and Sveltos can can do it for you. It's a it's it has support for cluster API. Initially, pretty much, like, I was the project we were working at Cisco, it was using cluster API to create Kubernetes clusters on on demand, on prem. And and that's pretty much when I realized that there was a need. At least I felt a need to have management for a fleet of clusters. And I know that there are other solutions there, but I think, like, Sveltos is a little bit different because, like, it's it's goal is to be, like, pure intent based and very dynamic in

5:40 in what it does. You know, it deploys, like, the phones and everything. Nice. Yeah. I think this is a common scenario in Kubernetes land. You know, we have a lot of choice. Right? There's there's individual problems, but there are lots of tools that come at it from a different way. And I think, you know, there's no one right or wrong answer. There's the one that applies for your use case. And, like, the more choice we have to make there, I think it's it's always a good thing, you know, being able to have a like minded solution to other people

6:09 and contribute and especially in open source. Right? Like, it's just a great possession for us all to be in. Absolutely. Yeah. I mean, the cases are every I feel that too much like it's everyone has a slightly different use case. And so there are, like, it's the more product that are out there, the better it is because there is definitely, like, it's gonna be one which is closer to solve, everybody use case. So it's good. Yeah. Thanks. So I believe you have some slides you want to run through to give us a bit more context and information.

6:35 Sveltos Architecture: Management & Managed Clusters

6:40 Yes. Left some slides just to give, like, a context of how Sveltos runs, and then we can go, like, into a demo of of the light demo. We like demos. Yeah. Perfect. Yeah. Okay. So I'll share some slides, and I go, like, in presentation mode. So we know that once we create a Kubernetes cluster, now that is just the beginning because after that, we have at the end to deploy a bunch of add ons to have, like, a production cluster or a cluster that fulfills, like, our needs. So what Sveltos does, the idea of Sveltos,

7:22 is that there is one management cluster. And in this management cluster, Sveltos is running, and there are a bunch of managed clusters. Some of the cluster can be cluster API, power clusters. Sveltos, like, fully understand cluster API concept. And but any other cluster pretty much can be managed by Sveltos. As long as there is network connectivity and Sveltos has a cube conflict to us is like the managed cluster, that's all it needs. The main CRB that Sveltos uses, it's it's called cluster profile, and it has, like, essentially four fields. The first field, which is the most important,

7:53 Cluster Profile CRD: Defining Add-ons

8:03 is the not the most important, which is like it's the the first field is the cluster selector. The cluster selector is just like a Kubernetes label selector. So we have a fleet of clusters. Each cluster says set of labels. These cluster selectors, in this case, is selecting all the clusters which match this label selector here. So any cluster which has a label EMV equals to a fig, which stands like for functional verification. Now those clusters, what Sveltos does, it gets all those clusters here, and then it deploys all the ones that are listed here. And

8:38 we have three different sections. There is an L chart sections where there can be one or more L charts. In this case, what I'm asking, like, Sveltos is to deploy Caverno version 2.6 in any cluster which is missing this cluster selector. There is another section which is a customization section. Essentially, here, you can defer, like, to one or more directories that contain customization file, and Sveltos uses, like, the customized pretty much to take the YAML and deploy the YAML in all the managed clusters. And finally, there is another section which is policy ref, which essentially is referencing

9:20 secrets and config map. And each one of the secrets config map can contain one or more Kubernetes resources that Sveltos is gonna deploy in all the clusters that matches the clutch the the cluster selector. So here I have one example of a config map that, contain in its data section two, Kubernetes resources, a gateway class and a gateway. So, what Sveltos will do if a cluster profile is referenced in this config map, it will deploy this gateway class and the gateway instance in any cluster that is matching the cluster profile that is referencing this config map.

9:58 So this is pretty much the basic idea, to have a single one or more cluster profile where you dictate what add ons needs to be deployed. And this is just like a slide that shows, like, how Sveltos we work. So if I post a cluster profile in the management cluster, and this cluster profile is a cluster selector. In this case, it just has, like, one single chart. And I do have, like, two managed clusters. As soon as I add the label environment production to this cluster, Sveltos will detect that this cluster is not a match for this

10:39 cluster profile. It will take all the add ons that are listed in this cluster profile and deploy those. So will be deployed in this cluster. And likewise, if I add the same environment production label on the second managed cluster, twice exactly the same. Sveltos will detect that this cluster is not matched for this cluster profile. It will take all the that are listed in the cluster profile and will deploy those. So Sveltos can take care of deploying, upgrading, but it can also take care of removing add ons. Because, for instance, if I remove the label environment production for this workload cluster,

11:17 this cluster is now not matching the cluster profile anymore. And so Sveltos, by default, is gonna remove all the add ons that it has installed because of this cluster here. This is the default behavior. It's it is configurable what Sveltos should do when a cluster stop matching a cluster profile. But the default behavior, it will withdraw all the add ons that as Sveltos is deployed. Now I follow you, like, on Twitter, and I think it was a week back. There was a thread that you posted, and I like the statement that you put it there. That everything that your application needs, beyond

11:42 Dynamic Configuration & Data Fetching

11:55 its own declarations should come from the environment in cluster. And I liked it because this is actually what is behind Sveltos' idea. Because, for instance, if I have to deploy Calico in all the clusters which are matching the label, the cluster selector environment production. Now Calico requires to be passed the port CIDR to be deployed. Now I know that if the cluster it's a cluster API type of cluster, there is a cluster instance which represents the cluster in the management cluster. And that cluster instance has exactly this information that Kalliko needs. So what I'm telling here, Sveltos, is

12:46 anytime you find a cluster which is matching this cluster selector here, before you deploy Calico, take the cluster instance in the management cluster, which represents the cluster where you are about to deploy Calico. Take these hider blocks from the, cluster instance, which sits in the management cluster, replace it here, instantiate this value section, and now you can go ahead and deploy Calico. So it is important that sometimes the information that you need when you are deploying add ons in bunch of clusters might be present in the management cluster, and you might wanna fetch it at run time

13:30 and then pass it along to all the managed clusters. Another example, like, by default, like, Sveltos, it's fetches like, you can fetch, like, those resources here from the management cluster. But Sveltos is not limited to that. Essentially, you can tell Sveltos to fetch anything and use it. In this case, I'm telling Sveltos to fetch a resource, a secret, which is present in management cluster, which is called which is in the default namespace. It's called autoscaler. And I want Sveltos to represent this resource with the autoscaler secret ID. And so now this config map here can actually use any

14:12 information which is present in this in this secret. If you see, it's like Sveltos fetches the secret, default autoscaler, it names it autoscaler secret. And now if I look at the config map, the config map, it's actually pretty much using the autoscaler secret and seems like, I wanna take the namespace of the secret, and I wanna take the data token of the secrets, and I wanna take the data section CA CRT of the secret. In this case, pretty much the secrets contains a certificate and a token, and I wanna pass it, like, to the managed cluster because

14:44 I wanna create a sequence in the managed cluster which contains this information. So it's very dynamic because that too much, like, it's it's it's the main goal one of the main goal of Sveltos. The other features that I'm gonna demo, it's it's what I'm gonna cover, like, here in the next two slides. Essentially, so far, what we have said is that the platform admin in the management cluster is the one that is in charge of adding labels to the managed clusters. But, ideally, I don't wanna do that. For instance, if I have two different admins, right,

14:57 Introduction to the Classifier Feature

15:19 I might have one admin which is in charge of deciding which is the Kubernetes version in each of the managed clusters. And I might have a different admin which might be in charge of deciding which add ons needs to be deployed. Now I wanna I don't want to require those two admins to talk to each other. So if I need to upgrade the cluster, I don't have to go to the add on admin and say, is there any action that you have to take care before I upgrade the cluster or after I upgrade the cluster? I want that to be, like,

15:48 taken care of by Sveltos. And so there is a second concept that Sveltos has, which is called classifier. Essentially, classifier is capable of classifying a cluster based on its run time state. What I'm doing, like, in this example here, I have a classifier which is saying, if you find any cluster which is running a Kubernetes version, is greater than or equal to one twenty four, but strictly less than one twenty five, does this cluster here is a match for this classifier, and I want you to add automatically this label to the cluster gatekeeper v three nine. Now if I can if I

16:22 pair this classifier with a cluster profile, which is saying take any cluster which has this label here gatekeeper with nine, and deploy those add ons. Now there is nothing else I have to do because what happens is that as soon as I post those two objects to the management cluster, Sveltos is gonna deploy an agent in each of the managed cluster. These agents here is gonna detect that those two clusters are both match and match for this classifier. Because they are a match for this classifier, Sveltos is gonna add the labels that he has been instructed to add 20 cluster that

16:54 match this classifier. So now those labels, gatekeeper 59, are gonna be added to both clusters. Because those clusters have this label here, now they are a match for this cluster profile. And because they are a match for this cluster profile, gatekeeper gets deployed in all those in both clusters automatically. Now let's say that I am the I don't admin. And what I know is that I'm fine with gate gate gatekeeper 390 if the cluster is less than one twenty five. But if the cluster, it's, upgraded to one twenty five or higher, I want, gatekeeper b three ten. So I can express my

17:33 intent simply creating another pair of classifier classifier. I'm creating classifier that says, like, if there is any cluster which is matching which is running a Kubernetes version better or equal than one twenty five, now I want the label gatekeeper b three ten added to the cluster. And then I say that, if there is any cluster with gatekeeper v three ten, I want gatekeeper n chart three ten zero to be deployed. So now let's look at this cluster here. It's running 124. I post select this pair of classifier and cluster profile to the management cluster. I upgrade the cluster to

18:11 125. So now because the cluster has been upgraded to 1252, this cluster stop being a match for the previous classifier and becomes a match for the new classifier. And because it becomes a match for the new classifier, the gatekeeper the the label is gonna be updated because the label is supposed to be gatekeeper v 310. So Sveltos update update the labels on this cluster only. The other cluster is still matching your classifier. Because the label on this cluster has been updated, this cluster becomes a match for this cluster profile and automatically gatekeeper and chart gets upgraded

18:46 to three times zero by Sveltos. So those are the two main features that I'm gonna demo today. Sveltos is like more of those features here and did the final goal of Sveltos, it's intent based. So it has an event driven mechanism where it can detects events and deploy a don't see response of events as well. Me exit the so I'm done with the slide. Don't know if there are questions or if you wanna make me to jump on the demo. I can go, like, on the demo twice. Yeah. Like, this really resonates with me about the way I want my

19:14 Initial Discussion & Q&A

19:28 GitOps pipelines to work. I love that you can pull out arbitrary resources within the target cluster and use them as part of the templating across the application delivery. Like, we've had, you know, we're all we're typically, we're all developers. Right? We've been writing code for, say, five years, ten years, twenty years, however long you've been writing code. We learn all these best practices. And I felt that every time we were adopted get ups, we were thrown all of that away. Like, you know, the fact that we have CICD pipelines with baked in hard coded environments like dev and staging and prod

19:58 infuriates me and they're doing it because they need to inject ingress host names and all this other fluff. But the cluster can provide that and like I'm sitting here but where has Sveltos been my entire life? Like I need this tool and magically your demo on it today. So I'm really excited. I think this is a really great addition and the way the the classifiers are working as well on top of that as a kind of you know, that ad hoc way like as my cluster evolves like we're always oh, no. Alright. Most people that I have spoken to are

20:31 using managed Kubernetes clusters. Right? They're not using infrastructure as code as strictly as we used to kind of encourage ten years ago. Right? Because JKE, EKS, AKS, they will manage the upgrade for us. Like, we don't even store the versions on our IAC anymore. And as these things are changing dynamically behind the scenes, our IAC doesn't really know how to react to that or do anything with that. And there are get ups pipelines don't have any awareness of that either. And this kind of bridges that gap as well. Guess the question I have Yeah. Right? Because

21:01 I'm excited though and I'm like, I wanna use this, but I don't have a fleet of clusters. Can I run Sveltos and the cluster that it manages? So it's for few things, it's the the real answer, it's I will say it's no because it's limited in what you can do, like, in the cluster itself. But it's actually a good point. Like, I've been thinking about, like, whether, like, you should be able to manage even the cluster where it's running because, ideally, it's feasible. I mean, it's all there. It's just a matter, like, of running the agent in the management cluster

21:40 itself, but I've not gone there yet. So Yeah. Think that might be something to happen to tie on and experiment with because, yeah, it could be cool. Anyway, yeah, no questions right now. Just excitement from my end. So happy to to see the demo and see what we're playing with here. Thank you. Thanks. So what what I have here, it's I have, like, I'm I'm running in the management clusters. And the only thing saved so far in the management cluster, it's cluster API. So and I'm using Docker as infrastructure provider. And I did create one cluster API cluster,

21:59 Demo Setup & Registering Clusters

22:20 which is called cluster API workload. But I haven't installed Sveltos yet. I will follow pretty much the Sveltos. We'll go to the document and we'll follow the document like to install Sveltos. And I do also have, like, two GK clusters, which I did create, which I'm gonna import and let Sveltos also manage GK clusters. So pretty much, like, I can show that both cluster API, power clusters, or any other type of cluster. In this case, GK can be managed by by Sveltos. So if if I go to this Sveltos installation, it's simply as deploying those to YAML. It has, like,

23:08 an end chart as well to deploy Sveltos, but I'm gonna, like, follow, those to simply pretty much, like, deploy these these YAML to the management cluster. So I'm gonna apply the first YAML. The first YAML pretty much contains all the resources and everything that Sveltos needs. I do not have permissions installed, that's why pretty much, like, there is this error here because that was pretty much like if it has Could we be seeing some light settings? I guess those are your slides. Oh, you're still seeing this slide. Oh, I'm sorry. I think yeah. So, like, let me just

23:42 I think I was sharing the okay. Sorry. That's okay. So I'll go here. Okay. So pretty much what let me see. Actually, I'll I'll share the the entire moment. That's why I don't like to keep switching every time. I'll share the entire screen. And it doesn't allow me to share the entire screen. So okay. So pretty much, I'll like I'll have to share and stop and share again. So what I did, I went to the project Sveltos documentation, and this is pretty much like the install section. And this is how pretty much I can install Sveltos in the management cluster. So it's

24:20 just posting those to YAML. And what I did I'll share my monitor now. You know? And want I like to share anymore. So what I did, like, this is my management cluster, and I would say that what I have in the management cluster, I did now right now, like, installs Sveltos as well. So you just like these Sveltos components are running in the project Sveltos namespace. And I do have the, cluster API, which are installed as well. And the, Docker is the infrastructure provider. And I do have a cluster which is being provided using just

25:02 move this a little bit higher so it doesn't bother me. And I do have one cluster which it's been created using cluster API and Docker's infrastructure provider. And I did install pretty much like Sveltos, and now I'm installing the default classifier, which is just need to, like, to make sure that the agent gets deployed in all the managed clusters. Now what I do have here is that I do have also two two GKE clusters, and I do have the cube config of those GKE clusters. And what I'm going to do, I'm going to register those two GKE clusters with

25:46 with Sveltos as well. So Sveltos can also manage those GKE clusters as well. So I'll go back to the management cluster, and I'm going to create a namespace, which I call GKE. And then Sveltos comes with the CLI. It's not required to use the CLI, but it's very handy when you have to do, like, when you wanna see what the ones have been deployed or when you want to register a cluster with Sveltos. So even to register a cluster with Sveltos, the documentation is on the project Sveltos. I can show where that it is. This

26:23 is very annoying to type like this. It doesn't allow me, unfortunately, to share the entire screen full screen. So I have to go here. But if I go back to this Sveltos documentation, there is a section which explains how to register a cluster with Sveltos. And there is, like, an example. Essentially, it all Sveltos requires is to get the KubeConfig to create a service account in the managed cluster, which allows Sveltos to deploy add ons and push, like, create resource on the cluster. And the commendation is just here. And for GKE, there is pretty much like this script

27:01 that I put in there, which does pretty much everything it takes to keep config from from the cluster. Does that work for GKE and EKS where they need, like, that second factor, you know, like, the g cloud? Does it need to get the service account, I guess? So it will work for any cluster which allows you to get the KubeConfig and allows you to use that KubeConfig process the cluster without requiring, you know, any third tool installed pretty much to to us. Sometimes, like, if you use, I think, some of the I forgot what it may be. Like but

27:43 let's say that you are creating, like, a cluster which one of the cloud providers, and they require you to install their own CLI. So even after you download, like, the KubeConfig, the KubeConfig internally, it's calling the this cloud provider CLI to to work. I mean, in that case, pretty much, Sveltos won't work. So Sveltos will work every time you have, like, a cube config that you can pass to Sveltos. Sveltos can load that cube config and automatically pretty much can assist the cluster without requiring any. Okay. Cool. So I'll go back to I apologize for this switching back and forth.

28:20 Don't worry about it. All good. So as I was saying, like, Sveltos has a CLI. So what I'm doing here is, like sorry. It's a that I'm gonna run the register cluster. So what I'm doing, it's I don't ever like to pass this anymore because, like, mycubecom my cube catalog is pointing directly to the management cluster, I can get rid of this. And Sveltos Kaddle is the CLI that comes with Sveltos. Again, it's not required. It's it's it's useful to I find it useful. I use it all the time. Clearly, I'm biased because I developed it.

28:59 But what I'm saying what I'm asking Sveltos is to register a cluster and create the instance that represents this cluster in the GKE namespace, call it cluster one. And the KubeConfig, it's it's the KubeConfig that I store, which is the just yeah. It's called GKE cluster one. So this is pretty much like the cube config that I took from DGKE cluster. So right now, I'm creating this one, and I'm gonna do the same for the other cluster as well. So I'll call it cluster two because pretty much it's how I call, like, on on. It's just like for simplicity.

29:45 And now if I can say that so there is, again, there is one cluster, which is cluster API provider power, but there is also two Sveltos clusters. Sveltos cluster pretty much like are using the cube conflict that they just passed, like, to manage those two g k clusters. Now I have prepared, like, a few I did prepare, like, one few configurations that I wanna show. So there is one cluster profile, which this is a very basic one. So what I'm doing, like, I'm asking Sveltos to deploy in any cluster matching like this cluster selector here, environment function verification,

30:06 Demo: Deploying Add-ons via Cluster Profile

30:30 and also to deploy the content of this content map. And what I'm gonna put, like, in this content map, it's it's a Caverno policy. So I think I download like this. I do have the Caverno policy here. Just a Cavernoplastic policy which prevents any image with the latest start to be deployed. So what I'm gonna do, like, I'm gonna create a config map with the content of this file here. So I'm creating a config map into the following space and then call it, cabernaletas, and the content is coming from the file that we just saw.

31:06 So if I'm gonna look at this config map here, Because essentially the in the data section, it has the cluster policy here. Now before I go and pause the cluster profile, if I look at the I want to add, like, the proper label for the clusters. So I can start doing the, like, an edit of the cluster API, power cluster. And it has few labels, but this is the label I'm gonna use, like, in the cluster profile, environment function verification. So I'm adding this label here, and I'm gonna do the same for the Bell plus clusters that I have. So it's

31:51 Yeah. Still looking at your terminal just as a reminder. Yes. Can can you see me editing? Like, just to make sure that I'm showing sharing the right record No. I I don't see editing. We're still looking at the get Sveltos cluster dash here. Okay. This is a bummer. Sorry. Stop sharing. And even if I switch tab, I think so, essentially, what I did, I I add I downloaded like this cluster policy. Can you see now the cluster policy? Yes. I see it. Yeah. Okay. And then what I did, I created config map that contains this cluster policy here. This

32:42 is like in the data section, it contains the config map cluster policy. And then the other things that I did, I went to the cluster which is created by powered by cluster API, and I added this label here, environment functional verification. Because I added this label because I'm gonna use this label in the cluster profile. So so far, I only have, like, this label here. There is nothing else done. And now I'm also going to do the same for the two Sveltos clusters that I created. So Sveltos clusters in the g k namespace. I think I called one cluster one.

33:27 And I'm going to add the same label to this cluster here. So environment is function verification. And I'm gonna do the same for the third cluster. So environment function verification. So at this point, all the clusters have this level here. Now I do have this cluster profile here, which I haven't posted yet, but, essentially, what this cluster is saying, it's it's cluster profile saying it it's instructing Sveltos to take find all the clusters which have this level here, environment functional verification, and to deploy and chart, and to also deploy the content of this config map here, which we saw it's it's

34:15 just like a cluster policy. So as soon as I post this cluster here, just gonna show something else. There is no cluster profile yet. So I'm going to and there is no cluster summary yet. I'm gonna show what this cluster summary is. But so I'm going to apply the cluster policy. So cluster because I profile with the. As soon as I apply this cluster profile here, what Sveltos does, it's it figures out automatically that there are three clusters which are bot matching. And it creates a cluster summary for every per cluster profile matching cluster. So I

35:01 have three clusters that which are matching this cluster profile. And so it creates, like, three cluster summary. This allows Sveltos pretty much to know per cluster what to do. So if I go and I can use, like, this Sveltos CLI at this point to show the add ons. So I'm simply like there's this Sveltos CLI. It has a bunch of commands. I'm gonna show you, like, some of the commands. And one of the commands is add on. And if I ask to show me what the add ons are, it's gonna display all the add ons that that

35:42 Sveltos have deployed so far. So when I look at it, you know, it's like I can see that so far, he has deployed Caverno version and chart version three zero one at this time here in the cluster API power cluster. And it has deployed a cluster policy, a Caverno cluster policy with this name here at this time, again, in this cluster here because of this cluster profile here, Caverno. Now I think it's still deploying in the other two GKE cluster. It takes a little longer for add ons to be deployed there. It's like but this is, like, now

36:19 it has deployed its Caverno and chart in the cluster one. In the cluster two, pretty much, it's done deploying everything. And the cluster one is still missing the cluster pull up policy, which hopefully by now is deployed. So, yes, now at this point, pretty much, everything is there. And I can see the exact time when it was deployed in each one of those clusters. Are these add ons just CRDs? Like, can you just run cube control get Sveltos add ons as Sveltos add ons? Is that how this work? So no. That's pretty much like it's this Sveltos

36:51 Exploring Sveltos State & CRDs

36:54 card or show add ons. It's it's pretty much just showing you what resources, what add ons Sveltos has deployed in each cluster so far. Okay. So I can actually Could I get that myself with kubectl? That's what I was Oh, yes. Yes. Yes. Absolutely. Yes. For instance, like, if we look let's go to to one of the cluster here. So I have we can take, like, the cluster API of our cluster. Right? So if I do this and I do, like, a QCalow get pods, for instance, you can see that the has been deployed. You see?

37:29 Yes. And you can see that the cluster policy, it's also been deployed. Like, it it's a it's at the Sveltos Kabo, pretty much like it's a it's it it runs in the management cluster and so it's a centralized place where you can go into, like, show me, like, call the I don't so you don't have, like, to keep changing your Qt config, like, to switch from one of the managed cluster to the other. Hey. Can you run That's to control API resources and correct it for Sveltos so I could see what what custom resources that deploys to the management cluster?

38:04 You can show oh, in the management cluster. Yeah. Yes. So pretty much like the resources that Sveltos is deploying so far are, like, it's it's I know. But I can you show me the output of API dash resources For Sveltos? Yeah. Okay. Yes. I just I'm very curious. That's all. No. Absolutely. Yes. Yes. Yes. Sorry. Don't understand your your question. So those are the all the different CRDs that Sveltos has. So so far, the only ones that we are using, it's the cluster profile and the cluster summary. And this is something that we can configure.

38:47 This is something that is not supposed to be configured like Sveltos configures in it on on its own to keep track pretty much of the progress on every managed clusters. And then the cluster configuration, it's if you wanna see pretty much without using Sveltos without using this Sveltos CLI, you can actually look at this cluster configuration here, and there should be one resources per cluster. And for instance, if we look at this one, it this contains all the add ons that Sveltos has deployed and why. And this is pretty much what the Sveltos catalog does. It goes and beats

39:22 this config this cluster configuration here, and it knows how to parse, like, this information here and display, like, in a human little pivot table. Okay. Yeah. That's what I was use, like yeah. You don't have to use this Sveltos Kaddle. Pretty much, you can use like, you can go and look at the cluster configuration, and it gives you exactly the same information that this Sveltos Kaddle does. Except that pretty much in this case, you will have to read the three of these because pretty much there are three different class that are managed by Sveltos. But yeah.

39:50 Absolutely. You don't have to use as as I mentioned, like, Sveltos Gaddle is fully optional. It's not it's not needed. There's a question from the audience, which I'm gonna drop in just now where we're we're chatting. So Harsh is asking, I'm trying to picture the get up story. Is it going to be Argo or Flux to manage the management cluster and let Sveltos manage apps? Is there like, how do you see Sveltos in a landscape of It's actually this is a very good questions. Thank you for the question. I the way I see it is that

39:57 Q&A: Sveltos in the GitOps Landscape

40:23 I would use flux to push the configuration to the management cluster. And once it reaches the management cluster, I will use Sveltos to send it to the managed clusters. The reason pretty much I will do that way is because the configuration that you push to the management cluster, you clearly pretty much wanna have, like, all the advantage of the detox approach, right, where you can do a review of any change. You can have a history of every change. It's you can have your CICD pipeline before pretty much it reaches the management cluster. But once it reaches the management cluster, I think

40:55 the the the benefit of using Sveltos is that Sveltos is capable of fetching information from the management cluster and use this information here to customize or configure the advance that are then going to be deployed to every managed clusters. And for instance, like, if a new cluster is added that is managed by Sveltos, you know, there is no configuration that needs to change the management cluster. Let's say that I create another cluster API per work cluster using Docker as infrastructure provider. As soon as I add the label environment functional verification, it means that this cluster should have and this cluster policy deployed,

41:34 and Sveltos will take care from there. So there is nothing else that I have to do. So I do see to to answer again the question, like, I do see flux as the tool to push the configuration to the management cluster. And once he reaches the management cluster, I see Sveltos as the solution to to deploy. The other reason I would use Sveltos in that case because Sveltos has features like, for instance, you may say, if in any of the cluster, you see that there is this event here, for instance, a service is created, this is the set of add ons that

42:07 they want to deploy in response of that service being created. So with Sveltos, you can do that. So it's more dynamic. Right? You can react to, cluster on time stage changes. You can react, like, to events in any of the managed cluster, which is difficult, like, to express in in a GitOps approach. But the configuration cost to manage my cluster should should should come with plugs if I were to do it. Actually, this is how pretty much, like, I internally do it. Okay. Thank you. There is since we are, in scale, there's one something else that I wanna show. For

42:36 Dry Run Feature

42:41 instance, let's say that now I I wanna make a change. Right? But I'm not sure about what the outcome is. So I want, for instance, to change the caverno from and chart from three zero one to three zero two. But I don't know what the outcome is going to be. Right? What I can tell Sveltos is, please go in dry run mode. Right? In dry run mode, what Sveltos is going to do, it's not gonna make any change to the managed clusters. But it's internally gonna run like the full logic that it will run. So pretty much,

43:16 like, it will run all the steps that it will execute when it deploys a once. It's just like it will skip the last one, which is actually the real deployment to the managed cluster. So if I change this cluster profile and I say it's like, I wanna upgrade but be in dry end mode because I don't know what the outcome is going to be. Now I can use again Sveltoscalo, and I forgot what the command is in that case. I think it's a dry run. Yes. Okay. So it's I can tell Sveltos, show me what will change if I were to commit

43:50 this change. And if you see what it's gonna say is that it's gonna say that the Caverno in it's gonna be the current version 30.1, and we'll move to version 30.2 in this cluster here. And same pretty much like, you know, the cluster. Right? So there would be like a Caverno and Sharp upgrade. But with respect to the cluster policy, there's no change. The object is already deployed, and there is no change. You you haven't changed the configuration affecting this object, so there is no change there. So pretty much, you can review, like, what the you can make

44:22 a change. You can review the change. And if you find the change, you can go back and tell Sveltos, okay. Now deploy it. Or if you're not fine with the change, just revert back to whatever you were. So this is something pretty much like that, especially when managing, like, a fleet of clusters. It's it's tough to and you wanna make sure that only a certain you you wanna make sure what the side effect is going to be. Like, it's it's nice to have, like, this dry run command. See, the sorry. Just related to that. Right? Let's assume

44:54 Discussion: Phased Rollouts

44:58 we wanted to change from dry run. There's also continuous. I've got two clusters about to be upgraded. Say one of them is considered lower risk by the organization. Can we tell Sveltos to apply it to that one first? And if it's successful, continue the rule, like, you know, like a phased rollout of a deployment across the add on. That's a not right now. So the way you will have to do right now is to have different labels in that cluster there. So, for instance, like, the way I will solve it, I will, I will call that cluster there, for instance,

45:32 preproduction and the other cluster production, let's say. Yeah. And then I will have, like, two cluster profiles which are exactly then one matching the preproduction and one matching the production. And then if you wanna actually see the change live in the preproduction, you just make the change to the cluster profile there. You see what happens to the preproduction cluster, and then you go back to the to the other cluster profile and change it there. But it's actually this is, like, it's one of the features that I would like to add. Well, you also tell Sveltos,

46:02 which is the order. If you make a change, I wanted you to go and you are matching, like, 10 different clusters. I want you to go in order. So go first to this cluster, then to this cluster, then to this cluster, then to this cluster. Yeah. But it's it's not there yet, but it's it's a very good point because it's it's something that definitely too much you wanna see. Cool. Thank you. Now the other things I wanna show, it's the the two so the cluster that I have, it's running one twenty six version. This is the cluster

46:25 Demo: Dynamic Add-ons with Classifiers

46:34 profile. While the two, the two GK clusters I don't know if I have this information here, but they are running do not have this information here, but they're running one twenty seven. So they're running at the different, like, Kubernetes cluster. So Kubernetes version. So what I created here, like, I have one classifier. And this classifier is saying that if you find a cluster which is greater or equal than one twenty six or less than one twenty seven, I want this cluster to have this label here, version b one twenty six. And then I'm gonna pair actually no. Actually,

47:15 here, I'm gonna pair this one with cluster profile for one twenty six. So this cluster profile here is saying, like, if you see any cluster with one twenty six, please deploy NGINX ingress version 18. So I'm gonna pause both of those. So I'm gonna pause first this one. All these posts are going to the management cluster. I'm gonna pause this, and I'm gonna pause the cluster profile What's the name? For b one twenty six. As soon as I post those two things, now what you should happen if I go in again and look at the add ons,

48:00 only one cluster, it's running one twenty six. So I expect the NGINX is gonna deploy only to one cluster. The other two clusters should not have, like, this configuration deployed. So so far, pretty much nothing is deployed yet. You know, I just deployed the classifier, so the classifier is gonna get pushed to the manage clusters, and then it's gonna come back saying whether the cluster is a match or not. At this point, I see that pretty much, like, this information came back to the management cluster, and it should say that the cluster is too much.

48:36 Let's take classify your reports. Yeah. So this is back. So now if I go and look at the on support deployed, you see that NGINX now has been deployed only to this cluster here. NGINX is not gonna be deployed to the other cluster one and cluster two because those two clusters are running Kubernetes version, which is 01/1972. So the only cluster which is running a Kubernetes version, which is greater or lower than one twenty six and less than one twenty seven is this cluster here. And so this is the only cluster which is matching this

49:11 classifier. And because this is the only cluster that is matching the classifier here, which is running a version between one twenty six and less than one twenty seven, this is the only cluster where NGINX is gonna be deployed. And and we saw it looking at the advance that, again, like, it's just like nothing else is deployed. This is the only place where NGINX has been deployed. And if I look at the cluster so I get to my edit. Just to, like, get cluster. If we look at this cluster here, we can see that Sveltos added this label here, version one twenty six.

49:52 But if we look at the Sveltos cluster, the other two Sveltos cluster, cluster one, for instance, the GK namespace. No. The label has not been added because it's pretty much like this cluster is not a match. So this is the classifier demo that you can say what advance you want to be deployed based on the cluster run time state. Nice. I wanna post here, like, if there is any question. Otherwise, pretty much, like, I have another features that I would like to demo, which is the event driven framework. But I will post here, like, if there's

50:21 Deep Dive: Classifier Capabilities (Resource Filters, Lua)

50:33 any question. Okay. I mean, what can we do with the the classifiers? Could you run a cube control explain on the spec so we can see what the what that looks like. Yes. So just see here. See what's the filters, and this is the classifier. Okay. So it's this classifier here so I can get the Alright. Is there any other command which is better Yeah. You can run cube control, explain, and then space, c r d name dot spec. I'm like this? Yeah. Perfect. Oh, alright. That's Yeah. You could do that with any nested field that expands. It's

51:37 really cool and my favorite command. Oh, that's perfect. That's great. Yeah. Pretty much like what the classifier you to do. It's we saw pretty much classifying a cluster based on the Kubernetes version, but it can also classify a cluster based on the the deployed resources. So for instance, now so if I if I take this one, it's gonna go farther into that. Right? Yeah. Correct. K. So you see pretty much, like, you can tell Sveltos classify a cluster based on take this group, this kind, take this label filters to filter object, like, of this group kind

52:21 version. You can say it's like if there is a max count or a min count of resources, and you can even pass, like, a Lua script. Okay. So you so you can pretty much, like, cusses, like, any field in any of those resources. And then you can say pretty much whether the you know, like, if this return is true, it means that the cluster is unmatched. So you can really look like at at any deployed can look at the cluster and decide the the classified cluster based on the resources that are deployed there. So it's it's Yeah. So,

53:00 like, using our example from earlier, if I've got let's just say a WordPress website. Right? And I don't want to deploy to the cluster until there's a config map that exists with everything I need, like DNS name and TLS or whatever. I would add a classifier that monitors for that config map and then when it exists, I deploy my application. And I think this loops right into a second question from Harsh. He says, is there a way we could order the app or resource installations? For example, make sure we install project x first because why I need CRDs,

53:32 etcetera or webhooks and a voice back off. I guess you would use the classifiers and the resources to do that too. Right? Maybe? Yes. Potentially, you can do with the classifier exactly. Pretty much you can say pretty much that when you see that this CRD installed, then at this label, which it's causing the cluster to be a match for a cluster profile, which contains your second set of accounts that you want to be deployed, and then you can keep, like, piping into that so you can have, like, a multistage deployment. The other things is, like, if you have

54:08 if all your resources are, let's say, in charts or customization file or resources, like, instance. If you put if you have two hand charts that you want to deploy and there is Sveltos is gonna honor the order that you put here. It's gonna first deploy Caverno. And if here, like, I have, like, a second after Caverno, here, I have, like, a second hand chart. So the order in which Sveltos is gonna be the it's gonna deploy the resources that they first deployed Caverno, then it passed to the second one, and then it passed to the third one, and

54:40 so on. And same to successfully deploy before moving through the list? Yes. Yes. They will successfully deploy. Otherwise, pretty much like if if you're three, like, it's it will pretty much first we have to successfully deploy before the third one, like, it's it's installed. Otherwise, we'll just before customization refs? Oh, no. Essentially no. So, essentially, the order is within any section. So Right. End charts because pretty much the way Sveltos does it, it's that to be fast, what it does, like, it's it it it takes, like, the end charts and deploys the end charts. Sorry. And then in parallel,

55:21 it takes, like, those objects here and tries to deploy this object here. And then it takes something else and, you know, the customization file and tries to deploy the customization file, like, in parallel. So those things do a customization for all of my CRDs, that would have to be labels because Yes. And then everything else that would be ordered within helm and customized within the cluster profile. Okay. Got it. Yes. Absolutely. Yes. Yeah. Or you pretty much will use the classifier, like, to to put the order there. Yeah. Got it. Nice. Yeah. I think I would have a cluster

55:55 profile for CRDs, deploy them. I have a classifier that one or us for the CRDs and then applies the labels that I need for my applications. I think that's a pretty nice workflow. Yes. Thanks. Okay. You said there was one more thing you wanted to demo? Although we've got a comment from Harsh just saying that's super cool. So there we go. Oh, thank you. Thanks. Thanks. Thank you. Glad to hear that. Thanks. Yeah. The if there are, like, more minutes, I wanna show, like, the event driven framework. So as I mentioned, pretty much, like, Sveltos has

56:25 Event Driven Framework Demo

56:31 the ability to also deploy add ons in response to events. So for instance here, like, I'm gonna go, like, super fast on this computer. Like I said, there are, like, only few minutes left. But, essentially, there is a concept of event source. So in this case, what I'm saying is, like, it's it's if there is service with this label here, Sveltos functional verification, this event, this is an event. Right? And now I have the second CRB which says, like, look for this event here. This is like this is referencing this event here. So it's saying, like, look for this event in all

57:05 the cluster which have this level here. And if you see this event happening in any of those clusters, I want you to deploy the content of this config map here. And the content of the config map is a network policy, right, which is using some information coming from, you know, the service, which is matching event. So I will deploy this going to the managed cluster. I'm deploying this into the managed cluster. Now if, again, if I look at the add ons, nothing is changing because so far, too much, I have, like, no such service deployed anywhere.

57:43 But now I do have okay. Yes. I do have a a service example. You know, it's very stupid example, but the important part is that this is a service which is has the right label, so it's matching the event that I defined. So for instance, I can deploy this one to the oh, no. Cluster two. I'll just deploy it to cluster two. Once again, nothing is there yet. It's like there's nothing. So now I'm going to deploy this one to cluster two. So to config. Git config. Git git Config. K. I'm gonna go let you keep

58:29 a dollar tie of this example service. So I'm deploying this example service into the config. cluster two. And now I what I expect is that Sveltos should detect that this event has up and, like, just in this cluster here and should deploy, a network policy in this cluster. So just gonna give it sometimes to So how is the case a little low. Event add on different from a deployed resource classifier? Like, how what's how do you know when to use either one? It's it's it's I would say, like, it's the the only difference, it's whether

59:12 you want to I would say that you use the classifier when you want Sveltos to manage the labels for you. And you use this event mechanism when you don't want Sveltos to touch the cluster labels, but you want Sveltos to see deploy your own in response to events. So because for instance, if you look here, right, at the event based alone. Right? I'm not telling Sveltos to add any labels. I'm saying, like, look at all the cluster which add this label here, environment functional verification. And if you see these events, then deploy those around here. So the difference is

59:50 purely whether you want Sveltos to manage the cluster labels for you or you simply want Sveltos to react to events. That is pretty much what will say the only difference. The outcome is the same. Adults are deployed in response to cluster runtime state changing. Okay. Got it. So if we look here, yes. Now we see that pretty much there is a network policy which has been installed just like in this cluster here and nowhere else. Because, you know, Sveltos detected that there is the this event has happened, like, in this cluster, and so it's gonna deploy the onset

1:00:26 we told Sveltos to to deploy in the cluster. What happens if you delete the surface? It should delete this add on here. So we can do that. So we can apply it and then delete. No pressure. But the service the service has been deleted, and, hopefully, we should see that, yes, my Sveltos detected, but it's not too much anymore, and so I removed, like, the network policy from the cluster. Yeah. So behind the scenes, that event add on detected the service, and it looks like it created a class a dynamic cluster profile. Is that right? Yes. Yes. Absolutely. Yes. You're

1:01:07 right. When pretty much like it created, like, a cluster profile with the list of add ons that you listed in the event based add on, and then you deploy those ones here. Because you already know pretty much, like, in the the the cluster work to deploy. But it did deploy just in that cluster. So it cleared, like, a cluster profile just for that cluster there. Very cool. I like that. Thanks. Yeah. That was it for the demo. I'll stop sharing so I can see your face again. Okay. Harsh asked the final question and was just

1:01:33 Post-Demo Q&A (Multiple profiles, conflicts)

1:01:45 curious if we have multiple cluster profiles to those run-in parallel if the labels exist. They do run-in they do run-in parallel, but Sveltos is an internal mechanism because if you have multiple cluster profile, If you have no misconfiguration, that's fine. They will done in parallel. They will deploy all things in parallel. All good. If there is a misconfiguration, for instance, you have a cluster profile that says, like, deploy Elm Caverno, Elm version three point zero point one in this cluster, in this namespace. And you have another cluster profile that says in the same namespace, in the same cluster

1:02:19 deploy Caverno version 2.7. Clearly, that is a misconfiguration, right, because you can have only one, right, in the very same namespace. And so what Sveltos does is the text that that is a misconfiguration. So the first cluster profile can go and deploy the cavernium charts. And on the second cluster profile, you will have, like, an error message saying, these other cluster profiles managing this cavernal chart. So you wanna make sure. So it's actually pointing you to where the misconfiguration is, where the conflict is. So it gives you the right information so you can easily fix the configuration.

1:02:56 Cool. But they do run-in parallel. Nice. Yeah. It's it's a really nice project. So thank you for coming on and sharing it. I've kinda mentally went through a bit of a journey. At the start, I thought, you know, this is and they get up space. It competes with flux and Argo, but I've kind of finished thinking that's not where it is at all. It's a completely different layer. I actually see a world where I could have Sveltos doing the classifiers, labeling clusters and then as part of the cluster profiles deploying, you know, flux resources or Argo CD resources to other

1:03:32 clusters as well and doing a handoff to a certain way. And I think the really neat thing about Sveltos is that dynamic nature, the classifiers, the event based add ons, even the ability to inject a little bit of Lua. I love that as well. Like, I'm a huge Lua fan. So, like, all these things working together That's great. Yeah. Just give you this really nice little dynamic run time on top of the fleet or Kubernetes clusters. Something I haven't seen like, I don't know if you remember the the gate project from like five, six years ago.

1:04:01 It came out of Microsoft and it was like a event based run time JavaScript framework to handle and react to events within the Kubernetes cluster. And Sveltos is bringing the best bits of that back out in a really neat way. So yeah. Thank you. Thank you. No. Thanks. Yeah. I mean, I like as I mentioned, like, I've been working, like, on distributed systems. So when I start with Sveltos, the idea was I've seen, like, the set of problems, like, in a different context, you know, like, a a data center. But but some of those problems here, like,

1:04:17 Future of Sveltos & Name Origin

1:04:30 can be easily applied to Kubernetes fleet of clusters if I say that the controller that sends the controller is the management cluster and the all these switches are the separate, like, Kubernetes clusters here. So, like, some of those problems here pretty much can be brought into this context from different context and then soft here. So and it has, like, other features, like, also, like, supports, like, multitenancy because sometimes, you know, if you have a fleet of but in these clusters, sometimes you might wanna say it's like, you can manage all the clusters which are with

1:05:01 these labels, and I can manage, a different set of clusters. And so at that point, there, pretty much it has a way to say is, like, if I'm configuring something, it verifies that I'm allowed to deploy those accounts in those clusters, whether, like, US platform admin has given me as a parent admin permission to do those things or not. So Sveltos like supports it as well. Cool. I think we have two more questions, one from Harsh and one from me. So I'll throw Harsh's up first. How lightweight can the management cluster be? Would it work on

1:05:30 any distribution? Connect a bunch of operators to it without worker nodes. So, like, can they run this on just a single node control plane setup? So let me see how many, like, nodes I do work here. In the management cluster, in this demo here, I was doing, like, three clusters, and I had only one worker node and one control plane nodes, which is running on a kind local. So, like, it's even, like, limited the amount of CPU and memory that you can take because otherwise, was like nothing else would work on my laptop. So it should

1:06:02 honestly, like, again, I haven't tested this scale with more than tens of clusters. So I don't know if you go, like, into the hundreds what the problem would be. Ideally, the design internal is done in a way that eliminates the amount of operations that it does in parallel, so it doesn't require, like, too much operation. For instance, you can configure, like it has this concept of a long queue job. Deploying an advance is a long queue job. Right? So you can tell Sveltos the maximum number of long queue operation that you can have in parallel. So

1:06:33 if you have, like, 100 clusters that you're managing and you're managing, like, one of configuration, pretty much like if you say it's, like, 10 long queue operations can go in parallel, that's what Sveltos gonna do. Like, it's gonna run 10, and as soon as one is done, it was gonna take the next one and do it. So it potentially has the way, like, of scaling well because but even Tesla, like if someone is is, like, hundreds of clusters and want to try, like, a gladly get any bug and fix it and improve it. But I don't have, like,

1:07:02 that many clusters to to try. Alright. Awesome. I hope that answers your question, Harsh. So let's finish up with why don't you tell us what's coming next for the the project? And if you wanna share, where did the name come from? The name come from because I was discussing with another guy, which is and he mentioned pretty much, like, he loves boats, and he told me that Sveltos in ancient Greek means a fleet, and that's pretty much, like, why I took, like, the project Sveltos. It's like, oh, yeah. Managing, a fleet of clusters. It means, like,

1:07:44 fleet at least unless someone I I trusted this guy, so I hope that I'm not it's the right information or not. With respect to the set of features, I do have, like, list I can tell you, like, the one that I'm currently working on. Essentially, I want Sveltos to also take care of let's say that you are deploying a deployment or a stateful site with Sveltos, and it's mounting a config map or a secret, which is also might also be deployed or not by Sveltos. What I want you what I want Sveltos to do is to detect when

1:08:22 the config map, it's changed and reload the trigger a rolling upgrade of the deployment. This is actually a feedback that I got, like, from the team. So, you know, it's a thank to them, Dario and team. And it made a lot of sense, so that's the one I'm working on pretty much. Like, so you don't have to worry about if you have a a certificate in a config map and a deployment which is mounting that certificate and you rotate that certificate. So Sveltos will be able to automatically trigger a rolling update upgrade of your deployment

1:09:00 as soon as this the text that the config map is upgraded in the managed cluster. So you don't have to worry about that as well. Because I I think, like, all those problems here are every time we have to ask the admin to do something in response to another event, it's when it becomes a real problem because, like, at least I have, like, a short span attention, and I I will pretty much, like, mess it up, like, unless there is Sveltos or something else behind the scene that takes care of Okay. That for me. Alright. We had a

1:09:35 Final Q&A (Multiple Helm Charts, Rollback)

1:09:35 question sneaking at the end there that'll throw up in the screen. And I'm really sorry. I don't wanna try and pronounce that name because I'll I'll make a mess of it. But the question is, can a cluster profile use more than one helm chart and how does Sveltos behave in the case that it fails to deploy one? Does it roll back all the changes? We kinda covered this a little bit, but maybe you could go into it in a bit more detail. So, yeah, I see, like, his name is Julio. Julio. He can yeah. He can deploy

1:10:00 he can reference more than one end chart. And the order, it's sequential. So if you have, like, three end charts, we'll deploy the first. And only if the first is successfully deployed, it will deploy the second. And only if the second is successfully deployed, it will go on the third one. Let's say that I deployed the first one and the second one failed. No. It's gonna it's not going to roll back. It it will keep retrying, trying to deploy the second one. So it will deport what the error is. So if any error pretty much that he sees, like,

1:10:35 it will deport back in the cluster summary so you know exactly what it is spelling, but it won't roll back. So it will be, like, up to the customers. It's like, okay. First one succeeded. Second, didn't succeed. So I'll just delete the cluster profile, which will be drawn on the deployed ones. Because the idea is that it's they keep retrying stuff. You know? The error may be temporary, and it will report the error on them. If we can move forward, we move forward. Yeah. It's Kubernetes. Everything's eventually consistent. So Yes. Absolutely. Alright. Well, thank you for taking time out

1:11:07 Conclusion & Goodbye

1:11:11 of your day to to join us, preparing that demo, going over the slides, all of that was was fantastic. It was great to see the project. I definitely have some use cases that I'm gonna have to kick the tires on it and play with. So, yeah. Thank you for your time today. Thank you. Thank you for having me here. Thanks. Alright. Thank you everyone for watching. We'll see you all next time. Have a great week and a good weekend. Adios.