2:27 Introduction and Agenda

2:27 Hello and welcome. Today's today's episode, we're taking a look at Fleet and k c s by Rancher Labs. You can't laugh at me in the first ten seconds of that. Sorry. I was already abused by the intro and then the error message appearing there. Yeah. I I really like that when I first did it and I'm kinda like, I'm getting really bored of it, but I have seen it every day many times. Anyway, we're joined by Bastian today from Rancher. We're gonna be taking a look at Fleet k three s. We're gonna be spaying up Kubernetes clusters and

3:00 having a bit of fun and seeing what these tools provide to this common workflow. And hopefully, make all of our lives a little bit easier. So hello, Bastian. How are you? Very good. And thanks for the invitation. It's great to be here. Yeah. I'm really excited for today. So before we pop over to Bastian, I have had some issues with audio lately. So if anyone's watching live and I can see a few of you now, please just comment and tell me if my my levels are too low or too high. Because yesterday, someone informed me my levels were too low,

3:30 an error in five minutes into the stream. So let's not make that mistake again. Now for now, what we're gonna do is, Bastian, would you like to share your screen and just give us a quick introduction to the tools we're gonna be playing with today. Certainly. Awesome. Okay. I can't see any you anymore. Am I sharing now? You are. Your slides are up. We're on the side. Perfect. Great. So what we wanna look at today is how you can manage multiple Kubernetes clusters running everywhere up to thousands of them with k three s Rancher and the project

4:00 Slides (What is Rancher, Fleet, k3s)

4:06 we called Fleet that is gonna be introduced in Rancher two dot five, which is scheduled to be released next week. And just to kick off off kick us off, I wanna quickly talk, like, five minutes or so why you actually would want to do this and how this all ties in together. One of the nice things about Kubernetes is that it's a common compute platform that you can use everywhere across any infrastructure. So you cannot only use Kubernetes to run it somewhere in the cloud or in your data center, like, with one, two, three, four clusters

4:39 to, yeah, power your websites and your APIs. You can also use it anywhere else from your development laptop to edge use cases where you run Kubernetes in factories, in shops, in windmills, in five g towers, wherever you have something where software is running. And as a developer or a DevOps person, the huge benefit you have from that is that across all these different infrastructures where you wanna deploy applications and software too, you have a common API to orchestrate all these deployments. You have a common API to care about backup and recovery, auto scaling, networking, service discovery,

5:25 role based access control, low balancing, and so on. And all the differences of all these different environments and infrastructures are evened out, and they'll all look the same for you. Still, from an operational point of view, you still have the challenge that you actually have to manage all these Kubernetes environments on all these different infrastructures. And whoever all of you who maybe tried to set up Kubernetes on their own at some point in time probably discovered that it's kind of complex to do. Kubernetes consists of multiple different micro services that all talk with each other. You have to

5:59 What is k3s? (Lightweight Kubernetes)

6:06 manage then each the d historic mechanism in a highly available way that also means more than one instance. You have to care about certificates and networking and persistent storage. And then even just orchestrating all the different components of Kubernetes and making sure that they are configured in a consistent way and can communicate with each other is challenging. And then, of course, you also have to upgrade Kubernetes at some point in time that can even make it more complex. To make this easier and especially in these use cases where you wanna have hundreds or thousands of clusters on all your different

6:45 edge devices, at Rancher, we created a Kubernetes distribution called k three It's basically a Kubernetes, and it's also completely standard compliant. It is certified to be a standard compliant Kubernetes, so everything that works with Kubernetes can run-in k three s. But the difference is that all the different components are bundled together in a single binary. That means all the Kubernetes microservice the services are now one binary, and it also includes either for a single node use case, an embedded SQLite database for storage, or for an use case, an embedded ETCD storage. It also directly bundles a container runtime, so

7:26 you don't need to have docker installed somewhere or container. It bundles a c and I plug in, DNS service, ingress controller, everything you need to directly get started. And because it's also all bundled in one binary, the whole memory footprint of running k three s is a lot lower than when you run all the single microservices. So, usually, you can get by with less than 512 megabytes of RAM for server, k three s server, and with less than 200 megabytes of RAM for a worker agent. And this also makes it a very good tool to run-in edge scenarios

8:05 where not only you have tons of clusters, but also these clusters are limited in computer sources that you have available. And, of course, being a single binary deployments are also very fast and very easy. So that can solve actually running Kubernetes on all of these edge devices and all of these small micro devices. Ah, a good point is also k three s also runs on ARM. So if you have a Raspberry Pi, it also works perfectly. But still, you have thousands of clusters, and somehow you have to deploy them, you have to provision them, you have to do backups

8:46 and recovery of them, you have to monitor them, and also you have to roll out authentication policies and so on. Of course, you can write your own Terraform scripts, Ansible playbooks, puppet modules, or whatever for that. Or and that where Rancher comes in, you can use the Rancher management server for this. With Rancher, you have a central management server from which you can provision and manage up to thousands of Kubernetes clusters on all the different environments and infrastructures. And then you have a central place to not only provision them and upgrade them, but also to set up authentication,

9:05 What is Rancher? (Multi-Cluster Management)

9:27 security policies, monitoring, and so on. So so just like a rough graph, what we are gonna set up in the next hour or so is we are gonna set up Rancher. Rancher is actually also just a workload that runs in Kubernetes. And from there, we are gonna create multiple clusters. And, of course, they are all on some VMs, but they could be running on any kind of edge infrastructure with different sizings and different size of nodes, join them to Rancher, and then we can manage all these clusters from there. So that solves managing clusters, but still we have another challenge.

10:11 We also if we have thousands of clusters, it's actually not trivial to deploy our our our workloads or applications or software to these clusters. Of course, we can write script for that scripts for that, and it's a lot easier because we have a common API to do these deployments. But if you're imagining a script that deploys to 1,000 Kubernetes clusters, first of all, it probably will take quite a long time. You have a scalability issue. Then you also may have a connectivity and fault tolerance issue because not every one of your edge clusters on a windmill, for example,

10:46 will be available when you wanna deploy something. You have to then care for that and handle all these different problems. And we thought, okay. Since edge and these edge use cases are very important, a lot of our customers are going into this direction. Let's make this easy as well. And there is their fleet comes in. Fleet is, first of all, a standalone project, but it's gonna be fully integrated into Rancher two dot five. And what it allows is to do multi cluster GitOps deployment on scale. The way it will work is you have here down here all your clusters that you

11:24 attach to Rancher. So Rancher would be this year, which also directly includes Fleet. You can organize your clusters in groups and use labels to tag them, and then you can tell with a GitOps approach, hey, Rancher and Fleet. Here are my applications, and these applications are called bundles in Fleet. And I wanna deploy this bundle with my applications and helm charts or Kubernetes YAML files or customized files to these groups of clusters. And you just do one API call to set this up. And then these clusters down here, they will pull the information asynchronously and update themselves.

12:04 So you have a more scalable and more fault tolerant solution. And if one of the clusters currently is offline, as soon as it comes online, it will connect to the manager again and pull down the new information what it should be running. So, yeah, that's was my short introduction. Let's maybe then get into actually setting this up and doing this. Or do you have questions already so far? Okay. Let's see what we have. So we have a Hello Bastion from Chandrika. We have a nice wave from Siam. Nice to see you again, Siam. Very good. Who also has a little compliment

12:45 there for you. Fleet is awesome. He has been Also, we're looking forward to get some feedback then about it. Yeah. Okay. Oh, and I think is that a question? Alright. Let me just pop that up. So they're saying that on a single Raspberry Pi at home, they're running Docker and Cortina. They have seen k three s, but wasn't sure whether there were enough benefits to run on a device. It depends on how much RAM you have. So Kubernetes and also k three s needs some of it. You have probably a benefit if you have multiple of the Raspberry Pi's. If you have

13:21 just one, then you can also just do docker run and Kubernetes doesn't give you that much benefit besides playing around with it and maybe getting used to Kubernetes. But it also depends on the Raspberry Pi. If it's a one gig gram Raspberry Pi, it's probably still very limited what you can do, and you waste too much waste too much memory for it. If you if it's a larger four gig one or so, then it's different. Yeah. I think that's that's good advice across the board. If you've got a single node, you probably don't need to be running Kubernetes

13:51 or anything like that. The difference is if you have 1,000 of these single node clusters, then it could be interesting again. Yeah. That's a good point. Yeah. I keep, you know, Rancher at the management platform for all those clusters. It's cool. So let me share my screen now and show you where we're at. So this is the the hands on portion there. So in order to do that, we have to provision a little bit of hardware. So through packet, I have provisioned as Rancher one, two, three, four, and five. These are all Ubuntu 20 o four machines.

14:00 Starting to gets hands on - a look at our hardware

14:23 All x 86 m d 60 fours. Good to go. Now depending on time constraints and how we can run today, I've added in a couple of extra machines that we can maybe try and take a look at. I added a CentOS eight box and I have added an ARM machine as well. So we can maybe see that management playing across the disparate clusters. So the first step is installation, I guess. Should I just pick the first box SSHN and then get started, or would you recommend something else? Yeah. Let's just pick the Rancher one box and install a small k three

14:50 Installing single-node k3s on our first machine

14:59 s cluster on it and install Rancher on it. This is, of course, not a highly available installation. Normally, you would need, like, three nodes for this, but we are just playing around. And the process of setting this up highly available is very similar. What you mean playing around? That's that's his prod for me. That's okay. This cluster is stuck in a row forever. Okay. Alright. I am on Rancher one. I do have some documentation here as well. So this is the k c s docs. Is this where I start? Yeah. You can either start with this one

15:29 or the installing Rancher one. It has the same commands to set up a k three s cluster. And that's This one. This one here. Okay. So we can probably skip the first step on the left side. There are, like, sub pages on that. So we set up already the infrastructure. So the first thing we do now is set up a Kubernetes cluster. And this is a single node installation we're doing. Correct? Yes. And then we already have to adapt the first command a bit because we are doing a single node installation with just the embedded

16:01 SQLite database. We don't need to point it to a MySQL database for storage. So one of the nice things also about k three s is that you can point it not only to the etcd or to to an embedded ETCD or to an embedded SQLite, but also to external SQL databases for storage. Especially, a cloud provider that could be interesting. Instead of having to run your own ETCD, you can just fire up an Aurora or something like that and point then k three s to it when you set. Nice. I like that. That flexibility to pick

16:34 the the back end store is pretty cool. So you're saying all I need is this curl s h server and then we're maybe gonna add something to that? For now, I would say we use the default version so we can yeah. That should be fine. Alright. Yeah. So we removed the data sorting. So that means that it's using by default an embedded SQLite. And what it's doing now, it's basically downloading what the script is doing. It's seeing, okay, which operating system are you running on, downloading the correct binary. In this case, Linux m d 64

17:14 binary, then setting up some swim links for kubectl and so on. It creates a script to uninstall case regs again, in this case, because we're in a bundle sets of a system d service. Hey. Cool. I mean Yep. So that's what was this? Like, twenty seconds, thirty seconds to set up a Kubernetes cluster? Yeah. I thought we were gonna have to, like, you know, make a cup of tea, have a better chat back and forward. Right away, I've got, like you said, under thirty seconds, I'm typing cube control get nodes, and it's telling me that my cluster is healthy.

17:47 That's pretty cool. I like that. Maybe you can run h top. We can have a look at bit of the memory consumption. That could be interesting. We're barely tackling this thing. Yeah. So We got 1.3 gig of memory of 62 being used. Maybe I over provisioned this cluster a little bit. There's probably also bit of bit of other stuff running on this. Yeah. Well, this is really beefy box. Okay. Nice. Yeah. We can we can maybe also have a look at the pots that are running in. Yeah. Maybe in all namespaces, and there you can see it already installs,

18:29 coordinates, and traffic as an English controller, metrics server, some local pass provision for local storage volume. You, of course, you can also switch this all out. If you don't like traffic, you can say on installation, disable traffic, and then roll your own English controller. It's a bit for convenience that it's there. And is that true for the CNI implementation, or do I just get flannel and You get by default, you get flannel, but you can also say, I want don't don't wanna use flannel and then install your own CNI plugin to install anything else. Because in the end,

19:00 it is standard Kubernetes. Okay. Nice. So we have we have our first k c s cluster. Okay. Then what we can use this one for is then to yeah. We have k three s running. So just out of curiosity, why is this prefix, this k c s? Kubectl is not actually not installed. It's also kubectl is also bundled in k three s in the binary, and that's just like a some link for this. Alright. So if I run should be a symlink to to the k three s binary. Alright. Cool. Okay. So So and you can

19:40 also say k three s k three s three s, and that will be the same thing. Always like to run the command. Okay. Alright. Okay. So it's all embedded in a single binary. Yep. And then the cubes the cube control I actually have on my machine is really just a wrapper or something around this thing or link to it. You can also use any of your own cube procedures just like to make it easier to use. Yeah. That makes sense to me. So is there a next step then to to install Rancher? Yes. So you can see the next one

20:12 would be this this location where it actually wrote the cube config. You could either download this or we can use just directly use cube CTL on this box however you want. Susan? This is a cube config? So I'm not No. This is just a normal cube config. Yeah. Yeah. Maybe don't show the total certificate in there. Yeah. Yeah. Alright. So let's get like this. Yeah. So maybe let's use this box tool just because we don't do we're not gonna use much cube CTL stuff on the in this cluster anyways. So the next step would be then to

20:48 install Rancher. Alright. Next. Let's see. And Rancher is used to install Helm, so we probably need Helm need to install Helm on this box. If you haven't you can, I guess, use it use Snap also to install it? Okay. Let me just check. Yeah. We're not gonna have Helm. Do I have Snap? Nope. Let's just grab the Helm. Okay. And then we have to it's just download the binary, I guess. I'm sure there's a curl bash here. I can just yeah. There we go. Yeah. X. Get help. Get help. Okay. So what we're saying is we can add

20:50 Installing and playing with Rancher

21:37 the Rancher Helm repository, and then we've got oh, we need a namespace. Got it? Okay. Now we have to make one decision because Rancher is a sensitive workload because you can manage Kubernetes clusters through it. So you need a TLS certificate. There are three options. Either you provide your own certificate in a Kubernetes secret. If you have your own CA or you bought an expensive certificate, you can use that. We could use let's encrypt for it, or we can just say, let's make it easy and let Rancher generate a self signed certificate. And for this for both let's encrypt and

22:22 self signed certificate, Rancher uses cert manager. So we would need to use in source cert manager. Alright. So that's going, create the names. I can just copy all of the comments. So I think, yeah, it should be copyable. Yeah. Yeah. It'll be it'll be fine. Okay. Is that the last Yeah. We have to want do one thing. Of course, the Helm binary doesn't understand the the this symlink to k three s cube config, so you have to export the cube config. Okay. So that's the export cube config equals slash Rancher? No. Yeah. ETC Rancher Case3SCase3SYamo.

23:10 Yeah. And then also the Helm CLI knows now. Perfect. Okay. So I think we have to wait now until swap manager has started because it has this nice webhook. Okay. We don't have k. I mean usually have locally. It's not, of course, you don't have this on VM. Yep. Exactly. Already there. Yep. I can either make that table 14,000 times over the next fifteen minutes or whatever or just alias it now. So Yep. Yeah. Why I said we have to wait is because this webhook needs to be started. Otherwise, the creation of the certificate will fail

23:50 because Kubernetes will directly try to call this webhook wherever a certificate is created, and the Rancher Helm chart will directly create a certificate. Okay. So next is then we have to install Rancher. And, basically, that's the command, but we have to modify it a bit because we want to install the newest two dot five release candidate to it. So maybe if you Yep. Go to text editor somewhere, that makes it probably easiest. Hey. Let's just quickly Perfect. I'm assuming I just copy this and change the number. Yep. No? Yep. Correct. I like it when things are intuitive.

24:37 So the newest one will be 250DashRC3. And let's hope that RC 3 will work. I have only tested r C 2 R C 2 yesterday. And the second thing is we have to change the host name. Yeah. And since probably we yeah. It would be a bit too much to create now DNS entry. We could use this zip.io magic DNS. Do you know that? Is that the IP address followed by this? Yeah. Correct. So Oh, I think actually from Basecamp, the service, and you that basically will resolve everything to this IP address. And maybe add Rancher

25:24 in front of it, like rancher. And then because we may wanna have more than one. That's great. You'll actually we're already getting the traffic for a four page. Because yeah. Yeah. Because English controller is there, and the default by default, the case is English controller just listens on port eighty and four four three on the host network. Okay. So what was your last comment there? I've been putting do you think Let's maybe do Rancher. And then the IP address just to have, like, a nicer URL. Yep. Happy with that? That should be yeah. Looks good.

26:01 If not, we can just fix it and change. Okay. So it's it's now starting, of course, three pots because that's the default. Well, it doesn't matter. They don't need some. We have a very busy machine. Otherwise, we could have set the replicas to one or so in this one note installation. So when this installs, I'm assuming that we get some ingress set up Yep. That's gonna make this resolve to the the Rancher UI. Correct. So it will it will directly one Rancher port is up and running. And, yeah, the Rancher Helm chart will directly set up an English resource as well as

26:52 the cert manager certificate to get a self signed certificate for the source. Is there a authentication by default, or have I just given people access to a Rancher UI? No. There's authentication by default. But you have to set up the set up the admin password first. One is faster and set up one already. We can but you can also reset the admin password then. Yeah. First. Okay. And we also gonna use the multiple cluster management thing because we wanna manage multiple clusters. Great. And we can just save this. That can be useful if you have a

27:38 reverse proxy in front, it may not may be able to detect the correct setting. And we have a Rancher installation up and running. Awesome. Yep. 2Dot5RC3. The local cluster that is already in there is the cluster where Rancher is running in. So this is also, by default, directly available. We could also hide this if we don't want to want users to see it. It is a good practice to have Rancher running in a single cluster just on its own, just to separate the management tool Rancher as a control plane from where you are running your workloads in in

28:17 different Kubernetes clusters so that your workloads are not refer influencing Rancher too much and that Rancher is not influencing your workloads more importantly. Okay. So I'm liking this, a nice visual representation of of my clusters and my individual cluster. We just have another quick question there from from Siam who's saying multiple management option is not there in r c two. Is that correct? It's still there. I think it's just in r c two. For some reason, someone removed the that you can choose it when you install it, and you directly go to the multi management setup.

28:52 And you could not switch to the just a single cluster management, which is new in Rancher two five. Awesome. But the multi cluster management is actually what you see in Rancher two four already or two three or two two. That should this is what you're seeing now. This is more new. And what you saw before is then what existing Rancher users should be familiar with. Okay. So just to recap, I mean, there's there's a lot that happened with that element still there. We have deployed the Rancher. I mean, it's is it just a UI? Rancher is just

29:26 a UI that allows me to manage, integrate, explore my customers. There's no other components to it. Is that have I got to understand it correct? Yeah. It's a UI and API and kind of like an operator, you would say. Okay. So Rancher did operators before operators were cool. Nice. Okay. So this is the cluster explorer. I mean, besides just taking a look at pods, jobs, all the other Kubernetes resource, Is there anything that particularly cool or important that you wanna cover on that before we move on to the Maybe go back to the cluster manager.

30:03 Upper upper right. Oh, there we go. Yeah. So where you can see all your the multiple clusters. And if you go back to on the left upper upper left where low where is local to global, you can see that you can add clusters there. Maybe if you click on it on the upper right side, you can now then now start to add clusters on any kind of infrastructure. And the cloud providers you see there are just the ones that that are activated. It's a plug in system. You can even add more to it. You can register

30:37 all the cloud provided managed solutions, or you can install a new Kubernetes cluster on existing node or just import an existing Kubernetes cluster. Directly, you've I'm not feeling the the packet love here, Bastian. In the end, it's Docker machine. So it's also not something that we completely invented, and I think there is a Packet Docker machine driver. Is a Packet Docker machine driver. You wanna edit? Yeah. Yeah. I'm gonna get that. I'll work on that next week for sure. I think one of the things is for all the supported ones, the usually, you also have, like, a nice UI then.

31:10 If you if you just use a plain Docker machine driver, you probably just have input boxes in the UI and not, like, drop downs or stuff like that. Okay. Nice. Also, what interesting there is that under if you click on security in the top box Yep. Yeah, and then authentication, You can see that you can configure all kinds of enterprise authentication systems, some are based ones, active directory, l dub, you name it. And then you can get all your users and groups imported for there so that these users and groups can authenticate at Rancher. But not only at Rancher, but also on

31:48 all connected Kubernetes clusters. And then you can set up directly centrally in Rancher, granuly all the roles that you wanna give people globally in Rancher, but also within one cluster. And these are then gonna be rolled out as standard Kubernetes role based access control to all attached clusters. So is this managing the deployment of decks across my Kubernetes cluster, or is it its own mechanisms? It's it's own mechanism. So in the end, every user will get a token, a dedicated token. This token was synchronized in every Kubernetes cluster, and it also can expire and everything. So it's

32:25 a token based authentication on Kubernetes level, and these tokens are then managed by Rancher and attached to this active directory users, for example. Okay. Nice. I like it. So I would say let's add more clusters to it to have some fun and deploy stuff for. Right? Oh, yeah. So we do have another question from my friend, Sam, here who's asking if we're able to update the k c s cluster from the UI. Yeah. Let's let's go let's do this next and install a bit older one. Right? I would say. Yeah. And yeah. Let's do that.

33:00 Upgrading k3s clusters with Rancher

33:05 So if you I would say let's add two three clusters for now. One single node cluster that can we can run-in a slightly older version, and then we can test the update. And then maybe also on three VMs, a highly variable cluster to have to see how that looks like. Okay. So the the process for them doing this. Now do I go on to each of these machines and provision k three s and then add them to Rancher? Or do or can Rancher do something with the IP address, like, on provision? Currently, there's a way there is a way for

33:38 Rancher to directly install Kubernetes cluster on these machines, but that would be an RKE cluster running on Docker. It's also standard Kubernetes, just a different distribution, and it's using the upstream images. So there's no modification at all. The downside of it, why I wanna go the k three s way is k three s is much faster. And with RKE, it's like five, six, seven docker images. It needs to configure each d and everything. It just takes longer. Alright. So let's And there's no benefit to it. So we first create currently on the VMs, a k three s cluster,

34:15 and then we can import it into Rancher. But then we can manage everything through Rancher upgrades and so on. Okay. So maybe can I post easiest as we is I just put the command instead of our private chat? Right? Then Yeah. Go for it. Just so that we can install k three s in a slightly older version. Alright. So this is the same k three s command we used earlier, only this time we are setting an environment variable to specify an just an older version so that we can That we can upgrade it. Yeah. Got it.

34:52 And it's now using the newest one seventeen. We could also specify the exact version if we wanted to, but I'm always too lazy to do that. I love this curl command that in twenty, thirty seconds, I get a cluster. That's just ridiculous. Okay. So we have a cluster now. And to add a torrential, if you go back to UI Got it. And you click on add cluster, you can now import register an existing cluster. No. Not on existing notes. This register an existing cluster above this and then other cluster. Got it. And then we can give it

35:35 a name. We could now directly add. If you have multiple users or groups, you could directly give them permissions in this cluster. Of course, you can also do this later on. And maybe let's directly give this also a label for late later on. And maybe label I don't know. Group also doesn't matter so much. And then one or Ubuntu. Yeah. Let's maybe Google Ubuntu. That's great. We can use this later on to fleet to decide then, okay, deploy to all Ubuntu clusters or deploy to all clusters. Ah, nice. Okay. And now we basically get a kubectl apply command

36:17 that we can execute. And we since it's self signed, we have to use the one below that because it is a self signed certificate. Otherwise, kubectl will say invalid TLS certificate. Got it. Okay. So this is the same as this command only. Yeah. Just that downloads the YAML with scroll and then pipes us into kubectl apply. Done. K. Yeah. Once these pods are started, they should then also build up the connection to Rancher. Basically, what this is is it insults a small agent into the cluster. And this is also in every connected Kubernetes cluster. The only

36:58 Rancher specific thing that is not standard Kubernetes is this one agent. And the job of the agent is to connect to the API and make it manageable. And the rest is standard Kubernetes. Yep. And now we have the second cluster there, and maybe we can directly upgrade it. Right? I'll take your word for it. Yeah. So can I click on this cluster? Yep. So first of all, maybe before we upgrade it, can you just click on the cube config file there? This gives you you're gonna hit on everything afterwards. Otherwise, it will show you a token.

37:33 Okay. Then let's not show it. This is also the way how you get now a cube config with user token that is dedicated to the user you are logged in into Rancher. So if you have had your active directory connected, you would now have active directory based authentication at this cluster. So that's cool. Really cool. But Well I mean, was fast enough. Oh, people can pause it, I'm sure. So Yeah. It's well, whatever. But I'll turn it just don't go mining crypto for the next half hour. Okay. We can now also yeah. If you click on the context menu

38:11 with the three dots in the upper right Got it. And click on edit. And there you have a drop down with the Kubernetes version. And, yeah, you can choose any new one. Click save. With the concurrency, you could have also if you have a multi cluster multi node setup, configure how many nodes should be updated in parallel. If you have 50 worker notes, you probably don't wanna go one by one, but update several ones in parallel. And it also can directly if a multi node setup, then drain containers away before updating. And I'm assuming because k three s is

38:53 bundled as a single binary, is that the upgrade process is just pulling down a new binary and then replacing the process. Yep. Like, that's done. It's also done already. Okay. So, I mean, I could run SCS. That doesn't show the version. Version. I think it's just for Of course. There we go. Oh, yeah. One nineteen. Our client and server. Alright. Nice. So it's upgrade. Oh, of course. Yeah. It also this is funded. It also directly up updates KubeCTL on that note as well. Yeah. So that was the single note setup. Should we directly add an assert to assert

39:30 Installing a multi-node / HA k3s cluster with etcd

39:38 cluster to it? Yes. Okay. So I grab this IP address. And And I would say let's do it slightly differently and set up a highly available multi node cluster and maybe also we see embedded SCD so that we directly have not only one control plane server node, but multiple ones. And is this where you tell me you're gonna give me the command for that. Right? Yes. So one you're on one on the first node for this cluster, this this embedded entity is only available in one nineteen. So I directly specified one nineteen. And to make it easier,

40:23 I directly specified a token secret that we need the the secret this token secret we need afterwards to to connect all these notes together. It needs to be the same in every command. And with this cluster in a variable, I'm basically saying initialize it's the cluster as well. Okay. And with this case, we have x x server saying this should be a server node and not an agent node. Yeah. But, basically, it's doing same thing, just one nineteen with x d included. And is this still gonna take twenty to thirty seconds? Or Yes. And the

41:11 second command, we do the so we probably should do a three node cluster because at city needs forum. Right? Okay. So we have two more nodes. So that's good. Yep. So the we need the IP on the two other nodes, we definitely need now now the IP address on of the first server node. And, yeah, I also send you the command for the other two notes. Alright. Got it. So And you have to change the 3 u l then to the correct IP address. That, I can just about manage. So So this is pointing, basically, 6443

41:55 is the port of the API server so that it can connect to it correctly, and it is all then authentication versus very unsecured token. Well, yeah. I was thinking that. I mean, people could join this cluster just by heading these IP. Let's hope that they're not destroying our demo. Right? It'll be fine. People are nice. Right? Yeah. So well, we add this arm one. Would that work? That's the third part. Question it. Yeah. Let's try this maybe. I have never had a never did did a setup with share with different CPU architectures in the control,

42:43 it should work actually. I don't see what I'm trying to think if there's any components that wouldn't run on ARM, but I think it should be okay. Yeah. I mean, it it I've seen the URL change there. It was definitely done with an ARM binaries. So let's So if you do QC to get notes, you should see that there are no yeah, two That one is just starting. The other one is still starting the system d service. So yeah. So it's not ready yet, but maybe it just takes a couple seconds because it also needs to install the c n I plug

43:21 in or yep. Now it's ready. Nice. So we have now a three node Kubernetes cluster with one r we basically, what we have now is three nodes, and all nodes have all roads. So there are worker nodes, it's a d, and master nodes or control plane nodes. Yeah. Every time I run this, I expect to see a fourth node, but you know? We could also add a fourth node to it. If that would be probably where we say, hey. We just add a fourth worker node to it if you want to. No. No. I'll I'll let our audience add that force

43:56 code if they're feeling brave. So Okay. Yeah. Then the next step would be also to add this to Rancher the same way we did before. Okay. So I just pop back over to global. I add cluster, other cluster. I give it a name. So this is our cluster. And I'll get this a p j. And even though it's not all Ubuntu, I just wanna grip that with the previous one. Yeah. Yeah. That would be nice that we can still connect it to it. And it has some Ubuntu notes in it. Right? It does. Yeah. I think was it group

44:34 that I used, the key? I think it was. I group. Yeah. Alright. We grab the insecure command, and we run that. And this is magically gonna show up again. Pending. Yep. Okay. Are we okay with two clusters to deploy to them, or do you wanna add a third cluster? No. I I think that's good. You know, we've got a single solo node one. We've got the agents It it would be basically the same. And also the command to add additional worker nodes, so non control play nodes to k three r s is basically also the same same

45:16 instead of saying server, you say agent, right, in the command. Okay. Cool. Well, that's important. Yeah. Maybe on this bigger cluster, we can see here in the UI that it already shows you a bit of information about what is kind of reserved as resources into the cluster. But maybe let's also activate the Rancher integrated monitoring to see a bit more metrics and dashboards. And, yeah, let's let's keep yeah. Let's keep it at this because I I'm fairly con convinced that this will run on ARM. Yeah. We can just use the default. It should be fine.

45:20 Using the Rancher integrated monitoring

46:01 Sorry. I was getting ahead of myself there. I'm just clicking buttons. That's fine. Maybe one interesting thing is that we are it was Rancher 2 Five, and it mentions this at the in the top that we are also provided version two of all this monitoring integration that is even more flexible. But let's keep the v r v r v one one because I'm not sure if the version two of the monitoring that is new already works on ARM. It should probably, but I'm not sure. So this is the the new one. Right? The new one is a new one. In

46:35 the end, it's it's using the premises operator. Both of them do. The new one is more flexible and more in line with what is upstream. But we can yeah. Let's keep it to this. If you go back to the cluster page, if you click on cluster yeah. Cluster. Okay. That's not completely ready yet. What does that's just deploying exports. Right? Am I am I correct? Yeah. It's deploying Prometheus and here we go. I think it may have scheduled something on the ARM note that is doesn't have an ARM image app. Yeah. It's some sort of operator, and it

47:17 is Can you try to re delete these ports and hope that they are or maybe let's let's cordon the ARM note for now. Rancher, tap in. Yep. And to delete parts. Long pasty window. The kettle the The namespace. The namespace is kettle well, Prometheus something. Kettle system. No. It's not this one. Okay. Kettle Prometheus. Yeah. So I don't think I regret my my poor choice of tabbing layout here. But No. You can also go to the Rancher UI. I think that was a a good point to have a look at the all these ports in the Rancher UI, then it's oh, there

48:23 it's up. So you can see here now, you have now a bit more metrics. And if you open up this cluster metrics box Where is that? At the bottom of the screen. Yeah. You can see it already starts pulling metrics of all the different notes. Of course, we take some time until all the data's there, and you also have a link everywhere to Grafana that gives you more detailed dashboards. And, of course, you can then extend this and monitor your own applications if you add an influx to be to it or so. You can also scrape this with Prometheus

48:59 at your own dashboards to Grafana. It's very flexible. Yeah. Nice. Very cool. I like it. And you get it set up with all the defaults with a sensible set of alerts, sensible dashboards for everything you need to discriminate to do the click. That is also quite nice. Okay. So I think that's a really it's it's it's cool. I like this. You know, we've got cluster management. Important that was if I even provisioning the clusters themselves was an absolute breeze. And we now have this single pane of glass for lack of a better term for working with my clusters.

49:37 As now, when we look at Fleet and start deploying stuff? I would say so. Or if you of course, you can look at other things if you want to first, but maybe Fleet would be kind of nice. Yeah. Let's let's get some work load running on this. Okay. For that, you have to go to the Explorer UI because that's where the fleet UI is located. So click on Explorer, maybe on the local cluster. I think it doesn't matter even. It doesn't matter. You should go to both. Click on cluster explorer on the top. It goes all to the same UI in the

50:13 end. Yeah. And if you go now into the upper left corner on the drop down there for our cluster explorer Yep. And go to Fleet. So this is the Fleet user interface. And what you can see already on the clusters, As you would expect, there are two three clusters already registered, the local one, and then the two ones we created then. And by default, it created the new clusters we added in the fleet default workspace, but we could add multiple workspaces and move all clusters around and organize them to workspaces. And if you go to cluster groups,

51:08 this is how you can group clusters together, and there's already one cluster group. Maybe we can add additional group for our one to 20 clusters. Alright. Let's see if I can work this out. That would be easy. Yeah. We're already in the default workspace and yeah. Have been to 20 clusters. The selector's gonna be Yeah. The labels. Selector. Yep. And, of course, we had connect we could also give that labels if you wanted to, but it's probably not necessary. Labels all the way there. Yeah. And then you should be if it works can you click on the group?

52:01 It's not there, is it? No. The group is there, but the clusters are not registered. Oh, that's oh, okay. That's just the workspace default. Got it. In the workspace default, you have the group there, but we have zero clusters. Why is that? I probably just messed up. So I'm gonna see Can you go back to maybe no. Go back to Fleet. We can also directly edit it from there and should see it. But if I was if I were debugging this on my own, like, the way I was I guess I would have to go

52:29 check the labels that I added to my cluster here. Yeah. If I had oh, that was in the edit menu, wasn't And then here I added labels. Annotations and the first, there are the annotations and the labels. So did I not add Okay. We'll add it that. I think I did, but maybe let's yeah. Let's go back to the fleet interface. There we can definitely edit. Excuse me? Interesting. I would have expected that this will work because I think I saw you doing this. So if you go to clusters Yep. Got it. And then you click on,

53:09 for example, one the cluster also. Yep. And you can now go in the context menu, edit it. But it has a label group Ubuntu 20. There you can also see it. Oh, yeah. I do see it. Oh, but Did we do a typo? Probably. I mean, whenever something goes wrong, it's generally my fault. So Okay. Or did we do this in the? So if I click on my cluster group and edit as form oh, my rule's not there. Okay. Can you edit this as YAML? Oh, no. It's the it's match labels. So k. Can you go to back to cluster groups?

54:07 Maybe it's just worked and we were too fast also. So this is all declarative. Right? I mean, I could have this in a get repository and apply it to my cluster without ever touching the UI. That's pretty cool. I like that. Yeah. It's all good based. Can you refresh maybe? Yeah. It definitely worked yesterday. Okay. Interesting. Okay. Let's not use cluster groups then and just go forward. What we can now do is we can so to deploy something with Fleet, it is a GitOps based approach. So you have all your resources, your applications in Git repositories,

54:47 and we can now add a Git repository to Fleet. Basically, is we have to give it a name, the URL to the repository. This is a repository that I prepared. And, I mean, I I didn't really speak to you about it. I just gone ahead and added some YAML to it. So it's just straight up Kubernetes manifest fails to deploy Telegraph and and FlexDB. Okay. Yep. Looks good. Then you could set up authentication if we had not a public repository. It is an either username based or SHK based. You can set up the branch that

55:28 should be watched, and then we can now add multiple pass to it. And Because I applied every pass will be will be a bundle in Right. So if I do enter For every bundle, it's kind of like in one application. And don't do I think you have to remove the leading slash. In my test yesterday, it only works without the leading slash. The example is wrong then. Yeah. People are heavy at work, especially at the UI. And maybe at a a second pass, we could for the second application for to it. I guess that that's important just to

56:06 remind people. We are running on a release candidate version of 02/2005. So these little these little things are expected. Okay. And then we can basically under deploy to configure which where this good repository should deploy to, and you can choose your workspace. And I think I know what happened with the cluster group. We added it to the wrong workspace. Can you maybe go open a new tab also just to check? And can you delete this cluster group that we added? Yep. And let's create a new one. And in the workspace yeah. You have to choose fleet default

56:56 because the clusters are in the workspace fleet default and not in the workspace default. If I just add my group into 20 label and then hit create. Yeah. Know we have That's good. So we could say cluster group and yeah. And this will then deploy should deploy this to all clusters. You can also click on the name, and you should see the events popping in. Maybe we have to wait a bit or yep. Now you see the resources are being applied in the cluster. Nice. So if I pop over here, pods and watch, And plug TV is running.

57:58 My telegraph is running. My arm one failed. And that's annoying because I checked that there was an arm in the manifest. Well, but besides that, my job is completed, which means I have authentication. So Is it maybe I don't know if wrong version also that we are using in the YAML or So if I do Or can maybe just try to debug it a bit and then fix the error, push it to the repository, and see it updating. That will pass. So what I would normally do is I could describe my my daemon set here and check the image, but I

58:36 guess I can just use the UI now. Right? Sure. You if you go back from the Fleet UI, you can choose no. You can either do that. At the moment, they are both two ways because we are in the phase of transitioning to a more flexible UI. So can go to the cluster explorer, choose the correct cluster, like the cluster that is correct, and then you can, for example, go to stateful sets. Mhmm. Oh, daemon's daemon's set. Oh, daemon's sets. Sure. Sorry. Yeah. Daemon's and go in there and you see all your ports and events

59:10 and maybe click on the trailing one. So I think yes. The Alpine one probably doesn't work on ARM. Ah, that could be maybe we have to use the non Alpine one. Right? At telegraph one so one fourteen definitely has ARM. Give that a second. Yep. There's ARM 64, but I'm assuming maybe the Alpine one doesn't. Let's let's just. Yeah. There we go. Yep. It's okay. So I But it's great that Yeah. I was like, I I tested that manifest. I know it's good. So let's take off the Alpine. So I'm just gonna make some assumptions here. Please

1:00:01 feel free to correct me. If I just push this to my GET repository, is there a auto sync that runs or do I need to configure that through Fleet? Nope. Push should be fine. That's your that's that's the whole goal of it to make this like a GitOps based approach to manage. Now we have two clusters. Imagine, again, you have windmill park, and on every windmill, you have like, on a turbine, you have, like, a small Kubernetes cluster and you have thousands of them. You wanna have such approach. Alright. So I hope that works. Because, actually, this is I had not tested

1:00:44 that. Maybe you can you should be able to follow it in the Fleet UI. Of course, it will also update here at some point, but maybe the fleet are out of sync, and it should now start synchronizing. Because it saw okay. There is a new commit. And the daemon I also saw that the daemon sets changed. Okay. Excuse me. K. Oh, there we go. Modified. Modified. No longer out of sync. So if I go to cluster explorer, daemon sets, we have 303. It worked. So that means I oh, I'm not on my machine. Let's port forward.

1:01:58 Oh, I'm gonna need the kube config, aren't I? Yeah. You could just SCP it or pull it. Or you can download one. If you go back to the cluster manager, you could go on cube config file and quickly copy it or download it. I'm gonna shut this down in, three seconds. Oh, go Not quick. Download link if you wanted to have that. Okay. Okay. So call this cube config. And if I grab my terminal here okay. So we can do export cube config equals cube config. That's not the right directory. Where did I save you to? Oh, it's

1:02:47 M Site Telegraph D S. Okay. That environment variable should still work. So in theory, I can run get notes. Nice. Can run yeah. I have my aliases now too. So get pods, port forward, and I wanna log in to InboxDB. I wanna see those metrics working. And also to just reiterate, you now have only the permissions that you were assigned in Rancher. So the whole if you had LDAP or active directory authentication, that will be would be all now reflected. Okay. Excellent. I like the authentication stuff being made easy because I'm very bad for just letting that slide

1:03:38 for convenience. So Yeah. There's one thing I need. So I create a secret in this, and I can't remember what the password is. So let's just decode that. You can also just maybe a trick go to the Rancher UI and copy it from there. Oh, nice. If you go, for example, to projects and namespaces this is one way. And now you have it in it was in in the default namespace created? Yes. You create in the default namespace. And click on the project default. Yeah. And then on the resources in the top, you have secrets.

1:04:17 And yeah. There you have and then you can just double click on this, or you can just copy it now from there. I think if you copy it, you have it in clear text. With some sort of what? Voodoo? Let's see. No. That didn't work. No. Okay. Maybe then just click on the upper right on edit. Well, will the cluster screen will give me a nicer view than you want? I don't actually don't know. We can check. But you can definitely go to it and then click There we go. Oh, yeah. It shows us in clear

1:04:58 text. Yay. Okay. I had explorer. Ah, there we go. Look at that. Maybe? There we go. Alright. Let's just drop that down to five minutes. I wonder why that's slow coming down the wire. I guess it's not important. Alright. I have my metrics. Nice. So that's taken a GET repository with my manifest, my applications in it, deployed it to the cluster based on my cluster group that we configured through labels. I mean, I guess, my question is I can just add extra if I added that label to my other customers, that workload would just be shipped out

1:05:50 to them all. Right? Yeah. Sure. Maybe we can even try this. Let's maybe add a certain an additional cluster and see that it's also rolled out there. So maybe if you go to if you go to cluster manager again, and you have one we have left. Right? I think. So we oh, no. I think we deployed to H E and Rancher 2. It's the same labels there, didn't we? So you wanted to see now that if you added an additional cluster to it, that it also deploys it on there or Yeah. So is Yep. I mean, it's it's

1:06:29 already done it. Like, both my both my clusters are running Telegraph and then FlexDB. I mean, if you added now an additional cluster and also gave it this label so that that actually met matches this this cluster group, it also would Fleet would directly add this new cluster to the cluster group and also deploy it there. If you want, we can demo that. Yeah. Sure. So what's the what's what's step on there? So we would add an additional k three s cluster, create a new one. We I think we have one VM left. Right? Alright. We have Rancher five, which I

1:07:00 don't believe we're using it. Yep. So I just SSH on there, execute our lovely little curl command. Yep. Let me just grab that from over here. We can just use a default one. Right? Yeah. Sure. I mean, let's just stick one nineteen on it. And then to add a cluster, I come to the UI. I click cluster manager. I click add cluster. I'm getting the hang of this. Global at see, this is it's just too easy. Yeah. Also, direct stop directly at the the group label. Yeah. Open to 20. We grab this this command here.

1:07:59 And then I'm gonna pop over here and we're now in a pending state. So what's gonna happen here is that's gonna bring this cluster under management by Rancher. Fleet is gonna detect the label, add it to our cluster group, and then deploy or get ops workflow to the cluster. That's that's that's the magic you're telling me about to happen. Hopefully. Alright. So I don't have access to cluster explorer yet. It's still importing. Let's just give that a few minutes. Yep. Let's see what we have on the comments. So we got a tada from Jason. Yeah.

1:08:38 I was very happy that it worked too. That was awesome. John says the stream looked great. Thank you, John. Appreciate it. And Sam saying here comes the influx the expert. Yeah. Whenever you find them, let me know. I don't know where that person is. And we have our cluster. We have a cluster explorer. I'm excited to the human set already. Yep. We've got our get ups here. If I go back is it there? It is. Yep. Too easy, man. Too easy. Maybe I sent you another link in the private chat that would be maybe nice to

1:09:17 just show the documentation page once for Fleet. Let's do that. Because what's also nice about it that for these git repositories, you cannot only put just standard Kubernetes YAML in there. It also understands Helm chart. It understands customize. And there is even, like, this fleet YAML file that you can add to it to make them customization based on cluster labels. So you can say you have you deploy this to the whole cluster group, but on all clusters that are labeled with something production, you configure it slightly differently to add to also have these use cases. So

1:09:59 it's actually very powerful. So if you scroll a bit down, you can have a look at the fleet YAML, you can see you can pass all kinds of helm values in it, and then there should be also below this customization. Yeah. These target customizations, you could say, okay. For these targets, please use these overlays and do different helm, values for it. Yeah. For the whole bundle. It's actually very very flexible. And if you scroll up a bit, it's also interesting. You can also configure a rollout strategy for all the clusters. So this rollout strategy is not for the

1:10:43 deployments or so. It's actually how many clusters should be updated in parallel and how many should be max unavailable or something like this. And unavailable means it's currently being updated. So it's an actual a gradual rollout of my workloads on a cluster basis. So roll out to two of my clusters before I send out to two all of my cluster. Yep. And that works for the modifications that I make to so when I modify my Gelb thing, it's gonna test it in one cluster then start rolling out. Right? Yep. Of course, what it's doing in the end,

1:11:04 What is Fleet? (Multi-Cluster GitOps)

1:11:15 it's doing helm upgrade, kubectl apply, or the equivalence for this. There is not something in that then that's kind of like load testing, but it's just checking that the applications are coming up. But if you have proper readiness probes and live disk probes, that would be reflected. It's pretty impressive. I gotta say, like Yep. What's the I mean, the golden question here is is Fleet's pretty new. Right? So have I should I be rolling this out to production? Do would you advise that we just keep an eye on the project? I mean, what what's the consensus on that at

1:11:52 the moment? So it's gonna be generally available and under enterprise support by us with Rancher two dot five. So the two dot two five o release is scheduled next week, and then a reset is gonna be marked as stable. So that usually takes, like, two five one or so until we say it's not the latest release anymore, but one that we consider stable and that also the community tested before. And yep. Then you can definitely use it. Awesome. So we have a a question here from Jason. Jason is asking if there are any hooks. I'm assuming Jason is asking if there's, like,

1:12:35 a pre install, post install hook similar to what Helm offers. The Helm hooks definitely will work. So if you use the Helm hooks in a Helm chart, these are gonna be executed. For non Helm resources, that's actually a really good question. Actually okay. Sayam says it's yes. It's possible. Okay. That's all there are. If you have a a link to any documentation on that, Sayam, if you wanna drop that in a chat, we'll pop that up. Yep. So it sounds like that is possible, Jason, and we'll try and get you more information. They have another question here.

1:13:16 So I'm Eric is asking Fleet without Rancher. Is that possible? Yes. So Fleet is a standalone project. You can use it on your own. Just for the Rancher Ads, it's like the nice integration with the UI. So the UI is not available as a standalone UI, but then you could just use Rancher for that. It is also open source. But if you don't wanna use it, you can also just use Fleet on your own in any kind of Kubernetes clusters. So if I can take any vanilla Kubernetes cluster, run the Fleet controller, point it to my Git repositories, and

1:13:48 get that same GitOps workflow without anything else. Yeah. You also quickly showed it that in the end, everything that Rancher does is just creating Kubernetes resources for Fleet. Right? So can I just do get They are in the local cluster? So in the first Rancher cluster, you can see. Oh, that was that was too many tabs ago. Come on. Okay. Let's jump back on that issue. Yeah. So for Fleet, there are customer source definitions for git repos and everything. So can you could say kubectl get, I guess, git repos also. There we go. So I can do a

1:14:05 Hands-on: Setting up the Environment

1:14:33 kubectl get repos. I think it's git repos. Or I think there also should be a cluster. And then in all namespaces, you can see, yep, there is this one git repository. So you it's in the end Kubernetes resources that you can create in any cluster. So my provisioning step for launching a new Kubernetes cluster with Fleet is to deploy the Fleet operator, add the CRDs to find my repositories, and just let it let it do the same. That's great. In the end, the UI is nice and fine, but you probably will not click this together but have if you

1:14:46 Installing k3s for Rancher

1:15:05 manage thousands of classes also have kind of like a pipeline for that and automation for that. Excellent. I like it. It's very cool. I think there's been a an awful lot to take in here today, but like I said, I'm very impressed by what we've seen. I'm gonna check one thing before we finish up, and that's the size of that cluster to see if anyone that take me up on the offer to add the fourth note. I'll be disappointed if no one did it. Where's my note? You can on the notes in the top. No fourth notes.

1:15:46 I gave you a cube conflict publicly as well. Maybe it was too fast. Oh, well. That's great, Bastian. Is there anything you wanna cover before we wrap up for today? I would say, maybe let's have a look at the questions if there was something else. Yep. If there are any more questions, you've got a minute to drop them in and then we will wrap up for today's session. Otherwise, it was what I wanted to show. Yeah. We we covered a lot. Like, that's an hour and fifteen minutes. We've covered, you know, provision in a k c s cluster with a

1:16:00 Recap

1:16:22 single command and under thirty seconds, which I'm still mentally impressed with. We covered the deployment of Rancher to that cluster. We then spun up more Kubernetes. The KCS clusters, one h e, one not h e. Again, by tweaking it with a couple of environment variables, which again ridiculously easy. We brought them under management with Rancher through one command, a cube CTL command and a curl command. And then we managed miraculously all of us in an hour to deploy Fleet and deployment get all these workflows to all of my clusters through labels. Like yeah. It's a nice setup. I can see

1:17:01 the the appeal here. I can see why people would want this. Yeah. I also talked a lot about of these of these edge use cases. That makes complete sense to use something like this because otherwise, it will just not scale and work. But even if you have two, three clusters or even if you have one cluster, it would be also, this GitOps based workflow can be very nice to roll out your applications to staging and production, for example. Oh, yeah. Definitely. I can definitely see that. I I I mean, I don't wanna rope you in because we're, you know, on a

1:17:29 livestream and stuff and have you commit to something, but I will. As I I definitely think there's a lot of room here for another session where we can dive into some of the RBAC stuff. I mean, that's an occasion things that we provide here. I think it would be really cool to do a deeper dive into that configure. But I won't make you see a guest on the stream, but I will chase you up afterwards. Looking forward to it. Alright. Well, thank you very much for joining me today. I think that's a fantastic introduction to three different projects

1:17:54 there. I love what I see, and I'm hoping I get to play with us more in the future. So thank you very much again, Sebastian. I hope the people have watched and tuned in have enjoyed this. And I will be back again soon. Thank you. Thank you for having me. My pleasure. Thanks.