1:04 Introduction and Housekeeping

1:04 Hello and welcome to today's episode of Rawkode Live at the Rawkode Academy. I'm your host, David Flanagan, although you well know me across the internet as Rawkode. And I realized that I said Rawkode three times in like five seconds. However, today we have an awesome episode as we're going to take a look at the k0s project, a Kubernetes distribution for all use cases. Before we dive into that, there's just a little bit of housekeeping. Please subscribe to the YouTube channel and the button's right below my face right now. So click that and tick the bell. It means you're gonna get notifications

1:34 for all new episodes of Rawkode Live. I'm gonna do my best to explore the vast cloud native landscape and produce more videos and more materials so that we can all learn this crazy mess that we are in together. If you wanna come and chat Cloud Native and Kubernetes and pretty much everything in between, there is a Discord server available at Rawkode.chat. Come and say hello, and I look forward to meeting you. We've also been kicking off the Rawkode Academy courses this month, taking a look at a complete guide to InfluxDB. So take a look at the membership options.

2:04 You can support this channel for 99¢ per month or become an incubating member, join the courses, and there's more guest lectures and other cool stuff coming very, very soon. Alright. Back on track for today's session. We're taking a look at k0s, and I'm joined by Youssef from the Marantis team who works on k0s. Hi there, Youssef. How are you? Hey. I'm I'm I'm I'm excited to join the join the show and and and and yeah. No. It's a pleasure to see how it goes. Pleasure to have you here. We're really looking forward to today's today's session. I think there's a lot of

2:17 Introducing the Guest and K0s

2:37 interest in and k0s and people wanna see more, so I'm sure we can give some lots more details and explore it in our hands on way that we do in this show. Before we dive into that, could you do us a favor and just introduce yourself and tell us a little bit about you? Alright. Alright. So so, yeah, I'm I'm I'm I'm usually and and and been working with containers and and Kubernetes for, well, quite a quite a few years. A bit bit of a historical kind of background. We we actually in in one of the

2:46 Guest's Background

3:09 previous companies I worked for, we we actually went to production with Docker zero dot six version. And and and if I would would would really have a time machine, I would go back to that day and tell tell myself that, oh, please please don't do it. We but but but I I I'm I'm in general, I'm I'm kind of a stubborn person, so I'm I'm still working with containers and and and Kubernetes and whatnot. So so so I'm I'm definitely definitely enjoying the whole cloud native landscape and and and the the the kind of all the all the possibilities that

3:44 it it provides and and, well, of course, all the trouble that it also gives us some days. So so Yeah. To be in the container ecosystem that early and still be in it today, you must have the patience of a saint, you see? No. I don't. I don't. I'm as I said, I'm stubborn. I'm just stubborn. So so I I I still I still enjoy working on on projects like like k zero, for example, which which kind of makes the technologies more accessible and more easier to use. Because as we all know, Kubernetes and and and basically all of

4:20 the all of the building blocks that we have in the cloud native native landscape, they are not the are not the ones that are trivial to grasp grasp. So Alright. Well, we already have our our first question on the chat, which I think we'll get to later. But we do see a quote, Jen, and we'll do our best to answer that at some point here. To kick things off, you're gonna get us through a little bit of slides, tell us little bit about k0s, and then we're gonna get hands on. So I am gonna Right.

4:40 What is k0s?

4:46 Throw your slides up. They're now live. Take it away, guys. Alright. Alright. So just I mean I mean, that's the that's the whole point of this this Rawkode streams is is to go hands on. So so I'll I'll I'll glance just go through this couple of slides quickly. Where the where where the name comes is is is basically a sort of a play from the from the, like, zero friction, zero dependencies. And, of course, as as in any open source, it's a zero cost. Well, that's a that's another discussion whether whether whether open source is really a fully zero cost,

5:26 but but but anyway, we we try to make k0s really the the like, the one of the easiest easiest options to to boot up a Kubernetes cluster. So you don't really have to have to be a, like, a, a seasoned expert and and veteran in the industry to to to really get get a, like, a production grade cluster up and running. And, also, the the one of the main main drivers for everything that we do is this zero dependencies. So so we we try to make everything everything in a in a stand alone way. So so you only really need the k

6:02 zero's binary, and and and that includes everything that that that you need to have in order to run Kubernetes successfully in in basically any Linux node. That's that's where the zero in the in in the name really comes from. Thanks. Of course, that I mean, that that there's there's quite a quite a few different cube distros out there already existing, of course, and and and it's it's not the first distro that I'm actually working working on either. But but one of the one of the main reasons why we why we kind of started to work

6:44 on k0s, like, about a about a year ago, What was really that basically, there wasn't a distro that could allow allow us to have, like, a super, super versatile distro that would fit the the from the, let's say, from the basic cloud use case where you have, like, VMs in a cloud and and and and whatnot, and all the way to the to the environments where you have, like, industrial PCs and and and lot of network segmentation on the on on, let's say, like, a like, on a factory floor, for example. And and and that basically

7:26 Control Plane Isolation Explained

7:26 took us took us to this this, like, one of the main distinguishing features of k0s is this full control plane isolation. What it means is is is really that the the control nodes by default are not really part of the cluster from the networking point of view, from the port scheduling point of view. So you we we we don't really have, like, or any any container d or or anything running on the controller nodes so that it's it's, like, fully impossible to schedule workloads either on purpose or by accident to the to the control nodes.

8:08 The Single Binary Approach

8:08 Always independence. So so we don't or or we didn't really wanna wanna get into into the into the working mode where you where you have to maintain, like, a lot of and deal with the dependencies between packages because we we've done that actually in the past, and and it was a it was a sort of a nightmare to to manage. I mean I mean, it's it's it's it's something that we we definitely wanted to avoid and and and really get into this one binary approach, which is not like like a new thing. K k three s has been has been

8:49 doing that already before, but we actually technically do that slightly differently than than k three s does. And then, of course, we we we wanted to have a, like, a pure upstream distro. What I mean by that is that that we don't maintain a fork of Kubernetes. We don't we just basically take the the upstream stuff and and just compile it as as static binaries, and that's it. So so you what what you get is is, like, pure vanilla upstream Kubernetes. Yeah. Batteries included. What do we mean by batteries included is that that we we we we have everything

9:18 Batteries Included & Extensibility

9:31 that you need to have in a in a Kubernetes cluster so that it it actually works. So we bundle, like, container d for the runtime. We bundle qProductor or Calico as the CNI. There, of course, etcd for for data plane for the for the data store as the as the state for the control plane. We've also added up that this client from the from the k three s side. So you can actually use SQLite or or even MySQL as the as the data store. But everything everything can be actually swapped to your favorite solution if you if you really

10:09 need. So you can you can bring your own container runtime if you if you have a a good reason for use, like like Docker, for example. You can you can still use that and and or or or something else. Then, of course, you can you can also bring your own, like like, CNI implementation for, for example, if you need to use or want to use Weave or or something else or Selium or something. One of the one of the sort of design design kind of drivers that we have is that we we wanna we wanna keep the core

10:49 k zeros as a as a sort of a bare bones. Like like, I I think one way to one way to describe it is to to to keep it as as sort of a un opinionated. So so that's one of the reasons why we don't, like, bundle a lot of different things. Like, we don't bundle, like, like, ingress these service meshes or or anything into the into the core k zeros because they they those those though you know, on on that level, you have, like like, way too many opinions. So we are not really in the position

11:23 that that we can we can make a selection for the for the majority of the users that, okay, everybody should use this ingress. We, of course, do have ways how to how to sort of extend the the the the core k zeros, and and and we've built in mechanisms that you can bay basically dump dump set of YAML on the on a certain place on the on the disk on a controller node, and and it'll be applied automatically. Or or you you can you can actually deploy Helm charts via the the k0s configuration directly. So that's

12:00 sort of a reconciler for that also available. So but, of course, in the end, it's just Kubernetes. So so you can, of course, extend it in in in in many, different ways. So I I already mentioned that that we we we have this k0s as a single binary. So so it's one way to look at it is is actually that it's it's sort of a self extracting binary. So so it's actually compiled or or built in a way that that we, of course, have the the k0s binary itself, which has the the all the logic

12:40 of of setting up different components and running those and configuring everything on the on the Kubernetes side. But do the basically basically, we append differ other other binaries at the end of the the the k zero's binary file. And then at compile time, we actually calculate the offset of different files. And and and when we run k zero's, we actually extract the binaries out at at at runtime on the fly, and and and then we can boot up to API container d and and all the other needed processes as as separate binaries. So this is this is something that that

13:19 that we implemented in a in a quite different way than compared, for example, to k k three s. So so on k three s, they they they basically compile everything into a single binary and then just run different things like you'd be APIs and others as as basically as go routines on the on the runtime. In our case, the the cube API and everything else, they are, like, real separate processes on the on the host. Can I ask a question? Of course. Does that mean that k0s is, like, really just a supervisor for these other processes?

13:54 Yes. Yes. Yes. It's a it's a it's a glorified process supervisor. Yes. I like that approach. Very cool. Yep. Yep. About the control plane isolation, the the because that's that that's something that that that really is is a different thing compared to all the all the other distros that I know of, at least. Somebody might correct me if I'm wrong if if there's some some other distro that implements a similar thing. So so as I as I mentioned, the the cubelet, for example, is is is not running on the controller nodes. So we don't really need any need to

14:36 play with the the and tolerations and whatnot to to to kind of isolate the control plane nodes from the from the cluster. And and and that is that is something something if you if you consider, like, your your typical typical cluster setup with, for example, with q KubeDM, you get this master node, which has these taints, and and then you have to play with lot of lot of tolerations and and and whatnot with the with the workloads where you when you're scheduling and and where you're scheduling things at. So so we we wanted to get get sort of

15:15 away from that that that paradigm. And and and there's no, like, like, standard or back controls that would allow to say that, okay. You see it's it's stupid enough that we don't allow him to to to deploy anything on the controller notes, for example. You don't really have that those sort of a standard controls for that in in Kubernetes. Of course, you could do your own admission controllers or or OPA policies or whatnot, but but we we wanted to wanted to really have a, like, a full isolation with things things like, say, scheduling to a control node is just

15:57 not technically possible. It sort of simplifies things quite a bit. What it actually allows us to do is is also have a lot of versatility on the deployment architectures. So we we we use this this connectivity component to to actually enable the communication between the controllers and the worker nodes. So, basically, the connectivity agent running on an on an each worker node opens up a tunnel to the control nodes. And then the control node, whenever like like, whenever the API API server needs to call the cube, let to to exec into a port, like, the

16:12 Versatile Deployments with Connectivity

16:45 logs of a port or or port forwards and and whatnot. The connection actually goes to through this tunnel. So so you can think of it as as as sort of a, like like, reverse SSH tunnel in a way. So what this really allows us to do is is is deployment architectures where your controllers can be actually running on a, say, public cloud and your workers running on your private data center with no direct access from the Internet at all. I actually have a setup on my desk here where I have couple of industrial PCs running worker nodes, the controller for those is

17:28 actually on a cloud. And my home connection definitely is not connected to the or, I mean, it's connected to Internet, of course, yeah, but but not from the Internet. I don't I don't have any port forwards or anything. So so it's a it's a sort of a neat neat way of enabling this needed communication and still having, like, a lot of versatility on this network segmentation and and whatnot. Nice. Alright. I think that's about it as a as a slide intros. We can, of course, talk about a lot of these these points while we go through the hands on parts.

17:58 Q&A: K3s, MicroK8s, and HA Control Planes

18:09 Yeah. Great. Thank you for that. That answered a lot of the the questions that I kinda had in my head coming into this. The firewall one and the way the communication works is fantastic. That was always one of my favorite things about SaltStack is the fact that, you know, the control plane or the SaltStack master as they call it, the workers only had to be able to open a connection to that. And then they use the zero m q messaging all the for all the stuff back and forward. So it was always the minions or the worker that was in control of

18:36 the communication. It just makes the firewall rules so simple because they just have to be able to speak to this one thing over here, and then you don't have to worry about anything else because of the the tunnel approach. Very nice. Exactly. Exactly. And there's the the the surprisingly surprisingly many features on the on the for example, that are really tested on the on the conformance program even that really must where the where the API server must really be able to open connections to the to the kubelet, for example. Okay. So we got a question

19:11 in the chat, and it's from Ty. We have a small follow-up from Daniel. So Ty is just asking if we can talk about maybe a comparison with k three's mini cube, and Daniel has added on micro kits. And I think we've kinda covered that. It was already your slides, but is there anything you wanted to add to that just to kinda add a bit more extra flavor? Yeah. Well well, I I think as as as we've kind of learned from the from the get go is that that the the most comparisons that we get is is with

19:42 k three s. And and I I think partially we we throw ourselves with the naming into the into the discussion, discussion, but but that's a that's another discussion. So so I I think the main main technical difference is the is is the fact that we have this true control plane isolation from the from the day one, and and and it's sort of a not really hard coded. You of course, nothing prevents you to run the worker parts also on the same nodes. Yeah. But then you have to take care of the the taints and tolerations again.

20:16 So it's it's not really the, like, the default way of of of deploying k0s. Maybe maybe one other other kind of real different differentiator is is how we run these cube APIs and container DS and whatnot. So so so in case errors, they are really running as as separate processes. So it's not like everything is embedded into a a single process. Of course, there's like like with any solution, there are pros and cons of of each of these. So so but in in at least in in in how we've been working with k0s now for the for the year or past

20:58 year or so, I think it is this actually having having, like, this vanilla upstream Kubernetes binaries. It actually makes us makes us able to really move fast. So so say that there's, like, like, a new patch release of Kubernetes. It's basically ours that we can ship new k0s version because we don't have to deal with the call module dependencies and and and whatnot because that that's always a nightmare when with Kubernetes. Definitely. Okay. We're gonna get hands on in just a minute. There's one more question in the chat. I'll throw up from Russell, who is asking, can you balance the control

21:40 plane across local and cloud servers? I'm assuming if you wanted a highly available control plane, can you distribute them? Yes. Yes. You can. You can. But, of course, if you if you have multiple controllers, you have to have, like, some sort of a load balancer that that balances the load between those those servers. Yeah. And be careful of cloud providers ingress and egress costs, Russell. Very good. Yeah. Alright. Let me get my screen shared. Keep the questions coming. We'll do our best to answer them as we go. We're gonna install k zeros now on a couple of machines.

22:15 Installing k0s

22:19 But I've got the homepage available. I have the documentation. I have my Equinix metal servers. So I've got four. I don't know what we're gonna do with them. I assume one will do a maybe a manual installation, and then we'll look at other options for the other three. Yep. Sounds good. Okay. So we'll choose the the getting started guide here. So it seems to be the popular pattern these days, but curl bash is the installation method of choice. Is that what, like I guess that's that's there is a convenience for people that want to experiment. Cut the

22:54 tires, have a play with this. Yes. Absolute I mean, of course, nobody should curl pipe bash in production or any any, like, real environments. So so yeah. But but but but it's it's, of course, it's it's, of course, convenient to to to have these sort of face scripts for for purposes like like today, for example. Yeah. Definitely. So this is a bit to 20 o four machine. I don't need to do anything. I can just literally run this and it's gonna work. And you mentioned on your slides that it it's OS agnostic. Yep. It can really just run anywhere. I

23:33 guess that's the beauty of just it being aesthetically compelled by anywhere. Like, it'll Yep. It'll run almost anywhere. Do you see people using k0s for like I mean, this is a sizable machine. You know? But I guess it runs on IoT and single board, Raspberry Pis, all that kind of stuff as well? But at at at least the worker part. Yeah. Yeah. Because, I mean, in in the end, we we we we have to remember that the the control plan is is running stuff like Kubernetes API at CD, which are, as we know, quite resource hungry.

24:10 So so they they do take, like, at least a gigabyte of RAM to to to be able to really run the control plane. But the the the the worker plane is actually actually quite quite slim in a in a way that that that on on worker plan, the k0s process itself, it's it's not really doing much more than than being the glorified process supervisor. So I I I think it's nowadays roughly 200 megs of RAM that it uses. Right. Okay. Yeah. Can get on board with that. How big is the binary? About 200 megs. Yeah. 70. One 70 five, it

24:54 seems. Yeah. Alright. Nice. That's just because we embed everything into the same binary. So Yeah. So this is your k0s extractor and supervisor. You've got container d. You've got the cube API server, and I guess in the cube scheduler, the cube controller manager. You got all of these things stuffed into there. But I I I really do like the supervisor approach rather than the the go routine approach. I think that's that's pretty nice. Yep. Yep. Okay. So we can use the k zero's binary. So it's obviously got some helper sub commands here. Yep. And this allows me to install k

25:26 K0s as a System Service

25:33 zero's as a service on my machine. Okay. Yep. Let's just see what let's see what we've got. There's there's quite a few quite a few different commands and and and help our functionalities on the on the binary too. So Yeah. I can see we've got an area gap set up. We've got the controller APIs. Oh, we've got backup stuff. What does that do? It takes a it it it basically takes a snapshot on the on the on the state of the control plane. Nice. And spits out a terrible file. And we got yeah. We got CTR, etcd,

26:16 install, keep control, start state. Sorry. Yeah. There's there's a fair bit on there. Let's run the so this is going to install the controller. Is that just does that mean control plan? Yeah. Yeah. But if you if you add that dash dash single option, then it'll it'll basically be a a sort of a special configuration where where the controller is actually also spinning up the worker parts because, well, it's a it's a single node setup. So it's a single node cluster. So it's it's it's mainly intended for this sort of a developer use cases where you where

26:56 you just wanna run a a quick single node setup for to test something, test your application, and and whatnot. And then it also it it also actually disabled disables some of the some of the components which we don't don't really need in this single node use case. K. Well, that's it? Alright. That's it. And we have a start command. Right? Yeah. K zero Yeah. Start. So that's just gonna run all of my components for me. Yep. And It'll actually just call basically, the the the system CTL to start the k zero service. Alright. Okay. Let let let's talk about that, but

27:44 I'm really curious to see if this getting all this is gonna work. It did, right? Yeah. Is that okay? So I'm curious now. When I do the k0s install controller, it's creating system d services for each of the components. So No. K no. It's not. Okay. It it create it creates a single system d service for k zeros. Alright. Okay. And the and and and and the process that that system d service manages is then the clarified process manager for the other needed Kubernetes components like API controller manager and whatnot. Oh, yes. So here's our process tree here.

28:30 Verifying the Single Node Cluster

28:32 Yep. Alright. So we've got our k0s, which is running cane with SQL lite. We got the API server, the schedule, the controller manager, container d, the kiblet, the proxy, the okay. Cool. Okay. I understand that. Yeah. Perfect. Yeah. That's that's neat. I like that. Yep. So so so, basically, the install command and start command are just like helper utility functions to get get get your get your system d setup easier rather than having to write system d units yourself because that's painful. Alright. My next question was why did get node return nothing earlier, but it's returning something nice.

29:12 I guess we were a bit too early in the API server and kubelet spinning up for that to respond. But Yeah. Yeah. And and and also as as as I mentioned on the slides, the the it kind of works as a as a self exacting binary. So when when it boots up first time, it it actually sees that, okay. I I haven't extracted the binaries yet, so it'll take a bit of bit of this guy or two to in the in the first boot. Okay. Got it. Awesome. Well, we now have a k0s cluster. We've got access to a status command. Let's

29:44 see what that does. Okay. Because of the version process ID, parent and the other thing. Okay. Nice. And I've already kind of done that. I'm not gonna uninstall it because I just Well, that's nice. Okay. So pretty painless. I guess that's the developer experience you're going for. Right? You just you just want it to be get out the way and just work. Yep. That's the that's the zero friction that we aim for. And this is a completely I mean, it's it's not that it's compliant. I mean, it is upstream Kubernetes is what you were seeing as well. Yep. It is.

30:21 It's not passing any I mean, it can go past the compliance test, but it really is upstream Kubernetes, which I think is pretty quick. We actually we we actually do run the the compliance test for for basically every single release that we do, which is, well, sort of my one of my favorite things to nag about is the the flakiness of the of the conformance. So it's it's sometimes it's annoying to get it passed, but but we we do run it for every single release that we do. I think just saying sometimes that it's annoying to sums up

30:52 Kubernetes in general for me, to be fair. True. True. So is there a 01/22 release of k0s? It's in the it's in the works. So, hopefully hopefully, within next few weeks, we'll we'll ship it out. And what's involved in that process for you then on on that side? Like, what are you looking for? What are you testing before you adopt a new upstream version? We do we we do, of course, the the full conformance testing, of of course, and and we we also also do bit of stability testing and and and make sure that everything works together nicely.

31:32 I mean, technically technically, if we if we think about from the from the, like like, k0s developer point of view from basically somebody from from my team that that does the actual actual Kubernetes version change. He's actually changing the version number in couple of files, and that's technically it. But then, of course, we have to make sure that everything still works. And and especially now in the in the in the 01/22 because there's there's actually stuff that that fine that is finally being removed. So we have to make sure that everything everything works clean and still and and and

32:05 but, hopefully, in the next couple of weeks, we we we will be able to do the release. And we, of course, want always wanna bundle in some bug fixes for k zeros itself and and maybe maybe some cool new features and and whatnot. So Cool. Do I get to select the version of k zeros when I install it? Or is there way can I, like what I'm curious, I'll ask the real question? It's like, could we maybe run one twenty one dot two or one twenty dot zero and like do an upgrade of a cluster?

32:21 Cluster Upgrades and K0sctl

32:33 The first question we got at the start from coach in there is they were just curious and interesting how they manage cluster upgrades, especially in production, if we could talk about that. So is that something you think we could run through? Sure. Sure. I mean, technically technically, as upgrading upgrading is is is is like just get a new version of k0s binary itself and and and restart the system d unit, and that's it. But, of course, when we are talking about production services, you have to do it in a, like, controlled way. And I think that's where that's where this

33:15 this k zero CTL helper tool comes actually in a in a in a in the play. So so k zero CTL can actually do it like a rolling manner. So it first goes through the controllers one by one and and and always waits that that the previous one comes back online and and whatnot. And then it'll it'll move into the into the worker nodes and and do the normal, like, upgrade upgrade drain, upgrade on sort of a dance, what we call it. Okay. Maybe that links us into another question from Ty then. Ty is asking if there

33:53 Q&A: Comparing K0sctl and Other Tools

33:53 is something similar to k c up or ketchup, I think it's supposed to be pronounced. Or is this where Feros comes in? Is k zero CTL like ketchup? Or is I'm not sure what Feras says. So Yeah. It is it is k k zero CTL is is is quite similar than than k three as up ketchup. There's too many numbers and and acronyms. To try to try to say k three s and k zero s and and k three s up in the same sentence, and your tongue is twisted. Yeah. I think I'll have a strong drink

34:28 before I try that, I think. Yeah. What's Pharos? Is that something I should be familiar with? Is that something to more interest? Or No. It's Pharos is actually actually something that I was I was working in the in the past with. So it was a it was a cube distro that we did in the in the past with with another company. Oh, okay. Okay. Got it. So so if if ties is resemblance between k0s and faros, it's it's mainly because there are some some of the same people are are behind both of these. So but I think from from technical point of

35:04 view, faros and and k zeros are actually completely different. So Okay. Got it. Nice. Okay. So should we take a look at k zero CTL or k zero's control or k zero's cuddle, whatever your preference is? We we don't have a preference. So so I always always use the the CTL, like, kubectl, and I guess that's the the the one of the most important battles in the in the cloud native ecosystem, whether it's kubectl or kubectl or It's just technology in general. We all pronounce something different. I mean, I can't like, sometimes I say SQL lite, sometimes I say

35:46 SQL lite, sometimes I say MySQL, sometimes I've given up trying to find any sort of rhyme or reason to this. Yeah. Yeah. Yeah. I'm I'm pretty much the same so. Okay. So we are going to install the k0s control and I know on purpose they're gonna say it a different way every single time. Is there a brew tap, or would you suggest I just grab the release from the GitHub? I don't I don't think there is a brew tap. So probably just, like, either go get it or just download it. Okay. Darwin. That's empty. K0s cuddle.

36:00 Installing with k0sctl

36:37 Here's the definition of cuddle. All close in one's arms is a way of showing love or affection. Thanks, Google. There we go. We don't have k0s cuddle installed on our machine. So do you wanna give us the the high level overview? What is this tool for, and when should people reach for it? Yep. So so so caterers cuddle is is is basically designed as a as a sort of a helper tool to to like like a special purpose helper tool to set up k zeros over multiple different hosts. So as as we saw when you when you

37:24 when you did that setup for a for a single note, that's, I mean, it's it's super easy. But but imagine you have have, like, hundreds of notes in your cluster or or even more than it I'm I'm at at least myself, I'm I'm gonna be bored after the second one. So and and when when people are bored, they make mistakes. So so, of course so it's it's mainly like an like an automation tool in a in a sense that that just automates the setup over multiple hosts. And then as as already mentioned, it provides, like,

37:58 also the sort of a day two operational benefits, like like the the seamless upgrades of the cluster and and and whatnot. Okay. Perfect. We've got a question from Alex in the chat who is wondering if there is a reason for the change in the default CNI from Calico to KubeRotor. Right. The main reasoning actually was the the resource usage. So lot of the lot of the use cases we we saw early on where k0s is being used is is is use cases where where the infrastructure is is or or has less resources. Like, these industrial PCs and and sort of,

38:44 like, edge computing use cases, whatever edge means for people. But but but but those sort of use cases where where you you really have want to want to save, like, like, lot of lot of the resources or as much as resources as possible. So that was one of the one of the main drivers. And and, actually, at sort of adjacent to that is is the fact that Calico at that time I I haven't actually checked the the latest versions, but but at that time, it didn't actually support 32 bit ARM at all, which gProuder does so. So those were the

39:07 K0sctl Configuration

39:20 main main two drivers. Perfect. Okay. So let's jump back over to the documentation. We have the tool, and we can use an edit command to generate, I guess, just a default configuration. Yep. Okay. So this is a Is it an actual CRD or just made to look like one? Like, can I apply this to a Kubernetes cluster with some definitions of No? Can't. Not yet? At least at at least currently, you can't. But it's it's it's definitely something something that we had in mind that that that that from the day one, let's make this look like

39:59 a Kubernetes resource. So if we ever want to do that, then we can. A cross plane provider that could use this resource would be really sweet and just have it go out and do all my upgrades and and stuff. Yep. Okay. So this just expects me to have some hosts, which fortunately I have got a user, a key path. Yep. And then I tell the version of k zero's that I want. So this is really a kind of an orchestration tool. I'm gonna I'm assuming I'm gonna do some sort of k zero CTL apply or converge or whatever the sub command

40:36 is. It's gonna read this file. It's gonna SSH onto all of these machines and give me back a multi node cluster. Yes. Exactly. Exactly. Alright. Okay. So let's drop in some IP addresses here from my Equinix console. So we've already burned machine one, so we jump for two. If I ignore the key path, will it use my host default agent? Yes. Good. Because I Well, we're about to find out because I've got a bit of a weird s s h set up. So I may have to Okay. Quickly jam some keys, but we'll work at what happens.

40:48 Hands-on: Deploying a Multi-Node Cluster with K0sctl

41:12 And I need to copy these five lines and one more IP address. Okay. And we're gonna install 121 by three and save. Okay. Yep. And apply. Awesome. Okay. Yep. What do you think the chances of this working with my SSH setup are? Are we feeling confident? I'm I'm feeling fairly confident. Yes. I just remember that we've we've had some problems with the with SSH agent based setups, but but I think it worked. My ad blocker is clearly blocking the title metrics, though. But other than that, I think we're okay. I think there's a there's a flag or

42:00 environment variable for you to actually just disable the metric stuff completely. Sorry. When I first brought in this ad blocker to my home network, I actually blocked the traffic for the stream software. So that was the the fun couple of days for me to try to debug that. Guess it's just doing this thing. How long does it take? I guess it's just going on to each machine. Does it do them concurrently? Like, at the same time, is it spawned on multiple processes or does it do the control plane first and It it it it does the the control

42:35 plane first, of course. I mean, there's no point of of going to the work or not if we don't get the control plane working first. So Yeah. Makes sense. I guess Oh my god. It's it's annoying to see these errors, actually. I don't really, really, really wanna get rid of that. I mean, if if if sending the metrics actually errors out to you as a user shouldn't really care or have to care about that fact. So Yeah. I guess I need to do I need to do something about that. I I definitely am supportive, though of open

43:14 source software having this at home. It's one of the most difficult things in the world is trying to understand what versions of people are using and and and they're still actively using it because you need to know where to apply effort and maintenance and stuff like that. Yeah. Exactly. It's just that projects are doing this now. It's I think it's better longer term. Yep. If I could enable it just for this offer, I would. I promise. So I can run what I see here is this is gonna download that cube config to my local director.

43:44 Yep. Or it it'll it'll actually spit it out on the on the screen. So you'll have to now probably pipe it to something. Yeah. We didn't get any error messages. That's okay. Wonder if there is a flag. Must be an environment variable. Okay. So And it said that it it might be like a like a hidden option. Oh, I'm gonna need to kubectl, kubectl, kubectl, Look at that. Yep. Everything's up and running and fine and dandy. But but here you actually see that that because remember that you have three hosts in the in the YAML.

44:26 Verifying the Multi-Node Cluster

44:38 The control plane is not listed. Right? Because it's not running a kubelet or even container d or else like that. Yep. Exactly. Exactly. Yeah. When when I first seen two, was like, oh, we're still waiting on one. And I was like, oh, no. It's it's the the control plane isolation. Yep. That that's probably the the the most asked question that we see on Slack or or even in in GitHub issues that that why don't I see my my note here? You shouldn't. So the the the it it basically works exactly like like if you if you get

45:09 your Kubernetes cluster from Amazon, say, you don't see the controller nodes. You just get the API address, and that's it. It's it's pretty similar here. Nice. Well, that's pretty neat. I like that tool, and it's pretty straightforward. You know? Just adding the IP addresses of each of my machines. I like to just use the the SSH agent. I was worried that wasn't gonna work, but it just worked. So, yeah, pretty solid tool. I like that a lot. Is there anything else with k zero CTL we should take a look at? I think that that that covers the that covers

45:51 the the the, like, the sort of basic basic stuff. I'm gonna run a platform. Of course, now you now you. Yeah. Well, it it should actually dump you a a tarball in your current Yeah. Directory. Yep. So what is that backup of? Is that the database? Yeah. It's the kind database, the the the CA certificates, and and and, basically, the needed state of the control plane. Okay. Awesome. I mean, of course, the we we we have to get that that CA into a safe backup place because if you if you wanna change the CA on the cluster, then, well,

46:40 it's a slightly more difficult exercise. Yes. Definitely. Let's tackle a couple of questions, and then we'll see if there's anything else we wanna just, we wanna run over. So we've got one more from Alex. It's just a follow-up from the Kaleckel cube rooter question. And I guess Alex is just saying, yeah, that makes sense. If we want to use on local and for a medium sized virtual machines, I can still use Calico. Right? So I think what you said is it's all swappable. Right? And all this yeah. And and and it's it's, of course, all

46:54 Q&A: CNI, Debugging, and Windows Workers

47:14 swappable, but but we do include Calico within k0s itself. So so k zeros itself has the capability to run either or Calico. But then, of course, there's a third option. You're, like, bring your own. So so you can deploy k zeros without any CNI, and then it's up to you to configure whatever CNI you you wanna use. But we we support out of box, Calico and Cube Router. Alright. Thank you. Is asking, so how do I debug the control plane? I guess if something goes wrong, what what's the options then? Is it SSH onto the

47:49 machine? Usually. Yeah. Usually. Yeah. Or use use whatever whatever mechanisms that you that you use to to connect the note to the notes and and and whatnot. So Yeah. So I guess as part of the you know, when you're deploying k zero's control plane node, you'll probably wanna stick some monitoring on that machine, get some logs out of it, and a few other bits and pieces. Yeah. Yeah. Absolutely. Absolutely. Okay. Yeah. I don't think Alex has a question there. I think Alex is just agreeing with the control plane not showing up on the list. So we're all good.

48:28 Alright. Awesome. Really cool project. I like that YAML format and just spinning up and deploying the machines. I think that will save me a lot of time and a lot of manual steps. Is there Yep. Anything else with k0s or a k0s CTL that you think we should cover before we finish up today? I was I was not really planning, but I was I was prepared to to show a demo with k zero CTL where I actually integrated with Terraform if if if that is something that that people would be interested to see. I think

49:00 k0sctl with Terraform

49:00 we would love to see that if you're happy to share. Absolutely. Absolutely. Alright. Your screen is up. We can see Versus Code and your terminal. Take a look. Yep. Excellent. Let me bump up the font a bit. The font is good. So, basically, I'm I'm I'm using a a smaller European cloud provider called Hetzner for this demo, mainly because they are super, super fast to spin up all the all the needed needed infrastructure for demo purposes. So it's it's basically a simple simple, like, three plus three case where I spin up three three controllers and and three workers and and with

49:52 the c x 31 type. So I think that, if I remember correctly, that's like four CPUs and four gigs of RAM. So but I mean, it's a it's a it's a typical three plus three case. And and as we as as we learned from the slides and the discussion, whenever I run AJ control plane, of course, have to have a load balancer in front. So so my Terraform also deploys this Hessner load balancer thing and connects it to the to the workers and the the usual cloud cloud stuff. Let's let's call it cloud stuff. People

50:32 get sometimes annoyed when I refer to com complex technical things as stuff. I'm okay with it. They're alright. Yeah. Yeah. One of the one of the neat tricks that that we we have in some of the examples and and and some of the documentation on case or CTL is this use of this Terraform output variables. So, basically, I have the the the hosts like you had in the YAML. I I I basically concatenate the list of controllers and workers, and I just have the similar structure or the same structure as you had in in YAML in in Terraform output.

51:13 So and then then I define that the output is actually a YAML encoded value of this Terraform variables. So what what in the end I actually get out is is pretty much the same that, like like, you did manually. So what this allows me to do is is is do stuff like like Terraform apply. Yeah. Yeah. Yeah. I trust what I'm doing. I don't really, but let's assume that I do. Yeah. You see see the main reason why I use Hetzner in in many of the demos because it takes, like, twenty seconds to boot up six

51:55 Applying K0sctl Config via Pipeline

52:07 six VMs and a and a load balancer connected to those. So Yeah. That was fast. Pretty neat. Yeah. So what I'll do is I'll I'll take the output as a raw because that that'll be the YAML encoded value. And then I I can actually pipe it to k zero CTL. Apply, and then we don't need this redirection. So, basically, I said said that, okay. Let's apply whatever you get from the standard input. So this is a sort of a neat pattern that we can use with k zero CTL tool. So imagine you could actually actually have, like,

52:53 a, like, a CI CD sort of a pipeline for your infrastructure and and for your Kubernetes clusters with with this. Yeah. I can see, like, a GitHub action that runs a Terraform apply, and it passes the the output forward. And then you got the k zero CTL kick in, and it off it goes and does its thing. So Yep. Exactly. Exactly. Or maybe in future, you could actually actually dump the dump the k zero CTL YAML in a in a Kubernetes API, and then some some magic some work kicks in then. Yeah. I'm looking forward to seeing that crosswind

53:32 provider. You'll have that ready for me next week. Right? I no. No promises. No promises. No promises. But to be honest, it's not the first time I've I've heard the idea. So Yeah. I think it would work really, really well. Because you could have like a k zeros that runs cross plane on a single node setup and then apply your other k zero's control YAML's to it and then have the controller cross plane or otherwise go and create more virtual machines and deploy. That would be a pretty nice setup. I like that. Yep. Yep. Yep.

54:06 Or then what the what the what the, like, the this true control plane isolation actually actually allows us to do is is you could basically run the k0s controllers in a pod because there's no requirements for kubelet or anything else. So what's it's it's just set of normal processes. So what's preventing running in a in a pod. Yeah. Definitely. So there's sort of a miter ship type of a type of a pattern. Right. Looks like it's done. Yep. It's done. I got my three plus three setup done and and yep. One sec. Let's dump it out.

55:00 Export. Keep config. Keep k0s. Get node pod dash a. And there we go. One thirty node k zero's cluster. Yep. And then h a h a control plane with load balancer and everything, and it took me, like, few minutes. Even with even with my typing speed. So Very nice. I like that. Yeah. But I think that that that of course, we or or it it doesn't, of course, matter how how people set up the k0s cluster whether you use, like like, Chefs or Ansible or Solstacks or whatever tools you use. It doesn't really matter. We just wanted to build this sort

55:54 of a, special purpose tool to help with the day two operations, like upgrades and and everything. So, it'll it'll make life easier in in many cases. Awesome. Maybe we can actually actually try an upgrade. Where did I have the version? It's variable. Yes. And the the virus fail. The virus file actually contains a secret too, so I'll switch. You don't see the screen now. Right? No. But I think it did show us a virus fail at the start. I I saw showed the example for Ah, the example. Right. Okay. Gotcha. Yeah. Really should learn to do these sort of things. That's alright.

56:08 Hands-on: Performing a Cluster Upgrade with K0sctl

56:49 I flash my secrets on the show a couple of times a week. People have been nice and and not hacked me yet. So thanks, audience. Yeah. Alright. So what I what I did actually because we we see that we actually have this 21 to two version running. So so I'm gonna actually bump it up to to 21 to three now. So it's it's, I have to do that. Apply for the Terraform first. You see that in in the Terraform output that the version actually changes. Mhmm. Doesn't change anything else. Then I'll do the output and

57:31 apply thing. We'll see what the the upgrade process actually looks like. Let me erase that a bit. Based on what you said earlier, what k zero's control is gonna do is SSH on to each machine, pull the latest binary, and then basically just flip them over? Yep. Okay. We've got a a question in the chat from. How's it going? As a Windows workers know, it's something that has been thought of or working on? We do have a experimental support for Windows workers too. So in the in the k zero's download release page, you actually see a k zero's

58:20 dot exit already existing. There you go. Try it out, and then I'll let us know how you get on. Yeah. It has an experimental label, so be very Nuno is not shy from experimental labels. He is always playing with That's what I've learned too. Yep. Yeah. But I think that's the the the the the whole, like, cloud native native world that that what I've been I've been basically telling everybody whenever working with Kubernetes or or anything. If you if you see, like like, something dash something slash v one beta one, well, just use it. It's stable enough. Yeah.

59:00 V one beta one's too mature for me. If it's not an alpha API, I'm not interested. Yeah. Yeah. Yeah. So beta bet beta is the new stable in the Exactly. Cloud native world. So Well, it was only 01/01/2016 where we could write a v one beta one for leak deployments and everything else. Like yeah. Yeah. Yeah. If it's v one, then it's almost like legacy already. Right. Alright. So so what what what do you see? What actually happened is that it upgrades the controllers basically one by one, and then it moves on to the onto

59:35 the workers. But because I have, like, three workers only, we can't really do that, like, in parallel upgrades that well. So by default, the the the k zero's cuddle actually takes, like, 10% of your notes and and runs the the the upgrade in parallel for that 10% at a time. So if you have 20 notes, it'll actually update two notes at a time and and and so on. So it and and then it actually does this this typical upgrade or the drain update on cord and dance for each of the notes. And and at at all steps, it waits that everything becomes

1:00:13 ready again and whatnot. Awesome. Very cool. I'm glad we stuck around for that extra bit of demo. The upgrade was nice. I mean, we got some love in the chat for the upgrade as well. Very cool. Alright. Well, that is k0s everyone. I hope you liked that. You've got five seconds to get any more questions into that chat before we say goodbye and let you say get back to the day. Well, if you have any questions, drop them in there. So I'll finish with a a question that I ask quite often. It's just like, is k0s

1:00:28 Future Plans: Autopilot

1:00:49 finished, complete? Are you just tracking upstream now, or do you have any new shiny stuff coming down the line? We do have we do have some new shiny stuff coming up. So one of the one of the things that we are we are working work actually quite early early in the process, but but we've started to work on a on a feature what we call, like, autopilot. What it'll it'll do is is is it'll take the cluster itself, and and it'll it'll apply the cluster itself will handle all the upgrades and updates and everything by itself.

1:01:25 So we'll we'll basically basically bundle this upgrade logic and all the control mechanisms between the node trains and and and on coordinates and whatnot. We'll we'll bake it into the into the control plane itself. So, basically, it's a it's a cluster that is on an autopilot in a way. Nice. Sign me up for that. We got one question that has snuck in there. So is asking it. It's can the 10% be controlled? I assume this is in relation to that upgrade. Yeah. Oh, I I suppose that it is. If it's not, then then we have to

1:01:49 Final Q&A: Upgrade Parallelization & Storage Issues

1:02:03 make it as an argument or some sort of a parameter on somewhere. Yeah. To request welcome. Right? Yeah. Yeah. Yeah. It's just changing one magic number to something else. Alright. There's maybe a a good idea for a a nice simple contribution that cook gen. So if it isn't configurable already, feel free to give that a go. Yep. Absolutely. Absolutely. Alright. Well, thank you so much for joining me today. Really good to see that. Our demo well, I say our demo. Your demo went off without a hitch. It was really nice. I hadn't actually used Tertra cloud before,

1:02:38 but the speed of spinning that up was pretty impressive. So maybe I need to check that out at some point as well. Alex, we can then move one final question. We we can do it. Right? Yeah. We got a little bit of time. Yeah. Alex says, I tried once to deploy Portworx to k0s, but I could not get it to work. Any ideas if it has something to do with the locations where it stores the files or configurations? I I do remember reading about that issue, but I I can't remember the the details where we where

1:03:12 we landed on on. And and to be honest, I don't I don't really I don't really know port works at all, how it works and what it does. So so but it it probably has something to do with that where where where k0s puts in in in the files and and sockets and and everything. Everything. So that that's why that's my best guess based on based on the information that I I have and and know about Portworx. Alright. No worries. Okay. We got a thank you from Ty in the chat. So, yeah, we're gonna finish

1:03:43 Conclusion and Wrap-up

1:03:46 this up. Thank you again. You see, really good. Love the demos. Very cool. Any last words before we before I let you go? Well, as as as in any open source project, just we we always appreciate feedback and and and both bulk reports and and feature requests and and everything in in in between and and even better if if you can pull up a PR. But, yeah, thanks for thanks for having me join the session, and I I I I really enjoyed the the format of of your sessions here. So Awesome. Hands on and not not afraid of of, demo

1:04:27 effects. No. No. I've I've I I look silly enough on a stream regularly, so I'm not fazed by it anymore. But Yeah. Thank you. It was great fun, and I'll hopefully speak to you again soon. Have a great day, guys. Alright. Bye.