0:18 Introduction

0:18 Hello, and welcome to today's episode of Rawkode Live. I'm your host, Rawkode. Today, we are taking a look at multi tenancy on Kubernetes through a project called vCluster. Now before we get started on that, there's just a little bit of housekeeping. First and foremost, please subscribe to the channel, like the video, click the bell. This will keep you up to date with all new episodes of Rawkode Live. It will also encourage the YouTube algorithm to share this content with the broader cloud native ecosystem. We also have a Discord channel available at Rawkode.chat. If you wanna request new technologies to appear

0:20 Introductions

0:54 on the show or talk about any existing or previous episode, that is the best place to do so. Feel free to jump in there and join the conversation. Alright. Now for today, for vCluster, I am joined by two fantastic guests, Rich and Lucas. Hi there both. How are you? Hey, really good. Awesome. Thanks for having us. It's my pleasure. I'm really excited to take a look at vCluster today. It's kicker tires on it, seaboard is capable of and share that with everyone that is watching. Can we just take a few moments, please? If you both introduce yourselves, tell us a little bit

1:02 Guest Introductions

1:27 about you, and then we'll talk about vCluster. And I'll throw it to you first, Rich. Yeah. So I'm a senior developer advocate at Loft Labs. I've been with the company for about three months now, and I'm really enjoying it. I've been in DevRel for a few years now, but have a long, long history of different kinds of ops roles before that, sys admin roles, SRE, all kinds of stuff over the years. So yeah. And Lucas? Yeah. Hi. I'm Lucas, CEO at Loft Labs. I brought Rich in a couple months ago and super excited to be working with him. And we launched vCluster

2:10 a couple of weeks ago, and, you know, I'm trying to hop on as many streams and calls as possible to to, you know, tell people more about what we're up to right now. Awesome. Thank you. And thank you, Noel, for telling us my audio was on during that video. After I assure everyone that that is not the case, I guess I made a mistake. Sorry about that. Oh, I guess I missed an opportunity to plug my thing, which is that I also do a podcast about Kubernetes called KubeCuddle. So if you just search in your podcast

2:42 player for that, you should find it. Don't you worry. I was I was gonna mention kubectl anyway. I'm a big fan. It's a great podcast, and I encourage everyone to check it out. Alright. Let's start then with a little tldr on what vCluster is. Which one of you would like to tackle that? I'll start off. So vCluster is basically a solution to the pain of Kubernetes multitenancy. So if you have used Kubernetes for a while, you have probably run into the the situation where you're using namespace based isolation, and there are things you just can't do.

2:53 What is vCluster?

3:22 Right? You can't manage cluster wide resources as a user in a namespace. And so this is actually something that was built into our commercial product, Loft, and then we kinda pulled it out and open sourced it recently. Lucas, is there more you wanna share? Yeah. I think I mean, that that was a great great intro. I think you said pretty much everything except, you know, vCluster is just is more than just multitenancy. It's really just an efficient way to also create self-service for for, you know, Kubernetes clusters. Because, you know, as soon as you have

4:03 access just to a namespace, you can basically now spin up your own Kubernetes clusters. I mean, they are virtual. They don't have their own, you know, VMs and nodes. But with this restricted access, you can essentially make yourself cluster admin for, you know, this virtual cluster, which is, you know, something that people often forget when they think about virtual cluster, and I think it's actually one of the most exciting parts. And, I mean, in terms of the technical end of it, you know, you've got a namespace on the host cluster, and inside of it, there's a k three

4:36 s API server. And that is basically your Kubernetes API, you know, within your vCluster. So it looks to the user like it's a full blown Kubernetes cluster. So what are some of the, like, more common use cases for this? What do you see people adopting vCluster to do beyond, like, just multi tenancy? Is there, like, a broader picture there? I think that a little bit into the details. Oh, sorry. Just go ahead. No. Go ahead. I I I was just gonna say, I think, you know, there there are a number of use cases, but to me, probably

4:50 Common Use Cases

5:13 the most interesting one is, you know, Lucas mentioned self-service, and I think, you know, this is great for dev environments. You know? If you don't wanna have, you know, different clusters spun up for different teams, you want people to be able to do self-service and just get on with their work as they need to. I I think that that's one of the better use cases. CICD is potentially another use case. You know? So instead of using something like kind and having to provision something, you know, every time you run a set of tests, you basically are just leveraging this cluster that's

5:50 already there and and, you know, just creating a name space on it, which I think is is likely a lot faster. Yeah. Definitely. I think, you know, just you're mentioning those use cases there and thinking about how, you know, CICD is one of those things that when you run it in your cluster, it normally needs quite a lot of privilege. Like it does some magic stuff. This is the building container images and it's access to the like host paths or it needs, know, service accounts with maybe quite broad permissions. And if you could toast all of

6:20 that in a virtual cluster, you're kind of protecting the outer layer but giving them you know, you're not slowing down developers and I guess that's almost a superpower to a point and that the development team can now just work with whatever permissions they need without having to constantly come back to the operations team or the maintenance teams, SRE teams and go, hey, I need this new permission or I need this. Yeah. And I think that I think CICD is really tricky, right? Because if you're in a situation where you run a lot of tests and they take a

6:48 while, there's always that trade off, you know, where you're trying to reduce how how long the tests take, and you're trying to do everything that you can to sort of, you know, shrink that amount of time before the developers get feedback. And, you know, you're often, you know, making trade offs. Like, do I wanna have a a VM with a cluster already baked into it? You know? Or do I wanna provision a cluster every time? And there are pluses and minuses to both of those things. You know? The more that you bake in, the more that you have to deal with,

7:22 like, you know, recreating images or updating them somehow. And, you know, if you provision stuff on the fly, then you've gotta wait for all of that to happen. You know? And so the the thing the situation that I think we've gotten into with the Kubernetes multitenancy problems is this cluster sprawl, you know, where people feel like it's easier to just make a new cluster for everything than to think about dealing with multitenancy. You know? And and that's really, I think, a a lot of what we're trying to address with this tool. Wow. Big mission. I'm looking forward to check it out. But

8:00 it sounds amazing and I think there's just you're right. There are so many different use cases. I like the CICD one. That one hadn't really occurred to me until you mentioned it. And then the minute you said it, it was like a little late switch. I was like, oh, yeah. He's totally right. Like, it's a great use case. Is vCluster something We have a actually Sorry. I didn't go rich. Oh, I was just gonna say, David, we've got a really big announcement today. Do you wanna talk about that, Lucas? Oh, yeah. Sure. Yeah. We're we're actually breaking the news as

8:18 Major Announcement: Certified Kubernetes

8:30 of right now in in this call. So vCluster is now a certified Kubernetes distribution. So one of the I think there's, like, 80 distributions out there, and we worked pretty hard on the, you know, past couple of weeks to pass all the conformance tests that CNCF put out there for certified distributions. And, yeah, pretty happy to report that they merged our pre request now. We're not the logo does not show up on the website yet. That's to come hopefully in the next couple of days, so this is really, like, breaking news. But super happy to report that. Yeah. That

9:06 is a big deal. Congratulations. That that's that's awesome. It's a fully conforming Kubernetes cluster running inside of a namespace on a Kubernetes cluster. Cluster. Yeah. We have to give a shout out to our have to give a shout out to our CTO and cofounder, Fabian Cram too, who has built a lot of this. He he's just a machine. He just codes like crazy and build such cool things. He's probably on it right now. So I'm assuming there was maybe a couple of things you had to to to pass the conformance test. Is there are any of

9:44 those quite funny that you wanna share with us? Anything that snuck through? Yeah. I think I think the most most tricky part was actually getting things done in terms of notes because, you know, Kubernetes has certain restrictions regarding you know, if you're thinking about a a virtual cluster that runs inside a namespace, what happens to the notes? Right? Do you see all the notes in your underlying cluster? Do you only see the notes that you actually have workloads running on? Do you only see, like, notes for certain note selector and all these kind of things?

10:22 So, yeah, we had to we had to add a couple of flags to, you know, essentially enable certain configurations to, you know, sync all notes from the underlying cluster and things like that because, you know, that made it much easier to run these tests that that, you know, the conformance tests essentially require. Awesome. Yeah. I hadn't again, I hadn't really thought of that. What I would expect get nodes to return inside of a vCluster. I guess based on what you just said, that's it's all configurable then. So me is the person deploying vCluster. I I decide what what should come back

10:58 there. Right? Yeah. Exactly. I mean, we we we're trying to add as many, you know, config options as possible. VCluster is actually pretty much I mean, you can either just deploy a stateful set, right, and a service, and that's vCluster. You can basically run cube CTL apply. So it's basically just what what arguments you provide for the for the entry point of the two containers. Right? We have a k three s container, and then we have the sync container that actually does the synchronization between what's inside the virtual cluster and what's outside in the real cluster.

11:01 Configuration and Flexibility

11:33 And, yeah, typically, you deploy it via Helm, and we try to expose as much as possible. So, you know, there are so many different use cases for virtual clusters. We talked to one company that started using our product to essentially, you know, kinda simulate clusters and simulate, you know, what happens. So they wanted to test, like, what happens if we run, like, 500 daemon sets. Right? And, you know, our first five hundred nodes, and then the question comes up, hey. Can we actually produce fake notes with vCluster? So that's something we're currently looking into. And, yeah,

12:06 there there's so many use cases that people come up with. It's incredible. And this is pretty new, you know, in terms of the open source project. We just released it during KubeCon publicly, but but this is technology that our users on our commercial product have been using for a while. So it's actually a lot more baked than you might think from, you know, a brand new open source project. Alright. Well, my interest is certainly peaked. So why don't we get this deployed to my cluster? And feel free to share all of your experience and history of

12:35 Getting Started: Deploying vCluster

12:43 the project as we go and I will do my best not to mess this up too much. So I have my screen here. This is the vCluster documentation. We're gonna work through the getting started guide And if nothing has broken in the last ten minutes, we should have a cluster. We do. Alright. Step one. I'm assuming I just deploy vCluster. So do I need the CLI? Is that Yeah. It's gonna be much easier if you're using the CLI. Well, I do like easy mode, so let's do that. Yeah. And, of course, you know, this is

13:00 Installing vcluster

13:31 a quick start, so it's not necessarily how you would, you know, productionize this, I guess. But Oh, yeah. That's that's alright. Quick start, sir. They're more fun. Mhmm. Deploying stuff to production is too hard. I'll deal with that deal with that in the future. Okay. So we have the vCluster CLI and then step two is that we wanna create a vCluster. I know it looks like you offer let me zoom in on this a little bit, multiple approaches to do this. So it feels like is the vCluster CLI just optional? I can do this through Kubernetes

14:10 manifest, but like you said, the vCluster CLI is gonna keep this really easy for me. Okay. I see he's nodding. Yeah. Why I mean, you'll you'll actually see the output in the CLI and it essentially runs Helm under the hood. So if you already have Helm installed, you're gonna see the command on there. Alright. So let's just copy this command And this is vCluster create vCluster one. Okay. So this is just the name of my virtual cluster. And then this is the namespace to run it in? Yep. Okay. So do I need to create that namespace first or is that created for

14:48 me as part of the vCluster create? You just need to do the command and Alright. Let's see what happens. It worked? Is that it? That's it. VCluster running now, hopefully. So and and, again, when we were talking about, you know, the speed difference, you know, within a workflow like a CICD workflow or something like that, This is literally just, you know, deploying a couple containers with a Helm chart. Alright. I mean, it's not supposed to be this easy. I'm just gonna throw that out here. Like We can we can screw it up. I didn't is this clustered?

15:44 I thought that this is Rawkode Live. Yeah. Yeah. That's right. You're right. Although, I'm already thinking that v vCluster can provide some pretty interesting clustered scenarios so maybe another day. Alright. Let's see what it's done to our cluster then. I know I'm going off script, don't know if the getting started guide is gonna actually guide me through this but the questions in my head are what have you done to my cluster? So I can see that namespace was created. I wanna see if we have anything running inside of our namespace. And it looks like we got

16:00 How does vcluster work?

16:20 a core DNS running and a Rawkode live zero. So that's the stateful set that Lucas mentioned earlier. Are those the only two modifications to my cluster or is there something else? You should see a service as well in your cluster. A service. Yep. Oh, yeah. I mean, I guess two services because it's a stateful set. You're also gonna need the headless service. But yeah. And then, Eric, actually, what you see in terms of DNS here, that has already been deployed inside your vCluster itself. So that's kind of how you see how things are met to the underlying cluster. So

17:05 everything you're creating inside the vCluster and the first thing that vCluster does, it deploys DNS to itself, essentially, is, you know, all the parts that are being created and some of the services that are being created inside the vCluster will be synchronized down to your host namespace, and that will be rewritten. So you kinda see the cube systems or the x cube system x here inside the name that actually shows you the namespace inside your vCluster because we're mapping, you know, your 500 namespaces inside your vCluster to a single namespace inside your underlying cluster, essentially.

17:43 Alright. I wanna digest this a little bit. I wanna make sure I understand what's happening here. So I think the first thing I wanna make clear to people is that if I run pods instead of virtual cluster, those are still just pods running on my host clusters. That's correct. Right? Yeah. Exactly. So the cluster has no scheduler. Yeah. So what it does is there's a sinker that syncs those pods to the host cluster. And so as we as we get into it more and, you know, you deploy something, if you have something you can you can

17:45 How vCluster Works: The Syncer and Resource Mapping

18:20 run inside there, we can look at how that looks. Okay. I I I wanna clarify a few more things before my you know, that mind blowing emoji. Feel like my head's gonna be doing that in a minute. So what we have in our Rawkode Live namespace here, that I'm assuming that Rawkode Live stateful set zero pod, is that an API server? Yes. I mean, there's a regular k three s API server. And, typically, it has a SQLite database, but you could also you know, we expose all the k three s flags, essentially. Right? So you could essentially

19:00 use etcd as a back end, right, or MySQL as a back end rather than SQLite. But SQLite is the easiest and, you know, quickest option for, you know, quick start, so that's why that's the default option for us. And then you also have a controller manager. It's all packaged in k three s essentially. Right? That's kind of the power of k three s packaging everything in that single binary. And then the only thing that's missing in terms of Kubernetes control control plane is the scheduler. Right? Because we don't have any separate nodes, and we actually wanna, you know, deploy things

19:35 in our underlying cluster. And instead of the, you know, instead of the scheduler, we're using a syncer. And that syncre monitors, you know, what's happening inside the virtual cluster and then synchronizes resources down to the host cluster to the same namespace. It rewrites certain things as you can see with these partners, etcetera, you know, to provide prevent naming conflicts because we're mapping to a single namespace. And on top of that, it synchronizes the status of these objects back into the vCluster so that if you're inside the vCluster, you see, you know, image pull back off

20:11 and all these kind of statuses that your pods can have, and it looks like a regular cluster. I was That's true. Mind blown emoji when I started playing with this too. So I definitely get the feeling. It's it's super fun. I I I think I've got a handle on it now. So I'm gonna walk through the life of a a pod on a vCluster and if I get anything wrong, please feel free to correct me. But you know, you've got this KCS API server running on the host cluster. I can add whatever configurations to that I want through validating web

20:47 configurations, mutating web configurations. I can run my own operators and controllers in it. I can apply all my resources to it and all of that just works and then the magic sauce is the sinker then. So it's just said in looking at the SQL lite kind database in the KTS and then propagating enough of that back to the host cluster to be able for the workloads to start running, but then you also pull back through the status updates from the host cluster. Does that ever lead to contention between what the host cluster wants to do versus what

21:20 the KPS ever wants to do, or does it just work very nicely and harmoniously? Yeah. I mean, it works pretty nicely because the responsibilities are pretty isolated. Again, I mean, it's basically just a different kind of scheduler. Right? A scheduler that instead of actually scheduling work scheduling workloads is basically just copying workloads to be handled by another scheduler. That's what it's doing. I guess what I'm thinking is like, if I have a mutating web configuration and the host cluster which is adding a set of labels and then the host for whatever reason has validating web hook which doesn't

21:59 allow those labels, You then have to kinda navigate a little bit of turbulence, but I think that is so contrived that it probably just wouldn't happen. But that's again, that's Yeah. I think it's actually a super exciting scenario because, I mean, that that's kind of the world where we're coming from. You know, we we created vCluster. Again, as which mentioned, was part of our commercial product. Right? Our commercial product is essentially provisioning development environments in form of namespaces for engineering teams in a kind of self-service fashion. So, typically, you have all these constraints that you just

22:09 Security Considerations

22:32 mentioned. Right? You have network policies. You have validating, mutating webhooks, etcetera. Whatever your admin teams defines to, you know, security operations of that shared multi tenant cluster that a lot of the teams are, you know, creating namespaces in now. And then when you do that, you're restricting your users. And that's essentially what we think when we were thinking about, hey. What if users need to do things across namespaces? Right? Now the network policy needs to be adjusted for that particular use case. You know? Are there things that we can do to make those things easier? How can they deploy

23:04 their own CRDs? You know, have their own controllers and work on those kind of more advanced objects? How can they mess around with r b, you know, just because that should be part of their, you know, helm chart that they wanna, you know, check-in to get later on or something like that. And that's essentially you know, the answer to those questions was vCluster because you just need access to namespace, then you deploy vCluster, and now you're admin of that vCluster. And, yes, you can add a lot of things to that cluster. You can add, you

23:32 know, validating, mutation control, etcetera, but you're still gonna be also validated and mutated at the lower level. Right? That's actually the beauty of this because, I mean, your pop gets checked twice. Right? Because the syncor connects to your underlying cluster and basically does a QTTL request. Right? And if that fails, you're gonna see that in the status of the of the virtual cluster as well. So it's transparent to the user. Right? But, yeah, I mean, there are different there are more layers now that, you know, you can add these, you know, constraints essentially. Yeah. I guess that's almost a double edged

24:08 sword as well. I can imagine a situation where people may think, I'll use the virtual cluster so I can run containers as root, but the host cluster may have a policy that doesn't allow that. So not violating the host policy, it's still gonna block it, but at least they can try in the virtual cluster, which is quite interesting. I like that. Okay. Yeah. And I mean I mean, I think as a as a user I think as a user, you would wanna get educated about what those policies are anyway. Right? Oh, yeah. Because eventually, the things that you're playing around with are

24:40 gonna hopefully, you know, land in production, and you're gonna be under the same kind of constraints. Okay. I'm curious. And instead of asking a question, let's let's do something with our vCluster and see what happens. I'm assuming and I know I should probably go back to the docs. In fact, I will go back to the docs. Unless it doesn't do what I want to do next, in which case I'll go off script again. But oh, so we can do a connect. That is gonna do it. Right? This is what I was curious about. When I connect to the virtual cluster and

25:15 Namespaces in vCluster vs Host Cluster

25:17 I create a namespace, what happens on the host cluster? I'm assuming you create a namespace there and there's a mapping between the two, but maybe I'm wrong. So let's do this connect command first. And this is where I'm gonna regret changing the name of the cluster because now I have to change all of my commands. There's a rookie error, but oh, I'm not gonna pressure turn on that because this creates a kube config dot yaml and I may have one of those. Do I? It creates it in your local directory though. Yeah. This is a cluster API

25:57 cluster, and I have a KubeCon. Oh, but not not a YAML. Okay. We should be okay. I was worried I was gonna blow away my KubeConfig, and that would have been the end of our session, but we should be alright. Okay. What do we got? We got a config written to dot YAML, so we're safe. We can access the cluster by using the config and it started up port forward. So do I need to leave that port forward running to interact with the cluster? Yeah. Okay. Yeah. You're gonna wanna do that. Come on computer. There we go.

26:42 So kip config. Look at that. I mean, that's a bit weird to forget pods all, but I understand why. I would be very worried if this was cluster and that's all I could see, but this is what we expect, so I'm happy with that. Well, I'm gonna create namespace. I'm gonna try and trip it up just because I like to do that. Alright. So that did work. Let's run cat names basis. And I'm gonna do that again on Yeah. Yeah. Okay. So interesting. It's not what I expected, so maybe we could dive into that a little bit.

27:43 When I create a namespace on the virtual cluster, you either don't create it on a host cluster until I add some workloads to it or you just don't create it at all. You wanna dive into that a wee bit? Yeah. We don't create it at all essentially. So what we do is we map your x namespaces inside the vCluster to the single namespace where the vCluster state for set is deployed to. So no matter how many namespaces you create, all the pods that you create in these namespaces end up in the same namespace in the underlying host

28:19 cluster. Of course. Because you can use vCluster when you only have access to a namespace. I didn't think of that. You you do confine everything to that boundary, which makes a lot of sense. Exactly. When when you create a name What is this? That's a request to the k three s API server, and it's stored in your SQLite or, you know, etcd or MySQL or whatever is the back end for your k three s API server, and that's it. Nothing else happens. Were you gonna say something You can literally delete that namespace and everything goes away.

28:54 Alright. So we also got a question from Rio in the chat, which I think we've just answered, but I'll throw up anyway and you can feel free to add any extra flavor to that as you want. But Rio says, so a user who deploys vCluster only needs to have admin role bindings to a single namespace. That is correct. Yeah. That's essentially the core story for for vCluster. As as long as you can create, you know, a stateful set and a service inside a namespace, you can and, I mean, you have to have the privilege to also do port forwarding if you

28:56 User Permissions Needed for vCluster

29:29 wanna connect to the g cluster, obviously. But other than that, yeah, you don't need any privileges. So if you have admin role binding in that particular namespace, then you can deploy the cluster. Alright. And Noel has a comment. I'll pop up, but I think we kinda just answered that as well. Noel is saying that the the sinker just manages everything from the vCluster namespace back to the host. Do you wanna talk, Lucas, about which which objects get synced? Yeah. So there is essentially we kind of differentiate a little bit between you know? And we have a separate docs

29:59 Resource Synchronization Details

30:10 page about that as well regarding high level and low level resources in Kubernetes. So we see the lower level resources, like parts and service, different from higher abstracted resources like, you know, deployments or stateful sets or even CRDs. Right? And what we need to sync is essentially parts, services, and everything that parts need to run. So let's say you mount a secret. Right? Then we may need to sync that secret. Right? Otherwise, we can't mount it. Or if you mount a config map, it's the same scenario. But all the other 500 secrets and config maps you may have in your vCluster don't

30:53 need to be synchronized. So we're pretty selective in what needs to be synchronized. But, obviously, we need to synchronize everything that's required to actually start a certain part, and we need to synchronize services just so that we can make networking between your parts inside the vCluster work and that they also, you know, can communicate, you know, via services rather than just on an IP based level between each other. Okay. That makes a lot of sense. I think it really smart that kind of selective sync based on whether something is is actually being consumed or used is really clever.

31:31 I mean, is there things that cause problems here? I'm thinking storage and CSI plugins like, I guess, that you have access to whatever the host has, or is there something a bit different there? Yeah. I mean, that's typically I mean, you can have so, essentially, the vCluster can have a subset of things that are available in your host cluster that depends on the configuration. Right? So, essentially, think about it like a storage class. Right? If we if you were to use a storage class and you wanna create a a persistent volume claim, for example, in

31:35 Handling Storage and Host Resources

32:10 your in your vCluster, you gotta have that storage class. Otherwise, you can't use it. Right? Otherwise, it does it simply doesn't work in Kubernetes. And that storage class, you know, obviously needs to be from your underlying host cluster because that's where that part actually got started. So you can't just come up with, you know, other resources not available in your host cluster regarding, you know, actual part workloads and volumes, etcetera. Alright. That makes makes a lot of sense. We got a comment from Rio who's obviously very excited for their go to custom Docker and Cain CI setup. So

32:52 we've got a we've got a vClusterConvert already. I'm really half of their end. That's awesome. I I do wanna say that we love kind. Like, it's an amazing project and it's done so much for the community. I definitely think they're you know, vCluster isn't necessarily the right tool for every situation, but but I do really like it for that CI use case for sure. Nice. We got a security question from Noel in the chat who is asking whether the KCS API server runs as a non root. And they're wondering about potential, you know, user breaking out of the virtual

33:29 cluster into the host cluster and whether that should be a concern. That's actually a pretty good question. I'd have to look pretty deep in the code myself. Don't know what the answer to that is. I assume if you can run it without root, it would be possible with vCluster as well. I don't know if that's it's the default for us or not. I guess they're still contained with the the namespace that vCluster was deployed to as well. So if you got your Rawk policy Rawkode policies correct there, you know, it's it's pretty strict, I would imagine.

34:06 I actually Yeah. I assume that have some is just a single binary as well. So even if you, you know, were to break out of the actual process there or something, you'd end up in a probably, like, distroless container. I assume that's how k three s is packaged, but I'd have to look in how they're packaging things because we are basically just using, you know, the image off of Docker Hub. So I don't know if it runs with the root user or, you know, what's inside of there. Yeah. I don't I don't know the the

34:34 answer to that question either, but I do have some friends that are pretty security minded that we're playing around with it. And the thing that they emphasized is the importance of doing admission control, you know, on the host cluster. Yeah. That makes a lot of sense too, definitely. Okay. I deterred this a little bit there, so let's jump back to our Get Started Guides. So we did the namespace, we checked that out, we created a namespace. Now we're gonna deploy our first workload to our virtual cluster. So I'll need to change this a little bit.

35:17 Oh, it's the namespace. Oh yeah. Because I decided just to trick it. See and then the image. Okay. Awesome. So we should be able to run get pods all namespaces. And we have an NGINX pod being spun up and a Rawkode live namespace inside a Rawkode live virtual cluster inside a Rawkode live namespace. Easy. It's a lot of Rawkode live going on here. Yeah. I'm I'm starting to Rawkode live inception. There's a lot of inception vibes going on here, that's for sure. So let's run. I just like seeing what happens on my host. I'm assuming based on all the conversation

36:14 we've had now is what I expect to see is in my Rawkode live host namespace. There to be a pod that's potentially called Rawkode live Rawkode live Rawkode live. Because it would be namespace vClusterName and pod. That sound right? Yeah. Pretty much. That should be what it was looking like. Yeah. Oh, we don't wanna do all namespaces here. We wanna see here. Oh, no. Okay. So I never called my engine x deployment Rawkode live. I should have. I missed the trick. But we got Rawkode live, Rawkode live. There you go. That is cool. I like that.

36:56 And yeah. So the secret sauce is the sinker. Right? The sinker is the one that does all of the magic here. It feels to me like there's so many there could be more applications to this sinker than maybe just virtual clusters. Like, I'm thinking maybe cluster federation or disaster recovery and being able to copy stuff over. It feels like that could be applied in actually maybe a couple of different ways. Yeah. It's actually pretty crazy. Again, I mean, since we it's even crazier since we open source vCluster, but even before as part of the commercial

37:32 product, it's crazy what people, you know, come up with in terms of ideas. And so far, you know, we started with essentially virtual clusters as a way to, you know, give extra privileges to users that are confined to a namespace, typically. But then the next question is is the one that that you are asking that that a couple of folks have asked us as well is, you know, can't we back at vCluster by different physical clusters. Right? It's a super interesting thought because the syncer, I mean, basically, just wants to keep CTL apply command. Right? You know,

38:09 technically. I mean, obviously, it's using client go, etcetera. But that's what happens under the hood, and it could technically, you know, route different parts to different clusters in a way or, you know, sync them to, like, five clusters in parallel. You know, they're they're essentially unlimited possibilities here, and it's super exciting to explore those use cases. But so far, that's not a main priority. Awesome. Okay. So we can clean up our vCluster now using vCluster delete, or is there a few things that we should kick the tires on before we do that? What else does the vCustr CLI Yeah. Do?

38:51 Right. Doesn't do a whole lot more. I mean, you can upgrade the the vCustr CLI. Right? Other than that, it's really just create, connect, delete. I guess that's all you needed to do. So nice. Alright. What should we do over a virtual cluster? Do have any any cool things that you normally deploy to this? Yeah. I think one thing on the checkout is essentially what happens with the nodes. You you do have a multi node cluster. Right? Yeah. We've got a control plane node in two workers. Yeah. Alright. We we could check out, you know,

39:11 Exploring Nodes Within the Virtual Cluster

39:33 what happened what happened with our part. Do we see nodes inside the vCluster? That is typically what a lot of users are interested in because that's a it's it's a non obvious question. Right? What what happens there? Yeah. We actually we we touched on that, and then I forgot oh, no. That's named spaces. No. It's Alright. So I see both the workers and the control plane is removed. I guess this is the the default configuration for how to expose the notes. I think the default right now is we actually changed the default recently. I think the

40:13 default as of today is that we show you the notes that are actually that you have workloads running on. So your pods get scheduled by the underlying cluster, and I assume that you have two parts, you know, at least or, like, one part on each one of these nodes, but not one on the other one. Probably because you have some kind of thing there that doesn't allow parts to be scheduled. Yeah. We got our core DNS on one of the nodes. We got NGINX on the other. And I actually removed the tint from the control play notes,

40:47 so we had three to play with. So I guess that means that I could do an edit deployment engine x. What do we call that? A demo engine x. Oh, there's a namespace involved too. But did I change again? I'm gonna have to have a word with myself. Alright. Okay. You need to learn the power of copy and Copy. Yeah. Copy and paste, David. I know. I'm just gonna scale up. See if we can get Yeah. That onto the control plane node, run the get nodes. I I like I like that it kinda tries to work

41:43 out which which nodes it has to expose. I can imagine in a scenario where I've got my host clusters running on twenty, thirty, a hundred nodes, whatever that is and people have access to vClusters. You know, you're you're really restricting the scope to what they need to see or they have the ability to see rather than just giving them a map of the world and going, here, here's all the things that we've got available to you. Yeah. It's actually quite interesting. I mean, we we already talked about, you know, mutating admission control earlier inside the host cluster.

42:15 And what you can essentially do in inside the host cluster, you can, you know, enforce certain notes for certain users with mutating admission control, you know, essentially just on a pod level for you don't need to avoid about a whole lot of other resources. You essentially just worry about the parts. And then, you know, you essentially, you know, rewrite these parts that the synchronizes down, and that way you can essentially restrict notes as well for you know, although you don't add anything to the vCluster just by adding admission control to your underlying cluster. Okay. I like this.

42:54 So what you were saying sorry, Andy. Go Rich. Oh, I was just gonna say I'm I'm with Noel. Noel mentioned in the chat that he's adding it to the log list of tools to check out. I I know that field. Yeah. The the cloud native landscape just seems to continue to grow at an absolute wild rate. There's just so many cool technologies coming out, solving a whole array of really cool problems. It's it's phenomenal to watch. Yeah. Yeah. One of the reasons that I was really interested in coming to Loft is that, like, I just know that people have felt

43:34 a lot of pain, you know, over over multitenancy and self-service and that these things are really big issues for people's productivity, you know, for the engineers and and for the platform teams and and people who are supporting them too. You know? And you used the word clever earlier to describe something about the design, and I I think that I think the tools are smart. You know? I think vCluster is a smart tool that solves real problems. Definitely. So something's crossed my mind though and I'm gonna throw it out there. I just ran get nodes on the host

44:09 Running Different Kubernetes Versions in vCluster

44:13 and we've seen the control plane and the two workers. Apparently my ability to remove a taint is well, as crap as everything else I've done so far, but the age on my host nodes is seventy minutes and fifty minutes. The age on the second worker node and the vCluster is probably from when we deployed engine x. So that's something that the KCS cluster is registering is actually registering the node with KCS when something gets scheduled there? Yeah. That's that's exactly what happens. You also see that the the the version shows k three s as well. Right? Well,

44:52 that was my next question. Like, can I deploy one two one as a vCluster? Can I drop down to one sixteen? Like, can I get really weird testing situations running across multiple Kubernetes versions regardless of what my host cluster is? Yes. You can. I think the only limitation that you have is the pod spec. Right? Because we're synchronizing your pods. So if there is a feature if you're using a vCluster of a higher version than your host cluster, and you're using a pod spec feature that is not available in your underlying cluster, and that is one of the fields that we're

45:27 syncing, so an essential field. Right? Then that may lead to a problem. But then you're just transparently gonna see it in that in the status. Right? So you can get around these these things. And typically, you know, pop and service, which are the core things that we're synchronizing, they're rather stable. You know, taking the Kubernetes API, not a whole lot of things are changing. You know? Whereas things like, you know, ingress or, obviously, I mean, CRDs change even more frequently. That's typically the things that that would break a lot more. But, yeah, people are doing

46:02 that. People run entirely different k three s versions on top of, you know, other Kubernetes versions, and that's actually one part where it's super interesting for CICD as well. Right? If you wanna test, like, 20 versions of Kubernetes at once. Right? You just spin up twenty week. That's just really quick. Yeah. One of the one of the use cases we didn't talk about is testing upgrades. You know? Yeah. Yeah. You could actually do that in a pretty safe manner, I guess, by rolling things out to a vCluster first, seeing what happens, and then going, alright. It

46:30 Deploying Different Kubernetes Versions as vcluster's

46:41 looks good or oh, crap. Oops. Things went wrong. And then, yeah, you got to stay in a nice, safe, and secure way of handling that. That's pretty cool use case as well. Yeah. I mean, again, if, you know, like Lucas was saying, you know, if you're if you're doing CICD against a specific version of Kubernetes, you know, you could do a scenario where, you know, you run the test against both for a little bit, you know, and make sure that that, you know, the newer one is working as well. Okay. But I'm curious about this version thing

47:12 Demo: Deploying Different vCluster Versions

47:12 there. Is this the flag I need? I just specify the KCS image, and that's the version of Kubernetes I get in my my virtual cluster? Yeah. That's pretty much what works. Yeah. You can also I mean, if you're looking at the documentation and you see, you know, the the cube CTL deployment option and the Helm deployment option, how these things actually get passed on. The CLI is basically just, you know, running Helm install or Helm upgrade on the hood essentially and passing those things to the to the helm chart. And the helm chart is basically just adding that to the there's a

47:47 extra arcs that you can pass, and they essentially get concatenated, you know, to the regular arcs that we pass in k three s. We don't need docs. I've got you too. Alright. I'm just I'm so curious to have this multi this version thing that I need to try it. So how old can I get? Let's see. No. KCS is now old. There we go. Let's try this. What are what's your host cluster? 120.four. So yeah, could I think I'll create a 116 and a 121 just for no other real reason that I think it's cool that

48:29 I can do that. Like that is this very exciting to me and I don't know why but go. I'll grab the 121, which is I think the latest is 1212. Yeah. There we go. Too much fun. That's what I'm saying. One thing I also wanna mention is you always change the namespace here. It's also possible to have multiple vCluster in the same namespace. That's technically possible as well. Typically, we recommend, you know, for easier visibility in what actually is happening under the hood to isolate them with the namespaces, certainly possible. I'm just sitting here popping up Kubernetes clusters

49:04 Nested vClusters and Multiple Instances

49:24 like nobody's business and having a rare time. So It's it's also possible to run a vCluster inside of a vCluster, which I actually did a YouTube video where we do that on the loft YouTube. It's pretty fun. What? So you've got a vCluster running on a host cluster. You create a vCluster and a vCluster and the sync just goes to the sinker and the sinker goes back to the host cluster. So it's just two layers of how many how many have you tried? How deep if you went, Rich? I've only done the the that second layer.

50:00 I mean, I'm gonna have to try it. So alright. So this is I'm gonna get so confused and forget where I am, but I'm I'm gonna this is me on VCluster 1, and now I'm gonna create another vCluster called VCluster 5. Okay. Looks like it may be working. Okay. Let's see what happens when I do a list. So this is inside of VCluster 1 and we have access to V Cluster 5. If anyone at home is actually managing to keep sense of this, you're much better than I am. I'm gonna unset my kube config which puts

50:40 me back on the host. I don't want to be in the host. I want to be on my remote cluster. Okay. I may need ten or twenty minutes to work this out. So vCluster less connect took me a while to work it out when I made the video. So this is really cool. I have four clusters on bare metal right now. I shouldn't do an all because there's too much running. Let's do this. Oh, man. We should have made it VM on the bare metal to run the vCluster's inside. The one sixteen failed, but we'll ignore that.

51:30 We got our engine x's. In fact, I think I just got all those images wrong. I've I've I've I think I've now got five broken clusters, don't I? Oh, I'm an idiot. You see when you when you know, you know. This is just a tag. Oh, yeah. You wanna use what Alright. Yeah. When we did the get nodes, right? I guess it should be rancher slash k t s and then the tag. Oh, okay. Yeah. Don't don't worry about it. Just run it again. It'll essentially run Helm upgrade. Oh, nice. Okay. So we can run The only thing

52:15 only thing to note is because they're stateful sets, the parts actually don't get recreated. You'll have to delete the parts, but then they get, you know, the new parts that will be scheduled. They will essentially have the new image. Okay. So let's see if we can upgrade and fix the broken image on Rawkode Live two, and we'll try and get a one twenty one cluster of this. Yeah. So there's a home upgrade, which means I should be able to run this. And you said I might just have to Yeah. You see that there's still three minutes

52:54 old or oh, no. That one actually got recreate. Oh, I think if the image is changing, yeah, then then restarts it. Yeah. It's running. Okay. Let me recreate my five clusters again in under one minute. One 16. Oh, that could suit. Okay. They added change the image for that. So let's do rancher case. Yes. And this can be cluster three. And I'm so confused. Now I wanna connect to vCluster and create another cluster for cluster inception. VCluster. Oh no, my connect's up here, right? Yeah. Cool. And this one will do five and yeah, so we'll just do that. Okay.

53:52 I have no idea how many v clusters I've got anymore, but we definitely have some virtual clusters. So we can do a v cluster list. We'll connect it to the current v cluster and we can run get pods all and this is a vCluster so we should only see apparently that's oh no, this will need me to delete the pod because Juggling too many things, but we'll get there. This should just not be this much fun. I mean, I have to say that's been my reaction the whole time since I started playing with this project is is

54:38 I just think it's super fun. And a lot of what we do, you know, as ops people and platform engineers and and, product engineers isn't necessarily fun. So so my theory is that you embrace the things you can do that are. Right? So if you get to play with a tool that, you know, is really fun that you enjoy, you know, run with that if you can. Couldn't agree more. Alright. Let's confirm our version of Spark and then I'll stop doing silly things, I promise. But we have a connect here and we're gonna connect to

55:16 vCluster two which I think is a one twenty one cluster. So I should hopefully still be on. There we go. We got a one twenty one cluster running on a one twenty cluster. That's pretty sweet. That's awesome. It is. I wanted to thank Noel too who said in the chat that he enjoyed that video that I was talking about on YouTube. I I appreciate that. Thanks. Awesome. And no one now suggesting that on custard, everyone's gonna be getting a nested vCluster. Potentially. Pretty nice way of doing it. I'm not I'm not sure if that would be

55:58 fair but Alright. Yeah. This this is really cool. That version thing was a nice little cherry on the top there as we went through this. I wasn't expecting really I mean, now that I've done it, I'm like, well yeah, of course we could do something like that but it just it doesn't occur to me and now that I know I can have multiple versions of Kubernetes API available, even just building operators and being able to run end to end test against all of these versions and do that super quickly, I guess, guess, that's really nice. I like that completely.

56:33 Awesome. There's one more topic that may be interesting, and that's actually networking. The reason, you know, why networking for in cluster stuff works out of the box is essentially because the pods are running actually on your host cluster. Right? So they use your regular, you know, network plug in that you're using for for your your host Kubernetes cluster. And then what we do essentially inside the vCluster is we just you know, because we're syncing the services, but we're also renaming them, again, to prevent naming conflicts because, again, we're mapping from five namespaces to a single host namespace. Right?

56:34 Networking and External Service Access

57:12 The only thing we need to do is adjust the cluster internal DNS and tell these parts to talk to the DNS server that is running inside our vCluster, which also creates part on the underlying cluster so you can basically also connect to it via regular IP. Sounds a little tricky, but, essentially, again, networking so in cluster networking, so connecting to any services inside your vClusterWorks. And you can also, you know, immediately talk to other parts via IP. And the really interesting part is actually regarding ingresses. Because, you know, frequent use case is that you you don't want every one

57:54 of your if you're in that use case that we originally come from. Right? You have, like, users in their isolated namespaces. Typically, they share an ingress controller. Right? Or they have, like, couple of them available in the cluster. If you're using vCluster, you don't want everyone to deploy their own NGINX and have to, you know, provision their own load balance servers. That's right. It's a little bit more tricky. So by default, vCluster's also synchronize ingresses, but that can also be, you know, turned off and on, etcetera. Okay. Yeah. That makes sense as well. Does that mean

58:36 if I say I bypass the DNS within the cluster, I just start using port IP addresses. That that network isn't going to work. Right? No. That network works. So, I mean, your part is so if you're inside the part, right, if you were to just run Kube CDL exec inside a part that you deployed via the vCluster, then you're technically not then you're technically inside a part of your host cluster. Right? And if you were to ping another part in your host cluster with the with that IP, that should work unless your network policy is forbidding you to do that.

59:21 Some interesting use cases there. I'm thinking like as me as operator of the host cluster, I can actually make some things available to vClusters using the host networking thing. Like I could say here's a http service that's gonna give you access to secrets or metadata or something and then they just know this magic IP address or whatever is is gonna work. I'm not sure what Yeah. A real world actually is there yet, but there's that there's something Yeah. No. It's it's funny that you mentioned it. That's actually one of the most requested features for vCluster is because

1:00:00 what you described right now is what we recommend as a workaround. Right? So we have the common use case that you have, like, a very bulky application or kind of like a test system or something that people need to connect to. Right? Just think of, like, you have a as a, you know, cluster admin, wanna, you know, make a Kafka cluster available for your engineers. Right? I think that is one of the use cases that one of the one of our customers, you know, was outlining, and they don't want everybody to deploy their own Kafka cluster because

1:00:33 they're pretty bulky. Right? They tend to break, and it it's very high maintenance. And they're already inherently multi tenant. Right? So they can essentially with different credentials, manage different users. So why not one central instance and make it available to your engineers? Right? And then the question is, how does your pod that you created in your vCluster now connect to that service? Because that service of that Kafka cluster does not exist in your vCluster. There's essentially two answers to that. One is if you have a IP which is not supposed to change frequently, right, because you created that service

1:01:14 for a Kafka cluster, then you can direct directly, of course, com communicate VIP. And the workaround would be to essentially just create that service inside the vCluster. So essentially, one of cube CTL apply to create that service to kind of have, like, a fake service inside your vCluster. That is something that works right now, but we're thinking about other ways, you know, when, obviously, operators wanna, you know, enable for the engineers is a more automated approach to this. Right? But they don't need to play around with things like that. So we're currently thinking about ways to synchronize

1:01:52 resources from the host cluster back into the vCluster essentially so that services start popping up that you have not created in the vCluster, but you want them, you know, to be synchronized from the host cluster into the vCluster. That would be a use case. Yeah. I was just thinking that. Like, if I could label my Kafka servers on a host cluster in some way that associates it with the vClusters and then the sync will create those services for me, that would be a a pretty pretty interesting setup as well. Yeah. So this is open source. It's on GitHub,

1:02:23 Contributing and Getting Involved

1:02:25 you know, GitHub.com/loft-sh/vCluster. And, you know, we have been getting some issues for people. People are kicking the tires a little bit, and it's really exciting. You just can't predict what people are gonna wanna do, you know, until you throw something out in the wild like this. Nice. Alright. Let's clean this up. I'm assuming this is gonna be really really simple. And I have no idea what you configure them on anymore, so let's work that out. Okay, that's good. So let's go back to this one. VCluster delete. Copy and paste, Dave. That's the future. Stop changing stuff.

1:02:51 Cleaning Up vClusters

1:03:24 Yeah. Since everything is contained inside those namespaces, so the vCluster itself, right, the stateful set and the service as well as the workloads that run inside the vCluster, it's actually really easy to purge a vCluster. You can just delete the namespace. Yeah. So you could literally, on your host namespace, delete or on your host delete those namespaces. Done. All gone. Yeah. That was much easier. I was going to go vCluster delete and work my way through them all. But, yeah, I can just remove that whole namespace. And that's that's the the great thing about vCluster is I only need the namespace and

1:04:05 I can clean it up with just the namespace. That's very cool. Alright. Is there anything else? I know I haven't really stuck to the documentation too much, but is there anything else that we wanna go through before we finish up for today? Maybe just a quick shout out to what Fabian has been working, our CTO at Loft Labs. He's been spending a lot of time on this. You see that they're operator guide monitoring and metrics. He's been spending a lot of time on, you know, actually making things like cube CTL top work. So, essentially, you know, having the

1:04:12 Monitoring and Metrics

1:04:42 capability to deploy metrics server inside your vCluster and then hook up monitoring via Prometheus and stuff like that. So that's really moving more from, you know, the depth stage towards staging environments, production setups, because they're very interesting use cases in that space as well, and we'd like to explore them further. But, obviously, we need to provide some more, you know, tooling for operators to gain confidence that they can actually, you know, run that, you know, in in staging and production scenarios, you know, because without monitoring and and metrics and things like that, that would not be

1:05:18 possible or not recommend that. Very cool. Yeah. Yeah. Hadn't even considered those kind of day one, day two things, but it's nice that those are being worked on. Yeah. The tool is very rapidly changing. Fabian's amazing, like we mentioned. Just wanna thank to, you know, the folks who have been, like, kicking the tires on this thing and and giving us feedback. It's so helpful. If if folks watching the stream, you know, play around with vCluster, we'd we'd love to hear about what you do. Nice. Alright. Well, that was a whole lot of fun. I'm I'm still a little bit shocked at

1:05:55 Conclusion and Final Remarks

1:06:01 the simplicity and how easy it was to really get everything started. You know, you said at the start using the vCluster CLI is just gonna make things a lot easier. Yeah. It was easy. Really, really easy. And the speed I wish you can spin up and shut down those virtual clusters is a sight to behold. So thank you both for for joining me and and guiding us to the session. That that was a lot of fun. I really enjoyed that. Do you Thank you, David. Do you have any final words or anything you would like to

1:06:31 share with the audience before we say goodbye? I I just reiterate what I said. You know, if you if you kick the tires, please please do let us know. I'm at rich Burrows on Twitter. At me. I'd love to hear about what you do with vCluster. Awesome. And definitely check out the kube cuddle podcast. Lots of fun there. Alright. I will put links to all of this in the show notes. Feel free to play with vCluster, reach out to Rich if you have some fun, let him know if anything breaks. I'm sure they'll be very happy to to help

1:07:12 you. As well based on the things we were doing today, it's probably gonna take quite a lot to break a vCluster. But like I said, I had a lot of fun. Thank you again for joining me and I hope you both have a wonderful day. Thanks again. You too. Thank you.