0:38 Introduction

0:38 Hello, and welcome back to the Rawkode Academy. We are continuing on with our complete guide to Teleport course today. Before we begin, I will caveat this with this is my first stream on my M1 Mac. As such, if anything seems weird, please let me know. I think my audio is okay. I think my video is okay. I think my scenes are okay. But feel free to jump into the chat and say, hey, something's not right, and I will do my best to fix it because we want this to sound and look as good as possible.

1:21 Alright. So what is on the agenda for today? Well, today, we are diving into another workshop where we take a look at running Teleport on top of Kubernetes. There's some weird things to discover when working through this, just that Helm charts are a little bit difficult to find. Some of them are deprecated, but it's not obvious that they're deprecated. And I'm gonna try and do my best to just kinda walk you through this from each step. If we pop open my screen share, we have the new workshop pushed to the Rawkode Academy slash courses repository on github.com.

2:03 You will find a directory called eight kubernetes with the README that we will be using for today's session. So feel free to work through this and let me know how you get on. Today's is a little bit different from the other ones I've done. Normally, I try and hide the answers. Today, I have to say we just kind of leave them all here, more of a follow along. Let me know which format you prefer and we'll tweak that for any future workshops. The prerequisites for today are obviously Kubernetes, easiest thing in the world to deploy.

2:29 Prerequisites and Workshop Materials

2:34 Everything we're doing today should should work on local Kubernetes setups, but you will have to bear in mind that Teleport really wants to be served over TLS. So, really it wants a domain name, and you may find you have to do a few extra steps to get this to work with Minikube or Kine or any other Rancher desktop, Docker desktop, etcetera. So, for today and to make my life a little bit easier, I am using managed Kubernetes clusters. I have two on Cboe and one on JKE. And I'll explain why I've made the decision with these two clusters, the three clusters,

3:14 as we work through the exercises for today's workshop. Okay. So we got we got people here. Hello, Moz. Hello, Russell. Thank you for joining us. If you missed the caveat at the start, let me know if my audio was okay. I'm on a new machine and I'm very worried because the audio is always the first thing to break. Okay. So the first thing you have to do when you want to deploy Teleport to Kubernetes is just to say what way you want to do it. Now, the Teleport team do provide some Helm charts that allow us to simplify this process.

3:47 Teleport Helm Chart Overview

3:58 So we're gonna run through those today. For me, the first thing I had to do was find the Helm charts. What I would suggest is go into the Teleport Teleport repository. They're not gravitational. Got that wrong already. Here we go. They're not in the most obvious place. Typically, projects in the cloud native space will have just a deploy or a chart directory at the top level. I mean, it's almost ubiquitous now across most Go projects at least. However, Teleports live under the examples repository. And that seems to be their examples of how to deploy Teleport rather than like their,

4:39 this is our production deployment, you can go and just use it as you wish. It feels like it's take it, maybe modify it, and make it suit your own needs. But we are gonna use them today. So an example of ratio, we have chart, and we have a whole bunch of things here. Now, most of these charts are deprecated, but there's not really any way to know that without looking into the actual chart dot YAML, where it will say deprecated true here. So while there are a selection of charts here, only two of them are presently

5:16 doing anything. I popped that out. We don't need to. So I've kind of broken it down in this readme of what these charts are for. So we have the Teleport cluster chart, and that's just sets up your standard single node Teleport cluster running on Kubernetes with a local authentication back end as opposed to OAuth, GitHub, SAML, etcetera, using a persistent volume claim for storage. And really it's the easiest way, right? You can just deploy this chart, update some DNS records, which we're about to do, and you get a working Teleport cluster. Now you do have to go onto the

5:57 cluster itself and manage your users, etcetera. But these are examples. So you can take that config map. You can swap out the local authentication and you can configure it with your GitHub application credentials if you're using open source. If you're using enterprise, hook that up to Google SAML, whatever you want, and go nuts. It seems to be that the Teleport chart, which was essentially the exact same as the Teleport cluster one, is deprecated. I assume this was just the first one that they started putting together and then decided to kind of deprecate in favor of the Teleport cluster. So

6:27 we won't be looking at Teleport. If it's deprecated, please don't be using that as an example. The next thing I found, and I actually got quite excited by this one, the Teleport daemon set. I was like, yay. We can run super privileged pods across all of our infrastructure and get us to switch access to them well, so I'll get an access to a Kubernetes cluster at the same time. This is also deprecated. However, I am going to be replicating it through my own dirty hack today. So you can't use the redeemer set chart, but it was good as an inspiration

6:59 for just a proof of concept that I wanted to build out together. Welcome to it up. Nice to see you. Okay. Let's keep working. So we've got the Teleport Kube agent. We are definitely using this one today. This allows you to if you have an existing Teleport cluster, possibly the best way to run Teleport is on bare metal and then expose all of your cloud stuff back to it. However, and the KubeAgent chart is the best way to get started with that because you can just deploy the Kubernetes proxy with that configuration to any Kubernetes cluster and then filter them all back

7:12 Teleport Cube Agent

7:40 to some other Teleport, and we will be looking at how to do that today as well. The next chart is also deprecated in the last chart, which is the Teleport Auto Trusted Cluster. This seems to be a chart which, again, is good as an example if you want to go and see how to implement this stuff. But Teleport works really well in a cluster of cluster mode, in that you have one centralised, pet like, super friendly cluster that you take care of, and then you have other Teleport clusters at the edge or on different regions, etcetera,

7:47 Teleport Auto Trustee Cluster

8:11 which can then use the root cluster for authentication, privileges, permissions, etcetera. We may actually do an episode on the kind of federated cluster of clusters approach towards the end of this course. So, tuned for that. I actually need to update this. You should not run a daemon set when it was deprecated. This is what I was going to try and encourage, but we're going look at doing this a different way. Alright. So, exercise one. We know where the Helm charts are. We know which ones are active, we know which ones are deprecated, and we have a good understanding, hopefully, of what

8:39 Deploying Teleport Cluster with Load Balancer

8:45 each of these Helm charts does. Now, the next step is we want to install Teleport to one of our Kubernetes clusters. So, I've included let me zoom in a bit. There we go. Can we go one more? Yeah. There we go. So, I've included a bit of a warning at the start of this task. The Teleport charts are not published through the Artifact Hub. If you're not familiar with the Artifact Hub, there we go, Artifact Hub. This is like the new Helm Hub or even a replacement or discovery mechanism from the Google Helm stable and Helm incubator,

9:27 etcetera, where you can just come on and fax it for something that's definitely not gonna be there. Like the reddest chart, and you can see that this is by the Falco team. We got one by Bitnami, the Bitnami chart is usually a pretty safe bet for everybody. But if you search for Teleport, there are charts here and they're all not official. They're contributed by other people, other teams, etcetera. So they are there. You may find them that way, but I would probably suggest that you use the ones provided by Teleport. And in order to do so, you can

10:05 just do a Helm repository or Ripple add Teleport with the URL. I've already done this. I don't need to do it again, but I can run Helm Ripple list, and you'll see that we have already added the Teleport repository to my local environment. You can also, if you want, just do like a shallow clone of the actual gravitational Teleport repository. When you're deploying and using Helm, if you don't want to include the chart name or repository slash chart name at the end, you can just use a local directory, especially if you do want to make modifications

10:42 to the example charts and to replace that local authentication with a GitHub authentication, etcetera. So, that's a really good way to do it. We probably won't do that today unless I'm getting particularly bold towards the end. We are just gonna use the example charts with a little bit of configuration. Cool. So, we're just going to copy this verbatim and deploy our first Teleport cluster using the Teleport cluster chart to my first Kubernetes cluster. I am using Kubei. Trying to at least. How'd you do it? Oh, there we go. I've got my three different clusters here. So,

11:28 I'm going to deploy to JKE first, and then I'll tell you why I'm deploying to JKE first. We'll make sure that I have JKE cluster. So, I'm looking for Yeah. Some of this stuff only really exists in a JKE cluster. Plus, what do we need? Oh, yeah. We're copying and pasting this command. Nope. I've now copied and pasted the wrong thing. Much better. So, we're going to call this helm release Teleport cluster. It doesn't really matter what name you give it. You just need to be able to know the name when you want to run helm

12:11 upgrade commands, to be able to tweak or modify the configuration. We're going to run with Acme on. So, one of the cool things about Teleport, and we covered this in the first installation video where we looked at running with Docker and with the compelled binaries, is that it has built in support to reach out to Let's Encrypt to get certificates. So in order to enable that, we just need to set Acme to true and then pass in our email address because less encrypt requires that all certificates have an email address with it so they know who's provisioning them, there are wait

12:44 limits, etcetera. And we give our cluster a name. This is going to be a fully qualified domain name. And for today, I am just using my Rawkode.sh domain, which is relatively unused to date. I'm telling this chart to create a namespace and that I want to keep all my Teleport stuff in a Teleport cluster namespace, followed by the chart name. With this, we can hit return. Wait just a moment. And it is deployed. So we can do Teleport cluster get pods. We'll see that's currently container creating, so we'll stick a watch on that and just

13:30 give that a moment. So this shouldn't take too long. It's just provisioning a persistent volume, pulling image and starting the container. We can see that it's running now and it's healthy. So, we're kind of almost done. Of course, I'm gonna show you it working. You don't need to take my word for this. But the reason we're using the JKE cluster first is because one of the things that the chart does by default I really should just set my namespace, we'll do it this way. One of the things the chart does by default is deploy a load balancer type

14:10 Deploy a Load Balancer Type Service

14:12 service. This is a Kubernetes service where the type is not cluster IP, it has load balancer. So, this does not work in all Kubernetes environments and the fact is very unlikely to work in Minikube, Docker for Mac kind because they won't have the ability to provision a load balancer. The service type load balancer is really a cloud construct with providers for AWS, GCP, DigitalOcean, Linode. They all have cluster controller managers that monitor for this service and actually reach out, get an Elastic IP, and then attach it to the service. So, I use JKE so you can see

14:49 that work. And the next step I need to do is to copy this IP address, jump over to my DNS, and add a record for the fully qualified domain name that I added to my cluster and save that. And we just need to give that a minute for that to propagate. And what should happen? Is we should be able that's my cloud instance, Teleport.Rawkode.sh. And we should be able to browse to this momentarily. There we go. Awesome. That's one of the reasons I use Cloudflare. The propagation of their two ENSE records is just ridiculously fast.

15:31 And we have Carlos with us today. Hey, man. How's it going? Nice to see you. Okay. So we have deployed with the Helm chart, Teleport cluster to JKE. It's got our load balancer. We updated our DNS record. Teleport can now satisfy the ACMI challenge, and we have a cluster. And we could tick our box in, yes, we've created our DNS record and it worked. However, you may be running a local environment or non cloud environments, bare metal environments, where there's no load balancer service, and I wanna run you through how this works too. Well, I do provide another command here,

15:57 Deploying Teleport Cluster (ClusterIP with Ingress)

16:12 and we will switch my Kubernetes context to Teleport one. This is a Siebel KCS cluster. And it responds. And it does. Awesome. So, I mean, we could do the same helmets document I just run. However, what we're gonna see would be the service Comment from Russell saying my audio is a bit muffled. Maybe I am just muffled. I'm not sure. I'll try and then see a little bit better. But thanks for the heads up. Okay, so we could deploy the We could run the first command again to this cluster. However, the service load balancer would set in

17:02 a pending state indefinitely. Civo Cloud, along with many other clouds, just don't provide load balancers as a service. They're also expensive. If you run a Elastic Load Balancer on AWS, I think each one of them is 20 to $25 a month. You can go down the road of using the application load balancers, which are a lot cheaper. Carlos thinks I'm getting sick. Maybe I am. If it's just that my voice sounds a bit nasally, perhaps I am getting a cold. I mean, I have a three year old who's been at nursery for just over, well just under three months and

17:40 she's pretty much been home with new snot and sniffles and coughs every week. So, I guess it's inevitable that I'm going to catch something. Okay. Alright. Stay track. So, yes. ELBs are expensive. ELBs are cheaper and require more setup, but you can just not use a service of table advancer at all, which is what I'm gonna show you guys. So, we'll paste in this command and you'll see that it's this Talm install Teleport cluster. So it's the exact same ACME true. We're still passing in my email address. I am gonna change the name on this

18:19 just so I can have multiple DNS records, and we'll call this civo1. Rockcode. Sh. And now I'm using the helm. Thanks, Carlos. And now we're passing in a set service dot type equals cluster IP. So now we're telling this Helm chart, don't deploy a load balancer, use cluster IP. I work out how to get traffic to this in my cluster. And here, I decided don't know why I decided to try and do this with the local chart, but I'm gonna just change that and stick to what's worked so far, which is Teleport cluster. So almost identically the same command, only we

18:59 are overriding the service type. This will deploy to Cboe and we'll look at how to expose this. So we've got the same namespace, But we can just keep that watch on. I don't like it when I see pending. I was very, very worried there that I had deployed my SIBO cluster without a CSI. I wasn't even sure if that was a thing, but we're okay. So, the pending was provisioned in persistent volume and the claim. Container creating now means it's pulling the image, and hopefully very quickly, we're going to see that this is running. And I can panic

19:54 a little bit less. Guess we just wait. Wait. Wait. Wait. So, as I say here, you will need to create some sort of ingress resource. So, this Civo cluster does get one public IP address, which is load balanced by default. Because it's a workshop, So Carlos is saying, why am I using Helm directly and not doing getups? I would love to be doing getups, but, you know, this I I just wanted to be able to run through the commands in an interactive fashion, like as if you were building a proof of concept, and then the getups can come later when you know

20:49 that it works. However, a great question. Say that again. Yes. So, the Civo cluster comes with one IP address with a load balancer. It deploys with traffic by default as ingress. I'm a fan of emissaries, so I just swapped out. What that means is, and we can see that our Teleport is happy, which is good. If we do get svc dash all, we have our ambassador, which is Emissary NGRIS, as a load balancer, which gets that one public IP address that is available. That can route all other traffic via the NGRIS or mapping objects as Emissary calls them

21:34 to the service inside of the cluster. You can see we actually have our cluster IP service here for Teleport. So, in order to make all of this work, we need to first create the DNS record for C4-one, I believe I called it. Point it to the load balancer and hit save, and then create an emissary mapping. Now this could be, if you're using traffic, use whatever traffic custom resource they have. If you're using Kong or you're using NGINX Ingress, whatever you're using, you can just go to that. You may even just be using networking V1

22:15 Ingress. That's fine too. So, I'm going to create my mapping. YAML. My host name is SQL one. No prefix. And I believe the service is called Teleport cluster. I'll quickly double check that. Yeah. Teleport cluster, Teleport cluster. And I can now hopefully just apply this to the same namespace. I call it mapping. Alright. We're debugging. So what happened there is my cluster is telling me that kind mapping doesn't exist, which I was not expecting. So, we can use API resources. And I wanna see what we get in the ambassador namespace. Ah, this version of ambassador

23:28 is old Because nothing ever goes simply. Alright. Let's upgrade it. I mean, could just do like a map in v2. But I I don't know what's the best approach. I'm gonna I'm gonna upgrade it. So we got ambassador here. Edit deployment ambassador. And we can see We have ambassador Edge Stack one thirteen ten. Updating that seems like really risky, doesn't it? Maybe I should just use the old mapping. Drop the version there. Oh, wow. It's really old. The problem is when I was using the Civo create cluster, they have like a marketplace, which clearly they don't update their applications because

25:06 this is, there's a substantial difference here. However, I can still go to these versions of the docs, which means can hopefully Yeah, this looks almost the exact same. So I'm just going to We're just gonna hope for the best get ambassador. I'll be two. In fact, that leads to alpha. Maybe this is better. And Teleport cluster apply dash f mapping. Alright. Port cluster. I didn't know that was a thing on Twitch, Russell. Russell suggests that if this was on Twitch, we could bet channel points. I'm not sure what channel points are or whether I break

26:07 my cluster. Maybe next time. Okay. Let's see my services and my mapping. This this this may work. I have I you know what? I have no idea. Let's find out. Civo one Rawkode SH. Well, first challenge. The certificate is the ambassador Edge Stack self signed one, which means that Teleport hasn't been able to satisfy its Carlos is asking, why didn't I use traffic from Civo? I could've. I just I like emissary ingress from the times I've used it. I don't think it would be too much of a hassle to swap it out. I am very much regretting

27:10 that decision. But let's just see what happens. So we're having emissary. So the DNS record is going to my ingress controller, which is correct. However, the certificate that we're getting back is incorrect. So we can do a cboe rockode dot sh and reset dash dash b cboe one. Alright. And then if we want to debug the certificate, I think we can add. I can't remember. Capital a. So I'm gonna try one thing before I move on to the next step. I mean, all I was gonna show you is that you don't need a load balancer. I've kinda proven

28:00 that, but I haven't got TLS working, so it's not ideal. But close enough. Right? I'm gonna look at the logs to see why our ACME challenge hasn't kicked in. Right. So management search is in ACME. Let's try just restarting that. When in doubt, a reboot fixes everything. Well, let's delete our pod. We'll force stack the challenge to come through. Now that the DNS is there, we'll see what happens. Feeling that and moving on. It's not a big deal. Okay. Follow log. Patience is a virtue. I've landed on another node and it's still an image, isn't it? There we go.

29:13 Okay. Must be quite a big image. Bonus though, because this is my first stream using the m one, it's also the first stream that when I'm not talking, there's actual silence in there. Like, I cannot hear the fans from the machine. In fact, they're not even on. Wow. Slow. There we go. Okay. Logs. So our TLS server is x two. Is that old or new though? Oh, it must be current. It's the M1 Pro top of the line, not the Max, but the best Pro option. I couldn't justify the Max, Russell. But I could have, but I would have

30:31 been bullshit to be honest. Oh, well. Alright. So here's some advice. Obviously, try things before the stream. The second, now that I think about it, the fact that we've been done the Emissary route, which also has support for Let's Encrypt, I should probably disable the ACME support and Teleport and let Emissary deal with it and that would just fix this. I think what's happening right now is that Teleport is trying to answer the request, but it's not really getting to it. So, we're just gonna skip this stage and say, voila, it'll work. Use whatever ingress controller you've got to maybe

31:16 handle the TLS certificate instead of Teleport's ACME. I will play with this another day and get it working and do a small fifteen minute tutorial. Awesome. Okay. What's next? It's annoying though. Of course. Her heart is the Emissary TLS. That feels like a rabbit hole. I don't really need to go down. Yeah. Doesn't matter. Doesn't matter. All know what cert manager is. We all know how ingress things work. I don't need to fix this right now. Okay. So let's close this one and close this one and pop back over here. So, the important thing was just you don't

32:12 need to have the load balancer, but just the cluster IP and then handle the ingress to your cluster, however you handle it already. I mean, you're already gonna have services that are getting less encrypted certificates somehow and getting routed via ingress somehow. So, just use whatever you know and maybe I'll do the same next time. So, the next challenge was, if I to pop open Teleport. So this is my JKE one, which I never created to use before. Let's do that quickly so that we can take a look at something. So let's go back to our JKE cluster.

32:33 Exposing Host SSH Access (DaemonSet)

33:04 Get pods. I'm gonna actually just exec into this for a moment and create the our first user. So we can do TTL users add roles access, editor, order, login, route, and the username, of course. Login. And then accept my own invitation. We can close this. We see getting started. Save the password. Test Teleport Rawkode. We'll copy the password. Yeah. I just have I how is that gonna does it really scan that already for me? That would be impressive. Normally, I have to do that manually. Let's see. No. Yeah. I don't know where that came from. Let's scan it properly.

34:09 Let me use this one. Okay. So, that first Teleport install that we did, you see that there's no server access. There's no applications of course, but we do have access to the Kubernetes cluster. So, I could run through these commands to do that. But we probably, depending on your Kubernetes setup, is that you probably still want that server access or at least the ability to SSH onto your node, your nodes within your node pools, node groups, etcetera, for any sort of debugging or manual commands, of course, that you want to run. Of course, we want to be in a

34:51 world where we never need to be able to do that, but we're not quite there yet. And Teleport did at one point support the daemon set chart, which of course is gonna run some extremely privileged pods inside of your cluster to expose that host through Teleport. I couldn't get the daemon set chart to work. And plus, I didn't really like what the daemon set chart was doing. Now, understand why it was doing it, but it was just dreadful. It was actually running a privileged pod, installing Teleport onto the nodes, configuring a systemd service, and then starting the systemd service.

35:27 At which point, the daemon set chart itself becomes completely irrelevant. So, it was just like a really hacky way to get Teleport running. So, I decided, well, let's just try and do this as Kubernetes as we can by using a daemon set that just runs the SSH proxy and see how far we can get. So we are doing things like hostpads and I'll talk about some of those annoying hacks, but for the most part it kind of works. So, the first thing we need to do, and I haven't done this on the JKE one, so I'm gonna get myself into trouble

36:00 now, but we don't have a working Cboe cluster. So, what can I do? You'll see there is a lovely warning here. Teleport definitely do not endorse this approach, but it's a good experiment, right, to see how things work. So, the first thing we need to do is modify the config map that the Teleport cluster chart deployed. The reason why is that we need to be able to provide some static authentication so that when we deploy our daemon set, is that it can speak to the Teleport control plane. The way that we do that is just

36:32 by adding a token. So let's do it out of our container, Teleport cluster, get pods, and get config maps. So our Helm chart deployed the Teleport cluster config map. We can go down to the off service here. You can see the domain name. You can see the authentication is type local. We're gonna add tokens. Now you can add tokens for a variety of purposes in Teleport. You you could add coop tokens, you could add an app token, you could add a DB token, depending on what you want to expose. We want to expose nodes. So this is

37:10 a token that anyone, anyone with that token can join your Teleport cluster as a node, which means exposing server access. You can set the string to anything you want. I am going to go with random. And I'll just quote this because I have no idea if that space would be included. I actually think it would be. And I'm not even sure that this is all this is this isn't a re this is one string. I'm just I'm yeah. I'm gonna do this. So, now this is a node token with the string random, not space random.

37:49 And because this is Kubernetes, that means that the pod, even though it is mounting this config map, has no idea that really it's changed, and I don't think Teleport is monitoring it using I'm notifier or anything like that. So we're going to delete the pod to force Teleport to detect that change and restart its server. Okay. So, if we run get pods, we are running, not quite healthy, but we are running. So, we can move on to the next step. So, I talk about deleting the pod here. You can use a label selector if you

38:27 want. We're going to deploy a new config map. So, this is to provide a Teleport config where the off service is disabled, the proxy service is disabled, but we do enable the SSH service. We're just adding some labels so that we know this is a Kubernetes worker node, and then we configure it to speak back to our auth server, which is on Teleport.Rawkode.sh, with the auth token that we created. And then we have a daemon set, which is host PID, host IPC, and host network. So we want this pod to feel like it's the host

39:02 from the Teleport Connect SSH. We're not I'm not doing anything with a fail system. If you really need to be able to do that, of course you can go ahead and do that. But really I just wanted to do some simple process exploration to see what's running on the machine and look at stats, CPU utilization, etcetera. So, you can extend this further. Maybe I'll update that with some more hacky mount stuff. For these kind of privileged pods, what we would do is mount the entire host file system onto slash host. You could definitely do that here and then you could

39:37 share route if you needed to. Of course, I don't think you should have to be able to do that. That doesn't work to what I'm doing, but still. Let's copy this and we'll call this ds. Yaml, Drop this in. Where's my token? And this is normal YAML, so the space doesn't matter. And I actually think I've done this in a way that I don't need to modify anything. We can now apply dash f d s. If we run get pods, this is just a one node JKE cluster because I'm cheap. But you can see that we now have

40:22 our pod running. Now we can take a look at the logs. What we expect to see is that it is attempting to speak to our Teleport cluster, and the join token should be accepted. There are no logs because I've just realised I'm telling it to go to a file. No big deal. If we hit refresh on this page, I'm hoping, there we go, that this now shows up. So, we've now enabled server access with DaemonSet running in an extremely privileged fashion. Carlos asking a question. That's the minimum, host pad needed. So, you can actually run this daemon set without

41:15 host pad, without host IPC, without host networking, and it will still work. But you're gonna get server access to the pod. I'm just trying to make that slightly more useful for its intended purpose, which is host access. Going so far, it's just not to mount the host file system. What that means is if I click root here, I can see everything that's running on this machine. Like I said, you could easily mount it to our host file system to slash host, But it's just not needed. We can run top, we can see what's going on in

41:52 this machine. Really, I'm just exposing enough that I feel I can debug something that's wrong with this machine. In fact, with the permissions I have now, I could from here mount the host file system. We have complete access to the blocked devices. So, you can do anything you want. I think that would work. Dev SD one. Yeah. There we go. But we don't wanna do that. So, there's no let me recap. There's no supported way from Teleport yet to really run-in this configuration. The idea, I'm going to speak for Teleport, what I think they're going for here. They're a very

42:43 security conscious company. I don't think they would encourage running privileged pods in this fashion. You'd probably, as you configure your nodes, would run the Teleport agent from there and keep it out of the Kubernetes system altogether, only running the kube proxy inside of the cluster. I think that's the ideal way. You can do it this way though. I've ran this as an example. Use it as you wish. Okay. Yeah, Mollis, you're 100% right. The security people would just not be happy with running the pods with these privileges. Really, as an example, you can do it

43:22 and only you will know what you need to expose from server access at this level. So just give yourself enough to be able to get SREs or ops people to do the jobs. They probably have SSH access to the servers anyway, but I don't run SSH in production. So I would probably give them best route because it's just all audited. Although I'd probably run Teleport through node provisioning rather than Kubernetes with a daemon set. But, you know, we don't always get to pick and choose. And Carlos, yeah, got it. It's just so that when you SSH in, you have visibility

43:54 to other bits here. That's the only reason I gave it up, is so you can run it like debugging that 101, right? It's to run PS and see what's going on with the system and then probably followed by talk to see what's using resources. So, I just wanted to enable that use case and that was all. Okay, so we're gonna take a look at our last step now, which is we have this other Kubernetes cluster Pop down here. Oh, I need to be in the same folder because that's where I stored my context. So this shell is now going

44:15 Connecting Another Kubernetes Cluster (KubeAgent)

44:30 to be Teleport three, which is another SQL cluster. We still have our JKE one on the top. There we go. So, can we run the other Helm chart that Teleport does support, the KubeAgent one, to give access to this cluster from anyone authenticated against this cluster. So, the first thing we need to do is create a token. So this is very similar to the static one that we just created, only this is the slightly nicer way of doing it. I can do this from t c t no. Because I'm not authenticated. I'm not authenticated against this cluster on the command

44:56 Create a Token

45:17 with TSH login. Although I could be, right? Yeah. Okay. Let's do it. TSH login proxy, and this is not my Teleport Cloud. User Rawkode. So, there I can get CLI access to this Teleport cluster. Provided I have this, my password and my OTP, which is the bottom one. Okay. So now we should be able to do TSHQLS, certify cluster. We should be able to do server, SSHLS. Something wrong with that tunnel. I'm just forgetting how Teleport works. Yeah. SSH is actually SSH. How do you list the servers again? Nodes LS. Okay. So TCTL. There we go. So, I now have command

46:31 line access to my JKE Teleport, which is where we wanna be. I'm gonna create this token, which is the type cube. So remember when we added one to the actual configuration file in the config map, we added a static token. This is a dynamic token, which will expire in forty eight hours. And it's basically one. There we go. So we can now use this token for the next two thousand eight hundred and eighty minutes to register a Kubernetes cluster with this cluster. Got it. Let's copy the fact. Let's get the command first. So this is the Helm

47:19 chart that we're using now. So, we're sending it to I'm getting myself confused. That means you must be confused as well, but I'll I'm gonna slow down. So, on our Civo cluster, we want to deploy a new Helm chart called Teleport Kube Agent. We're gonna create a namespace that's gonna be called Teleport. This time we're setting the roles to be Kubernetes. This deployment is only going to expose our be a Kube proxy. We could also do DB and app. So, if there were databases and applications that we wanted to expose, we could do them in this fashion. But right now,

47:53 we're just focused on Kube. Our proxy address is not gonna be my Teleport cloud account. It's gonna be our JKE cluster. So, we're gonna use teleport.rockwood.sh on port four forty three. We then need to provide our auth token, which we have access to here. We just created this. Delete the dot. And we give this cluster a name, which is CVo3, and then we're specifically deploying the Teleport KubeAgent chart. Cool. Alright. So, now we can take a look at the Teleport namespace. We can see the pod is pulling. We're gonna have to wait for that image one more time on a fresh

48:32 cluster, unfortunately. But if we wait ten to twenty seconds, what we are going to see is that this proxy registers with our primary cluster, which is available here. And in fact, it's already there. There we go. So now, even though I'm authenticated against my Teleport JKE, I can use that to query any other Kubernetes cluster within my infrastructure. I don't even think that was running yet, but it is. So, we can pop over here. And what I'm gonna do is, you know, if I run kubi, kubi ctx. You know, I do have access to all

49:12 all these clusters just because of the directory that I'm in. I'm gonna go to the temp directory. And if I run kubi ctx, we don't have anything. We have no access because we're not in the right directory anymore. In fact, I don't think I can even run. Although that may still be configured for the JKE one. But that's okay. It's the JKE one. We're gonna try and preview the other one. Maybe I should just unset all of this. Was that too bold? Where? What? Where are those two times coming from? Okay. That should definitely feel alright. Yeah.

50:31 Okay. So now we have access to no Kubernetes cluster. So we're gonna click connect on this. We don't need to log in. We've already logged on to our Teleport cluster. We do a TSH kube login to c v three. And kubectl get notes. This worked last time. Yeah, Russell. It was. It was my QConfig was pointing to that local file, which is why I was so confused because I had dropped out my QBContext. Qb is a command I just discovered today. Really cool though. Okay. Ten minutes to debug. Why that authentication failed? Do we wanna try that or no?

51:27 During my tests, this just worked. Right? So when you do a TSH kube login, what that does is pull down some Kubernetes. Okay. Let's do config view. Yeah. When you do the TSH kube login, it configures you a kube config. It sets up the certificates, points it to the cross points it to the cluster. And then you've got some exec helpers where it's using TSH just to validate your config and stuff like that, your your credentials. And then normally when I do get pods, it works. However, this is telling me that the user cannot request Kubernetes access,

52:19 has no assigned groups or users. Now what that means is what this means is that my user here doesn't have permissions to be part of that Kubernetes cluster. So I could come in and take a look at the rules here. And we can see, and that should be okay. I actually don't have access to the Kubernetes cluster. This must be a Teleport bug. Alright, I'm gonna try and fix it. I will reach out to the Teleport team and see what happens. However, what's important from this final update the access rule for Kubernetes. Yeah, nice one, Miles.

53:20 Carlos saying Rawkode is missing. Moz is just telling me to go all smashing on it and give myself everything. So, this is filling at the Teleport level, not the Kubernetes level. In fact, we can prove that. We can connect to my GKE cluster. Run get pods here. No. We wanna speak to the We wanna speak to the SIBO where the the proxy is. I'm only gonna spend a minute on this. So let's export. Where's my SQL three? Teleport three. Teleport get pods. Log slash s. There we go. So, this is showing us, this is the Kubernetes

54:34 proxy on CvO3, which is actually telling us that a client connected. So this is, this was my request coming through. If we do Is that gonna do it? No. I guess because I already had the session. It'll only log in and log out. It looks like I am speaking to the proxy and that's going okay, but then Teleport itself is saying that I don't have the permissions to do what I wanna do when we run get pods. This user cannot request Kubernetes access. So my Teleport permissions don't really allow me to do what I wanna do, and it's telling me

55:16 that because it has no assigned groups or users. I haven't modified this before. If we modify the access one here maybe Moz is right. Maybe we could do this, which would get me root in a cluster. So on a Kubernetes cluster, this is my JKE one. We can run Kubernetes get cluster roles. And there's admin. It's not system master though, is it? Grap. Yeah. System. I guess I could just do admin. Basic user may be healthy. Let's try admin. My service accounts. I feel like I'm going a bit rogue here. Grap system. I don't know what service account we'd wanna

57:12 try and be. Let's just try the group. I'm gonna save that. Now the next thing I need to do is TSH log out. Log in because you need to log out and log back in to refresh your permissions. And this is going to be TeleportRawkode.s h user Rawkode. I need port. Password and token. Kube login. We got a little bit further there. So now I'm getting user Rawkode cannot less resources pods in API group. So that horrible little hack half worked. Like, I now have convinced Teleport that I have access to this cluster. We're now querying

58:24 the Kubernetes API. However, have now broken the authentication and let things in the user Rawkode rather than something else. Now, I mean, I could just go down the road here of creating a service account called Rawkode and giving it whatever cluster pending I want, but I'm not gonna do that because I am out of time and I'm just going down a really horrible path of showing people things that we shouldn't be doing. So what I'm gonna do is I'll reach out to Teleport, I'll work out why this doesn't work out of the box, but it has

58:54 previously with Teleport seven. So I think this is a change in Teleport eight and we'll try and get that bug report logged and resolved. Phew. That's quite a lot for one hour. So, yeah, a few comments. Carlos is asking, am I impersonate in the service account being used by Teleport? Kind of. Your Teleport user is getting mapped to a service account with some rule bindings in the cluster. You get to configure all of that. By default, it just can't work. I'm not really sure what's happened there. Russell is asking, if I only have to log out if you add new rules and

59:04 Conclusion and Wrap-up

59:33 not change prefs. You're right. I didn't have to log out. Good catch. In fact, is right there behind you. No need to log out. Okay. As Carlos says, thank you. On how to hack yourself. Yeah, there was some terrible advice in today's workshop with regards to running privileged daemon sets and modifying permissions forever. Yeah, Moj, I could create a service code called Rawkode and give it cluster admin in the cluster. But that shouldn't be the approach of what people take home from this workshop. So, will log the bug. We will get it fixed. We'll do every follow-up in a week

1:00:16 or two whenever it's ready. But thank you for joining me. Had a lot of fun running through this. It's on GitHub. Feel free to do it yourself. If you have any issues then you can just open them on the repository and I'll do my best to get them fixed. Nice little laughter there. Yeah. Carlos is saying, don't forget to join the Kubernetes happy hour this week. Carlos and I are hosting the Kubernetes office hours tomorrow. That is at two p. M. GMT. So, come and join us for that. And Russell, you're not getting admin. Although maybe I

1:00:50 will federate some Teleport access. It'd be a bit of fun one day. Alright. I'm logging off for today. Thanks for joining me. Have a good one, and I'll see you all soon. Bye.