0:00 Holding screen

0:27 Hello, and welcome to today's episode of Rawkode live. I'm your host, Rawkode. Before we begin, I've just got a little bit of housekeeping to sort out, and then we'll move on with today's session. Today's session is a Kubernetes security lab slash workshop. Now I would please encourage you to click subscribe on YouTube and click the bell. This will get you notifications for all new episodes. As I explore the cloud native landscape, you get to do it with me, and we can all learn together. We also have a rather active Discord server available at Rawkode.chat. If you're not watching live, that is the

0:42 Housekeeping & Sponsor

1:00 best place to come and ask questions, suggest new episodes, and get involved with the community. Please feel free to jump in there, say hello, and I'll speak to you soon. And lastly, I wanna thank Equinix Medal. They are my employer, and they allow me the resources, the time, and the energy to invest into this show producing this content. So thank you very much. Thank you very much, Equinix Medal. You could check out Equinix Medal with a $50 code. Use Rawkode dash live. $50 is roughly around one hundred hours of compute. Use it as wisely as you wish.

1:31 Introducing the Guest, Rory McKeon

1:31 Okay. Now I know nothing about Kubernetes security, but but fortunately, I'm joined by someone who does. Rory McKeon, a cloud oh, I forgot. A cloud developer advocate and security at Aqua Security. Is that close? Yeah. Cool. Yeah. Yeah. That's very good. We we went for cloud native security advocate. We were trying to work out what we're gonna call this that I do, and that's what we've come up with. Cloud native security advocate. Awesome. So why don't we just quickly start with you telling us a little bit about you, and then we'll talk about what we plan

2:00 What is the Security Lab?

2:02 to cover today. Sure. Yeah. So I said I'm an Equinix, a cloud native security advocate. So my role is basically trying to educate and inform around around cloud native security, Kubernetes security, containers. That's why I'm really happy to be doing this today. I've been doing container security now for about five years. Started with my previous role, and I've been doing a number of things around that, working with CIS benchmarks, which if you do not come across, well worth looking at. But the other thing I did was I did a training course, and that's where the lab that we're looking at today came

2:34 from. As part of developing the training course, a common question from people on it was, how can I practice this after the course? Now I thought, that's a good question. And then thanks to the wonders of kind, which we'll see, and Ansible, we worked out actually, it's not too difficult. We can give you a little lab that you can work on anytime you like. I actually practice some, like, attack techniques in a safe place, right, where you're not potentially attacking someone's thing that you don't own. So and that's where this came from. Awesome. Well, I must say you have your work

3:04 Environment Setup with Kind and Docker

3:06 cut out for you today teaching me security stuff. No easy task. But I'm looking forward to playing with this lab and hopefully not skating myself too much, but, you know, it might be par for the course I think with this. Okay. I guess we should just share my screen. We'll give everyone a link to the repository. We're gonna be using Docker for Mac for this today. So hopefully that is something that you can all do at home as you follow along. So how how old is this lab? How long have you been working on it? Two

3:40 year I think it's about two years. So we started doing the training course. The first time we did it for a conference was Black Hat twenty eighteen EU. And, it was like the a couple of runs after that, people started saying, hey. This is great, but I want to practice at home, which I can which you understand because you do these training courses, and it's a lot of information in a short space of time. So, absolutely, people want to take it away. And I was at the time, was playing a bit with with Ansible and then also

4:03 getting into kind. So if if people on the have not used kind already, use kind is the one that you should take from today. Kind is awesome. If you don't have anything with security, take that because kind makes this work. Yeah. I guess we could quickly cover what kind does, and then people will also see an action as we start to spin up our first kind of Kubernetes clusters. But kind, I believe, stands for Kubernetes and Docker and allows you to have a fully functioning multi node, which I think is really cool, Kubernetes cluster running purely in containers.

4:32 I think it makes really good for this kind of thing because you want to, you know, you want to throw away container. Right? You don't want to you don't want these things hanging around because they're deliberately insecure. You don't want them on the Internet because they're deliberately insecure. Having them in a nice isolated kind environment, I think, really helps. Yeah. I couldn't agree more. So what's the format for this? You know? Do we just need to spin up our first cluster? Is it gonna get me through I know I should probably read the documentation, but I'm always particularly impatient. So so right.

4:51 Getting Started with the Lab Format

5:01 So, essentially, we've got a couple of things to start with. You scroll yeah. You see the getting started there. So, essentially, what there is is there's a a kind of client machine container. There's a little container that what you can use as kind of the client when you're actually attacking the clusters. And then what we've got is we've got a series of different clusters, each one with a different vulnerability. Right? So it's gonna be each one has got something wrong with it, the name the clue is generally in the name, and we can kinda go through and attack

5:20 Overview of Vulnerable Cluster Scenarios

5:28 each one. But you can do some of the stuff straight from, like, the Mac, but it's easier if you kinda go into this client container because it's got the tools installed, which is kind of gets rid of why I don't have a tool to do that with. So yep. I was just gonna say, I I get is this teaching, like, best practices in defensive security by giving people a kind of attackers' playground? Is that the is that the the goal? Yeah. The idea I think is to an extent, I think people some of it's quite obvious.

5:56 Right? Some of it's gonna be you'll see something, okay, that you would never do that. Other ones, it can kind of inform you of, like, oh, I didn't realize that, for example, how easy it was to go from create pods to I control the whole cluster. So if you got a a developers who've got create pod access, we can say, hey. This is how easy it is to go from there to they could own the whole cluster anytime they wanted to. And I think that kind of, you know, that's all not always understood. Yeah. Okay. That

6:20 Launching the Client Machine

6:23 makes sense. Alright. So the first thing I need to do is it to run the client machine? Yeah. So that that that should bring up the client machine. So that's This is our acid test for is is everything working in the prereqs? Should be. Oh, we're gonna find out. Okay. Let's see what happens. Let's find out. That's fine. Actually, yeah, I set this up because I was having problems with my CI. Oh, application warning. Oh, if you do just a straight create cluster, I know what that is. That is and this is the first thing that that that

6:26 Setting up the Client Machine

7:02 I forgot. When kind first comes up, it creates a Docker network for kind. Ah, right. And so if you've never run kind on the machine you're using so usually, of course, I've run kind for something else. So I don't hit that, but that will be I'm trying to create something. That's that's I should take notes or something. Bugs that come up when someone else tries this because I know what I know I know what goes wrong when I try it. I think what threw me off there was just that really noisy deprecation warning for the

7:31 one really sensible error at the end. The one actually useful error at the bottom but it's like because I was like, oh, shit. We're not gonna be able to fix all that. I don't it's just that one little thing here which makes a lot of sense. That's just something Ansible that changed down the line, actually, probably. That's on my list of oh, that's gonna break at some point, but not yet. Do we have to wait on this kind cluster? Do you think it's already Probably will I don't think it'll create the network straight off. It shouldn't take super long. We got

7:59 a preparing note. But it's got there. I'll put this later finish, I guess. What's the And you you've got you've got the Mac terminals. You get the nice pretty graphics. That's, like, kind of They're not standard, though? On some terminals on a lot of terminals, you don't get them. Like, some of the emojis don't exist. So you get, well, just little gray boxes or whatever. Yeah. Yeah. I guess you gotta have, like, a font with the glyphs being done or power line symbols or something like that. That's it. You've gotta have some kind of and

8:29 I I know some of the it's obviously the Mac that has it, but but some of the terminals I've used, you have to go and find a specific font and install it to get the the shiny graphics. I run the Cascadia font Cascadia code font from Microsoft, which ships with the power limbo power light symbols baked in really cool. Love that. I mean, I I I think I spend more time tweaking my terminal, my color theme, and my fonts than I actually do any real work. Mhmm. And I think it's a slippery slope. I don't know if other developers are like

8:59 that or if that's just something that I waste my time doing. But I've I know what I do. I I I spent a whole time creating like a whole Ansible playbook to set up my terminal to do all sorts of stuff. At least I went to the point of well, I I did the bike sharing going and actually automating it. So let's try that now. Hopefully, that should now be happier. Yeah. I'm sure it will be. I've got nothing but confidence in the sounds of all people. Yeah. I've got yeah. So it's gonna complain. Here's where it complains about the deprecation.

9:30 And it's going and pulling the image in the background. Think this is So that's client machine, and then we have to let's just carry on with the instructions, I guess. Yeah. So So what we do is once we go on that, if you have to scroll down, we can look at our vulnerable cluster list. So these are the ones this kind of thing where you can implement lots of these. These are essentially all the different ones, and and this this these these examples were built essentially from the course. These are things where we cover this on the

9:47 Selecting the First Scenario

10:03 course, so I thought, right. Time to write a playbook. And you as you can said, you can see from the names that that each one of them's got kind of a descriptive name. So, for example, we got unauthenticated a p unauthenticated API server, then near the bottom, which is an API server that has anonymous access possible. And we've got unauthenticated dashboard. So you'll know that, like, people like Tesla, some years ago, in the early days of Kubernetes, they were using the Kubernetes dashboard and left it on the Internet without authentication, and that ended badly for them.

10:35 So this is just an example of that. So Yeah. The Kubernetes dashboard for so long never shipped with any authentication anyway. Yeah. Yeah. So so that that one so, yeah, there's there's there's there's there's various things that actually work kinda like in the old days. In the old days of Kubernetes, we're actually at the defaults. So the kubelet for a long time I think there's about three versions the kubelet shipped with no authentication possible. So you just you had to know the port was there, and you could always compromise the machine, which was fun. That's actually how I got into Kubernetes security.

11:00 Lab 1: APIServer Insecure Port

11:07 I I literally started looking at it. It was 2018, and and someone said said to me they sent me a link and said, this is possible. I said, no. That can't be right. I didn't believe that. And I was like, oh, that works. And then I went and found the GitHub issue, and it did work for, like, three versions. Wow. So, hopefully yeah. So what we're gonna do is once we've once the client machine's up, we can just pick one of these. What should we do to start with? So these are This is simple. All Ansible products.

11:34 Setting up the Insecure Port Scenario

11:34 Right? I can just run Ansible against any of these. It's gonna create a new cluster with that vulnerability. With that on it. Probably the insecure port is, like, the easiest. Okay. So it's it's happy now. Yeah. So the insecure port is probably the most simple. Okay. This is gonna give me a client cluster running Kubernetes with no authentication. Or no. It's gonna be using the because that's a real Yeah. Insecure port we're running. Yeah. The old parameter insecure port was, I think, only got deprecated in 01/20, I want to say. Yeah. That was It was surprisingly recent. It was recent. Okay. Ansible

12:09 and secure So it's Ansible the Ansible dash playbook. I'm sure there's I'm sure there's docs there. If if if it's just it's just the same command as it's ansible dash playbook and then that that file should do it. Yeah. And then the insecure. Yeah. So that should do. Okay. So how do I test that my hacker tools are working? So what we can do is once we've once this has come up, we can just docker exec into that client container, and that will be that's got all the tools and everything. And it should be on

12:35 Attacking Scenario 1: Insecure Port (Nmap & kubectl Access)

12:46 and it's on the same network, so we don't have worry about anything about what Docker for Mac is doing in terms of with network weirdness, which is kinda why I did this because Docker for Mac and Docker for Windows use some very clever but moderately complicated things with networking. And and I just thought that's going to cause things to fail in kind of odd ways. So it's easier if we just have a client machine on the same network as the the target cluster. Yeah. That makes a lot of sense. I guess we'll just give this a a

13:16 few seconds. So all those Yeah. So this stuff should be cached. Right? It's Yeah. Once once they come down, once they're cached. And what I did was I've I kind of, like, created custom kind so kind of a node image, and I've just customized that. And because, I guess, as versions go on, they'll stop supporting the old broken stuff. So as things get better mean, to be fair, I'm seeing about kind of things that were really bad for security in Kubernetes history. They are going away, you know, one at a time. You know, insecure port is going Kubel always needs

13:45 authentication now. So things are improving, so I'm kind of trying to keep the old stuff so we can kind of demonstrate still demonstrate these things because a lot of companies still do run older clusters. You know, companies do not always move on to the new shiny that quickly. Well, yeah. I think it was just yesterday. The Kubernetes project announced they were slowing down the release cadence, just three releases a year to try and help organizations. You know, four is is a big ask for some organization to upgrade their production clusters four times per year. Yeah. So, I mean, I one of the

14:16 one of the the things I always said at every I think every talk I've done and definitely the training for enterprise customers is you do realize this thing will need upgraded every single year. So I used to work for large UK banks, and I know that is not their way. That is not the way they do things. They don't upgrade whole systems every year. So so having a new thing that, you know, you're gonna have to upgrade every year is going to be a bit of a kinda culture shock, I think, in some cases. And even even enterprise things like OpenShift now,

14:45 they they've got the same release the same support life cycle as Kubernetes. They used to have, like, a really long one, but not anymore. Yeah. I remember, you know, back when you used your groups there a thing, I used to speak to people that worked at banks. They would always tell me that, you know, it's not a job for everyone because you have to get comfortable writing a line of code and then waiting six months for that to hit production. Yeah. That's the Yeah. Yeah. A lot of the banks are running systems that have been there

15:10 literally since, like, the seventies and eighties, and they're still running the bank. And their big problem now is that the people who write who manage them are all retiring. So if you ever want a really stable job, learn how to code COBOL on mainframes and go work for a bank, and you will never be redundant. Which is not great, Cobalt. Decided to bake my feet, so I fired. There you see. I've evicted our cat because otherwise, he would have made his presence known. So so now we should be up and running, and it looks so our client so our

15:46 our server tells you there at the bottom. The last thing it's doing is printing it saying the target IP address of our, you know, vulnerable cluster is of that. So we can now docker exec into the client machine. Is it just called client machine? Yeah. So minus IT client machine and then bin bash should work. Where's the name? Oh, just go client. Sorry. Oh, client. Yeah. Consistent naming, I never get it right. So now so now we're in our client, and we now have access to the same network as our target. We know that our target server

16:23 has got is on my IP address. So for this one, this one should be really fairly straightforward. We can check first thing you would do if you were actually attacking a cluster or if you're looking at machine, you'd probably port scan it. Right? So you you'd say, okay. I want to know what services are there for me to attack. And in the lab client, we've got nMap installed, which is kinda like the usual port scanner. So if if you just nMap should be installed there. K. Yeah. So if we do nMap minus small s, then capital t,

16:59 minus v, minus n, then just the IP address. It should there we go. And you can see we've got port eighty eighty o, which is our insecure server insecure port. And so now we know we've got access to that. So if we were, you know, looking at this machine, we go, okay. It's port if if we got an idea that it's Kubernetes server, we know, okay. This is, this is what we're probably going to want to attack. And so then we need to actually tell kubectl this where you get you you see how your memory for kubectl,

17:35 flags is. Okay. And I and I'll I'll what I'll do is I'll mention actually actually a pretty important point for the repository. In the in the GitHub repository, there's two folders. If you scroll, there's one called scenario setups and one called scenario walkthroughs. So the scenario setups is just like how you get it running because there's a couple that have a bit more to do. And then in scenario walkthroughs, there's kinda like the cheat sheet. Alright. Okay. So so if we get stuck and and I I probably would remember all the things to do with these,

18:05 we can always go in there and it remind us. So for this one, it should be super simple. So this one should be, like, awesomely simple. We just do kubectl minus s and then HTTP and then the IP address, 8080, and then anything. Yeah. And oh. Oh, here we go. That's come back. And we're okay. Yeah. So was gonna thought before. Yep. So we got access. Now this is the thing with the insecure API port for people who might not have encountered it is it's an old thing, and it allowed completely cluster admin access. Right? If you get access

18:48 to that, it's cluster admin. You are cluster admin. This was still in use in the default install of some Kubernetes distributions until, like, a couple of versions ago. So it's not amazingly unlikely you might find this. And we have when I used to be a pen tester in my previous role, we did find clusters that had this running. People would use it for, like, monitoring. So they'd say, okay. I I can't be bothered, like, configuring, like, rules and and creds for monitoring services. I'll just use the insecure support. I'd like that never happened, but it did.

19:18 I I'm definitely guilty of throwing my monitor in the end secure port. But I don't know if I was being really naive or just misunderstood, but I also thought it was read only. I'm assuming it's not. No. It's totally close. It's cluster admin. If you get in secure port, that's it. It's game over. Of course. Your entire cluster is yeah. It well, this is one of the problems is that not super well known, and it's not always super well documented, some of the stuff. I mean, actually, that's not that's not fair. The Kubernetes documents like documentation site is amazing. Knowing

19:20 Discussion: Insecure Port vs Read-Only Kubelet Port

19:48 where to look to find everything can be a challenge if you don't know. So if you don't know that that's a problem, you don't know to go looking for it. So in terms of, like, the way the lab is structured, what I set up is, like, the target, and it's in in the read me. It says, the target for every attack is to try and get the certificate authority private key from the the the cluster master. And and the idea for that is that that's the golden key. Right? If you ever get that on a real Kubernetes cluster,

20:00 Lab Objective: Obtaining the CA Private Key

20:20 you've got persistent backdoor access to the cluster for the lifetime of that key, which is usually measured in years. So if you ever give someone read access to that file, that's your cluster is theirs for as long as they want it, until that key expires. And most clusters I've seen installed will go somewhere between two and ten years. So, realistically, the life of the cluster. Oopsie. So, yeah, that's this is a a fun this is, like, my big one of my big things, again, people should know about your security is protecting that file. Like, read access to

20:53 that file is super, super sensitive. So I'm assuming then that with this secure port enabled, I can just create a pod spec with a host path mode for slash, let's say, Kubernetes PKI and then everything I need is sitting there. Yeah. We can And so the fun Probably the easiest way of doing this one is you can actually because it's QBADM that runs all the clusters, the control plane components run as pods, so you can just exec into them. Of course they can. Yeah. So what we can do is if we there's no there's no yeah. Oh,

20:58 Attacking Scenario 1: Obtaining CA Key via kubectl exec

21:31 yeah. My dash s. Okay. Yeah. I should really just export those but I don't think I'll be in this cluster too long. Oh, nice to be s. Yep. It will be for all the other ones also. So you're saying I could just jump into the API Yeah. I'm in the API server. There won't be a binary there, will there, Shell? Bing Shell? Probably Bing Shell, I reckon will work. Trash. Try Bing Shell. I wonder if Bing Shell will work. I thought Bing Shell. Yeah. They all seem to be they also I think, I mean, those there's some plans

22:09 to not. So in, yeah, in PKI And there we go. That's the c a key. So yeah. So if if you literally read access to that key is and that's, a big thing that that people should know about running Kubernetes is be awesomely careful with that key. Because if you give someone This one? Yeah. The the reason why you need to be so careful is if someone can read that key, they can create a new client certificate and sign it. We think that we're using that key to sign. And because client certificate authentication is always turned on in Kubernetes and you

22:43 can't turn it off, You can create a a credential. Just say, want this credential to last for five years, and I want it to be member of group system masters, which is cluster admin access. So if anyone ever reads that key, it's yeah. So that's why. I guess from a, like, a defensive point of view, like, if you've got a you'd want to have your audit controller log stuff like this as well. Right? Is that is that possible? You can you can put yes. Yeah. You can definitely do things like, for example, tools like Falco, I'm guessing, or other runtime

22:58 Discussion: Defending Against CA Key Access & Backup Risks

23:20 security tools. So there's a variety of those. Tracy is there as well. We can alert on you doing things like exacting shells and containers. So if you exact a shell into a container, then you're going to you're gonna get you're gonna get caught. I think, for me, probably the hardest part, though, is if people want to do, like, backups of their clusters. I've never seen a great idea for you know, you're gonna want to back these files up because you need them, but you have to be really careful where you back them up. And I have seen I've seen at least

23:51 one distribution that actually puts them in a config map. Oh, wow. I was gonna suggest, like, a public accidentally public SD bucket, which I've seen before as well. That will I have also seen again, when I used to do pentest of clusters, I have seen people check them into GitHub. Private repositories, obviously, but still checked into GitHub. But, of course, then you don't know who's cloned it. Right? So someone's cloned it. You have no idea what they've done with it once they've cloned it. Then they leave the company. They move team. Yeah. Yeah. Because the only way to fix that is rotate

24:23 everything. Okay. We got a question. You happy to take questions as we go? Yeah. Yeah. Questions are absolutely fine. Noel was asking, you said the docs are extensive. Are all of the exercises that we're gonna do in this repository doc? Are they mentioned are the docs mentioned, I guess, in the the lab? Yeah. So in the setup and walk throughs, I think I've got walk throughs for every single one apart from two. There's two that aren't done yet. But so so everyone we do today, definitely, the walk throughs will be in that folder. Okay. Perfect. And Cool. A random comment from

24:28 Q&A: QBADM Defaults & Distribution Security

25:01 who's asking if we can show how to install Rancher. I'm afraid not, but there is a video on the YouTube channel. Please feel free to go check that out. Alright. I do Rancher. I have done rancher. I like rancher. Rancher's really cool. It's actually a different episode, actually. The jukebox episode where I just kinda stream and say, right, what what are we doing? Let the audience Yeah. I was I was a great fan of the of TGIK. Yes. And they do that sometimes where they will say, let's try this. I haven't done it before. And and that you you you

25:29 tend to find some fun stuff there. Oh, definitely. Okay. So that's our first one. Right? So we've we've got the kind of concept now. Have cluster, attack cluster, find the key file. Okay. Should we do another one? I would love to. Should I start shutting stuff down? Is that important? Yeah. You you if you just if you exit that, then I it just if you shut down the if you you can get clusters, and it'll tell you the name and then just shut down the insecure board one. Shut it down or kill it? I'll just

26:02 Cleaning Up Scenario 1

26:02 delete it. Minus minus name I think it needs. Yeah. It's like minus minus name for the Alright. Depends on I think it one of the problems I've got these is my machine is so powerful. I end up leaving these running and I come back and I got like 10 kind clusters. Yeah. I'm I'm not sure my Mac software running the streaming software and Yeah. At least my fans are probably all I've got headphones on, I'm not sure. But I'm pretty sure my fans are already going mental. So Yeah. I I went the whole desktop route now

26:33 and now it's like a four gig RAM. You just don't notice. I left an OpenShift cluster running and didn't notice, which is impressive. That is pretty good. I like that. Okay. So Well, I have to pick one. Do you wanna pick one? Should we should we do kubelet? Kubelet's so there's one there r w kubelet no auth. Yep. I I I kinda like that one because, again, the the the the kubelet has its own HTTP API, but go try finding documentation on it. It's one place where there's not you can go read the code, but it there's not, like so because it's

26:46 Setting up the Kubelet No Auth Scenario

27:05 not intended for external use, but it does have one. So we can we can have a look with that. That's that's quite a fun one. Yeah. Okay. So that means I have to run Ansible playbook, pass in this, and let it spin up our new cluster. Yeah. So and it should be faster this time because it's gonna have the image, I think. We'll see. So I actually wasn't aware that the kubelet had an API. Why? Yeah. What's the intended use? Do we know? It's the API server talking to the Kubelet. So the API server has to tell the

27:25 Discussion: Kubelet API and its Use

27:35 Kubelet to tell the CRI to go do stuff. Yes. And it's this API and it's actually it's an interesting one because it's one of the places where the TLS configuration of, like, QBIDM clusters is not perfect out of the box. It uses a self signed certificate. So, usually, that's, like, signed in the certificate authority. But for when the API server's talking to the Kubelet, the certificate is by default just self signed. So there's a risk of, like, man in the middle and stuff. So the ship the Kubelet doesn't subscribe with a watch then, like, all the other component?

28:06 It goes that way. But if the but the API server, somebody seems to go the other way and say, hey, mister Kubel, go and do stuff. I think that's my understanding of it. I will accept corrections on that point though, for sure. Alright. Okay. Well, you know, I I love to learn on these things. So that's something I had no idea about, which is cool. Yeah. But now we're gonna take advantage of it, I guess. We're yeah. We're gonna show how you can take advantage of it. And this this one's this one's a little bit

28:29 fun because we we're gonna do all our hacking for this one with curl. Curl. Okay. I I know a few curl flags. I yeah. That's one of the fun things I one of the reasons I like I'm a great fan of HTTP APIs, and and I also get annoyed when things go gRPC from a attacker's perspective. HTTPs are so many tools. You know, you can just use anything which which talks that. And and then with gRPC, it's into, like, what proto do you have and do you have all the other things? I guess even with issue two p, if

29:02 the only tool you had was Telnet, you could probably still make that work. Yeah. Exactly. Yeah. But you're always gonna have Carol. Yeah. The Carol author on Twitter always posts, like, messages about the release and the Carol birthday and and all the new contributors and stuff. And just the sheer age and size of that project always amazes me. Yeah. I think it's 23 years old this year, like, this month it was. Yeah. It's been going such a long I he's got some really great information about things like how all the different weird things you can

29:31 put in URLs. So people are ever thinking of doing, like, regex on HTTP URLs, be really careful. It's it's astonishing the weird stuff you can put in URLs and how it's not part of a spec. So you can say I'm spec compliant, and it still will do something odd. That's great fun. Alright. Okay. So I'm gonna copy this IP address. I'm gonna assuming I'm going back into my Docker, my client. Yeah. That's the one. And, I mean, in theory, I could just run the same command to see if things are running. Right? But I'm assuming that's not Yeah.

29:51 Attacking Scenario 2: Kubelet No Auth (Probe & Nmap)

30:03 So this should fail horribly. It should say, no. I'm gonna I'm not yeah. I'm not gonna play ball with that. Okay. But if we do if we do the actually, if you go up, we should look at nmap. Actually, I don't know. Or just retype that, actually. We can do nmap again, we can show that the Oh, it might not oh, it might not know the port. If you give it minus p, just before the IP address, and then do minus p 10250, Let's see if hopefully, it should find it. Yes. Is that is that not the read

30:35 only port, actually? That that that's read write. So read only is 10255. 5 5 o. That was deprecated, wasn't it? Yeah. I was gonna say that's gone because that was again, it was one of the early ones and and it was could not authenticate the read read only port. There was no option to do it. So it was it was great information disclosure. If you're a pen tester, hit that port, and you get quite a lot of kinda useful information, definitely starting points. I in the early days, there was cAdvisor as well on four one nine four, but

31:03 that that went. I think it went a lower in the browser. Yeah. The the one zero two five five is actually the the read only port I use for monitoring. So now I feel less guilty that I wasn't I I am I'm happy that I wasn't using the API servers and secure ones. Yeah. Oh, the API server and secure ones. Yeah. Well, you've seen that's that's how yeah. The the read only keyboard port is not terrible. It's not I mean, as a pen tester, I liked it. I I, you know, I was I was always happy when I saw

31:06 Discussion: Kubelet Read-Only vs Read-Write Ports

31:29 it. But it wasn't like a game over. It wasn't I I would have finished my job today and that's me done. Yeah. You have all the information that you need. Right? Almost? Yeah. But each host? Yeah. It tells you things like, you know, what images am I running? What flags is the API server running? If you had control plane node? Lots of cool stuff that might otherwise lead you, you know but, yeah, it's not game over. But this is this is the read write one. So this is this is the one with the with the funky API.

31:54 So to testing that, what we can do just to get started, if you curl minus k because it's gonna be HTTPS. And, yeah, it's HTTPS on that. And then And you should get four zero four. Right? So that that's fine. The the base URL says, no. Go away. I'm not I'm not want but but because it's given us something, we know it's running, and we know it's not authenticated. Because if it was authenticated at this point, it would say, no. Go away. You haven't given me the client certificate I was expecting. So from attacker's perspective, just seeing that 404

31:57 Attacking Scenario 2: Kubelet No Auth (Curl Access)

32:27 is, like, that's okay. I've got something to look at here. So then you need to know then you need to know the the end end points. And then the first one is pods. Nice and simple. And, yeah, Jake is there for that reason because otherwise, you're there forever. So now we we we know every pod that this kubelet is is running on the machine. Every pod, every security context, every volume amount, everything. And this is kinda what you got out of the read only port. So it is actually kind of it's kinda gives you an idea of the

33:05 level of information disclosure. But what we need to do now, and this is this is the fun part. We might need a text file to, like, copy paste bits and pieces, is all the information we need to get access to that key is in the information we just got. Because there's an endpoint on the kubelet API called run, and it lets you run commands inside pods. But we need some bits of information to make it work. Okay. Let's get that text Yeah. Open. Documents. I will you don't need to see it. Right? I can just leave it off screen.

33:42 Yeah. Yeah. Okay. So the first thing we need is namespace. Now that's always gonna be cube system because we wanna try and get the API server pod. So if we do the endpoint is is run, and then it's gonna be slash. And the next thing you put after this after slash after slash, then it's cube system. Then what we need to know is we need to know the pod name. And so the pod name, we're gonna scroll up until we see if can find the API server pod in that output. It will be there somewhere, but it might

34:09 be a way up. Let's hope the scroll back buffer doesn't Oh, my god. Was there from pods. Correct. Okay. Network. Oh. Wonder whether Why does the network? Is it is it API test server or Yeah. Correct. Okay. That's it for API probably wasn't my best move, but we we are we do have it kinda restricted. I wonder if we can find out the where is it going to be? Yeah. There we go. So that's that named cube dash API server dash. That'll be that one there, I think. Yeah. Think that's pointing the screen that you guys

34:57 are pointing. I pasted that into my thingy. Okay. Yeah. So you got that. So we have that, and then after that so, yeah, paste that in and then slash, and then the name of the container, is always kubit dash API server. That one's pretty simple. We've got a nice case. So now we need to tell it so the way this API works is you're posting. So you post the command you want it to run. So we do minus capital x post. Mhmm. Yeah. And at the end of it all, we want minus lowercase d and then space,

35:42 open quotes, c m d equals let's do who am I to be to start with. Let's see if that's gonna work. Yay. I think Noel has maybe done this lab before. Otherwise, he's just been doing bad things to other people's kubelets, but I kind of suggested that. I can add there. Okay. I have the ability to run any command as you were inside the kubel API. Yeah. So this is this is why leaving the Kubel open is really dangerous because, basically, you are essentially, Kubel controls Docker. So you think about I've got Docker exec, that's what you've got, essentially. You've got anything

35:55 Discussion: Kubelet Exploit & Crypto Mining

36:23 you can do with Docker. Yeah. I see it up here. And that's backdoor access to the cluster. So anyone leaving the kubelet open and there have been cases. So, like, Aqua have a research team who have honeypots, and they look for people scanning the Internet. And one of the things that the the crypto coin mining community have worked out is that cubelets allow you to execute code and execute commands on machines. So they absolutely do look for these. And if you leave so if anyone ever left a cubelet port open on the Internet, you can basically expect someone to do that

36:55 quite simply. But that that that vulnerability was actually that was the thing that got me to Kubernetes security. So that was in 2018. That was that always worked. That worked on every cluster. And, like, for about probably the first couple of Kubernetes security reviews I did, that was my oh, look. My cluster. Well, that's eye opening. This is at least, like It's a it's a fun one. I like that one. It's So it's not a well documented API. So simple as well. Just Yeah. A quick curl command. As a fact that I understand why the kubelet run API endpoint is there,

37:31 but just super dangerous. But that's just now authenticated only. Right? That's what you were saying. That's that's how Yeah. So it it by default, obviously, you can turn that off. Right? So it is it is possible if you misconfigure the Kubelet or if someone tries to do something with Kubelet that they could make a mistake. And it's kind of a good reason why it's worth, like, periodically having these things these these tools, like, Kube Hunter and some other tools, which will, like, probe these. There's also CyberArk. I've got a tool called Kubelet Cuddle, which will automate that process. So instead of

38:04 using curl, they actually have a tool, a Go binary. You run the Go binary against that API, and it does all this for you, which is really super neat. So, yeah, it's worth kinda doing those periodically on your clusters. You should just get back nothing. Right? You should create, nope. You're not authenticated. Go away. Okay. We have a comment say in our use of the word crypto community. It was maybe a little bit harsh. I don't think it was intended to be Crypto mining. Crypto mining. Oh, I said crypto mining community. Sorry. I understand. Yeah. I I mean, crypto I said

38:34 crypto mining community. Sorry. Maybe I wasn't I wasn't clear. My accent might not have crypto. No. I wouldn't say crypto community, but crypto mining community. The people who go around, compromising people's machines for crypto mining for clarity. Yeah. I I I know what you mean. There are a lot of people out there that are scripting all of these weird and wacky ways to be able to run crypto miners on any sort of computer across the entire Internet. We mean those people. Yeah. Those people. The people who are yeah. So for clarity, the people who are doing

38:59 that look for cubelets because they provide compute in the same way as the API server provides compute or the Docker socket provides compute. It I mean, that port number, I guess, you find that open anywhere, you're it's pretty much just a a green flag. You know right away you can probably get access to it. Yeah. It yeah. Unlike eighty eighty can be a bit you know, a lot of things use the eighty eighty because it's, the alternate web port. But the Kubelet port, one zero two five o, that's there's not a lot of other things. So yeah.

39:27 And there are scanning services that will hit that now. So I'm pretty sure Binary Edge, which is another similar thing to Shodan, it will it it indexes one zero two five o. So I should I should run it on port one zero two five one then. So up here. Yay. Absolutely. Actually, no. There's something that runs on one zero two five one. Weirdly, all of the components like scheduler and controller manager run on ports, generally just for health check. They only look at health zed end point, but you you find there's loads of ports now.

39:57 But, yeah, you could run a different port, obviously, authentication. Yeah. Just keep that turned on. Don't turn off. Don't turn off. Don't ever break it. So I'm gonna have to get the list again. You know, I'll I'll leave it running till my computer start crying, and then maybe I'll start shutting down. Yeah. Let's start spinning up the next one at least first. So we've done n secure with a read write, kubelet. That was a scary one. Do we want to do a kind of a user one or do want to do an x c d? X

40:25 Setting up the ETCD No Auth Scenario

40:29 c d is like the other one for, like, server stuff. It's like the other like, obviously, the database where things are, or you wanna do an SSH to things one. Well, I I guess the attack factor with x e d is, like, I if I put something into x e d, like, the compute the API server is gonna make that run somewhere. Is that the attack factor there, or is it something else? So if you allow the if if you either get creds for the etcd server or if someone puts etcd unauthenticated so, for example, they turn off it'll put

40:57 authentication properly on etcd, then, obviously, you can dump out anything's in etcd. So it's a question of how do you go from iPhone port twenty three seventy nine, which is the XCD port, to running code on the cluster? And and there's a couple of steps to that, which are kind of fun. We we can if you want, we can look at that. That's the last of the big server port attacks. Yeah. Why not? Let's do it. So we do have a question, which I'll get to in a second, but we'll spend this one up first.

41:25 Let that run. Okay. Noel is back with a question. Are any of these vulnerabilities that we are looking at or or part of the labs still shut by QBADM by default? No. QBADM QBADM, I've done a lot of work. And and this great props to the QBADM team. They've done a lot of work on, like, the defaults so that this doesn't because, obviously, this would be super dangerous if if if this is allowed by default. I would say that all the major distributions now get this right. If you'd asked me that question two years ago, I would not be saying the same

41:27 Q&A: QBADM Defaults & ETCD Security

42:01 thing. But now, it's it's it's it's getting a lot better. That said, you know, the there are, like, a 20, I think, certified distributions. Do all of them get all of these right all the time is a good question. And if someone has a load of time on their hands, you could totally go and find out probably, but you'd have to have a lot of time in your hands. So, I mean, I guess, it's q QPDM just must be the best way for people to go. Like, if they're not going down to OpenShift or any other distribution Yeah. Like, people

42:34 shouldn't be manually configuring all of this themselves. They should just Yeah. Use QPDM. Right? I I would say that. Yeah. I I I QPDM, there's a lot of, like, weird edge cases that you don't know. Actually, ETCD is a good one. So the way ETCD works with its authentication is it trusts any certificate issued by the certificate authority it's been told to trust. So you have to have different certificate authorities, one for XCD and one for Kubernetes. You can't use the same one, or anyone with a client cert for Kubernetes would be able to get to XCD.

43:10 And SCD has no authorization, so it just gives everything to anyone in the way that Kubernetes sets it up. So it's that's a that's a kind of a weird edge case. You wouldn't have thought of that. I wouldn't have thought of that. But knowing that, yeah, that's the sort of thing that could be a DM have done the work. They've they've worked out what's needed and done the hard work to do that. Oh, yeah. I can imagine, like, you know, certificate authority and certificate management is a pain in the ass the best of times. If

43:33 I was hand rolling my own Kubernetes clusters, it makes it insensible to just say, hey. I'm gonna have one CA. I'm gonna provision and send a bunch of certificates and not really think beyond that. But then when you realize that, oh, anything why am I clustered now has the ability to speak to the database that powers it. That's And I've I've seen I've seen distributions made that mistake. I've absolutely seen distributions made that mistake because it's not something that's well known. You know, if you don't know that, and it's and it's not obvious. And the

43:58 thing is what I find with what in my kind of I'm I'm just a security person, but from from what I've seen of how people use Kubernetes, it's a big complicated system. When you take out the box, you use it as provided. You don't go playing with it because you want it to work. Yeah. That's true. Okay. Let's see. Oh, it's still spinning. Okay. It's still spinning? Maybe your laptop's fans are actually, it should be kinda is quite nice and light. It's it's Oh, yeah. They're they're taken off. Maybe I should have cut the other one.

44:30 Alright. It's it's got past the hard part. We could still shut that down. So delete cluster name and that was a redrary. Yeah. Okay. I would have laughed if I shut down s t d no off, but I'm okay. Yeah. Alright. So we have our IP and I'm gonna jump back into our oh, client machine. There we go. Are we we end mapping? No. We can yep. We can end map. We can end map for Port 2379. That should be our our port of choice here. Oh, it's 0 5. It is one up. Shut the other one down, it's it's last

45:00 Attacking Scenario 3: ETCD No Auth (Nmap & etcdctl Access)

45:20 digits five. Very good. Cool. And it oh, nMap knows it's nMap knows about that port now. Because that service line just comes from nMap's database. Yeah. So it must have, like, decided it knows what SCD is now, which is cool. Making hacker's lives easier everywhere. Yeah. You know, you gotta use mMap. I it shows you that I spent a large number of years using nmap that I've got because I can remember all this. The one it's the one command I don't forget the flags on. Everything else, I'll have to look up. But nmap, I'm fine. No. Nmap and TCP

45:54 tumper tools I reached to as a very, very last resort. So they're definitely not something I'm comfortable with. Yeah. So what we're gonna do so obviously, etcd is gRPC. It's not HTTP. Although, there is an HTTP API in there. It's just a bit hidden. Should do the job. And the first thing we need to do is the really weird thing that I always forget, which is need to export an environment variable to tell it we're using version three. Yes. So it's export alright. So export. X c d c t l underscore API equals three. Yeah. I have a Otherwise cheat sheet that

46:50 I always use to configure s c d Yeah. When on clustered especially just because I could never remember those that Yeah. The temptation to get it working. I never remember that and I always wonder why it doesn't work. And then I go after about three or four tries, oh, that. So now it's like one of those things I think I've done enough that burned into my brain. So we should now be able to talk to the servers on o five. So we need to give it some flags. First one is dash dash insecure dash skip

47:21 TLS verify because who cares about verifying TLS? And then we need to say minus minus insecure dash transport equals false because it is on HTTPS. Then we need to tell the endpoints. This is the HTTP cuddle's got a lot of flags. So minus minus endpoints, plural, equals yeah. It's HTTPS. I think you might need to get HTTPS for the start. It's not an example there. So I I kind of omitted it. But Yeah. I think yeah. I've I've I've just kinda got Let's see. And then at the end, we we just say get slash. No. Are we do if we yeah. We'll

48:04 we'll try get slash minus minus prefix. I'll try yeah. Minus minus prefix. It's minus minus keys only. This is why I wrote all this down. We go. And so that is what that's done is that's just dumped out everything you ever like, that's everything SCD knows about in terms of cluster. And as you can see, one of the cool things that's in that list is secrets. And this is why access to SCD is so incredibly dangerous because that's every secret in the cluster. And are those available in plain text, or is Kubernetes not providing any level of encryption there?

48:36 Discussion: ETCD Secrets and Encryption

48:48 The they are but we're going getting them over the API. So it's good the by default, it's in plain text. You can get the API server to encrypt, but it's not default. So guess what? People don't turn things on. They aren't false. Very often, you should turn it on. But, obviously, if you go via the Kubernetes API, you'll still get them in plain text. Right? Because you have to be able to get them back. But, yeah, we this case is a default install, so it doesn't encrypt them by default. So we we can get them out in plain text. Well, kind of

49:20 plain text. So what we're gonna do is we need we need our our a token that's gonna give us, like, access, like, good access. I'm gonna shout out to one that used to be really great. There's a there's a secret. In Kubernetes, it's the only service account token that you can get cluster admin without out of the box, which is the cluster role aggregation controller, which I bet I had never heard of. And it used to be there's there's it used to be cluster admin. And it's not cluster admin anymore, but it has impersonate rights so it can

49:25 Attacking Scenario 3: Obtaining Service Account Token from ETCD

49:56 become cluster admin if it wants to be. We have a complex one to play with, though. So what we've got, if you scroll up a little bit, I think I put one on here called admins account. Yep. So if you get that path and then we yeah. But that just get that, basically. And this is where we get the the the light that is gRPC. But if you see where it starts e y? E y j That's j w t. Right? Yeah. And down to the point where just for the hash sign. It's like a hash sign Yeah. So

50:40 they are up to e y j. If you get that and then do kubectl dash dash insecure skip TLS verify because we don't have the the the TLS setup. Minus s HTTPS, the IP address 6443. What was the IP address? Sorry. 172 0, 0 5. 0 5. It's not 18 or 20. I think it's 143. Yeah. And then minus minus token that. And then, like, get p or something. We'll get nodes. Yeah. I mean, just maybe get an IP address wrong. No. It's not it works. So that is literally how you go from so we we're now we've

50:51 Attacking Scenario 3: Using the Stolen Token

51:35 now got some access. If you wanna know how much access we've got, the best my favorite one of my favorite commands is auth can I minus manage list? So we just yeah. Auth can I? It's the off space can I and then Off space can I? Yeah. And then that that's And we have the magic at the top, which is star.star. Star Star. Anything. Which is anything. So this is a cluster admin account, which just so for for for for ease of of of doing stuff, this was just cluster admin because we're trying to demonstrate the

51:39 Attacking Scenario 3: Verifying Cluster Admin Privileges

52:13 ACD part, not the, like, provest part. There was also a great quote that Iain Coldwater did, which I use in all my talks, which is we're all made of stars, but your RBAC shouldn't be. I like that. It's yeah. I love it because it's it's so true. There's a lot of temptation. Again, Kubernetes is complicated. RBAC is complicated. There's a lot of temptation, I think, for people who really have a problem with it to just say, give it cluster admin. I was actually looking at installs for various open source projects last week, and I noticed

52:16 Discussion: Dangers of Cluster Admin and RBAC

52:47 that a lot of them said, if you're doing this in GKE, you won't have enough rights to begin with, so just run this command to make yourself cluster admin. And I was like, it didn't ever say after that, and by the way, remove cluster admin from your account afterwards. It just said, make yourself cluster admin. And I was like, yeah, that's really super super dangerous because cluster admin can do anything. I think every JKE tutorial I've seen, whether it be installed in Celeb, installed in Rawkode, any of these things, they all start with, oh, you have to elevate your privileges. That's

53:17 the first thing you have to do with JK. And and they always say cluster admin. Don't say elevate it using this profile that we've designed to only give you the things you need. It's just go get cluster admin. And And then that rule is just lying there around forever. Right? It's like a Yeah. And then someone if someone gets your account or gets your token, you're gonna have a bad time. And and as more and more things go in as CRDs, so more and more of, like, people install things like, say, my Kubernetes cluster is now

53:43 good to manage all my databases through an operator, you're not just giving them Kubernetes access. You're giving them database access too to all the databases you're managing via Kubernetes. Or even if you're doing OpenShift, to the machine nodes as well because, like, machine in OpenShift now, you're managing the VMs via OpenShift. So, yeah, that is my one of my rants about why cluster admin is super dangerous, but and we see it too often. Yeah. That makes a lot a lot of sense, and I'm gonna definitely think twice about those sneaky little oh, just do this example.

54:16 It's it's it's it's yeah. It's super common. That and the other one I've noticed is operators. So this is a, like, a QBADM cluster. And as I said, there's no one who gets cluster admin by default. But to give you an example, an OpenShift four five cluster, I think, has 42 accounts that get cluster admin. Oh, wow. So there's 42 different operators. They're all the operators. They're all the operator service accounts. So if this if I was running this command against an OpenShift cluster, we would have a load of different choices that we could use to get cluster admin because I said

54:45 because I think it's 40 something. I just it was over 40 that have cluster admin. So it's it's it's one I'm kinda hoping they might fix that, and it's just, like, early days ago. Because it it it it's kinda like, you know, in the old Windows world saying, do people need domain admin? And the answer is you should never need domain admin. You should need you know, so there must be some smaller quantity of rights we can you can give people than than that. Yeah. That makes sense. So for this one, if you wanna finish

55:11 Attacking Scenario 3: Obtaining CA Key using Stolen Token

55:13 it off, we can just do the exec command again. So we can just as we get p o and then make cube system names Oh, yeah. Yeah. Okay. To get the the CSR or mission. Yeah. Our mission. They probably want to I think we'll to get PO first in in KubeSystem to Yeah. I was just thinking that. Was like, oh, I I wouldn't remember that name. They they are predictable names, but because kinda's kinda our Kubernetes is good like that. I got it. So that's it. Right? That's yeah. Yep. That's another another backdoored cluster. Another cluster you can

55:59 you can keep access to forever. Of course, until you delete it in, like, one minute's time. Cool. I am learning loads. Thank you. No worries. Learning a lot I think what I like here is I've I've learned a lot of things that I really need to start checking on my production clusters. Little subtle changes. And I think because I use Kubitiam, I'm probably okay. But it's still nice to know which things there need to be checked on a regular basis. It's yeah. The the the port ones I said, for all the mainstream investors, you shouldn't have

56:10 Discussion: RBAC and Installer Risks

56:31 a problem. RBAC rights is a different story. The thing that kinda gets me about about RBAC is it's so easy for some install programs. It's like what you said. It's so easy for some installer to say, just do this. And I've seen installers that do things like give the default service account in the default namespace cluster admin. I was like, you might you wouldn't unless you read the manifest, you'll not notice. Yeah. And then forevermore, every single pod that ever gets deployed into that namespace gets cluster admin, which is, of course, it's gonna hit you

57:00 Lab 4: Privileged Pod

57:02 sooner or later. It's just how long. Nice. Alright. What what do you wanna do next? What else is this one? We've done a lot of server stuff. You said there's other stuff. Yeah. Let let let's do a let's do a kind of a a privilege escalation one. Should we do to create pods easy? That one's that one's alright. So SSH to create pods easy. What's wrong with create pods hard? What is wrong with pods hard is harder. And we we can do pods hard but we're gonna have to do fun stuff with like reverse shells and things.

57:08 Setting up the Create Pods Easy Scenario

57:30 Let's start with easy mode and then if we have the Yeah. I was gonna say we'll start with easy and we'll see how we go on because because pod's hard, it like it gets a bit harder. Yeah. I think this is just Like, you know, there's there's so many here. Right? We're not gonna do all of these. Yeah. No. We're not we're not all of them. And and definitely, if if anyone's, like, watching and thinking, hey. They haven't covered x. Issues or PRs are always extremely welcome. Is the automation to build the vulnerable images all here for people to piggyback on? Yeah.

58:03 So the the the images are all in the if you wanted to create one, it would just be I would just look at one of the existing playbooks. They're all fairly straightforward. So these are these are manifests where you've you like, pick, like, the XCD no auth yeah. These are the tasks. If you just pick one of the playbooks, the XCD no auth one or something Yeah. Okay. Yeah. They they you can be able see it's not. So all we really do is say, right, start up a a kind cluster, and then we just have a little task to say,

58:34 you know, in this, give it a one you know, I'm gonna do a a custom client cluster, and I've got a cluster you see there's a cluster config there? This is etcd no auth. Mhmm. So what kind lets you do is it lets you it lets you customize how QBADM is installed. So for example, you can change a port number, you can turn off authentication, you can change an API server flag to do something it shouldn't. And k that's one of the things I really like about kind is you can just write these configs. They're a little bit funny

59:00 to get used to, but once you're used to them, they're they're really easy. Awesome. Yeah. Definitely. I'm gonna play with that. One of the things I've wanted to do with, like, the clustered stuff is, like, provide reproducible ones that people can do and try and fix on their own time as well. And maybe doing that through kind with this config approach would be a a decent way to do that. Yeah. For, like, for, like, Kubernetes level stuff, I think that that would that would work. Mean, obviously, if you're doing that low level things kind, the one place that doesn't shine. But it

59:06 Discussion: Reproducible Labs with Kind and Ansible

59:27 it absolutely for for for things like this, you know, you could you could get clustered kind of broken clusters quite easily, I think, for anything like OS level or not. Yeah. The people that modify the kernel and the directory and the proxy directory. I I don't like those people, and I will not build reproducible versions of that because it's Yeah. I guess I guess there's I I've always wanted to do more with, like you know, because kind is great to a point, and then for anything like Kubernetes level, fine. But if you wanna do low level

59:53 stuff, I know there's there's good automation for, like, building raw machines Yeah. And doing that. You that's what you'd need to do that, I guess. So this one's doing a bit more, it looks like. Yeah. So this one, what this is doing is is so the idea of these SSH to something ones is to simulate an idea that you've got you're a legitimate user, and you've got some level of access to the cluster. And how and and so it's so it's it's this a lot of these are basically things that are in our back. I'm like,

1:00:05 Lab Goal: Privilege Escalation Scenarios

1:00:26 what can I do? So if I've got create pod, what can I do? If I've got the ability to read pod logs, what can I do? You know, all that sort of thing. And it's how easy would it be for someone who has that access to to go and get, you know, cluster admin. So what we need to do with this one I'm gonna get my setup thing up for this, actually. We need to SSH into the place that we have the so it's SSH minus p 32,001, and it's SSH user at and then just the IP address.

1:00:40 Attacking Scenario 4: Create Pods Easy (Initial SSH Access)

1:01:00 All one word on the ssh user. No. I passed the dash l so we should be okay. Oh, that's the the user's ssh user all one word? Not ssh dash user? Cool. I should do it. And password's the same. So we're now what I've done is I've given we've given this service account in this pod a level of access. If you do, like, a command, it should it should be able to talk to the API server. But you don't have right to get nodes. You do have right to get pods. I have some access. Yeah. So you have some

1:01:25 Attacking Scenario 4: Checking Initial Permissions

1:01:43 access, but not all the access, which is if you think about it, that's a it's more likely to be what, you know, somebody who's trying to get extra access can do. So if you can see what we've got here, we've got right to create pod exec, which is a subresource of pod, and we can get and list pod logs, and then we can get all those new things to pods. But that's all we can do. Right? We can't get to nodes. We can't get to anything else. So the question is how would we get access to

1:02:14 to to access the underlying to the underlying node? And and so there's no pod security policies on these clusters, which means that that, technically, if we can create pods, we can do bad stuff. So there's a couple of ways we can do this, but let me get the where are we? The easiest way. So this is the easiest way. It's kind of a neat one. If in the in the root of that machine root of that pod you're in there, there should be a manifest called key dumper pod. I think it's the root directory, if I

1:02:18 Attacking Scenario 4: Identifying Exploit (Create Pod + Host Volumes)

1:02:47 remember rightly. Key dumper pod. There it is. Yeah. So if we count that out or LSR or whatever, what we'll see is this is just a neat way of demonstrating if you give someone create pod, what can they do? And we'll see what that what that does is it it's just gonna do like, give me a pod within BusyBox, mount the Etsy Kubernetes PKI directory in, and then just cap the key. Yep. So if we create that that pod on the cluster, It should. Fun part is it's going to say oh, what else do saw? It's say create create

1:03:18 Attacking Scenario 4: Creating the Malicious Pod

1:03:32 create create initially then. Once it's done it, I think it'll go to crash back off loop. We don't actually care. There we go. Hey. And there's your key. Nice and easy. But if you anyone so think about it. If you give anyone create pod on your cluster and you don't have a PSP that stops them from mounting host volumes, there's nothing stopping doing that. Yeah. Because every worker everything that runs, like, Kiplit has that CA key available. Is that right? Yeah. Well, everything every control plane node. Every control plane node. Yeah. So the this is

1:03:44 Discussion: Create Pod + Host Path Vulnerability

1:04:13 actually a fun one, and I and I was I was actually discussing on the Kubernetes Slack today is what's the security trade off of running the kubelet on the control plane nodes? So kubelet m by default will ship will run all will run the kubelet on the control plane nodes, and will run the API server inside a pod. But it means you can get to that node through the Kubernetes API. And if you can get to the node through the Kubernetes API, then you can do this. And there's obviously, there's tolerant there's tents. People always taint their control plane nodes, but tents

1:04:46 are not security control. Yeah. They're options. Yeah. You can just tolerate any taint. So whenever I'm doing this, I just put tolerate everything. Tolerate all the things I don't care about taint. So anytime anyone is running their control plane components in pods, if you have any users who can create pods and can mount host volumes, then that's possible. Is it possible to tell the kubelet to only run static manifests and never accept other workloads? I don't think so. Good question, though. I I don't think so. That would be a cool PR. I'm sure someone Yeah. We get interested. Yeah.

1:05:09 Discussion: Mitigation Strategies (Admission Controllers)

1:05:22 It's it's kinda like one of those things where like, if think about it, you need things like CNI and probably logging solutions Yeah. And everything else. And it's whether you want to go down that line of, like, the the management overhead of of doing that yourself with, like, system d or wherever else to run these components on the control plane. So it's it's one of those areas where I think there's definitely a trade off. You know, there's ease of use on the one hand, and there's, like, you if you do it this way, you have it depends

1:05:53 on who your user base is. Right? If you trust all your users or maybe you deploy everything to your clusters via CICD and there are no interactive users, this may not be that bad because you can have a review process in your CICD pipeline that says, hey. Post volumes. I don't think so. I I guess you're probably more likely to have some sort of a measure controller that just that wouldn't allow this anyway. Think that would be like, something like Kivernal running in your cluster would automatically stop. Yeah. Yeah. Kivernal and and are probably gonna be the two ways people will

1:06:22 stop this going forward. One thing I I kinda noticed though is you need to be you need to be whitelisting acceptable volume types, not blacklisting unacceptable volume types. And the reason is that new volume types come along. So you might get something like a new volume type that comes along and allows you to do this, like ephemeral volumes or which are coming along. And ephemeral volumes so if you say I'm gonna block host volume, host path volumes, and then ephemeral volumes come along, if you're using a blacklist and you don't keep up to date on the

1:06:53 fact that ephemeral volumes just came along, you won't know that and that may allow a bypass. Yeah. So, yeah. Tricky. Yeah. I don't think I like security in the Kubernetes landscape anymore. It's it a lot of it comes comes down to, like, what you're trying to do. I I've seen and and there is quite a lot of push for this at the moment looking at multitenancy and especially, like, hard multitenancy. I would be really super hesitant to say I could do hard multitenancy in the Kubernetes cluster. I I'm sure that it's possible, but I'm also sure it's really difficult to

1:07:07 Discussion: Multitenancy & Alternative Runtimes

1:07:28 get right, and you have to be super careful. Yeah. I I I think I agree. I guess I I I've never had to worry about multi 10 say, and I Kubernetes cluster as well. I I don't feel like I have any experience there to comment, but, yeah, I think I'm on your side of the fence there. Just even thinking about that and keeping us secure seems like a bit of a main field. Well, you need to get your R back completely right. You need to get your pod security policies, what you would have used as it's

1:07:54 going away either the new I haven't given the name yet, I don't think. Hierarchical The new replacement. Well, yeah. I I think something like that where either hierarchical namespaces or cluster API. If I was going to try and design a Kubernetes multi tenancy thing, I would probably more inclined to try and do it via cluster API where you're giving your tenants a cluster each, not a namespace each. There is actually a cluster API, a nested provider, which runs Kubernetes clusters on Kubernetes. Maybe that's an option people could go for more hard multi tenancy. Yeah. That or or if you can

1:08:33 don't think I actually quite like the idea of is clusters where the nodes run, like, Firecracker or something or, you know, some other, like, serverless style container where there's no real node to get to. And, actually, a fair point to make and it's probably worth mentioning for these is things like this obviously aren't a problem in managed Kubernetes because in managed Kubernetes, you have no access to the control plane. So if you're running AKS, GKE, EKS, there's no key to go and get. There are other problems and there are other things to think about, but this isn't one of them.

1:09:00 Lab 5: Helm 2's Tiller

1:09:02 Yeah. Definitely. I do like the firecracker approach as well. I actually have an episode with the people that we've worked coming up and looking at their fire cube project which is Yeah. Firecracker I Yeah. But I think that that kind of the ultimate container runtime approach where I mean, Docker's great. I love Docker. But it's it's got what I always call a flexible security model, and it's flexible. And if you want to and you have the access, you can just turn it all off. You know, privileged containers are a thing. Yes. You know? So so that's the problem with

1:09:32 flexibility. It's really cool, but it can be removed if someone has the rights. Yeah. I think there's there's other run times out there that are maybe targeting the more security conscious clusters. Cata containers is the other one that kinda pops in the head. I think they're And gVisor as well. And gVisor. Ones that yeah. Those are are other ways of achieving. So things like that, I think, if I was going to someone said to me, you're engineering you're engineering multitenancy clusters, and I wasn't able to run away, I I would be looking at at at something like that.

1:10:03 That's cool. That's the kind of segue in conversation there. I like that. Yeah. Why don't we we compromise this one. We got we got the CA Yeah. That was done. We've we've got that one. That that that's that's create that was create pods easy. Alright. How about we look at one more and then we'll we'll leave the rest up to the viewers to try it over time and I'll I'll do the same and another bit of play. Well, if you could only pack one more to first to take a look at what we see. What do we want to do?

1:10:27 Setting up the Tiller No Auth Scenario

1:10:30 I'd say Tiller. I I was gonna say Tiller, but not Tiller because no one uses it anymore. But I I was I was really talking to an ex colleague of mine. He said he found it this week. So Alright. So we're gonna take a look at Teller, which is the Look at the the Teller from Helm v two. So this is now this is now deprecated. But, again, as we were saying before, in large enterprises, something being deprecated does not mean it will go away tomorrow. If you are a a security engineer or you are a penetration tester and you are

1:10:31 Discussion: Tiller (Helm v2) Vulnerability

1:11:02 dealing with the enterprise world, you will regularly come across things that are very old and have been in there for a long time because it took two years to get them installed, and people are not going to uninstall them just because they're deprecated. Yeah. And Tethr was so ubiquitous. You know? I I I can't think of a single Kubernetes user that wasn't using Helm two Yeah. A couple of years ago. Like, it was just everywhere. And for a notable reason, like, Helm makes a lot of things a lot easier. Deploying third party applications that you don't

1:11:28 know how to operate. Helm has a chart for it. But unfortunately, the teller was this kind of cluster admin operator setting in everybody's Kubernetes cluster available to do bad bidding. And and for a year, for one glorious year, it was how the people in the team and myself compromised most clusters. I'm not I'm not even joking. Literally, for one year, most Kubernetes security reviews we did ended with, we are cluster admin because we found Tiller. That must have been just like your as soon as you see that, right, you're just like, okay. This is this is the game

1:11:56 Discussion: Tiller Authentication Default

1:12:01 over. Right? The thing is that Tiller does have authentication, but the default installed didn't turn it on. It it warned you. There was a line, to be fair to them, in the thing that said don't do this. But if you hold the the the the getting started thing, it didn't put authentication on. And as a result, we saw I mean, I'm talking production clusters, not just dev clusters. I'm talking about production clusters. We found them and they people had installed it and they hadn't, you know, turned on authentication. Yeah. I mean, I think I I've definitely

1:12:33 done the hell install and let Tethr just do its thing. It's I I actually wasn't even aware that I could turn on any Rawkode authentication on it. Yeah. It's it's one of those things. And I said, yeah, for one glorious year, that was, like, you know, a a not inconsiderable number of clusters that we would write a report. And it was, like, critical at the top. You're running Tiller authentication. Yeah. We got a a comment from who says, Helm template. A really good way to do it. The downside is you lose access to the hooks, but well, not with Helm three,

1:12:58 Discussion: Helm, Operators, and Permissions

1:13:07 of course. But with Helm two, you don't have access to, like, the post install hooks, the pre install hooks, all those things actually make the chart work. So, like, 90% of the time, was cool and it worked, and I did it as well. But there's always that chart that it just caused a lot of problems. And but a good way to do it. That was it. And the the yeah. Helm is great and it it simplifies complex and sold a lot. But Yes. I mean, they've changed it now. Right? And and it's all CRDs. Not not not I

1:13:32 will say that CRDs don't me a little bit, mainly because of what I said before about people giving out star rights. So you start seeing more and more things as CRDs. Things like, for example, in traditional infosec, you might say read only rights are sent are safe. Like, saying, oh, you know, I'll give someone read only. That's okay. But not in Kubernetes if you start putting sensitive things in your cluster. And a lot of things I've noticed don't use secrets reliably. So you can't say give someone read only to everything bar secrets because people will put the sensitive thing in

1:14:05 a config map or they'll put it in a OpenShift deployment config or something like that. Yeah. And there's something I've seen is, like, even though Helm two is now deprecated, most people are moving away from it, they're moving away to operators which aren't exactly given much less permissions than what the teller probably was originally anyway. As I as I said, yeah, OpenShift operators definitely the quantity of cluster admin is high. It's not I'm sure it's not all of them, but definitely it's a good number. So it's it's one to watch if you're going to operators is what are they doing with

1:14:34 RBAC? I mean, how much rates are they getting? And and what does that mean? Because they're designed to work in most clusters, which means they're probably overpromised up front. And if you really want to reign that in, there's gonna be a lot of, probably, work to be done there. Yeah. And I'm surprised when people are designing operators that that's the approach they're taking because it it it is complicated. And as you said, realistically, people want it to work in as many clusters as possible. Yep. Definitely. Alright. We have our new Kubernetes cluster. I'm assuming there's a teller involved here, so I'm

1:15:09 gonna jump into my client machine. Go back into client. Yep. And what we're gonna do here so oh, I should have checked. So what we're gonna do is we're we're going to first, we need to find the teller. So the first thing we need do is find Tiller. To do that, we're going to port scan the service port range. So by default, Kubernetes has a range of ports that assigns for services. So if you expose a service or a node port, it'll say these are the ports I'm gonna use, and it's 60,000 to 30 to to three two seven six

1:15:11 Attacking Scenario 5: Tiller No Auth (Nmap Scan)

1:15:50 seven. And if we do that, it should there we go. So in this case, it's gonna be on thirty nine three four. So another handy tip if you are security reviewing a Kubernetes cluster is scan that port range. So anyone who's put a node port service into the cluster, it'll turn up there. Bad node ports. Bad. Bad node port. Yeah. Bad node ports. But, again, I I like Convenient. Really convenient. Yeah. Really convenient. Really handy. And and one of the the the big things I've found with Kubernetes clusters is that people will deploy services inside them thinking that

1:16:16 Discussion: Node Port Risks

1:16:29 the network they're on is secure, and so they don't have to worry about, like, authentication on that service. So looking for node ports is always a good kind of point of attack. So now that we've got this, we can we can try talking to it. The primary in there is we can use Helm there's a vendor called Helm two, which I I I did because there's no Helm three as well. So I've Helm two binary, and we need to tell it where to go. So it's host and then 172 well, yeah. It's 1721804. The port we just got, which was 3935,

1:16:43 Attacking Scenario 5: Accessing Tiller via Helm Two CLI

1:17:05 I want to say. Instantly forgotten. Oh, it's 30 was it 30935? I think it was 3093 And then version is a good command to get it to to tell us. Let me check and see if that's what it is. 94. Oh, 94. So close. Still can't type it anyway. Okay. I'm do that. And that so that if you find if you find it, then that's that's a good way to go. Interestingly, if you're inside the cluster, so if you're, like, starting from I'm actually inside the cluster, you can also get it through DNS because all services get registered with DNS,

1:17:41 and Tiller is then on a predictable name. It's like, I think, it's tiller.Kubernetes no. .Kubernetessystem.service. You can just dig for that name, and if that name pops back, you know Tiller's there. You can always find tiller. There's no good way of hiding it unless you break DNS or something. Damn. So we've got that. If we do l s, so the same command with l s at the end, it'll Shady Helm chart here? There's a there are in so where is it? It's in charts. There is indeed a thing called Priv SSH chart. And am I just gonna install it? Yeah.

1:18:19 Attacking Scenario 5: Installing the Malicious Chart

1:18:19 If you just install that chart, the Priv SSH chart one. Name Priv SSH. It should done. Hey. So that's now installed essentially a privileged SSH server into the cluster. This is if you phoned Tiller on a on a a a cluster, that's that's it. You know? Because if he's got no authentication, that's what you can do. So what we're gonna do is it's on a port, 31846 there. Not local host. It's 172180431846. Yeah. Oh, I'm gonna get that PRS wrong? Oh, it's minus p. You have to put the if that's the h, you have to give

1:18:35 Attacking Scenario 5: SSHing to the Privileged Container

1:19:07 it the port as a minus p. Oh, of course. I always get that wrong. Every time. The s h always gets me. And the the password is really insecure, all lowercase. Because I I put this image up on the I put this image up there, and I was like, you know, I I wanna make really sure people understand this is not something they should ever use in a real life situation. So I said, let's make the password really insecure. Don't do this. And now we have a service account instead of a Kubernetes cluster. Now we're now we're actually on the node.

1:19:39 Attacking Scenario 5: Accessing the Host File System (CA Key)

1:19:39 So we're actually s c h onto the node. If you you look in the yeah. So if you look at we're actually on the the we haven't got the if you do share route slash host, actually. There you go. You're now completely on the node. That's the whole node. That's the API server node. Okay. So that that SSH is actually running a It's running a privileged container with all the host namespaces and everything else. So that's the full node. That's if that was a real cluster, you'd be on the on the a p on the master.

1:20:12 And there's our Sneaky. I thought you were just gonna get me inside of a poet and then maybe do some sneaky stuff. Yeah. But the fact that No. A teller could just go ahead and actually do that. Yeah. So that's if you think about how that worked, that was a combination of teller being there, Tiller being cluster admin, which is the default, and no pod security policies. So pod security policies would have made that harder at least, although, probably, cluster admin could bypass them if there was any pubs you know, PSPs that allow it. And it's as it yeah. We

1:20:16 Summary of Tiller Exploit Chain

1:20:48 did this is something that I did to real clusters because if you have till it was there, this was possible. Awesome. That was very, very cool, but also very, very scary. Just so many different I get I mean, the Kubernetes is a large distributed system. There are just so many different ways for someone who is either a hacker or a pen tester or even just a security advocate. There's so many different avenues there that you can just go and knock on a door and say, hello, are you here? Yeah. And and and there's new ones coming along,

1:21:02 Reflection and Future Security Considerations

1:21:23 and this is why, you know, one of the things you'd always do, and I definitely still do is each Kubernetes release, I go and read the release notes. And I say, what's coming this time? What's what's new? What's what's changing? And sometimes it's it's stuff that's hardened stuff down. Now I'll be going, well, I'll be ticking something off going, right, that's off my list because that's been closed. Another time, it'll be, oh, look. A new feature. What does that do? There there was a fun one in in 01/21. It was already in beta, but now it's

1:21:50 stable, which is you can use external helpers in KubeConfig files. So if you look at external authentication helper Oh, I saw that. Yeah. Yeah. If if someone gives you a kube config file and you don't trust them, read it because you can execute any binary off the person's regime when they run kube kettle. So as soon as I run kube kettle, it'll just call that binary and run it. Yeah. They use that for EKS and JKE authentication. Now it reaches out to the Google Cloud SDK command line and the AWS command line to make sure that you get an SDS token or

1:22:19 whatever. It it but but there's no, like, validation of what the binary is. So the binary can be anything. So so so the the rule the that was just a new one for me. Was like, I will always read every kubectl file I get before I use it from now on. Because if someone wants to be like, to make a bit fun, they could make it do all sorts of fun Well, yeah. They could curl their own endpoint on the Internet, pass it through your SSH private key or anything like that. Yeah. Easily. Yeah. Or or or

1:22:46 download a backdoor and run it. You know, just download a you know, curl something and and and and run. And away you go and that's their machine, you're like, oh. Awesome. So, yeah. I'm gonna I'm gonna have to play with the rest of those scenarios. It was just Yeah. It's too interesting and too much fun. So, you know, thank you for guiding me through the first four or five scenarios that we did there. Thank for even having that repository online and making that available to people to kind of play with. And I definitely I'm gonna encourage myself and

1:22:58 Conclusion and Call to Action

1:23:15 others to maybe subscribe and watch that repository and see what new stuff gets added with each Kubernetes release. Yeah. And and if anyone's got any ideas, they wanna see something, I'm I'm I'm open to issues or PRs. Yeah. Definitely. I'm gonna check out your kind tooling for the animal stuff. I think that's really cool for preparing other scenarios like this as well. Yeah. Alright. Well, thank you for joining me. Have a great day. Congratulations on your new role. I'm looking forward to us hopefully doing more stuff like this in the future. And maybe we'll even get to meet up when there's

1:23:37 Outro

1:23:45 no lockdown. I know. Could be a beer at some point. I hope so, mate. Alright. Thank you again. Have a wonderful evening, and I will speak to you soon. Cheers. Bye. Bye.