0:49 Introduction & Housekeeping

0:49 Hello, and welcome to today's episode of Rawkode Live. I'm your host, Rawkode. Today, we're taking a look at hacking Kubernetes, playing some fun capture the flags organized for us by our friends at Control Plane. Before we get started, there's just a little bit of housekeeping. So first, please subscribe to the YouTube channel and click the bell. This will get you alerts and notifications whenever we have new episodes of Rawkode Live. Next, we have a very active Discord server. Feel free to jump in there if you wanna chat cloud native, Kubernetes, and anything in between. And lastly, thank you to my employer, Equinix

1:21 Metal. They allow me to do this on their time and leverage their hardware when required. If you wanna check out a bare metal cloud, use the code Rawkode. This will get you $200 of compute, which is around 400 hours on one of our more modest instances. Alright. That is a housekeeping. Today, I am joined by Andrew Martin, the CEO of Control Plane. Hello. Hello. I just I've looked up and I've seen your big cheesy smile and I was like, I was like, I don't I won't say anymore. Do you wanna tell us a little bit about yourself? And then we'll get started

1:34 Guest Introduction & Background

1:54 on today's session. Sure. Yes. Hello. It's a delight to be here. I'm Andy. I'm alcohol free at this point in time. I, I promise. And, yeah. Hi. I'm the CEO at Control Plane. We are cloud native security, which basically means, we do DevOps, and we try to do it well because DevSecOps and that layer above is just solid engineering. So very proud to have a very smart bunch of colleagues, doing interesting things. We have plenty of positions open if people are interested, of course. And, yeah, my background is in low level engineering and distributed systems.

2:36 I came into security as a product of getting access to these systems and realizing that there was less than perfect coverage. Of course, a hundred percent is never possible. So I was just able to follow my interest and keep on going down rabbit holes and untangling balls of twine. So that's how we find ourselves here today. Awesome. Thank you for sharing. So what are you drinking? It looked like a beer. Oh. Yeah. I mean, I I'm almost carbon neutral, so I'm now trying to go alcohol neutral. If they would like to sponsor me, this is

3:12 the most crisp and beautiful of the unfiltered lagers. It is 0.5%. Nice. It's a couple of really nice alcohol free beers that I like actually. The Erdinger one is really, really nice. Mhmm. And the Brewdog ones are are particularly tasty too. Yeah. I Sorry. The the other silent magical beer is the the Jupiter. And for many years of drinking in Belgium, I've accidentally bought cases of non alcoholic beer before. So now at the other end of my existence, I'm happy to type it says actually really quite good as lagers go. Nice. So today, we're taking a look at some of the

3:53 Introduction to the KubeCon CTF

3:57 CTS scenarios that you and your team at ControlPlay and put together. Is it the KubeCon ones? Yes. Indeed. It is the scenarios from last week's epic KubeCon, frankly. The, the livestreams that we did, one of which you were a part of, were just fantastic. Really thoroughly enjoyed doing them, and we didn't have enough time to finish everything. So we will run through them all end to end. Awesome. Well, I actually didn't have a chance to partake in the CTF last week, except for what we did on the the livestream in between where we had a little play.

4:03 CTF Platform & Availability

4:32 It definitely gave me a wee taste for it. I like, oh, I should do more of this stuff. So I'm really excited that we actually get to sit down together and do this together too. So you have sent me some rather sketchy and dubious bash stuff to stick in my shell, which I did without blinking an eye. So there we go. We're gonna see how we get on today. I've got this link from the CTF. I mean, can people still do this on their own time, or is this something that they'd have to send you a little money for?

5:02 There are two versions, of course. We are open core for all of this. So there is an open source project called Kubernetes simulator, and that fundamentally is a provisioning engine, aka Terraform, of course, and a load of nefarious scripts that run through and misconfigure, I guess, security contexts and the definitions of what's running in a pod and various various other things that miscreants may or may not take advantage of. So that open source piece is used for the CTF. Yep. It is beholden upon me to patch the changes back into that, so that will be done next

5:44 week because I had to take a breather. Yes. And to actually run this at scale, we've built a load of, distributed orchestration around it. So we've run that for conferences. We've run that privately as well. But, centrally, all of the learning experience is available through the simulator, and all the scenarios that we ran for, KubeCon twenty twenty north sorry, twenty twenty EU, no. It was North America. It's continental confusion are available on the open source version. So everything will be there, but no one can cheat for the CTF so we don't publish them in advance. Got it. Perfect.

6:24 Setting up Scenario 1

6:24 Alright. So this is the dodgy stuff you sent me. Untars some configurations and SSHs me in to something. So I'm just gonna hit k s play, let it do its thing, and we are n a shell, either in a container, on a VM, and somewhere. Right? Is that is that all I need to know at this point in time? That is a correct observation. Each the the penultimate well, the last line of text there where your cursor is tells us what the starting position is. Actually, you're you're kind of right about the layers of abstraction,

6:30 CTF 1

7:05 but yeah, we only care that it's a pod right now. Alright. Okay. So I'm starting in a hashtag pod, which is in some namespace. It wants me to follow the captain and prove out his attack path to find a flag. Alright. Yes. There is this arch nemesis sort of archetypal sailor of the binary maelstrom, and all this other nonsensical rubbish, who is this character that I've constructed to write a book around, essentially. So that there's a book coming out in November called Hacking Kubernetes. And in order to give kind of threat modeling and attacking stuff more teeth and something

7:10 Scenario 1 Context (Captain Hashjack)

7:50 a bit more realistic, this guy, captain hashtag exists. You will see on the fourth lineup how much effort I put into the, kind of UTF eight spelling of the word Hashtag. And joy of all joys, it renders in our terminals. Truly, we are in the future. Yeah. So this guy, captain Hashjack, has done numerous nefarious things with his motley crew of scoundrels. And the point of the CTF is to chase down what he's done and understand how he's performed these attacks in order to understand for ourselves how best we secure the cluster, how we chop off those branches on

8:31 an attack tree, and how we model the threat model in an abstract sense and, of course, in the concrete sense of actually putting these controls in place. Okay. So should I just start pushing buttons or Mhmm. Yeah. Alright. Okay. So I do see a process table that makes me think we are in a container. Confirmed. This looks like a host disk. Alright. So it appeared to me that we can just mount the host disk. That is entirely correct. The the telltale signs here are the container runtime would normally mask dev and so not give us full access to

8:43 Scenario 1: Recon - Discovering Host Mount

9:36 everything that is available on the host. Because, of course, dev is a virtual kernel file system. It's provided because everything in Linux is a file. And in a container normally, we would not see lots of TTYs. If we know David's not a big fan of Unicode. Yeah. There was a custard episode where a fell Scotsman actually, Guy Templeton ruined broke the core DNS config by swapping out the old for a Unicode old that looked very much like it. Yeah. That's the the original typo squatting, I guess. Painful. Anyway. Yeah. Big time. Yeah. So so with normally some very restricted

10:15 CTF Strategy & Flag Format

10:17 but but but yes, you are you're bang on. This is exactly where we want to be. So I guess I mean, I've got access to the desk. I can just share it this. I'm assuming I've know Kubernetes. Yeah. I've got the rootster. That's always a good sign. I've got ability to speak to the cubelet or even probably modify no. Thank you. I was starting manifest. Alright. Okay. So am I supposed to be looking for a flag or am I just trying to take the roots there? It's like, I'm looking for something. Right? Yes. There are

11:02 flags littered across all the clusters. In this case, there's only one. Okay. The, there's a vital piece of information which I should, in the spirits of Fairplay, share with you, and that's that the flags have a well known format. So flag underscore c t f, open curly parentheses, a hex block, and then a close parenthesis. So it is possible if you have the time and inclination to find these flags. Some of them, it takes a a little bit longer to find. So sniffing around in some places is useful. I mean, I just check things that I

11:40 Scenario 1: Searching for Flag 1

11:50 would assume would probably be obvious. So what you'd like, I mean, I've never done like a real CTF. Right? So if I was partaking in this, are there any hints or advice that you would give to people? Like, what should actually be doing? Like, once you get access to the the root disk, other common places, paths, commands they should be running, things they should be looking out for? That's a really good question. The playing a CTF and avoiding or evading intrusion detection, are are quite different things. So if we were actually attacking a host system that we

12:32 were, we were red teaming, we had legitimate reason to be, to be attacking the system, of course, which is the disclaimer on on all of the the CTF stuff. In that case, we would probably be trying to tread quite carefully. So we wouldn't necessarily be grepping for stuff, although that is, frankly, the the best way to solve a CTF a lot of the time. We would try and perhaps, depending upon our level of, of defense evasion. I was just watching, we were doing a Sands call earlier, and there was a really great presentation on emerging threat actors,

13:12 and they popped up a SolarWinds slide. The SolarWinds slide, the the attack began getting on for a year before they actually dropped the Sunburst malware the Sunspot malware. So they were super stealthy. They were just doing one thing really cautiously at a time because, I mean, you don't wanna trigger those alarm systems. In a CTF, I haven't played a CTF with IDS. I think it's it's a great idea and watch out for next KubeCon CTF, I guess. But, yeah, in in this case, speaking personally, I go and look in places that, I would drop things myself, I guess.

13:55 The root directory is normally somewhere that an administrator if I've just got a Snowflake server and, as you and I can probably recall from the days before wide automation and DevOps being being a thing almost, we used to have to go on servers, and we used to have to move keys around. We used to generate our certificates from the server itself. And the slash root directory is a it's a place where root trusts they only have access. Yeah. So so something in here is likely to cause cause problems for a flag defender maybe. Alright. Well, I love that I could see

14:43 the kernel with AWS and I could set the metadata API. That would be fun. But I'm looking in this root thingy. And you said you wouldn't grab? Why would you not grab? In a CTF, I would. Alright. But on on a non excellent work, sir. Hate that I don't have my all complete. I know. Cdf dot star. Was there was something there. There we go. Very good. That is it. Oh, look. There's more more UTF eight. What a joy. Alright. So if you just if you grep for the flag again Yeah. It's in the authorized

15:07 Scenario 1: Flag Found & Analysis (Persistence)

15:38 keys fail, wasn't it? Yeah. You got it. That is bashing this thing. There we go. So the the flag is this nasty person has added SSH keys to the authorized keys, so they have access to the host going forward. That is exactly it. Yes. And this is one of the one of the ways to achieve persistence if you are just mounting the host file system. Because, of course, without using NS enter, we've only really technically charrooted or entered the the host's mount namespace. If you run a a PS again here, you'll see you'll see it's still in the

16:26 container. Oh, because of the bash. Yeah. Chirutz told you the Oh. The the magic, I think. Yeah. So so yeah. Precisely. So you you're able to navigate that file system and pull all the things from it. Mhmm. But but we're still yeah. Exactly. As you see, we're still in the process namespace of the container. So the only namespace we've managed to break into so far is, is the mount namespace, and that means to actually break out onto the host, we need to do one of a few things. The simplest is, well, let's say the simplest.

17:05 We can do what we've done here, which is to drop an author a public key into the host dot SSH authorized keys for root. If we wanted to use that attack route on a modern system, we'd also probably have to enable route login for for SSHD because that's not always enabled anymore, but we can do loads of things. Interacting with system d units is a bit more difficult, of course. Sorry to interrupt. I I was I was only gonna say we we have access to the etcetera kubernetes. We have access to the kubelet in the

17:39 static pod manifest. So we mean, we could run a privilege pod in there, which would get us access to the all the namespaces on the host. Yes. If we had n a center in that pod, then we could just blaze into all the namespaces. But because we've just because we're in a pod with that, slash dev access, which is normally because it's privileged anyway, That's the route we've taken. But, yeah, exactly as you say, we could just run another container. Mhmm. We could just drop in a shadow pod that has the tooling that we want to break

18:14 out onto the host. We could drop in a cron job, which fires off a reverse shell. We could mess with some of the system d units, although they might need a daemon reload and that would take longer. It's game over for multiple different games simultaneously. Nice. Super cool. Excellent work. Okay. Let us progress. Alright. Do I need do I need new stack new things? Or am I done with this shell? Yes. That shell is thoroughly popped and no longer very used to us. So here we go. And this highly advanced, tar g z based access provisioning mode is is how we run

19:00 CTF 2

19:04 the the KubeCon CTF as well. And all credit to the attendees, many of them did question what do we think we were doing, just throwing around bits of bash and tarballs, which is put it on a USB key and give it to them. Right? It's coming in by pigeon. Alright. So we're now in a second shell. I'm assuming like what, you know, the new people they're joining c test. Right? The first five commands you should run. I'm assuming are look at the processes, look at the main points, look at maybe wanna see if you've got an sophisticated

19:23 Scenario 2: Initial & Advanced Recon

19:48 network or no access to the network. Like, what would what would you run beyond that? Yeah. That that's awesome. I I have some useful collateral that I will share in a second. Alright. But but the first thing is just to exactly like you say, check the mount points because they bleed so much of the abstraction of a container. From there, it's worth checking. We can see in in the process table if if we are actually in a process namespace. Yeah. Precisely. So all good. The the network as well is super useful. We should be we should have a pod

20:27 well, rather, we should have a local Ethernet adapter that is on something that looks like a private range. It should probably also look like quite a small range because we assume it's a pod network. If it's not, then maybe we've got access to the host adapter. There's also a super useful, bit of kernel jiggery pokery that we can do. So because we're, I I I checked ID as well just to see, what, what we are. Okay. So if we now cat proc self status, then we can see some stuff in here that also looks a bit like what AMI

21:09 contained brings us. AMI contained is the canonical internal container observability tool written by Jess Fraz, who just shipped all the container security stuff right at the beginning. But we do have to pull a binary into our pod if we want to run am I contained. So proc self status will tell us quite a lot of things. We can see that we have no set comp enabled on the, like, seventh or eighth line up there. We can see some of the capabilities that we've got. Although broadly, I I don't decode those. And then if we go up,

21:47 there is, we can see the UID. Of course, we can see that through ID as well. And it's some interesting information. To take this to the next level, the AMI contained tool, which will run, in a in a later attack, will also tell us it will take a best guess at which, which container orchestrator we're in. So let's try and guess which orchestrator we're in. Well, that wouldn't be doing justice to clustered and its obvious spelling hints perhaps. But if we cat proc self c groups singular. Sorry. Then we can see that the name of

22:37 the c groups that have been created also give us a bit of a clue. And it's all this stuff around leaking abstractions that make containers at once super fast to spin up and beneficial and portable and useful, but then they're not a perfect abstraction. Okay. So is that the first the first thing to do is really just to gather as much information as you can before trying to move towards the flag. Right? Yeah. Exactly. Alright. I'm I'm gonna show my naivety here. I didn't see anything here to reveal maybe what CRI implementation we were using. I do

23:21 see system d and freezer. Does that mean we're using container d? That's a really good point, actually. If these are stood up I I mean, yes is the answer, but I'm not I'm not convinced that I'm not sure that queue pods changes for different run times. I haven't I I can't actually tell you that offhand. Certainly, if we were to run this via Docker, it would tell us that we're in Docker there. Right. Okay. So I I think you're you're definitely on the right path. Yeah. I've that's how I handle most things. I just say I don't know and guess. So

24:03 but I should just be prodding around. Right? This is I did see the mounts were It looks like we've maybe got access to the host again that way. Yeah. Maybe let's see. Oh no, we don't. Okay. That's how it should look. Alright. So what do we have then? Not a whole lot on this one. Is there I I don't know what system this is. Well, let's see if I can work that out. Maybe that'll help me. Not an LSB file normally. I'd like to go asterisk rel into asterisk. And that yeah. That's it. Oh, actually. Yeah.

25:00 Alright. Okay. But it's a bit different for different I mean, older OSs don't quite have it. So that star rel always tends to work. Nice. Yeah. I normally look for the l s p underscore release, but I guess star rail would cover that too. So handy. Alright. So Yeah. When you started off, you used the mount command. Mhmm. Which is super useful because it gives you all that extra context. And and if we go right to the top I say right to the top, just slightly further up, then it also leaks the overlay f s. Yeah. Exactly. Yeah. There's the docker

25:37 run time as well. Yeah. Okay. And and so so that gives us a bit more it's just something else that's leaking. However, this is super noisy because it tells us things about where c groups are mounted. So I always run both mount and also d f. And that tends to scream a bit louder when there is a service count. Yeah. We've got the secret in here. And it's kind of most of the time, people don't unmount the default service account, which is obviously not a great look. It also means that a service account may just be

26:05 Scenario 2: Checking Kubernetes Permissions

26:21 unable to do very much by default, let's say. Someone else who came to left me control in this container. Yes. Sometimes thinking of the user. Does that mean I can run oh, look at that. Bazinga. At least there's some RBAC going on. Oh, we got some crypto miners going on as well. That's fun. That is the badger. And the the context for this is that we've got unexpected workloads. Now notably, just some background with this one, the this is more about attacking Kubernetes' workloads and side effects rather than misconfigurations of the orchestrator. Doesn't really mean very much, does it?

26:27 Scenario 2 Context (Unexpected Workloads)

27:23 Okay. So I need to work out what my next move is. Right? Yes. I I can start poking around. I could start exactly into other pods looking for maybe better privileges like in the interest of time. Yeah. What should I do? Those pods should not be running and stealing our compute resource. Let's shoot them in the head. Oh, wait. Are we the good guys or the bad guys? I thought I was trying to get ripped. We're the good guys following the bad guys. And this is what the bad guys have done. Okay. There are a good number of routes targeted

27:25 Scenario 2: Goal & Strategy (Stop Miners)

28:11 flags, but this is not all of them. Okay. So we wanna delete some pods. I was gonna spin up some more, but whatever. You've you've avoided the mistake that is easy to make, which is we are in fact in the hashtag pod. Yes. Because my my first instinct was to do a delete pod dash dash all and I thought, no, I'm in this namespace. So although now that I've just realized I've deleted pods, which is probably rather silly because they are tuples, which would suggest there may be something else get deployed. Deployment. I don't have access to that.

28:16 Scenario 2: Deleting Workloads

28:59 But if you check the pods Yeah. They are coming back. So not stateful sets. It could be daemon sets. I don't have access to that. You're very much in the right direction. Right. Okay. So my my goal is to get rid of all these pods. Yeah. And between you and me, they are demon sets. No. They're not. That's instinctive. I'm so sorry. No. They're deployments. Alright. But I don't have the access to list the deployments on my current privileges. But you could probably elucidate their names. Oh, so I just don't have list access. Yeah. That's right.

29:50 Oh, look at that. Alright. Incidentally, I'm looking to launch all of those coins. Really, I should have been a bit smarter and just says, auth can I list? And I would have seen that I can delete very nicely left as a hint there. I was just being rather silly, but there we go. Got it. So that should mean they're all gone at least now. Getting there. Is that it? Is it done? Almost. Almost. At the point that they make themselves scarce, something will probably turn up in secrets. Oh, there was a hashtag secret. I looked

30:15 Scenario 2: Workloads Stopped & Flag Found

30:36 at that earlier. Oh, there's a flag. Alright. Okay. I think, potentially, this is managed by the world's oh, gonna make a big claim. The world's shortest operator, perhaps? I found my flag. Oh, the one I missed there. There you go. Excellent. Excellent work. Okay. Let's keep on rolling. Here is your next scenario. So in this scenario, we have we're running internal container registries, and the pirates have turned up, taken the registry down, and left a secret in one of the pods. So this is a this is a kind of screaming backflip through a flaming hoop kind of

31:29 Scenario 3: Initial Recon & Permissions

31:35 exploit, where, again, we're just using what Kubernetes gives us in an unusual way. So is it hacking? Well, of of a sorts. Okay. I don't wanna run that, but echo hashtag. Okay. Is that important? Should I be looking at that or is that just there for the is that an Easter egg? Yes. It is not important for It's not important. I was hoping you were gonna wreck Rawkode me, but Just you wait, perhaps. Alright. So I can see Docker CRI again. We have our overlay. We've got our secrets. We've got some disks. We have a service account.

32:42 No IP command. Alright. So Do we have control? We do. People are always nice leaving all these tools lying around. It was a yeah. Feedback from last time perhaps. So they have to curl down the binary in every single pod. Alright. Let's see what I have access to here. Well, we can create deployments. We can get logs, but we can get pods too. Okay. Let's see what else we have. Namespaced, private tier one, two and three. We do not have the ability to exec into another pod. So I'm assuming we're create do you want me to create something? Is

33:40 Scenario 3: Strategy (Run Compromised Image)

33:52 that why I have the ability to create create deployments? Yeah. That's it. There's the interesting thing here is the contents of the image. We can't get to that image, but we can run a pod using that image as the base. So it's then about and this is the, the screaming backflip. It's then about trying to it it's almost a blind injection attack. It's not quite it is blind because we can't see the file system contents. It's kind of an injection, but we are still just running kube control. So that's perhaps a overly dramatic wording. But if we

34:30 if we have a look at what's running in those private ear pods, so dumping them as as YAML and grepping for image. They are running Excalibur. That is the badger. So the goal here is to run a pod using that image. Do you maintain your own registry? This is the the secret internal registry for the purpose of this. Alright. The answer is absolutely not. Sorry. I cut you off. Okay. So No. No. It's all good. Right. Let's push me along a little bit here. I'm not sure what my next move is. So what we want to try is cube

35:14 Scenario 3: Executing Commands Blindly via Logs

35:26 control run and set the image to be that crazy, practical image. And then yes. Nice. We want do we want a terminal? I'm not sure. Maybe just try double double hyphen. I'm not sure if we yeah. Okay. And then I don't know if IT will work on the run, but I figured I'd try it. It's it's worth trying. Yeah. What's there? Yeah. I can't end because I don't have blood exec. Checking a thought was gonna happen. So the way that we can do this semi blind maybe it's kind of monocalled half injection is to check the log outputs.

36:30 And the the the tool that I really love for logging is called Stern from, a company called Worker that Oracle bought, like, three years ago. But this tool Stern, I'll drop a link in, allows us to essentially specify a wildcard that lets us grep across any existing or future pods in any or all namespaces. We can't do that right now. We'll just have to use cube control logs. Well, we could, but let's not bother. And okay. So where are we? I have to admit missing the previous command. Sorry. I was running the logs against our

37:15 hashtag too, but I don't seem to be getting anything back out. And then I just decided to check the logs of private tier one, two, three, but let's just sleep. Oh, yeah. It's going round in circles. Okay. Let's have a look at this injection command. Could you pull up the cube control run again, please? Yep. Yeah. This. Okay. So if we replace bash if if I could get bash let me think about this. If if you can remove the IT as well, because I don't think we need an interactive terminal for what we'll do, and then double hyphen.

37:56 And then we'll we'll run bash again, bash space hyphen c, and then just put an ID in there and see what happens. Alright. Okay. Gotcha. And we've got a fun game here because we need a unique Yeah. Pod name, of course, every time we do an injection. So, yes, apologies. This is where Stern is useful. Alright. Rip, Rip, and Rip. Cool. Okay. So Noel has very helpfully posted, another useful GREP. So if we replace ID with the GREP r flag CTF search, then we will find and I think, again, for the purposes of of speed searching in user share.

38:54 So up to, please. That's the badger. And then, just Noel's last, last chat in the that's the badger again. And then if we use user share as instead of the root slash. That looks good to me. Joyful joys. And we have a flag. Very nice indeed. Interesting points to note here. The rickroll will be from a different line of the song every time. If you actually and why under god's bright sun would anybody ever create a directory name, which is a comma? Who who only knows? But, yes, congratulations. We are rocking through the flags. On we go to the next one.

40:21 Setting up Scenario 4 & Cluster Stability

40:21 There we go. Thank you. And we are in number four. Very nice. Now it occurs to me because of my exceptional timing, I may, this cluster may self destruct around this. Can you just check? This is an interesting experiment. What does uptime say? 55. So this is this will go in a a blaze of glory very shortly. There's another one coming up, so we'll be able to transition between clusters semi seamlessly. But let's You can send me the my service account again. I'm assuming we're gonna be following the same pattern here. Let's just go straight in there.

41:03 Scenario 4: Initial Recon

41:24 Oh, that's ripped. Yeah. Loved into a false sense of insecurity, I'm afraid. Alright. The the back chat for this one is the supply chain is compromised, and code has been merged into an application library that developers use. The library runs in the in a pod. Attackers have used that reverse shell that the pod threw to jump into the pod and find secrets mounted on the host. So somehow, we need to get out of this pod and, and into a mount point. Okay. Cool. And we have four minutes. And I and I have to say, I, again,

41:37 Scenario 4 Context (Shared Process Namespace)

42:33 lulled you somewhere because if you go back to the PS at the top now earlier on, we both jumped into the same pod, and that meant that we had two bash sessions the two bash processes running. Actually, what we've got here with two sleep sessions, processes is potentially an indicator of misconfiguration. Okay. Oh, there we go. Oops. See you later. Okay. What should we do? We will have the next one up very shortly. And if you just okay. Yes. So so just looking at that. We've the the first thing to do, please, is to scroll back up to the, description.

43:15 Scenario 4: Multiple Containers Hint

43:23 And and then just slightly further down. Now just before we've typed uptime, it says use kube control, describe, blah blah blah to see all of the containers in this pod. And the line above that, defaulting container name to audit, that comes from kubectl exec. That is not part of the scenario description. And probably in the spirit of fairness and goodwill, I should put a dividing line between the two so that's clear. Right. But concretely, what that means is there's more than one container in the pod. KubeControl used to not tell you this information when, when you jumped in or or give

44:07 you less information. But now with kind of one two one Yeah. Yeah. That's that does you can set the default container now. Right? Yeah. So it and we go toward it. But, actually, the, the container that we're looking for is in the same pod and somehow accessible. And that's, suitably vague, not be of any further use without a cluster as well. I don't know if we've got any more. So if I just send you one that we've been let's, let's reverse back to the last cluster. We only need another couple of minutes before this comes back up again,

44:28 Cluster Setup Issues

44:54 but at least we'll have something to poke around that mind you saying that, they were probably all spun up at the same time. This is where I miss Lewis. He never makes these kind of rudimentary mistakes. Alright. So I'm deleting the old cluster, and you want me to redownload the previous cluster, which you've just sent me. I'll do that now. Although we suspect that that one may also go kaboom. Yes. Which is nice because I'm actually I'm wearing my Marvin the Martian t shirt today. So things gone kaboom is encouraged. It's only up ninety minutes. There we go.

45:33 Well, that's is that what you expected? Yes. Because I was better at the beginning of this video than I am at the end. So this might be an interesting time to pull am I contained? Fortunately, Jess does an awesome job on the releases page. Genuine tools, am I contained? That's the badger. And if you go straight to releases, she gives you a checksum validation. I mean, this is how software should be delivered. Right? Everybody who is running code cove and not checking and when I say everybody, everybody, even up to HashiCorp, the big boys. So

45:37 Security Tools & Concepts Discussion

46:14 we've got a permission denied. This is totally standard in containers. We often can't hit the file system locations that we want, but what we can do is yeah. You got it. We can write to temp, and that's we can we can grep the mount points for the string RW, and that will just tell us if we've got a mount if we've a writable mount. As long as we don't have no dev Oh, wait. I was trying to download. Oh, there we go. Let's Yeah. Repeat that. And the before as well Yep. Twice. So this kind of cat and mouse game

46:55 of immutability on partitions is really not solvable. Applications need to drop PID files, lock files. They probably need temporary disk. You've probably got access to dev SHM, shared memory segment. Really, even GKE, which is super hardened and well configured by default, you can still find places to write runnable files. It becomes a lot easier to IDS detect the thing, but boom. Okay. Awesome. So if we remember what we had in self status, we've got a a lot more visibility here. We can see has namespaces pid? Yes. That's a good thing. Username spaces, no. Hardly anybody

47:36 has username spaces right now because they're very complex. They cut across a lot of other namespaces because everything is a file, and every file must have, discretionary access control, users and groups, and blah blah blah. Doing a lot of work to bring usernamespaces to Flatcar, which is the spiritual successor of container Linux or CoreOS as it used to be. That will be super awesome when they ship that. What else do we have here? AppArmor profile, docker default. You beauty. Very rarely is AppArmor on by default in Kubernetes, so this is a very happy day. And we can see our bounding sets of

48:16 capabilities. This will give us an indication whether we're privileged to start off with, but but also tell us things like we've got deck overrides. That means we can change permissions on files in the same way we can with churn. We can make nodes. We can do routes. We we've got a lot going on in there. And then yep. Sorry. No. No. I was just agreeing. Yeah. Stuff. Thanks. App Armour, SC Linux, love all that stuff. It's so easy to operate these days in 2021. Tell me about it. And this is where the vendors make the money.

48:53 Yeah. There's there are loads of things that I I know are just good practice, but then, you know, it's hard. Big time. Yeah. There's there there are tools that will extract runtime security profiles from running applications. But unless those applications exercise the full sets of production runtime behaviors, they're not gonna build the right security profile. So there's this mysterious gray area wherever a security operates, I suppose. And often, there there's some awesome tooling that's come out to support this, actually. The, Aqua Tracy does some of this stuff. Red Red Hat have a, but they have multiple things. They have,

49:48 a man, I'm just I have tool override. So there there's a there there's a tool that will use eBPF to trace system calls to generate profiles. And Oh, nice. And so could maybe do bits of that as well if you got that running on your cluster. It's absolutely the same space. Yeah. I guess Falca reverse engineer. Yeah. And seccomp, I think, you know, with the seccomp operator and all the work that Dan and Sasha are doing there, it's it's getting a lot easier to run seccomp on Kubernetes as well, which is which is nice. I just wish people would do that for

50:23 AppArmor and SE Linux. Otherwise, I I'd maybe stop disabling them. Dan wolf's cries every time someone disables extract SELinux profiles from deny events, similar to Audit to RBAC that Jordan Leggett built. I just can't remember what it's called. I'm sure someone will find it. I think Noel has come in. Is it Bain? Yeah. But Bain's great, but it's actually a a preprocessor for, for AppArmor, I think. He says, not having used it like that. So perhaps Oh, I think I missed his first comment. So I think that's also from Jess Rizzell, and it's a tool for dynamically

51:00 CTF 4

51:06 generated tech comp profiles for a binary, which is nice. Or unless I've just stashed two random comments together that aren't related. It's it's in the right direction. Bain focuses on AppArmor, and it's less of a reverse engineer. It's more of a a DSL. So you can just so you don't have to write AppArmor, basically. You can write kind of higher order language and and have it compiled down. But, yeah, I I mean, the this is where container security vendors the next cluster's with you, by the way. This is where container security vendors have been operating for the past,

51:44 like, five, seven what year is it? Eight years. And another great I'm just dropping stats that I heard in a call earlier. Security operation centers that use open source tooling have a markedly higher talent retention rates. People like open source. Yeah. True. It's awesome. Alright. So we're getting close to the hour, and I know your calendar is always a nightmare. So I don't want us to go over that hour. So why don't you guide us through this one if we can do it in the next five minutes and then we'll do a quick wrap up and then

52:12 Returning to Scenario 4: Shared Process Namespace

52:25 maybe I can schedule more of your time in the future to do a bit more CTFE fun. Absolutely. SE Linux audit to allow is the one. Good spot. We'll lead. Okay. So what are we doing here? We are So you've told me that we have a second container and this pod and we need to find a way to get access to the other container. Right? Yes. That is entirely correct. No. I know that pods share the net namespace but I don't seem to see any way to confirm that, I guess. So yeah. I'll You're totally in the right kind of area. Right.

53:08 Okay. Go for it. Because we've got those two sleep pods, we want to just have a poke around. And while PS shows us that we are in a process namespace, the question is who else is in our process namespace? By default, Kubernetes containers in a pod are in the same network namespace, the same interprocess namespace, but not the same mount namespace, and not the same, and they have their own container file systems, of course, and not the same process namespace. In this case, we have a shared process namespace between containers and a pod. So if we go cat proc

53:47 12 because 12 is the pit of the other sleep process. And we could probably have found that by just doing an LS and seeing that we have a few numbers here. So is that how you would have done it? Yes. It should always correlate to It would have shown me it here. Right? You got it. See, sometimes I take the hard road. I don't think Right. Okay. Sorry. We're in prog 12. Cool. Okay. So what we can do from here is see the root file system of the other process. So if we let's start with cat and

53:52 Scenario 4: Accessing Other Container's Filesystem via /proc

54:24 go into root, please. Just yeah. Root within this directory. Alright. And then just start auto completing. So we've now got the root file system of the other container because the PID that is running in the other process namespace is now leaking back to us as well. This is by design. It's What? It's necessary. It's because we've shared the process namespace between the pods the containers in a pod. So, I mean, I would say I'm relatively familiar with the product namespace, and I had no idea you could navigate the root file system for each process in this way.

54:59 Scenario 4: Finding the Flag

55:08 Okay. Normally, this is a highly I mean, we we're roots, so we we have access to privileged system stuff. And from here, where do we want to go? This one, you may want to look in temp. So in the temp directory of the other container, and then into temp again, and then into logs again. You might just want to also complete for a bit. Oh. I should have just said no. Read them all out to me one by one. Yeah. I I feel bad that I've used the same, file name twice. The directory name rather.

56:05 There we go. Excellent. Fantastic. Yet there are two more, and there is a, a mysterious bounty flag hidden later. So let's let's cut it off there and find the time to get the others through and then maybe run through some of last year's as well. Alright. Well, thank you very much. Like, I'm learning scary stuff, which is always a wonderful thing to be doing. Like, that route director and the prog thing. I I took that that's wild. I I don't know why that's a thing, but I'm gonna have I'm gonna I'm gonna have to look look that up for sure.

56:25 Wrap up Discussion

56:45 It's a thing because when the kernel and I'm I'm gonna butcher this, and please, colonel devs, correct me. But when the kernel is managing processes, it wants all of that information so that it can do its own introspection. They're just SIM links to places. Everything in Linux is a file, so it makes that all available through the proc virtual file system. When we're root inside the container, and here's the you know, just say this sort of blur in the face, we have access to all those privileged operations. And so root inside the container can do

57:17 an awful lot because we're still just using Linux, and that's why we want to elevate to a non root user. And would that be something that disappears with user namespaces? At least user namespaces perform oh, man. It's a mouthful. Subordinate group and user mapping. So where we've got kind of we used to have one to 64 k. We've now got one to 2,000,000,000, possible user IDs. There's so many because you can then so you've got your root user namespace. You can then create another user namespace on top, and that maps zero inside that namespace to 50,000

57:25 User Namespaces & Mitigation

57:58 on the host. And then you can do that again, and you can do it to a point of brain melting complexity that we won't think about. So does it go away with these namespaces? Some of the privileged host mounting stuff does go away because we need that permission on the in the the root's username space. What we did just now oh, that's by design in inverted commas. Cool. Alright. Well, I'll definitely be losing sleep and turning off all of my production infrastructure tonight. You did a great job. Alright. Well, again, thank you for taking some time out of your

58:25 Final Remarks

58:32 day and walking us through this today. It was great fun. I definitely am gonna get more involved with CTS, I think, and just use them as a learning experience. I think this is a really fun and engaging way to learn really lower level components of the kernel and containers that I've probably just been neglecting for a while. So, Andy, it's always a pleasure, but thanks again. Have a great evening and I will speak to you soon. Cheers. Totally enjoyed.