1:44 Introduction & Sponsors

1:44 Hello, and welcome back to the Rawkode Academy. This is Custard. Today, we have two amazing teams who have very kindly or badly broken some Kubernetes clusters for the other team to fix. Before we introduce our first team, we are going to say hello and thank you to our sponsors. So Teleport have been sponsoring Clustered since almost the very beginning, and we've been using Teleport since the very first episode. Teleport is an amazing tool, allows us to commoditize access to each of these bare metal clusters. The fan grained RBAC controls and a pairing ability, which you will see us using on

2:25 every single episode of Clustered. So thank you Teleport. Also, thank you to Equinix Metal. They've been providing the hardware again, since the very first episode. These bare metal machines are awesome. They have tons of cores, tons of memory, tons of fun to be had for people that wish to break and fix clusters. So thank you, Equinix. It's always a pleasure. All right. Let's introduce our first team. I'm gonna pop over to here and we're gonna say hello to team ambassador labs. Hello all. How are you? Very good. Thanks, Dave. Very good. Awesome. I love to I do this every time and

2:54 Team Ambassador Labs Introductions

3:00 I haven't found a way better to do it. Like, I say hello to a group of people and then they all just like, oh, which one who's gonna speak? Like However, I'm really excited to to have you all here. And why don't we start at the top right, work our way around clockwise, and each of you to say hello and introduce yourself, please. Hello, everybody. My name is Alex. I'm a software developer at Ambassador Labs, and I'm based out of Montreal. Cool. Hi, everyone. My name is. I work as a developer advocate at Ambassador Labs, and I'm based in Lagos, Nigeria.

3:40 Hey, everyone. Daniel Bryan here, head of DevRel Ambassador Labs. Previous life as a Java developer, and then moving across the ops. So I'm hoping my Linux knowledge comes flooding back today. Right? It will. It's alright. I'm sure nobody ever attacks the Linux machine. Right? They just do a nice easy breaks on the Kubernetes cluster. That's what I'm saying. Alright. Well, you are up against a returning team, team fair winds. We will see them in around forty five minutes time. But let's get started with our cluster for today. So we are gonna pop open my screen share.

4:10 Start of Team 1 Challenge (Ambassador Labs vs Fairwinds' Break)

4:17 I've already connected to the control plane node. So if you could each please join the session and just type echo hello or anything to let me know that you are there. And then I will wish you best of luck and you can kick this off. All right. We got a hello, two hellos. Let's see if we can get there we go. That's there's London. Awesome. Off the flying star. That's that's the hard bit done. So I'll let you just take it from here. Set up your cube config, check for a control plane and best of luck.

4:53 Initial Cluster Diagnosis (`get nodes`, `get pods`)

4:53 Awesome stuff. Shall I drive folks, first off? Then you can shout out stuff. Yes. Please go ahead. Cool. How half the Fairwinds folks been we love Fairwinds. Have they been nice to us? Right? Let's see. We'll find out soon. Notes. Oh. Oh, we have start. For a control plane. Well, supposedly. I know they've done some tricks in the past. I do get notes. I guess I get pods. It's looking pretty good. Right? Yeah. It looks like everything is running as well. Yeah. Restart on the controller manager. Interesting. Yeah. Conveniently, like, an an a day ago, right, when you would have been playing

5:34 around with stuff? Let's Okay. David, can we see if v one is running, please? It's then crash look back off. So if you take that for less, I've I've screwed up as well for the audience, but the cluster pod is is a broken pod. Ah, thanks. That was great. Wishful thinking, You know? Yeah. Yes. Worth a try. Right? Shall we Yeah. Describe on that one? Yeah. So that's where the problem is. Sure. Get the events. Yeah. It seems every other one is running aside from that. So we got waiting, crash loop, back off, terminated, oom killed.

6:14 Identifying OOM Killed and Memory Limits

6:15 Java developer. I know that all too well. Spinning up Spring. I love Spring, of course, but often gets him killed. Look at the limits on this pod. He has 10 megabytes. Well spotted, Alex. Well spotted. Yeah. Back off, reset. Liveness probe failed health as well. What's the recommendation, everyone? What do you think? Let's try and bump up the memory limit. Yep. What's the command on that one? That's feel free to type if want it as well. I don't recall by hand. I'd probably do a kubectl edit. Yeah. Sounds good. Right? Should do that. Yeah. Edit deployment. I think that's your best

7:01 Fixing Memory Limits

7:03 bet here. So what you're in prod. Right? Yeah. The deployment then. What's it called? Cluster. Right? It's just called cluster dash. Right? Yeah. And I will show this through here. It's a weird bug. There we go. Oh, yes. Can you demonstrate to my screen? Oh, gotcha. Got the checks there. There's the limits. Right? So I added them? Yeah. What what are we going for, do you reckon? I think we should remove everything. Exactly. Sledgehammer approach. We don't need limits. Come on. Yeah. And yeah. Happy with that? Yep. That's not good. There you go. Resources column.

8:08 Supposedly running there. Right? It is. It does appear to be running. Do I love that Rust application instantly fast fire instantly fires out. Right? Would you like me to test it? Yes, please. There it is, please. Alright. That would be we're we're clearly expecting it tonight. It's gonna go in flames. Right? The Fairwinds people are far too devious. So normally oh, there's my LinkedIn. Normally, there is a node port service in 30,000. But can you just double check that for me and run out a get service? Run out what side, dude? I'm sorry. Get services. I just wanted to

8:23 Testing Application Connectivity (Initial Failure)

8:54 make sure that Yeah. So I would have expected that to work. I will check one more thing just in case it's my setup because I am also prone to a lot of errors. I'll get the public IP address. Oh, it's not there. Oh, yeah. Too many clusters. So just in case. They left you a message. Gotcha. It says gotcha. I was just gonna do a carol, right, on the Oh, yeah. We could've just done a carol look was, but I'm jumping around looking at it here. So No worries. And that's why I was just like, just talking. I lost

9:30 Discovering the "Gotcha" Message & First Hint

9:44 connection. No. It's back. Here we go. Yep. Yeah. I was gonna do a call local host. Yeah. Freaking Fairwinds Insight can help you with your done team. Bet you save your resources. Actually, on that note, have we got we have got some hints if we need them. Right? Thank you, Fairwinds folks. I believe it's hints there if we we need them. Cool. Well, what are you thinking now? Should we check for network policies? Anything like that? Yeah. Because, obviously, something is answering our request, but not the application that we're expecting. So maybe you look at the the ingress configuration

10:11 Investigating Networking (Ingress, Network Policies)

10:29 or the So we couldn't find node port, Alex, aren't we? So no ingress. Although there is m s s s ingress deployed to the cluster. Just throwing that out there. Emissary ingress is a nice ingress. Who works on that? We love emissary ingress. Right? Yeah. And I I one of the first things I did when I saw emissary ingress there is I was like, what what listeners are set up? What mapping is set up in my cluster and of course didn't find any. And then I figured out you're using no port to expose these things. Yeah. I've

10:56 been improving the setup over the last month, and I just brought in ingress. So, yeah, I will be exposing over some shiny domain Know where we are, David. I can help with those listeners host some mappings. Emitter ingress is one of the if not the best ingress out there. Right? Cool. So we've got yeah. So it doesn't look like emissary is being used in terms of mappings or even probably it's worth just checking listeners because that would be very cruel to leave it. Yeah. Nothing I suppose I should do. So looks like it's not a

11:27 panemissary ingress thing. No. But, I mean, you're having an application that looks like my application. I mean, what should we be checking next? Yeah. The image, I think, is a good good idea. Right? Yeah. Definitely. Checking the image. So if we just do we've already done the describe, actually, on the on the deploy. Right? What's the line? On the pod, not on the deployment. Yeah. If I just keep what do think I should do on the pod or on the deployment? Well, we did it on the pod initially. We can take a look at the deployment

11:38 Checking Application Image

12:13 as well. Because it says, right, image g hcr.i o Rawkode clustered v one. So the image looks legit. Right? That's what we expected when we were messing up with our cluster. Mhmm. Yeah. Screw to it. I like it when things look legit. Yeah. You know there's something we could try bumping it to v two. Right? Because that the ultimate goal here is to try and get to v two. If you see what happens there, I reckon the Fairwinds team might have left us a nice little surprise there as well. Well, that is the that is

12:36 Attempting to Deploy V2

12:44 the voice of a experienced cluster or as well, and you've not been on before. Yeah. Let's give it a try. Right? Yeah. That's a good idea. Edit. Deploy. Plastic. Well, It is running. It's restarted. If I just do a local host again, see what we've got there. Uh-huh. It's gonna be alright. Boiled. Yeah. So do we do we think potentially someone's messed around with the registries? Actually pulling them like a Are you can you describe the I would doubt it. Walking into I heard it's always full policy on the deployment. So I would imagine that we are getting the

13:16 V2 Still Shows "Gotcha"

13:50 image from the registry, and it's not preloaded on the machine. Oh, yeah. Image full policy always. Yeah. I'm with you, Alex. If you describe the part as the image the same? You get. Kubectl. My pod. Pulling image? Yeah. But it it does appear to be pushed down correctly. So Yeah. It looks like it's bit That's fun. What if we do big on g h c r? See if that's Do you know the IP address of that off the top of your head? I do not, but I was thinking if it was, like, loop back or something, it'd

14:50 be crazy, like That looks pretty legit, actually. But I'm just thinking, where's that going to? Because that that would be the just thinking, like, to your point logically here, Alex, like, is something intercepting that port before it's even getting here? Or is it the image? What do you think of the junk? Like, I wonder if I'm we might be getting led astray. Right? I mean, I I know we're actually talking about, like, network policy. Right? And then you mentioned there was an existing note, so there's no way to track that. Because ideally, that could be the problem. Maybe they set

15:31 some commands that prevents us from accessing it directly. So Like, for the people watching then, right, if you had a NodePort service, can you follow that thread? Like, what endpoints does it have? Does the pod IPs match up? Like, you you could kind of work that out. Right? Yeah. Interesting. What's the perk folks' thoughts on that? So the Let's look at the the service node port and probably the endpoints being served by this for this service. Do you wanna drive, Alex? My Internet is a bit flaky, so I think I'll I'll You're shout shout what to type in that

16:06 Checking Service Endpoints (Matches Pod IP)

16:25 case then. Let's start by listing all services. Yep. I like your thinking. So we have a note for on the clustered service. Yep. Mhmm. Let's look at the kubectl get endpoints for this clustered service. You've got a cuddle get endpoints? Yep. K. We can just run that as is if you wish. And now you've got 8080? Yeah. Well, you've got a port you've got an IP address. You may wanna Then we can, yeah, get pods dash o wide to get the pod IPs. Yep. You have a match. It does match. So that rules out the hypothesis.

17:37 Nice thought, did it. Nice thought. It's a cool effect, this. Image compose it always. Image name appears to be correct down the stack, but the image Is it car port? It's all that, I wonder. Sidecar container, sorry. Didn't see anything, did we? No. It it I I don't believe so. It says one of one ready, so one container. Couldn't I couldn't do anything clever with the in it containers or I know the Fairwinds folks have used all the Kubernetes real estate in the past. So let's assume that the request is set in the correct pod, but somehow the Fairwinds team have put

18:22 something in place that well, Kubernetes believes this is the correct image, that they're actually doing sneaky sneaky stuff. So what's responsible for pulling the image to the machine? The Docker daemon or Yep. Docker container d c r I and the kubelet has some involvement. So maybe start I may have to start poking at those, I think. How was your Didi Dong, Alex, what's your experience like with the with the cryo stuff? The the Fairbench team are also reminding us we have hints. It's alright to use the hints. Thank Kevin's team. Appreciate it. Sweet sweet Yeah. Cat them with that. Good idea. So

18:31 Discussion on Image Source & Lower Level Issues

19:11 we got some chat from the comments saying mutating webhook. Now, it doesn't feel like mutating webhooks because what we're seeing in the state and the edit pod would be different. Right? If they had mutated the the payload. Yeah. Yeah. So it's a good you know, they're they're sort of Oh, people do love mutating webhooks on clusters. I do gotta say. And I got one, I think. It doesn't have to take a look at Xeno, but I think you probably fixed the error rate. Oh, look at that. Some ASCII art. What's that say? Fairwinds versus ambassador's ass.

19:48 ASCII Art Message Found

19:55 I don't know if you get my Yeah. Unfortunately. Okay. Quite mean. Ambassador, I think. Right? I'm gonna Let's out. There we go. Oh, nice. Nice. That is nice. That is nice. I like that. Very cool, actually. Yeah. Kudos for that one. I mean, not a hint. Just I mean, if we're sticking to the rules, I'm just saying. To be fair, think your rules do say zero one is the first hint, David. Little bit like Built in system namespaces can hide all sorts of junk. Built in. So, like, static static pods. Yeah. Like, those namespaces that come directly with

20:17 Hint 1: System Namespaces

20:33 Kubernetes. Yeah. I wonder if we do l s e t c Kubernetes manifests. We're gonna see who we expect. Right? API server controller, keep the case of that VIP. But I wonder if each of them have got do you think I'm gonna rabbit hole here? I'm wondering if each of them have got what we expect. We could cap them. I'm also could dim them and have a look. Or is there an easier way to go in system namespace? It's gonna have all sorts of I mean, you could trust the timestamp. Running in the cube system names.

21:04 Sorry. Sorry. Sorry. Folks, what was that? Yep. I think you mentioned something around system. Right? So, like, you still get namespaces and then find try to, like, get things in that namespaces, see what exists inside of it. Because it it seems like they hit something inside the that particular namespace, if I'm not mistaken. I think you're right, Eddie. Yeah. Just look to David's point. That's a suspicious March 2, yeah, yesterday on coop controller manager. Right? Yeah. Exactly. That's a good hint, I think, there. We just go with him. Right? Interesting. So that one's definitely for folks watching, should contain

21:12 Exploring System Namespaces (`kube-system`, `kube-public`)

21:54 some YAML. That says did you put the path in or just the fail in him? Oh, that's cheers, dude. Yeah. I've you're right. I just my my badge will be oh, yeah. Just do ahead of myself. Right? Fox boys. Great work. That's more like it. Right? Yeah. That that's what you expect. Shall we actually, I'm trying think what's the what's best to do here? It go go to the top? I mean, when I open a fail and my customer doesn't start in lane one, I would maybe look at what it was around. You what? So David? When I opened a

22:37 file of them and my cursor is 200 lines down, I would probably check there first rather than go to the top of the file. Hundred? That's that's a good question. You're right at the end of the I was it's the driving mentality. I'm so in in the terminal. Yeah. Was it right at the end? That is Oh, it was. Okay. Yeah. Yeah. I think they might have thought that. Can I use control f and b on this one? B. There we go. Chat if anything looks suspicious, folks. Yep. Okay. That part, is that is that supposed to

23:41 be healthy? I'm not sure. Yeah. A lot of them are healthy. There's a good point, though. Some of them are health some of them are healthy. Right? So, yeah, you you're right. Oh, okay. There's little bit of suspicion, but I think I know the XCD ones are health, for example, versus healthy. Yeah. If you I think you'll find if it comes from a Google developer, they've got a concept of z pages, then you'll see health z. Oh, okay. Okay. Actually, David, I wondered where that rule came from. Yeah. It looks alright. Yeah? It does. It does.

24:16 Yeah. Any thoughts, team? Any thoughts? Because it does it looks pretty legit. Right? Yeah. It's good. Is it It does look good, Harry. Yeah. Sorry, Alex. Well, suppose it is. It's probably an ad, man. Now it does look good to me. There there was one thing that covers you, but I I don't wanna send you down any rabbit holes. So Oh. But time to speak like, something Didion mentioned, you're about looking at namespaces, Didion. Right? We try that. Yeah. Let's run the system. Namespaces. So, like, what we're thinking? Kube CTL? Gets namespaces. So just yeah. So

25:05 maybe, like, Kube CTL gets all and then specify the Kube system namespace to see what's inside of it. How do you do that again, do you know? Keep you still get all dash n for that? Yeah. Yes. Yes. And then keep system. Think. Oh, not that much. We got a bunch of Cilium, CoreDNS. There is something wrong with KubeDNS. Seems like there's been multiple replica set of could they have been trying to deploy a new version with a new configuration of core DNS that would have forced Rawkode replicas? Yeah. That would concern me too, definitely.

25:49 Interesting, Alex. Yeah. Interesting. Yeah. DNS, right, could be pointing in the wrong direction. How best to it's funny how we're trying it off camera. Was like, one thing I don't wanna do is call DNS. I'm really not very good at call DNS. Right? I thought the cluster the Fairwinds team must have been like, yep. We're just loving that. So there's a few different train of thoughts right now. Okay? You got the DNS, you got the hint which alluded to stuff in hiding inside of a cube system or system based namespace. But the effect that you're currently dealing with

26:23 is the fact that the image isn't the image that we want it to be. Yeah. So I think you need to just pick one of those and let's let's focus on it. Hnt o one probably should be the thing we focus on. Ever fixing things in order. However, you could just go straight for the image. So So you're thinking the namespace stuff, David? I mean, that would be the journey that Fairwinds want you to go down just based on the order of the hints. But, you know Yeah. The mission is just to get v two running. So it's entirely

26:53 up to you. What are thinking, team? Yeah. I think we'll go ahead to let's just start somewhere because we don't know for sure where the error is. Yeah. Let's look at that head again. Yeah. Let's let's remind ourselves with that headset. It was one. It was one. Yeah. Yep. Sorry. Built in system namespaces can hide all sorts of junk. Built in system namespaces can hide all sorts of junk. Yeah. So, I mean, it's possible that it's not exactly the cube system. I think when I saw system, I assumed it was cube system. So it could be any other v built in namespaces.

27:41 Yeah. I think you're right. I would I would take a look at those namespaces that we always ignore and see what's running. I think that's a good idea. So what are we thinking, Max? We look at what gives CDL get an s? Yes. Let's check each other one of them. I think we we could try public. I just want to kubectl get all thinking? Get all. Yes. And then dash namespace. Dash n, then new namespace. Interesting. Postgres there? Kube public. Because you said the Postgres be running in default. Right? That is well, yeah, that's what

28:06 Rogue Postgres Discovered in `kube-public`

28:26 I wanted. Yeah. Should we continue down this should we have a quick look at node lease? Just see what's Yeah. I think we should just check that out. Even though node leases usually used to keep the notes. No resources. Right there. We've already done system. Right? Yeah. Is it worth just checking? I mean, it'd evil. We mean even to, like, put something in emissary. Right? I think that looks pretty legit. Right? That's Yeah. This looks legit. I wonder what the image in that Postgres pod might be. Yeah. It's a good good thing you're nice. Yeah. Let me just

29:10 matches what we would expect to find. Superhackerimage.one. Yeah. I like that. I you're I like yeah. So and this is yeah. Right? Low balancer. Yeah. So kubectl. Yeah. All was it it was what's the name of the The cube dash public was the one that was suspicious. Yeah. There was cube dash public. Yes. Yeah. Cube public. So can you describe on the deployment, do you think? It doesn't look like a deployment. Yeah. I think it's deployed. Default set. Okay. So that's all set. Oh, yeah. Oh, yeah. Cool. So Yeah. Yep. Keep CTL. And it's only twenty five hours old. That's

29:59 suspicious timing. Indeed. Should we just what's the idea? Describe on the portal? I think describe should give us more context. Yeah. Oops. Alright. We're halfway through. Image Postgres 13 Alpine. Docker. Liveness. Exact p g ready. So I'll give you some information about how my setup works because I think it's pertinent right now. So the postgres at the default namespace is not a stateful set. At least I I don't think it is. Or it may be, but it doesn't use any persistent volumes and the data is all Oh, okay. Is all loaded through in in a container.

30:43 Understanding the Postgres Setup & Data Injection Idea

31:02 So it could be that they have provided their own postgres with different data to inject onto the page. Interesting. Let's In which case, we should be able to configure which Postgres is being used by the application. I'm assuming there's a configuration environment variable or something. There is a Postgres running in the default namespace. Right? Yep. I think that's the one, if I'm mistaken. Yeah. It's also 25 years old. Interesting. Yes. Yeah. Interesting. So I'm wondering, you in your config there, Dave, for that for that deploy I think you use deployment to inject some config for the Postgres?

32:01 It may very well be entirely hard coded to just Postgres. So if you run get services, I'd be curious to see what we see there. Standard cluster IP. Yeah. And it's two days old, whereas the other one was twenty something hours. Yeah. We just do we we don't describe that one. I mean, if we do describe on right. Just keep detail. Describe on the service. Think scrolling up through. Hopefully, folks can see this. Oops. Yeah. So my application is hard coded to speak to a service called Postgres and the same namespace. So if we cannot change the endpoint

33:19 for that thing That's if it's my application running. Yeah. Which I I'm not convinced of. But maybe there's something to do with core DNS? Yeah. We've seen those replica sets. Alex, you were suspicious about them. Right? Yeah. They could be forwarded onto something else. Good catch. Onto another namespace. Right? Yeah. That that makes a lot of sense. How would we go back yeah. My core DNS knowledge is somewhat limited. How do we go back Yeah. Finding that one out? Everybody says because why would you why would you need to configure core DNS? It just works. Right? Yeah. Exactly. Like, I

33:30 Investigating CoreDNS Configuration

33:55 mean, it's always DNS, right, as the as the joke goes, but why do we need to configure it? Let's maybe describe the core DNS pods and figure out if there's a config map or kind of configuration that's There is. There's a config map in KubeSystem called core DNS. Or, yeah. Maybe cube DNS. I can't remember. But if you've listed config maps in cube system, you will see what you're looking for. In cube system. Yeah. CoreDNS is one. Does it just get config map or describe what is it? It was it KubeS? I thought it was cubes public. Was

34:42 it cubesystem? Was it not cubes public? We so Okay. Well, cubes public is where the rogue postgres is, but the core DNS configures in cubesystem. And you can just system. Alright. Add dash o yaml to that if you wanna look at it. Yes, Oh, sure. I'll check lame duck five seconds. That's interesting. It's a rewrite. It's tough. Interesting. So what do we think it I think we should Should be like not rewrite this. This should public. Oops. Just to edit. Edit. Yeah. Go straight to edit. Go for it. Yep. Yeah. It's just in, like, start getting

35:17 Fixing CoreDNS Configuration

35:31 the one and keep public. And then what we thinking? The Let's comment comment out this rewrite block so that the Comment it out? That's a bit cautious. Alex. That that's Alex all over. That's why he's awesome on the team. He's super cautious because, yeah, I was thinking, like, d d. I'm just gonna press up d. Exactly. That's what d d and I do in DevRel. Right? Yeah. Brilliant. Cool. You over that? Yep. And now Let's maybe restart the core DNS server. I don't know if the configuration map will automatically reload or if we should force a restart.

36:22 Yeah. What's the what's the best command for that one? So if I do TTL get is it, like, in the name of this cube system? Just we just restart? The pod probably. Can see I'll get pod. Yep. Alright. You just have ten minutes. Yeah. So we've two pods there? Yep. Just delete? Or Delete. Delete. Yeah. Can yep. CTL. Delete. What's usually take a while to okay. That that was fast. Yeah. That's pretty good. Did you want yeah. If I change the final one there to TBH4S4S. Yep. And then we probably wanna do the same thing with our application.

37:35 Restarting Application Pod & Testing (New Error: DB Connection)

37:38 If there are connections established already to the wrong Postgres instance. Like it. Forced to reconnections. Good point. Yeah. And see if could restart with that Rust app. Right? So should we try calling local host? Different error. Failed to connect to database. It's a different So one down. What's the next one? So it's a hard coded value in the app reaching out to that. There was a very pertinent comment from the chat. It was the endpoints, I believe, for this. That's right. There we go. It was the Narla Patel there. The Postgres service has no endpoints on the

38:42 Hint from Chat: Default Postgres Service has No Endpoints

38:52 default namespace. Oh, interesting. Well, thank you So probably label selectors or something of the sort. Yes. That's the best way to figure that one out. Get endpoint but no. Yeah. We can do some analysis or look directly at the the labels that are applied to the Postgres deployment and compare the label selectors in the Postgres service. Or should we describe the service for Postgres? Mhmm. Q o. It's just Postgres. Oh, No endpoints. Good catch, Narla. Nothing. None none. Right? On the labels on the endpoints. Oh, I'm sorry. There is actually on the label selectors, app equal Postgres.

39:13 Identifying Missing Labels on Default Postgres Pod

39:53 Sorry. Yeah. Alex, what was that? There yeah. I was just pointing to the app equals pro equals Postgres. So Oh, I gotcha. On the selector. Sorry. Yeah. Yep. So you're gonna wanna check that pod and see if the labels match up. Mhmm. Yeah. On the Postgres pod. Label's none. Alright. You wanna see that one at the top? So do have Kubel edit? The old favorite. Go for more time. Exactly. One more time. There is I can just find it. How much of the pod spec is immutable, though? Oh. I don't remember if labels are or not.

40:35 Investigating Default Postgres StatefulSet

40:59 Yeah. So we probably should modify the statefuls, sir? Deployment for the cluster one. Yeah. The deployment. K. Keep CTL, edit the cluster service, did you say? Deployment. The deployment. The final label. Clustered one, but the Postgres one. Yes. Sorry. It's a stable set for STS. Yeah. Oops. Press q l. There we go. We'll get it. Awesome. We're in. Yeah. I wrote these manifests. I don't even remember. It's fine. There we go. At postgres. Oh, is there a mutating webhook or something? That's changing that. Could be. Yep. More for Luke. Yep. Got five minutes or just under five minutes.

42:17 I think the Fairwinds team might have a specific on this one. What is it? Get mutating webhooks? I can never really Mutating webhook configuration. Nice long one there for you. That's a long one. No recesses found. Any thoughts? I would encourage you to use the hints given how little Yeah. Time there is. It doesn't hurt to use the hints and and get maybe catch one more time. Ejecting HTML is so easy when you control the database. That was your suspicion, David. Right? Yeah. I'm gonna have to sanitize my inputs. Lesson learned. We do it like another

43:10 we sort of need that. I've never seen that SQL injection go the other way. On the database side. Yeah. Nice. Should we do the other? Three. DNS. That's it. Right? DNS. There's no way to that. Right? DNS. Yeah. We're good for you know, you can just name a naked pod the same as what a control expects? I did not. Right. Did you Ah, okay. So let's just delete this pod. Yeah. And let the country actually It's the Postgres pod. Postgres Yeah. Quick. Nick it with fire. You've got you've got it's three minutes. New CTL. Delete

44:00 pod. What was the Postgres, yeah, cool URL dash o one. PostgresQL, o one? I should And maybe Postgres. I don't know. Yeah. Let's just if you get supports. Yeah. Yeah. Oh, just deal. O. Yep. Yep. Lost my turn up. I don't know. Interesting. Should we just do a Yeah. Yeah. Go for it. It failed. Well, it may not be running yet, to be fair. Yeah. So this is Oh, it's gone. That's intriguing. Right? It's just default set replicas set to zero? Is the deployment set? The stateful set. The stateful set. In in the other namespace or No. In this

45:08 one. So what's the what's the command to run? I mean, I would just edit the stateful set and then search for replicas and see. So kubectl edit. S t s. Yeah. Post SQL. There we go. Touch four. Zero replicas. Ah, yes. I see it. That's alright. That's That's alright. Click here. Yeah. One minute. You're kinda close. Yes. We do we do have me too. Right? But kinda like a like. Oh, that's that's b one? Your bag, your bag is not a success. Yeah. Yeah. It looks like it worked. It worked. But we didn't is it got b two,

45:34 Finding Replicas Set to Zero

46:07 though? Let me show us that. Yeah. Yeah. And where we changed the v two? Did we reverse back to v one? Not sure. Oh, Pairwinds insights can help you with resource risk in limits. I I think it is. I think it's just a video. I think I remember what resource. There you go. No. Oh, it is. Yeah. It's just my it's just my my browser being a little bit weird, but you got the video. It was all. Awesome. Great work there. Well done. On the nick of time. Yay. Awesome team. Right on time. Late than never

46:47 Team Ambassador Labs Debrief

46:53 used slap. Alright. Well, thanks for that. You said great. You just got it fixed. You had like ten seconds left. So you know loads of time for debug in there. I'm gonna ask you kindly say thank you, but I ask you all to pop off and we'll get the fair win team and they'll see what you've done to your cluster. So thank you again. Well done. Thanks, David. Thanks, David. Alright. I'll just pop over here until we get the Fairmans team over. Get my cluster back up. Ambassador Labs. Alright. Cool. They're all getting here now.

47:10 Transition to Team Fairwinds

47:41 We've got Brian, Andy's back. Hello. How's it going? Oh, and Robert. There you go. Oh, we got a fourth. And Robert. Oh, someone is still watching. I would encourage you to close your YouTube tab. Oh, someone is still watching. I would encourage you to close your YouTube tab. Oh, some of those were still watching. Oh, two of you are still watching. Okay. I'm gonna mute all of the fear of wins team until someone walks out where that tab is. You got it? Cool. Alright. We're back. Sneaky work there. Well done. Before we talk about that, why don't we do

48:30 Team Fairwinds Introductions

48:32 a little bit of round of introductions? We'll just start from the top right with you, Brian, go around clockwise. Please feel free to say hello and share a little bit about yourself. Sure. Hey. I'm Brian, customer reliability engineer here at Fairwinds. I joined two months ago, but I actually worked here for several years previously. And really excited to be back and and part of this team here. Alright. Sam? Hi. Thanks. Samantha McAuley here. I've been with Fairwinds for a few months. Also a, SRE with, Brian. And just happy to be aboard and and having fun

49:21 here. Thanks. Awesome. Thank you. Robert? I'm, Robert Brennan. I'm a I, I'm a software developer. I lead the team building Fairwinds Insights, which are our commercial platform for doing, Kubernetes governance. And, Andy, there you go. Hey, y'all. I'm Andy, director of r and d and technology here. I've, been doing Kubernetes for so many years now, and it's just super fun to be back here. This is a total blast. So thanks for having us. Yeah. Welcome back. Doing a reverse SQL injection into my application was particularly sticky because in my head, I was trying to fix the wrong thing, I

50:07 think for a while in there, trying to work out how you potentially replace the image. I like that. That was very sneaky. All right. I think it's only fair now to hand a cluster over to you. So, I'm gonna pop my screen share back up and I will open a connection on Farewell. Oh, no, not Farewells. Ambassador Labs control plane one. Please remember to use activity, active sessions and join rather than creating your own session, just so that we're all typing into the same window. And if you could all give me a echo hello, that would be awesome.

50:18 Start of Team 2 Challenge (Fairwinds vs Ambassador Labs' Break)

50:45 And I will get one more extra picture of on the screen because we're missing one. Yeah. Just do this all ad hoc. I don't need me. I'm good. All right. We've got a few hellos there. So feel free to set up your KubeConfig and aliases, check for a control plane and best of luck team. But who have we got driving today? Oh, yeah. That would be me. So yeah. Feel the curse everyone feels when you screen share screen. Everything suddenly goes to nowhere. So where is this one, Andy? It was in Etsy. Etsy Kubernetes admin dot com.

51:13 Initial KubeConfig Issues (Typo in "keys")

51:52 There we go. And, Brian, I understand. I drove last time, and I still don't remember all the details of what happened because it was just, like, total blur. Uh-oh. You have no control play. Let's take a look at what's in that cube config. It looks like Yeah. It's missing some information. Client keys data. Is that the right key? Is it supposed to be client key data? Client key must be specified. Client key data. Yeah. There's client key data. And it says client keys data. Is that supposed to be plural? I don't think so. I think Not

52:47 based on this error. Errors of the very wary, I think. Yeah. I'm very wary of typos after last time because we spent, what, fifteen minutes staring at that YAML trying to find a typo last time. Alright. So I'm just gonna ditch that guy that one there. Oops. You can specify aliases. That's right. Ex what is that? Export. I'll do that later. Looks like we're still not getting it. Yeah. It's asking for client key data still. Yeah. Thought we are we in the right config? Let me get out of there. It's there. Where is client key data?

53:23 KubeConfig Error Persists

53:44 It's in the right place? Is it, yeah, is it indented properly? So can't we just use QADM to give us a new config so we don't have to try and figure out all the things that might be wrong? Sure. Remember the command for that? Oh, no. I think it's let's see. I think it's just kubeadm kubeadm config. Yep. Kubeadm kubeadm config. Let's see. Would it be user? Have you used the Internet as a generator config? Or is the user command necessary? No. I gotta have something there. You can try Kubernetes dash admin. But I think without the user.

55:15 Oh, without user. No. No. Wipe user. I think that is the user. Yeah. Gotcha. It might want a flag. Oh, I got the flag. The flag. Again, screen sharing. Apologies. I know. It's user and then dash dash help. It must want maybe dash dot. I don't know if it wants a flag for the Kubernetes admin or not. Oh, right. Right. Right. Right. Right. I don't know. I've never used this this command before. Oh, what I'm let's see. I'm looking at this. No. I see. Q config user dash dash client dash name That's it. Equals dash admin.

55:59 That's But we need the q a d m config file for this cluster. So we need dash dash config, and I don't know where that is. K. But you can try it without the last flag and see what it does. K. Config not set. Alright. Alright. Let's see. There is no Kubernetes and config left on the machines after the provisioning step, I'm afraid. Okay. Alright. Then let's let's look at because Why yeah. Why is it still complaining about the the key key not being there? Good question. Let's take a look at that thing again, shall we?

56:55 Yeah. Yeah. The admin.com file? Two. With which kubectl. Okay. And let's check for aliases too. Okay. That's sneaky. Because, like Yeah. That would be sneaky. You could just type alias. Yeah. That will list them all. Just type alias. Yeah. Up to date. I don't see anything there. Someone suggested looking for a backup of something. Wonder if it's hidden somewhere. I don't know. Let's go back to what what we were looking at the queue. The admin conference. Right? It's not red hair and YouTube part of it. So you have client key data, client certificate data. Oh, wait. No. That's

58:00 cluster. Contacts. User your Kubernetes admin. That matches the name. That oh, check your contacts. That's right. Yeah. Kubectl yeah. The name of the context. Is that correct? Do a kubectl config get context. Oops. It does look okay in the file. So we specify for Kubernetes admin. To use the client. Why is it doing that when you say kubectl config, though? That says to me that something's been, yeah, messed with here with the kubectl binary itself. Wow. Cat user bin kubectl. Uh-oh. Oh, alright. I mean, I if I was gonna be really mean, I would replace it with a

58:51 Suspecting the Kubectl Binary/Command

59:08 bad script. Can you type reset to clean that up? I I did the same thing, and it worked okay. My my apologies. How do we get out of this one? Guess I could've used file instead. That would've been Yeah. Yeah. I think I'll just close this terminal. Alright. So I let's close this. The session will disappear. Okay. We have a I missed it. There you go. Okay. Connect. Oh, I can't spell control plan. Apologies. Okay. We have a new session. Please join that. The one that says could just close the one you're in just now. Just jump into the one that's been

1:00:31 open for a few seconds and reconfigure your config and hope for the best. And for the record, that is two people that have captured a binary file in the last two episodes. So Do you want me to use file on that, Andy? No. I mean, we know it's a binary now. Oh, yeah. Yeah. Yeah. I guess it could be the wrong binary, but we would I was gonna say maybe. Download a new maybe? Yeah. We could get a new just grab a new one just in case. Let's see. It helps if you use grab. Me in

1:01:16 the other room. Sorry. Yeah. Yeah. Yeah. Grab. It's been there for a while. Okay. Yeah. Hello. Yeah. Oh, yeah. There must be hints. Yeah. Was gonna say, are there other hints? It's functionally similar to Carta. So a typo? Don't count don't count them all. There's That's what Carta did. There's an element of the eighties soft sell song here. Brian, you're our music expert. Tainted Love. So there's tape somewhere. But That sounds like that sounds like a typo that we're missing maybe. So let's maybe take a look at that admin conf again. Okay. I got Cluster, I mean, right, is there

1:02:10 Investigating KubeConfig Keys & Typos Again

1:02:31 is there more going on, like, above here that I can't see? Here we go. Clusters cluster name, Kubernetes, context cluster user, Kubernetes admin. Contact's name is Kubernetes admin at Kubernetes. Current context matches that. Client's I wish you all the spaces and tabs. I have curiosity. Say that again? Is YAML fine with spaces and tabs? I'm guessing. You know, like Oh. Like What what have we got before the client key data about, could it could it be a tab character or something? Could be, but I don't think the actual cares. Does it? It could if it's

1:03:37 oh, no. Yeah. I think you'd get a different error message. But it's weird that it's saying there's no client key data. That is weird. Like, if if can you if can you edit the file with with them just to, like, see if they're letting any hints? It's that little add add add at the bottom. That's just them. Oops. Is that the right number of spaces in front of client key data? I can't actually tell from here. Yeah. It looks like it. Looks like four spaces. Can you Yeah. Go down there. Let's see. One, two, three, four.

1:04:36 Yep. That's four. I'm looking at a cube config for a kind cluster locally. Yeah. So suggested our client certificate data and client key data swapped. They could be I'm not sure how we would let's see. Just, like, chain change the keys back and forth? Like, change the lines to try it. Data. Okay. Do wanna oops. Like that? Yeah. I don't think you ever I saved that. Did you export the you export the Kube config again? Probably No. Copy of this file. Then why is it still doing this thing? Oh, you're right. Export econfig equals Etsy.

1:06:03 How is it still giving us the same error no matter what we do? I think This is not right. I think need a new There's alternative ways to set the cube config. What about the flag on the cube control? Yeah. I I would try that. We checked. Yeah. The dash dash cube config equals Yeah. And let's just rule out some environment trickery. And then the last thing I'll suggest is there are other cube configs in that directory for the other controller managers, etcetera, that may be able Yeah. That would be yeah. It's true. We could try any one of

1:07:00 those. I mean, if we wanna grab a new kubectl binary, I have the command here. You could also do apt get install dash dash reinstall kubectl if you wanna refresh it. It's like that one? Yeah. I don't know if it'll more of it reinstalled flag beam before install, but give it a shot. K. I'll put that at the end. Or Yeah. Take it there. K. Same error. Alright. So it doesn't Should we try one of those other KubeConfigs? Or What does k version show just to make sure we're using one dot 23 dot four? It's 23 dot three.

1:07:58 But something's a weird, but it should be Oh, k k version dash dash Is it that's just client only or that's just client? Client on wow. It's not liking any of that. It's not the real c t l binary. There's no way. Yeah. Something's weird here. Beautiful. Just here. Can I download that binary to Loco? Oh, you could always just instead of using alias and keep control, like, fully, like, do slash then keep control and then you're guaranteed. Or you could do that. Try the one. Try try the one in that slash. Although that has

1:08:28 Bypassing the Hijacked Kubectl (`/usr/bin/kubectl`)

1:08:43 been really fast. Try that slash q back tool. You need to plus exit. Oh, yeah. Yeah. Yeah. Different. We swapped the Yes. We we did that. Right? The key the key and the certificate. Yeah. We need go swap those back or use a different KubeCon thing. Alright. Should I move this file over to user Yeah. Join? I don't think it's gonna make a difference. I think what's in user Ben is correct. I think you'll find there's something in the bash profile possibly that is being very sneaky here. But if you fully qualify, could control all the time as slash user

1:09:30 slash bed slash control, you should be okay. You can you can tell us to this current one. Yeah. Maybe something in bash or c or dot profile. It could be anywhere, to be honest. Oh. Mean, we've we've gotten around it. We don't have to know what it is. Yeah. Exactly. Gone out of this cluster and move on. There you go. It's the function. There it is. Yeah. Okay. Oh. Oh, man. That's oh, man. That's so good. That is exactly what we did to Carta, isn't it? With a function instead of an alias so that they wouldn't be able to find it. That's

1:09:51 Finding the Kubectl Function in `.bashrc`

1:10:06 so good. You need to do that. Sorry. Yeah. You're on it. Okay. That doesn't am I on it? Yeah. The source I thought would have like, you could do unset cube control. Yeah. That would be good. Okay. There we go. There we go. Okay. Back to wow. Man. We're We're out. Alright. So just swap these over. Yeah. Think y'all counted on our memories being better than they are. Alright. Back to our regularly scheduled program. Alright. Let's see what we got for pods and deployments and such in this name space. I love that they watched that episode. They

1:10:50 Diagnosis: Pending Pod and Node Taints

1:10:58 used that trick against you. Oh, so good. Apparently, I should've done the same. Got one that's pending. Alright. Alright. Well, based on hint number two Oh, yeah. You no taints. Right? Tainted love. I bet we're gonna see something around tolerations or taints. Yeah. So There's nothing embarrassing about knowing that soft sales can take a good lump. I'm not I'm just gonna throw that out. No. It's a wonderful thing. Let's see. There we go. What do you see? I see a taint here. Right? If you can use the last pager, it just means the audience can follow along as well. Oh,

1:11:53 sorry. Okay. Are you all bits are you all able to see that taint? Yep. Yeah. It's there's a no no schedule taint. Oh, alright. Yep. Should I just delete that off those nodes? Mhmm. Yes. K. Is that through edit, or is there another way to do that? There is a nice sweep, but nobody remembers it. I would edit. I'm just speaking for a second. I'm just saying that. It's like, I I I know how to create one, but I don't remember how to remove one intentionally. It's it's the same, but you put a dash on the

1:12:13 Fixing Node Taints

1:12:33 end of the tenth. That's it. That's it. Definitely not gonna remember that. Yeah. I'm still relying on keep control edit these days. It's unbelievable. Okay. And then worker two. Alright. That's good news. Running. Let's edit the deployment to v two and see what happens. I suppose we should look and see if the app's even running on v one. But Yeah. Let's do that first. Was that a curl command? Or Local host 30,000. That's not fun. Yeah. That's not working. Import 30,000. Check the endpoints. How much is that right? Yeah. Yep. That's the right pod or that's the right

1:13:04 Testing V1 Application (Still Not Working)

1:14:00 Is it the right port? The port is correct. Yes. Okay. K. Cluster runs on eighty eighty. Okay. Alright. I'm gonna assume Daniel with his love for the JVM has stuck it on a machine, and now all networking is broke because the JVM is stuck in all the Should we look at hint three? What did what did the curl sound there? I don't think you need hint three. In the return. Alright. But what's the symptom here? Right? You're you're trying to curl a service. It doesn't work. So Right. Let's check for network policies. Yeah. Good shit.

1:14:47 Discovering the Network Policy

1:14:53 Do a default log. What can we do that as as the animal? Twenty three hours old. I don't think that exists normally. I mean, the name would it looks legit. Right? The name looks legit, but I I don't don't think you need that. Let's see. Policy rights egress. No rule. Pod selector is nothing, though. But that Doesn't that select all pods, though, on a network policy? Yeah. It seems like that. I'll just delete it. Yep. The network policy. Yeah. Nothing good ever came from leaving something lighter. I'd like that. K. Let's try our curl again.

1:15:42 Testing V1 Application (Working)

1:15:44 Hey. Yep. That's v one. Did we edit the deployment already? I can't remember. Not yet. Like that one? Yep. Check curl. The new one wasn't ready yet. I'm not sure. Do we do we get get network policies again? Wow. Came back. Oh. Okay. That's what Is it It would it would have a control plane name if it was a static manifest. We learned that while trying to break the other cluster. Let's do a get pos dash capital a and see what else is running. It's probably something recreating it maybe. You're right. Unless it is a a static

1:16:37 Network Policy Reappears - Investigating Recreators

1:17:10 pod or well, no. Let's Kube VIP. I don't that doesn't look familiar. That's a bare metal BGP broadcaster and load sponsor. You don't need to worry about that. That's supposed to be there. I mean, as to it as a KubeFit. I don't I I right now, I don't know. But Yeah. Yeah. I can't remember if there was a Cilium operator or not. There is that look I mean, besides the twenty four hours ago restart on it, it looks okay. It passes the eyeball test. Controller managers. Cube scheduler ambassador labs control Plane 1. That's correct. I just remembered that the cluster is

1:17:53 called ambassador labs, not heroins because I was Two core DNS pods, cluster pod. I mean, does the sneaky thing have to be a container? No. It could be a process system as well. Yeah. Yeah. Maybe edit the cron job file? It could be a cron job. Yeah. Do it. Can repeat it again? Cron jobs. Oh. I mean, like a Oh, on the system. On the system. The system crontab. It could be a system crontab. Yeah. Yeah. We spent so much time in Kubernetes. We forgot how Linux works. Yeah. Crontab dash e, you think? Yeah. Oh, there it is. Nice.

1:18:40 Finding and Removing the Cron Job Entries

1:18:44 Nice. Oh, and it's gonna automatically set that too. Right? Yeah. Both of those. Yeah. Yeah. So just delete those? Yeah. Make those go away. And then Okay. Have to reload that. Right? Edit the deployment. Right? Yeah. Use just the cron tab anymore. Okay. We don't have to be My screen is not letting me see everything very nicely. Okay. There we go. And then Okay. And then curl. Still says we want Oh, he's dancing now. That may be my browser cache, which is always really bad for this. So is it v one or v two, is it? One in the title.

1:19:41 Confirming Team 2 Success (V2 Running)

1:20:05 Oh, that could be a bug too, to be honest. Yeah. Should we start loading it up in the background? How do we know? I I I think it fixed it. I mean, if you describe the pod and it says v two and my browser says v two, I'm happy that it's v two. Dude, dude, just grab it, Will you? Please. Maybe I'll update the title for the next episode. I kept wondering about that when I was testing it. I was like, it still says v two in the title. I just assumed it was browser cache. Alright.

1:20:35 Yeah. That says v two. Successfully build. I think you have fixed that cluster, and you had a whole nine minutes left to spare. There you go. That was fun. Awesome. That was very nice work. Ambassador folks. That was cool. That was great. Yeah. Some dark arts we don't usually deal with. I mean, dropping down to content on the Linux host was pretty amusing. Okay. Alright. Well well done. You smashed that. Like I said, nine minutes left. That's that's rare on custard. So well done. Good team effort there. So thank you for for joining me. And

1:20:38 Team Fairwinds Debrief

1:21:12 I'm sure we'll have you back for a third time in the future. Awesome. Thanks for having us. Alright. Thank you all. Alright. That was two great clusters, two great teams, everything fixed. I mean, that's a pretty good episode of Clustered. I'm really happy with that. And it was fun. I was laughing all the way through that. So thank you for joining us and watching all your comments and the chat. I just want to say thank you again to Teleport. You've seen us use this tool on this episode and every other episode. It's just an amazing tool that I really genuinely

1:21:19 Conclusion & Thanks

1:21:44 believe everybody should have installed on all of their environments. So you can support the show by checking out rockode.liveteleport. And also thank you to Equinix Metal for providing all the bare metal machines that we use for this. We will be back tomorrow. We had to reschedule last week's episode because everyone had more important things to be doing, but we will have another episode tomorrow where we will be joined by Adrien and William from the Discord community. So we will see you all then. Have a wonderful day everyone, and I will see you all soon. Thank you very much.