1:18 Introduction and Welcome

1:18 Alright. Welcome back to the Rawkode Academy. Today, have another episode of Clustered. Woo hoo. Feel free to say hello in the comment section. I'll pop them up on the screen. My name is David Flanagan, but I go by the handle Rawkode. And today we have a solo edition of clusters. So we've got two great guests with us with two broken clusters that we will attempt to fix. Remember, it doesn't matter how our guests fix the clusters, all you need to do is get that v two image running, which I've made some fixes this week. I

1:52 haven't added logging, but I forgot. But I hopefully have introduced enough cache busting so that we'll actually see v two without 10,000 refreshes or at least fingers crossed. Okay. Before we get started, there's a little bit of housekeeping where I wanna thank our sponsors. Teleport have been sponsoring Custard for a very long time now, and we've been using Teleport since the very first episode. You see it in action every single week. It is an amazing tool that allows you to commoditize and secure the access to your Linux machines, Kubernetes clusters, applications and databases. So you'll see us using it, Go check

2:05 Sponsor Thanks (Teleport, Equinix Metal)

2:29 it out by visiting rokode.live/teleport. And also thank you to Equinix Metal. They provide all of the hardware. We run clustered on bare metal machines with, I think, like 32 cores and 60, maybe even a hundred gig of RAM. I can't remember. They're pretty big beefy boxes. Why? Well, because it makes it more fun. So thank you to EquinixMetal for that as well. Now I'm gonna pop over and introduce our guests, and then we'll get the comments going, and then we'll fix some broken clusters. Alright. Welcome, guest. Today, I am joined by Rawkode and. Hi there. How are you both?

2:51 Guest Introductions (Borko and Bojan)

3:06 Hello. Alright. Yeah. I don't know if he's nervous or excited or a little bit of both, but let's start with some introductions there. Barco, why don't you go first? Tell us who you are, and then we'll move over. Hi. My name is Barco. I'm currently working as a software engineer at RenderCloud. Big fan of the cluster series. I think it's a great resource for people learning Kubernetes. So I'm happy and humbled to be here and try and fix some clusters. Yeah. You also participate in officers with me, sharing more Kubernetes knowledge with everyone. So thank

3:46 you for that as well. Alright. Bojan? Yeah. Hello. Nice to meet you. And, yeah, I'm a software engineer at Digital Ocean. Is my identical twin brother, so it's a quintessential twin thing that we're working at the computing companies, I guess. And not really super familiar with this show. I've only watched it once when participated once before. And but, yeah, looking forward to, I guess, trying to fix what he did. And yeah. Maybe a little bit of expectation setting. I think mostly what I do is interacting with Kubernetes programmatically. So, you know, manually configuring control plane is

4:35 not something I do on a regular basis. So maybe I don't have the operational muscle memory for everything involved there. But, yeah, looking forward to learning maybe something new myself as well. Awesome. Well, thank you very much for sharing. This is the first ever identical twins we've had on custard, so that's pretty cool. And we're gonna see if the twin tuition. Is that a thing? Right? Twin tuition? You both can just read each other's mind now? I think we have found that not to be true, I think. Would you agree, Rawkode? I don't know. I don't know.

5:13 Alright. Well, we're gonna find out if you could protect each other's breaks. So I'm gonna pop open my screen share here. I'm gonna pause the timer just because we're not quite ready. I'm not even decided who's going first. We have our teleport server here. Does do either of you have a preference for going first? Yeah. Barcode, you suggested. Right? So Yeah. Let's pop open control plan. I'm gonna start the timer. Okay. If you could please join the session. Just type echo hello, anything to let us know that you're there. Start setting up your KubeConfig, and we'll take it from there.

5:42 Borko Begins Debugging

5:54 You guys see that? We do indeed. Good luck. Awesome. Okay. So first thing I'm gonna do is get my completion. No. Even setting up the KubeConfig first. It's just like I want my auto complete. Just Mhmm. Yeah. Is that okay? Yep. Alright. And then we will So first thing I'm gonna do is just check if we have nodes or namespaces, and things seem to be I mean, we're connecting to the cluster. Yeah. You've got a healthy control plan. A good start. So we don't have our application running. Let's see if we have I mean, things seem to be okay.

6:01 Initial Cluster Check and Pod Status

7:19 Okay. So I will so here we have so we have invalid image name. Yep. So I'm guessing he so it just to explain things, this is a deployment. So we can see that there's a deployment. So we'll take a so we'll take a look at this deployment. And take a look at if he changed the image. Is that I I feel like that's probably not correct. Is that supposed to be the right? Yeah. So it has been updated to the Rawkode Academy GitHub organization now rather than my Rawkode username. So that does look alright. Yeah. That looks fine.

7:26 Debugging "Invalid Image Name"

8:29 Okay. Interesting. It passes the a test, at least. I'm not telling you it's correct, but I'm gonna say that it looks correct. So I don't think we've seen invalid image name before Yeah. On cluster. That's not a, like, image pulling error. So I'm trying to think what does that quite mean. I'm also curious. Are you on a vertical screen by any chance? No. No. No. Or just a very large screen? Like, your screen is massive compared to mine. Alright. It doesn't matter. I'm I'm scrolling frantically to keep up. But there anything I can do here to fix

9:21 it? No. We're we're we're all good. It's all good. Don't worry about it, mate. So just because I'm I'm curious about what we're seeing. Right? Is that we have two clustered replica sets here or at least two pods with different replica IDs. So it's as if the image has been updated. It's now being considered invalid by the kubelet. But when you go to the deployment spec, it it doesn't appear to have been modified. Right. And the image pull policy is always Let's take out the sledgehammer already. Oh, there's lots. The sledgehammer's coming out. So you're gonna delete all the replica sets

10:07 Attempting a "Sledgehammer" Fix (Deleting Replica Sets)

10:39 and force the deployment to recreate. I'm not going to be trying to figure out if that's a cause of the issue or not. What's the l equals clustered. I'm not sure if dash dash label Oh. Work. But dash l one l. Well, one dash. And a d on custard. Oh. Sorry. It's the it's the rule of typing when people are watching. Right? Okay. Wow. Image name. Okay. I mean, this is Oh, yeah. I was laughing at it. Is there idea. Why don't we just describe the pod? Is there maybe there's something else in the event?

11:39 Analyzing Pod Events and Image Format

11:44 Yeah. Failed to apply default image tag. I mean, this sounds something like container d. Like, something's different. Like, I almost feel like something's off there. I don't know. I have never seen this here before, but invalid reference format. Right? Like, that for some reason, my inclination is saying that it's not Kubernetes. What what were you what do you think? So it's fact field. I mean, it passes the eye test, and I've been burned by this before on clustered. I'm half tempted to guess that there's maybe a Unicode character in that image name. It's kinda where my my brain that has been

12:53 hurt in the past by breakers is going to right now. So I mean, what I would do would be to edit the deployment again and retype that whole image name from scratch. Okay. Yeah. Let's do that. I mean, it's just a guess. Like, I I'm wrong more often than I'm right. So it's your call. Well, Robert Cable said, is that even a valid URL? It certainly looks valid, and Russell is saying that the Unicorn is against the rules. Only unofficially. It's never become an official rule. Like like the no e b p f rule. Like, you can use e b p

13:17 Retyping the Image Name (Fixing Unicode?)

13:57 f. It's not not a real rule. Why did that jump over? Sometimes when does that, it's particularly annoying. Abhishek is saying why people talk very low about DevOps. I can talk louder if that's what you mean. I'm not sure. Hey. Okay. That's interesting. Does it still stay invalid? Well, I updated it to v two because that's the goal, and we type the image names. So I don't know. I mean, things seem to be working. So can we open the page and see if the I suspect we have more things to fix. But Sure. Hopefully hopefully, my breaks are correct.

14:21 Testing the Application (Database Timeout)

14:46 Yes. We have a database connection time out. So You want I was using the only code in there. If that if that was against the rules, my apologies. I didn't know. But It's it's not against the rule. It's an unofficial rule because it hurt me so much last time. I literally spent thirty or forty minutes pulling my hair out. I have no idea. So it's it's not against the rules. You can use it. Or something. Was it the k? Because I thought the k looked It was it was the c. The c was the serialic c or the

15:16 serialic s, I guess. Yeah. So It was the c last time as well. So that's just that hurt even more. But there we go. Alright. So we've got a v two, but we cannot reach the database. So the issue is now your application needs to connect to our database. And if I recall correctly, that's hard coded in the application. Correct? It is hard coded, yep, as Postgres. Right. Okay. So let's that stateful set. Yep. That stateful set called PostgresQL just to make it more annoying. But the service is Postgres. I should fix that too. So you're going straight into the service definition.

15:29 Debugging Database Connection Issues

16:26 Yeah. I just want to see if there there was something off here because last time I was in Cluster, they put a startup probe to change the password. So I thought maybe he was doing the same thing back to me. But there doesn't seem to be a startup probe that's that's doing anything. Is that correct? Redness probe? That looks alright to me. If that were failing, the pods would not be healthy. Right. And I think the pod was healthy. Yeah. Correct. Okay. So I'm just going to take a look at our pods and services and endpoints.

17:30 It looks good. Yeah. Got a service. Got an IP address. We've got an endpoint. Okay. Did you say we have logging now or something? No. I didn't add logging. Oh. I I added cache and validation so that when we deploy v two, we'll actually see the video rather than the browser. Because I used the same video name on both. It should be it should work better now. So Okay. What would cause the cluster pod to be unable to speak to the Postgres pod? Let's check if there's any network policies. Yeah. Good idea. This is a Selium cluster. How's your Selium

18:26 Checking Network Policies

18:38 knowledge? Terrible. Yeah. So I'm just checking, like, what other network policies may exist. So we have some ceiling specific network policies. Yes. We have CMP and CCNP, which are cluster cluster network policies and cluster cluster wide network policies. Is that correct? I don't think so. Right? Well, I think we need to take a look at it. Yeah. I mean, I'm I'm just going to get that. Yes. We have an Anchorage deny on app cluster to Port 5432. Yeah. That would probably get in the way of things, wouldn't it? Oh, there's that too. You can just do CCNP

19:30 Discovering and Deleting a Deny Policy (CCNP)

19:58 as a sharp or, yeah, tap to put that out. That works. Okay. Maybe let's give it another try. I'm afraid that it's still unable to connect to the database. And I I love the laughter on the face there. So what we had some other policies maybe we should Yeah. There's also CNPs. And there's our first DNS comment on the chat from Bradley. Okay. So We pointed out that the error message was slightly different. Oh, was it Failed to look up address. So yeah. So this was not a failure to connect. This was a failure to look

20:51 Investigating CoreDNS Configmap

20:56 up address. Good catch there, Bradley. I don't know if that So this sounds like DNS. So if I I think it's in the config map. No. Right. Is that where the DNS might be configured? There is a conflict map called core DNS, and they could just have namespace. Yep. Bradley has seen the previous Unicode break where Guy Templeton put the Unicode c and stayed resolver for the Kubernetes namespace. Yeah. So I I see something here that does not look right. What do you see? Yeah. Right. Yeah. So let's fix that. You don't like that rewrite then?

21:49 Finding and Editing a CoreDNS Rewrite Rule

22:21 I do not. Would you like me to or that? What what do you want me to do? You want me to try again? Yeah. Alright. Look up failed. Did I okay. This is actually a question we get on officers a lot, if that helps you. I modified a config map. Oh, we got we got we got to restart the Everyone's biggest gripe with config maps. Yeah. Yeah. Yeah. Oh, what is this? That's CT blank control point one. That's not good. That's okay. That's that's because it's a matter of pods. So it inherits the name of the control plan.

22:53 Restarting CoreDNS Pods for ConfigMap Changes

23:23 Right. Okay. Yeah. All the comments are coming in there. Restart, Gordon. Yeah. Yeah. Yeah. The audience are always much better than us than I am. Alright. Would you like me to try again? Let's yeah. Let's give it another try. And we have to Dan. Alright. That was good. Well done. I know. Very pragmatic. It I think it was good learning debugging exercise for sure. Yeah. Definitely. I got some clever ideas. I want to try to make the API servers that are returning, like, I'm a teapot status code for, like, image or, like, random stuff, but I can't

24:22 Borko's Challenge Debrief

24:55 get that working. So Yeah. I I reckon if I haven't if if I didn't mention the previous unit code break, we may have been looking at that error and valid image name a bit longer. You think you would have got the Unicode given enough time there? Me? No. Yeah. Probably wouldn't have any side of that. Because there's a truly evil break. Well done there. Alright. Let's Who is the evil twin, I guess? We'll find out. Well, let's say that right now. I'm gonna open a session on Rawkode. Control plan one is now there. I have shared my screen again.

25:26 Preparing for Bojan's Challenge

25:40 So, Rawkode, if you can join our So I'm joining yeah. Okay. Join the session. Just take echo so that we can see it on the screen. We know we're in the same session. Yep. Perfect. And then set up your Kube config tech for our control plan, and best of luck. Oh, the alias key. That's the season pro right there. I don't know about that. So I don't know. Let's just trying to get all the parts, and this is not looking good. It is not. I do not think I think we know if we'll be able

26:17 Initial Cluster Check (Control Plane Issues)

26:26 to train now. Oh, no. K. That's running. Alright. So we do have a Kiplit over the pier. But there there are some banners. Running. Now let me just is it queue? If we got the the CTL, system control status, or are you trying to check something else? Well, q v p I is here. Where is it? Oh, yes. So that runs under a static manifest, so there won't be a system d service. Yeah. But we do have some error messages Here. Okay. Error get Says our container runtime network is not ready. Okay. Never plug in. Oh, okay.

27:40 Okay. So c and nine. I mean, I'd be running. You can always just use PS to see what, like, if you a running. Right? The PS and and filter for Kubernetes is probably a good step right now. Because we have no eye no idea what's running. So we have HCD. I'm going to save for now. That looks okay. Yep. We have a scheduler. We have a controller manager. We have a kubelet. We do have an HD. I don't see an API server. Yeah. So yeah. So that I guess. Is that even okay. I mean, should that

28:28 API Server Missing from Processes

28:54 be No. I wouldn't expect to see that there because it runs inside of container. So the kiplet is responsible for starting at c d and the control manager and the scheduler. And those all started fine. So your next step is maybe to go look at the static manifest directory. Yes. And we do have an API server. So I guess Untouched fail snaps. Although that means nothing these days. But still, Rawkode wouldn't be that mean, would he? Sorry. We'd okay. I I don't know. I did break at CB last time. So There's some suspicious output right in the middle

29:13 Examining Static Manifests and Suspicious Processes

29:48 of our screen. I don't know if you see that. I don't know if I should mention it, but I have now. Do you see the process is running? These looks suspicious. Oh, yeah. I do. Yeah. Okay. I know what that is, but, like, I guess, let's yeah. Rawkode's like, shit. I've I've been cut. I can't have said that PS comment. Okay. Well, I'm just gonna oh, wait. First, I'm going to delete that. Yeah. That kinda so the idea was that you'd check IP tables, but I have a lock in it so you can't flash them.

30:37 Identifying the IP Tables Lock Script & Cron Job Break

30:46 So you'd have to figure out, like, how to remove the lock or find this file and just stop the crunch up or something. Alright. Yeah. Like, we do IP tables at Let's do IP tables dash f, capital f, and see what I've never seen a lock on it before. A capital f. Sorry. Oh, nice. Even if, yeah, even if you go, like, flash w, right, because it's right now running as a cron job, It would rewrite. So, like, you might get you might, like, flash them and be able to access the best like, delete them, and

31:32 then it would recreate them. Sneaky. Very, very sneaky. Cool. So you've already just kinda given away. This is the cron, so we're gonna have to edit the system cron to stop the script. So I I have You can do crontab dash e. It'll drop again. It's straight in. Yeah. So I have seen, like, this error in in, like, live, which is kind of it was inspired by, like, an instant I had to deal with previously. But it was when you have a lot of if you're using IP tables, if you have a lot of rules,

31:47 Editing Cron Tab to Remove the Script

32:40 the the IP tables can get pretty large, and then they take a while to update. And then you can have a lot of if you have a lot of pods or a lot of things changing quickly, you can get into some weird behavior where processes start tearing out because they time out trying to update the IP tables where another process has it. So I wanted to kinda replicate something similar. Unfortunately, I couldn't do anything with network policies because we're using. So I just kinda had to just do a lock on it, I guess, using the fake command.

33:26 Yeah. It was nice. I like that. Yeah. I There's something not working. Let me oh. There we go. Thank you. There we go. Alright. So the crown has been removed. Maybe we should check the process table just to make sure this they're they're actually gone. That that looks a bit better. Yeah. Okay. Okay. You wanna check our control plan again? Well, the point where IP team will still be there. So we'll we we will need to flush them again. Alright. Now surely we're have a control plan. Well, let's just oh, yeah. Oh, yeah. We still don't have

34:05 Still No API Server; Flushing IP Tables

34:46 a API server. Yeah. Well remembered. No. This is interesting. I don't think I broke the UPS server. Oh, man. Alright. Oops. Am I? So it could just be that with the the drops on the table, the API server, maybe under some sort of crashing state. You could try restarting the kubelet to encourage it to restart the API server again. That's I don't know. Oh. Bradley wants to know, Rawkode, if you changed the default editor to Nano. No. I think that's just when you do cron tap, it, like, asks you what you want to open. So the default is Nano.

35:28 Restarting Kubelet

36:06 Alright. Okay. So Well, can you check the IP tables? We're now getting a connection refused instead of a timeout, which makes me think the IP tables are now working correctly, but we're not getting the APIs ever started. So Let's run the PS one more time just just to be sure that we don't see the API server there. I see it. Yeah. It says there now. Okay. Good. So I think it just the Kubernetes just needed a bit of a bit of help. Okay. So let me see. Maybe try to get pods or something and see if it's actually working.

36:15 API Server Starts; Checking Pods (Cilium Crash Loop)

37:10 Yeah. There we go. Alright. Celeum is crashing. But, like, I guess I'm assuming one of our notes is maybe offline or maybe it's just this node that's offline and it's just because it's not restarted those ones yet. I'm not sure. Oh, right. Namespace. Now I'm just going to check. This looks sane at the let's see if there's any events. No status here. Okay. Let's just check. So, yeah, the Are those are those pods still crash the back off? And so one of those come back. I think if I think if you kick kick the crash loop back Yeah. They do take

39:07 Cilium Recovering; Custard Pod Healthy

39:13 mhmm. I think it's because this node the API server was offline, and it just It's cool. It up, actually. So it should be sync three. Oh my god. Yeah. Do a wait. Oh, no. That's a worker one. Okay. So Potentially, he's been on worker two, causing some pain. No. Okay. But, I mean But we don't we don't need worker two. Right? So I I mean, possible because I did do try to do some other stuff that might have, but I don't think Alright. Our custard pod looks alright. Shall we shall we try the update? I mean,

40:02 Testing Application V1 (Database Timeout Again)

40:06 can can we just see whether the app works? Can you open whether version one works first, I guess? Yeah. Sure. Yeah. Alright. Connection tagged out. Speaking to the database. Okay. Which is interesting. Okay. We've we've got ceiling working. Okay. So that's not okay. That is so that's that's not an issue anymore. So yeah. Okay. So we're kinda at the same problem that he had Yep. To assume. Hopefully, he didn't do the same thing I did. That'd be very train intuition type of thing, I guess. Definitely. So yeah. Yeah. So let I mean, Postgres is running.

41:14 It does appear to be. Yeah. Yeah. So let's just, I guess, get Let's take a look at oops. So I'm, like, attempted to rewrite the image here, but I know that's not wrong. So redness. I mean, that's fine. Right? Because there's no oh, wait. Start approved. That that looks a bit interesting. Right? Yeah. Okay. Let's I think you got I don't know if it was just intuition or luck that you described that deployment instead of looking going down the rabbit hole of services and endpoints. So that's definitely to intuition that you went in there. I think so.

42:10 Discovering and Fixing the Postgres Startup Probe

42:40 Because I wouldn't have got that. I I would have been debugging the network stuff. So that was you setting the domain. So the postgres lookup was gonna end up its way down custard.com. Very sneaky. Wait. Did I did I delete too much? I No. You're I mean You're you're good. That's fine. Oh, okay. Let me just make sure this all looks me go to the source code and Make sure this all checks out. Okay. Sorry. You don't trust trust your brother? Not in this area. No. So I miss probably speak yeah. Let try that case. Sure. He's speaking.

43:46 How can I obviously? I do think that's, like, my code right now. I think I think this is alright. Yeah. Okay. Let's give it a shot. Oops. What happened? There should be an error at the top if it was or maybe I was wrong when I said that you you cut it properly. I think I might have deleted something too much. Yeah. I think you've been doing it. Click and delete. So what's the shortcut to go to the okay. So no. No. You didn't delete so much. Oh, no. You didn't delete enough. Oh, yeah. The failure threshold looked into the

44:00 Correcting Edit (Deletion/Indentation)

44:38 resources, and I thought it was okay. But yeah. So we have to delete four more lanes. Yeah. And then Yeah. That that's better. There we go. Okay. So that should restart it. You've got v one? Oh, awesome. Okay. So let's now change it. Okay. Almost. Yeah. There's just I mean, I Well We got V2. Cable stuff could have taken a really long time. Right? I did not expect to be discovered so quickly. Well, mean, you you you don't even try to hate it in the process table. They're like, he ran the PS command and it

45:36 Bojan's Challenge Debrief

45:49 was just that rare. Yeah. That and that's probably not my forties, so he was definitely hitting on a weak spot here. Yeah. That was interesting. Yeah. That was good. It was that was that was fun. I like that. I mean, I'm not entirely impressed with the Unicode, but it's it's now that I I've seen it. It's good. I I I really thought the Unicode is gonna be in the DNS as well all over again. Okay. Well, I guess, not surprising that other others have come up with that idea. Yeah. Think you undersold yourself by saying that this

46:30 is not the yeah. But Russell's still there. Nice work. Because then you said you weren't used to this type of work. Like, you were you were working through that pretty quickly. Yeah. So I think you can see the contrast. Like, I'm more used to, like, debugging workload related things. So that's kinda where I was gravitating towards, like, things like network policies, things that would affect functioning of what the app was supposed to be doing and and how it was supposed to be running in its environment rather than the cluster itself and yeah. Just probably why Barco went

47:02 for the Linux route and the the control plane because Yeah. Because you you knew that's where he was weaker. Right? Yeah. Yeah. Yeah. I was just one inspired by, like, the issue that faced, but I couldn't really I want to get it through Kubernetes somehow, like, with network policies or, like, creating a lot of IP table rules that way. But, like, it didn't work with. And then I was like, okay. Whatever. I'll just make take a look on it, like, through a script and then put it in the cron job. And then I probably shouldn't have said you

47:42 can use PS. It's running. That's right. That kinda gave it away because I suspect it would have taken a longer time to figure it out, but well. Well, I felt it only fair to mention the the script that I put in the PS consider. I also mentioned the unique codes. So I think that I I even talked to match a little bit there. Alright. Well, thank you both for joining us. That was really cool. Thank you for taking the time to to break these clusters and then to come on and share your knowledge with the audience. It's it's not easy. Thank

48:05 Conclusion and Thanks

48:13 you. So I really appreciate you taking the time to do that. Yeah. It was fine. It's fine. Awesome. Alright. Thank you to everyone in the audience for your comments and chat throughout. As always, user always catching things that is missed by us on Clustered. If you want to join us on Clustered and you're feeling brave, then drop me a Twitter DM or open an issue on Get Help or email me david@rockode.com. Thank you to our sponsors, Teleport and Equinix Medal, and thank you to both our guests. It was a pleasure having you. I hope we'll see you both soon. And have

48:44 a good day. Thank you. Cheers. Bye.