3:16 Introduction, Show Premise & Housekeeping

3:16 Hello, and welcome to today's episode of Rawkode live. I am your host Rawkode. Today is our seventh episode of clustered, a show in which me and I guess will attempt to fix broken Kubernetes clusters. The catch is these clusters are broken by members of our community and sometimes they're nice to us and sometimes they're downright evil. No idea what we're gonna expect today, but hopefully it will be educational and fun for all. Now before we get started and I introduce today's guest, please remember to subscribe to the YouTube channel and click the bell. You will get alerts for every

3:53 episode of clustered as well as all the other educational materials that I'm putting out there in the cloud native landscape. I would also encourage you to join the Discord. We're doing something a little bit different today. If you're on the Discord, there is a Discord stage, a feature which launched yesterday. People in there can communicate and discuss the episode live over audio. And if we get stuck during today's episode, we are gonna go to the audience for some help. So get in there, get involved. Also, I'd love to thank my employer. They allow me to spend time and energy and

4:23 compute resources putting these educational materials together. So thank you to Equinix Medal. If you wanna check out Equinix Medal, you can use a coupon code, which will get you $50, which is around one hundred hours of compute. That code is Rawkode9. Awesome. That is the housekeeping done. Now, I can introduce today's wonderful guest. I am joined by Guinevere Zinger. Hi, Guinevere. How are you? Hey. Good morning from Seattle. I'm doing well today. Thanks. Thanks for having me. How are you? Yes. I'm very well, but as always, I'm clustered, slightly scared, worried, paranoid, and excited all in

4:40 Introducing Guest: Guinevere Zinger

5:03 one rolled up into a ball. So you're I don't think you need an introduction. Everyone is familiar with you in the Kubernetes community. Why don't you give us the well, give us the TLDR and anything else that you wanna share, and then we'll move on and get started on today's clusters. I am the loud, bouncy one that asks all the questions. So I'm I don't know everything about Kubernetes. Some people think I do. I I I usually know who to ask for help. I consider that my secret power, and I just I'm here to share that it's okay

5:45 to not know everything and learn together and find out. So, that's that's I think. That's me. Yeah. I live at Seattle, which is beautiful right now. We have awesome spring weather, mountains on three sides when we all can. Again, you should all come visit. It's I yeah. I have a kiddo. He's maybe going to poke in his head at some point. And, yeah, I'm really excited. I've I've not watched one of these before, so I'm I'm I'm gonna be like, wait. How does this work? And yeah. I'm I'm I'm looking at this and I'm I'm really excited. I love pair programming,

6:30 so pair debugging is my next favorite thing. Well, yeah. We'll be We'll be doing a lot of debugging. I'm pretty sure. Like I said, we have two broken clusters. We're gonna use teleport. We're gonna pair in the same session. We're gonna work out what's wrong and hopefully we're gonna fix it. I love that you said, know, you wanna show people that you don't have to know everything. I feel like I've done a sterling job of that in the first six episodes. I have I've definitely shown that you know, you can work in this space for a long time

7:01 and work with Kubernetes daily. There's still things that catch me every single time and I think that's true for everybody. Right? We wanna set new norms that you don't know everything and then it's just about experience and learning together. It's awesome. Yeah. Yeah. I mean, what was it? Yesterday? No. Two days ago, I learned that the default back off limit on jobs is six. I learned that the hard way. I kept waiting for my job to fail and it didn't. Well, there you go. Today, I've learned that as well. So Right? We've already started off

7:38 on the right foot, which is exactly what we want. Alright. Let's well you've not seen this before so like, I mean you don't know what all the other breaks that people have been quite evil about. So we're gonna we're really gonna see today whether these people like us or not. So we should introduce our first broken cluster. So we're gonna start with cluster 14 today, which is broken by Phil, Phil Weltz, who is a member of the Discord community, active on Twitter, and as European. So we offered to do this cluster first so that you can spend a

8:00 Kluster 014

8:10 little time with us and hopefully give us any advice if required. Well, when required, let's just put that out there already. Now, let me pop open my screen share. We are now floating bubbles. We have our teleport. I'm gonna jump on to the control plane node of cluster 14. I will zoom in just a smidgen and if you can just join that and type echo hello or something so we know your web is and we will get started. As always audience, please feel free to leave your YouTube comments for anything that we missed, things you want us to check. If you're

8:44 in the Discord stage, we will try and get you involved as well as we get started. So, how is your session looking? I might not have joined your session. Let's see. Let me disconnect active sessions. Options. Root session is in progress. I'll join yours. I joined my own session. That was smart of me. Right. Well, Fell assures us to that he has treated the cluster kindly. We are gonna find out. What normally what I would do first here is just kinda get a set up. Well first I always do my alias otherwise I'm gonna type that 14,000 times and get that

9:34 wrong. The other thing I wanna do is Sorry. So that we don't have to specify dash dash kube config every fourteen seconds, I will just export that too. So now, in theory, we can run k get notes. I'll let you jump in and get that done, and we'll see if Fella's kind enough to leave us an API server. That rarely ever happens. Maybe, you know, maybe this cluster isn't broken and it's an April fool's joke. We've broken something, but we will so do our best to give this cluster a little bit of diligence and see if we can

10:15 find something that is wrong. Now because you've not seen this before, I'll just kinda run through the objectives. Normally, in our default namespace, we would have a couple of workloads. An application that has a little video of me pointing to my watch going we better fix this. It would appear we don't have said workload. So we'll need to work that one out first. Although I'm curious about the Postgres zero. So it looks like we do have the database, but we don't have the web application. And normally, I'd want to deploy an upgrade to the application, which schedules a new shiny

10:18 Initial Cluster 14 State: Missing App Pod in Default Namespace

10:52 video of me dancing, but we're not in a position to do that yet. Let's see if we have pods hiding in other namespaces, though. Yeah. Go for it. Oopsies. I could type in front of people. Oh, yeah. Did I fail to mention that? So there's a bunch more stuff. Yeah. It looks Let's see. One of the staff pods Oh, no. Appears to be hanging. I'm just gonna run that again because my scroll seems to there we go. Scrolls running. Good. Yeah. Okay. Awesome. So there's a Cilium operator that's been going on that is in a

10:57 Checking Pods in All Namespaces (Cilium, Kiverno, etc.)

11:50 Cilium namespace. I don't even know what Cilium is, so that's cool. Cilium is the CNI plugin that we use on these clusters. Awesome. So and then, Qiverno, also of interest. That one looks like it was installed eighteen hours ago, so that's different. I don't know. Again, I don't know what that is. And and then we have a hanging two minutes thirty six seconds. Okay. Also, of the operators had some restarts. Oh gosh. The CSI plugin has been restarted 17 times, 14 times. Okay. There might be some clues there. Yeah. Definitely. I think we have to follow, I guess

12:42 Investigating Stuck Pod: Cilium CNI Plugin Error

12:45 what if a bread crumb. Right? The only container in this entire cluster which is not currently running is our Rookseth detect version is stuck on a in a container step. So I reckon we're gonna have to get some logs out of there. The Kivernal that you're not familiar with is a policy agent which probably has some admission controllers in the system that we wanna take a look at later. That definitely is new. I did not deploy Kivernal to these clusters. So I think we we pointed that out. Qyvernal sounds like the main character in a

13:19 high fantasy novel. Like, you know, winter is coming, but where Qyvernal's curse? I I don't I actually have never I have avoided Game of Thrones for its entire existence, so maybe I shouldn't make jokes about it. Let's see. Okay. So where is my prompt? I think I lost my prompt somehow. Nope. There it is. I had a scroll. Excellent. Let's get some logs out of there. K. Log. Is it log or logs? I always forget. I think it's logs. I tend to use logs. But you'll need to specify the names. Okay. It's waiting to start pod initializing.

14:14 What happens if we, like So if we wanna get the edit container log, we'll need to provide the name of the container. So we're probably gonna have to Oh, wait. Describe it and see what's actually going on in that back, I think. Okay. Script pod. Cool. Failed to set up network for sandbox. Failed to find plug in cilium c and I in path up CNI bin, and there were a bunch of restarts on that Cilium container. Yeah. That's weird. Why would I'm not sure why our CSI plug in is It's a CNI plug in. They don't know what they have to Alright.

15:20 Okay. So the CNI can't send out the sandbox for this. Okay. Okay. So I think they've done I think Phil has done something to our CNI. I'm curious. How brave are you feeling? I am terrified, but that doesn't necessarily contradict the bravery. Do we want to just like kick a pod? Like kill it? Well, yeah. That's what I'm thinking is I I in order to confirm that this is not related to Rawkode and Seth and this is a CNI thing, I've got a funny suspicion that if we kill any other pod, it will also fail to star. It's kind of

15:55 what I'm thinking. Interesting. Yeah. Sure. Let's see. We have detect version. That one is, like it's literally just detecting a version. Let's let's not kill the operator one. What what what what's harmless. Which one do you want which one do you wanna boot out? Let's kill one of the metal l b speakers because we're not even using load balancing. So what is the what was that namespace called? Metal l b Yeah. System. Oh, system delete pod. And then Oh, it started just fine. It started. Well, let's take let's take a look at the logs. Yeah.

16:56 Sorry. I I guess I could type too. Alright. We'll we'll we'll take our fair share of typing. That's that's all fine. No. Is that that p g p Oh, yeah. No. That's true. Yeah. It's okay. We can ignore that. That's echoic thing. So maybe so it's the init container. Right? So it could be something Let me No. On you go. What's your what's your train of thought there? Let's take a look at let's take a look at one of those, like so we have the CSI plug in. Wait. What's? Okay. So first of all, what's the difference between CSI and

17:38 CNI? Let's let's talk about it. There's too many acronyms here. Container system interface? A storage. Storage interface. Okay. Thank you. And then container network interface. Right? Yes. Correct. Okay. Those ones have seen a bunch of restarts, but they're in the same but we might wanna we might wanna describe one of those and see why they got restarted so many times because that might be related. So the Rawkode namespace seems to be having some trouble. Maybe there's something messed up in the namespace. The other thing that we might look at is the operator logs. Okay. So yeah. You're right. If we'll

18:26 the 14 restarts, five restarts, 17 restarts. Yeah. Maybe there's something that is confined this Can we get the previous logs for that pod? We could. Although, we just had a spanner thrown into our our hypothesis. Philip has commented saying that he has not touched the CSI. Okay. Good. This is potentially just a red herring. Alright. Alright. Okay. I'm gonna so there there should be a workload running on our default namespace which hasn't Yes. I'm gonna redeploy it. Yep. Just to see if it works. I'm gonna do that from my own machine because that's just where the manifests are,

19:14 which I know you won't be able to see, but I'll do just workload and pop back over here. Uh-huh. So that was all unchanged. I really hope I'm working on a rate cluster. But we can't see anything. So if we don't see pods, I do expect to see Alright. We have a deployment but we don't have pods. Interesting. I guess Yeah. Let's describe that deployment. Let's see. Oh, okay. Rolling update strategy, 25% max unavailable. Available false. Minimum replicas unavailable. Progress deadline exceeded all the replica sets. Yeah. Okay. We've got a couple of failures here. It seems to be saying

20:36 I think it's unable to create the replica sets. Yeah. I also And there's zero pods. What if so we cannot okay. So we try to redeploy this? Can we So I think Can we do Yeah. So we tried to redeploy it. It's all unchanged. Right? It's the cluster thinks the deployment is fine. It looks like we do have the replica sets. We're not getting pods. We did see the kivernal earlier. So I think we're probably gonna wanna see what kivernal has done to our cluster. But I'm just gonna run and get events and see. Yeah.

21:25 Uh-huh. We've got a whole bunch of errors here. Okay. And the mutating web configuration. Always a pleasure. Conor ref invalid namespace. Okay. So it's it looks like it's a permission thing. Does not exist in namespace empty. So we're in the default namespace, not the namespace empty. Empty string. I I think. Right? It's I can't remember how long it's in space. It's not something I've ever tested before. I I think I would hope. I don't even know. Maybe an empty string should be default, but then that error message makes it look pretty weird. That error message makes it look like somebody

22:13 is just giving in a blank namespace. Why don't we see if we can what's the I can never remember it. It's the cluster Oh, no. API resources. Right? We can also just get all. I'm gonna grab that for mutating. Yeah. There we go. Let's let's see what is in our cluster. These seem to be twenty six seconds old. When we deploy. Wait. That's that's not when we deployed. That's, like, more recent than that. That is more recent than that. Okay. Let's describe these because these those are those are, a little suspicious, don't they? I know nothing about admissions policy. I'm just

23:11 used to having permissions everywhere. Does not exist in namespace empty string. So Okay. So it looks like a lot of noise, but Clients can fit. Although the policy on this is ignored. So even if this webhook failed. Although I wonder if this is mutating the namespace. Interesting. So the owner ref garbage garbage collector controller? What happens if we delete the Qiverno pod? I think that's a great idea. Kill it with fire. We'll need to delete the mutating webhooks. I mean, we can delete the kivernal part, but I don't think it's the mutating webhooks will still be there although

24:31 I guess the failure policy is What do mutating webhooks even do? I've I've gotta I've gotta So gotta ask Google. What these do? It just means that anytime that the API server wants to create a new resource within the cluster, it will send that request to all of the Mutant Webhooks who can actually modify that request and the cluster without us knowing. I suspect what's happening here is that one of these Mutant Webhooks is deleting the namespace on all of the pod objects, potentially. So I'm gonna say go away. Like, we do not want any of these cavernal policies in

25:14 our cluster. Oh, thought I missed it. Yeah. Oh, oh, not that shit. We can give that a few seconds and maybe we'll start getting a pod up. We may have to actually encourage that to happen. Why don't we delete both the replica sets and try to speed speed that up? And hopefully the deployment controller will yeah. There we go. There's a replica set. Although we still don't have our pod. Oh, well. Do okay. So can we describe the deploy again? See what happened? Should have some events. Scaled up replica sets to one. Okay. So So

26:10 So not quite an April fool, but Phil has commented that he installed Hiverno just for distraction. Uh-huh. Nice. Thanks, Phil. So I'm looking at I'm looking at some of this. So if if we are having a replica set that says we want this thing, but it's not coming up at all. Well, Waleed At all at all. We're getting a lot of comments from Waleed in the chat. Shall we jump over and ask him what he thinks? Sure. Where are we jumping over to? Oh, sorry. I'll I'll bring Waleed over here. Waleed, do you wanna jump into the Discord

26:53 stage and we'll get your advice? And she's right. I wish I knew exactly how long the latency was between YouTube because I could see that, and I don't know if he's gonna hear it within one second or, like, fourteen hours. I'll magic him along. So I see a failed create for the replica set. And then. Oh, there's Walid. Replica failure. Failed create. Hey, Walid. Can you hear me? Waleed. I can hear you. There we go. Gwen, can you hear Waleed okay? Yes. Alright. You were typing into the chat quite a bit there. What are you suggesting we

27:41 look at next, Waleed? I'm not sure because Phil says he installed the Kiverna as a distraction. I would like to check if there are Kiverna policies. So if you do okay, Kubectl get the cluster policies or c port to just It looks like we have a cluster policy called disallow privilege escalation. I'm gonna assume unless that has been very sneakily named in order to take our attention away from it, but that's probably unrelated. Alright. Okay. The latency on YouTube and Discord is a bit funky. So I'm just gonna mute that now, and Gwen and I are

28:29 gonna march on. Okay. Let's see. So I'm looking okay. So, basically, I see that, like, the okay. So No. We deleted all these webhooks. I'm I'm are these just filling up our event log and really making things weird for us? Oh, validating too. Right. Oh, okay. Look. There we go. Hot security policy. It's just clogging up the logs. Yeah. So Nice. That's I see that kavernal distraction as Phil called it was actually hating and obfuscating all the debugging stuff that we needed. Alright. So Eric Hating Pods is forbidden. Pod security, unable to admit pod empty. Okay.

29:38 There we go. So we have the RIC privilege policy. I think it's just part of Rawkode and Seth. I think that's okay. I'm gonna assume this forty two one which is such a random name is possibly doing something a bit funky. Why do I Describe that. Yeah. Can you do PSPs? Let's find out. Oh, you can. Nice. Okay. Okay. So this is It has a path prefix. I wonder. Will we just delete it? Sure. What does it belong to, though? Is it is it managed by something? Who created this pod? There's Look at the namespace.

30:36 It's namespace. There. It has no namespace. There's no metadata on this which I think it just means that bad actor has decided to drop this in and cause a little bit of carnage. So I'm just gonna sledgehammer approach and we're gonna delete that. Okay. Do we think we have a pod? No. Still no. I'm wondering who created this pod security policy. There are some extra operators in here. Aren't there? Right here. Cilium operator. Hubbell Relay default. Yeah. I want to know what the Cilium operator does. Maybe this is totally off. So that manages the CNI plugins.

31:45 I see. I see. I see. Okay. I think that should be all right. Okay. I wonder that's the thing with this show like, you think you're onto something and then nothing happened. I'm I'm going to delete the replica sets again to try and Sure. Try and encourage your pod to be recreated. Because if the pod security policy did block it, the replication controller, I'm not sure when it would try again. That's fair. No. Alright. Can we describe the replica set that we're getting? Yes. Oh, we still have the pod security policy. Pod security policy. Replica set controller.

32:44 I didn't think those are no. They're not named so I wonder Well because these ones I saw were okay and happy pod security policies are actually sneaky sneaky pod security policies. So what are the caps there? I don't know what that column is. I That's the that's the capabilities, Gavin. Yeah. See, I see caps and I think capybar That's just please send me capybarra pictures on Twitter, everyone. Okay. So sysadmin, Rawk privileged. And then yeah. I don't know what's let's let's describe those, I guess. Can we do that? Yeah. Well, I mean, we could have, like, a

33:38 tape, but there we go. Okay. Run as any manager q CTL client side apply. Allow privilege escalation false. Okay. Is this the one we're using, though? Is this the one we're under? What was the pod security policy? Hang on. Let me scroll back. Oh, gosh. I I scroll back, everybody. No. No. When you scroll, it's it's it's it's you're scrolling as well? I was. Yeah. Looking for any sort of hint. Okay. Pod security policy. Oh, wait. From replica set controller, can we, like, can we get logs from the replica set controller? Is that a thing?

34:41 Because I would like to figure out which pod security policy is broken. So we could go and take a look at the API server I think that would be a good idea. Because Sledgehammer deleting pod security policies feels like maybe now we're we're removing the debug abilities since none of this gets exported to any external logging system. Okay. There's more Hiverno. Yeah. Let's let's try and get Hiverno as a picture. Nothing. Okay. Why don't I just open it and then Okay. Alright. This seems to be all Kivera now. Authority set to yep. Yeah. Like Okay.

35:59 Alright. Let's just clarify. We have an API server. We got a proxy. We got a schedule. Everything we expect. Keyboard's running. It seems quite happy. It's it's happy. We're communicating with the API server. This is a permissions thing. Yeah. For some wacky reason. It's claiming a pod security policy was unable to admit our pod even though we have no pod security policy. And there's an empty list. Like, it it feels like there is an error message that's not that we aren't getting at the end there. Right? Unable to admit pod. Yeah. I'm I gotta say, I'm feeling a little confused.

37:16 What's the okay. So I don't know anything about pod security policies. So I'm just going to Nor should you. They're deprecated. Which which which Kubernetes version are we using? One twenty, I think. They're deprecated but not removed, but they're still available at the moment. Okay. Let's see. So this is our API server configuration. Yeah. We can disable pod security policies. Is that is that too harsh? No. No. Let's do it. We want this to work. Alright. Let's see if how that restarted. QP API server. Yeah. It did. Make sure there's no I'm gonna give that

38:37 a second and check the pod again but see if there's anything suspect in here. Do you see anything? Sorry. I know I'm scrolling fast but that looks normal. Yeah. Maybe. I love that everything I see is in the form of a question without being a question. Like, is it? Or What's a what's the insecure port thing? I don't know. I think that can be I think no. I don't think that does anything anymore. I think insecure ports, again, maybe Well, we have an insecure port equals zero, and we have a secure port equals six four four three. So I guess

39:34 I hate this show. What have I done? I broke the API server. Yep. Shit. Shit. Oh, wait. Oh, is it restarting? Oh, I think because I dropped out of them and it's just restart. Okay. Phew. So why do we I'm also confused. Why do we have two roles on the API server? Is that normal? Which ones do you see? Well, when we describe when we when we describe the I'm sorry. I just noticed that the Kube API server also has is when you just describe the nodes. Every time I acquire them, it restarts it. Oh,

40:48 sorry. No. Off you go. You can type. And I mean, maybe. What? What do we want? Yeah. Don't think it's restarted yet. Alright. We're get oops. Sorry. We're getting some advice from the audience here that there was an alert privileged false flag in the API server manifest, which needs to be changed to a low privilege true. So Okay. Wait. So we're having too many privileges. So a privilege true. Yeah. I'm I'm actually not entirely sure. I'm assuming the API server is saying that we are low privileged pods. I'm not entirely sure what that flag does.

41:38 I think I'll need to let the audience tell us because I have no idea. Oh, there we go. That was it. Dang. Alright. So we'll give that a second. So post the container image, we'll do our upgrade and hopefully that is enough to get us onto our second cluster. So there was not a lot changed What container creating? Still sneaky. I'm hoping it's pulling the image. No. Okay. Didn't we didn't we get that before? We were seeing that with the Rawkode stuff. But everything else seems to oh, no. We we've lost this we've lost our Cillium pods.

42:43 Oh, I'm such an idiot. When we ran get pods in the Cilium namespace earlier, there were only the two operators. Right. Now it's just occurred to me that, yeah, there was not actually any c and I agents running on the nodes. And now, I see one in an error state and I'm even more worried. But I should have picked that up. Why did that fail? They're coming, maybe. No. Enter. Logs. Let's see. Could not open. Subsystem, read only file system. So what are the system control files here? Not open. I p v four. I p

43:58 forward. I p forward. So sounds like we need to change the permissions on that. What? Let's try edit daemon set cilium. I mean, is that not a daemon set? Oh, it's in the wrong namespace. Edit. It's in the Cilium namespace. Thank you. I'm pretty sure I set that out loud and I continued not to tape it. That was useful. Okay. What is going on with this demon set? Enable debug false. So maybe, first of all, we wanna enable debug true. That may may be healthy. Wonder if these read only is true though, should just be removed. I don't know. Yeah.

45:17 I I I I don't really know if those are supposed to be or not. That's the thing with this show. People are cruel. Like, what was the file system? Was it The there was a read only file system, but the file system, I don't think were any of these paths. Yeah. I don't think it was it. I don't modules. Oh, wait. Tip Cilium config map? No. But there were underscores in the read only error. We can just pull it up again, can't we? Let's get Yeah. Yeah. It's like, proc sys net I p v four.

46:06 Wait. Disabled support for due to missing kernel support. I guess what we could always do, I'm guessing. Let's take a look at this filling and config. See if something in there has been changed, not that I would even know that I had. So the operator should monitor this conflict map and rule out the agents for us. So, really, I guess, we just wanna see I don't see anything. It's monitor aggregation interval. No. That looks alright to me. Yeah. Sidecar is still Yeah. I don't think I don't think that's been touched. So why is I mean, the only reason that it could

47:17 write the process net would be if it wasn't privileged. Maybe we Do we have a Cilium config map? Yeah. We just that was the thing we were just done. It doesn't really seem to have too much in it, which means we're kinda back to looking at those redone lasers. Now should we just remove them and just see what happens? Let's see. So this is the Cilium. Sorry. I just did some I could just redeploy Cilium. Is that cheating? I don't think so. Alright. Let's just do it. Oh, I need to tell it not to create the namespace.

48:29 Create namespace. Okay. We'll just reapply it. We'll see what happens. Unchanged, unchanged. The operator was changed and the demon set was changed. Okay. So, something does happen on those two. I'm curious if we edit the daemon set now if those read only file systems are still there. They are. Okay. So something had changed in here. Something had changed in the operator, And we're getting running Acetym agents. Yeah. Good. Phew. I was getting panicked. I'm curious what you changed, Phil. Feel free to let us know in the comments. So does that mean Does that mean we have our pods?

49:27 Pending. That's different. Right? Was that the same pod? Oh, no. No. Pending. Yeah. Okay. Node role Kubernetes IO master. Yes. I was going to say here when we did k get nodes. I think I just looked at it a little bit. But see this? Uh-huh. The control plane node has two roles. Why does it need two roles? And Why does it need the master role? I don't think it does need that. I think that's the a legacy thing because it was renamed from master to control plane. So it's maybe just left around because there's still things that depend

50:25 on that label being there. But we're in Kubernetes one twenty. Seriously? I So Taint zero to four nodes are available. Nodes one node had taint, but the pod didn't tolerate. Okay. That's fine. Three, too many pods. Too many pods? Does that does it say too many pod? Oh, okay. Yeah. Ah. Rachel, let the the pod limit on the APIs or the Is it the cubelet or the API server? I don't know. Let's find it. I think it's the cubelet, but I'm gonna check. Yeah. Okay. It's not in here. Let's find the cubelet. So the cubelet is starting for system d.

51:25 System d. Oops. Sorry. I'm about to drive. Oh, no. You feel free if if you wanna open the Qubelet. No. Okay. So this just tells us where all the configurations are. So we'll just pop these open one by one. Let's look at Q bled config. Yeah. Nope. That's just the thing. Let's try this one. Pod. Anything. Nope. Okay. Next. Could be QPDM flags. No. Which one did I miss? Etsy default keyboard? User bin keyboard? No. Maybe it was that first one. No. I must have missed one. The one for connecting to the cluster. The kubelet

53:01 we just opened, I've opened it three times now. The controller manager is not the one I'm looking for. It has to Why am I so bad at this? CD bar lib kubelet config. Not there. Maybe I just don't know what the flag is called. Maybe I should just read them all. Can we restart the? Yeah. But we need to find that config first. Are you sure it's max pods? Unless it's not the kubelet under control. Oh, I'm such an idiot again. It's gonna be the kubelet on the worker nodes. He's modified the kubelet on all the

54:12 worker nodes so that they can't run anymore. I've joined another session if you wanna pop over hopefully. I'm finally starting to come to my senses a little bit here. There we go. I am having super I'm going to drop off, I think, and reconnect because I cannot see. It's very fuzzy. Your screen is incredibly fuzzy. Right? Oh, no. Yeah. Feel free to drop off. I'll some Be right back. While you do that. Perfect. Var lib shiplit config. Oh, a different number on each host. That's fun. Thanks, Phil. Welcome back. Hooray. I can see you. Okay. I have removed all those very sneaky

55:10 max pod settings. And I'll probably need to restart the kubelet on all of these. We'd have to start stacking salt stack and all these machines so this is easier. Okay. All those kubelets now been restarted. Uh-huh. And hopefully our pending changes. I hope so. I hope there's nothing else on this cluster. Failed to sync secret cache. Timed out waiting for the condition. I'm hoping those are old. Yeah. There we go. Oops. Sorry. Off you go. I would I won't touch the keyboard anymore. It doesn't log anything. It's a it's just a really Nice. Okay. Do you wanna modify

56:08 that deployment and change the version to v two? Which version? On the image. So just k edit deploy clustered. API version. No. Change it on the container image. So we're gonna upgrade my application so we can see my events. I see. I see. I see. I see. I see. Container image. Yeah. There we go. On help container registry. My new favorite registry. Uh-huh. So we should be able to change that v one to a v two. I'll put forward from my local machine. We'll see the dance, and we will move on to our second cluster.

57:08 Thanks, Phil. Let's dash f. Okay. Our new version is almost running. There we go. Okay. I'm gonna Ariminating? Okay. I'm port forwarding. I'm now gonna one, Pat. Something weird networking there. No video supported. Well, we're not getting to see my dance this time. That's annoying. Alright. Fixed. Let's move on to our second cluster. Alright. How was your first cluster? Whoo. Steve. Alright. Alright. Close all of my 14 tabs. We're now on Cluster 12. Thank you again, Phil. For something that you said you'd went easy on is, took us a little bit of time to find what we got there.

58:00 Kluster 015

58:37 Second broken cluster is by Guy Templeton who is the co chair of SIC auto scaling and an engineer at SkyScanner. And happens to live very local to me. Hey, guy. How are you doing? I have created a Auto scam. I've created a teleport session on the control plane of Cluster 12 if you wanna pop over. I will set up our cube controls stuff for alias k. I should really just bake these into the image, shouldn't I? Fascinating. Is it broken again? Nope. Wait. I keep doing I keep connecting to my own session. Alright. Here we are.

59:26 Maybe. It's thinking about it. Oh, excellent. Hi. There I am. I will give you the honor of running k get nodes, get pods, whatever you fancy, and see if we have oh, two clusters of an API server. People are being very kind. Although at this point in time, I actually think it's easier for me to debug the API server than stuff Yeah. Yeah. Alright. Okay. So we're looking for the cluster deployment. We are. And that is running. So is Postgres. And there are some Smerle tools, which what is Smerle? I have no idea. I am gonna Let's

1:00:14 ask Google. Smerle. Smerle, connected counters for social media. Surprisingly powerful communication tools for your business. I wonder why we have Smeril on our cluster. Sam count display in real time is my first result. Alright. Shall we run get pods in namespaces and see Yeah. Oopsies. See if there's elephant's effect. Ping back in and see some of those logs. There's three completed jobs. So why are there jobs here? There's this system, cube scheduler, speaker. Okay. So this this looks like somewhat less terrifying. We have a pending sleep pod. So that just looks like something that is sleeping forever.

1:01:36 It does. And timing out due to, like, health checks or something. It's also the weird smell stuff, which probably shouldn't even be there anyway. Yeah. Let's do Let's describe the deploy and see why there are extra pods in it. Yes? Yeah. Yeah. Why let's see why this one's pending, I guess. Oh, no. Yeah. Custard is fine. It looked like it was running. Yeah. Should we just try and apply the update? Well, we should switch it to version two. Maybe maybe this cluster's working. Rolling update. Yeah. Yeah. Sure. Alright. Progress deadline seconds. Oh, I can't scroll. Right.

1:02:15 Failed App Deployment: ETCD Request Too Large Error

1:03:11 Uh-oh. Could not be patched. Request is too large. FCD server. Oh, shit. Okay. I don't like ETCD problems. Deployments.apps could not be patched. Request is too large. So I'm wondering. So what what are these so we have these extra pods in the namespace. Right? How big are these pods? And they've been sitting there. Right? And the sleep pod has a pending status. No logs. You may want to scrape it and see. Yeah. Exists tolerations. Not ready. Execute exists for OP. Exists for three three hundred seconds. Okay. So five minutes. Tolerations. Unreachable. Execute. Something tells me these are just noise to

1:04:37 get us looking at the wrong thing. Maybe. So when we try to modify the clustered deployment with the new version of our image, we got an error from etcd saying that the request was too large. So Yeah. I think we wanna go check our etcd manifest and maybe the STD logs. Yeah. We should probably look at the STD logs. Right? So cool. Where do those live? Is it varloggedtd? They'll be under varlogged containers and then there should be an STD file hopefully in there. Is it directory? Yeah. Yeah. That's the one. Alright. Lots of hell.

1:05:33 Okay. Wow. I hit control c just so we can wow. Health okay. Okay. There's a ton of health checks here. Yeah. I Somebody is sending so many health checks that right? There are like what? Every ten seconds. Yeah. There's a health check. Okay. That's okay though. That shouldn't So I think this is strangely Okay. Good because it means that STD probably isn't broken and just misconfigured. That's kinda what my hunch is So why don't we jump into slash e t c slash kubernetes slash manifests? Kubernetes every time. And if we do yeah. Take a look at that cd.gam.

1:06:17 Fixing ETCD Max Request Bytes Configuration

1:06:37 In fact, let's see if it's been modified. Do I do a l s dash l and see the timestamp on it? Sure. Oops. I really hope it's been modified. March 30 Yeah. Last night. '23. Yeah. We see you guy. We know what you're doing here. Alright. Let's pop this open. And hopefully, it's as simple as a flag he's passing them to restrict the size of the request. Do you see it? I hang on. Max request bytes five. Shall we just delete that line? We can either delete it or make it really big. I'm happy to delete it and

1:07:24 put it back if we need. We'll remember my request by or comment oh, no. If you yeah. Comment at it. Let's do that. Good idea. Can we do that? Does that even more how do we comment? Oh, wait. Wait. Wait. It'll be a hash symbol. Yeah. Or a pound Sorry. If you're American, which well, the audience maybe. Never understood. Oktothorp. Oktothorp. What? The German word? No. That's the that's saw you were telling me the German word for, like, a pound symbol or some or a hash symbol there. Yeah. The German symbol for the pound is the same. Oh, alright.

1:08:00 The number. The number's the thing. Yeah. I only know this because my grandmother's old recipes were still in pounds. Got it. A German pound is 500 grams. How much is a British pound? I mean, like, the weight, not the money. I have no idea. I wish I knew that. I'm gonna modify deployment just to see if it works. Maybe this is gonna be nice and simple. I always say that was such hope. Well, the good news and bad news It saves get But it's not doing it. But it didn't get edited. So the edit does show it, but yeah,

1:08:54 we're not Shall we redeploy it? What possibly delete the pod first? Yeah. Why don't we delete the pod and then I reckon we're probably gonna have to check replicate could be the same problem as the last one. Probably it could be pod security policy or mutate webhook. But let's delete the pods and see what happens. Hanging. That should be okay. I think it's thinking about it, hopefully. I'm gonna control c it and just see what's happening. Yeah. I think it's trying. Okay. Why Terminating. Why don't we Why is it Why is it terminating forever? Either something It won't. Mess with again or

1:09:06 Pod Stuck in Terminating State

1:10:05 it's just being particularly slow. Let's describe our replica set. Okay. No. We don't see anything here. There's no PSP. And I can never remember the mutated web. I think this is the same as the last one. It feels like it. Right? Sure. It's not No. Well, it's a little different though because the replica set, like, it's just literally not leaving. Like, do we not have delete permissions? Like, why is it taking forever to terminate? Don't do. Okay. It's definitely Warning. Node's not ready. From node controller. Node is not ready. Let's let's describe the nodes. Yeah. Good idea.

1:10:19 Node Not Ready / Kubelet Issues Identified

1:11:19 I oh. Yeah. Alright. So these are all of the nodes. Maybe I should just describe them in in turn. Non terminated pods, 15 in total. What? The kubel isn't posting node status anymore. There's something wrong with the kubelet. Do you see that? Memory pressure. Kubelet stopped posting. Yeah. There's something wrong. Okay. Which was also actually, the cubelet would explain why we're not getting our new pod created either because Yes. It's just well, it's running. There's that's a I guess that's a star. This is we're still on the API server node. Oh, yeah. Why don't you hop over to the other

1:12:32 node and check on the cubelet? Good idea. Okay. That's good. I guess if we can get it working on one, we should be able to see it. It depends where it's scheduled. Let's check that first. So get oh, boy. So this terminating pod was scheduled on l zed. Is that the one I happen to go on? No. Of course not. No. Okay. Let's go on to l zed. And there's no kubelet. Oh, that's not my it's a system d service. So Yeah. Exact start is missing. I don't know if the order there is some No.

1:13:05 Investigating Kubelet Systemd Service File and Logs

1:13:37 I don't like the look of that, so I'm gonna remove it. I don't want an enthusiastic star. It looked alright. I would at least unless it's crashing. I see a status. Right? We should let's trust system d. Okay. It says oh, it's telling me that it's been really Active activating since Thursday 1810. That is literally right now three seconds ago. Right there. Active. Keyboard service. We just Oh, yeah. Service has more than one exact start setting which is not allowed. I just removed that. Right? Yeah. Why did maybe my daemon reload didn't work. And then restart Can you restart kubelet?

1:14:33 Unit kubelet service has a bad unit file setting. More than one exact star. Okay. So I'm I'm assuming one of these other files. There must be another system, the overlay. There's a way to get that as well. K. You know, I think we need to like stop the cubelet first. And then restart it? Because environment file. There's a lot of environment file. This drop in only works with cube, Adam. K. Okay. Let's stop to keep it. Why does it keep Wait, I did modify it. I put that exact start back in because I wanted to confirm the message.

1:15:55 Stop kubelet. Let's start kubelet status kubelet. How did I get that without the pager? Okay. So this is complaining now that who knows? Let's go to logs. Failed code accident exception failed. Oh, there's quite a lot in here. Right? Oh, god. It just speed up on our screen. Go routine. I don't see an actual error yet. And ah, there we go. So the CPU, CPS quota period of one second requires a feature set feature gate to be enabled on the kubelet. Okay. So we can either enable this feature or enable. That makes sense? I I I don't know.

1:17:00 Finding Kubelet Feature Gate Error (CPUCFSQuotaPeriod)

1:17:17 I've I've got Yep. Yep. I've got Kiplet blindness now. Yep. Let's go and add that custom. Are you in this session? I am. Sorry. Maybe I am? Wait. Hang on. No. I'm not in that session. Sorry. You're in the node. I I haven't joined you. LZ. Right? Yes. There we are. Okay. That we broke. Okay. We do have the CPUCFS quota period. Should we just comment that out? Yep. And we'll restart. Maybe that'll help. Although I don't know if the fax is just to disable the feature or enable the feature gate, but we'll just we just wanna

1:17:25 Fixing Kubelet Config Error

1:18:22 get the system all running. It's doing stuff. Yeah. Let me check on the other thing. So, on this side, both Postgres and our clustered workload are, terminating. Bunches of things are terminating. Uh-huh. We are down to pending. Interesting. We broke it. Okay. I'm gonna check the kubelet on the last machine. Did I join name two? Eight nine. No. Okay. Let's just see if we have a kubelet on here. Yeah. We do have a kubelet. Okay. So we now have a kubelet running on all of the worker nodes. However, we're still having a problems terminating and scheduling

1:19:00 Redeploying App and Discovering Deployment Failures

1:19:46 the cluster pod. Right? Available. Run get pods again or just does it still just say terminating? Oh, no. It's not. Anything? That that's better. Okay. So why is Right. Postgres pending? I don't know. And oh, thirty four seconds. You wanna try to describe and post this? May maybe we're just being impatient. Maybe it's it's coming back to life. Every time. No events. Absolutely no information there. State till set. Node, none. Stat Oh, there's no scheduler. Where did you see note none? Note none. Right here. Oh, yeah. Yeah. Okay. So Oh, sorry. I'm, like, highlighting it. Do

1:20:39 Confirming Pods Have No Assigned Node

1:20:55 me use the screen sharing. Yeah. There's no node, and there's no IP. So we do have a scheduler running. I'm gonna grab the logs of it. Copy and paste. Schedule appears to be okay. Shoot. Okay. So our schedule is running, but we're not getting pods assigned to a node. I'm not sure. I'm looking at this. And the the notes are reported healthy. Right? Yeah. Yeah. What about taints? Do the notes have weird taints on them We can that wouldn't allow us to schedule things? Invalid disk capacity. Oh, yeah. Good catch. And then so note is now

1:21:17 Finding Admission Controller Errors (Mutating Webhooks, PSP)

1:22:39 so So our Updated node allocatable limit across pods. K. Sorry. What are you thinking? Yes. So when you pointed out the valid capacity zero and image fail system, that's obviously a problem. And that's on the let's notice that. L zed. L zed. And the other one oh, no. Wait. No disk pressure. Sufficient memory. Okay. So the other nodes aren't reporting the same. But it's just that one node that seems to be reporting that. But that wouldn't explain So it only happened once, which means it's not recurring. It's not repeated. Right. Node has no disc pressure.

1:23:56 Five times two over It's happened a couple times, though. I know it has no disc pressure. Right? And it still has, like, non terminated pods, Five in total. Okay. So there's five pods on there. Yes. That makes sense. Limits Thanks. Memory. Julia, annoying me. So this is really weird. Why are so I'm looking at the allocated resources, and it's saying it's 0%. Like, if you if you look at the note description, the request and the limits, like, 200 I mean, those aren't, like, super huge. Right? But, like, it's all 0% of the note. Why? Is it just

1:24:46 Deleting Kiverno Mutating Webhooks (Distraction)

1:25:14 does it just get, like this looks really weird to me, but I I I might not know anything. Like, CPU requests are zero for the cube system? I think that's just because the resources that are applied to the cluster are requested. So small. Resources. Okay. I'm not really doing Okay. Sorry. I was I'm just looking at that. 00% looks weird to me. In production, I see, like, you know, massive percentages. So like that. Okay. So the thing that's really bothering me is if we describe in fact, we're not even getting yeah. Well, let's describe the postage for And

1:26:05 and randomly these clusters died. I mean, these pods, not clusters. Randomly, pods just, like, went away and didn't come back at all. Like, not even pending. Was it? Yeah. Yeah. Okay. That's a good point. Let's go back to our we must be missing something obvious here, I'm hoping. Let's describe this new replica set. Why is nothing happening here? Okay. So this is very similar to the last cluster. The pods aren't showing up. They're not being created. Let's check our events. No. It is not ready. It says and tool sleepy too. No. It is not ready.

1:27:03 Tools. No. It is not ready. And then post pop yeah. It all just says node is not ready. Node has sufficient sorry. I'm just reading a starting Kubelet. Why is the Kubelet starting on that node? Should we ask guys for some help since we're approaching it? Like, look at the logs on this replica set. Why is the Kubelet not what why is that even on there? Invalid capacity zero on large file system. Yeah. That's the batching guy. Scrolls. And how does I never use the time. I'm fine. Okay. My coworker, who I'm supposed to be meeting with

1:27:29 Hint from Guy: Revisit Scheduler Configuration

1:27:58 in six minutes, is currently waiting for his vaccine. So he won't be there in six minutes. So we just oh, we got the cluster events. Alright. And then we started the kubelet. Right? That was eight minutes ago. We established weird. No. It's not ready, but now set up failed. Failed mount. Failed mount. There it is. Eight minutes twenty four seconds. Yeah. I see that. Killing. Stopping container tools. Yeah. The whole note is not ready. That one, we fixed. But, the volumes that are in the Postgres pod are not valid. And then there's a tape manager eviction

1:28:07 Identifying Pod Security Policy Errors

1:29:05 for the Smurl tools, which I don't know that we need to worry about those necessarily. But let's fix the Postgres one first, which is a right there, taint man oh, it's also a taint manager eviction right there. And then a failed moan. Oh, that's twelve minutes ago. But but yeah. Where did you? Twelve minutes ago, pod right. Slightly up. Marking for deletion pod taint manager eviction. Why why did it get why is that a taint manager eviction? We just tried to restart the I don't see tint manager eviction. Go up go up to go up to

1:29:55 two lines twelve minutes ago. Two lines from your highlight. Yeah. Oh, scroll moving your cursor down, down, down. Oh, yeah. Okay. Alright. There it is. And there's a bunch of those. And that happened when we deleted it. Why does it report a paint manager eviction when we just wanted to delete the thing? Didn't we just, like we edited it. We edited the let's look at the let's look at the deploy description. That could have gotten edited. Right? I'm kinda worried. I don't even know. This header here seems really consistent, don't you think? Failed to set secret cache.

1:30:00 Deleting Suspicious Pod Security Policy

1:30:55 Yes. Time to wait until the condition. Wait until the condition. What the hell is the secret cache? I'm gonna Google this. Yeah. Let's let's Google that message, actually. Good call. Beast me guessing for the next five minutes. Partner is Rvac permission. Joy. Let's get ahead. Are you there, Guy? Forgot what's it. Latency. Oh, so they shouldn't be because he's in the thing. What is the point of a live audience for support if they're not available there to give you support? Cassette. Desired two, current one. Yes. Thanks for showing up. This is all the slightly freaky with the

1:32:01 Pod Security Policy Still Blocking: Checking API Server Config

1:32:37 wait and see. Hang on. I'm gonna mute the YouTube. Alright. Give us a hint because I'm over I think I think you've got a bit sidetracked on the the errors and the events. There are two things still going on here, just to add to the confusion. One is why are pods stuck in pending and not getting scheduled? You are on the right tracks having a look at the scheduler. I think you want to have a closer look at it. And the other, you should see once you fix it so it starts scheduling pods. Okay. And was the CPU CFS quota thing being

1:33:26 commented out if I made that worse or was that a fix? No. That's fixed. I I was trying to I I was trying to make a pod a node very slow by putting the CPU up really CPU, CFS quarter window up really high. The API server won't know. Well, thank you for the hint. I hate you. I'll speak to you later. Thanks, guys. I don't hate you. This isn't this is amusing. I I I enjoy watching Scottish people hate doing things because it's it's it's wonderful. Okay. So there's something let's bake about our scheduler here. In fact,

1:34:03 Discovering Custom Scheduler Name Configuration

1:34:07 yeah, we I think if we in fact, we can see here all these files have been modified. So I'm not sure how we missed Yeah. Air and air. So this is the scheduler manifest. Let's see if we can spot Oh. Special scheduler? I'm gonna remove that. Thought We we should be overriding the scheduler name. Special as it might be, not today. Initial delay seconds, time out seconds. Okay. Let's let's just change one thing at a time though. Yeah. That's great advice. Definitely. Although, I yeah. I I get a little bit gun hold sometimes, so it's good to

1:34:23 Fixing Scheduler Configuration

1:34:56 change one thing at a time. Alright. Do we have a scheduler? Maybe it just takes a minute. Alright? We go. Zero to one. So that that affects our scheduling problem. Maybe. If it comes Has postgres zero been assigned to node yet? No. Okay. So let's pull up the So maybe kill the replica set. Kick the replica set. Yeah. It's still not running though. Yeah. It's not oh, yeah. It's not healthy yet. So it may so we just acquired the leader lease. So in theory, this has been scheduled. Alright? Alright. Cool. It has. Oh my gosh.

1:35:30 Cluster 12: Pods Now Scheduling & Starting (Scheduler Problem Solved)

1:36:04 It's on that node too. Our favorite node. Alright. So we'll override the scheduler name. Obviously, I beg no no. We don't do that. Sweet. So now we are still not seeing Oh, sorry. On you go. Clustered, and we're not k. Describe. Nope. Okay. Scaling replica set. Scaled up replica set. One desired. One available. One unavailable. Alright. So Yeah. I still think That's our that's our clustered. Yeah. Check those replica sets again. I find it really weird that we're like, it's not like the pod has been created and it's unscheduled. It's just there's just no pods. Right? And

1:37:18 Yeah. We're still desired one on that second replica set. Okay. So there's something wrong with the product. Wait. What's happening here? Three days, eight hours, we have a replica set. But the pod but it says desired and current, but I don't see any current pods. Yeah. I think he's What's happening there? I think he's modified the controller manager, and it's making all this It's lying to us because I don't see any pods. What? It's not. No. You're right. Right. You're right. So there must be something in here. The fact that they could the fact that

1:38:00 the replica set isn't even creating the pod for it to not be scheduled or rescheduled is a big warning sign. So what has our sneaky Scottish person done in here? Look at all those controllers listed here. That's a large attack surface to change stuff. What else is there? Can I scroll? No. Maybe I can't. No. That's right. Oh god. Weird. That's I'm good at them. I promise. Okay. What's happening down here? Scheme, h t t p s port. That seems fine. So I I think it's the that controller's list. One of the weird things you can do

1:38:06 Disabling Pod Security Policy Admission Controller in API Server

1:39:12 with Kubernetes is I only know this because of an earlier episode is is that you can do weird shit like, you know, disable the replica set controller and stuff. But one of the other things is you can just tell it to run them all like this. So I think he's purposely typed all of these out to kinda hide the fact that the replica set controller is missing. And in fact, if we set there is a replica set replication controller. Replication. Is that same? No. I think let's say that. Let's do the star. Like, what's the what's the worst that can

1:39:50 happen? Let's let's try and let's try and spin up a version. I mean, somebody could later take over this cluster after you deleted it from Well, there is that. I'm gonna I'm gonna ignore that. We just I didn't that wasn't me. And then I'll do my impatient thing when I run PS every fourteen seconds or every fourteen milliseconds. Okay. We have a controller manager. And I better see a pod in a second. I wonder if we should delete like build replica sets and try and encourage it to Yeah. I was thinking that. Although, why is there an ancient replica set?

1:40:42 It's gone there. Oh. Oh. And we have a Okay. Look at that. Wow. And it's talking about terminating pods. It's terminating pods that don't exist? Interesting. We didn't know about before. Alright. So that's the hopefully, that's now pulling the image. And it's running. Wait. It is running. Alright. Let's go into Oh, yeah. Did we deploy our update? I can't remember. Yes. Oh, we we tried to. Right. Okay. So we can port forward cost. There we go. It should be edited. Yeah. I think it was one of the first things we did and then we realized

1:41:09 App Running But Cannot Connect to Database (Connection Refused)

1:41:35 that everything was broken. Yes. Although we have a second. At least like we have a second problem though. And that Oh, no. Wait. It it can't hit the database service. So there's potentially one more problem or maybe it's unrelated. So I'm just gonna describe the service and see if we have any endpoint. Shit. We do. Is this an intended guy before I start Wait. Let's let's double check let's double check that we did in fact edit the deploy. Yes. Oh, yeah? Okay. Okay. So why can it not speak to a database? Yeah. It definite it definitely can't speak to

1:41:38 Cluster 14: Pods Start Running (Initial Problem Solved)

1:42:49 a database. Let's look at the logs. I don't think there are any. You can try. What? It's a very terribly written Rust application that I wrote. So Oh, I mean, I wonder if it can't connect because they're out of sync. It looks like it's DNS. It's always DNS. Right? It says failed to look up address information, name not found. So let's check Kube DNS is working. Do you wanna run a get pods on the Kube system namespace? Have to DNS is running. Well, why do we need to So DNS is really bad for logging. In

1:43:15 Checking CoreDNS Pods and Logs

1:43:53 fact, I don't expect to see anything come out of that container. But it also says not found. Oh, you're it's just missing the cube system namespace flag. Ah, right. Thank you. I'm like, it's broken. Right? Okay. I'm I'm losing I lost I lost connection here. Ah. Wait. Maybe? Go ahead. You type. Yeah. Was just gonna say it's a lot easier to spot tables than missing words when you're not the one typing. No. I lost connection to the I think I Oh, sugar. My Alright. Connection refused. Failed to watch v one service unknown. Get namespaces. So it looks like it is

1:44:49 trying to speak to the API server. I'm getting connection refused. Maybe not. I don't know if there should be a selector on that service. Let's describe the service. Yeah. Okay. So it's it's fine. There is an endpoint. The service is available on port four four three. Why can core DNS not speak to our API server? Because core DNS never talks to anybody. What? Okay. So this is a Selium cluster. Okay. Which means there are no network policies. Okay. Sugar. Why are there two pods for cordianess? Why are there two cordianess pods? It just runs for redundancy.

1:46:10 That should be okay. Maybe. Okay. So core DNS has a pod. It's scheduled. It's running. It has no connectivity across the cluster. Shit. And there's a connection refused. If it's not a network pause, it may be either kip proxy or the core DNS deployment could have been modified in some way. Okay. When was it edited last? How would we know that? Is it in the manifest? Yeah. If we do a if we do a cube edit deployment cube dash n cube system core DNS, we should be able to look at the metadata or status that may have a data in

1:47:06 it. Failing that, I think we're gonna have to either look at yeah. I mean, it could even be IT tables. It depends how evil Guy was feeling. Yeah. So there's a generation four on this. So I think it has been modified. Replicas to generation four? Tell me more about that. Oh, revision four. Yeah. Every time we modify it, the generation Creation time stamp. That was two days ago, though. 03/29. I'm just scrolling and hoping something jumps out at me. Schedule new name is the default scheduler priority class name system cluster critical. I don't know what that means.

1:48:18 Okay. I wish I had an idea. That's weird. No events on that? The endpoint is a public IP address? Get I I don't think it should be. So if we do a get pods wide our IP server has. There's the one forty five though down there. Packet cloud controller manager and cube proxy. Okay. So some of them are getting. Okay. Wait. Wait. Okay. So wait. Wait. Hang on. The cube proxy for the cluster that we were trying to schedule on, L Z 6 N Q, am I am I, like, super out of whack here? Because

1:48:41 Noticing Pods/Services on Different IP Ranges

1:49:54 that is the only one that doesn't have a public IP. I'm I'm I'm just sorry. Like like, right on the third one from the bottom. Right? But the one right above it is the other worker node, and it has a public IP. Okay. So you're So the service let's look at the service, the Kubernetes service. I think wait. Kube proxy, right, on the broken node. Are we on that node? No. We're not. We're on the control plane node. So I think So we should look. See that? I I think these are okay and these why

1:50:47 does our core DNS have a different structure? Why is it a 19216845? That's not a CIDR address that we're using in our cluster. Right? Like, our pod IP space is 1012. The public IP address is I think is typically okay. This is fucked. Right. And also but, like, I bet we can't reach that particular node because it also doesn't have I should be better at this by now. So right now my hypothesis is that the IP address of these pods has been some something ski with there. Right? That that shouldn't be the IP address of those pods.

1:51:47 So Yeah. I I don't know. Why is it getting those IP addresses? Let's kill them. How'd you feel about that? That sounds great. And see if it comes back with those funky IP addresses again. I I I'm too quick to delete pods. I know it's a terrible practice but sometimes I think just let's just do it. Okay. So it has come back with those IP addresses. Now if it's on a if it's on a a network of its own, it's not gonna be able to reach the IP address of anything that within the cluster. Like it just wouldn't have the roots

1:52:42 at all. So what So what is the I'm having to just to delete all these the shit and the default namespace that we know shouldn't be there. Unless you've you've got some ideas. I'm drawing a loss again. Is it You happy with that? Yep. Oh, no. Wait. They're of course, they're deployments. Okay. Good. Gone. Let's wait for them to go away. The only I I don't even know. I'm completely guessing. I think hopefully we can delete core DNS. I I don't think those IP addresses or ranges. I think something's messing with it. We did look for admission controllers

1:53:50 earlier. Well, I mean, we also have no trouble. Wow. There's like, it's telling us twice that it is terminating. Look at that. For the same one. See that? The the the the deletion log is weird. Yeah. But I just blame Kubernetes for that. Yeah. I'm gonna delete this and then you're on your own. I have no idea. I'm just gonna I wanna see core DNS come back with an IP address that I know exists in my cluster. No. Okay. So the CNI plugin is responsible for giving everything an IP address. Let's check the So then config map.

1:54:40 Checking Cilium Config Map (Pod CIDR vs Host Network)

1:55:01 There Oh, there's a cluster at P V 4 Pool Of 1921680016. Is core DNS I don't know what that means. Means I may have done something very bad. Shit. I think he's changed the cider of our network at Medflight. Like, right now? Okay. I just I I I I'm so mad at When when I see a get pods all and I'm seeing 10.12 here, I'm seeing 1 9 twos here. Are these node related? Let's do a grep for 89. Wait. What did I get? Oh, that's it. Silly. Okay. So this is all of our pods

1:56:35 with IP addresses. I'm gonna grep on a worker eight nine. Nothing. That's not right. Oh, no. A dash a 89. Yeah. See, we have multiple CIDRs across I mean, that just seems fucked. So let's check out I don't know if you're still watching my screen or if you're watching my screen or if you're on your I am. Alright. Okay. So Yeah. I'm gonna pop open Versus Code, and then I'm gonna remember which CIDRs are used for cluster creation. So here's the no. Cluster dot YAML. Oh, there's gonna be a token in here, but we're just we'll deal with it.

1:57:25 I'm super out of my depth here. Cider blocks. Yeah. Okay. So 192168. I weirdly enough then. However That that looks right then. But only for core DNS and a few other things. We have a whole bunch of other stuff on its own network. Right. I don't know. So we just we're gonna have to just check all the obvious stuff then. Cluster size Oh, service cluster. Service cluster IP range. 172 down there. Service cluster IP address range. I don't know what that means, but I all all I see is different. It's different. I think that's just for Kubernetes

1:58:07 Starting Debugging on Cluster 12 (Broken by Guy)

1:58:20 and core DNS. I'm not even sure. Okay. Okay. I'm confusing myself. That was wrong. 10 or 12 is the actual bare metal network. So I'm going to describe this pod. I bet you it's just got both. Rawkode. Is it just got multiple IP addresses and get pods that's just pecking a random one to show me? No. That's it's only IP address. Which means it's either running a host networking or doing something else. So I think that's a completely different thing, and I'm debugging the wrong problem. So which is very standard. So now back to this. Core DNS cannot

1:59:00 Initial Cluster 12 State: App & Postgres Pods Running, Seeing Unknown Pods

1:59:16 speak to the Kubernetes API server. It's getting a connection refused. I don't think it's our back. It literally cannot hit the service. We've checked the service. It has endpoints. They look alright. What about the cube API server? Yeah. Let's check. Because that one has been up forty three hours, which is not the same time as when the cluster was made. Well spotted. Okay. So I mean, we could just look at this as a static manifest. I shouldn't look at it in the cluster. So we're back to suspecting something in here. Mhmm. Kubernetesdefault.service.cluster.local is a service account.

2:00:13 Do we have it? Why? Does that make sense? Why does a keyblade have a preferred address type? Enable admission plugins, node restriction. Is that normal? Yes. I've fallen for that before. I tried to delete it, and it turns out that is normal. Normal. What about the advertised add address? Yeah. I'm looking at that. So we wanna correlate that to the service IP address. Right now, I'm gonna assume it's okay. But definitely something I'm gonna check-in a moment. The rest of this looks fine. I mean, we're speaking to the API server. Right? It it it is working.

2:01:10 Controller and other server seats are scheduling pods. They're speaking to it. It's only Yeah. It's only core DNS. Is there something funny about core DNS? Yeah. Let's go back into this. I feel like we've missed something. I've missed something. Oh, I missed something too. So where are we in right now? This is the core DNS config. I'm really looking to see if this networking stuff is wrong. Now he could have changed the security context. I'm not sure what core DNS needs, but I'm assuming that being service is probably alright, etcetera and all. It doesn't run a DNS

2:02:12 server. But it drops all? The DNS policy yeah. That should be alright. Although, I think there is a DNS policy of host firewall. Wait. Wait. Okay. So default is not the default. You know what we could do? CoreDNS Kubernetes manifest. What's this supposed to look like? And you can see I've been on this before because clearly people keep fucking with CoreDNS. But so what was it we said was potentially suspect? We wanted to look at DNS policy. Yeah. Default seems to be okay. Okay. That's fine. What was the other thing? I'm not sure about enable service links. That sounds bad to

2:03:13 me. Yeah. Because that would imply that it's maybe yeah. I don't There's something that's missing. Priority? What about priority? Two bajillion? That looks huge. What is that even? It's not set in here, but the priority class name is. So, yeah, let's let's comment that. The class name seems okay. Preemption policy isn't set so we're gonna take that out too. Schedule the name security context. Like you said, let's change one thing at a time. Let's let's see if any of that helps us whatsoever. I really wanna be on the viewer side of this one one week. Right? I just

2:04:06 not be on the side of it for sure. I'm gonna have to get someone else to host this. Okay. So we're deleting core DNS. I'm hoping this is the enable service links or that priority crazy big number whatever that's doing. Maybe guy would join us if it fixes it and tells what the hell that was doing. Let's see if I can port forward again. I just forget maybe it's still broken. Should we restart the whole Kubernetes deployment rather than just delete the pods? Well, let's check the logs of it first in case of something. Sure.

2:05:03 No. It's healthy now. Whereas I was complaining a lot last time, wasn't it? I don't remember. It was. There were We deploy our clustered deployment. Yeah. Okay. So it can pick up on things. I I just like starting with a fresh slate sometimes. Yeah. I'm just gonna check both of these. Yeah. Those both look good. So let's try your suggestion. Actually, just like delete the whole deploy, like, redeploy it. Okay. Well, since it's up, I'm gonna just check it. See what happens. Yeah, we still have no cluster DNS. Kubectl s has endpoints. I'm assuming Kubernetes has

2:06:25 endpoint. Are they scheduled on different nodes? Okay. That's after eight. Yeah. Let's see if Guy wants to come back and give us last bit of advice before we we wrap this up. Yeah. You there, Guy? I am indeed. Very close. You've you've not looked at the configuration of core DNS. You've looked at the pods, but not how the pods are configured to run the DNS. Okay. So the you want us to look at the core DNS conflict map? Yeah. And you can I I nicely left you some small tools pods so that you could if you didn't

2:07:00 Hint from Guy: Check CoreDNS Config Map for Small Typo

2:07:29 have anything in the cluster to exec into and run decks against things in the cluster and outside the cluster, you could use that? You didn't seem to like my present much. I I didn't know where Smurl was. I just was like, hey, go away. Because in in my experience of these episodes, people don't leave me helpful things. They leave me bad nasty things. And I just wanted rid of anything I didn't know what it was. So I mean, that's that's reasonable. Alright. Thank you. We'll give this another few minutes to see how we get on. Thanks,

2:08:04 guy. I like my phone a friend button. That's coming in quite handy. Okay. Unfortunately, we had this YAML open so why don't we go through the config map? So we're looking to see errors of ready. So I'm I'm I'm I'm impressed to see that lame duck is a real thing. Great. Kubernetes cluster dot local. That's different. Cluster domain reverse sliders. Yeah. I think these are substituted apply time. Oh, sure. Sure. Sure. So that's fall through. We've got bad stuff here. Do we? Oh, TTL. And TTL. Yeah. Let's get rid of those. Is there anything else?

2:08:57 Prometheus forward cash flow. Prometheus forward cache list reload load balance reload load balance. Yep. So I just like What about forward at c resolve con? Yeah. I think that's just telling it to fall back on Yep. Both. Okay. So now we've modified a config map. There's no core DNS controller. We will need to put those pods. And then let's just hope it fixes it. It's always this slow. I should start doing a force kill with a grace period on it. I hate config maps. Just I really just don't like them. I'm like, no. I don't think that helped.

2:10:05 Well Okay. While he told us to try that, let's try it. I guess that's the nicer way instead of deleting the pods. But we already deleted the pods and we still weren't getting the work in DNS. Although I guess the server never really checked to check the service was actually responding yet. Bad me. Okay, we have endpoints. We have put DNS. Yeah. Just keeps coming. Alright. Did we miss something? Is that errors supposed to be at the top? Yeah. Errors. Health lame doc. Ready Kubernetes. I wonder if this lane is wrong. Is it custard at local dot service?

2:11:36 Identifying the CoreDNS Config Map Typo (`clouster.local`)

2:11:41 Oh, no. Let's let's I'm gonna see cheat. Let's I'm gonna jump on to cluster 14 and grab the config map and see what's different. Yeah. I don't need an invite. There we go. Okay. You think after seven episodes, I would know what this config map is supposed to look like, but I just do not. Have cube control, cube config. Okay. This one works. Cool. Cluster dot local. Oh, and the pod and secure and the TTL are actually there too. The forward. This is the same contact now. Right? Right? Yeah. I think so. Let's pull them side by side.

2:12:53 It was in the same. And put the lane one. Yeah. So we had to move to polygen secure and TTL. So let's add that back. Sure. What about Yeah. These are the exact same fail. So he told us to look in the core DNS config and then there is nothing wrong with the core DNS config. That's just cruel. Yeah. Alright. What about the You got an idea? Okay. I'm I want to get seriously. K. Guy has given us a helpful hint on the YouTube chat saying there's a few pixels. This is a white space error in the

2:14:19 core DNS config. I will be mighty frustrated. Do you mind if I Cool. I don't know. Go for it. Yeah. Go for it. No. Just feel free to open it. Do the thing. Yeah. Okay. Let's just pull up in that config map again. Unless it's whatever's off screen. No. There's nothing. Wait. Hang on. Which one's the so this is 14. Okay. So this one's 14 here. And in fact, I'll I'll explain it. Might not be a white space error, and this is 12. K. Resource version is different. Uh-huh. K. Flip back to 14. Keep flipping.

2:15:31 Scroll up. Okay. Scroll up to where it says data core file. Damn. He's mocking us now because he said you're jumping back and forward showed a slight difference. This number seems arbitrary. Oh, you're rich. That's the one that's so different. And I was looking at the time stamp. So that's The res the UID is different, and the resource version is like So the resource version, I would expect different, but it's wildly high. Right? Yeah. But what I don't even know what resource version means in this case. Is this a where does that come from? Could fig map

2:16:29 resource version? I'm gonna delete it and see what it puts in its place. Yeah. Our big huge number. Okay. Six digits. Did you just replace it with the previous one or with the one from the other cluster? I'm I'm just gonna look up wait. What the I don't even know what that means, resource version. I'm assuming. There's like a weird bug and I'm sure he's found it because he works at Skyscanner and they've got a level of scale that most people don't have. But something to do with this number going so high is maybe causing it to load an

2:17:15 old config map in the pods or something. I don't really know. But the number being so high worries me. But we're now at the stage where I have to go and pick up my daughter. So Yeah. I'm just gonna bring Guy in to tell us what wonderful magical What we should this is. Yeah. Alright, guys. Lay on us. So when you were jumping back and forth between the working config and the non working config, If you look really hard at the c in front of it in the Kubernetes config Oh, you bastard. The c? Yeah. It's not a c.

2:18:04 Fixing CoreDNS Config Map Typo

2:18:10 No. It's a it's a character which looks a lot like c. What? So the Kubernetes plug in became authoritative for letter that looks like cluster.local. That is cruel. Yeah. I'm feeling a bit bad about it. And it works. I mean, my app doesn't work, but it worked. Alright. That was really harsh, guys. Thank you. Sorry. I think I only need my best to meet the person. Alright. I really really do have to go get my daughter before she gets really grumpy that she's not in her bed. Guy, I'm gonna cut you off but thank you very much. That was painful.

2:19:01 Conclusion and Thanks

2:19:11 Thank you. I learned a lot. Okay. This was really this was really fun. Thank you so much for having me. I'm I learned a ton. I hope I was helpful. Yeah. You were. You definitely were. It's always nice just to have somebody help you, talk to you. We've got some really good tips from you, you know, changing one thing at once. Just walking through the whole process. And again, we have to make sure that people know that doesn't matter how much you work with Kubernetes day and day. There's just so much. It's such a huge complex project with thousands of

2:19:47 contributors, millions of lines of code. You don't need to know all. So You just need to have friends to jump on clusters with. Sadly. Friends like Gwen to come and help you whenever you need it. So This was super fun. Yeah. Thank you for joining me. Have a a really great day and I will speak to you again soon. I will. Enjoy bedtime. Bye. Bye.