2:26 Introduction & Housekeeping

2:26 Hello. Welcome to today's episode of Rawkode live. This is the clustered series. A show in which me and a friend and guests will do our best to fix broken Kubernetes clusters. The catch is they are broken by members of the Kubernetes community and weird, wild, wonderful, and scary ways. Now, before we can begin today, I've got a little bit of housekeeping. First, please subscribe and click the bell on YouTube so you get alerts and notifications for more episodes of Clustered and the other cloud native content that I am producing. Also, if you are Discordian inclined and like to chat, we always have

3:05 an active community of people chatting about cloud native Kubernetes and a whole bunch of other stuff. Come and join us. And finally, I need to thank and want to thank my employer Equinix Medal. They provide the time and resources for me to invest in this show and hopefully produce learning materials so that we can all learn together. If you wanna check out Equinix Medal, you can use the code to Rawkode live. It will get you $50 in credit, which is around one hundred hours of compute. So enjoy. Let's see if I remember how to make

3:32 that disappear. Hey. There we go. Alright. Now today, I am joined by. Hey, Duffy. How's it going? Quite well. Quite well. It's going great. I'm looking forward to this one. This will be super fun. Great to see such an awesome audience. Good to see you, Sam and Rich. I think Sam might be giggling glee in the background, but I'm not sure. Yes. Definitely. We've got loads of comments coming in. I'll get a few more of those on while we chat. Do you wanna just give us the quick TLDR on Duffy, who you are, what you're up to, and then

3:35 Welcoming Guest Duffy

4:03 we'll get started. Sure. I'm a I'm a CNCF ambassador. I was actually pulled in as a CNCF ambassador last year after applying for quite a while, and I'm I'm super excited about being here and helping kinda get the word out for Kubernetes technologies, cloud native technologies, that sort of stuff. In the past, I've worked for companies like and and and really helped a number of of folks just kind of learn about things through things like TGIK and stuff like that. It was like it it was such a blast, and I'm I'm really looking forward to doing more of

4:33 it. So I'm really glad to be here. Thank you for having me on. Oh, no. It's definitely my pleasure. You're a font of knowledge when it comes to all of this stuff and you've been not only just helping me in in private for DMs whenever I get stuck on stuff, but also you come on and, you know, join the comments and help us all out during the episode. So I'm I'm soaked for today's episode. Okay. Let's I think we just need to dive right in. Our first cluster is in fact, we have a comment from Siam already. So,

4:56 Starting Cluster 11 Troubleshooting (Siam is the Breaker)

5:02 Siam. Siam is the first breaker cluster 11. So we're gonna pop the screen share on. We're gonna use my my I think it's my favorite tool for doing this, but teleport, which just gives us the ability to share a session and pair on this in real time while showing and sharing it with people that are watching. So Duffy Yeah. Yeah. Do you Oh my goodness. Do you have it opened? I can't believe I I don't know if you have this word in The US. Do you have to do you know what a is? Is that a Scottish word maybe?

5:38 I'm not sure. Okay. What this is? Like a tag. Alright. Siam is definitely is really pretty. I wonder if that was done with, like is that under bash, I see? I'm not curious. Anyway, never mind. Alright. So Siam has broken our first cluster, and we're we're gonna do our best to fix it. How how do you wanna start? What have you got? I'm gonna I'm gonna link in. Oh, yeah. Let's confirm you can type it, type it. That's me. Let's see if we can see you. I'm gonna do I'm connected and typing in cluster 11. Do you probably click connect?

5:55 Initial Cluster State Assessment (Nodes & Pods)

6:15 Can you go to active sessions and join my session? Sure. So we should have covered this. That's my fault. Oh, and Sam is there. So while you join that session, Sam has told us that each node has something different. So I'm gonna quickly just share that. What we got here? Where do I see session? Subscribe to Rawkode. So if you look at my screen here, you'll see there's activity then on activity, there's active sessions. I gotcha. Alright. Okay. Let's just see what these other two notes say. We got welcome Duffy. Nice. And the last one,

7:00 Duffy rocks. Duffy does rock. Definitely. Is this working? Do you see me saying hi? Yeah. There's a there's a bug when you joined that it means that I have to reload, but I can see it. You're typing. We are good. So you're some linking something. Yeah. Normally I just export it but it's nice to see other ways of doing this. Oh, we're already good. See there's already four nodes. We're finished. Yeah. As you were on this show that we are able to even run and get notes at the box. So, you know, thank you for

7:47 taking that easy on us maybe as I am. Although we don't know what's in store, so I shouldn't I shouldn't cut my check engine. Oh, it's not looking good for Rawk. They're not happy. See if that's okay. So what I'm looking right now what I'm doing right now is I'm kinda, like, doing a a look around just to see what we're looking at and that kind of thing. One of the things a lot about these sessions that I think is super interesting is that because we're using cluster API and cube ADM together, we can make a

8:16 bunch of assumptions about how things work. And I wanted to, like, talk about that for a minute because I remember, like, really being surprised by this. It is actually real it would be really really a fast fix to do a thing like grab the the cube ADM configuration from the config map inside the cluster and then leverage that to just reset all the manifests. So if what we were doing was, like, erasing the clock to get this cluster back up, that is exactly how it would approach this. Right? I would jump into the I would

8:50 I'd grab the the kubeadm configuration from the cluster, and I would leverage that, do Kube ATM reset, do Kube ATM and hit, and just blow everything way back blow everything back to a healthy space. But we don't do that here. Right? Here, we're trying to figure out how it broke. So I kinda dig that actually. Yeah. Definitely. It's you know, I I I like I like to think that it's not just about fixing the clusters, but it's about really trying to dig in and see what went wrong and and understand flags. Like, one of the really cool things

9:23 for me on this five episodes and those six episodes is I've discovered flags on the controllers. I had no idea that existed. And just exploring the API and the configuration in that way, I find super interesting. It's just Very cool. And part fun, you know, definitely part fun. Okay. Let's talk about what you're typing right now. So we ran a get node, so we seem we had two unready and one looked like it possibly had a taint on it. We've ran a get pods. We're seeing that Rawkode is pretty unhappy. I'm assuming possibly due to the

9:52 not ready nodes. Then you're taking a look at the pods in the default namespace and we can see my my workload appear to be working, but we've got a couple of random workloads also setting there. And then you've run a cry control p s as well. Yeah. And then I was gonna make cry control p s work real quick. So let me just pull up the thing for that. What should I do? I'm not sure. There's a thing that we did in I swear I swear I know what I'm doing. I've hit the wrong button.

10:10 Fixing crictl Access

10:28 There's a thing we did in kind, which I think is super handy, and I wanted to share that with you. See if I paste in here. Copy. K. K. Okay. Sweet. Rory McKeon is saying we should take off a nuclear cluster from orbit. It's on the way to be sure. Exactly. But looking at would be, like, super fast, but it wouldn't really be super educational. So what's it what I'm doing now is I'm setting a configuration file in Etsy called c r I kettle dot yaml. And if you conf and if you populate this, like, with with something that looks like

11:29 this, runtime endpoint, and then you point it at the container d socket, then you can just use c r I kettle p s, c r kettle pods. Could you I think there was also some news this week as well, wasn't there? I think that NerdControl was donated to the container d and cloud data foundation project. It is. Yeah. It has been brought into the project, but I think CI kettle is is not going anyway. Right? C r kettle is more like around kind of the implementation of what the container runtime for cube looks like. NerdKettle's

12:03 project is actually even cooler by a curious Akahirsuda. And what he's doing is he's writing a effectively, like, an abstraction that makes it look and feel and work like Docker. Because, like, I don't know if you've ever tried to use container d directly to, like, interact with Docker produce. Like, can you start a container, pull a container, mount volume, exec? There are a bunch of those things that we would normally use in troubleshooting that exist in c r x kettle. But doing those things in CTR is is definitely a cognitive load to, like, get that

12:35 to work. And NerdKettle looks to offset that because with NerdKettle, could just use NerdKettle like you would Docker. But instead of going to the Docker obstruction, you're going directly to container D. It's pretty cool. Nice. Alright. So let's see if your workload is working. Can you you know, you wanna try doing a proxy and seeing if your if your website is open? I'm actually kinda curious if if Sion broke that first. Let's see. We can do a port forward. Plus we have a new sample workload now. No longer just a boring old WordPress system. Oh, I wanna hear more about this.

12:51 Application Connectivity Test (Port Forward)

13:17 I can't remember what port I'm running on, but we're gonna find out if it's port 80. I think I would have done port 80. Do you have a port forward to the control plane? Because right now you're running on the control. Yeah. I should know better. Okay. Let me do that from here. This is cluster zero 11. I'm gonna export. Make sure I can run get pods. There we go. Okay. And I've had eighty eighty on that previous one, so I'm gonna do it again. Let's see. Hello? Maybe not working? We're not getting a connection reset on the

14:05 port forward side, which makes me think I do have the right port. I think my application Ah, let me zoom in. Oh, my. Alright. We have do it. Yep. Alright. We have our first clue. Yeah. So there is something weird. So let me explain the sample workload then. It is a Okay. A very simple Rust based application that prints a video on the screen of me, but I wanted to make sure we had some connectivity within the clusters. I added a quotes feature that pulls in some quotes from people in the Kubernetes and cloud native community

14:42 and puts them on the screen for us. That look up to the database to grab a quote failing, which tells me that our database service is unhappy or potentially some networking issue within our cluster. So we should try more Could it be I mean, it it can't be DNS. Right? Like, it's never DNS. It's never DNS. No. I I don't know why we have that saying that's all. But we can see from our get pods that we ran there that, you know, the postgres DB that we're running in the same namespace does appear to be running and healthy. It's passing its probes.

14:53 Investigating CoreDNS & Controller Manager Pods

15:15 So, I guess we should maybe take a look at the service before we dig into anything else. Oh, you're you're good. I know. Or just let you type away. Oh. Take a look at cube system. Core DNS is terminating three days ago. Why would it be stuck in terminating? That's not a good sign. Yeah. That's interesting. But we do have two working copies of code for DNS. That means it's not DNS, by the way, just FYI. That's maybe too bold a statement to make this early on in the episode. Fair. Have we got too long terminating?

16:07 Interesting. The packet cloud controller manager seems to be dying three days as well. Are you sourcing the completion? Yeah. There we go. Oh, rocket syntax. Is that what it's called? The rocket syntax? That's the way I that's the way I learned about it. So what's your hunch here? You're you've taken a look at the pods and cube system. You're now pulling out a conflict map. What are you thinking? I'm thinking that that somebody has messed with the configuration of core DNS at least, and it or and it might be because it's terminating. Terminating is super interesting. Like, that is terminating.

16:53 I wanna I wanna look at the events for that and figure out, like, why it's terminating, why it hasn't actually already been deleted. So there might be also something happening with the controller manager not doing cleanup or something else funky happened in there. It looks like the controller manager has restarted a few times, and so there might be something defined in that position. So my plan of attack is take a look at the core DNS configuration, see if we can figure out what's keeping it from working. And then I wanna go and look at the manifest on the as a Kubernetes manifest

17:06 Analyzing CoreDNS Configuration Map

17:23 directory and check those for errors or weird stuff. And then, you know, we'll go from there, see if we can get things working. Yeah. We got a comment from Jason just saying, you know, are all the nodes up, which we know two of them are unready from the get nodes command area. That probably would explain the the terminating? I guess, if that be yeah. We can maybe dig into that if that seems to be the case and explain why that happened. That's a very good point, actually. If they're if the nodes aren't reachable, then, like, if

17:51 they've just kind of, like, gone away, then the pod will the Kubernetes will keep those things around almost indefinitely, which is always kind of a surprise. One of the things I've noticed as we do more of these episodes as well is that age on our conflict maps and other things is always very misleading because it's not edit, but the actual usually the creation timestamp. Yeah. Steve Wade says, hey, Duffy. Steve. I know you're trying to think and stuff here and I'm just gonna keep throwing comments at you, but So we're looking at the config map for

18:40 core DNS. Let's see what we've got up here. I mean, I really should be getting a bit more familiar with this though, but We do have health already Kubernetes. Yeah. Is there a way to go in Go ahead. I was just wondering, is there a way to know if this has been modified since the creation? Is this timestamp operation update here? Does this because this is the same as our creation timestamp, can we assume this hasn't been touched? Is that correct? Creation if it's been just directly replaced, the creation time stamp would be different than the

19:21 creation time stamp of the cluster. Yeah. I mean, I'm, like, at a generation with conflict maps. Right? I don't see that here, though. Yeah. Looks like it's three days old. Yeah. I And it looks like so 03/2004. Yeah. It looks like unless unless it was just directly replaced three days ago. This is probably not the thing I'm looking for. Yeah. But it's can I see it? Can I see it? It's not DNS? Well, it's not this part of DNS. That could be any number of other parts of DNS. Yeah. This all looks fine. Okay. So

20:00 so let's see. Oh, you mind if I end this all big real quick? No. So anything that we need. It's not big on a bunch of systems. It will be bind utils, I believe. And dash utils. Right? I never remember. I always end up both trying it both ways. Try no dash. Oh, bind nine dash utils. There you go. Let's let's try to suggest it to us there. Bind nine dash util. Yeah. The amount of times I've done apt install deg, apt install dns utils, apt install dns utils, apt install bing utils, like I just

20:40 I really should get better at remembering the package. So then you go to a fedora central system and it's all different again. Okay. So we're now gonna take a look at the service. I think we're gonna describe the postcard service to see if we actually have any endpoints. DNS would probably still resolve, but What did DNS resolve if there were no endpoints? What would it resolve to? And I'm not sure. Postgres. Oh, I typed it wrong. That's that new keyboard of yours. It is. This is what I mean by rendering. Oh, here we go. Alright. One more sec.

20:52 Checking Database Service Endpoints

21:29 I wasn't doing it. There it is. Yeah. One extra r. There we Alright. Oh, it has endpoints. Is it? Yeah. I think I do think it's gonna be DNS. So what I was gonna do was If let's see if built in DNS is broken. Sam asks, who's eating popcorn? I'm gonna ask him that's rhetorical question and you're sitting there giggling away. Watch we know that Jason is giggling. Thanks, Jason. Thanks for the support, mate. What then? Just Jason, the timbers laughing. That's all. Error from server pods bash is forbidden. Post service check. Eight eight eight check. Oh, nice.

22:31 Pod Creation Forbidden: Discovering Admission Control

22:38 Alright. Okay. So what what happened there? Okay. We've tried to run a bash image and and we couldn't create a resource. Ah, nice. Very nice. So this is gonna be an a p this is gonna be an admission control thing. So you're opening the manifest because you want to know if it's dynamic or if it's a static admission controller. Is that right? Yeah. Yeah. I'm looking at the admission controller flags and also any other configuration in admission to see what's been turned on. Looks like note our back is here, but there is an admission control config

23:17 file, s e kubernetes demo admission dot JSON. Yeah. We is let's see. Kubernetes. I don't see I don't see how this could mount. It's interesting. So I bet you it's just erroring out because it's not present. Let's see Kubernetes demo admission control. Let's take a look at it anyway. Yeah. Why not? What has he done to us? Can we find out? Alright. Here we're applying some certificates. We have a service check eight eight eight check image, which is probably gonna be a a thing running inside of I mean, somehow, it's gonna have to resolve this, but it doesn't it doesn't look like

23:22 Inspecting API Server Admission Control Config

24:32 it likely is. So a quick way to resolve this particular problem and move past it would be for us just to remove this admission controller because it wasn't part of our original cluster. So can I just clarify can you explain something to me here? Okay. So there was an admission controller JSON flag that pointed to this JSON file, which points to a kube config file with some parameters. Oh, my teleporter. Oh, there's a disconnect. Oh, yeah. I think it's just because you're oh, you open then. I thought my teleporter has crashed, but we're good. Oh, no.

24:41 Removing ImagePolicyWebhook Admission Control Config

25:03 But I'm I wanted to I wanted to look at the the API server's configuration so we could look at what's happening in here. Right? So what I was pointing out before or what I was curious was whether the the file was actually mounted into the into the manifest because remember the API server is gonna be running as a static pod. So all of the resources that that static pod needs have to be mounted in just like you'd mount anything else into a container. Yeah. So because if we go up here to the admission line where it says admission control config file,

25:35 this this line is pointing at that configuration file, which kicks off the whole set of things that allow the API server to interact with that admission webhook and see if it could get and and and see if the image that we're pulling is allowed or disallowed. I don't know what Siam put in the allow list or the denial list. It might still be in that demo directory somewhere, and we'll look at that a little bit more here in just a minute. But I think for now, what I wanna do is drop this line. Yeah. Agreed. And

26:09 then just down here on enable admission plugins. Uh-huh. Can you see the The image policy webhook on the end? And then That's the thing that tells it that's the it's like a two parter in this particular admission control plug in. Right? The first part is to enable the image policy webhook, which tells the API server we have a thing that is just gonna do image policy validation, whether it's allowed or denied. And then the configuration file tells it where to go find that thing and how to ask it. It's that. I know. Do you know what node restriction

26:45 Explaining Node Restriction Admission Controller

26:46 does? This is so cool. It is really cool. Yeah. Go for it. Let's let's see here. What does it do? What node restriction does is it it's responsible for so when I first started playing with Kubernetes, typically, what we what people would do is they would, like, give every node, like, almost the same certificate. Right? The same identity certificate. And all of the nodes are represented by a group called system nodes, and you would apply permissions from to that system nodes object. That means that any node in the cluster effectively was like god mode. It could

27:25 read anything inside the cluster. It could do anything with its own credential. It was effectively unlimited. Right? As long so if you could exploit the underlying node, you could easily take over the cluster because that credential that the node had, that was a high value credential. What node restriction does is it enforces that a node, as identified by its host or by its by its node name can only access those resources that are attributed to that node. Right? So if you were to try and do KubeKitle get secrets all in fact, we let's just try this out real quick. It'll

28:02 be really cool. Get secrets dash. Man. Working on it. I'm gonna get it. Apple a dash dash config. This is node restriction. This is what that admission plugin is doing. Without that admission plugin, then all of the secrets would have just flushed to the disc would sort of flush to the screen. Ah. Pretty cool. Yeah. That is cool. Okay. Yeah. It's like a really nice view. For a little while. I've seen that at mission yeah. That mission controller on a couple of episodes now. I wasn't sure if it was evil from one of the breakers or part of the setup,

29:00 and I just kind of ignored it. So I'm glad you've taken the time to explain that. Definitely a good thing. One of the other things but you can do things like get nodes. You can do things like other you can there are a bunch of other, like, more generic cluster wide commands that you can still do with that credential. So if you were a if you were a an attacker and you only had, like, a user specific or namespace specific credential, this is a way that you could escalate your privilege to have a cluster wide

29:28 read certificate if you took over the node. So interesting stuff. Nice. Anyway, so Okay. So we've removed that from our API server. Does does that mean you should now be able to run that batch image, and is that still our our plan going forward? That's still my plan going forward. Okay. What? We go deeper. Disable pod, check for labels, validation errors. So I am has disabled pod creation on your cluster. Rule, check for labels. Failed at pass. Metadata labels app. So that means that we have yes. PRD. Oh, nice. Oh, Sam. What what what do you see? Oh, I

29:37 Pod Creation Blocked by Validation Policy

30:26 see chaos glitch. I would just like to I would just like to point these things out. Yeah. Go for it. Wow. Oh, that's beautiful. Look at it. Okay. So what I just ran here was I did keep getting CRD, which is because I know that it's not gonna be an admission anymore. Right? The admission plug in pieces, we've effectively taken care of. So anything else that's gonna eat our lunch is gonna be deployed into inside the cluster, And those things that are deployed inside the cluster are almost always gonna have to be registered with the cluster for them to work.

30:40 Identifying Caverno and ClusterPolicy CRDs

31:03 There's a couple of different ways that we can see those things or that that Kubernetes can be extended to be able to support those things. Like, one of them is custom resource definitions. Right? And and that's what I'm listing here is kubectl get cluster resource definitions. Let's me see, you know, what is the other stuff that is available inside of this cluster that I could use or that has been registered with the cluster? And what we see is a bunch of good stuff. Right? We see a bunch of, like, custom resource definitions that end in chaosmesh.org.

31:36 We see that Caverno is enabled, which is probably the thing that is currently blocking us. Yep. And we but we also see that there has been some goosery. You know? I use that term goosery to describe what has hap stress chaos. That's probably the best one word thing ever, stress chaos. Stress chaos. Yes. Also really good. Alright. So good old. Yeah. And this so likely, he deployed this into something. Right? So we have a couple of different namespaces here. We have Caverno. You get all get NS dash r. It's like get Dash oh, A Dash N.

32:27 Would you like me to do some typing? And now the thing that's giving me is the dashes. I gotta figure out how to remap that. What? Oh. Oh, you did, like, get dash a instead of pods. I think it's dash all that I wanted. Well, no. Because you're specifying the namespace with dash and Kiverno. You've not set a resource. I'm assuming you want pods. Oh, you wanted actually all. Okay. Alright. So we have our service running. We have a deployment. How does convert will get populated? Get all get API resources. Okay. So we're gonna have cluster policies, which

33:30 are CPOLs. Yep. We're gonna have cluster report change requests. We're gonna have generate requests. I'm looking probably for CPauls. So you can all get CPaul dash We'll take a look at it. Yeah. I think that disabled port policy is definitely something we wanna take a look at. Alright. What do we got here? We have background tool, rules match, resource kind for pods, check for labels. Validate data. The message is this. Match resources. Damon sets deployments jobs, stateful sets. Oh, I'm tempted to do a replica set just to mess them up because it didn't get caught.

33:51 Examining the 'disable-pod' ClusterPolicy

34:28 And then disabled for that. Blocked cron jobs. I think replica replica set was the only one he forgot. Looks like Yeah. I think But for our purposes but I was gonna say, I guess, we just wanna delete this terrible policy. Right? That's what I wanna see. Like, is there a way those match validation failure action. Probably something that we could edit to make it approve or something or ignore. No. What's the subject again? It's a c poll cluster policy. Yeah. Hey. Alright. I love when people actually do this. It's not it's not everyone that does it today,

35:47 but it's really cool. So the explain command allow us to get the documentation on the open API definition for the policy, if I'm not mistaken. Is that close enough? Yeah. Like, you can't do it now. It didn't used to be that you could do it, but you can do it now, which is very cool. So are you looking for a way to allow this to fail rather than deleting it just because you're being nice, or you wanna be a bit more methodological? Okay. There's that word I can never see on this team. Methodo methodological. It's a good word. It is a good

36:09 Deleting the 'disable-pod' Policy

36:25 word. I can tell you. Yeah. What I'm wondering is oh, I mean, let's just we could try nuking it, but I was wondering if we just turn it into something that will just allow us to do all the things rather than deny us to do all the things, whether we could just leave it in place and keep going and till we can play with it later. But let's let's try and nuke again. Let's see what happens. Yeah. But there was only one policy. I didn't see a policy to block the deletion of a policy, which would be a nice

36:46 trick in itself. So That would be admission. Or, I mean, I guess you could use OPA or something. C all dash I think it was called disable pod. Disable hyphen pod. Oh, the sledgehammer approach. Terminates. It's my favorite technique. And it's gone. Boom. Okay. Alright. Let's see if we can get in there. Ta da. Oh. Oh, I thought I thought it was that yeah. Oh. No. No. That's probably okay. It's probably gonna just take look at Maybe. No. I think we've got a network in issue. The bash image shouldn't take that long. Right? Probably not.

37:25 Pods Still Pending: Suspecting Node Readiness/Scheduling

37:53 But we did see some other stuff, didn't we? Like we saw the chaos thing. So I wonder if we're Yeah. There is the litmus kiosk deployed to our cluster. We got two nodes in a not ready state, which I'm assuming could be a side effect of this litmus stuff that's running. I know that the litmus stuff has labels. I think that opt you in. And what I'm wondering is whether failure to me. I don't see any labels there. No. I don't see anything. KLC. No. But not ready. Not ready. Not ready. That's probably not a good thing.

38:43 So what what makes Kubernetes return are not ready when you're doing the get notes? What what does it consider a ready note? Do you know? Usually, ready ready oh, what's that? So ready notes are really about the configuration of the cubelet. Right? So if you this in the fact, you can even see to some degree, like, why why it's ready or not ready. So piddle five node. See worker a t. So ready in this case, looks like requests. The method's nothing being used here. So this is a big part of ready right here. Right? Network unavailable, false. Memory pressure,

38:44 Diagnosing Unready Worker Nodes (Kubelet Status)

39:35 unknown. Keyboard has stopped posting node status, which means that this node has just kind of, you know, withered off into the into into the into the history books. It's not connect it's no longer reporting in. So we have to go look at why that node's not reporting in. Right? So in this case, ready means that. But ready could mean any of these things. And with, like, some of the more recent stuff in cube, Kubernetes now has the way or or this is now extensible. You could have things like, you know, node problem detector indicate whether a node is ready or not ready

40:07 as well and kind of and and extend that. But for the most part, if you look at this condition status here, that's gonna tell you what not ready means. So let's go jump into our other nodes and see what we see. Do you wanna pop open a connect a connection, or shall I? Yeah. You can do it. I'll just join. It's all good. Okay. Click it all. I'm gonna go to Target first. Get notes. I'm gonna jump into 7694F. Let's see. Servers 7694F. Subscribe to Rawkode. Do you see that? Yeah. Thanks, Sam. Appreciate the assist.

40:18 Troubleshooting Worker Node 7694F (Kubelet Logs)

41:10 Alright. So we done a system control status on the Kiplit. It looks like it's loaded, but not running. That's what I'll be. That's yeah. We can see it's an activating auto restart seven seconds. So keep it. Not happy. Exception. Yeah. Let me figure out the journal logs here. Dash flu? Yeah. Interesting. What is going on here? So it looks like Object. Yeah. Words. The Kubelet config object API version missing. Nice. K. Wirelib kubelet config dot YAML. That's actually a super handy error. Yeah. If only all other messages were itself, what was that? Look at the typo.

42:38 Fixing Kubelet Config File Typo

42:40 Cheeky Siam. Do you see it? Yeah. I'm trying to get over to it. It's a I I. What's a I I? Yeah. There you go. Yeah. RP. And while I'm in here, I already see something that looks a bit mischievous, but our client c a file five lines down is c a two dot c r t. I wonder if that's either been renamed or it was just thrown around them too in there too. Confused. It is surprising. I think Kubernetes p k I c a two. Not even leaving them. Look at that. See, that's there. There we go. Yes,

44:00 I am. We got it. You've dropped into recording mode. That's how I roll. You know, I like that. 1722076Of10. Yeah. That's correct. For for k Q D N S. I have the address in green, then we know. Else looks pretty okay. This is actually one of the things that always trips me out. You see this line right here? Static PodPass? Yeah. This means that with the cube ADM based cluster, static pods work on any node. Isn't that interesting? They should only have to work on the control plane nodes. There was an issue open for this, but never got really resolved.

44:18 Restarting Kubelet on Worker Node

45:00 I didn't know that. So on a worker node, I could create that directory, add the config and then just run whatever workloads I want. Yep. One of our friends and my current colleague, Dan Fennerin discovered that if you put the namespace into static manifest, that namespace doesn't exist, it will still run the pod and it's invisible to keep control commands, which I think is ridiculous. That should that shouldn't be a thing. And and and and I have been up to some pretty good onks that way. That is keep it. Yeah. Looks happy. But I'm just gonna jump over to our

45:36 under other tab and run our get nodes and yeah. That that one is back and happy. Alright. Let's jump over to number PMC q h. PMC q H. I'm gonna start it and he'll join or Yeah. Yeah. Go for it. Okay. Welcome, Duffy. I I feel welcome, Flanagan. Alright. Let's see what we got here. It does say not ready schedule and disabled, which makes me think the Kubelet oh, no. It's not gonna be okay because I can see it restarting already. I thought we make an okay Kubelet with something that should get on, but not the case. We have another exception.

46:28 Okay. It'll minus u. Keep it. The u is for unit, l is for wrap lines, f is for follow. Yeah. I tend to do Do remember? X e dash u. I don't know why I separate them though. I feel like I feel really silly now that I didn't think of that, so I I'm stealing this from you. And I always used to do no pager for the rap lines, so that dash l is awesome. That's probably the best thing I've learned today. Well, that and the other thing you taught me with Chrome, which is I guess you

47:01 could Oh, yeah. The trick is self sign certs. This is unsafe. Good trick. Yeah. I'll let you read the logs and when I explain that. Like teleport as running with self signed certificates and Chrome I just assumed that let me do that because it was trying to be secure and then Duffy taught me that if I just type this as unsafe. Yeah. This is unsafe into the browser. It loads it anyway. It's a awesome little trick. I think our problem. Right here. Container d is not run. In fact, no, the path to the socket is

47:31 wrong. You see there's missing a k. My face is almost covering it, but hopefully people can see that. Yeah. Where is that system? I don't see system the cubelet service d 10 cubic d m config. Yeah. Let's see. System. System. Kubelet. Barlet, Kubit, Kubit config dot emo. I really wish there was an easier way to work through all the different environment fails. There are too many of them, but yeah. I mean, like, a bunch to one place in violin kubeletconfig.yaml has been a boon for sure. But I think what I'm looking for is this one.

49:03 Violet kubelet kubelet mflags.env. So c d Violet kubelet. There it is. There it is. Another trick. Boink. Oh. Oopsie. I'm not supposed to make it worse. Come on. Fix it. Alright. Yeah. Yeah. I've lost track of it a bit of times. I've made a cluster worse than this show. Alright. Let's see. Let me jump over. I'm trying to watch. Done that weird thing. Online again. It's now ready, but scheduling is disabled. So I'm assuming it's got some sort of tint on it. Oh, you just corded. Oh, let's go. I'll cord and check. Okay. I'm gonna jump back over to that.

50:08 Node Becomes Ready but Scheduling Disabled (Kubeadm Taint)

50:14 How should we do kettle? P m c a p m c q h. Worker. P m c q h. It notes. There. You You wanna run a get pods all and see if all those random broken things are coming back to life? Those dashes are causing me I know they're really getting I know. We still get a lot of no. Maybe. Maybe it's getting better. Still well, it's looking pretty good. I think they're running. Let's take a look at your test app again, if you don't mind. Yep. Let me reload. I really need to speak to the Teleport

51:03 Application Still Failing: Re-evaluating Connectivity

51:10 team and get that fixed. So let's see. Get pods. Or forward. It's not changed as it nope. Get a refresh on that guy. Yeah. It's but I think it's gonna time out again on us. So we're still unable to speak to Postgres. Connections. Okay. Yeah. I think it's gonna time out on us. Alright. There we go. So we're still gonna there was something weird going on here. Okay. Keep it all. But we do have all the keyplets online. Let's progress. What's your what are you thinking at the moment? I am actually curious how it's configured. Like,

52:26 you know, if there's is DNS busted or anything else like that? The other thing I'm thinking about is just trying to see if I can resolve DNS from inside there to see if that's a see if there's a real problem. Well, That's the bug. But where control run you did for the bash part is it's still pending, which is something I think we need to fix in order to get the connect to have it even on the cluster working. Sure. Yeah. That's not good. Yeah. You know what I think is happening? Go for it. I'm curious.

53:14 Yep. We still haven't figured out what he was doing with all the chaos stuff. Oh, yeah. Should we just delete all that chaos stuff? Surely that's just there to trip us up. Although if it has caused chaos deleting, that may not be the best approach. We probably wanted to undo the chaos. So controller manager is looking okay. I think we should take a look in the the litmus name space then that was created. We both know Litmus Chaos is a CNC project, Chaos Monkey sales thing. Sam is a way to fetch another popcorn. We do have the chaos operator running.

54:14 We have chaos operator metrics running. We have a deployment. Should we just nuke it? Let's just nuke it. Let's just nuke it. Nothing good ever came from running chaos there. It's actually kind of an interesting thing. I wanna find out my reasoning here. Like, in the deployment, if we have a pod that's running that is the chaos operator, then it's the chaos operator that's eating our lunch. Right? It's actively finding our lunch and eating it. But if we disable that pod, then it can no longer do that thing. Right? So when I'm when I'm deleting that deployment,

54:51 I should be able to completely disable the chaos operator from doing what it's what it was doing before. Unless it's running as a static pod somewhere, in which case, I will be so so pleasantly surprised, I am. I can't even really put it into words. But this is the reason why I'm totally done with turning this off in this case. Let's see what happens. Bye bye, KS. Hold on. No reset. Perfect. I'm gonna reload my app. Do we think that the chaos just disappearing is gonna fix I don't The bash one is still pending. Should

55:36 we maybe describe that? I wanna see what's happening. Get all bash. Yeah. Still says pending. Not very useful. So it's like maybe the schedule is not there. Yeah. Strange that we're getting nothing. Bet if there's another table in one of the in the scheduler manifest or something. No. It looks like it's there. It's restarted a few times. Yeah. So in another window, I have, like, a kind cluster stood up because kind is also KubeTM based, and that can help me, like, understand what the configuration looks like for a working scheduler. Mhmm. Employment. I'm curious about looking at that real quick.

57:22 Keeps scheduler 124. Looks right like the right image. I don't see anything particularly bad about this scheduler. Looks okay. What I wonder is there is a way to extend the scheduler. Big maps. That's in two system. But I don't see anything extending it. What if we did keep it off? Create deployment test. And then flag image. What you talking about? Get off. Yeah. I'm not sure. Although I don't often bust a a create deployment to be fair. Normally, just use a kib control run. It's different. I wonder if I text it or something. It worked.

59:05 I created the replica set. Yeah. I'm sure we Yeah. Nothing to plug yet. Pending. Pending. Not being scheduled, though. The scheduler is definitely not doing the same. Our problem is the scheduler. Shall we take a look at the scheduler logs? Oh, okay. Before I do this, I wanna make a I wanna make a bet that it's that it's permissions. Oh. K. So the scheduler oh, look at the port number on that. Yeah. There we go. Feather, Siam. Wow. That's pretty. Okay. Where is that configured? Oh, that is configured here. Oh, you just jumped at the manifest directory.

1:00:36 I don't know if you meant that. No. I did. Oh, scheduled. Ah, okay. So in the in yes. So good. So good. So good, Saya. You should find one to say. Alright. Let's run that get pods. The scheduler oh, well. When we change the configs of the scheduler, do we need to restart up pods? Yeah. That's the thing. Right? Like, that's the thing about about most of the thing most most things in app in the application space. Nothing unless it's implemented specifically, nothing's gonna trigger to the fact that the file system has changed. So move

1:01:26 manifest to here. There's a cool trick last week that someone pointed out that you can just touch the fail in order to restart it even. It was Jason. I learned that last week. I did not know that last I did not know that. Jason just popped up with a comment. Hey. We need to restart that. Yeah. Was a good trick. I like that one. In fact, he's already jumped in. That would work well. Yeah. That was a great thing I learned last week. I love that one. Wonder if there's like another typo in so that's obvious. There we go.

1:02:20 Hey, yo. Back in I think we're I think we have I think we have this cluster. Well, no. We don't have a database connectivity yet. Still not resolved our DNS issue. But we are getting short on time. Oh, and the pods aren't creating either. Hopefully, that's just Hey, Ted, puppy. Let's go. Pull it. I mean, I know I'm impatient. I think we're gonna have to describe that. Yeah. That's not right. That is scheduled, but unable to run. Alright. Well, let's see what this has been associated with. This is on Worker 7 6 9 4 F.

1:03:05 Test Pod Unschedulable: Investigating Scheduler

1:03:41 Oh, let's see here. Docker library bash failed to do request. Dial TCP lookup register on 1477520720. What is that? Is that a DNS name that you actually know? That is 14775 is the facility or region that I have deployed this to. So no. Okay. So I'm gonna jump back over to I'm gonna jump back over to this guy, this node here, which is the 7694F node. And I already still have my session open, I think. Oh, yep. I'm on it. There you go. Okay. Girlgoogle.com. I don't know what this is supposed to be. Let's just throw 8.8.8.a

1:05:01 in there. Let's get it Well, let's see if it works from the curl google.com. I think it works from here. K. Let's see. Resolve.com. Which session are you on? I just went back to the master real quick just to look at this resolve.com. I'm gonna compare. So 207207207208. Oh, yeah. He's taking the numbers off the end. Oh, alright. It's obvious you know, it's it's a it's death by a thousand cuts. Seven. Alright. And then I'm gonna go back to them. I'm gonna do curl google.com. Hey. I'm gonna jump back to that master. Yep. You can we'll get pods.

1:06:00 Test Pod Running: Database Pod Failing

1:06:06 Ash is running. Test is still kinda mostly unhappy. Oh, that yeah. I send back off so it might just need to kick. Mostly to to get all described pod. Actually, pulled it. Yeah. Okay. There we go. Now what does it look like for the database? Nugget. So I think that's gonna say that again. Alright. K. K. Should we jump inside of the container, the custard container Yeah. And try and execute a few commands? Okay. You may wanna just check that's working. That that exists, I mean. Do you have a script in here that does, like, the testing?

1:07:00 Testing Network Connectivity from Inside a Pod

1:07:35 No. I'm afraid not. You're gonna have to do an laptop there, I think, on that. We have no connectivity. Wait a minute. That doesn't look right. You said it was something else before. Oh, wait. Is it? +1 722026010 is yeah. That is right. Yeah. That's good. Okay. Right? Yeah. Yeah. That's right. You can get all Yeah. Get get services. Cache and system. Yeah. There we go. Yeah. That's okay. Do wanna describe it? Make sure we got endpoints. Right. K. I'm gonna pull another batch trick. Nice. It has in punch too. Okay. That looks good. So our exec

1:09:08 Network Works from CoreDNS Pod

1:09:18 interesting. So when we're in here catcresolve.com. 20 6 0 ten. Curl Google It feels like a network policy to me. You're probably right because you're using Cilium. Good call, David. I get them right once in a blue moon, so hopefully. Hopefully, today's my day. No. Well, that must maybe it's a misleading name. I don't think that's a deny ingress. Let's just delete it. Alright. I'm gonna jump over to my app. Look at that. That's a nice catch. May your bag be bountiful and your success great. PS, you're very smart and hot. Signs, Steven Augustine. And

1:10:15 Application Connectivity Restored

1:10:31 here's my lovely video for y'all. Found us that we got time. We need to fix this. There we go. Should we deploy the update? Yeah. Good. Image version two. I don't have an alias key. I have no point in that now. We have our new version coming. Let's pop over here and get a new port forward. I think we fixed everything, hopefully. Image isn't too big. It should be real quick. Very cool. Thank you, Sam. That was fun. There we go. We have our new container port forward this one and make sure our upgrade worked.

1:11:35 Cluster 11 Fixed and Application Upgraded

1:11:35 I think it's just not ready yet. There we go. And if the video doesn't load, I think that's just my my application. Fight for your limits and sure enough they're yours by Duffy's mom. It's true. I love that statement. One of my favorites. Right. My knowledge is the best knowledge. Thank you, Siam. That was awesome. We're not gonna get my dance in video because clearly I've messed up v two of the image, but the quote worked. We have our connectivity. We did deploy our update. We are ready for cluster 13. You happy? I do. Good. Good. Right.

1:12:16 Transition to Cluster 13

1:12:16 I'm happy. Thank you, Sam. That was good fun. Let's get that closed. So I don't make because I've got a very bad habit of going to the wrong cluster. Alright. 0 13. Export to config and then we'll come over here. Okay. Let's jump onto the control point. So this cluster, I kinda left as a little bit of a surprise for you. Alright. Alright. We we had two returning breakers, former colleagues of yours. Can you guess who that would be? Oh, no. It is Jason Was it and Jason did that be it? It is Dan and Jason. They decided to

1:12:28 Starting Cluster 13 Troubleshooting (Connection Refused)

1:12:58 put a little special together for you. You know? So feel like I feel like I feel like I'm opening a love letter here right now. You know? Like, that makes me feel awesome. Alright. So maketheyear.com. Oh, cool. Okay. I'll be back at at just one second. Apologies. Alright. Uh-oh. Oh, no. Oh, it's not a good sign. Sorry about that. Sorry. Okay. Did we have an API server, or is that too much to ask? We I think that is too much to ask. +1 6849. 60 8 40 9. Alright. Well, that's the first test I wanted to look at. K.

1:14:13 No Kubernetes Components Running (crictl Check)

1:14:27 Which line again so I can remember what it is? Jason's saying no API server. I didn't expect her to be one, to be fair. But, you know, I can remain a little bit hopeful at time. See you at kettle. Yes. Oh my. Oh. Oh. Oh. Oh my. A little sparse there, isn't it? Yeah. It's looking a little anorectic. It's true. Kubernetes manifests. They're like Interesting. When you go when you go to, like when you're tab completing and you realize that there's more to a word than you think there is, you know you know that feeling? Yeah. I'm

1:15:12 Missing Kubernetes Manifests

1:15:33 having a feeling. Well, let's let's see what the left is in the Oh. Not working. Unless I'm just gonna take a quick peek at these Mhmm. Because it that's, I think, probably a good idea. Just take a quick little look here. Okay? Nope. That's all. Okay. Okay. Looks normal. Maybe Suspiciously. So Oh, even SED is gone. Yeah. It's a good sign. Okay. Totally gross. Also oh, what? Does that look okay? Does that look bad? I mean, I don't think the SED, the daemon, actually generates a thing that's called walled up broken. I'm just putting it out there.

1:16:11 Inspecting etcd Directory (Suspicious Files)

1:16:56 You know? Oh, yeah. Like, we be recovering some STD stuff here. I need to start adding rules. No break in STD either because Violet etcd. Violet etcd. 2381. '20 '3 '80 '1. '20 '3 '70 '9. They don't it doesn't look like they've been too mean to me here. Like, this all looks right to me. So what I'm gonna do is I'm gonna start by doing this. Move to FTD. Hold on. What's that? Okay. Slash. Should we run like a PS or something first? Just because maybe they are running and No. Because cry control would show us down

1:17:11 Restoring etcd Manifest

1:17:49 at the lowest level. Right? There's no way to hide stuff from a cry control. Well, there is. Oh. But, you you have you gotta be a little more creative. I I don't think they did that. I don't think they would do that per se. Okay. I trust you. Oh, we got a fairly amazing trying to pop up there. Ready. It's a good sign. Right. Okay. That's a reasonably good sign. Okay. The next thing I'm gonna do, I'm gonna move the API server over there. So you're just moving things back one by one because you don't trust these two. Right?

1:18:30 Restoring API Server Manifest

1:18:52 I don't know what you're talking about. Trust these I trust these guys with my life. Or at least the life of my pets. I guess it could take a little bit longer for the API server, but let's see. Oh, yeah. There we go. So when we do a a cry control get pods there and it says state ready, that's just as far as the container goes and not readiness program for that because it's not it's not aware of that level. Right? Yeah. So it's like, pods shows you the sandbox and PS shows you the process.

1:19:47 Okay. Oh, SCD did in fact bounce on this. Oh, yeah. Perfect. Okay. So I'm gonna move h m s to a back. Or should we delete that broken file? Maybe that's causing us easy to filter initialize. Maybe it's just like a random text file that's got some, you know, profanity in it or something. I like I like where you read that, but I wanna see. I don't know. Yes. That's a I wanna see if you can grab the logs from this puppy. Get all bugs. I guess you could also go to our log containers as well. I don't know.

1:19:53 etcd Failure: Discovering File Permission Issue

1:20:42 Cray control could do that too. That's cool. We got a oh, okay. Permission based. Yeah. Doesn't look bad. You see anything that's suspicious? Looks okay to me. Hold Open bar, let it see the member snap. Is it not is it not running as a root? We got a security contact. Is that what you're looking for? What I was wondering was, like, if it was mounted read only or something. It doesn't look like it was. It looks fine. And it keeps saying My low pitch. In my weird high pitched voice because I have no idea. See that error message There

1:21:53 Identifying Immutable File Attribute on etcd DB

1:22:41 it is. Now open database, Violet etcd member snap DB. Operation not permitted. Open Snap t p. So let's do a look at Snap t p. Okay. Oh. Oh my gosh. Check her. We wanna see if there's any weird attributes on the file that would Flown away this is what they've done. I can't even remember what the syntax is to look at the permissions that we're seeing. Hold on. Jason's left his very sneaky comment there. There are more to permissions than the files. Yeah. Shutter. How do I but I wanna see the permissions. LSetter. That's what it is.

1:24:06 I think we need the translator. You dirty dog. Can you explain that to me? Hold on. That's beautiful. So in the file in the attributes file system attributes. Right? It's a thing. It's like a security feature for the underlying because in in Linux, everything is a file. So you can actually mark a file immutable. And if you mark that file immutable with its attributes, then that means that nothing can effectively mutate it. What's neat about permissions here is that we're so used to looking at, like, whether I'm allowed to do a thing, not think of

1:24:09 Explaining Immutable File Attributes (chattr)

1:24:51 the think of, like, attributes more like the authorization rather than the authentication. Like, I'm authenticated to do the thing. I have readwrite. I'm good. I can I've they it knows me. I'm rude. I'm a I'm good. But Chatter says, but no matter who you are, you cannot do that. Right? So it's, basically setting setting the permission to block it. So I want to disable minus I, I think. Should we just set a a man change adder and see if it lists the different attributes? Would that work? I think I want shatter minus I. No man d b. Oh, well. Never mind.

1:25:10 Removing Immutable Attribute from etcd DB

1:25:46 So that I is an immutable flag and we think that we've removed that and now it should start when it restarts correctly. Yeah. That was two minutes ago, so we might just have to Oh, I'm looking at a one. Yeah. Exit two minutes ago. Does that mean the touch didn't work? Or did we move it back out of the directory and into the I have not moved it out of the directory. It is right where it's supposed to be. I know this works, though. Yeah. Don't think you moved it. You only put one dot instead of two.

1:26:58 Restoring etcd Manifest Again (After chattr)

1:27:29 You see the HDD pod drop. It goes. I believe STD back into manifest. Yeah. There's conversations in the chat right now about SE Linus and AppArmor. Yeah. I'm not dealing with that. Yeah. There's no one else to ban things. Houston knows how I feel about AppArmor, so that would be that would be particularly cruel. I feel like I'm getting some lag here. See where I cut all. Well, we don't know if they haven't done other stuff to this machine. I wouldn't be surprised if there was something timed inappropriately. Yeah. Uh-oh. Yeah. That's not coming back.

1:27:58 API Server Failure: Out of Memory Error

1:28:56 Oh, look at the load average. Hello. Dang. We better stop that quick before we there's access to this host. Key swap d. What was your thought process in running mount there? I was trying to see if we could do LS swap and see if we have it. But I don't remember what the trick is, so, like, swap. We could just just swap off all. I think it's the dash dash all. So in the swap. So you're not using a top memory. We do you see the cubelet freaking out? Yeah. The cubelet. Interesting. You wanna try a p s a u

1:30:44 System Memory Restricted

1:30:55 x and get a fill path on that k swap d just in case it's just particular just in case it's named that way, but not actually. Or you think it's cubelet? I'm just gonna take the cubelet out for now because it's freaking out. So I wanna, like, try and reduce the load. If the cubelet's taking up so much juice, then maybe that'll help us a little bit. There we go. Let's get that. Nope. Nope. That's not what I wanted. Message. Secret part of memory process called API server. That's because the cubit died, and that's okay.

1:31:50 Identifying Restricted Memory via vm.min_free_kbytes Sysctl

1:32:03 You know what? I'm gonna move FS API server. I'm gonna move the API server back out of here. Because I don't know for sure if there's not something else happening. I haven't looked at the image or anything. Okay. CTL. So I'll do a few. And that minus. Gotta really find a gotta really find a way to do that. Choice of a new keyboard. Right? Yeah. We all do it. Alright. Let's see what the keyboard I'll say then. Memory, c group out of memory error. Total VM. Oh, they messed with the c groups. I think my teleport has

1:33:45 it's not crashes. It's very slow. Yeah. Okay. So whenever we start the Kubelet, the or the API server, bad things are happening. Yeah. Also see so a secret out of error message out of memory error. That looks okay. A little bit in the file. In the unit file, though. Yeah. Yeah. I'm pulling up a command that should give us the ability to see, like, what the top consuming c groups are and see if there's maybe a c group that has been started up in another namespace or some other way. Does that say that we only have 1.6

1:35:38 gig of memory? It does. Oh, I just got kicked out. Boink. Hold on. Oh, this machine's definitely slowing down again. Is it restarting or 1869999. There's something weird with the machine. We got 62 gig of RAM, but we've only got one available. Perfect. Oh, Jason's commenting. What does three dash h show? What's the h channel and no g and m? Human readable. I can't even I can't even get back in right now. We've only 1.1 giga RAM available to the machine. But that means that So how is it to restrict the avail memory of it? But that would have to

1:37:07 be a kernel boot arg, or is there another way of doing that? You could do cat cat proc c m d line and see if something was defined there. No. It's okay. Yeah. Think it's gonna be the dev cat. Something is the be the way that Sorry. I need to go. I think it'll be something in the way the c groups are configured. Like, the c group itself has been manipulated for the root c group. Now is that root c group gonna be process one? Like, can we see it by broadening around it? Yeah. I can make the cat Let's take a

1:37:52 look. I think I'm gonna need some help from well, Dan, of course, you know, this is probably gonna be a thing that is a little bit on the edge of my experience. So somewhere there But you can modify the kernel through the SAS interface. Right? Is there a way for us to pull something out of here? It would probably be under Secret. Yeah. Under secret. Maybe not. Features. Maybe it's m m. Oh, you think it's secret features. Hold on. Alright. It's really painful to type. I can't even get in though. Jason has said We even get in. Not secret related.

1:38:40 So there must be another way to restrict the memory on this device. They haven't modified the kernel bit parameters. We got a comment in chat, proc mem info. Let's check that out. Oh, I got booted out. Hey. When you invited me, what username did you give me? Your Twitter handle. Okay. Which I'm not sure how to pronounce. Is it Maui lion? Maui? How would you say it? Maui. Maui likes the island I grew up on. Maui. Lion likes the big cat. Maui lion. Maui lion. Got it. Alright. We're we're we're gonna have to gotta handle this memory thing because

1:39:31 otherwise debugging this cluster is not gonna My intuition is telling me that they they did something to, like, the root c group that limited the available memory to the root c group, the top one. What I think has happened. It's told us it's not c group. Well, I can't even get back in. So there's that. Yeah. I'm gonna try and SSH from my terminal and see if that helps. There is a run time tunable. Yeah. We may have to pull up Google. Alright. Okay. How Unclaimed. What if you do cat proc mem info? Anything? Oh, that's what I tried to do before.

1:40:24 It completely died on me. VM allocated. No. That looks I mean, unless I'm missing something, this is all looking I think our problem our consists is in another castle. That's what I think. So he says it's a runtime tunable. Let's take a look at the configuration of container d. Can we do system CTL? But my host is showing that we don't have memory available. I don't think that's his container d. Right? Like Well, everything is in a namespace. Right? So it's gonna be I mean but you just said so, like, we have two options. Either

1:41:23 we're missing something in the way that c groups themselves are configured at the kernel layer, and we need to go chase that down and find out where that is. Or runtime wise, there's something messing us up. So my intuition tells me that it like, I want this to be something where they configured the root c group, limited the amount of memory available to it. You can also you can tune the kernel virtual memory. Kernel virtual memory. Oh, it's SPM. Okay. So I'm impressed. Tuning virtual memory. Can you take a look at sysctl dash a pipe grip?

1:42:22 Comparing Sysctl Value to Healthy Host

1:42:29 Let's see. What do I wanna see? Mem to dash a just in case. Yeah? Maybe that's It's probably VM. I wanna see VM min three k bytes. Let's try VM instead of mem. Maybe there's something missing. User reserve k bytes 131072. What's what's up there? Which one? Sorry. Min free k bytes is five zero one. Oh, so that's saying that we have to keep this number of kilobytes available, which does not means they can't be allocated by the kernel. So we could probably just do echo zero over the top of that. Oh. Well, there's a there's a CTL way of resetting

1:43:39 it to its default, which is not what you're doing here. I like the sledgehammer approach. I would just have cat echo as you know, right over the top of that and assist our system. But how can we reset with this CTL? You'll actually Hold on. Read values from system directory. Yeah. That's that's system. Alright. Let's see if that changed our VM minimum allocatable. No. It's still ridiculously high. At least I I I think that's high. Let's try the sledgehammer. Let's let's jump into another I would jump into another host to see what it is. So

1:44:31 it's probably it's probably something that's reasonable, but it should be set at. I'm a big fan of, like, having an example of something that should that is working the way it's supposed to and then comparing that with the thing that is not working with it. It. I'm not happy with the performance of this machine either, to be fair. Yeah. And free So let's make it that. I'd be happier with that change. Alright. So echo oh, I didn't copy properly. 7584. You can echo it in, or you can also do system d system CTL dash w,

1:45:00 Resetting vm.min_free_kbytes

1:45:15 and then the argument equals the value. Either way either way works. What was it? Men three Three k bytes. Okay. Alright. Okay. Cool. And that's already snappy, like Now, that's like a top. Yeah. Okay. But In fact, we could run out back in. Free dash h. Yeah. We've got available sexy kick there. Okay. We're back to business. Let's see if you can join the session again. For this. And I am options, join session. Back in business. So sneaky. So sneaky. Alright. So I guess what we wanna do now is get the Kubelet and API server

1:46:26 Restoring Controller Manager and Scheduler Manifests

1:46:29 back into the manifest directory and see if we can find some sense of normality here. One hopes. Okay. Hey. Copy? Yes. Let's see. Thanks, Jason. Normal is relative. That fills me with great hope. Let's see. I'll be right I'll be right back in just one second. Okay? Of course. If you wanna see if the API server comes up, that'd be awesome. Alright. No worries. Alright. So the API server is back in here. I guess I should use crack control. Nothing there. Oh, why it just slow? Am I being very impatient? No, there we go. Nice.

1:48:09 Control Plane Healthy, Worker Nodes Unready

1:48:10 Okay. So we have an API server. I can't remember if we took this link, but let's see if we can do We can. Okay. We've got two nodes not ready. Ship system namespace. Obviously, missing a couple of things, but it doesn't look too bad. And my actual application, database is terminating, let's try and move it over the rest of these manifests. Like I definitely took things nice and slow and one by one and was trying to verify things and I just gonna it'll be fine. Okay. Scheduler, good. Controller manager, good. Oh, running. Once they could just

1:49:19 Let's see. Don't keep system namespace. I fixed it all. Get off the hook. Oh, you can't hear me yet. Oh, you can. Alright. So I've moved Alright. Job done. Too easy. Moved those manifest back. The API server came up. I put the controller manager, the scheduler back end. They appear to be coming up. We can speak to the API server. Control manager's not quite healthy yet, but I think it's getting there. We still I ran and get nodes. We still got two of them down. There's still two worker nodes that need a little bit of CPR.

1:50:01 But the Kubernetes control point does seem to be okay. I will say okay. Fine. Okay. So we probably wanna jump on to D R 44 and Q P Q Unless there's something else on here that we need to check out. Dang it. Was really close on that one too. You know? That's Getting better. Yeah. There we go. Alright. So I think you're right. I think what I wanna see just look at these other two mental. I think I would like to just real quick for my own verification. Is that controller list that suspect? No. That's maybe okay. I just that star

1:50:40 Troubleshooting Worker Node DR44B

1:51:11 kinda threw me. Yeah. That looks okay. Okay. And then cat. Pretty simple. Okay. Oh, okay. Now it's okay. Alright. Just out of curiosity. Nothing suspicious there. Oh, you're looking for pod security policy. Yeah. I was just curious. These are places where I I would, like feel like one of the things I should have done earlier was this one for sure because we are on a Cilium cluster. Oh, alright. Okay. Well, let's go see if we can get our notes back. Get get all get notes. Let's jump over to D R 44 B. There's a session open if you just wanna

1:52:49 join that one. Oh, cool. I'll just join that one. Do you wanna start a session on the other one too? Yep. Got it. Alright. We have sessions. I'm on DR44B right now. I'm not sure what I'm on. Hold on. Close that. Config here. KDR443 active sessions. There we go. Now I see it. Alright. So 443. We have a kubelet. It's active and running. Nothing suspicious. Logs. Request that. Reload. Okay. We have a kubelet, but we're not ready. It's a good sign. No errors. No obvious errors. That's true. Oh, there we go. Cannot connection refused on the docker socket on the container d

1:53:32 DR44B Kubelet Error: Containerd Connection Refused

1:54:28 socket. Yeah. It can't speak to container d. Okay. Yeah. That's not running. Think we're don't think be that easy though. No. I don't think that's gonna run for long. That's that thing again. Okay. I'm gonna add that to my automation for spinning up these clusters so we don't have to configure every single thing. So a couple of things we've been doing in this session, which I've really been enjoying. Right? One is that we are in our own hurry. We're not in anybody else's hurry. Right? I know that that's hard to accomplish sometimes, but it definitely brings some

1:55:08 DR44B Node Becomes Ready

1:56:05 I'm enjoying the to what we're trying. The chillness and the the thinking and the the talking process. Yeah. It's it's it's working well. You know what our targets are? Like, we're moving through it. Like, one of the things that is, like, one of the things that that I think the show probably could use a little bit of is, like, just remember that, like, no matter what, like, everybody in this audience, shout out if you're with the audience, is is rooting for us. Like, everybody is rooting for us. We can't we literally cannot fail in this scenario. There's absolutely no way

1:56:36 to do it. But no matter how hard it looks like it's going to be or how difficult it is, like, know that you're that that that things are always gonna be on your side. I'm gonna jump back over to this master and see if we've cleared this problem. Okay. Session. Odes. Yeah. K. We got E r 44 b back. Let's take a look at q p So starting container d was all we had to do there? I know. I know. I'm I'm I'm nervous. I've got suspicious eyes for sure on that one. But alright. Let's I'll take it. Well, it's it's

1:57:06 Troubleshooting Worker Node QPQH

1:57:25 a subtle trick. Like, they probably expected that to be more subtle than it was because, like, the way it works, like, it would look like things are still running. Right? Pods are still active and running. Why isn't it working? But you if you didn't know that container d was broken, then it might take you a while to get there. That's true. So And it's easy to overlook the obvious. Definitely. I think I've done that time and time again. Alright. What are you thinking? We should jump out to q p q? You wanna get that one? Yeah.

1:57:52 I'm a jump on the q p q. Exactly. There's a active session there if you just join it. That is where I am. Oh, looks like our oh, wait a minute. It says it's killed. Why did it say killed? Yeah. Would have killed? Oh, I see. It's a reload. Uh-huh. Why why is that being killed, do you think? You're looking for suspicious processes? I am. Smiley face from Jason. Assuming that's, like, a suspicious processes. Yeah. I think that's our teleport. Why is it saying killed? Something is the other possibility the other one was so easy, and then this was really where they're

1:58:04 QPQH Kubelet Status: Killed (SIGKILL)

1:59:29 gonna make it shine for us. Alright. So let's see. Something is sending the cubelet a kill. A sick kill. Yeah. That could be fun to find. Hoping that I would see, like, you know, some dumb little script somewhere, like, sending a kill. Yeah. Well, Jason did leave us a comment. He said, why honk with a suspicious process when you can have the system honk for you? Fair. Fair. Memory max 64 k. Yeah. I would do it. Yeah. So good. Terrible people. Oh, they even reload. Well, the sick kill wasn't on because of a bad setting.

2:01:08 Removing Kubelet Memory Limit

2:01:29 And I agree Bill Gates quote there from Jason in the chat, who needs more than 64 k of memory? Right. I mean That was amusing. That was amusing. I'm gonna jump on to the control plane just to run get nodes and we've lost a D r d r 44 has died on us again. Our our starting of container d I do not think was enough. Excellent. Excellent. But we do have q q ready, if that helps. Oh, we'll leave that one. We'll leave q q working itself out for a while, and then we'll jump back over to

2:01:50 QPQH Node Becomes Ready

2:02:12 I think that would have been as swift as possible. But let's jump on to D R 44. Yeah. That's 99. I still have a d r four. Yeah. There we go. Nice. So we're seeing oh, network namespace. Too many open files. I feel like when we start to container d, we've let list a dragon. I think you might be right. Too many open files. Check network namespace. Close. CNI. Too many open files. Okay. So That probably wasn't what I wanted. I want let's let's see. We dash I. Oh, you're trying to do the I nodes

2:02:34 Investigating File Descriptor Limits on DR44B

2:03:40 as d f dash I. Yeah. At least get dot 2%. So freeze. I mean, we maybe it's a good limit? It could be. Could be to max, what's it called? Oh, fail max. Oh, no. Okay. That's that's pretty big. Too many open files. File ticket usage for snapshot. Oh, wait. You know what? Usage for snapshot is probably a a red herring. Is that in the logs? Now this one is kinda checked, though. Where are you Yeah. We see too many oh, sorry. I was scrolling back up in the history. Yeah. We don't share our scroller portion of that.

2:05:12 So Yeah. That is unfortunate. Yeah. I don't think it I don't know. Doesn't particularly like that. Yeah? Lots of too many open files. It shut down again. Right? No. It's still running. Yeah. It looks okay on my end. I'm just scrolling, scrolling, scrolling. I'm take a look at when we started container d. Right? So we need to know what containers are running and maybe find a bad apple. Yeah. Possibly. Oh, you know what? You were just saying this yourself. Let's see Kubernetes. Okay. That's good. Try dash here. I mean Do you put It's the right place.

2:07:14 What if this starts with a dot? Wait. What? I was just curious if they had a head and dot fail. Oh, I don't think we get caught. Alright. Could they have over could it be a fake fail system? You know what? Let's look at this real quick. Let's go back to master for a sec for the control plane for a second. Sorry. Keep it all. Subscribe node. Oh, you want to see what states Cisco okay. That's a good idea. Saying node ready. Rawkode notes again. Hold on. Reading this. Memory pressure, false. Disc pressure, false. Grid pressure, false. Ready, true.

2:07:36 DR44B Node Becomes Ready Again (Intermittent?)

2:08:25 Keyblade is posting a ready status. Did I go to the wrong one? I don't get notes. Having that dining on red herring kind of feeling. Inter intermittent problem. That's fine, really. The pods. Alright. Should we test my app? Yes. We should. Remember on your you're on 13 now. Yep. I'm setting up the port forward, heading the web, and we got version one. Okay. But that means we're hitting the database. It does. So now we can do a edit deployment clustered. Version two. Continue creating for past scheduling. Mental note to pre fill the images. Although, people may mess with them, I guess.

2:09:00 Testing Application on Cluster 13

2:10:03 There we go. Okay. Yep. Are you are you using fish or c s h? C s h. That's all. There we go. Oh, it's actually gonna I worked on this cluster. Nice. We got the happy dance even though I'm not smiling. I like the happy dance. May your bag be bountiful and your success be great. I love it. Is that us? Well, heck. Jason, how are we doing? I think we got all awful. It says down to the final honk, but then it says the. Something's weird here. Yeah. Especially with that d r 44 node, which was

2:10:40 Discussion: Intermittent Issue & Calling it Done

2:11:00 ready, unready with too many open files and then ready again. I'm sure if we left it long enough that our d r 44 b would maybe disappear, but for now it's healthy. Our app is working. We did our upgrade. I'm calling that a win. Yeah. I think we're I think we're gonna fall into the trap of, like, other operations folks. Those nodes are looking beautiful, Jason. Look at how happy they are. They're all ready. There's no, like, unreadiness. Yep. They look ready, Jason. I think one one of your geese may have fallen down a hole.

2:11:39 Jason says that no What time Bob? What time hey. It is it's working right now though. You know what I mean? And, like, it's time for lunch. And so we're gonna go have lunch, and then we're gonna, like, you know no kidding. Like the classic the classic the the classic it's working here. It's working for me problem. Works on my machine. We deployed our update. We've seen the happy dance. The happy dance is the aim of the game. I I I think we smashed it. Well, you know, I didn't do a lot of the heavy lifting, but still I

2:11:53 Wrap-up and Thanks

2:12:12 was here to witness it. So Yep. Happy. Alright. I had a blast. Thank you very much. I look forward to breaking or fixing again. One of the things I explained to you before that I would totally be happy to extend to you again is that, like, you can always if you're if you're getting a situation where you feel like you wanna dial a friend, you got my number. Appreciate that. Definitely. Thank you very much, Duffy. Thank you for I know we went way over what we plan to do there. But thank you for your patience and persistence.

2:12:43 Thank you to our breakers as well. To say I'm Yep. Dan and to Jason. We know there's a time bomb. If you wanna close chaos, you gotta do it harder than that. We're gonna run away before before it breaks again. But thanks everybody. Yep. Thank you, Duffy. I'll speak to you also and have a have a wonderful day. Thanks. Have a bunch of fun.