1:01 Introduction

1:01 Hello, and welcome to today's episode of Rawkode live. I am your host Rawkode. Before we get started, here's a quick reminder that you should subscribe to my channel at Rawkode live and click that bell. That means you're gonna get all of the updates and there are multiple streams per week. So hopefully there is content there. We can all continue to learn this vast cloud native landscape. Also be aware that we do have a discord chat. Please come and join us. The conversation is always fun and there's lots to learn from everyone involved there too. So cool.

1:32 Get involved. Now today we are gonna take, we're gonna attempt to write a Kubernetes controller to satisfy a random idea that I had in December last We'll talk a little bit about that in just a moment. But I am not smart enough to write my own Kubernetes controller so I have solicited some help from a friend of mine, Sahil Patel. Hey man. How are you doing? Hello. Great to be joining you. Thank you for the invite. Yeah. This will be fun. Hold on. Too many buttons. There we go. Yeah. I'm really looking forward to this.

2:06 I have written Kubernetes controllers before. I think you've got way more experience than me here. So So maybe we can, we'll do introductions first, but you know, maybe we'll talk a little bit about the different ways to extend and modify Kubernetes and get people a little bit of insight and flavor into how that works. So let's start first with you. Why don't you give us the Sahil pitch? Hey. Thank you. Hi. My name is Sahil. I am a software engineer. Currently, I work for a company called Monza, where I help build a better banking experience.

2:40 I've been using Kubernetes for a number of years now and been involved with the CNCF ecosystems of software. So things like Prometheus, Jaeger, and all of these other tools, Kubernetes, Envoy proxy, and all of the other tools that are involved in the CNCF ecosystem. And yeah, essentially, I want to make infrastructure like, want to represent infrastructure from an end user perspective. So like getting involved from within the end user community and stuff like that. Awesome. Very cool. All right. So Kubernetes controllers. What are they? So think of a Kubernetes controller in the sense that you use with Kubernetes, you declare

3:19 What is a Kubernetes Controller?

3:29 what you want, and the Kubernetes controller does all the actions necessary to take what you want done and actually make that reality. So essentially, if you create a deployment in Kubernetes, you specify lots of YAML. And you might specify like a deployment object. It will take that, and it will then essentially process that, see the state of the current cluster to see, for example, if that object already exists with that particular name. It takes what you desire and gets it to the end state. So for example, in a deployment, it will take the number of replicas. It will create the

4:12 replica set if necessary. It will pull down the image, and it will create the required pods. It will do everything that is necessary to satisfy what you declare essentially. Alright. And I mean, is it fair to say that everything in Kubernetes is a controller as it is just now? Pretty much everything. Yeah. Like, you know, a lot of the common objects that you interact with such as deployments, replica sets, all of these standard Kubernetes objects are essentially controllers. And they're getting down to the lowest level Kubernetes abstractions, which is things like pods and containers to the very end. So essentially everything is

4:29 Discussion: Controllers in Kubernetes Core

4:59 like layers of manipulation between to get to a state where you have pods and containers running in your cluster. Okay, cool. Very cool. All right, we have a couple of comments so let's get them on the screen and then why don't you and I just talk about the problem that we want to solve today and then we'll talk about the different ways that we could potentially achieve that then. So Vignesh welcome back, hello We have a excited. Yep. I am excited too. That's for sure. And a hi from Moz. Hey, hi Moz. Not sure why that font is a colored

5:30 What are we going to build?

5:32 black. I'll fix that later. Okay. So when we chatted about doing this we had rough idea that what if let me throw this back. I wish I'd had the foresight to find this tweet and maybe I'll try and find that in a second but I think it was in December I tweeted an image where I had taken I had taken the image tag within a pod spec and made it look a little bit weird. What I did was put the image name and then for the tag I actually provided like a semantic version constraint. Maybe that's best visualized actually.

5:36 The Problem: Semantic Versioning in Image Tags

6:12 Let me pull up my screen share. Assume this was a pod and then the image nginx and I think what I had was like greater than that's a more Versus code. Something like this. Where it's not valid and that this is not a correct image name or tag and this has no semantic meaning within a Kubernetes cluster. There's no way to really tell what this means. There's no way to really I mean, this would just fail. Right? Yeah. Because I wouldn't be able to pull it back. So what if we could write something and the Kubernetes land

6:55 that would notice this unique syntax for the image and then resolve that for us at runtime and update our pod spec. That makes sense? Yeah, absolutely. So this was quite interesting because from this tweet, I actually gave this a shot to see how far you could get, you know, whether even Kubernetes will accept this as like valid input. It does. You put this into your YAML. It does, yeah. I was quite surprised. I thought it would immediately complain because it's not a valid image name. But because it does accept it, that means it is storing that somewhere. So it's storing

7:35 that state in the Kubernetes storage system in etcd. So for example, we can then essentially extract that and manipulate it to convert it into something that is usable and that will essentially resolve and run on our cluster. Yeah. So there is there more than one way for us to make this work? Yeah. There's actually a multitude of ways. So what we could do, for example, is we could say that we want to go completely custom, and we want to write a custom controller that will look like a deployment. For example, we might give it a a kind of

7:56 Exploring Extension Points: Admission Controllers

8:17 like a version deployment and, you know, that would take the or like a versioned pod, for example, that would look exactly like a pod spec but would manipulate the image field before essentially interacting with the version pod controller and converting that versioned image name, like what you specified here, into a concrete image version and then pass it off to a pod to continue its pod lifecycle. Now another way, and I think the way we might possibly want to explore in this session, is Kubernetes provides two kinds of hooks when objects are created and when pods are

8:59 being built. Actually, this works for any API object, I believe. You have mutation controllers and you have validation controllers. So go into validation controls first. Validation controls are pretty much like what they sound. Essentially, you can provide a completely valid Kubernetes pod spec or deployment spec or any sort of spec. And you might want to validate it in some ways. So for example, you may want to look at the state of your cluster and say, oh, we should only be running this many deployments with this particular label. Or this particular deployment is valid, but it's missing

9:00 Admission Controllers: Webhooks Explained

9:34 this particular annotation, for example. And you might want to validate that and deny that resource from being created. So for example, when you create a pod spec and you add something that is invalid using kubectl, essentially you will get like a denied error if you write something invalid. That's essentially the validation kicking in. The other thing that you can do is essentially like a mutation controller. So mutation controller, actually, it'd be quite good if we pull up the documentation on this. So if you could pull up maybe the admission controllers for Kubernetes. Yeah, that one. Yeah, that one is fine.

10:28 So essentially both mutation and validation controllers are types of admission controllers. And what a mutation controller allows us to do is essentially hook into an object that's being created as part of the creation lifecycle and apply patches, apply mutations. So similar to how you can do kubectl patch, you can do that add creation time dynamically. And the way that they've implemented that in this particular controller, in this particular admission controller, is allowing for like a webhook style system. So essentially every time a pod or a deployment or something, whatever object you want to watch for gets created,

11:12 It will essentially call into a custom webhook that we can specify. And then we can then mutate the output and add anything we want or tweak anything we want before it goes on to the rest of the pod or deployment or replica set creation lifecycle. Awesome. The good news is I also found my tweet. There we go. This is my bold idea. November 13, if I have a search in December, this was wrong. But we're gonna try and make something like this work. So that was a really great explanation of validating webhooks and mutating webhooks.

11:54 I think you know, there's a lot of flexibility, lot of power just in those two methods have been able to extend Kubernetes without you know, getting into the nitty gritty of actually like writing a you know, a controller with a loop and reconciliation and ownership of CRDs and all that stuff like you know, you just maybe you just want to extend like we want to today. A primitive that already exists like a pod and we can do a cool couple of tweaks there. So is it fair to say that at a really naive level that these

12:18 Webhooks: HTTP Handlers & JSON Patch

12:25 are just simple functions that take an HTTP request out and then spit something back out? That's exactly it. Yeah. Essentially right now I believe admission controllers only give you the the object as JSON, and you will read the object as JSON. And, you know, you can use the Kubernetes core v one to unmarshal all the JSON that's provided to you. And you quite literally write, like if you've used customize or anything similar, you essentially write a Kubernetes patch. So you'll have like an operation. You'll have a path. And you'll have a value. And essentially you specify a series of

13:05 patches that you want to run. And can specify no patches, for example. You could just say the same thing that's come in is the same thing that's going out, for example. And yeah essentially that webhook is essentially just ferrying some patches which would then be applied as part of the mutations and then it will go on to the rest of the life cycle. Okay. So is that our first step then is that we wanna write some go application that we can deploy to our cluster register as a web hook, validating or mutating and just take on a request and just

13:42 pass it on. That's step one right? Yeah exactly. So yeah, it's pretty much exactly as you described. We're going to write a really simple Go application, a Go binary, and get that deployed. And it gets deployed in the cluster just like any other deployment would. So we're going to create a Kubernetes deployment. You can run it with multiple replicas. You know, it will have a service backing it and yeah, that's basically it. Awesome. I love how you say we. I feel that you're just taking pity on me because you know, I'm gonna be leaning on you an awful lot for

14:19 today's session. So let's get started then, we have a Versus Code live share set up, we both have the ability to pair program on this and type code. I'm assuming we're gonna wanna just initialize a really simple Go repository, pull in client Go that we'll probably need later and then build a Docker image. I'm guessing feel free to just tell me I'm wrong and that's not the first step. No, no. I think that sounds really really reasonable. Essentially we'll get, it'll allow us to like interact with all the different things that are involved and like talk a little bit about the

14:54 prerequisites before we get into the really, really like quite finicky stuff of getting the actual webhook installed into Kubernetes itself towards the end hopefully. So yeah, let's start it with some code. Well we also have a question. Are you ready? Because this I wonder if you've got opinions on this. Go for Mozz has asked, what are the differences between a controller and an operator? I see an operator as acting on top of a custom resource. So for example, I would write an operator if I want to interact with something outside of the Kubernetes ecosystem. So for example, if I wanted to

15:17 Q&A: Controller vs. Operator

15:42 tap in to a Cassandra cluster and cleanly shut it down as part of let's say that I got an event that's saying that a pod is about to be terminated, and I wanted to cleanly shut it down and reconcile that with my Cassandra cluster, I would term that as an operator. Essentially, it's like extending the Kubernetes lifecycle to do things with especially stateful infrastructure, for example. With the controller itself, it's more taking and interacting with your Kubernetes system itself. So essentially, it's taking something that you want done in Kubernetes and using Kubernetes as a way of making that

16:25 happen. So I would typically have a controller interacting with standard Kubernetes objects, deployments, replica sets, pods, secrets, config maps, essentially describing those primitives. A controller and operator under the hood are very similar things. You're going to have one loop. You're going to be checking for mutations. You're going have like a Kubernetes watcher at some point, maybe listening for events. So in implementation, they might look quite similar. But I think it's just a matter of terminology. And to be honest it is personal preference like depending on what you read, where you go, the terminologies are really mixed up.

17:16 Okay, first I just wanna point out that see your ability to answer and explain stuff is phenomenal. So thank you very much. Secondly, would you, I'm gonna throw a statement out there. Would you say that all operators are controllers but not all controllers are operators? Does that pass? All operators are controllers but not all controllers are operators. Yeah, I think would agree with that. You see the thing is in an operator, there's no reason why an operator again, taking, for example, the Cassandra example there. There's no reason why the Cassandra operator can't interact with both the Kubernetes ecosystem

17:59 itself. Whereas if you're writing a controller, I would probably want to scope it to just the Kubernetes aspect itself and leave the actual management of third party infrastructure to something like an operator. Yeah. Awesome. Alright. We'll tackle one more question there. Fair coming in today. And then we will actually write some codes. So Sachin has asked, is there any difference between writing controls from versus using tools like cube builder, operator framework, etcetera? That's a really good question. So with the pre generated code that you get, it comes with a lot of niceties out of the box.

18:23 Q&A: Building from Scratch vs. SDKs

18:43 It writes a lot of the boilerplate for you. It gets you started on a really nice layer. It gives you all the dependencies you need. It gives you some boilerplate YAML that you can start applying to cluster. Actually, later on when we're writing this mutating webhook, it's going to be really interesting to go through the Kubernetes phase because there's so many finicky things that you need to do in order to get these mutating webhooks hooks running. So and you know all of these things are not extremely well documented. Know so the interesting love when you generate

19:20 something using like the operator framework that might be the best source of documentation for running a Kubernetes controller. Know, because essentially that is part of the Kubernetes project and they expect others to not go it alone. Essentially they expect everyone to be using that sort of framework. Alright. Would it be fair to say that you only need to use cube builder operator SDK all these other tools when you're registered like I always think of them as being tools that simplify you know creating the CRDs applying them to the cluster providing the ownership like when you're

19:57 working at that level it would make sense to use one of these tools but for today's example where we're doing you know a web hook that mutates an existing spec that you don't particularly need those tools. Yeah that's that's exactly and also I think by doing it from scratch it's a nice way to understand how all the pieces fit together. You know all the pieces you need and also I also want to demonstrate hopefully, especially the code side, it doesn't need to be that complicated. Hopefully, we'll have to see how we get on. Hopefully, lot of the primitives look familiar to

20:33 like anyone who's written Go before and maybe not played around too much with the Kubernetes SDK side of things. Okay. Great. There's lots of context there, lots of knowledge. I hope everyone is enjoying the session. I think we just dive straight on now. We start getting some code together. If you're watching and you have more questions, please leave them in the chat. We will do our best to answer them as we go. Alright, let's do this. I'm gonna assume we just need to run a go mod in here. Yeah, that's right. We're going to need to

21:00 Building the boilerplate for our admission controller

21:04 pull in the Go SDK at some point. I've created a main. Go. So yeah, we're going start with a package main. Essentially, what we're going to have is we're going to have a HTTP server listening on TLS. So a mutating webhook or any sort of webhook that interacts with Kubernetes doesn't interact over a non TLS connection. So we're going to have to do some certificate faff a little bit later on. But we are going to be listening on a TLS connection. Now, as I say that, I have realized that we will also need some keys at some point. So for now,

21:44 I imagine we're going to hard bake some paths and then we can inject those in as a Kubernetes secret later on when we're setting up the Kubernetes side of things. Yes definitely. Okay so Mars has given us some moral support here. Let's do it. Thank you. Now this is just we're just gonna use net issue to be ready from the go standard library so I mean I'm not gonna remember how to type that off the top of my head I'm gonna assume you probably may be able to type off the top of your head but we're just gonna grab that

22:16 from Google Yeah. I'm gonna go for the time. You're gonna go for it. Right? Yeah. I'm gonna go for it. Let let's see how much I can remember. So essentially, we're gonna need http.handleFunc, I believe. And essentially, what that's going to look at is we're going to have a mutate endpoint. And that's going to call into a handle mutate function. Is that mutate endpoint convention or explicit? You can specify it in your YAML, but it is convention. Yeah. It is just what all the mutating webhooks seem to be using. So this is gonna be a

23:06 response writer, and we're going to have a HTTP request here. Hopefully, I got the order of that right. Value of type func. Oh, sorry. I think I just want to handle. And we're going to have a HTTP listen and serve TLS. Have listen on port four forty three. And that's a cert. Server oh, sorry, webhook.cert. And webhookkey.pem at some point. So we're going to have to inject those in as a Kubernetes secret. I wonder why this is not liking. I believe we do need a handle function. Is that correct? Hold on. If I get the other complete, I'll

24:20 just show people let's say pattern and handler was accepted here. So let's see. What's the error message? I think I've got in the pointers the wrong way around. Cannot use handle mutate value of func blah blah blah as we should be handler value and argumentation handler message server HTTP. I had the pointers of the arguments the wrong way around. The HTTP response writer is just a standard struct and the HTTP request is pointer. Sounds me right for not Googling. Cool. Now what we get in, if maybe we quickly pull up the documentation for an admission

25:03 Processing the Admission Review Request

25:11 controller. So if you maybe search for admission review, maybe it's on this page. Nope. No. That's the admission review. We just want that struct. Right? So Yeah. That's That's right. There we are. Oh, actually, that's perfect. So webhooks are sent a post request with content type application JSON, and it has an admission review API object in the admission.case.io API group. So essentially, that's what we're going to be getting in. And as a response, we'll also be sending back out and it says later on in the page an admission review response. So admission review has oddly

26:07 non Go standardness. It has an admission review and then a request inside it. And it also has the response inside it as well. So we'll get an admission review, and we'll send back an admission review. So let's write a really, really simple one where we're just taking in the admission review and marshaling it, and then just spitting the same admission review back out. And in order to do this, we're going to need the API sorry, the K8s API. So if we do go get. Uk8s.io API, That would download the latest API and also put it in our Go mod. And that

26:50 should be ready for use now. So if we have input as admission v one, admission review, this is the struct we're gonna drop into. Let's see if the PLS more like this. Admission v one is case.io slash API admission v one, which is where the actual admission lies. I think that's just resolving on my end. So while it's resolving, I will continue coding. So essentially, what we want to do is we want to decode body that we get from the request into this admission review. So that's going to be auto body and decode review. For

27:59 all my girlfriends that are watching, we're gonna be quite liberal with the errors that come out of this. We're just gonna send them back out. Interestingly, what you can do is within the when you're writing an admission controller, if it does return an error, you can specify as part of your mutating webhook whether you want the admission to fail because it couldn't resolve or to ignore. So essentially to just continue on with no mutations being applied from the mutating webhook. So when we initially set it up, we're probably going set it to ignore level because

28:32 we're testing around. But then we may switch that to a more stricter policy of failing if it encounters any errors. So I'm just gonna define maybe a send error function, which is gonna take an error. And what that's going to do is that's also going to take our response writer as well. Writer. We want to send back content type application JSON. And we also want to send back a status code. Think the autocomplete has sort of broken internal server error. That's alright. I'll just continue to type while it catches up. And we also want to marshal the error at

28:40 Basic Webhook Code Walkthrough & Error Handling

29:42 some point. So Marshall match. I think this compiler caught up, so hopefully you have your auto complete back. Oh. Although it keeps saying add our loading workspace. Okay. So go some needed disabled by mod read only. Alright. Okay. Go mod's just about unhappy. We'll try and fix it when you're you're ready. Cool. So what I've done here is I have output that error, And there's not much we can do if we're already in the process of sending an error and we have an error whilst we're trying to encode the error. So hopefully that will succeed all the time.

30:58 And we want to write that out. And we are done there. Let's see. I probably needed to download all the dependencies. Yeah. Hopefully that keeps it happy. Undefined JSON. Undefined JSON. Yeah. So we probably didn't import that. Thank you. And I think we're missing our listen and serve String string string handler. We need to specify a handler. Well, I think I might need to wrap this in a in a serve mux. We need a server. HTTP server. Free. Handler is going to be Linux. And then serve this and then serve PLS. And just specify the port because that's in

32:16 the address. I've got a saving main dot go here. I imagine that's not saved yet. Can include actions from Go. Thanks to s code. Very helpful. I'm not sure what it's Oh, okay. I think that just saved. So It says we have string string string and I want string string string handler. Yeah. But we don't have http.listen and serve t l s anymore. So I wonder if it's even reading that code. Let's have a look. Main.go. Yeah. So it's still got our old code there. Don't think Versus code has saved that. Let's see what is this saying.

33:34 Cancel. Let's try just maybe it's just unhappy because it's trying to run something. Let's make sure we have our dependencies. Save. There we go. Okay. So we got package JSON. That's good. Does it need to be encoding JSON? Yeah. Think you're right. Am I am I on that line? If you just add encoding slash JSON. There we are. Sweet. Let's give that a build. Okay. We're closer. We're getting there. Not enough arguments in call for HTTP error. That's line number 22. Ah, cool. Okay. So that's just on our error line here. So essentially, what we want there is we also need

34:49 to specify specify the oh, the return code. Status internal sub error. Yeah. That should work. What else was around? Handle mutate. It's got a few bugs here. Yes. Okay. Cool. So that should not be reviewed. That should be input. And if does not equal mail, send. Response writer. To the people watching, we'll we'll go over all this in a moment once we get it compelling. Yes. Yes. Could not on Marshall. Ricky says, I can't believe there's someone who can code without ID auto complete. Yes. It's not me. I'm glad it's a hail that's for sure and I would have

35:45 copy and pasted all of that http handle stuff. You were very brave there. Or silly, who knows we'll find out. Yeah, I think I think it's probably the latter there. Brave I think is a is a a I think it's giving me too much credit. You did well you were pulling that at the top of your head and it's even compiling now so. Oh wow. Amazing. Alright so Let's just take one minute to go through this talk about what's actually happening. Know it's mostly boilerplate I think that's should be encouraging for people we've not actually done

36:21 anything yet. This is literally code you could if it works copy and paste and use it for all of your admission controllers. So I'll let you just take a wee bit of time to run through this since it's your code. Yeah absolutely. So starting at the at the bottom there into our main function, that's going to be our entry point And essentially, what we're saying there is we've defined one endpoint, the mutate endpoint. And that's going to call into the handle mutate function there. We're listening on port four forty three. That will listen on local host. So we need

36:57 to set up the container port forwarding in our Kubernetes manifest. And essentially, we are setting up a TLS server. So what Go will do under the hood is it will set up a TLS wrapped HTTP server. So it will do all the TLS validation for us, everything that is necessary. We probably wrap this in a log. Fatal, just if it quits for whatever reason. And this will essentially block. So it will continue listening forever. And I've just wrapped it in a log dot fatal to print out any errors if it exits uncleanly. Yeah. So that's our main function there.

37:47 If we scroll up a little, we've defined a mutate handler, a HTTP handler, that takes in the two traditional arguments for every HTTP handler, which is a writer and the request. Oddly the request doesn't come first as an argument. If I was the implementer of this I'd put the request first just because that's how I think about it. Request then response. But here you get it the other way around. So yeah you have just a standard HTTP Go boilerplate function, which is taken in our request. The actual payload itself is in the body. So we have R. Body.

38:31 And essentially, we need to write out any sort of response to the ResponseWriter. Now ResponseWriter is quite literally an IO buffer under the hood. So if you were using Netcat or something like that, you would have to send out your response in the same order. So you need to write the status code. You need to write any headers that you need to write first. And then you write out your payload in the end. All right. Now I guess I have a question. In our handle mutate function, we're not returning anything? Yeah, that's right. So the

39:09 handler has no mutation arguments. Sorry, it has no return arguments. You know, it doesn't return anything like an error or anything like that. Think about it from Go perspective. If a Go is serving a HTTP request, within the boundaries of that particular function, anything that could error, you know, would have to be like a panic level error, like where the entire program just crashes. Otherwise, all the other errors are meant to be handled in the request response lifecycle. So for example, if you encounter any sort of logical error with the request that comes in, we've written a boilerplate function to send out

39:51 an error. Because we'll be using this quite often throughout the handling of the mutation itself. Especially when we're encoding things back and forth, there's always places where things could fail. And in Go, it's good practice to always handle the errors as soon as they come in and not ignore them unlike Java where you just have like a big try catch block. Okay. Could we just explain like, okay, so if there's multiple admission controllers registered with our Kubernetes API server, you know, and we're one of many. How do those interact with each other? How are they modifying the request? Is there

40:35 ordering? Is there priorities? How how does that work? That's a really good question. The ordering of admission controllers is well defined like for within Kubernetes itself. I'm not quite sure how ordering works if you have multiple mutation controllers and if there is any preferred ordering. So I may need to look that one up. I'm not sure, actually. If I'm gonna keep throwing hard questions at you. We could look this stuff up. It's I think that's pretty normal. If I mutate the object does it rerun all the others because the state has changed or does it continue from

41:20 where that happened? Do you know that? I believe it continues from where it left off in the life cycle. So it just continue going as like one big pipeline, like end to end. Like a middleware layer of some Yeah, exactly. Yeah, like a middleware layer of some kind, yeah. So essentially you would want to order these so that you have your, the thing that you want. If you require any specific ordering and if ordering is possible, that you have ordering in the sense that if something in the last in the order relies on something from

41:54 the first, then you specify in that way. Alright. Well, I think the end of that will come out as we actually apply to our cluster. So we do have a go binary right now which I expect to fail because of those serps. That's right. Okay. Well, did nothing. Alright. We'll fix that in a minute. Now we want to provision certificates, build an image, deploy it to Kubernetes. Yeah. That's right. So what we want to do is essentially give Kubernetes everything it needs to get this up and running. So why don't we create a folder with some manifests?

42:26 Preparing for Deployment: Docker & TLS

42:46 And then we can start creating an image and getting that applied. Actually, we should probably first create an image, a Docker image. So I imagine we're probably going to want a Docker file. So we set our working directory, we copy everything to it, we run a go build that should give us a binary and I'm not gonna worry about adding like a smaller image and stuff like that. We happy just to deploy this one as is. Yeah. Absolutely. So Kubernetes, semantic version. Terrible name but we'll go with it. Yeah. Let's go with that. So we should be

43:43 able to do an image belt. Let's call this Rawkode. We're not using semantic. I'm just gonna take all the credit and you know, image sync. Absolutely. We built this and what did I get wrong? Oh, it my get the Go image wrong? That seem like a. It's go Lang. Is it go Lang, yeah. And we'll just do Alpine. Keep it as small as we can. There we go. That build should be pretty quick. Pulling down our objects. Alright, let's tackle a question about that build. So okay, mods gave us a comment and then a question. So why is auto complete

44:27 and not working? Have I got it disabled and what Versus code theme am I using? Autocomplete was not working because it was dead load and dependencies in the background and then it was failing to run some sort of compile step or something. It seems to be working now, so hopefully we're okay. I think this is the plastic steam. People always ask for teams, people love that panda syntax. There we go. This is panda syntax. And then was had a question does a certificate or should this certificate signed by the Kubernetes CA or self same work fine?

44:52 Certificate Requirements

44:59 Yeah. So we're gonna have to issue a certificate that's signed by the Kubernetes CA. You can provide a CA bundle, but that will also need to be that will be validated by Kubernetes itself. So to avoid a little bit of faff, we can I think it's best if we use the Kubernetes CA, because that's already going be automatically trusted by Kubernetes itself? Alright. So question time then for me this time. We're gonna write Kubernetes manifest to deploy this. Yep. As the standard way for admission controllers to have a deployment with replicas one, do people use a straight up pod

45:00 Creating the Kubernetes manifests

45:35 or staple sets? Like, is the the facto here? Yeah. So we'll create a deployment and we're we you can have as many replicas as you want. So we could start with a replicas one. But imagine if you want this to work on a highly available manner, especially if it's stateless and idempotent, then essentially if one request fails, for example, because the pod had died, the mutation hook itself, then it would just reroute to another pod. So yeah, just a standard Kubernetes deployment. And we'll also have a Kubernetes service as well, which is going to be fronting our

46:15 webhook. And then gets the interesting bit, because we're going to write a mutating webhook configuration. Alright. Did that look okay? Yeah. I think so. Yeah. Alright. Let's see if we can apply our alt Kubernetes directory. Lovely. What did I build it wrong? Yeah. I spelled it differently. No. Kubernetes context? There we go. Let's reapply. No container. Okay. So we have an error but we did deploy successfully. I think that's just because when we run our binary, nothing happens. And if we pull the logs, it's just unable to open our certificates. Yeah. I'm not quite sure why why it's not

47:41 why it's not running properly on your machine. I don't know if it's actually erroring out. But that's the the the line there is what I expect. Oh, wait. Yeah. It runs successfully locally. Interesting. Could you try running it again? I'm just gonna go build on the local machine. Oh. There we are. That's I don't think we had the latest binary built on the local site. Classic. Cool. So we are going to have to inject in these certs. And in order to generate them from the first place, we going to use a tool called CFSSL. So this is a set of tools that

48:17 Generating TLS Certificates with CFSSL

48:22 Cloudflare have provided, where it allows you to specify a certificate configuration in JSON format. And sidesteps the need for open SSL. I can never ever remember the syntax for open SSL. I always have to Google it. It's really obscure. Whereas with CFSL, at least you can reason about what you've generated. So let's go and create that. So we're gonna generate a certificate signing request. So maybe if I create a new directory in here, Do you have a CFSSL installed and also CFSSL JSON? Suppose so Well, I I just installed CFSSL. Do I need to install another package?

49:11 Yeah. You'll also need CFSSL JSON. No dash, I think. Let's try do that. Does it come by default maybe? I'm not sure. Let's find out. So we wanna generate a CSR. So if we do CFSSL JSON, and what we're gonna do is we're gonna specify the CSR in JSON as in c f s s r JSON as in one command. Oh, one command. Yep. It probably might come by default. Oh, it does. There we go. I don't know if my because I've not reloaded my shells, I'm not getting auto complete on the commands, but it is there. Okay.

50:01 Cool. So if we go back to the editor, I'll show you what the CSR looks like. So essentially and I am copying and pasting a little here. Essentially, what we're going to do What should I be looking at here? Sorry. I've created a new directory with And then there's a file called CSR. Json. So what we have here is the declaration of the CSR that we're going to create. So what did we call the Kubernetes service? Was it semantic? Semantic, yeah. Yeah. Okay, cool. Semantic. And essentially, we allow listing those particular hosts as part of the certificate signing request.

50:51 That's going to be the common name. And yeah, I think we have everything we need to start generating the actual certs themselves. So what we're gonna do now is if we do SSL JSON. So if you cat your CSR dot JSON yep. And then pipe that into oh, is that saved properly? Hopefully I've just saved it. There we are. Yeah. If you pipe that into CFSSL gen key, because we're gonna have to generate a key first. And you also need specify a dash. And then we're gonna pipe that into CFSSL JSON dash bare, b a r e, bare.

51:52 And we're gonna generate a server key. So if you what did we call the you're gonna need one more argument, which is the the name of the the output file. And what did we call it? We called it webhook. So if you call that webhook, that's just webhook. It will generate the the required prefixes because it can generate two files. Cool. So that's now generated two files, hopefully. I did. Webhook. C s r webhook key dot pem. Yeah. Great. So what we're gonna do is we're gonna take the certificate signing request, the CSR, and we're gonna base 64 encode that, and

52:17 Creating and Approving the Kubernetes CSR

52:36 we're gonna create a certificate signing request for Kubernetes. Okay. So we want the CSR Yep. Base 64. So That's right. And if we we need to create one more manifest which is the certificate signing request for Kubernetes itself. Yeah. I don't think I'm gonna have a magic template for that. So let's put that here and then let's go find Kubernetes CSR YAML. Let's see if we got one here. Some RBAC stuff. And there we go. So we want this base 64 included version. As part of the request. Do we need to modify anything else within this?

53:42 Yes. We're going to modify the metadata to be the name of our program, semantic. The name is going to be semantic. We also want the signer to be the kubelet itself, the serving kubelet. So that's going to be Kubernetes.io/kubelet- serving. Yeah. And, essentially, we want the mutating webhook to be authenticated with the system itself because it's gonna be interacting with Kubernetes within the Kubernetes certificate framework. Yeah. I think that looks good. So if we apply that we run get CSRs. We have That's right. Yeah. So you have a CSR that is pending. And what we will need to do is

54:40 we can, within kubectl, approve it. So if you do kubectl certificate approve, I believe. Certificate approve. Then the name of our certificate. Amazing. Cool. Now we'll need to extract out the It says failed. The certificate. Oh, could you describe that? Not particularly useful as far as debugging event score. No. Interesting. Yeah. There's nothing there. I mean, the certificate does look valid to me. Oh, I think I wonder if we need to add a few more usages because I noticed that your your there's only one usage there, which is client auth. Mhmm. I think we need

55:05 Debugging Certificate Approval Problems

55:54 KeyCypher and digital signature as well. Okay. So just delete the CSR and then we'll throw it back. Yeah. If you delete the original CSR. Let me see if I can add the other two quick. Just make sure that's correct. So what I've done is I've added the last two lines there into the usages, which is digital signature and key in Cypherment. Now let's give that a shot. Failed. Let's see. What else do we have here? Your request does look valid. Let me just base 64 decode this. Bay is a new lane or something. Well, the certificate request looks valid to me.

56:59 Yep. Looks good. Interesting. This was gonna be a faff. Yeah. Oh, I'm nervous that Docker for Mac thing. Maybe I shouldn't have skipped over this R back stuff. Well, no. Because I'm just root on this. It doesn't really matter. Okay. Docker for Mac CSR failed. Let's see what Google tells me. I think you do need to remove the new line. It might be a little bit picky. I'm just looking at the Kubernetes documentation. So when you base 64 encode it, if you could pipe that into t r dash d and remove the new line. Okay. So we want to cat the web

58:11 hook CSR but we want it to be our dash d and then base encode. I think we want to base so according to this, it a base encodes first and then oh, actually then it's just removing the new line. Yeah. From the b 64. Okay. Maybe it's not that then. I mean, let's see what does that do. Well, you also need to specify delimiter. So you want to specify in quotation marks dash n. Yeah. Oh, no. We don't want that. Yeah. Just remove the new line. There's a kill check from no base 64 dash w.

58:53 I'll look at that in a moment. I think because I casted the CSR and base encoded, I actually think that it should be alright. And normally if there was a new line character and on the end you would see a really familiar like we'd see the two equals with the padding and something. Think that's Could you possibly specify in the metadata? This is a long shot in the metadata, specify a namespace. So if you do semantic and what namespace are you in default? I think you need to specify that in the name. So if it would just be semantic

59:34 dot default. This is really much a long shot. Let's give this a shot. We like long shots. Alright. Let's delete CSR semantic because you failed. Let's reapply all Kubernetes. Let's get our CSR. We can see it's pending. Yep. Certificate approved semantic. Not default. Not default. Yeah. And get CSR. Oh no. Alright. Let's see. What have we got in our cube sys? I mean, there must be logs on this. Right? Yep. Likely. So if we get the logs on the API server. No. I'm not sure what would be responsible for this here. Yeah. Don't think there's anything useful there.

1:00:48 Let's try controller manager. Nothing useful here either. Let me give this a shot on a local. Okay. Let's try Kubernetes admission controllers CSR. I'll just Google random stuff just now. Let's see what happens. Oh, I think the signer is wrong. Is it kubelet server client kubelet? Or no, you said serving something. Right? Yeah. Kubelet dash serving. Yeah. That does look okay. Yeah. Do we have system modes? Organizations are exactly system modes. Common name starts with system node. Did we get that correct? So our common name is let's just make sure that our CSR is correct.

1:02:22 Semantic.default.service. Semantic Default Service Cluster Local. That's fine. System node I wonder if that's different on Docker for Mac. Oh, possibly. Ricky suggests we just restart my computer. Maybe not today. Yeah. I wonder if the docker for Mac as cluster DNS is different. Let's jump into this doesn't work because it's a funny distro list or something. Let's try. Yeah. It's not gonna work. How can I get the full thing? Let's go. Let's run something. I'm just gonna get a debug container in here somewhere. So Yep. Like a busy box or something. Yeah. Something I can do a DNS lookup.

1:03:27 I'm not sure if there's a quick way to get the actual cluster DNS name. So and from Docker from Mac at least. Let's just run a bit to 20. Whatever Ubuntu gives me by default. And then my old favorite sleep args infinity. Alright. Get pods. Should've maybe have picked a smaller container image. There we go. It's running. So let's get inside of this. Run bash. Curl? W get? Tech? Alright. Install Baint tools. I can never remember the name of the Bain package. Bain tools, DNS utils, DNS tools is one of them though. Yeah. I guess I can just do curl. It's not

1:04:32 really that important. So we wanna confirm So what's your current theory at the moment? That Kubernetes dot default dot cluster dot local won't work. Default dot s p c dot cluster local. If that was DNS, it would probably fail for another reason. Right? No. NS tools. Details. Vague details. Oh of course. By nine. Yep. Okay. Maybe on. It should I wish that package name was just consistent across every distro but I seem to fight with it every single time. Eight twenty seven, that's annoying but alright. Dig Kubernetes. Default. Svc. Really? Ubuntu take command package. I wish I could say that's the first

1:05:56 time I've googled that. Banger Tails. Did I not try Banger Tails? Did I try everything but it? And it doesn't stall that. Do I have NS lookup? Big. I can't believe this is what I'm now feeling on. DNS utils. DNS utils. Seven has asked if I'm on an m one. I'm not on an m one. Unfortunately not. Yeah. So that's I think seven's confirming my hypothesis is that the cluster DNS is actually different. They're suggesting doc or dot internal. So I just want to confirm that. So Kubernetes default SVT. Dot cluster. That does the result actually.

1:06:52 Interesting. Could we have a quick look at the could you describe the the CSR again? Semantic dot default. Right? The CSR name should be semantic dot default. Yeah. But that system node at the start, what is that? Because that doesn't look like a valid common name to me. Is that wrong? I so according to the documentation, that is what it should be. The common name should start with system colon node, and then it should have a colon with the name of your your full common name. So name starts with system colon node colon. Yep. So the organization needs to be exactly system

1:07:55 nodes, which we have. Yeah. I wish there was The certificate signing request looks correct to me as well. Yeah. So the cube controller manager does handle this. Let's try the logs again. Logs cube controller manager. Nothing there. So what I've done is I've generated a new a new certificate signing request, like a an absolute long shot. Well, that's the face we got from Moz. He says regenerate the CSR. So let's do Exactly. Now that's going to generate a new base 64 encoding. Yeah. And what we're gonna do there is let's remove the original object from Kubernetes

1:09:43 and recreate it. Okay. So delete. So it's our semantic. It's gone. Yep. And default. No. That that one failed with an error. Yep. Can we just check to make sure it's still not there? Perfect. Yep. All good. So now we want to cat our CSR. Yeah. I've called it. No. I've already done that. So I've called it server.CSR this time. Previously recorded webhook, but I called it server. So, yeah, if you copy the base 64 What was that thing we got dash w zero? No. I think that command should be correct with the t r.

1:10:39 That will remove the new line. And then we can if you pipe that into p b copy, that will copy onto your clipboard. Yeah. Okay. That's it. And if we go to search.yaml and put that into the the request. K. Has that been put in? Yeah. I don't think it changed which means that it's probably the same. No. I don't I don't think it should be. It would have generated with a new key and a new algorithm. Right. Well, it's maybe just something in the middle that's changed and the end is consistent. So let's try applying it.

1:11:32 So what I've done is I've just taken the output from the terminal and copied and pasted it myself, and it definitely changed. Weird. Interesting. Let's try and apply the I hope it will work. Oh. 10 block type must be certificate request. That's just come out there. Into base 64. Yeah. That's a certificate signing request. That is a pen request. Correct. To do this kind of Oh, yeah. After Kubernetes. Yep. The certificate spec request has invalid value. That's really bizarre. Maybe you're right. Maybe it didn't change. Yeah. That seems better. Cool. That's now created it. I'm wondering whether we truncate in the new

1:13:08 line didn't work properly as as I'd expected. Now let's try and approve it. Alright. That's long shot. Approved. Oh, no. Okay. I don't know if this is a Docker for desktop thing. Let's let's let's change clusters. Right? I have other clusters available. So You have a mini cube cluster? Yeah. We can create one pretty quick. I also have a fully fledged nine nodes cluster on Equinix metal that we can always resort to have to do it. Oh, that'd be great. Maybe slightly overkill for building the mutation webhook but would be fun regardless. Let's see what many cube gives us in.

1:14:07 We'll just apply this entire certificate thing right over the top approve it and I bet you magically it's just gonna work and this is some weird Docker for Mac thing. Yeah. I can't think what else it is. What we have looks like what's in the Kubernetes documentation so let's Exactly. Okay. Come on, minikip. While you're getting that up and running, I'm gonna try and see if I can get Alright. So if I run get nodes, I just wanna make sure we're on Medikube. We are let's apply. I should be able to run get CSR. Are you running the latest Kubernetes

1:15:00 on mini cube? It was running one eighteen four. Do we need one twenty? I think we'll need one twenty. Yeah. Alright. Let's turn that down. Start Kubernetes version. Alright. I'll just take another moment. The image isn't too big. What version were you using on your Docker? Was it one twenty as well? It was. Yeah. So according to this random post I found on the Docker forum, always to be trusted. Right? It seems it wants the signer to be suffixed with docker dash desktop. So as a weird Docker for Mac thing then. Yeah, possibly. Alright. Well the only thing that changes now

1:16:10 anyway is we just have to rebuild the image using the many cube context. Other than that we should hopefully once this spins up, we're back on track. Fingers crossed. Cool. Almost foiled by certificates. It's just the way it goes. I did mention at the beginning that it would be a bit of a fad. So once we have these certificates, essentially we'll have everything in place for Kubernetes to interact with this webhook and ready to intercept some requests. Yeah. We've not even done the hard bit yet. We've got a pars and resolve that random semantic version thing I've thrown at

1:16:54 us. It won't be that hard, I'm sure. So there's a really great Go package that I came across a couple of weeks back called Semver, which is really quite nice because you can specify a semantic version and specify, like, grab a list of other semantic versions and see if they match and also sort them. So essentially, you can say, oh, if you have more than one dot 15, and then comma less than one dot 16, it'll accept everything within that range but not 1.16 itself. So everything above 1.15 but not 1.16 itself. So it does all the hard work of

1:17:37 the stuff that we actually need to do. Exactly, yeah. It does all the hard work for us. Yeah. Oh, it failed. Did that one fail as well? Yeah. I'm gonna describe this other one that we have in the mini cube cluster though. What's different here? The sign is different. Kube API server client Kubler. That's the only thing I can see that's different. Semantic dot default. We added the dot default. I guess we could always remove that. Right? I think I know what the problem is. In the usages, the documentation says you need server auth and

1:18:41 we've got client auth. So let's remove client auth and change that to server auth. I would hate for this to be the issue. Alright. That's the least. Well, you know, these things are here to teach us things. So let's approve. Yay. Oh, dear. Oh, no. Oh, well. Oh, no. Okay. Cool. Right. We we we're we're back on track. Do you wanna give that a very quick shot with the docker version as well? Or would that be too much faff? No. It's alright. Let's just pull it back to Docker desktop. Reapply. Because then we'll be able to avoid

1:19:30 the having to build the image into Minikube itself. Approved. Oh, well, that's an old one. So let me just Yeah. That's the old one, I think. Apply. Approve. Oops. Approve. Go back the array amount of commands. Issued. Yeah. Yeah. Amazing. Cool. So from there, what we need to do is we need to extract out the certificate itself, the certificate authority data. So we've approved the certificate. And now we need to extract the CA from Kubernetes itself to put into our bundle. So what I'm going to do is I'm going to put the command just in the file that you're looking at

1:20:00 Creating our MutatingWebhook configuration

1:20:20 Mhmm. For you to copy. Oh, you can just type it into the terminal if you want. Oh, yes. Actually, yeah. That would be good. A weird wraparound thing. That final pipe should be there. Right? Yeah. That final pipe shouldn't be there. It did a weird wraparound thing for me, but Okay, cool. So this is the certificate authority that's for your Kubernetes cluster. And what we're going do is we're going to reference that as our CA bundle. So this gives us the last thing that we need. Our Kubernetes cluster now trusts that certificate. We've done all of that.

1:21:09 This is the last bit that we need on our side. So that's So am I just copying this and then creating a secret of it? So yeah. What you're doing is you're you're grabbing that and you're creating your base 64 decoding that as well. So we need to base 64 decode that. So why do we need this? Because this is already in the cluster. Right? Yeah. So this is the certificate authority that has signed the request. So we're gonna have to put this into, where do we need to put this? We need to put this into the webhook itself.

1:21:48 So we basically, we need to specify the certificate authority bundle that has trusted our webhook certificate. So imagine it like this, right? Imagine that we didn't use the Kubernetes CA to issue the certificate for our webhook. We need to tell Kubernetes, where do you go look up the authority for that certificate itself? So you can specify that as part of your configuration. So one other piece that we need to create is a mutating webhook configuration, which is, hey, Kubernetes, go and inject this webhook into my cluster and be ready to start calling it. So maybe why don't we create that first

1:22:10 Defining the Mutating Webhook Configuration

1:22:28 and then hopefully this will make sense why this is necessary. Okay. So let's create admission.YAML. That's right. And what it is is it's gonna require a little bit of boilerplate. If you look for mutating webhook configuration Let's see if we've got one here for kind. Nope. If not, then I can get us started with the boilerplate that I have on my end. If that's easier. That's weird. Yeah. If you've got it there, we can just do that. Don't know why my YAML search didn't work there. Cool. So we've got an admission.YAML. So essentially, what we're doing is we're starting

1:23:22 with the mutating webhook configuration. We need to give it a name, like all Kubernetes resources. So we're gonna call this semantic, just to keep things consistent. We don't need a label, I don't think. Yeah, from there, webhook itself. Now, this is going to be the name of the service that needs to be called. So for us, that's going be semantic.fault.service.cluster.local. Does that look right to you? That is right. Yep. Cool. We're also going to need some rules. So essentially, we need to tell Kubernetes when this when should this webhook be called. Like what kind of operations are we watching

1:24:12 for? What kind of resources are we watching for? What API versions? So here I have just pasted a set of rules for we're watching for the creation of pods. And the good thing about this is that it's not going to look for just manual creation as well. Like if another controller is creating pods, it will also pass through appropriately. I mentioned earlier that we want to specify, like, a failure policy. So what happens if the mutating webhook fails? Like, you know, for example, the the webhook doesn't exist or, like, whatever. Now, default policy is to fail,

1:24:55 which means that our cluster will essentially error out and say that we couldn't call the webhook. I have now failed the creation of the pod. We probably don't want that from the get go. So we're going to start with ignore. So what this will mean is that it will try and call our webhook as like a best effort thing. If an error is returned or a failure happens during the call, it will just ignore that and move on. Now, the other thing that we need, lastly, is a client configuration. So this is the configuration for

1:25:11 Webhook Configuration: Rules, Policy, Client Config

1:25:30 how our webhook should be called. I mentioned that we need to provide the endpoint and the port. So we need to provide the service itself. We need to provide the path and the port. So essentially, because we've specified a mutate endpoint, we're going to pass in that mutate endpoint there and port four forty three. And we also need to provide the name of service as well. So the name is going to be semantic. Now, I mentioned earlier that we're going to need the certificate authority itself. That's also part of the client config. So when you copy the certificate authority data of

1:26:17 your cluster, that's where it goes, the base 64 encoded version. And that essentially lets it know what certificate chain it should be looking for. And because we're using the certificate authority of Kubernetes CA, we need to paste in that chain here to let it know that that's the certificate authority it should be using. Alright. Got it. To validate our cert. So do you have that copied? Could you paste that in there? I'm gonna copy it again just in case. So oh, yeah. Didn't run the command. There we go. So that should be it there. Cool.

1:26:53 What was I complaining about there? Missing property namespace. Oh, I wonder if we need to find namespace as well. Oh. Says this must require entry side effects. Effects, yeah. So when you're creating a mutating webhook, essentially you can specify whether your webhook is going to create any sort of side effects. So is it going to interact with anything that's not been passed in? Now for us, because we're taking in actual pod that's being created and just mutating that as part of our patch, our program has no side effects. So we're going to say side effects,

1:27:35 none. Now, admission review versions. This is quite an interesting field. So right now, what we have is we have resources that are specified as part of our API versions, like v1 pods. Now, when Kubernetes came out, a lot of these resources, like deployments and replica sets, were in alpha one, beta one, beta v1. What admission review versions does is that it will cast older and newer versions into the version that you are requesting. Now because pods is currently v one and we want it to remain v Essentially we're gonna say admission review versions just v one.

1:28:23 Does that make sense? Yeah. I think so. Can I just check with you? Are you okay to keep going? Cause I know that our session was supposed to end around now. So if you're available, we can keep going. If not, we can always schedule a part two. What works for you? I'm totally fine to keep going. Yeah. As long as you're okay with that. Yep. Perfect. Cool. It would be nice to get this mutating webhook up and running and then we can possibly leave the semantic clip for part two or whatever works. Cool. So that is all we need for this

1:29:01 particular resource. So let's try and apply that. Alright. So I'll just do admission. That is now created. Cool. Now if you fetch that resource Oh, mute it in. Describe it? Yeah. If you describe it. I I don't think it's gonna be very informative. Essentially that's now created our webhook and our service will now be called each time. Now I think the last thing we need to do is just to set up our secret so that we are injecting in the certificates for our pod. And then we should be good to go. Then the whole life cycle should be up

1:30:00 Deploying the Webhook Application

1:30:02 and running. Okay. So what we want to do is add a volume. Yep. No, don't all complete that. We want secret. Let's see if I can get nope. That's not it. What am I doing? Defining a volume. Right? Okay. So Yeah. You're defining a volume and that volume will probably be injected from a Kubernetes secret. Yeah. I don't know why that's not what it offered me a second ago. There we go. And the secret name, so We'll just call it as semantic cause we haven't created a sequence yet. Oh, haven't created it yet. Semantic. And

1:30:51 we're just going to take everything and then we want to add a well, this is my debug one as well. Let's copy that. Oh, yeah. I think you're in the debug manifest. And then we want to add volume mounts where we mount our semantic no, certs. Certs. Yep. And we wanna put that in slash certs in the root directory. Done. Sweet. And we also need to create the secret itself. Yep. I call it semantic. And now we need our certificate. Yeah. So we're gonna have two things here. We're going to have the key itself, and we're also going to have the

1:31:53 what else do we need? Cert. We need the cert itself. Yeah, we need to extract the cert from Kubernetes. How do we do that? I did have that command safe as well. Grab the cert from the CSR. So let's get the key in first. So the key is going to be webhook dot what do we call the binary? In the binary, we call it webhook dash key dot m. So what am I doing? Sorry. So what you're doing in the data is where it says key, you're probably going to want to rename that to webhook-key.pan.

1:32:46 Yep. Got it. Yep. And then that's going to be a base 64 of the server-key.pan in your in your root directory. All right, next. Cool. And we also need to grab the certificate from Kubernetes itself. So we submitted a certificate signing request. We now need to go and fetch the certificate itself. So if you do kubectl get CSR and then semantic dot default, you will need to yeah. Actually, I wonder if it does come in yeah. There we are. Certificate. So if you just copy all of that, that will be our certificate itself. Amazing. You think Kubernetes would provide a slightly easier

1:33:46 way to access the certificate signed with its own CA system just to, you know, instead of extracting out creating a secret where I could just say, hey, pass this through to testing but that I'll run a bit, maybe we can wrap it earlier. Okay, so we now have a secret. These names have to match what we have in our main dot go which were webhook.crt webhook - key Pem and webhook.crt. Amazing. Which means if I don't mess it up and do an apply of everything this time, I guess we can just do the deployment that should create our secret update or deployment

1:34:00 Deploying our admission controller

1:34:25 with the volume notes and we should see our new pod scheduled and it is running. Oh, amazing. One thing we didn't do is we didn't add any sort of logging to bug plants. We're not going to know if our mutation function gets called. So let's do that very quickly. So Oh. Very, very fancy. Yeah. I'm a serial print line debugger. I very rarely step into like GDB or like any sort of like code dev or anything like that. It's done me well so far. Yeah. I resort to print debugging and then I'll cave and jump to a debugger if I feel

1:34:57 Testing the Basic Mutation

1:35:23 that I'm wasting too much time like when I start doing I am here 76. I'm like okay maybe I should get a little bit more sophisticated there and I lie, I never do. I am here seven to six. Put swear words and it's Oh yes. Yeah. I am I am the speller debugger for sure. Alright. So we built that image. All we need to do is delete our pod and what we should see now is if we run logs and create a new pod that is called, right? That's right. So if we maybe just, know, kubectl create pod with like NGINX

1:35:58 or something, we should expect for it to be called. Yeah. I'll just pop in here. I'm just gonna make a chain. In fact, no. Because this works in the pod and not deployment level, right? I should be able to just delete the debug pod which will get recreated. Yes. If was tied to a deployment. Ah, lovely. Amazing. It all works. That means all of our certificates are correct as well, which is great. I don't think we need any more certificate debugging for the rest of the stream. Alright cool. This means that the like end to end chain works.

1:36:00 Modifying the Pod spec

1:36:35 I think how far do you want to go? We could get all of this up and running and now it's just pure writing code. Well, why don't we, if you're happy to work till on the hour that gives us around twenty minutes to hack together something with the semantic version and you know, I don't think we need to show a complete end to end. We can, you know, we just want to know show how we iterate on this. We work with a client go We'll do a little bit of that and then maybe we can finish it offline and push

1:37:01 it to get up for people to to kind of play with in their own time. Cool. Amazing. That sounds good. Sweet. So rapid coding. I will try and talk through all of the bits that we are doing in really rapid fashion. So essentially, what we have when we last left off this code is we've got an admission review. Now if we have a quick look into that admission review itself, if you click through to the admission review oh, no. No. No. If you click through to oh, actually, well, yeah. There it works in the in the Go docs as well. What

1:37:37 you get in the admission review, if you command click on if you escape that, if you command click on line 15 where it says admission review, yeah, it will take you to bit of code. That's right. So what you have there is an admission request, and you are meant to specify admission response. If you click through to the admission request itself oh, actually, yeah, it's just a few bits down. What you have is you have the UID of the request that's come in, and you also have the object itself. Now the object has a type of raw extension,

1:38:17 which is like that might as well just say interface, like Go interface. Essentially, you can cast it to the type of object that you expect. So we're going to unmarshal it into the type of object we expect. So essentially, what we want and what we expect from our mutating hook, the contract that we've established with Kubernetes, is that it will send us pod objects. So we can try and unmarshal this into a pod object. So if you go back to the code itself, what I'm doing is I'm initializing a pod object. And I'm going to try and JSON

1:38:54 dot unmarsh all of that. Review actually, that's going to be input object. Raw into the pod itself. And if that fails, same deal here, could not unmarshal pod. So here, what we have right now is the Kubernetes pod. We're probably going have to add the import as well. So that's going to be API core v1. Yeah, that's right. Now what we need to do now is essentially try and figure out what images are in the pod to see if there are any images that we want to semantically modify. So within the pod object itself, if we

1:39:09 Implementing Image Mutation Logic

1:39:51 click through to the pod object, you have the pod spec. And within the pod spec, you have containers. And within each of the containers, have a container object. And within each container object, have an image. Does that make sense so far? Yep. So what we're going to do is we're going to simply iterate through, just like you would in standard Go, if you have a slice of, well, anything really. Pod dot spec dot containers. And what we're going to say is, let's say that we just want to take that particular container and apply any sort of patch to it.

1:40:40 I was just going to suggest, if we detect or why don't we just replace it with NGINX latest just to kind of show it working? Does that make sense? Yeah. That's exactly it. So what I'm going to do is I'm going to create a map with our patches itself. If we had a bit more time, I'd probably use a proper ghost struct. And what we're going to say is, patch equals append patch map string string. How I mentioned, similar to how you would do a Kubernetes style patch operation, you have three fields. You have the operation

1:41:25 that you want to do. So here, what we want is we want to replace the image name. You have the path, which is the name of the object itself that you want to replace within the YAML. This starts at the very top level. So just like you'd have a JSON path, you start the spec. And then within the containers, you have a list. This is where the index comes in handy, because we can inject the index in. And we probably want to sprint off that by the image field itself. And what we want to do is we

1:42:02 want to set its value to just NGINX latest. Useful. Does that make sense so far? Yeah, quite useful. Yeah, I think we just need assignment. There. Can't this is the patch Yeah. So we need to assign it to the patch map. So that was a bug on my And we don't need the assignment there. We have what we're doing now is we've generated a list of mutations. What we're saying is for every container that you see, replace this image tag with NGINX latest. Very, very helpful. And what we need to do is we need to

1:42:47 wrap that up into a series of bytes. Send response. We'll deal with that in a second. Now, like I mentioned earlier, every single thing that we work within admission controller is essentially an admission review. Yeah, thank you for correcting that. Is an admission review object. So we're going to generate a response, which is our response review. And that's an admission v one admission review. And that's going to be we can copy the meta fields from the input. TypeMeta, because we want the same output to go out. And the response is going to be an admission v1.

1:43:48 Admission response. And within there, the UID, which is going to be input do it. But there's an interesting flag that you can specify, which is with the mutations that we've seen, or with the input that we've seen, do we want to allow this request to go through, or do we want to deny it? Now, we want this request to go through, but with a series of what we're saying here is, this request looked fine to us, but make a few tweaks, please, for us. And you don't need to specify this, but just for niceties, admission viewable.

1:44:39 JSON patch. If they add any support for any other patch types, So we've now generated a response review. We need to marshal that again. That code, REST flights. to head to oh, actually, similar to what we have here. And need We need to that's going to be respite. JSON. This is status Okay. I've got some error handling to do here. That just Could not generate patch. And could not the rest view. Could not generate very helpful error messages is here. Okay. So let's have a very, very quick walk through of what we've done here. Essentially,

1:45:55 we have taken in our input object, unmarshalled it to a pod because that's exactly what we expect. We've generated a series of patches. We have created a response set of patches. We have wrapped that up into an admission review response object with a list of patches. And we've told it that we want to allow this but with a series of patches. And we're now sending that off into the wire. I think it's important to note for people watching, like the only thing that's really bespoke and not boilerplate here is just the patch that we want to apply to the

1:46:32 manifesto has come into our admission controller. Everything else you can copy and paste pretty much character for character. You're a machine for typing that out so quickly by the way, that was awesome. This is where your magic happens, whatever modifications you want to make within your controller, this is the secret sauce here and this is JSON patch format, you can Google that, there's different types of operations and ways to work with it and I know it looks a little weird at first sight but you do get familiar with it over time. Now this is obviously the most useless contrived

1:47:03 example that we've put together but we could build this now and regardless of what we apply to our cluster is gonna overwrite it with the NGINX image and what we need to be careful with when we deploy this is that none of our system pods or anything like get deleted. Yeah and same with the hook itself. Yeah. That'll be fun. Alright. Let's rebuild our image. We'll delete our pod. We'll let that come back up and then we will make a change to our debug pod and we should see that image change from the Ubuntu one that we used

1:47:27 Verifying Basic Image Mutation

1:47:40 earlier to NGINX. That's right. I'll just give Google a little second here. I guess we are pulling in a few extra dependencies now as well. Yeah. The core v one has got to put in a lot of Kubernetes. So if I run get pods, we can delete pod semantic. So that's gonna get the new version of our mutating webhook. What I'll also do is just pull up the logs for that here. Let's see if we get the new one. We do. Excellent. And just to show, let's describe our debug pod. We can see here that the image is

1:48:31 it been too. So assuming I've not got my understanding incorrect here, when I delete this, a new create pod is going to be initiated and the image should be changed. Right? That's what we expect. That's That's what we expect. Yep. Alright. Let's see. So delete delete. I don't need to wait for that. I'm just going to go straight into describe. We should get a new ID here. FC. And it works. Oh yes. Success. We now have a mutating web hook admission controller making modifications to every pod and in real time as they had our Kubernetes cluster. That is awesome.

1:49:00 Resolving the semantic version constraint

1:49:13 Yep. Not useful but very much awesome. And I wonder if that sleep command is even gonna work. It has. Right. Okay. It's quite happy. NGINX is probably based on up into anyway. So awesome. Cool. What do want to do then? Do you want to pull in the Semver library and see how that works? Do you want to pause it here and resume another day? How are you feeling? Alright. So we've got ten more minutes till the top of the hour and let's see how quickly we can get the December up and running. Alright. Take a look. So

1:49:39 Adding Semantic Versioning Logic

1:49:47 there is if you pull up the Chrome window, sorry, browser window, and we see the library I mentioned earlier is called mastermind slash semver. Let's give that a shot. So, yeah, this is a library that allows us to essentially process semantic versions. And what we're gonna do is we're gonna use that and call out to I imagine we want to call out to Docker Hub to see what tags are available and see if we can do some sort of reconciling with the two. So let's try and get that running as quickly as possible. Let's cheat,

1:50:05 Introducing the Semver Library

1:50:35 right? Because we don't need to reach the docker. I mean we can definitely do that but I don't think are we gonna be able to make that in ten minutes? I I don't think so. Fair enough. I mean are you feeling brave again? You're the one doing most of the coding here. I was just gonna suggest that we just provide an arbitrary list of versions and have it work against it but I will I'll let you go. Yeah. Alright. Let let let's go with that. Let's go with that. Let let's make it easy for ourselves. I

1:51:01 think we've had enough difficulties as it is. So yeah. Well I think for me the important bit is seeing how this amount of version thing will work rather than you know, an HTTP request at Docker Hub. I'm happy to you know, we can do that offline or whatever But Yeah. See. Let's just make a go for it. Let's see what happens. If I just pull up the NGINX versions on Docker Hub, a few examples in there are actually valid images that are going to be pulled. So we have one dot 15 onwards. One dot 15

1:51:17 Implementing Semver Resolution Logic

1:51:39 one dot nineteen dot six. Yeah. One dot nineteen dot six. One 18 zero. And let's get one more just to give us a bit of wiggle room. Let's go back a bit. One seventeen four. Hey, we've got plenty of options here. So like we can set our constraints now. Yeah. Exactly. Cool, amazing. So what we're going to do is that should be good enough. So what we're going to do is we're going to take the image tag that's come in, do some string manipulation to get the constraints there, and then use that to figure out what version is best.

1:52:34 So a little bit messy, but we're going to do all of that in this particular for loop. Now what we want to do is, let's see, import the library first. That'll be quite good. Yeah. And I'll prepare an example. So let's assume we can satisfy I guess we wanna Do you have a terminal on your end? My terminal seems to have Alright. It's because it's running the logs. I see. Could you go get the Semva package? Cool. Amazing. So what we're gonna do is we're gonna take our image name, which is going to be container.

1:53:55 Yeah. Container image. I'm going spit it on the colon. And we're going to want two parts. And the image name is going to be on the left. It's a bit on zero. And the version constraint is going to be on the right. And what we're going to do is we're going to I've copied an example actually from there. We'll use what they can already parse, so we don't need to. Amazing. So that's going to be version constraint. So this new constraint is going to give us a auto complete is still loading. New constraint returns a constraints instance that a

1:54:51 version instance can be checked against. So essentially, what we're doing is we're passing in the range to the new constraint. And then we will iterate through the list of our versions and see which ones satisfy. Yeah, That's done. So what we want to do is for version in version range versions. More errors for now. Don't do this in production, folks. Test version as a string. What we're gonna do is Do we need to look the versions? Does that not take a list and return the best one? Does that know how the library works? I don't think it provides a function for

1:55:45 that version. No. It wants you to iterate through them. Alright. Okay. So what we're going to do is send for new version, which is our version. And what we're going to do is we're going to say, if v actually, if hook constraint matches, I think, is the name of the of the check. Oh, there we are. Yes, check. So what this is saying is, does our version in our version list match our constraint? V. Then we're going to say best version equals version. So far and then over here, instead of using NGINX latest Spring. Spring.

1:57:02 F. I nginx spring best version. Cool. I think this should be what is required. So what we're doing here is we're taking the image name from Kubernetes as it's passed in, splitting at the semi colon I'm sorry, at standard colon, initializing our constraint from there. Yeah. Thank you for that. We're iterating through our version list, checking which ones match the constraint. And the one which is the most latest, assuming that this is an ordered list that matches, will be put into the best version string variable. And then we're now interpolating that into our value. So let's

1:57:56 give this a shot. Do you think we should very quickly just not do anything if we get an error? Yeah. We can probably just send an error out. So yeah, if you add one there. I'm just thinking like if we wanna confirm that this works. I mean, we're about to redeploy this controller which is already gonna break it potentially. No. Because we have the allowed failure. Okay. Maybe maybe we'll be okay. Oops. Return. Alright. Let's build this. Let's do it. Yeah. We're gonna happy? Yeah. I think so. I just changed the error to output it to standard out. I'm

1:58:49 sorry, output it to response. Okay. So we're gonna build this, redeploy it and then in theory we're gonna try and add our constraints to our debug container which should be resolved for us. All with thirty seconds left. If this works first time around then that's it. It's like mic drop off. Yeah. That's it. Just walk away with like job done. Mhmm. Yeah. Very impressive. Okay. Delete pod semantic. So now we have a new version of our mutating admission controller running hopefully. Yep. So let's, the reason I wanted this here is what I was going to suggest is we have

1:59:08 Testing and Debugging Semver Resolution

1:59:44 this debug thing being deployed. Right? Let's take a look at that, which right now is just using this. There's no colon. So I just wanted it to, you know, it can't powers it just to continue. So now we can add 20.04 and that should still fill which means it should just go untouched I would imagine. Do you agree with that? Hopefully. Possibly. So we can describe our pod which is this new one here and it should be untouched. Correct. Awesome. Yes. Correct. Now we want to apply our magic syntax where we say NGINX and we're gonna say greater than 1.16

2:00:30 less than or equal to 1.17. Which should require I think I need a comma there. No. We don't have a one sixteen. Okay. I need to go up to one eighteen. What should give us one seventeen four. So Yep. You're gonna need a comma in your yeah. Between so more than equal to one sixteen comma less than equal to one eighteen. That wasn't an example. Yeah. You can just do Oh, actually, yeah. Maybe you're right. Okay. That's right. That should work. Right. I've never been so nervous. Container creating. Alright. Okay. Let's see what happened. Okay. So we didn't parse this.

2:01:23 Could you try with the with the comma? So you want the comma here? Yeah. And also could you remove the space after the after the colon? You happy with that? Yeah. Let's let's try that. Okay. I think we're gonna need to quickly pull out some logs. Yeah. Nothing to replace. Let's try semantic. I don't think we have much going on there yet, do we? Okay. You got five minutes to debug? Will we go for it? Yes. So is our semantic program running properly? As in, is mutating webhook running properly? Because that looks wrong to me

2:02:29 when you put in the logs of the semantic. Wonder if it's running engine neck. Yeah. Describe. That would be terrible. Yep. Oops. Okay. So we should be able to delete that now. Yeah. The latest version, no. So I actually did so this will work but for a reason that is that's quite interesting. The webhook will fail because, you know, NGINX is providing garbage. And what it will do is it will just fall back and ignore the error, whereas previously the semantic hook was correctly functioning. So if you describe the semantic pod now, it will it should be running with the correct thing.

2:03:26 I think that's pulling a new NGINX. I think it's worked. Can you describe it? It has. Oh, yes. And it resolved. Success. Amazing. That Siri celebrating? That is Siri celebrating with us. Alright. That was awesome. So you know, let's talk about what the next steps are. Let's finish up. Thank you very much for pretty much writing this entire thing and that was really cool to see that from start to finish. So let's assume I really want this to be a thing, right? So we wanna take this code, we wanna productionize obviously we should add tests and how do we

2:03:55 Next Steps & Conclusion

2:04:15 ensure that we don't run into that problem we have with the semantic controller. I guess we have to build checks and constraints into this loop here, right? Yeah, exactly. So the within the mutating webhook in Kubernetes itself, you only have namespace level constraints being added. So saying, don't call this webhook if the namespace matches these particular labels. So for example, we could add constraints that say, don't run this for cube system. And we could put the semantic program itself in its own namespace, and also say, don't run this if in these particular namespaces, or have these labels,

2:04:57 or whatever. We will probably want to add checks within this program itself to state that if the version doesn't look like a constraint range, then just continue onwards. Like, be graceful in that respect. And yeah, like adding tests and stuff like that. The good thing about this is that this is a standard Kubernetes API program. So there's no reason why you can't generate some mocks and interject them as like a HTTP, using HTTP mocking library in Go, yeah, you just pass these objects across. It's just JSON at the end of the day. You don't need any sort of Kubernetes

2:05:39 framework or anything like that to test this, which is great. Nice. Because we are operating on a pod level as well, we'd probably want to cache those resolutions I guess and memory or even on disk somewhere because as the pods are moved around or killed or whatever, we don't want it to resolve new versions for some pods and older versions for other pods. We still got a few things here that just make this a bit more feasible but excited. Yeah. Very, very excited. It it's also interesting because we may want to want this to run on a replica

2:06:13 set level instead of a pod level. So imagine that you are rolling out a a version of your system, even if it is a minor version, you probably don't want an upgrade to be happening. Like just because a node died, for example, we might want to say, Okay, when you're ready to roll out a new replica set and you do the kubectl rollout restart, that's the time it decides, or maybe I should consider if a new version is available that satisfies the constraints. Because then at least it's human controllable. At a pod level, it's probably too granular.

2:06:50 Yeah. Awesome. Well, I will push this code up just as reference for anyone that has been watching and playing along with us. I will speak to you afterwards. We'll work out how to clean this up, make a few tweaks hopefully publish it for people to consume and test out. Happy to see my world idea actually working. Thank you so much for just being able to explain everything that we were doing as we go so well and for churning out all of those code debugging certificate nonsense everything there. I think that was really valuable for me.

2:07:09 Farewell

2:07:21 I'm sure it was very valuable for others. Thank you very much for that. Yeah, thank you everyone. No, this was was really really fun. Yeah and I hope this is useful. It's a fantastic idea and I really hope that we can actually turn this into a proper plug in that people will use. Yeah definitely. Watch the space is coming soon definitely. Yeah coming soon. Coming soon to a Kubernetes cluster near you. Alright have a great day again. Thank you for allocating that extra time. I know we went over and I'll speak to you again soon and goodbye. Thanks. Yeah. Bye everyone.