1:00 Introductions

1:00 Hello, and welcome to today's episode of Rawkode Live. Before we get started, I wanna take a little bit of time just to say thank you to Equinix Medal. Equinix Medal is my employer and they donate the time and resources that I need to put the show together. So thank you very much Equinix Medal. Equinix Medal is a bare metal cloud and we are offering you $50 in credit so you can try it out yourself. The code is Rawkode dash live. This will allow you to spin up our smallest instance for up to a hundred hours,

1:30 but of course it's much more fun to play with the larger instances and take advantage of those NVMe disks up to 400 gig of RAM and crazy CPUs. So use the money wisely or unwisely depending on how you're feeling. If you wanna chat about anything that we talk about in today's episode or any other episodes, do have a discord server. Feel free to join us and you know, keep the conversation going. Very useful if you're not watching live. Please remember to subscribe to the channel and thumbs up the video if you can and you enjoy it. This just helps other people

2:00 discover the content too. Today we are doing part two of the Cilium episode. Last time we had a few problems which we've now correctly fixed and we're gonna explore Cilium and see how it can make our lives better. And for that, I am joined once again by Ilya. Hello, Ilya. Hello, David. How are doing? I'm very well. Thank you. I'm excited. I wanna, you know, properly kick the tires on Cilium this time and and see what I can do for my my Kubernetes c and I. Yeah. I'm I'm glad that I was working now. It's just at

2:02 Welcome and Episode Context (Cilium Part II)

2:36 the end of the day, we have the extra parameters and and and and it's working. The thing about it was that we are basically bringing it up in a new environment that hadn't been tested before. Right? Exactly. Yeah. I mean Or even Bare mail is not as simple as as other environments, you know. There's a lot of extra bit of tweaking that needs to be needs to be done, we've got that working. I think it would be really cool if we just start and talk about the the news that I've seen dropping on Twitter yesterday.

3:00 Cilium Funding and Enterprise Announcement

3:05 Oh, yeah. Of course. So let me do that. Oh, no. Yeah. That one. Yes. Very cool. Yeah. Ice Cilium has raised 29,000,000 in funding from various investors. This is series a, and we are launching our enterprise offering as well. If you if you could could show the the website in the next tab there. Yeah. So if you go to icll.com, you can find out about Cilium Enterprise and how it differs from Cilium open source. You can also see some of our customers and read about the team and, you know, find out the the openings that we might have,

3:50 and we do have a few. Yeah. We cannot time these two episodes as turned up as well, I guess. And the the last time was just after Google made their announcement about, you know, Cilium on on JKE. And then randomly, yesterday, the news dropped up at the CDC and I was like, wow. Really? A good timing. Yeah. Okay. Yeah. So, yeah, Cilium Enterprise, you know I mean, we're we're gonna be playing with open source today, but maybe you wanna just give a quick touch a bit about it. Mostly. Cilium Enterprise offers additional features for for enterprise security teams,

4:26 We're effectively enabling flow logging to systems like Splunk and and a few other feature features that are summarized on the on the product page. So if you if you click on the top navigation product so just at the bottom there, there's a there's a there's a long summary of of the enterprise features. It just keeps scrolling. So, like, for example, here so in in terms of core functionality, it's all the same. Then as you get into to our functionality, Cilium Enterprise has additional support for for for third party BGP integrations. And in some of the other sections, you

5:16 will see that there are exclusive enterprise features. Yeah. Nice. Yeah. No problem. Yeah. Alright. So today, let's cover where we are. So I'd I like to do very little upfront on the show. But we do need a Kubernetes cluster. And assuming nothing is broken since I spun this up. Good. We are currently in a position where I've used cluster API. We only have one control plane node and all the worker nodes are currently not ready. That's because they won't be considered ready until we deploy Cilium and there's networking in our cluster. One thing I did different from last time

5:28 Cluster Setup and Cilium Deployment

6:00 is that I think this may be useful to other people, so this code now is all available on getlab.com. Really needs to stop the tab suspenders, shouldn't I? Getlab.com/rawcode/equinox-metal-examples. There's a Cilium directory with the cluster API code and the Cilium configuration, so people can check that out and run it themselves. I'm just using helm template, which is spitting out the Cilium.YAML here. So step one for us, we're just gonna get this applied to the cluster. We're gonna run the connectivity checks. This is stuff that we did last time. Only this time it's gonna work. He says confidently.

6:30 Installing Cilium / Connectivity Tests

6:36 So let's get Cilium on this cluster. One, two, three. Oh, it helps if I create the namespace. I think that would help. Alright. There we go. So let's just keep an eye on that for now. We have a little watch. Did you say you could also use resource sets for this or what time do we try You definitely can use cluster resource sets. I just didn't use them here because it would automatically deploy it to our cluster and I wanted to do the apply it live. I should use the right namespace. Sure. Yeah. You're supposed to be keeping me

7:18 right here. Come on. Right. There you go. Okay. That's that's happening. Yeah. That's happening. So we cut our use cluster resource sets, and I will actually put an example of that up Mhmm. Tomorrow because Jason and I are taking over the CNCF Twitch channel doing Oh, cool. API stuff. At least for an hour anyway. So the quotes from that will be also in the repository. Now it looks like it's happy. Oh, great. Yeah. So now we can apply. I I added adjust target for this. That is just Well deploying. Yeah. This So that your adjust file and, like I

8:07 mean, does it support a subset of makes insights? Yeah. So I I I I I generally try and use make files whenever possible. But there are some really cool semantics adjust that just you can't really do. Like, you know, specifying and laying bash scripts is really cool. Which means like like an interpreter because when I tried to do this with raw make, read an echo uses the built ends of bash. And I know I could have changed the shell and make but then it's Yeah. Sometimes it's just nicer to use just file. So Yeah. It's a cool tool.

8:40 Verifying Cilium Installation

8:40 Okay. So we wanna confirm that those tests are happy. So no restarts all running. This is what we expect. Right? That's that's me that's to you as a healthy running Cilium install. Yeah. Looks great. Yeah. Yeah. All of them are happy as in ready. Yeah. There you go. Nine minutes and then we've already made a lot more progress. Nice. Okay. Let's just delete all those resources. I didn't add a just target for that. I really should've. Yeah. And they're gone. Okay. So let's pop over to the docs. Now, the last time we did this episode,

9:26 Star Wars Demo Application Setup

9:36 we started to talk about the Star Wars example. Is that where you wanna start today? Or Yeah. Yeah. That means I'm gonna have to remember where that lived. Yeah. I think I think it was policy. Oh, maybe. Yeah. Yeah. That's that's right. It should be there. Yeah. No. I think I was wrong. Let's try. Yeah. Getting started. And then I'll find it. I'll remember. There we go. It was this, wasn't it? Yeah. K. Deploy the demo application. So this is a Star Wars inspired example. It has three microservices, a duster, a tie fighter, and the x

10:00 Cilium endpoints

10:24 wing. Mhmm. Okay. So if I just copy the first lane and then we can talk about what that's actually deployed. There we go. And before I go deploying things to my local cluster I should just make sure that I don't accidentally do that. Okay. So off it goes. Okay. I like it when things work. So I just want me to pull out the pause and the s v c. So we'll just make sure that all that look good. We got a desktop service. Excellent. And this one's means list. Yeah. I don't understand why why it's asking

11:22 Cilium Endpoints and Identity

11:24 me to do this. Oh, we're trying to confirm the endpoints. Right? Oh, yeah. That's right. And we're using Cilium. And I'm using a different namespace. So Yep. So that list my Ciliums, and then we'll just do the endpoint list. Yeah. Oh, I need to pick a pod. Okay. Mhmm. Okay. Alright. Do you wanna walk us through that? Because I have no idea what that just did. Yeah. Of course. So this shows us no. Hold on a sec. So this is for for the Star Wars demo. Yeah. So it it just shows us the kind of underlying representation of

12:21 of the these demo pods in inside of Cilium. There there's also a CRD that to represent the same. But, essentially, what what it is about is that Cilium allocates a an ID of its own for for these apps. Yeah. I'm trying to just get the same information if you have one. Keep it all get CEP. Okay. Is it namespaced or is it cluster scoped? Would be namespaced. Alright. So we'll get CEP. Right. So this this is this is effectively the same. What do you call it? The the the same objects that represented at CIG. When we talk about a Cilium

13:15 endpoint, we're just talking about some network target, some pod, and Yeah. Network cluster. Yeah. Okay. Yeah. And we can see here that we've got our two desktop pods that we deployed. We've got Mhmm. They've got identity. They've got IPV four addresses. They've got some sort of state. I'm not sure what that means. Like Yeah. So this just means that Cilium is aware that these pods have an IP address and our endpoints that could be requested. That's right. Yeah. Alright. Yeah. Okay. Yeah. Now, Well, let's see what the tutorial tells us to do next, and then I can throw

13:51 questions. Yeah. The the tutorial just highlights that Cilium allocates an identity, which which can be seen in that identity column. Okay. And just because I see it on that list there, like, we've got enforcement disabled. So I guess this is like, there are no network policies being applied to any of these endpoints at the moment. That's right. Right. Okay. Cool. Alright. So now it's telling us that we wanna check for access. Mhmm. So alright. So assuming this wants us to have the x wing service blow up the desk there. Yeah. Mhmm. Yeah. Yeah. That's a good example. Okay. So let's

14:16 Testing Default Network Access

14:33 grab a copy first lane. There we go. Eventually, I'll stop doing that. So we're gonna exec into the x wing. We issue a curl request. Yep. And it's sending it to the desktop service. And we're just requesting landing and we get ship landed. Okay. Making sense so far. Next, it wants us okay. So we oh, yeah. Now the tie fader wants to request the landing. So let's just Okay. Got it. Cool. So we wanna try and protect the death star and we're gonna do that through applying policies. Mhmm. Alright. I'll copy and paste this, and then

15:13 Applying L4 Network Policy

15:23 maybe we'll just quickly write No. The the command down below. I think that it just that that would do the same. I'm gonna do that so many times. Okay. Yeah. So this is our network policy. Let's get some syntax highlighting going on here. Yeah. And we have some rule. And this is an LCD l four policy to restrict the desk or access to empire ships only. Okay. Mhmm. So when we're applying this rule, describing this rule, we need to have an endpoint select or and we're just staying whenever the organization is Empire and the class is death star that we

16:07 are gonna allow ingress from Empire organization ships and we can even restrict the ports that we allow them to access. Okay. Mhmm. Does it just want me to apply this? Yeah. It does. Yeah. Yeah. It does. Okay. Cool. So apply network policy. Mhmm. Done. Easy. So that means that the tie fader should still be able to request landing and our x wing should know well, our x wing should fail. So let's try a tie fader first. Alright. We can land that ship all day. And our x wing. Nope. Awesome. It's just gonna time out. Right?

16:37 Testing L4 Policy Enforcement

16:57 Yeah. We don't need to wait for that to time out. Nice. Okay. So that was pretty painless. Yeah. So yeah. And this is this is so what we've done is created a Cilium network policy object that specifies these rules using Cilium's own policy syntax, if you like. The we Cilium also supports the traditional Kubernetes network policies, which which have less features to them. Right? So some some of the things that that we will probably see soon, if you keep scrolling through this tutorial, we I believe we should have an l seven policy down below, which is

17:10 L3/L4 vs L7 Policies (Envoy Integration)

17:52 something that you can't do with with the the traditional network policies and Kubernetes. So let's just go step by step. Okay. So we apply the LVL for policy. Alright. So No. You you're on the endpoint list again. Yeah. You could you could probably use capital c at c e p. Oh, this one. So we can see that policy is being enforced. Right? Did I run the wrong command? Here. Enable. Enable. This should be enabled for something. The the output is a bit funky here, to be honest. Yeah. Let me I mean, I can make it small. I think I think you know

18:47 what it is, David. I think because, actually, this Cilium endpoint list will only will only show the endpoints for the given node because this tutorial was assuming mini cube. Right. Or The number worked. But, yeah, if you do get CEP, that that will show you, a better version. So now, endpoint state. The policy enforcement is actually not shown here. Right? Or Yeah. No. I I don't see it. So I'm gonna find the worker with the desk stars on it. That's that's where the policies will be. Right? So Yeah. That's right. That's where the the endpoint will be for that.

19:42 Which means we wanna filter the namespace on Cilium. And I just wanna see what the one is with the four g s m l. And run the endpoint list again against Mhmm. That Cilium, which is VWO. Yeah. That's right. So There we go. So we have enforcement enabled here. Yeah. That's right. So can can you do Cilium one point list dash dash out? Completely web host. Okay. Right. Yeah. So this this is this is just local to the host. I'll have to have a look into why why the CRD doesn't present this. Maybe maybe it is actually shown

20:32 yeah. Actually, if you do scroll back up, you know, where the get c e p up was. Okay? Mhmm. So it's it's for the desktop that we that's that's the the the desktop that that we wanna see. Right? So if you just do a if you put a get on one of those desk desk star endpoints. Let me send this back in. It's just straight on one of those. Okay. So Quick picture. Maybe you could do a describing this. Yeah. Actually, describe doesn't show that stuff. So Okay. So describe test star. Let's pick the first one.

21:35 Alright. So we can avoid the managed fields. Yeah. I don't see anything about enforcement here. Okay. So I got something for me to take a look into. Because in the We test it. We've seen it working. We can see it working in there. If I take on this, we see it on the node one. Right? So the enforcement is enabled. So Mhmm. Cool. So let's carry on. In fact, it's telling me to do get CMP. Take a look at that. We have oh, that's just our rule one. Okay. That's right. That's the one we've created. And then it

22:17 wants me to describe it. So as everything that we do with Cilium I mean, does it always expose these custom resources? That's just the way it works. Right? Everything is not me. Yeah. That's right. Yeah. Yeah. Okay. Yeah. As I said, those are there is support for the traditional Kubernetes network policies, which are inbuilt, but the the these Cilium customers also are are obviously encouraged. They have more features. Nice. So can we what's the difference between an l three and an l seven policy then? Oh, yeah. So well, l three and l four policies are implemented

22:50 Layer 7 network policies

23:03 in on the effectively enforced to the kernel level, while l seven policies are implemented with Envoy and custom filters that we have for Envoy. So, effectively, when you do define an l seven policies, Cilium will speed up an instance of Envoy that that will implement that policy, and and we will be attached to to the so that that that'd be like a you know? And and we would be transparently attached to the to the pod in effect. Okay. Yeah. We we can dig into that if you like. But maybe we could define the policy part. So this this demonstrates how

23:54 that policy would work. Okay. So let's run this now. This is another curl command from a typhi error, and Boom. We blew up the death star. Okay. So we don't want that to happen. Right. So we want to define an l seven policy. So this is like, l seven just means that it's pro protocol aware, so it actually understands HTTP. Right? Okay. That's right. Yeah. Yeah. Exactly. So this is this is just an amendment to the the previous policy, I believe. Alright. It's it's using the same name. I see. We have to to have a new object.

24:11 Applying L7 Network Policy (HTTP)

24:38 Okay. So we've got the same selectors only this time. Yeah. So we can now add this and say, hey, we don't Try. Yeah. Is this allowing? So, yeah, ingress Uh-huh. Supports Yeah. Cool. Rules. Okay. Yeah. Let's reapply. And then back to this. Okay. And now if we do request landed. That'll work. And if we try the exhaust port, we get access denied. Yeah. So yeah. I mean, I don't know. Do you wanna dig into how it works? I wanna take a look at the annual process. I mean, confirm that there is one now or something like that.

25:26 Examining L7 Implementation (Envoy)

25:32 I I well, I I don't understand if if I'm being really silly. Yeah. Oh, oh, no. Right. Okay. This is the request line. Okay. So we're only explicitly allowing access to the request landing. It's not that we're blocking access to the exhaust port. We're only allowed in a sec. Okay. Would that be possible? Can we say allow everything except for this endpoint? I wonder actually. So yeah. I think you could use regexes actually. Yeah. So you could you could have regex or the method or regex for the path. Okay. Yeah. Okay. So do you wanna dig into this

26:14 a little bit then and and and Sure. Why not? Yeah. So what's so where is this end volume? Yeah. So oh, yeah. Let's let's let's have a quick look at that. So if you if you go back into that pub where you were Cilium pub where you ran Cilium endpoint list earlier. You want me to go and save the pod? Yeah. Or you just you just try to replace Cilium endpoint list with p s. Can you say that again? P s p s minus a f or something like that. P s. Right. Okay. No. I'm not saying the Cilium. Right.

26:54 Oh. Yeah. No. Okay. Maybe it's not a f. Well, just l s proc, for example. Okay. Right. Yeah. No. You can see it now. So if you if you l s proc slash six six eight. And take a look at argument, for example. Yeah. Well, it's command line. Isn't it? Yeah. Exactly. So so you can see that, you know, this Cilium envoy binary that that has the these arguments. There is a file in the bar run. Cilium bootstrap b b, and and and it's logging somewhere. We'll be logging back to back to the Cilium log, I guess.

28:03 Yeah. But I it what what you can see is that there is now an annual pro process inside the the Cilium part that is Okay. So when I deploy this policy. So they enter my cluster. Right? We have an agent on every single node. Yeah. And that agent is running the Cilium agent, Cilium health. That's I think that's been cut off. I'm not sure what health part is, but and an envoy or envoy. I'm gonna change the way I say that every 10 minutes, I think. And it runs an an envoy as well on all the nodes. So the agent

28:34 is doing like what the health checks and the rules and the enforcement at the l three and four level, but all the l seven stuff has been proxy by Envoy and that's being configured. By the agent. Okay. Nice. Okay. Yeah. That makes sense. I got it like very cool. Mhmm. And I guess I mean, that's is that just a vanilla envoy? I can use all of the different features of envoy for the the proxy here, like No. Well, I mean, you you could you could do something there. I can't I I I'm I'm actually not sure myself.

29:10 But it is definitely not a vanilla version of Envoy as as as we build it ourselves with some custom filters because, yeah, we we have to have some c plus plus based filters that that have to be built and built on. But other than that, that's not like a fork or anything. Right? We build an upstream plus all our filters too. Well, it turns out I think the next part of the tutorial was actually my thinking anyway when I was like, well, I I don't wanna explicitly allow a single path. I really just wanna block

29:44 the exhaust port. And I think this is Yeah. This is kinda what it's saying here. It's like, hey, we can match the exact URL or we can use some rejects. Yeah. We've also already described the network policy. We've kinda seen that. Okay. So let's try this. So we can also do all of this with the Cilium CLI as well. So Yeah. This is saying I can run Cilium policy get. Cool. Right. And that's just the same route. Perfect. Yeah. Exactly. Yeah. Alright. What what do you think of next? Alright. Now it wants oh, Cilium Monitor?

30:27 Introduction to Cilium Monitor (CLI)

30:28 Yeah. But perhaps yeah. Well, let's let's have a look at Cilium Manager first. Yeah. Well, I'm gonna have to And if you know, I'm not gonna reference to it. Yep. So let's try it. Those port. Oh, yeah. Yeah. We can actually see that being done now. Okay. So the monitor just gives me kind of visibility into how the network policies are enforced. Yeah. By the request landing. Yep. That was allowed. Oh, no. That was denied. Mhmm. Did I get the thing? Request landing post. Oh, no. Because it's it has to be a post, Alex. Okay. Yeah. That's

31:24 fine. Just keeping you on your toes. There we go. Shipland, dude. Okay. And we get a Yeah. Yeah. Okay. Cool. That's really cool. I like that. Yeah. So if you now have a look at how the UI, we should actually see some of the same information there. Okay. So when It's in the monitor was the like, the the the pretty sensitive to Hubble if you like. Okay. So we have two parts deployed here, Hubble Relay and Hubble UI. These were enabled in the helm Mhmm. Just global Hubble really blah blah blah. And that's deployed a UI.

31:40 Hubble UI

32:06 So if I do service port forward ICC Hubble UI and that's just port 80. Mhmm. And that gives us Yeah. So if you try and select the namespace where those whether the desk star and friends are. Yeah. You should be able to observe, for example like, try and make a request from the from the X Wing to to the you know, try and try and destroy the desktop from the X Wing. You should see those policy denials in in the log below. Okay. Let's do that then. There we go. Alright. So that's unlanded. So let's

32:18 Accessing and Debugging Hubble UI

33:05 try and hit that exhaust port again. Now it's done now. And from from as well? Should we be seeing something here? I think not for h because that that hasn't been enabled. There are additional options that we need to do for that. But if you do this Oh, okay. Alright. So the Hubble UI. Yeah. So now now these buckets are being dropped. We should see these here. Try and update. We're not seeing it. Okay. How about, how about we take a look at the policies first anyway? We should see our policy here in the UI.

34:00 So this this is the the policy that we've specified. Yep. And if you go back to flows it's not showing it here. So they should be showing Yeah. Traffic that it's blocked. Yeah. That's that's that's some well, did you Have I forgot to enable something? Maybe. I don't know. Did we did we debug this last time, didn't we, offline? We did, didn't we? Because the last time I deployed this, but I think I was missing really last time, whereas we definitely Okay. Really Have a look at the relay log, for example. Yeah. Sure. Hubble. Really?

34:59 Okay. All complete. Not planned. Yeah. Oh, because I can't spell selling. Yeah. Logs f Hubbell Relay. There we go. Yeah. Okay. So, yeah, it looks like this is still not connecting back to the back to to the Huddl port. Yeah. I think that was a yeah. Remember we had to enable that port for 22444244? Yes. I'll I'll I'll find this like message that sent you. Yeah. Yeah. Yeah. That's ringing a bell actually. There was some phone with that port. Okay. Yeah. I did. Well, I think you could do what could you can do is edit the Cilium config config

35:54 map now. Yeah. Let's do it. ConfigMap Cilium. Oh, no. Keep doing that. No. ConfigMap Cilium. K. Have a look for Hubble. Enable Hubble. And what else is there? Hubble listen address. That one. Ah, yeah. It's all coming back to me. Yeah. Yeah. Right. So column four two four four, I believe. Right? Yeah. Yeah. Okay. Alright. So if I just kick Huddl now. Right? So then Should I just kick everything? Oh, just the the the the Cilium agent pods. Just the l kits up. Okay. So the pods kits equals. Alright. So I You can remember to add something here. So

37:06 expose list and address. I'll I'll do that before I push this code up. And we run get paused because Cilium happy again. And if I have logs on Hubble. Might take a little moment. K. Let's see if we can get exhaust port. I'll need to restart Hubble shortly. Right? No. I think it's retrying. Right? So if you look at how we're gonna log again, you see now it's now it's connected. No. Yeah. So I think now it's now it's actually trying to do something. I think I think it's doing something that looks just about right.

37:51 Visualizing Network Flows in Hubble UI

37:51 If you reload this once again Oh. Yeah. There you go. So you can see the drop bucket over there. Yeah. Okay. So Cool. Alright. Okay. Yeah. Because I was worried that I missed something. I'm always worried, Stora, but Surely this surely this wasn't an enterprise feature. Awesome. So Hubble Yeah. Okay. There's a there's quite a lot going on here. I just wanna make sure I understand this. So Hubble is a UI. Right away, we can see that spaceship speak to the death star on Port 80 HTTP. The death star has no egress. The spaceship I'm assuming that's just because I used one

38:15 Exploring Hubble UI

38:36 of the pods. Yeah. This is the x one class spaceship whereas we also have a class type of spaceship. I'm just gonna run that as well. Let's do exhaust port will be denied but we do have request landing. Should I just refresh this? Maybe that rebuild the oh, there we go. Nice. Very cool. So I can kind of a nice high level to really visualize the traffic here. Yeah. I I love that it's even just showing me this, you know, the actual endpoint that was last hit. We've got a 200. Really cool tool for debugging our network. I

39:14 can see, like, I mean, all the times we have problems in my cluster. Yeah. So and the reason you see the HTTP, as I mentioned, to to to enable full visibility of HTTP, you need to to to add a few extra options, which are not enabled by default. But you can see when when because because we have the the Envoy proxy running, we we can actually in Cilium intercepts the those h t p sessions already. So so we can see the the status codes here. Yeah. I'm assuming there must be other traffic going on here too. Right?

39:52 Possibly. Does it monitor itself? Yeah. It does. Perfect. Yeah. Of course. Yeah. Yeah. Mhmm. Alright. Really cool tool. And I'm glad it's not enterprise only, so I can use it. Right. Yeah. Yeah. So in in the enterprise version, you you have additional features around export and additional filters and various other things. We can do do a session on that later on if you like as well. Yeah. Definitely. So I think yeah. That's the end of the tie fair thing. What what other features does Cilium bring into the to the table that we could maybe explore?

40:36 Is there anything else we should cover here? Well, I guess one thing that that we could cover later, but that will require changes in config as proxy replacement. So Cilium has a more performed load balancer that that you can use instead of Q proxy. So it's implemented in the VPF course, and it doesn't use IP tables, which which have scalability and moments issued. Yes. I have definitely run into those problems in the past in a former life. So what's so that means that when I spend on my Kubernetes cluster, I can say, okay. Don't deploy Cube proxy.

40:41 Kube-proxy Replacement with Cilium

41:19 And then what what does Cilium do then? How do I? So yeah. So, I mean, there there are a few ways to do it. You can you can deploy it into an existing cluster and and and this so you can you can now basically enable a few options in Cilium and delete cube proxy. Right? You can do that perhaps if you if you if you feel brave today. The the other option is, of course, to to to to not even deploy a queue proxy in the first place. But I'm not actually sure whether in cluster

41:57 API you have an option for that. And, firstly, your cluster API doesn't allow us to integrate with the skip phase section yet. I know that there's some ongoing work to make that happen, but not yet. But we can always just, you know, nuke the cube proxy. I mean, it's just deployed as a static manifest anyway, I think. So we can always That's right. Yeah. So the there is an yeah. There is a guide for this that you'll In fact, yeah. Yeah. It's just like here, delete that's demon set. Yeah. And in the in the one nine release, we have a few

42:28 new cool features. Actually, if you wanna pull up the blog, Cilium.io blog, there's the one nine blog post. Today, we used one eight as one nine came out just yesterday, and we haven't updated all the all the the flags, you know, configs there, David. But the one nine comes with Maglev support, which which enables consistent hashing inside the load balance effectively. So you could hit the same part if you like. Okay. Cool. Wow. Quite a lot of changes there. Okay. So just, like, for sticky sessions and all that. Right? Alright. So let's let's try and replace a cube proxy

43:10 DNS network policies

43:23 maybe in ten minutes. That because I'm my concern is I'm gonna get just kill my cluster, and then we're done. So Yeah. Yeah. Let's try. What about I've seen something else in here. Right? So there was something about DNS. Yeah. So you could get DNS visibility as well. We can try this one, definitely. So this allows us to document a lockdown external access with DNS based policies. So the DNS based policies policies. Right. Cool. Yes. There there are few there are few things. One one option is that to to sort of get DNS visibility so you can see

43:38 Applying FQDN Network Policy (DNS)

44:08 what is going on in in cluster DNS, but you could also walk down access to some some domains, for example. Okay. That that's that's something to try, definitely. And I don't need to enable anything. Right? This is gonna work with the another install. Okay. So I am going and I just you'll load this. I'm just gonna copy it. Feeling pretty brave now. Things are working. Fine. Oh, no. Okay. Good. Yeah. Yeah. You you missed API version. Let me just let's just append that in here. Sure. Yeah. So this is a FQDN network policy, very similar to this network policy

44:50 of it. Yeah. You're now sitting API version. Yep. Have a look at the page where you copied this from. This this is work for v one. It's also v two. Okay. Great. Now oh, so it's also using the same labels, which is cool. So this is still an org empire except now we've got some sort of media bot and this policy is going to allow traffic to API.twitter.com for QDNS. Okay. Yeah. Let's apply it. Still feeling brave. Yeah. Alright. Now what that means is I can run it's got CMP. I can describe FQDN. And in theory,

45:47 Hubble should now be aware of our new policy. It is. Yeah. Okay. Now let's see. So this Yeah. Let's go back to the document for a sec. See, I'm trying to make my own destiny path here, and you're trying to get me to follow the docs. So I'll listen to it. Well, that's what I do there. Is what I'm just I'm just trying to Alright. It makes sense. I don't something out again. What's the okay. That's applying the policy which we did. Okay. So now it wants me to use media bot and try and curl this

46:26 API. Did we deploy was media bot deployed automatically for us? I don't know if I remember seeing No. I didn't see the media bot anywhere. Yeah. I think we need to find it. Okay. So where was the apply for media bot? There we go. Okay. And I'll just change that version in case that's important. I guess, yeah, why mhmm. 8.5. We just want it. Alright. I want 8.5 works too. Of course, it's a tag, you know, a brand. So there we have the media bot, and we're gonna try and exec exec a curl request instead of it and see

46:50 Testing FQDN Policy and Debugging DNS Resolution

47:08 if we can okay. So it's got a few URLs here. Let's try. I'm gonna time out. So all of those are gonna turn out except for Except for API.rep. There you go. Sweet. Good boy. Okay. So I guess it this wasn't an l seven rule. This is not Envoy blocking this. Right? This is the pod is not able to resolve that host name. Is that right? Yeah. Yeah. I I actually I actually don't know that much about the implementation details of the PDM policy. Alright. Let's have some fun. Let's see. Do we have bash? We do. So do we

47:59 have dig? No. Do we have apt? No. Do I have APK? There we go. Add update DNS details. I can never remember the name of that package. I'll do yeah. I I remember that it didn't exist. Oh, so I've just Back in the day, you couldn't get dig on Alpine just easily. Well, right now, it can't even resolve this thing. Oh. So let's fix that first. This is fun. Alright. Yeah. Now the Alpine CDN to that. So we can just add a new match name. Let's allow it to do that. Drop it out. Yeah. And then let's see if that works.

48:44 Okay. Cool. David, I worry that in Alpine, you don't have this package. Alpine dig command. Think you'll find that oh, dig. Oh, oh, there's dig. Okay. Easy. You see? Unlike the other district. No. It it lied to me. Okay. Oh, Bains dash tools. Okay. Okay. I wish it was just like, there's just some tools I need to be consistent across distributions. Like I've been to read out alpha. Just want dig by DNS utils, it's standardized. Yeah. Alright. So that means Yeah. That resolved. Well, that wouldn't resolve anyway. That was just me being really silly. Well, that resolved too.

49:40 Okay. So actually, we Oh, wait. Because I am not using cluster DNS maybe. No. I am. Okay. Yeah. Oh, and I think but they they try and try and curl that. I think what it actually comes down to is that it blocks it a layer below. It just takes the DNS into account. Alright. Okay. Now do we have visibility into its hubble as well? Okay. Yeah. We can see this. I like that it shows you that this is outside of your cluster as well. Yeah. Nice. Okay. Yeah. And let's see. What do we see in here then for those requests?

50:11 Visualizing DNS Traffic in Hubble UI

50:26 At the top. Yeah. So, like, you can see that, like, for example, that one's been dropped. I'm not seeing the DNS look up at all though. Right? Is that expected? Right. I guess, when I'm just throwing a little stuff at you, like, explain this. Why doesn't this work? Like Yeah. So you might want to have a look at Yeah. That is only HTTP traffic, I think. Right? We're not seeing non HTTP traffic here. Well, no. It's it it's it's not that it's h two p. I think I think you should see the VPN here as well.

51:19 Maybe this is going could you could you look at, for example, the system namespace keep system namespace? Yeah. I think, actually, DNS visibility was a separate feature that that had to be enabled. It it's not important. I don't wanna Yeah. Yeah. I'm I'm just enjoying exploring the networks through this Hubble UI. Like, I'm sure there's other settings probably need to be Yeah. That's right. Effectively, if you if you use a PDM policy, the reinforcement is at the at the CP level. Okay. And then it's got an example where we actually combine these rules so we can

52:13 see. Yeah. You guys were gonna allow cool. And we okay. Nice. Okay. That makes sense. So what does this this Okay. So this looks like That's that's port 53 on TCP or UDP. So Right. I mean, there is something DNSC wise going on here, and this rule is. Yeah. So if if you have a look up above let's let's have a look a little bit above the the the document should explain. Alright. Okay. This is all about. Yeah. So what did we apply to the cluster then? Is this am I let's Is it yeah. Is that part of it? Yeah. Yeah.

53:08 Okay. It is already. Yeah. To end point. Oh, but we allow anything. Okay. So we're not actually restricting DNS here. Right. This is this is egress. Ah. Yeah. So you're just allowing allowing topic to give DNS. Okay. So we do allow all 53 DNS lookups as the egress that has been restricted to certain domains. Yeah. So I'm assuming if I change this pattern Oh, yes. Actually, we're only allowed to resolve Yeah. To star.twitter.com. And then once we apply You know, you have to dial find packages there. So I should be able to do Yeah. And

53:55 That's actually right because the star would upload. So that'd be that's really cool. Try dub dub dub or something that Yeah. So that should work, but Google should fail. Right? It doesn't resolve. There it And Alpine should fail too. Oh, yeah. I've I've not got much hope for that running anymore. Yeah. It's gone. It's gone. Okay. Let's have look at Hubble now. Let's try. No. I don't think it's there must be something we need to enable on the Hubble side, I'm assuming. Yeah. The Yeah. I think there's there's DNS visibility, future over So it does have this, like, the example

54:47 filter here does kinda suggest that I can filter on DNS. Yeah. That's we could you could do that for for the in the destination column, the the DNS names there. So Oh, there we go. Oh, there you go. Okay. So this is probably going that you That post names based traffic. Right? Yeah. And wanna show DNS lookups by default because you're probably gonna get a lot of that in a real cluster. I'm glad I clicked that button randomly. Alright. So let's see what we so here's the DNS lookup. It was dropped and we were trying to look up

55:29 something. Another one. Yeah. So I I don't know if we're gonna get the name of what it's trying to look up, but I'm sure that's something I'm doing. I've had no idea. There's there's forward dates. I'm gonna run it again. Hold on. Let's try that. What do we get here? Update. I don't know. Okay. Just give me a second. Definitely see DNS queries here and Alright. That's a really cool feature. I'm I'm really glad we persisted with that and got to see that working. That was nice. Great. You you found it. I didn't know it.

56:22 Yeah. Okay. So I I like that. DNS based policies. That's a really cool selling point as well. Right. You know, especially if we were trying to secure these clusters and stuff like that, you know, if they can't even resolve, like if we provide a wait list of things or a low list that we allow to be resolved in our cluster. That's pretty sweet. Okay. Alright. Are we fit that we've got as confidence because we're obviously just owning this now. We're gonna do the queue proxy replacement. I'm sure it is gonna work. Okay. I mean, there there's other stuff we

56:41 Implementing Kube-proxy Replacement

56:54 can do here. I I mean, there's a lot of really cool things like security you know, Cassandra and Elasticsearch. But I really just wanna start messing with this cube proxy now. Yeah. Okay. That's what it says now. Let's search for Kube proxy. Kubernetes without Kube proxy. Let's make this work. So we are not gonna kill Kube proxy daemon set yet. Let's see if we can No. Let's see what we need to do. So I can tweak the helm command. So, yeah, you could do that. And I think there's we could have a look at the

57:36 at the Helm chart source code and and see what the values are for this to to set in the config map. In fact, if you if you go in and try and edit the Cilium config map now Yeah. Yeah. Let's do that. You've already added it added to it earlier. Yeah. I'll work out how to get that back into the just file when I push this code online, but we we can just do it direct for now. So this one says keep proxy replacement. Have a look at that may be already set. Yeah. Prob. Yeah. Try and set it to strict. Strict.

58:20 Okay. And then do we need to provide the service host and service port? Yeah. That's right. So because because normally, we connect to the API server using the using the service IP that cube proxy provides. But since we're about to cube cube proxy, we're not gonna have a service in the first place. So, yeah, that's that's the the one that we'd want to tweak now. Is this is this here in the config map? Or Okay. So Have a look at API, for example. I think I think it's probably not in this config map. So let's have a look

59:07 at the yeah. Let's have a look at the Cilium chart. So if you go to the Cilium repo I mean, I can just rerun this. I've already. Oh, yeah. But we are in well, you'd want to oh, yeah? Yeah. Because that's this is the home template. Right? So I can just add these parameters. Oh, okay. And we can just rerun it and reapply it. You're not you're not passing the throughput, pulling me or whatever. Right? No. No. No. This is just Yeah. And you but you also you also want to change this for one nine. So if you'd switch the doc to

59:44 118. Yep. Okay. So let's see. Does that change? Yeah. Yeah. Global. Alright. So And you you want the Hubble port back as well? Yeah. Let's do that so I don't forget. So Yeah. If you go to Cilium. Yeah. And then deploy folder. I already have the same as this last week. Install. Install. And then yeah. Oh, it's like, they've can have a change that annoying thing. Maybe I get a bit faster. Let me just get faster. I know. I was just celebrating that it's not hiding the folders anymore. That that was like the worst change to get happen in

1:00:30 a while. Anyway. So I changed it one eight. Well Yep. One eight. Mean, one eight. So I know one eight five for the last time. Yeah. Not values. Let's Charts. Oh, values as well. Yeah. But want the lesson at four two four four. Yeah. So global Hubble lesson address? Not to is it global Hubble? Oh, yeah. It's global Hubble. Okay. So, yeah, we've got rid of those global values in Okay. So now we need to work out the Kubernetes service, hosting service port. What what how do I get those values? Does it tell me? So you could get those

1:01:22 from your cube config file. Alright. Okay. Okay. Gotcha. Is that is that if if the if you know what's the private address, that's probably preferred. But So it's Yeah. Probably probably fine. Yeah. Yeah. K. Nope. Wrong file. So we want this. Yeah. 644 the IP address. 6443. Yeah. Oh, service service has been serviced. Yeah. I think so. Yeah. Okay. Well, I think this is good. Just just take a look at the docs once again and So are we sure that it's not just the actual service address? Well, yeah. Because No. Look I need to do the right

1:02:16 namespace. Let's have a quick look at the docs just just to to refresh because naming is not a trivial problem, like we all know. Replace API server IP. Yeah. No. Because the actual you see, it's just the API server IP and API server port. Oh, you're in Usually, it's 6442. Okay. Yeah. We've got the right values then. Okay. Nice. So that means naming. So we I can just we don't need to edit that config map anymore. I can save as fail, and we wanna regenerate. So just what did I call that? Cilium. This regenerates our helm What

1:03:00 did I get wrong? Nice error method. I mean Alright. So now I can reapply. Yeah. And Cilium. Ugh. My spelling. My typing. Cilium. Most of that will be unchanged. The operator has been updated. We may need to kick a few things, I guess. Delete. You had the first first command. Oh, no. It's already restarted. Oh, okay. New operators. Oh, yeah. And also the also the agent. Good. Okay. So we can have a look at the So is that Yeah. Right. Can I go to Cilium proxy? Cilium proxy. Yes. Let's try and get get daemon sets.

1:04:05 All namespace is cube proxy. Alright. So delete and cube system. Daemon set cube. Now we're confident. Yeah. Done. Could possibly go on. So now we have no cube proxy on any of our on any of our nodes, which means if Cilium is not working as we expect, we have no networking. So let's see if Hubble is still alive. It is? Mhmm. Let's try service. So And our ships are landed. Let's try the Twitter. Okay. Still good. Well, take a few properties to around. Maybe maybe it's, like, still terminating. Okay. So let's run get pods, cube system.

1:04:37 Verifying Connectivity (Kube-proxy Removed)

1:05:07 It's gone. It's gone. That was really quick and painless. That actually worked. David, how did we not screw it up? So that means there's oh, no. No. No. No. No. No. No. Will still be IT tables on our notes. No. They wanna have a look if there are any IP table rules. Yeah. Yeah. Let's Jump into a node and have a look. Yeah. Oh, yeah. But I can just get IP addresses from here. Yeah. Let's go on a control plane node. That was Right. Working nodes. Well. Okay. IP tables. Yeah. IP table save. I mean, these look empty to me.

1:05:22 Checking and Clearing IP Tables

1:06:09 There was something in the docs, wasn't there? Yeah. Let's let's let's have a good look there. So done. Validate is set up. Alright. Oh, yeah. Let's do what it tells us. Alright. Let's let's not celebrate too quickly. So Yeah. Right. Yeah. So let's see what that's I know the pod name's different, of course. Any of them? Shouldn't tab work? Different. Here's Set it in my face. Mhmm. There we go. So We've got strict enabled on a key proxy replacement. We don't need an engine x. We've already got an enough things going on, I guess. Yeah. Yeah. No. We can we can definitely

1:07:02 tell that this is so you can resend them service list. I'll just type it rather than changing those bits. Sir, service server service list. Cool. Yep. That's good. I see numbers. Yeah. So I think it'll save. Okay. Let's try that. Yeah. So it wants me to do that. We can run that and still import, I think. Or on the note if you still have the session somewhere. Oh, Still not gonna run. Empty. Clear IP tables. Oh, just reboot the node. But that will take five minutes. Damn. That's what I forgot. How to delete and clear?

1:08:08 Woah. You can try a firecracker. How quick can we do it? And the immutable upgrades and such things. I remember sort of thinking about the design where you would use bare metal and single VM per host with essentially something like Firecracker just so that you could get upgrades by means of images and and faster reboot. Okay. So on that. I've now deleted all inputs forward and outputs on test note. Like, table save was the one that we tried. Right? So now it's just I'll just do it on all the notes. I think maybe that's a little

1:09:06 bit quicker. Mhmm. So let's do this one. Copy this again. My finger is my SSH key now, so Alright. Gone. Last one. And that means that our worker nodes are all IP tables lists. So then we can test our traffic. Yeah. Alright. So now we have no IP table rules on any of those nodes. So let's go back to the typefader. That still work. Let's try Twitter. It it it works. Oh, cool. Yeah. That's Nice. Very cool. I mean, the only challenge in bit there was just removing coop proxy. It wasn't quite as trivial, I

1:10:05 guess. But you're right. I could just have repeated those notes up at the being. Yeah. That that would have taken longer. So I wonder why they didn't do tables through sat around. I'm not sure. Awesome. That was really cool. Yeah. Yeah. Let's pop that away. So yeah. I'm really impressed. Like, you know, I knew that Cilium did network policies. I know there's the e b p f rate and that's one of the hottest things going on right now as far as kernel security and networking because of the just raw performance numbers. So Right. Know, and that integration is just really

1:10:22 Conclusion and Wrap-up

1:10:47 fast. Removing IP tables, you know what? That is really cool. I now need to go and upgrade some production clusters. And then that the cherry on the the cherry on the top of that was those DNS rules. I didn't know that was possible. I just caught that in the docs. I'm just like, that's that's a really good feature. Yeah. There's there's more various things that one can enable as well. I think that table is definitely something that we should be able to part with in the Cloud native age. Yes. Definitely. I I completely agree with that.

1:11:23 Well, thank you for for joining me again. We didn't break anything. Things worked. I think we explored, you know, a lot of the high level concepts of Cilium. Hopefully, it peaked that people's interest in it. Congratulations on reason money enterprise product. Very very cool. And hopefully, can do some more exploration of Cilium and and then your features at a future date. But you know, that was that was great, Elliot. Thank you again for joining me. Thanks, David. Alright. You have a a great day and I will speak to you soon. You too. Thanks. Bye. Thanks.