1:11 Introduction & Recap of Part I

1:11 Hello, and welcome to today's episode of Rawkode live. I am your host, Rawkode. Today, I don't even know what episode number we are on, but we are continuing our exploration of running Laravel in a production fashion on Kubernetes. Last week, I was joined by Alex and Kieran, and we had a little bit of trouble or problems getting a Laravel metaware to expose or export Prometheus metrics to be scraped to allow us to horizontally scale. Fortunately, our friend Leo is here to help us today. Who has? And, funnel, I don't know how sad it is this time it was, but you

1:44 Introducing Leo & Laravel Prometheus Middleware

1:49 actually had written this exact bit of middleware that we needed just prior to that episode. Is that right? Yes. So I was talking to you, I think, like, the day before or something, and I was looking into Prometheus. And so I wrote that, and then I completely missed the episode. And I was rewatching it after, and, you know, as soon as I was like, alright. I I just wrote this yesterday. It would have been real good to watch this live and just, you know, link to link to the package. Yeah. That would have been invaluable, I think,

2:18 that day. We we tried two different packages, Alex and I, and just really just didn't get anything to work. Yeah. Nothing work. We're parkour net, and nothing worked at the end of last episode. But So every taste Yeah. Different stories, though. Yeah. No pressure, Leo. Alright. Okay. Forget my manners. Let's do a quick introduction. So we'll start with you, Alex. We'll move over to Leo, and then we'll get started again. Cool. Yeah. I'm Alex. I'm a PHP Laravel developer that doubles a bit in Kubernetes and then getting a bit closer towards dealing with that more production, hence, doing these streams with

2:55 David. Yeah. And I'm Leo. I'm the CTO at Jobberle. We build recruitment software. We're a very small team, so I do pretty much all of the DevOps side of things, which is why I've been doing the monitoring and Kubernetes things here, which is why I'm now here to hopefully not break things. Fingers crossed. Alright. I guess what we should just do is get the screen shared, quickly run over in under three minutes where we are again, and then just let's try and get your library imported running, and then we'll tackle the scalability side with Kubernetes.

3:21 Environment Setup & Initial Deployment Overview

3:36 All right. Let's see. So this is the repository, github.com/alexpowers/laravel-example-project. I don't think I even pushed last week's failure to it, but we'll do that after today when it works. I have Versus Code. Leo has kindly joined me. Good. Because you'll be fixing everything that I do. And we have a make file, which has our build targets, which I did this time build ahead. So we do have two container images. We can deploy this to Kubernetes. So let's just do that first, confirm that it works, then integrate Leo's library. So if we open our Kubernetes directory,

4:26 I think we can probably just reapply all of this. I don't think if we yeah. We did add some Prometheus stuff that I am gonna nuke. And we'll do that clean. Do we not want to keep any of the Prometheus stuff? No. Because the service monitor's weren't working last week, and it looks like it's just a bug in the the operator that we were using. So I have gone ahead and deployed my own Prometheus to the cluster and we're just gonna configure it the old fashioned way. We're not gonna use the service monitors on the operator

4:56 this time. So that is just, I'm more familiar with it this way. I don't wanna have to try and fix the bug in an open source project in our stream and I don't have time before prior. So, let's apply our operations Kubernetes directory, and this should get our application up and running in just about ten or twenty seconds. So we'll give that a few moments, confirm that it works. I guess what we can do just now is start to pull in your library, Neil. Yes. You can start pulling it in, but that will not work. So what you will

5:27 Installing PHP Dependencies (APCu Extension)

5:30 have to do is actually install the APCU extension in PHP. So that is a user level cache, which the Prometheus the underlying Prometheus library uses that instead of Redis. That is a choice I made in the library. So instead of Redis, it uses the APC user cache, which allows you to get kind of an in memory store to store between requests, which is how it's storing the request count and stuff like that. Alright. I'm I'm so glad I built all that stuff beforehand. Yes. But it is it should be two lines in the Dockerfile, so I think we're good.

6:11 Yeah. But I oh, actually, we don't have a working cache if I remember right from last time anyway. Yeah. Okay. So No. The stuff resolved. Yeah. We will fix the cache now then. Right? So we can just do this. We'll see if this builds. Normally, there's a small why is that copying it to such a random place actually? Oh, that's just where we copy everything. Okay. No worries because we're using the Rector. That's where I'm okay at this stuff. Okay. So we have the composer dot JSON. We're gonna do the composer install. Sometimes, I think Laravel does fail with that

6:40 because of the something, but let's just see what happens. Alright. Can I just do run Docker PHP install a p c go? I am not sure if it'll work with PHP install. I've used peckle, so PHP extension library manager, whatever it's called. We can try it with PHPX install. It may or may not work. Otherwise, there is a simple way to do it, which will just be two lines instead of one. Well, why don't we just do a entry point bash, jump inside this? Surprised that our image doesn't narrow because I've built the image. Yeah. It does. Yeah. It's

7:24 just this low. And this is all paints. We wanna run ash. Okay. So Docker extension install as a p c u on list. No. Okay. So we're doing the peckle approach. Yes. And you're saying the command is PECL install a p c u? Yep. That's all. And then what you wanna put in the Docker file is also Docker PHP EXT enable a p c u. Alright. Well, we had our first nag installing that. Of course, we did. You need to install auto comps, I guess. So Yeah. We'll need to build There is. Yes. There is actually, if memory serves, a dollar

8:12 PHPIs underscore depths that you can use in the build step, which has all the PHPIs dependencies. So it's an environment variable that you can use with the Alpine images. Okay. So you're saying if I run the peckle install again, but with an environment variable? No. Sorry. So it's it's just a list of packages that you pass to APK. Alright. Gotcha. Okay. So PHPIs depths. Yeah. Looks good. Third from the bottom. Yes. Alright. How did I not see that? Okay. So it's just APK ad PHPIs deps. Yeah. I don't know why this is three ads, but whatever.

9:01 I'm gonna say that It was because I I did them one at a time, I didn't wanna have to redo the old ones which you've already done. Yeah. Presumably, the build cache. Right? Yeah. I will need an update. If we have to rebuild anyway, then I guess. Yeah. Okay. So we want to use Leo's library. We need EPC here. We're using that instead of an external Redis. I've added the PHPIs depth. This should allow me now to run a pickle install EPCU. And let's just confirm. Like, I'll run this You need to delete the line nine as

9:35 well because you've got the pipeline in there still. I will just do that. There we go. Line six. Yeah. Thank you. Okay. Echo. And we expect this to work. Hopefully, it's nice and quick, and then we can rebuild our images. I don't really want a question. I wonder if that's gonna block my build. It will not because it will default when it's non interactive. Good. That's what I like to hear. I know this because this code runs in production for me, so it it should work. And we do use 7.4 FPM Alpine as the base. You're you're already throwing all

10:20 of your excuses out the window for when this doesn't work. You need to need to save some of these. Okay. I am fully confident. There's no help anyway. So I'm just gonna go with that. And then you said we wanted to run a Docker PHP extension enabled APC. Yeah. Yep. And that is what adds it to the PHP in the file. Okay. So we need one more command. Docker PHP extension enabled APC. Alright. So this modifies our base image. This should mean that anything we build subsequently down will have it. We have moved this composer dot up. This

10:58 this composer is all up to try and keep our cache here, but we are about to make a change to composer dot JSON. So why don't we do that first so that we don't have to do two builds? Yep. So I'll let Leo do a little bit of typing. You wanna show us how to install your library? Yes. I need to actually remember what it's called. There we go. The the easy way to do it is just going straight into the documentation and and following that. You want me to build up the docs for your library? Where is it,

11:08 Integrating the Middleware & Fixing Docker Build Steps

11:30 by the way? We should tell people. So github.com. Yes. It's github.com/jovila/laravelcloudnativeutilities. It also has a bit of defaults for, like, logging configuration. I kinda just tossed everything into one place. Alright. So But I have added it now, so that should be good to go. This is the release I just tagged a moment ago with support. So that shouldn't break anything else. Shouldn't is my favorite word on the show. Telling you now. Okay. Let's try and build this then. So we've got two main targets. I'm only gonna build FPM. We don't need to worry about NGINX again.

12:15 Don't really just wanna check that Docker file. I wanna move artisan up because I know there's weird behavior if that doesn't exist. We had that on a previous episode, but I think we'll Yeah. Save us out the rest. You can get around the artisan problem if if you wanna do that as well. I mean, I'll just talk through it because we're waiting for the build anyway. Yeah. The way you do that is to run dash dash no order loader, which will prevent running the scripts, and then you run dump order load after copying in the app files.

12:44 That way, we'll be able to run all the scripts and order loader and stuff, and it doesn't need artisan ahead of time. Yeah. There was another weird issue, though, with Yeah. Laravel mix On the assets. Yeah. Requires yeah. The JavaScript's compiling stuff for webpack requires artisan as a file to exist. It uses that to find out the Laravel root, which took a long time for us to find on the stream. But yeah. Alright. Well, well, hopefully, as image builds, so okay. If I remember right from last week, it's the random ninety ninety seconds mark, so

13:20 it's not not too bad. And then we should be in a position where we can just start instrumenting. Is that is that all auto instrumenting then, this this library? Yes. So there is a configuration file for it. So, yeah, the installation docs do have there is a command you can run to publish a config file. By default, it's gonna expose a slash metrics endpoint that I think it has quite sensible buckets of 0.1, zero point two, point three, four, five, point seven five one two and five seconds or something like that. There's a lot of

14:00 buckets. There's, 10 of them or something. Alright. Well, that should give us a little flexibility then when it comes to adding our horizontal pod autoscaler. So I I did also yeah. Sorry. I I did add just couple days ago the support for changing that in the config. So if you do wanna change the buckets, you can publish that configuration file as well. Well, you know I'm gonna test that now. Yes. Of course. I have not tested it, but that that'll be interesting. We'll see what happens. I'm curious. Like, you're using this in production at at Jobula. Right? So are you

14:32 using a horizontal pod autoscaler in your Kubernetes setup at the moment? We are not using it at the moment. Largely, we have a very stable load, and we are I mean, we we survive fine on honestly, like, one pod one replica, maybe two. We don't really need to scale up a lot. We do have got a lot of resources for each pod. Yep. So it's running quite fine. Also, I did write this library, like, last week. So, you know, I I just have not had time to do it. But I'm making up the excuse that, you know, we don't need

15:10 it. Are you again? You just got to get a little done early. So at the moment, are you basically using it in production just to gather the metrics rather than to make use of the metrics? Yes. Literally gathering them because we barely have a functioning Grafana setup as well. I probably shouldn't be saying these kind of things on the stream if when I try to, like, work with me. No. We have a we have a beautiful environment. That doesn't look good. The requirements could not be resolved. It looks like you require a certain guzzle installation

15:49 in one of your currencies. Yeah. I don't yes. Because we have requested 70.1, and they're saying 6 Yes. So that's requested by the Prometheus client library. Alright. If in doubt, six dot star, why not? Is that how you do it? See, I was trying to remember, but it's you know, I haven't read it right You can just do greater than six point o. Yeah. Okay. Yeah. I don't know if does greater than work with the asterisks? I'm not sure. Also, it it is the up caret yep. All good. Alright. But I should start from our composer step

16:29 anyways. So let's see. Yeah. That's true. Yeah. No. It does not like that. Okay. You can't do it as chevron and the star. Just do chevron Zero. Yeah. We're professionals. We know what we're doing. I'm not. I'm relying on user. There we go. It's happy. Right. That looks better. And I'm pretty sure there was nothing in my code that was using Guzzle at all, so nothing else should break. There's the artisan issue. Okay. You need to copy Bootstrap. It might be worth just putting the computer installed back where it was. Yeah. Let's just do everything. I don't wanna fight with that.

17:12 I don't wanna fight with that today. I fought with that in the past. I don't have it. I don't remember the solution at the top of my head. We'll deal with that later. Yeah. I think it's probably what Leo said before. Make it so it doesn't do any of the scripts afterwards, and then when them yourself have to copy the rest of the files. Yeah. Let's just get this deployed. I wanna see this metrics endpoint work. I wanna configure the buckets, and then we'll start scaling. Oh, I already did the dump porta load after the fact as well. It's interesting.

17:46 Oh, well. Okay. We have our images. Now we're just building the same tag. So that means if I do get pods, all I really need to do here is delete our Laravel pod here. And we're only running one. This will trigger it to use the new image. I'm gonna port forward at metrics. In fact, we've not we've only added it to composite dot JSON. Is there any other steps, Leo? No. That should be it, or it should do something. I I guess we'll find out if it does. Right? I mean, I'll need to add it to, like, some configuration

18:08 Testing the Laravel Metrics Endpoint (/metrics)

18:23 or No. Laravel has auto discovery. That's what was mentioned last stream when we had to start finding which file to add the references to when somebody was saying about auto discovery. That's one of the things that came in like Okay. So we have our posts. We have our home page. Do we have metrics? That's the moment of truth. We do. No. We we did hard code some metrics last week, though. Have you removed the hard coded ones? Because I think we put our own No. This the hard coded one. Yeah. So what happens is the library registers its own route.

18:59 But if you register the same route afterwards, then your route will take precedence. Yeah. Which is going to roots and it'll be in web. Web. Web dot PHP. Oh, yeah. That's my Yeah. Just delete that. Yeah. Right. I was like, oh, it works. My library works. Everything is fine. Does work. I had we're not confirmed it yet. Don't worry. It's coming. Alright. Quick rebuild then. Yeah. And this is where we need to do some more work on the composer on the Dockerfile. Because we haven't changed any of our dependencies. But because we are gonna change this time

19:35 and time again today. So Yeah. We're only gonna really change app and routes. Right? Yes. I was like, maybe we can get away with that. Mhmm. Probably case. Oh, no. We're dot probably We're gonna configure the the buckets, and I'm assuming that'll be in config. So why don't we try it with these two? Just and if it doesn't work, we'll just have to deal with the rebuild every time, I think. So let's check out the one more. And while that does the I'm gonna color pod. We'll check this metrics endpoint. So that's deleting. We can do a port forward.

20:23 I really should have checked the name. Oh, that failed again. So I I think we're just gonna have to deal with the thing. I don't think it worked. It worked. I saw that looking at the store. He was worried. I I told you. I I had full confidence in my code here. I think the challenge was maybe the patient on the first load, but I think we're good. Okay. So that worked. We have metrics. Let's jump back over here and we're yeah. We just have to unfortunately deal with that. So let's try not to make too many

21:07 changes on the PHP side. And if we do need to make some, we should probably just make them know and kick off that rebuild. Yeah. But we do some of the you want, we because we're only dealing with the metrics now, I suppose we could also just do the no auto loader step right away because we're not dealing with any issues related to resources probably. At least there's no build step related to mix. So if you just do dash dash no auto loader, you can have it right after the composer JSON copy. Yeah. That will help us. Yeah.

21:45 I just wanna make sure we don't spend too much time phasing it this. Yes. I'm hoping this should be quicker, though. Alright. It looks like it's maybe getting a bit further than that. Is that happy? Is that our last ever compulsory install? Oh. There we go. Nice. I think it's very optimistic to call it your last composer install. Ever. And done. Okay. I'm really happy with that. So what do we need to do now? Well, we wanna start scraping the metrics. Let me grab Yes. We need to get a data into Prometheus, I guess, aren't we?

22:22 Configuring Prometheus Scraping (Kubernetes Annotations)

22:36 Yes. Now the old school way of doing this is to modify the deployment. Oh, we're not using like a d anymore. I don't know. I just prefer that approach. Fine. These are the three annotations that tell Prometheus to scrape within Kubernetes. I think we're running on port 80, so we'll just leave that open. Is that the port inside of the container? It is. Yes. Yeah. Port a t. Yeah. So we just tell previous that we want to scrape by saying this to a string through. We tell the endpoint to scrape in the port number. Most of these are at the fact. I'm

23:10 pretty sure a t and slash metrics are the defaults and I could remove them, but I'm not going to. Just trust me. That that's always worked out well. We're gonna reapply our Kubernetes manifest. So we're gonna get pods. There's there's still stuff in there about PON. Are they still needed? Because they were put there last week for our attempts. Sorry. Can you say that again? What's in here? Go to the top. Line fourteen nineteen. Are those still necessary? Because we put those in last week. No. That was when we were trying to target the service monitor and

23:51 it was that bug. Yep. Okay. We can reapply and actually oh, you can't change them. So they're immutable. It's deploy Laravel. Let's just make it and then reapply with it. Alright. Let's see where we are. So give that one more second. Okay. It's running. I wanna make sure we have some metrics that we can scrape with from me, so I'm just gonna put forward one one more time. At least for now, one more time at the metrics endpoint. Do you need to hit a different URL to get metrics first? Yeah. I think that's the issue.

24:15 Troubleshooting Prometheus Scraping

24:36 There you go. You might be correct? Yeah. Yes. So there's just no metrics upfront. That's all it is. No no nothing's came through and hit. Not a big deal, but we do have some stuff here. So what we really wanna check now is that if I browse to Prometheus, one, we should see that there's a target that wants to be scraped. Two, we should see some metrics and three, maybe that'll all work. So I'm not gonna port forward to that anymore. Instead, we're port forward to our Prometheus server, which runs on port nineteen ninety. Local host

25:17 1990. We have our Prometheus UI. Now if we come into target oh, come on. Oh, there we go. Is it there? I saw some proof eighty eighty on there. But we're targeting 80. Right? Oh, shit. Yeah. Yeah. That's what is going on there? Yeah. That's a Kubernetes service. This is where my expertise ends because I set this up with a service monitor, and I was all happy. Yeah. Yeah. Yeah. So this this is on David to figure out. There's another tab on the status called service discovery. Would that that have any met bearing on it or not?

26:09 Because I'd assume that we'd be listed as a service on there. Right? Maybe it's just not picked it up yet, to be fair. Looks like we have anything here. No. Chrome page. Yeah. So we should be showing up here based on those annotations. So let's quickly check we haven't I haven't missed anything up. So we got a question that I'll throw to Leo. William asks, should normal setup health checks generate some initial metrics? So, yeah, the probes if we had probes on our deployment, I can't remember if we do, those should probably populate the first source.

27:02 First They would. Yeah. They would. And I think a question here as well is, would you really want that? Right now, the way the library works is it excludes all requests that hit the slash metrics route, but nothing else. So it might make sense to add support for excluded routes. I'd imagine you might not want the health check timings necessarily. I don't know if you want that in your metrics or not. Okay. Good question. I don't have any labels here. Did I put them in the wrong place? Why is my old tab not working? There

27:54 we go. Yes. Yep. You did. It should be on the template. Yes. It's meant to have to one more. Yeah. Okay. Right. Okay. I I'm sure everything will work now. God. Whatever This is your first stream, I guess, then, is it? It's the first stream where I'm participating. It's the first time I collection Everything has worked so far aside from not being able to build the container and a few other things. Except for my stuff, you mean. Thanks for that. Okay. So that's now running. All good. Okay. So we'll need to hit a few routes again to get some data back on

28:42 it, won't we? So you'll need to port forward it again. That is correct. Port forward this part here. There's a there's something called kubed forward. Terminating. Alright. There is a project called forward that actually exposes all of your services locally. I really should just start running that for these streams, but we'll make do it for today. Alright. So let's have post, home page, and then we should see some nice metrics in here. Perfect. What I wanna see in the targets now is our Laravel application. No. We do need to maybe wait the scrape interval. How long what is the interval? I I

29:29 believe I have it set to ten seconds. Okay. If that doesn't work, I'll gently nudge Prometheus and we'll see what happens. But we are in the same namespace. Alright. I know I'm getting a bit tab crazy now, but let's see. Everything is in default. So I would expect the previous to be scraping our Laravel application just now, unless I've got anything wrong. I see the labels are on the pod. I have annotation. Sorry. Yeah. Looks alright. When did start hating me? Alright. What have I got wrong? Let's nudge Prometheus. So I get pods. Delete pods.

31:01 Should be restarting now. Yep. We can forward. You spelled forward wrong. Thank you. I saw something there slash metrics with no port on it, which would imply, I guess, No. It's monitoring itself. So Oh, is that yourself? Oh, okay. Yeah. Our stuff is not showing up. Fabulous. Is it definitely on the same namespace? They are on the same namespace. Yes. They are. Do you need to change anything on Prometheus' side to tell it what labels to look for? No. No. But we do have buckets. So this is my application and these match what we have here.

32:17 So if I don't retest that just quickly, root equals slash post. Did I get that wrong? There's no slash on it. Oh, it's yeah. Just post. Okay. We are getting our metrics. It's obviously just heading in our target somewhere, and I'm scrolling past there. Oh, no. It's there. It's there. Yeah. Yeah. That's what I was on about. Yeah. Yeah. Yeah. I'm not getting insane. It's all good. Alright. Good. Everything is working. We have our metrics. We can, in theory, apply an HPA and scale this based on the response time rate. And one of the things that I think is really important with

32:50 Discussion: HPA & Scaling Based on Response Time

33:04 pod auto scaling, You know, we talked about this a little bit last week, but scaling on CPU utilization or memory doesn't give you any indication of what an experience your end user or customers are getting. Always always better to scale on the real response time. Like, I'm assuming, Leo, do you have any SLAs or an ideal response time that you expect your customers to get a response from? So we we don't enforce any SLAs, but we are trying to now to keep a 95 p of two hundred milliseconds on the API. I would like to have that

33:40 down to one fifty ideally. And I would I would agree that especially with PHP applications, it's even more important that you don't scale particularly on memory. Because if you want to optimize your setup, you wanna be running with a static number of FPM processes. So you're basically gonna have static memory consumption. Yep. Alright. And do you wanna tell us what about p 95 is? Yes. So the ninety fifth percentile, so 95% of the request going in under that amount of time, so two hundred milliseconds for 95% of the requests. Exactly. Alright. Now before we go throwing the HP on this

34:00 Adding Load with Siege

34:25 then, why don't we run siege against this post endpoint? Like, right now, we can see that our buckets are or nothing is responding in under five hundred milliseconds. So, really, we're gonna start to see hopefully plus five seconds plus infinity buckets really start to fill up over the zero and a half to 0.75. But why don't we try and emulate this in a way that maybe is real world day? Is that a word that is today? So we just see each last time. Right? And we just we don't really tune any parameters. Is there a way we can just throw

35:03 change. Sorry. Go on. That's alright. How can I just I I just wanna send a couple of hundred requests to this? I don't wanna overwhelm the system. I don't wanna I just wanna see a few buckets fill up with some data. Is there a flag? There's a You can dash c. Right? Dash c is concurrency. Dash t is how long it should run for. I don't know if you can set the specific number of requests. I'm not sure about that one. Well, I mean, I don't make control c in that, I guess. So I know you

35:32 can in Apache benchmarks. How big are reps? Reps sounds usually. Dash r. Well, I'm gonna add dash p, so I wanna print stuff. Concurrent is only 10. I'm okay with that. Let's just do a concurrency one so we can control what's happening. Let's do reps of let's just do 1,000 request. That's gonna take, like, a thousand seconds. I can currently see one. Let's just do 10 here. 20 requests for now. Time, I don't think we need to worry about delay. I don't think we need to worry about I don't think we need to worry about anything else listed here. We happy

36:07 with that? Mhmm. Alright. Oh, the p is actually printing at HTML. I I just wanted a summary of the request. I don't know I actually want to output. You get you get the summary of the request once you cancel it off normally. I I was hoping I'd get a running count or something. Yeah. Make it a bit more visual. That's not a big deal. Okay. Cool. How big is your history on your terminal? Your scroll history? And because all the oh, okay. Right. Right now. But I'm not fast. We're we're not gonna do another print. And now I'm gonna

36:44 do this one more time on the correct endpoint. But what's cool there is we can actually see a request. We actually have a a spread, a pretty even distributional spread to be fair, even at a hundred requests. So I don't think triggering a scale is gonna be too difficult. I'm actually really surprised at the currency of one sending 20. That's not right because we've got, like, we've got a lot of requests here. Does it send more than 20 or does your code not work, Leo? My code works perfectly. I it's a good question. I'm a little

37:22 concerned about 21 showing up. Exactly the same. We either have it's either a very even distribution Or a bug. Or maybe problem. We only have bugs in here. So we have 20 transactions, all responded correctly, and do we have a average time response time of roughly we think that's three hundred milliseconds. Yeah. Oh, I think there is a bug. Showing yes. So if they're because there's 20 requests. Right? And we have 21 on everything. Excellent. I I knew something would happen. It's not a big deal for today. We can still we All we really wanna do

38:09 is inflate these metrics anyway and we can still do our testing. But yeah, it looks like the library maybe has a wee bug in it. Alright. Yeah. That's Here's the total request down here. We got 21 on each end point. Yeah. And a few 20 one's gonna spread across there. Weird bug. But alright. I thought I actually think the 13 requests that went onto into here are probably correct. Not sure what the other ones are. Maybe it's just a I'm I'm confused about it. I am relying on the Prometheus PHP library, so I'm a little concerned about that.

38:47 Uh-huh. See how he just delegated that bug channel I'm not sure I'm not sure if I am just misunderstanding how to use the library. That is quite likely. But you know what? I don't right now, I'm just happy we have a metrics standpoint that is somewhat working, and we can work with today. So We actually managed to install it this week, which is a progress over the two that you tried last week. So Alright. So you'll work on that. I'm gonna work on the service monitor bug over the next week or two, but we can get away with that right now by

39:15 using the old school annotation approach. So we have metrics. Now, if we want to be able to scale based on these. So I'm just gonna recap a few things from last week that I think maybe makes sense. Let's pull this down. So we're running Kubernetes. We installed a metrics server. If you didn't see that, go watch last week's episode. The metrics server is what allows us to actually get metrics out of Kubernetes. It gave us resource utilization metrics on CPU and memory. If you want to augment that, you have to use a metrics adapter. And we don't have that installed yet. That's

39:20 Recap: Metric Server

39:54 what we have to do now. We did that last week? Yeah. I think so. Because we put the metrics server on there, and then we tried to get data into it from Prometheus, and that's where we failed. Alright. Okay. So this should just be a case of installing the helm chart version of the Prometheus adapter. I don't think I need to tweak the values in any way, but we'll see what happens. The Prometheus adapter gets installed to the cluster. Actually it gets encapsulated with the metric server and provides extra endpoints so that we can query the metric server for the custom metrics

40:00 Deploying the Prometheus Adapter

40:38 from Prometheus. I know it's a really weird setup, but you kind of get used to it. In fact, we can just see if we have any custom metrics available here. Might not be available yet though. Yeah. Not quite ready yet. So give it a few more seconds. While we wait for this, I can also confirm I am now 97% certain that I am just incompetent at using this latest library, misunderstanding how you think the bucket is. How to report buckets. So I I think that will be an improvement that I will be making after the stream.

41:26 Alright. You don't wanna fix it right now? I'll I'll try, but but not make any promises. No. I'm make us spend too much time on pulling in version zero point one point five, one point six, seven until it's fixed. Alright. Well, it looks like we're okay now. Our pod is no longer pending any probes. We actually get a response from custom metrics here. There is this really long man that will allow us to create a pod metrics from Prometheus. We'll see if it works. So I thought I should just share the link that I'm getting this from as well. This

42:13 was a stream I did a month ago with my friend Guy, who is the chair of SIG Auto Scaling or co chair of SIG Auto Scaling Kubernetes. And this is that auto scaling example workshop that we did live on the stream and we're reusing most of these components. And the docs says this command, which allows us to actually query. So I just need to tweak this. Right? So namespace default is fine. I'll pause, and we're looking for h two p request per second, which I believe we don't quite have. Let's copy total. Also, I can confirm

42:55 I can now tell you exactly how the library works. I just looked at the code, and it is kind of correct, actually. You should be getting 21 in all of them because 0.5 is a longer duration than 0.3, and that's why you're seeing 13 at the lower numbers. I wouldn't bucket them that way. No. But as in that is the way the library buckets them, so that's why they appear that way. Alright. I mean, I think that's a bug, but it's not a big deal. Alright. We'll label selected match. Oh, yeah. Okay. So we want

43:00 Debugging Custom Metrics API Access

43:31 app. What was the syntax there? Equal. They had a percent three d, I think. Did it? I think it was, like, a the entity first slash, I think. Yeah. I've got it over here. Interesting. The server could not find the metric HTTP request total. HTTP request total for pods. Alright. We'll move on for now. Hopefully, it's not a problem. It could be, of course, there's always something is that maybe I do need to point this towards our Prometheus. Let's see. Do we have a service? We do. In fact, let's just pull up that chart. Let's take a quick look rather than speculate.

44:35 So it is the Prometheus adapter. Here. In fact, it was the Prometheus community. This is the chart we installed. We can go to the repository. Nope. This one. Once it's in place. Okay. And do we have the values that we can tweak? Where's the chart? Why? It's just manifest. That's the actual code for the adapter and not the helm chart. Maybe it was this first one. This deprecated one. Maybe it still just lives in there. Let me face adapter. Okay. Oh, there we go. That's it. That was a long way about to get there, wasn't it? Anyway,

45:19 Fixing Prometheus Adapter Configuration (Prometheus Service Endpoint)

46:02 now we can go into the charts. We installed the adapter. We wanna look at the values fail and just see if there's any configuration that tells us where our Prometheus live. So we can see here, it actually does expect there to be a Prometheus service, which I do not have. So we can redeploy it with a small tweak to this and it should should exit. So let's create a quick Prometheus adapter values. Yml. I know I should probably move that into our ops directory, build it there. And ours is called server. That's right. Yeah? Are you happy with that?

46:51 Yep. Yeah. Prometheus server. Oh, port 80, the service. Why port 80? Because you were forwarding port ninety ninety before. Yeah. That's a hard port, but this service is mapping it to port 80. So Oh, okay. Right. Okay. So I'm gonna reinstall this using a values file, which is there. Oh, and we already installed it. Upgrade. Okay. Let's wait for that one to get healthy. I guess we'll wait a bit longer then. A lot of waiting in Kubernetes. I was gonna say this is really the Kubernetes experience. Yeah. It is definitely. Now there we go. Okay.

48:11 So we had curl request that we No, we had a kubectl raw request that we fired away. First one was just to hit this and that's looking a lot better actually. Let's see if we get our request total. No. I'm gonna ask if it's working, though. Yeah. Is this like so definitely up slash and not just Laravel example project? That's okay. Because the selected that we the label we put on the actual deployment doesn't have app slasher, I think. The label itself is app. Right? Let's just work it out. Let's just stay up the old scale. I'll see what happens.

48:58 I mean, we can always take a look at like, we see quite a lot coming back from here. That wasn't coming back last time. So I'm confident that it's speaking to Prometheus. Now, when we use the raw URL, we can extend that to actually Google deeper into it. And that's just supposed to actually proxy a request on to Prometheus to give us a response. That's HTTP request. Total is a metric name passing in the selector of app Laravel. I mean, I would expect that to work, but I'm not gonna let it hold me up just now. So why don't we create

49:30 our HPA? We're the same page? That all makes sense. I don't wanna jump ahead here. I think it makes sense. Well, we'll come back to that if it doesn't matter. Okay. So here is our standard HPA. Now what do we need to tweak here? We'll just keep continue calling this Laravel example. We have to tell it what target. We are targeting an employment. Ours is called Laravel example project, I think. Mhmm. Yep. We're gonna go from one to 10 depending on how we need to scale. Metrics are gonna come from pods and then our metric name. So we can copy this.

50:23 I mean, we can do it from I'm not put forwarding anymore so we can't. But let's copy it from just because there's a bit of unknown around how these function, I'll use request total and we'll just scale depending on our aggregate account of the number of requests because that's a really smart idea. And what we wanna say is that oh, no. It's not really gonna work because of the target value. Alright. Let's let's go with the So what's what's the time frame on the request total? Is that always increasing, or does it reset back to zero, like,

51:07 daily, hourly? Like, what's the deal on that? When the pod is restarted because we're using APC here, then the total will be reset with it. Right. Okay. So That total will actually reset. Well, yeah. We're actually gonna be collecting that from all of the pods. They're all gonna have different values. So be a little interesting, but we should get an aggregate We only have of it. We do only have one just now. Yes. K. Let's just see what happens. We'll set it to we expect it to be five. We'll tweak it as we we play.

51:46 Okay. Let's reapply all of our Kubernetes resources. We should see that we have a created HPA. We can create this and we can describe it. HPA Laravel. I think our selector is maybe wrong. It seems to think we have nothing. What did I get wrong? Employment. Laravel. Is it worth checking our deployment to see what the label is on that? Yeah. Yeah. It does match up. I'm looking at it and It's it's right. It it should work. Well, that's the story of my life with this Laravel stuff to be fair. Alright. Let's see. Unable to fetch the metrics in the custom

52:52 Defining Prometheus Adapter Rules (PromQL Series Queries)

53:09 metrics API, the server. Okay. So it's trying to request these metrics. Okay. So it's not yeah. That makes sense. So why is it unable to get those metrics? We were unable to get them too with Rawk. That's not good. Maybe we just got something wrong. But we can always clear the big messy one. Type it to jq and try and understand what we're getting back. I think we're gonna need to go through less. And then I lose my colors. That's annoying. Okay. So not interesting. Not interesting. I wonder if I should just filter it on

54:04 Laravel. Oh, we're not getting any metrics here. So I think that's a problem. Let's just quickly do dash a pod. Because they were coming into Prometheus. Right? Well, let's just double check that's still working. That's for sure. Port forward Prometheus server. Yeah. It's still the same, which means I should be able to refresh this. And we come to graph, then we can do HTTP request total. And we have our 21 requests. If we take a look at our buckets over twenty minutes, graph. Oh, I guess we don't have enough data. But we do we do have a value.

55:10 Yes. This here, we got app equals Laravel example project. I mean, in theory, we can use the raw here and just use a metric name without the selector to try and see everything. In fact, if we do that doesn't work. Like that. Yeah. We're not getting anything here. So how are those being exposed in the metrics API? Is there some additional mapping you need to configure? You're asking me? Yes. Is I'm asking you. Is it worth killing and re redeploying the metric service so to make sure it picks up Prometheus? Could I don't know. Turn off and on

56:04 again. I thought we did. No. You killed Prometheus. You didn't kill metric server. But then didn't I do a Metric server's eighty three minutes. Upgrade the adapter. Yeah. I'll try it. I think. Why not? Where is our metrics there for Oh, down. Yeah. Let's just we restart both of those. Oh, we didn't add the rules. There we go. My fault. Leo, your your Prometheus package, does that handle CLI stuff in the queue as well, or is it just HTTP requests? Right now, those metrics come from a middleware, so that would only be HTTP requests. I'm not entirely sure

57:18 what you would want to get from the CLI. Like, what what metrics we're expecting to get? Yeah. From from a queue, I guess, you would expect how long each queue job took and how often it ran. Yeah. Adding it for the queue makes sense. So Yeah. Yeah. CLI depends whether you've got parts of your system that invoke the CLI, I guess. But then yeah. Less common than other stuff. Yeah. Queue support would make sense. Alright. You can put middleware on that as well, can't you? Can you? Pretty sure that this middleware for queues, because you can there's the queue, like,

58:08 facade thing has a before and after, which you can hook into. So Yeah. I know I know you can hook into it. So yeah. Maybe I'll put that on the feature list after I fix the bug of Put that after fixing. Yeah. So what are these bold things for you to put in then, David? Sorry. Yeah. I forgot a step. So when you configure the custom metrics, it doesn't actually consume all of your metrics within Prometheus. What you actually have to do is tell it which metrics to pull out Prometheus and then to expose them

58:37 in a way that can be used in auto scaling. So what we're actually saying is the series queries that we want to use, Let's just pull out all of the HTTP request total. Probably don't even need that. I probably get away with just that. We can take off this and then we make this available as a name. So we're just gonna say that we want to work this out as a rate per second. So we do a sum and a rate over the series, etcetera. Right. So if you had, for example, request per second, request total,

59:12 a few other things you'd need to expose multiple different series queries. Well, yeah. You would just put in your Prometheus. You can match on the labels. I mean, we could I don't know if we have these labels Kubernetes namespace. But we can just see. Yeah, we do. Okay. Namespace and pod name. So, those are available. We can query on that. That means that the metrics server will pass in. And then we actually don't want to query on the request total. That's actually saying that we want to use that value, but what we actually want to

59:42 do is calculate the rate of change over a two minute interval. So, we're gonna calculate how many requests are made every two minutes and then sum them together across the label matchers, and that gives us a total value of how many requests we get every two minutes, which means we can say if we get more than 10 requests in a two minute period, we wanna scale it or we can do a thousand requests in a two minute period. And that's actually gonna work out really well because we can control those number of requests, so we can we can actually

1:00:09 force that scale up on that. That makes sense? Yeah. That metric that metric's query, is that a Prometheus, like, query language, I think, that's So this here is a Prometheus query. So it might still put forward in. Let me just show you in the Prometheus side of things. I think that'll help, hopefully, cement it. So although we don't really have requests coming in, so let me get one more tab. Let me get some requests coming in with siege again. Okay. So we can use HTTP request total. We had go, and then this tells us the request

1:00:48 that we have. We can see we actually have multiple series here. Why is that? Different routes. Yes. Okay. Thank you. And one's on post and one is on slash. So what we can say here is okay. We actually want, you know, this is gonna claim what we have 41 and it's finished, hasn't it? So we got 21, 40 one, 50 five. So that's gonna continue to claim, but that's not what we want to scale on. Like, we don't actually care about the big number. We scale about the value over time. So we can do a rate

1:01:27 like so. Oh, what did I get? Oh, it might be worth mentioning actually. The numbers that you're mentioning go, like, the 55 and stuff. I think our faces are covering because there's no visit on the stream. Are they? Oh, yeah. Just to throw that curveball at you. Sorry. Oh, I'm doing it right now. Okay. Let's go back to the start. So request total gives me how many requests were made on each endpoint and we can see the 61 there. I couldn't put this off for a minute. On. Bye bye. Bye bye. Okay. That's better anyway. I'm sure everyone will agree.

1:02:14 So this is just the request total on each endpoint. We can see we have 21 requests on the homepage, 61 requests on posts. Now because that is a eternally incrementing number, at least for the lifetime of a pod, is that you don't want to scale on 61 or you don't want to scale on 7,000 or 80,000. What we actually care about is the change across the interval. So that's where the rate function comes in, where we can say, okay, let's work out the rate of change on a five. We don't have a scrape at the full five

1:02:48 of 30 seconds. We can see zero. Well, that's right. We're actually probably not pulling in any metrics right now. So let's do 200 requests. And hopefully if we get this enough time for it to first scrape multiple intervals, which is ten seconds or twenty seconds, then we should be able see a rate of change over thirty seconds as a positive value. There we go. So we can see that we had 1.4 revive requests every thirty seconds. Now it's a four requests every thirty seconds and that will probably continue to change for as long as that siege command runs.

1:03:22 Because we have multiple endpoints, multiple pods, etcetera, we can also wrap that in a sum to get a single value, which tells us the number of requests across the thirty second window. And we get to say that that triggers enough, enough anything to scale. Does that all make sense? Yeah. That does make sense. Yeah. Sure. Now, of course, we're using request tool right now because the buckets don't quite add up to what we want. Let's see what we've got. You know, we probably could get away with using that. We can talk about that in a second though.

1:03:56 But right now we can say, right, we know if we send 200 requests to this, we're gonna hit roughly five before it starts dropping down. So we can actually say, well, if we get more than two requests on a thirty second interval, that is enough for us to want to be able to scale the number of pods up. Cool. Okay. So we're happy with this. We've seen this clearly run on the Prometheus side. Now what we can do is apply the, redeploy the metrics adapter with that value that has our custom rule. We should be able to query that using

1:04:32 the raw API again. It says hoping and we do do a rename on this actually. So what do we rename it to? Request per second. Yeah. Issue to be request per second. Request per second. You hit me so much computer. JQ grep dash I HTTP. It is there. Okay. What did I get wrong? Was this just not there yet? Too impatient. Okay. So now we have best custom metric available for the HPA to use. The last step. Not that easy scaling horizontally in Kubernetes, is it? That will lead me into something I'd like to talk about next and we'll maybe do

1:05:25 Adding Our Horizontal Pod AutoScaler (HPA)

1:05:37 an episode on that down the line, but for now it's okay. And that's the project. This simplifies scaling by adding more custom resources that you can use, but that's not for now. Okay. So now we look at our HPA. We have this value and no we don't. We have HTTP requests per second and we know we can definitely trigger a five. So I'm gonna say if we had a three, we'll say 2.5 is good. So if it goes above that, it should trigger a scaling event. And when it falls below that, it should scale it

1:06:21 back down. Does that make sense as well? Yeah. Is there some kind of, like, cool down period where, like, it doesn't so if it's hovering around 2.52 to three, it won't just, like, create, delete, create, delete, create, delete, or does it just do that and it's fine? Yes. There is ways to configure that. So what's the best way to check that? Because I can never remember. But I do have all If memory that is configured on the cluster level. Right? I don't think like, if you run-in a public cloud, you I don't think you have access to

1:06:39 Discussion: HPA Scaling Logic & Cooldown

1:07:02 change that. There are defaults for it, which is, like, three minutes to scroll down or something. I don't remember the numbers off the top of my head. Yeah. Do well, just to prevent people from hammering their systems, I guess. I was hoping cube control explain would show me it, but it only has the standard spec and not the custom spec. Oh, that's because that's all scaling v one. V2 beta one. Okay. Auto scaling V2 beta one. Is that gonna work? I have no idea. There must be a way to provide dev API version. Okay. So if we specify the

1:07:46 version on the right one, what we want is auto scaling b two beta one. We can now see that we have this. I I don't know. Maybe that isn't configurable then. I assume it's not very much of an issue because Kubernetes handles the scaling side of it for you anyway. So Yeah. It's definitely something that that KADA offers. I thought you could do it through custom metrics, but I guess I'm wrong. Anyway, let's let's let's see at scale. Right? That's why we're here. So we have a single Laravel thingy running here. What I'm gonna do is I'm gonna just

1:08:18 Demonstrating HPA Scaling with Load (Running Siege)

1:08:25 pop open some new split so that we can run a watch command. I recently changed this alias from g h, which I know and understand to g h b because I wanna use the g h c l I, which is g h. I know that's completely irrelevant, but it's been frustrating the hell at me all week. Okay. And it's only Tuesday. Okay. So we can continue to scrape. I reapplied you. Is that old? Old or new? I think they're old. Age, twelve minutes, three minutes. Yeah. Hopefully, those are just old. That's not even the name of the

1:09:05 metric I use anymore. So for a second. Yeah, I think we're good. So we can tell you to describe that. We should see the scale and events happen. Now what we wanna do is run siege again. And we also want I'm gonna do one more split as we wanna track the pods as they change over time. So we're on a watch on that. We're gonna run siege here and I'm going to describe this every couple of seconds and see if we can see the scaling in real time. Assuming I've not fucked any of that up.

1:09:39 By the way, the default downscale window is five minutes. It's in the documentation about horizontal port auto scaling under support for cool down and delay. It's an argument you it's a flag on the cube controller manager, so you do need to have control plane access to change that. And you can only specify the duration of what they call the downscale stabilization time window. So I think upscaling is instant, but it will wait five minutes before scaling down. Okay. Yeah. That makes sense. So it's it's, the public cloud, like, manager will be what handles that unless you

1:10:21 roll your own Kubernetes clusters. Yeah. I am not entirely sure off the top of my head. Some providers might allow you to pass arguments to the different control plane components, and that would be how you'd configure it in that case. Yeah. But that's per provider, so it's down to who we use, I guess. Okay. Fair enough. But I assume that whoever you use will have whatever they consider reasonable defaults, so it should be fine. Okay. I think our c is finished before we try and use trigger that. I'm running it again. We can see we're about 17

1:10:58 Observing Pod Scale Up

1:11:00 here now. We oh, good time in there. We have a week. As soon as I ran that describe, we got a rescale event based on our that value going above the 2.5 that we are trying to hit. We can see that it's worked out that we probably won another four. Unlucky for it, it's gonna continue to keep scaling up, but still, it's added four new instances. And if we run and get pods, we'll see we now have all the Laravels run. Oh, wait. That's the completed ones. All the Laravel projects running. I'm gonna quit siege.

1:11:35 We'll run a watch on this again and we should see within about a minute or two. Well, it's gonna be five minutes based on what you just told us. That'll scale then. So you've got five minutes of dead air to fill. Go. Now the first thing I wanna say there is it's five minutes by default. We are now assuming that Docker desktop actually relies on that default. I have no idea. We we will we will find out if it's five minutes. Well, we're gonna finish up. I can't believe that has been an hour and a half

1:11:41 Observing Pod Scale Down & Conclusion

1:12:07 almost already. That has So, Benko, what's that doing there now that started new ones, has it? Well, the value, I think, on aggregate over time is still above the 2.5. It's just fallen, but you'll probably find it's actually scaling more up. And if we describe the HPA, it has done another rescale. Oh, no. That's the first rescale. I think maybe some pods just died. I saw that we're terminating there. Yeah. Then we get pods, grab, Laravel. Yeah. We still got the five. It just looks like we lost a few for whatever reason. Oh, I know why that's gonna be. The

1:12:52 probes on it are gonna be failing because we've not got any metrics, probably. It's probably given a client error or something. Oh, because there's no actual requests hitting anymore. Yeah. Maybe. I mean, that's complete speculation at this point in time, but it's possible. Alright. I mean, I think we've we've shown everything we want to show. I think it's great that Leo has, you know, packed up this project and providing people with a working almost. It will be working. I I'm debugging it. I'm I'm not gonna delay the stream with the debugging. Instead, I'll just make it work, and then,

1:13:31 you know, by the time someone rewatches this in a week, it's just magically gonna work when they pull the latest tag. Yeah. No. I think you're doing great work. I'm really glad to see that there's Prometheus auto wiring super Laravel component that has exposed in the metrics that we need. And I think it's it's still enough there. You can still calculate the number of requests over a given interval and scale just as we have done today. And, of course, I would actually like to monitor and keep my plus five plus infinity below 10% or whatever, but

1:14:04 that could be something we do in the future. Maybe I can help contribute to the library that you're working on too. Then we got the metrics, we hooked up Prometheus, we added the annotations to the deployment, we scraped the metrics, we deployed the metrics server, we deployed the metrics, the Prometheus adapter, and we deployed the rules that expose the custom metric, configure in the HPA, and that's all it took to get all scaling on Kubernetes for Yeah. You you make it sound so easy. I think it would be awesome to take a look at the Keter project and then a

1:14:37 future stream. If you both wanna join me for that, it's not something I'm overly familiar with. I've used it. I like that. I haven't got it in production, but maybe a little bit of an exploration episode where we see if we can simplify this at all and take another look at the configure of the buckets and stuff. I think there's a lot more we can do here to make this easier for people, so I'm excited to to do that in future episodes. And for that. Sounds good. Awesome. Well, thank you both for joining me today, working through

1:15:05 that. I hope the people watching got something of value from at this time. I know last week was very frustrating for the people that watched two hours only to be like, what the fuck was that? Hopefully, it was fixed at this time, and that library was very helpful. Is there anything either of you would like to say before we wrap up for today? The library will work correctly eventually. That's all I have to say. And it's open source, accepting contributions. Go check it out. Try it out. Let's help, Leo, by working out where the cracks

1:15:34 are and failing issues. Accepting pull requests. Alright. You too have a great day. I'll speak to you both soon, and thanks everyone for watching. I'll see you later. Thanks. Bye. Thanks a lot. See you.