1:19 Introduction

1:19 Hello, and welcome to today's episode. I am joined by Dmitry Kalining from VMware. He is the maintainer of Carvel, an open source tool chain for deploying to Kubernetes and making our lives easier. Hello, Dimitri. How are you? Hey, David. Thanks for having me. I'm pretty good. Awesome. I I'm looking forward to this. Deploying to Kubernetes is is far from an enjoyable experience depending on the tools that we're using, and I'm I'm hoping that you're gonna show me some stuff today that's that's gonna change my life for the better. Right? Hopefully. Do you want to just then spend the

1:54 next minute? I mean, feel free to tell us about Carvel and what problem it's trying to solve and the different components that make it up. Sure. Sure. Carvel started about a year ago. It was actually known as k fourteen s before. We recently renamed it to Carvel. It it started out as a few projects that I was working on trying to improve what I saw as not a necessarily great experience deploying and managing configuration for Kubernetes. I think at the time, I was trying to deploy Knative to my cluster. And, you know, at the time, I think Knative

2:02 What Problem Does Carvel Solve?

2:38 had a hard dependency on Istio. And so I also had to install Istio, and then, you know, Knative had about a hundred different Kubernetes resources. So as you update throughout, you know, versions, it was kinda hard to track what should be deleted, what should be created, updated, etcetera. Right? Does some of that stuff, but I just didn't quite feel like I had, you know, fine grained control over that. And then, you know, as I was kind of playing around and trying to deploy things to Kubernetes, I've also dealt with a, you know, several helm charts,

3:15 and I always found it to be quite cumbersome to deal with textual templating. So I ended up kind of digging in into that area as well. And so the first tool that came out of all that was YTT. This was the the the structural templating tool for YAML. The second tool, I believe, came out was CAPP, or I guess some people call it CAPP. That's for deploying configuration to Kubernetes and kind of managing it over time. And then there's actually a few few other tools. They're probably less known than these two. K build is probably the next one that

3:58 folks hear about. It kind of helps you integrate your Docker building into this Kubernetes deployment workflow. So, yeah, that was that was kind of a small history of of Carvel. Yeah. So when we when I said tooltune earlier, I mean, it's definitely something instead of tools that are trying to kind of improve the developer experience across the entire lifetime of a Kubernetes application, It's kind of the vibe I was getting from all the different moving parts and stuff like that. Yeah. Well, one one of the key points that I was trying to to explore is how do you actually have

4:37 this separate tools that do these distinct things like deployment versus configuration management? How do you keep the how do you keep them separate but yet have them play very well together? So kind of following the Unix philosophy. Yeah. That makes sense. Total sense. So let's just pull up. I have a website shared here. And what we can see is that you kinda broke this down into two sets of tools. So we got, I guess, what are the core tools on the top, and then you've got some more experimental stuff, I guess, that you're working on on the bottom.

4:54 Core Carvel Tools Overview

5:13 That's right. So this is actually behind Carvel. There's now a team at VMware. And after various deliberations and whatnot, we decided to kind of separate this high level, quite mature tools from something that we're experimenting or, you know, messing around with. Yeah. That makes sense. So today, we're gonna get hands on with these tools, which I'm very happy about. But before we do that, I guess, would it be fair to say that these tools directly compete with customized and helm and just write in straight up YAML? Like, this is this is a tool chain to replace

5:50 all of that from people's lives. That's right. Yep. Yep. Okay. I'm definitely excited then. One of my biggest think I ran out of it. That's probably on a weekly basis on Twitter, But just using Go templates inside of Helm charts and fighting with wrangling YAML value files to me is just is is quite cumbersome. And I'm hoping that there are some of the tools that we look at today can make that a lot better. Do you use Go templates language? Is that a thing? You mean outside of ITT? Or occasionally, you know, still have to deal with

6:26 various configurations here and there. Yeah. I I I should probably just bake the bullet and start learning it and get better at it instead of complaining about it. But then what would I what would I run about on Twitter otherwise? So That's why I gotta have some material. Alright. So step one of every every episode that I do here is how do I install the tools? What is the the fact to best definitive way for me to install the Carvel toolchain? You can scroll down a little bit down to the page, and there's a a few installation methods there.

6:42 Installation

7:02 Since these are separate tools, you know, you can either use a script to install them all together, or you can select individual tools from their GitHub, download releases pages, or you can, you know, use Brew, for example, if you're on Mac. Or actually, Brew works on Linux, I found out. So some someone someone wanted to use Brew on Linux. So it's quite caught off guard there. Yeah. I mean, I'm I've been I joined I started a new possession recently with Packet, and I got a Mac as part of that role. And I'm still fighting against it.

7:38 I like, I am a Linux user and trying to get myself accustomed to brew as package manager. I mean, I come from Arch Linux. I've got the AUR there, and it's just everything in the world is packaged. Yep. So I'm no stranger to Carvel Bash either, but I will play it safe with the brew tap. At least then I'll get some more chats. Hey. It's actually as you can see, the the brew tap is still called k 14 s. We've been slowly updating the names. Recently renamed the Slack channel in our Kubernetes, I guess, in the Kubernetes Slack from k 14 s

8:14 to Carvel. So what what inspired that name change then? Inspired might be a wrong name for that, but within VMware, there is certain naming guidelines. And so this was one of the one of the things that we wanted to comply with and be friendly to marketing and other folks within the organization. Yeah. The the k fourteen s name was just something that, me and my friend picked as a short thing to type. But, you know, it was never meant to be a kind of a top level marketing name. Well, if the Internet is to be

8:20 Why did it change name from k14s to Carvel?

8:57 believed, the k fourteen s is Kubernetes tools. That is right. We actually counted those letters, but then somebody told us that you're not supposed to count the space between Kubernetes and tools. So at that point, we just said, you know what? K k 13 s would be unlucky. So we went with k 14 s. I like it. Alright. So now that my brew is finished, this should mean, in theory, if I could access the YTT. Yeah. K b l d. Oh, k. Excellent. I guess now and it would help if for once in LA if I remember to start Docker before,

9:42 you know, the the call. But we'll give that thirty seconds to get healthy, and I should have a working Kubernetes cluster. What what's the what's the choice over here for how you run Kubernetes? Do you typically use a built in Docker built in Kubernetes and Docker or something else? So I keep changing. I've been playing a lot lately. I've been battling with Kines running inside of Docker for Mac, which works pretty well. I'm also I really like MiniKube. I just like the direction that they're going in, and I've been trying to use that with the new VMware fusion 12.

10:17 But Oh. Because it got the new player release, which is like a week old or whatever. Mhmm. But it's free for personal use. So now I can use that for hopefully a slightly better experience. Yeah. I'm a big fan of Minikube as well just for the sake that it comes with a Docker instance that you can, you know, directly build and it's available to the cluster. Well, yeah, I have run into a few problems running kind inside Docker Docker for Mac. So edge cases usually, but they quite prominent. Alright. So we've got the tools installed. I

10:49 am assuming now I have a cluster I could spell. That is my Docker from Mac cluster because I see VPN kit and I've got oh, I've got some cluster API stuff running, but we'll just ignore that for now. So when we were chatting earlier, we're talking about the direction that we wanted to play with these tools. Now we could have jumped straight into YTT, which is the the templating side of the YAML aspect of it. But you were also saying that it may actually make more sense of trying to play with key app first to deploy a workload to our cluster.

11:20 Hands on with kapp

11:28 So That's right. I think let's go with that. How how do I use KeyApp, and how do I deploy something to my cluster? Well, I guess one way to to look at CAP is that it's meant to track the resources that you're deploying to the cluster. And the way it tracks it is by applying a label, and then it associates a name to it. So you should be able to take any plain YAML out there, maybe some kind of a Kubernetes example of sorts somewhere, and just use the deploy command under k app to to deploy it.

11:39 Deploying with Kapp

12:07 Okay. So one other thing, I maintain some helm charts for influx DB. I did template out some YAML, which we can use with. But for the sake of the key app stuff, what I'm gonna say is let's just keep it really simple for now by NGINX with the most basic configuration I can think of. That will be okay for the demonstration purposes. Yeah? Should be. Should be. So what you said there was I can use key app, and it has some sort of deploy sub command. Yeah. Yep. And does that just work like my like,

12:45 the role that It's gonna be yeah. It's gonna be dash f. And but in addition to dash f, you'll have to throw in dash a, which stands for the name of the app. I have to name my app. So I can just call that NGINX? Sounds good. Yep. There it is. So this is telling me that it has to make some changes to my cluster. So already, this is doing some sort of, I don't know, preemptive reconciliation. It's looking at what's in my cluster, looking at those labels you mentioned, and trying to trying to work

13:06 Kapp's Deployment Planning & Applying

13:18 out what changes are actually gonna happen. Right. So it's so it you know, I I believe most folks are familiar with Terraform. Right? And within Terraform, there's this idea of planning and then applying the change set that it planned. Right? And so we've done same exact thing in CAP where we first do a planning session of what what's going to happen. And as you can see over here, we have a single resource, and the operation of it is going to be create because CAP didn't find that resource in the cluster. And then it just presented you that plan.

13:52 So you can go ahead and apply the plan. And CAP promises you that it's going to do only what's in the plan. Right? So it's not gonna go off and, like, change something else. So you can be a % certain whatever is shown to you. That's what's gonna happen. Alright. So I can see that it's created my deployment as then waiting for changes. So it's actually observing the replica sets in the pods being created? That's right. Yep. We wanted to make it a little bit more friendly to finding out what's actually happening in the cluster. A lot of times, right, you

14:29 just deploy a bunch of stuff and you have no idea when it's done. So CAP has some effect to logic for checking out various resources and you know? So this this case, I think it's deployment. Right? So it knows how to wait for a deployment based on the deployment status. Okay. And, hopefully, it's useful information to to know. You know, it looks like the container was pending to be created, and then eventually they're created. So, I mean, the first thing that's jumping right into my head now is that what if I break this as what does Keyapt do? Like, let's say

14:58 Observing Deployment State & Errors

15:07 I come in here and and let's put those resource constraints back in. Right? And if I set them far too small to the point where NGINX won't be able to start, does that trigger anything with CAP? It might. It depends what the what the resource, I guess, is bubbling up to its status. Right? So it depends if the deployment is showing that kind of information. CAP may also sometimes look it into the associated resources, like the pods that are related to the deployment and raise some of their information, you know, bubble up that to the user.

15:43 We actually recently I think I had a what was it? I don't remember. There was there was actually a request in improving some of this logic to show this other more rarely seen condition pop up. But it it depends. I mean, we can check it out and see what happens. Okay. So this is our first update. It's now Yep. Actually created the labels, and it's telling me You can you can even do maybe if you control c out of this and then run the same command but with dash c, you'll you'll see some, hopefully, diff over here

16:23 that shows you what's actually being changed. That's nice. I like So that diff is per resource. Okay. I like that flag. Like, I actually see in the the depth essentially before I have to apply. It's pretty nice. Yep. So it looks like NGINX is pretty happy. Well, I saw I was I'm attached to NGINX. In terms of it just didn't care. So You may you may have to really get this. I'm not even sure if that's gonna yeah. What if we pick a tag that doesn't exist? Will that trigger some sort of failures? Yeah. I think so. I mean, there should

17:01 be somewhere somewhere image pull. Yeah. We should get an image from Microsoft. So we can see that if again, which I I really like that feature. That's pretty nice. And then we go. I see a four zero four. Yeah. So ending list. Yep. So by by default, capital eight until so this is kinda interesting. Right? Because there's no you know, Kubernetes is continuously reconsidering. Right? Because maybe the the the engine x 44 is gonna show up in the next five minutes. Right? Who knows? Right? But various resources, right, have defined their own rules, like deployment, I think, as a a condition

17:43 that indicates that the deployment isn't progressing after some time. Right? And you can actually tweak it, I think, on the deployment. So CAP follows those kinda rules just by observing the status, but it also has a global time out for fifteen minutes, I think. So if we see the air for fifteen minutes, CAP will tell you that, you know, nothing is really progressing. It's probably time to give up. Obviously, that's configurable, but best case scenario is that you wanna be actually changing you wanna be changing the your resources to have this tight feedback loop. Right? The most common might be, like, adding

18:21 a readiness probe to your pod, right, such that they don't just immediately become available. They, you know, they they take their time to start up properly. Okay. Well yeah. Let's not watch it for fifteen minutes. So I just reverted reverted that image change, apply that to my cluster. I've now got I've got it just kinda saying, yeah, we're good. If I run that again, it's just saying, hey. There's nothing to do. It's in the walk, and we're good. Yeah. Cool cool other command actually is cap and stacked. If you give it a dash a nginx,

18:51 Kapp Inspect & Resource Management

18:55 you should be able to see, you know, what's what's part of this app. We might even see the old pod. Not the old pod is already gone. But you can actually see that Kubernetes actually keeps few replica sets. Right? And I think that depends on, like, the history parameter of a deployment or something like that. I guess it's for fast rollback or something like that. Yeah. This kinda gives you a maybe a full view of what's what's cap seeing under this app. Oh, yeah. I guess we've got a bit of a contrived deployment right now, a very

19:30 basic engine x. There's not a lot going on, but I can see how that inspect passing in the app name and getting back that list of resources that have been deployed together. I guess, in a real world or production like scenario where I'm deploying a real microservice, which maybe has a back end database, has contact maps and secrets, like, this inspect view is actually gonna be much more useful in that scenario. It's definitely I've I've actually learned quite a bit after digging in on what kind of various resources get created. Like, you know, a lot of people know

20:04 about, like, kubectl get all, but get all actually doesn't show you everything. Right? It only shows you the things that are tagged as all or whatever, I guess, the proper term is not tagged probably. But there's there's quite a lot other, you know, sets of resources. And so inspect actually is quite quite interesting to look at. Yeah. Awesome. So is there anything else within KeyApp that we should take a look at? Well, it it kind of the way to put it, there there's a lot of kind of opinions that I've baked into it working with area

20:30 Advanced Kapp Features (Annotations & Ordering)

20:45 various other systems before Kubernetes. I I've been, I guess, in the deployment and configuration game for some time, and so Cap tries to, I guess, bake in a lot of interesting features that are helpful. For example, one of them is a lot of people run into a challenge when a resource needs to change, but Kubernetes doesn't allow you to change that resource because, I don't know, it's the field is immutable or something like that. Right? And so from a kind of a declarative perspective, right, how do you actually deal with that? Right? Nobody wants to be sitting there and doing all

21:25 kinds of manual changes to your cluster. Right? So cap, for example, has special annotations to control various behavior of what it what it can do. So it it can say, for example, if you see that the resource is invalid and you've marked off explicitly that resource this resource is okay to recreate for updates, then it will automatically do this kinda work for you. So, yeah, that is you know, I guess as people start using CAP in their environments, they they start, you know, asking interesting question. Like, how do I order, for example, running a DB migration job before actually updating

22:07 the deployment, things like that? And so cap, for example, as a feature for ordering the the changes to be applied to a cluster. So, yeah, there there's there's a few interesting sneaky, I guess, maybe features that that are not that you don't have to use, but you can kind of opt into. Okay. So let's talk about this with a little bit of a wider scope now then. Do you see CAP as a tool that people would use to deploy normal Kubernetes manifest like we were doing? Or you you probably see it in a way that

22:45 people should be having YTT manifests with cap deploying them as a resource. Does that make sense? Or have I got that wrong? So the beauty of, I guess, embracing this you know, solving these problems in, you know, in multiple tool, right, is that really it's up to the user to decide what kind of complexity they need to tackle in their workflow. Right? Some users, of course, wanna be able to have configuration that could be applied to multiple environments. You know, it might be slightly different in production. They may decide to use customizer. Maybe they decide to use YTT or maybe they're using

23:23 Helm charts or whatever it is. Right? And so the way I look at it, actually, I've I've explicitly said that it's it's not, I guess, it's not something that we wanna constrain. Like, this this tool is only for deploying, I don't know, plain YAML or something. It's it's really just tool deploying Kubernetes resources. And if if you happen to generate them from YTT, great. If you happen to generate them through a customized or Helm template or, you know, I don't know, Pew or whatever else. Right? Maybe even some JSON net tooling, whatever. Yeah. We we should be happy with that.

24:07 But, yeah, I I do see it as, you know, I do see it as a tool that the regular app developer can use when they're deploying their own software that they've written. Or they could be using it to deploy something that the third party have, you know, provided them, or it's something that maybe some of the open source, you know, projects like strip manager or whatever else is going on. Right? So I see it as kind of a scaling with your needs. Okay. So that leads me on to two different things then. I think just because

24:41 I'm a particularly curious person, what I'm gonna do is run this deploy because I wanna see what happens when I point to add those manifests I generated earlier. I just wanna see that preview with slightly more going going on. Okay. So I can see the the r back, the cluster rules, persistent volumes, and services. Cool. I really do like that feature. That's, you know, that's particularly useful. Yeah. One one interesting thing actually that I look at this diff is, you know, as you can see there, there's a few cluster resources versus there's a few resources that go in

25:00 Introduction to kapp-controller

25:20 the namespace. Their cap cap doesn't really make a difference between a resource that's namespaced versus a cluster level one. You know, it doesn't really care if it's a CRD or not. It'll just be managing it in the same exact way. So there there's that benefit to just having a, you know, same behavior applied to all of these resources. Alright. Cool. So one more thing on CAP then. When we were on the website, there was a controller for CAP. Do you wanna That's right. Put a bit of insight in that? Is that something that runs in the cluster

25:49 Kapp Controller (GitOps Style)

25:59 at a GetOps fashion or something else? Yep. Yep. So the idea there is if you There we go. If you think about what most people are doing, right, there's some kind of a fetching of configuration going on. Right? There is some kind of a mangling, like templating or, I don't know, overlaying or whatever going on. And, ultimately, there's, you know, the last step deploy. Right? And so we've we've decided to codify that workflow in a CRD. And so you get to specify a app CR. Actually, if you scroll up here, there's the examples folder over here. Yep. There might be

26:42 some some of them might be a little like, maybe Concourse Helm or something or NGINX Helm. Yeah. Yeah. NGINX Helm. Let's take a look at that. So, yeah, AppCR effectively split that up into these three sections, fetch template and then deploy. Deploy, obviously, is happening with CAP. Fetching and templating are actually they're decoupled steps, and there's various strategies like, you know, fetch it from Git, fetch it from your Docker registry. So we we also are believers of putting configuration into Docker images. That way, all your assets are together. Or or maybe it's fetch a Helm chart.

27:25 Right? And then the templating step could be also separate. So for example, in this case, we're we're using Helm template to template the the Bitnami NGINX, but we're also using YTT on top of that to change how the the service is configured for NGINX. And then, ultimately, we do a deploy at the end of that. So it it kind of it's a it's a opinionated way to do, you know, your your deploys to the cluster. And because it's a controller, it keeps on doing them. Right? So you can decide whether you wanna float kind of on

28:01 the latest reference or where you wanna look it down to a particular thing. So, yeah, that that's a it's a controller that's meant to represent the that 95% case. K. Cool. Let's pop back. Actually, may maybe one thing to note is more after after kind of building up the controller, I figured, well, you know, you can use all these tools locally yourself, right, to kind of replicate that workflow if you wanna debug, for example, if you just wanna iterate on it quickly. But I was also thinking making a CLI that's effectively the same thing that the cap

28:40 controller does. Right? Maybe literally even the same code. Right? But just happens to run locally where you can have a maybe, like, run it once kind of a thing. So kind of a an interesting idea of making it more, I guess, localized when you're developing stuff? Yes. There is one of my colleagues, Jason. Sorry. He's actually working on a proof of concept right now of a local binary that emulates Kubernetes API server without having any API server. So I'll drop you that link. It may be something interesting for you. Yeah. I think I've seen it actually on Twitter. This this in

29:21 in our case, I think we actually don't need Kubernetes. We just need that inner loop that does the reconciliation, so it wouldn't have to fetch any resources. But I I I see what you're getting at. I think there's a there's a lot of things that could it well, I mean, we want the Kubernetes API because it it's nice and declarative, but we don't necessarily need to cluster. We just need a daemon that could run locally or be run once. I think there's a a good market for applications there. Alright. Let's let's move on over then to

29:46 Introduction to YTT (Templating & Overlaying)

29:49 this templating thing. We've got YTT here. So I understand key app now. That makes perfect sense to me. I like the functionality that that brings to the table. The key app controller, think, is really sweet. I'm looking forward to playing with that at some point where I can, you know, have that running inside my cluster. Now I wanna get rid of of Helm or at least the template a bit or the customized a bit. And this is where my ITT slots right in. Yeah. Yep. That's right. Templating and overlaying. That's the that's the goal for this tool.

30:00 Hands on with ytt

30:18 And and templating, we're we're talking about the exact same thing that Helm does. And when we talk about overlays, we're talking about modifying or tweaking those generated templates per environment. Right? Yeah. So it's I've over the years, I guess, I've come to a conclusion. I don't know if people agree with that conclusion or not, but I I think you you have to use templating and overlaying together to to cover the use cases that you have. Right? Because we've seen that if you don't have the overlaying, then a lot of people as Helm chart, let's say, gets more mature. A lot of

30:57 people keep on asking, like, well, let's add one more configuration now between values. Right? And so you end up with a thousand values there. And, you know, those values are probably used by like, each one is used by one person only. And so, really, I think, you know, 95% cases that are common between people should be something like, you know, values, should be just templated by the author. There's no, you know, crazy direction that they need to deal with. It's just right there inside inside the template. But there's, let's say, a 5% case where an end user

31:32 wants to actually, you know, modify something. Right? There's no way that they have this data values because it's not a common thing to modify. Maybe it's very specific to their cluster. Maybe it's specific to their environment, whatever it is. And so having a single tool that has the consistent kind of a methodology for both of those approaches, in my mind, I think, is is actually advantageous. So, yeah, YTT tries to do both. Okay. Excellent. I mean, that thing you just mentioned there about adding one new thing to the values dot YAML, and it's probably consumed by one person.

32:10 It's definitely a problem that I face with with the influx DB in the Telegraph helm charts, particularly Telegraph. Because every time Influx add a new plug in to Telegraph, someone comes along with a pull request, adds a new values format to enable that plug in for their deployment, and there's over 200 plug ins. And you end up in a really unreally hellhole, you know, I'm just gonna call it a hellhole, where you have to try and convert YAML to TOML for the Telegraph configuration. Mhmm. And it's nothing but painful. So let's see what we can do here then.

32:39 Now what's the best way for us to kinda play with this? Did we just start writing some So there is a there is a playground here that's interactive. It just kinda gives you a feel for what the tool does. Actually, quite a few people have have really liked the playground. So that was a good call on my friend's part on making that a priority. That's. He works now at, I think, AWS. But, yeah, the the playground is a good way to start, kind of learn. There's a few examples there if you if you scroll up just a little bit.

32:41 YTT Basics: Plain YAML & Functions

33:21 Yeah. If you you think that you wanna kind of take maybe some YAML and try to make it maybe template a little bit. That could be a good start. Okay. So let me let me throw a question at you first then. It's like assuming I am an application developer that's already deploying stuff to Kubernetes. So it's just basic YAML manifest right now. Is the approach to adopt to my ITT to incrementally swap bets in and out of that or to just rewrite everything as YTT? It's a great question. Yeah. One one of the one of the

34:00 interesting side effects of being a templating tool that really deals with structures, right, and more specifically, YAML, is that really a a YAML file is a valid YTT template. It's you know, YTT just reads it in and spits out the same exact thing. Right? There was no, let's say, variable substitutions or anything like that. The the simplest thing, you know, to kind of level up, let's say, there is some kind of value that you wanna insert. Maybe it's different for two different environments. Right? The the simplest thing is just literally throw in, you know, one little usage of YTT within

34:40 that YAML. And, you know, it might be one line or two lines, and that that's how you really get started. So, yeah, the the intent is definitely not to change any of your YAML configuration. It's more just start out and just add some of these directives over time if if you need to. Yeah. Then what I would suggest is, I mean, right now, I could run y t t against this engine next dot YAML, and you're saying I'll just get the same YAML back. That that Should be. Yep. Yep. So digest the dash f again like this?

35:14 Yep. That's right. Yep. Okay. So let's then tackle the first one of my many annoyances with Kubernetes channel is the duplication between the selector and the labels. Is that something we can then template out with YTT so we only define it once? Yep. Absolutely. So YTT because it's dealing with structures. Right? And, you know, as we can see on line twelve and eight, right, we we have a single map item. Right? A map. Right? So we can pull that map out into a function and just call the function from two different places. And all that really requires is,

35:54 you know, cutting out the app NGINX and moving it up to the top level of the document and wrapping it in a in a depth. So you might this this probably referring yeah. Exactly. Referring to an example is a little easier to explain there or to show at least. The the key there is that YTT directives are all YAML comments, so they just leave along the the YAML nodes. And everything that's prefixed with, you know, pound at, it's it's annotation, what we call, is in YTT. And these annotations, in fact, could tell YTT where certain things are kind of a like,

36:38 this is a definition of a function or this is a four loop, let's say, or this is whatever it is. Right? So we're using a Pythonic language called Starlark from Google. Great language. Great library to use. And it's actually the YAML gets converted, you know, at runtime to to a template that's fully written in Starlark, and then Starlark gets executed, and you get the output back. So, yeah, if you grab, like, line three from from this example over here and just drop it in before. Yep. And then we also will close it. Yep. There you go. So now we have a

37:19 function. And all it takes now is to call that function from multiple places. And so to call it would be if you go to line 11. Actually, if you so if if you write it out right here yeah. So if you do pound at labels yeah. So what's what's happening here is actually it's not gonna quite work because we don't know if you mean that the value of much labels should be the result of the labels function. It could just be calling a function in the middle of the YAML. And so if you drop it on

37:52 the same line as much labels, then YTT is going to know that, hey. You're actually calling a function. And, also, you'll have to throw in the two parentheses at the end of that as a kind of a regular function call. Alright. Okay. Gotcha. Yep. Alright. Is there a way to keep my editor happy, or is that always just gonna moo and at me whenever Yeah. I'm not quite sure. Is this Versus Code? Yeah. Yeah. I'm not quite sure what it's complaining about right now. We we do have a Versus Code plugin that we've recently started out and, you know, we'll we'll probably

38:27 iterate on. Yeah. I'm not sure if what exactly it's not happy about, but oh, yeah. I guess it's value. So, I mean, if I drop this n, will will YTT then replace that value, or is it gonna YTT is actually not gonna be quite happy with providing the static value and the value that's gonna get generated to YTT and mostly because it it might be just confusing to users, like, what what what's happening over here, and so we wanted to prevent that. We may open that up later. Not quite sure. We're we're trying to be as strict as possible

39:04 today in terms of the rules, and so we can always loosen up. But, yeah, today, YTT is gonna complain at you saying that's not allowed. Okay. So let's let's do that. I I I like to see things work. So I can see those labels being injected. You're saying that's just gonna tell me no. Yeah. Yep. So it's saying it's best to jam on node to either have, like, computed or blah blah blah. Because it's got both. Right? Okay. That's right. Okay. So I will Yeah. And, of course, now that we have this function, right, we can

39:38 call as many times as you want. So, like, on line 14, you could also call it. Should be all happy. Right. Nice. Okay. That makes sense to me. So we're using Starlack language to define functions. We're going down the comment approach so that the YAML can still be read by other YAML files, I guess, or set back highlight works. The it this is kind of an interesting world. Right? Just because the template is YAML, you do get some benefits from other tools. Right? But for example, lot of tools, they strip out comments, right, because they're just not

40:20 aware that, hey. I should keep the comments, which over time, I guess, that may get fixed through improvements in YAML libraries, like the Go YAML v three. But, yeah, the main motivation for making the template in YAML itself was specifically for being able to read the YAML structure and then overlay the metadata on top of that. YTT Lint is actually one of the there was a there was a person from SAP. He was experimenting with how to do some type checking inside this kind of a world. Versus CodeYTT is from Eli. He that was his experiment and kind of

41:15 improving that. But, yeah, we've we've we've got some work to do there. Well, I mean, I've installed that that Lint one, and right away, my little red squiggle of death has gone away. So I'm pretty happy. Fair enough. Alright. Cool. So something that jumped out at me and the the playground example here. What is this load statement at the top? I couldn't really work that out right away. So in Helm, I believe you can just say, like, dot values dot something. Right? It's kinda like the values is globally available. We've tried in YTT templates to be very

41:55 explicit about where things come from. So for example, if you wanted to move a function into a separate file, right, the the the labels function that's been written, you would have to do an explicit load of that file to grab that function. Right? So explicit export explicit imports of of whatever variables or functions that you care about. Same applies to the built in YTT modules. So YTT comes with various modules like data one is to get the the data values that the user has provided to us. There's, you know, various ones like base 64 to encode or decode things. Maybe there's a,

42:35 for example, overlay functionality in YTT is actually implemented as a module so you can programmatically do some overlaying yourself. So, yeah, there there's various there's various modules that it includes. And so here, we're saying, please import the data module. And under data module, there's data dot values, which allows you to use what the user has provided to us. Okay. That makes sense to me. So the next thing then I think that makes sense for this example would be I'm thinking of the helm. What are we doing now? I so I would be able to define things globally that

43:05 Sharing Logic with YTT Libraries

43:17 could be applied in in multiple templates. Is there a way to do that with YCP? Yes. When when you say globally, do you mean like a same function or something like that? Or Oh, yes. So if I, you know, say this was across two files and I've got a service dot YAML I don't necessarily want to define the labels in each. Got it. Yep. Makes sense. Yep. So yeah. So at this point, you would say so you would have a service dot YAML that has that service, and we would create actually a separate file called

43:49 the maybe, I don't know, deployment dot lib dot yaml. And that's where we'll put our function so that we can import it from both places. Okay. So I can just let's I'll call it a b c dot yaml. That's alright. Yeah. One thing, you do have to call it dot lib dot yaml as the extension. So for now, we'll put that restriction to avoid kind of a people importing templates themselves from other templates. And so everything that's, you know, labeled as in dot YAML is importable by the regular templates. Okay. So I have my dot lab dot

44:24 YAML, and you're saying I can just Yep. This function over there? That's right. And then I can send that function from here first. Because right now, I've got my my y t t lens. It's kicked in and went, hey. This this doesn't exist. So Yep. So, yeah, you would use a pound at at the top, then space, and then type in load. And then you would open parenthesis. Maybe it's, like, double quotes at y t t. Or actually, no. No. At y t. Yeah. It's a b c dot a b yamo, I bet. Yep. And then the second argument would be

45:01 the name of the function. So it's gonna be a string as well. Yep. So that should make the labels available. Yeah. I'm not quite sure if YTT Lint has been I I think it's been used for a long or it's been experimented a long time ago, so it may not know certain things. But, yeah, the this this should be okay. Okay. So this based on this syntax here, if I had 10 different functions inside of my library, YAML, would they all need to import it individually? Is there a way to just bring in everything? So it's kinda interesting. The Starlark language defines

45:42 this so load is actually not a not a function, but it's a special, I guess, keyword or directive. And it does require you to specify all of the symbols that it imports. However, the the trick here is that if you're a b c lib yaml has lots of functions, you could actually create a little structure that holds them all. And all you gotta do is import the little structure over here. So it may be, like, structure a little a b c, and then you would say a b c dot labels. So it's actually kind of bundling it up

46:16 into a little object, and then you just import the the little object altogether. We actually consider adding a different way of doing a load to kind of be more closer to what, I believe, JavaScript folks do when they just assign a return of the load to a variable and just user of that. Might be might be a little bit more friendlier, I guess. So just to kind of elaborate on what you said there. I mean, were you suggesting that I would do something like this? No. No. So you would still keep the you would still keep the labels. So we

46:55 we are importing a symbol on line one, the label's name. So you wouldn't wanna be changing the YAML itself. So let let's just say that you have a second function over here. Maybe you can copy it and have two of them now here. Yeah. So now we've actually have two symbols, labels and labels too that you have to export. Right? And so instead of instead of the user on the other side importing them manually kinda one by one, you can create this thing called a struct. So it's effectively a better looking map, if you will. So I

47:29 don't remember if we have an example of that. But, actually, we might have an example in the YTT repo because I believe someone was asking about this. Might be in the docs somewhere. Let me see if I can find it real quick. Load. Oh, yeah. There it is. So I'll send you a link over here real quick. If you scroll down just a little bit, keep on scrolling down. So there you go. So it says right under your cursor, it says to load the cell functions from a single file, you can do effectively that. So

48:09 what what that allows you to do is just make a little object that has two fields, you know, label and label as one. So right now, it's little verbose, I would say. Definitely wanna improve on that, but that would be one way to go about it. Alright. Okay. So I guess random stuff just keeps popping up in my head now. Do I need to define this as YAML, or can I use dot star to then, you know, get some text highlighted and other things from my function? So it's an interesting kind of a choice to

48:43 make. Right? So when you're writing templates, you might be dealing with functions that return a bunch of structures. Right? To me, let's say, you're defining a structure, it's way easier to just, you know, maybe write it in YAML or copy paste it from somewhere that's already in YAML. Right? And so a lot of times, I will say, since you're dealing with YAML, I would just stick with Lib YAML. Right? Now if you're writing maybe functions that are more, I don't know, compute related, like, maybe you're, like, I don't know, multiplying things and adding things and maybe, I don't

49:13 know, appending strings or whatever it is. Right? You might actually not be using a lot of the YAML structures at all. Right? So at that point, I would say it makes sense to, let's say, it in the dot star file, and then, you know, it is the same exact kind of load directive from the other side. In this case, I guess, the example that's have it as is the star file where you just have, you know, returning two integers for whatever reason. But, yeah, it's really up to the to the end user to decide if they're dealing with YAML

49:41 and stick with LibyaML. If they're not, then star is also fine. And I'm assuming because this is, a I think it's a subset of Python. I would just return a a Python deduction right here, and that would return me something that looks like YAML? You you could you could definitely do that. Yep. So dictionaries will be converted to the regular YAML maps. Actually, since since since we mentioned this, there is another type of templates that YTT can deal with is textual templates since in you know, when you deal with Kubernetes, sometimes you have to actually do some textual templating,

50:13 which maybe, I don't know, for whatever reasons, is more appropriate for, you know, whatever you're doing. And so you could actually have a lead dot TXT, right, that has a function that's really doing some kind of a nice textual formatting. Right? And so that's kind of another another, I guess, consideration to make when writing those templates. Okay. There's a lot going on here. Like, I think there's a lot of options. But what you're saying there, quite a succinctly, was just stick to the LabiAML approach. Use use that until you need to break out for something

50:51 that's a bit more complicated or fits a different purpose. That's right. Okay. Cool. I will stick with the YAML. Okay. We're back. So let's we have we have kind of looked at the functions. We've imported the functions. I mean, that all works pretty self explanatory. What about now if I wanna tweak this NGINX deployment for a stage in our production deployment and their number of replicas is gonna change? So let's set replicas to 100 Mhmm. Which I don't I only wanna prod. So how do I go about overlaying this to make this tweakable? So if we do wanna use overlay,

51:03 YTT Overlays for Customization

51:30 we would actually have a separate file. It could be, like, overlay dot YAML or replicas dot YAML. Okay. Overlay dot YAML. Mhmm. And the tricky, right, is that the world of overlay, at least in my mind, is split into two kind of steps. Right? One step is to find what you're trying to overlay on top. And second step is what the operation you wanna do once you found it. Right? So for example, we wanna find a particular resource, a deployment in our example. Right? And we also wanna find spec dot replicas, and then then we wanna replace the value of it.

52:14 Right? So it might be a little easier to copy from one of the examples. Let's see if we got a overlay example. I believe we do. So, actually, let's see if we got it. There's overlay files. Yeah. Okay. Overlay files. Example 16 might be what you're looking for. So here, the overlay is, I guess, in there's actually two overlays here. One is an ops one YAML, and one is an ops two YAML. So we can probably grab the entire thing in ops one YAML and just modify it to our need. So on line one, right, we see that

52:55 we're loading in the overlay library. That's not always necessary, but in this case, because we're using an overlay module functionality on line three, that's that's why it's needed there. And then on line three, we're saying, let's find something that has metadata name example ingress one. Right? So in our case, we probably wanna be doing some other finding. Right? Like, what are we actually looking for? So I guess we could do metadata labels. You might have to wrap it in in another set of hash then for the app. Right? Yeah. So and then this was app engine x.

53:40 Yeah. Like, there's something like that. Right? That's right. In fact, we could've even used our function over here if we really wanted to, right, for the labels to to get the same label in here. A lot of times I've seen people actually do is use the kind and the name of the object to kind of identify it uniquely enough within their configuration, but this this will also do. So overlay subset just tries to find whatever is kind of a shared. Confirm So and then so yeah. So once we found this object so by default, we are we're going to or by default,

54:16 YTT overlay functionality expects that it's going to find one object. So in some in forever whatever reason, we don't have that one copy of that object that matches this subset, then it's going to tell us, hey. Hold on. Something is wrong. And then so instead of the whole metadata annotations removal, we'll we'll be just doing spec replicas. Right? Yep. That's right. That's all there is to it. Okay. So there's something I wanna confirm there. What I think you suggested was I could use the labels function here. That's right. Yep. And so I would just have to copy the import

54:55 Yep. Into my overlay. So did I mess that up? Yeah. I'm There's one more. Yep. There's one more missing there. There you go. K. So from the command line now, how do I piece this together? So there's an interesting property of YTT is that it doesn't allow any kind of so the the the execution environment is very self, you know, contained. Right? There's no way to access time. There's no way to access file system, no network, no nothing. Right? And so because we're doing all these loads, we actually have to give the files to YTT that we're trying

55:37 to load. So in this case, it would be like YTT dash f nginx dash f overlay dash f a b c lib yavl. Yep. Alright. So That that error you said was gonna happen if we didn't get a match. It says it it's a number of match notes to be one, but was zero. Ah, you know what? That is true. Yeah. The labels that we're actually trying to find is not inside the metadata. Right? It's elsewhere because we didn't we didn't have anything. Yeah. Well, let's just make that work. Right? Yeah. Why not? Don't I I guess it is better. Yeah.

56:17 This this is actually one of the kind of a one of the goes back to the idea of being as strict as possible. Right? Because you wanna find those error message errors as at your configuration time, not at the time when you're deploying and things are not working. Right? Oh, yeah. Definitely. I think that's a good thing for it to catch. And the alternative to me, hovering that together there, would just be to update the overlay and spec selector and then match labels. That's right. That would be an option. Yep. Think you have to throw in one more.

56:56 Yep. Spec, selector, match labels rather than just throw in new labels into our metadata. That worked. Nice. And then we have our replicas of one. So from a a, I guess, some sort of deployment pipeline where I was piecing this together, my environment my deployment pipeline would have to be environment aware that it could just fill in the correct overlay, and it's just gonna work. That's right. Yep. And a lot of times, we actually we we see folks which instead of using individual files, they just use directories. Right? So you can have a config directory that contains

57:11 Organizing YTT Files with Directories

57:36 NGINX, YAML, and a b c lib YAML. And then they may also have, like, a production directory that contains whatever overlays you need to contain. Right? So you're not really concerned about, you know, which files are individually included. Right? You're just relying on, you know, the file system to tell you what's what. K. So you're saying I can move this into a deploy directory. I need my library here. Okay. So my command could then You you you you do wanna move the library into the deploy as well since it's since it's being imported by NGINX YAML directly.

58:14 It's also being imported by our overlay. Does that mean I have to put the path here like this? So, actually, no. Yeah. The the this should work. You just have to remember yeah. You just have to remember to include it through dash f. Yeah. Right. Okay. So the first thing I would do is to include any libraries that I need. I could then specify my deploy directory followed by my overlay directory. Yep. I guess, and this still works. Okay. Yeah. It's it's the what's interesting is that, you know, I've previously been involved in various tooling that was always very strict. And so

58:53 this time, I've I've tried to make it just kind of a little less opinionated in terms of how you integrate with your workflows and Unix environment and whatnot. Because a lot of people out there, they're all doing all kinds of DIY installations of of Kubernetes cluster, and they have their own opinionated workflows. Right? So we wanna cater to that. Right? If you really wanna kind of separate them, you know, may maybe you have 10 different ways of how you overlay things. Maybe it's like dev and then, like, dev extra and whatever else. Like, it's kind

59:27 of up to you how you manage the files. There are benefits and downsides to that approach. Right? Yeah. Definitely. And but I could I I'm really starting to get a good patron of of how these tools come together, how they work together, and the kinda responsibilities for each, It's I think it's starting to make them look very, very interesting. So we're kind of approaching an hour now. So what I would like to say is, like, we can definitely demo anything else that you think is important for people to see, or maybe we could just do a quick

59:58 few minute recap of the other tools and what their responsibilities are. And then I will thank you and say this has been really cool. So do you wanna show off KeyBuild, or should we just quickly talk about KeyBuild, Image Pack, and vendor? I think we can talk about them. I think if folks are, you know, interested to look in, you know, it's it's there. They can also, you know, jump in our Slack channel, ask any any questions. We're very responsive there. So I guess so k build is is trying to solve well, it started out

1:00:28 Introduction to Kbuild (Image Building & Referencing)

1:00:32 as solving one particular problem. Right? So we actually have this problem right now. Right? We're just using image NGINX. Right? And so we never know what's gonna deploy. Right? Because NGINX latest will keep on moving. Right? And so what KBill does, maybe we can actually showcase that real quick, is if we pipe the YTT output into into cap. Right? Right now, it just gonna say, let the cluster try to find the latest thing. Right? But cable actually finds the image references within your files and switches them from non digest URLs to the digest reference URLs.

1:01:15 And what that ultimately gets you, right, is that that immutability of the config. Right? So let's say if your deployment has been running for a long time in your cluster and for whatever reason, some of the NGINX pods need to restart. Right? There's not gonna be a chance that when the pods do restart, they pick up an entirely different version of NGINX since the time you've deployed it in the first place. Right? So by using this digest URLs everywhere where k build finds the image reference, you're kinda locking everything down. And on top of that, you know, kind

1:01:47 of the second step to that was, well, when you're developing locally, you actually may want to get, you know, run your Docker build process, right, in the middle of all that. Right? You have your configuration. You made some changes to your app. So you wanna run the Docker build, and then you wanna do a cap deploy. And so that's, of course, cumbersome for you to do. So k build with a little bit of configuration will actually go ahead and run the Docker build for you and replace the image reference with whatever got built. Right? So k build is really, I guess,

1:02:18 focusing on how do you integrate into your deployment, you know, pipeline this image finding and and building. Today, can use, like, Docker build and pack build. So that's the build packs folks. Okay. That makes sense. I I I think what it I think what a lot of people don't realize is that when you deploy something to Kubernetes and you're using an an image tag rather than content addressable shares, is the if that pod or that deployment runs on multiple nodes and that tag changes at some sort of random time, you could actually end up with two different versions of

1:02:53 that image across your cluster or more depending on if a pod crashes, it's rescheduled on to another another week later and pulls in that that tag, which has been overwritten. I see it so many times where people end up with this completely weird cluster configuration with different versions of their image. Right. Right. That idea of key build resolving those at deployment time and then pushing that to the cluster. Yep. That's yeah. That's that's that's kind of how I end up with that, you know, separation of the concerns because you you do want it to make it pluggable

1:03:26 for all the other folks out there, right, that let's say, use customized or or helm template or whatever it is. Right? And so kind of a cousin of k build is image package. We're actually moving some of the functionality of k build into image package right now. The functionality that I'm talking about is ability to take a set of images and move them to your private registry. A lot of people call it relocation. So image package kind of combines the concept of you wanna keep your configuration in a Docker registry right next to your images. Right? So for

1:03:34 Introduction to ImagePackage (Bundles & Relocation)

1:04:02 example, if you decided to say InfluxDB version 1.1 is ready, you just push it up to the registry. And now it's under, I don't know, apps slash influx DB configuration or something like that. But influx DB configuration is also referencing, like, 10 other images probably. And when you, let's say, dealing with a environment that needs to be all firewalled away from public Internet, Right? You wanna take all of that configuration and move it into their, you know, registry. And so image package allows you to do that. Well, right now, it's not quite there yet. We're actually like I said, we're moving

1:04:39 that functionality from k build. But image package introduces this concept of a bundle where it's a image configure it's a application configuration plus a set of images that it references. We kinda see it as one thing. And so that that kind of allows you to to manage your environment more confidently in terms of where this you know, where the configuration is coming from, where the images are coming from. Excellent. This all sends really cool. It's pretty much a tool to make all of my Kubernetes painful experiences that little bit better. So it's great that these are all being worked

1:05:11 Conclusion & Resources

1:05:20 on. So I think we've covered an awful lot today, and, you know, there's obviously so much more that people can pick up. So what I would recommend is there was a talk at KubeCon a few weeks ago. Is that correct? That's right. Yep. Check that out. There is the Carvel channel on the Kubernetes Slack, there's obviously the get repositories as well. There's plenty of examples for people to go and start the experiment on their own and understand how the Carvel tools can help them. Is there anything you would like to add to that before we finish up, Dimitri?

1:05:52 Thank you for having me. It's been it's been pretty fun. You know, I really like the the format that you have going on. Yeah. No. That's pretty much it. We welcome anybody in our Slack channel to chat about various problems that they may have. And if it happens, then we can help. That's that's even better. Alright. Awesome. Well, thank you very much for joining me and taking the time out of your day to kind of show me the tools and walk me through it step by step. It's really helped me have a better understanding and and hopefully,

1:06:21 the people that are watching on YouTube as well. So thank you. Have a great day, and I'll speak to speak to you soon. You too. Bye bye.