0:09 Cool. Maybe let me just send a message. But we are now on YouTube. So don't say anything about me that says I'm unprofessional or rubbish. Alright? Let's save that for afterwards. It's really frustrating, but we are live. We're on YouTube. We'll get started in just a moment. Alright, everyone has been emailed. Let's see what happens. Sorry about that, but we are now gonna get started. So let's say hello in the chat. Alright. Well, Stefan. Thank you for taking time out of your day to join me on the session, talk about Timoni and work on a live

1:56 demo. So could you please take a few moments just to let people know who you are, what you're up to, and then we'll kick things off. Sure. Hi, folks. I'm Stefan. I'm a long term Flux maintainer. If you don't know Flux, it's a CNC project that does GitOps thingies. And, yeah, roughly nine months ago, I started a new site project called Timoni, which is like Helm, but not Helm. Doesn't do YAMLs, but QLang. And yeah, works quite nicely with Kubernetes. Uses all the new shiny things in the Kubernetes API that I've been, yeah, working with inside the Flux project for the

2:57 last two years. So, yeah, I'm very happy to share with you my project. And, yeah, hopefully, people will get excited about it. Awesome. Well, I do have lots of questions, but I know you've got some slides. So why don't you share your screen? We'll start with that, and I'll throw my questions at you as we work our way through that. And then for everyone watching us on YouTube, the demo will follow shortly after the slides. So we will be getting hands on and showing you how Timoni works in practice. Cool. Yeah. So, Timoni. I called it a package manager. And we

3:38 are going to see through the slides what makes Timoni a package manager. But before I'm getting there, I wanted to frame a little bit how I'm thinking about Timoni. And I basically put people in two categories. The first category is software makers, like, I don't know, software vendors that are trying to ship their cloud native apps onto clients' clusters, and of course, open source maintainers. There are so many open source projects out there for extending Kubernetes, like CID controllers and so on. And also platform engineers. So this group of people and organizations will want to, you know, distribute applications

4:36 onto end users' clusters or environments. So that's one category that Timoni tries to, improve their workflow there, how you alter your app definition and how you distribute that to everybody. And the second group are end users, Kubernetes users, can be developers, operators, people that want to use well known defined upstream packages for a particular app or controller or complex systems. And they want to make use of that to deploy it on their own infrastructure. So I think these two thinking of users in this way, it's quite common for any package manager out there. Someone creates the package for an application and someone

5:40 else uses it. But with Kubernetes, it's not always like that. Usually, these two groups can be a single group. If you are a platform engineer or if you are a developer in your own organization, you may have to create a distribution for your own app. And sometimes you also manage that app. You deploy it on different clusters. So this group can be one. Okay. So what is Timoni? As I said, it's a package manager. It's basically a command line tool. It's written in Golang. It's a static binary built for Mac OS, Linux, Windows. And it has no dependencies

6:35 on anything on your operating system. And it embeds three core technologies. QLANK. So basically, Timoni comes, built with the Q engine inside. So it heavily relies on QLANK for when you write your app definition, you'll be writing it in queue. When you want to deploy that particular app, you can also write the deployment, how you configure it, and so on in queue. But for end users, CUE is not a hard requirement. You can also write the configuration in a YAML file or a JSON file and so on. So that's one thing that powers Timoni, Keylang.

7:25 The second one is the Open Container Initiative standards. And what standard Timoni heavily relies on are OCI artifacts. So what are OCI artifacts? There are these things that look like a container image, but inside is not a container image, can be custom configuration. In Timoni's case, what's in the OCI artifact are CUE definitions. Timoni has its own OCI artifact type and is compatible with almost all container registries out there. They got to a point where OCI artifacts are are the standard, and they are accepted almost everywhere. And third, of course, is Kubernetes. Timoni deploys everything on Kubernetes.

8:24 I didn't get Timoni to Lambdas or anything else. Get. You mean get. Yeah. Yeah. Never know. The idea here is that Timoni tried to use this Kubernetes API using the latest and greatest innovation inside Kubernetes. And more specifically, Timoni relies on Kubernetes server side applied, which is a thing that it's generally available in Kubernetes since two years now. So you shouldn't have an issue using Timoni. You're probably running a Kubernetes version that understands this type of operations. So how this makes Timoni different to other tools out there is the fact that Timoni, every time it tries to deploy an app,

9:23 it will ask the cluster it will ask the cluster for the parent state and it will compare the cluster state with the desired state to what's in the app definition. And it will only apply changes by correcting the state. So you have like hundreds of deployments, but you only change one of it, unlike Helm and other tools, Timoni will only apply that change. So it will be very, very fast if you do small changes and frequent changes. Okay. Next. So what makes Timoni a package manager are these four core concepts around applications. So first, it allows you to define

10:16 an app. You distribute that definition. You can compose an app from multiple definitions. For example, you have microservices. The app concept is a bundle of all these microservices. Maybe you have a distributed monolith. It's also a composition of things. So it has this feature of allowing you to compose things and that's how your app comes into being. And finally, lifecycle management, which is the core thing that any package manager should do. Should be able to install things, upgrade things, uninstall it, roll back, and and all these things which we can call lifecycle management. Okay, so let's do a deep dive into

11:04 those. So app definition, what that means. How do you define an app for Kubernetes in most cases, normal cases, right? You'll have a bunch of YAML files on the disk with all sorts of settings in there, like Kubernetes deployment, the service and so on. But that representation, all these YAMLs on a disk are one way of how that app can be deployed on a cluster. You need to change things. You need to allow your end users to fiddle with limits or enable some feature in your app in the config map or things like that. So how do you achieve it in our

11:55 current day? You have to have some kind of templating engine. Right? Or you can use customizing and ship a bunch of overlays to your end users, which I haven't seen it done that much. So it boils down to be able to have a good app definition engine, you need a templating engine. And the templating engine that Timoni chose and is built inside is QLang. And what that brings to Timoni's Timoni's users is having type safe Kubernetes templates. I'm going to explain a little bit what that means later on. So you define your app with typesafe Kubernetes templates,

12:44 then you allow your users to customize it in all possible ways. But you have total control of what the end user can change to your app. You control what you expose, what you allow them to do. And more importantly, you also can configure defaults, good defaults for your app and so on. So have the app definition then of course, how are end users going to use that app definition? Well, the package manager has to have some kind of distribution mechanism. The distribution mechanism for Timoni are container registries. And I think it's kind of natural these days because

13:31 any app that runs on Kubernetes in a way has to is composed out of one or multiple container images. The app runs in a container, that runs in a pod, and so So all software vendors out there already have a container registry, or they are pushing their apps to the client container registry. So in order to distribute app definitions, you can reuse that registry, which is already there. You have figured it out how to scale it, how to run it, and so on. Right? So you can push to the same container registry where your app images are,

14:14 the Timoni packages. And Timoni packages have two properties. Timoni modules are semantically versioned. And this is enforced by the system. Timoni will not allow you to publish a new app definition without having a valid Sembr for it. And why Sembr is important is not because people have figured out Sambr. No, it's not that. For me, Sambr is more about ordering than anything else. You can determine what's the latest version by just looking at and it's quite easy. Now, using SemVol the right way, you, oh, I'm making a breaking change. I'm bumping the major version. Version. Well, not many people do it.

15:09 I mean, doesn't. It's on one point something, and any final release comes with some breaking change. But still, server as a means of telling your user, this is a release candidate or this is a stable version, it's quite important that I went with that. Timoni also wants to ensure immutability and a container registry does not enforce that by default. If you use, like Timoni uses the version to set the image tag in the registry. And you can override it. You can push now version 1.0.0 and later on say, oh, going to push again 1.0.0. This

15:55 is also an issue with Helm and container images in general. So how Timoni tries to solve this, it allows the end user to specify the version, but also the digest. And the digest in a container registry is immutable. And what Timoni will do, will compare the version with the digest that the user expects. And if that doesn't match it, it will not deploy that modulator and it will say to the end user, hey, you wanted this version with this digest, but this is they are no longer matching. So it can just delete the version from there and work with Timoni

16:35 just to digest. I know this is not human friendly. You can't just look at the digest and understand it, but for machine is really great and you should be pinning things using digest and Timoni tries to make that not also for modules, but also for container images. And I'll mention that later on. Yeah, and also modules can be cryptographically signed. That's also very important and you can verify it. Timoni fully integrates right now with cosign keyless or with static keys. So how does a bundle look like? How does a module look like? What What the

17:17 software vendor, what the open source maintainer has to create inside the repo? So you'll have this kind of structure. It is a directory. You can initialize all of this with a simple command, Timoni Mode in it. Creates this kind of structure. And if you look at it, it resembles a little bit with Helm. I tried to preserve the templates directory, but that is the only resemblance. And of course, it has a README where you should definitely put in there all the documentation around your module. And the structure here is very specific to QLAN. What is a Timoni module is, in fact, a

18:01 QLAN module with a more open ended structure. QLAN is, if you're not familiar with it, is this language that tries to unify all types of configuration. You can generate configuration, you can validate configuration, you can also generate code. You can do even scripting with it. So it's very flexible. It allows you to do a bunch of things. But this kind of flexibility also means that, you know, it's quite hard to get started and figure it out and so on. So Timoni comes with very strict ways of how modules are defined, how they are structured. So

18:48 it brings some kind of it's uniform. No matter what repo you'll be looking at that has a Timoni module, have to have this structure and it will have to have particular files with different functions. For example, you need to have a templates directory and in every templates directory, you have a config. Q, which contains a Q definition of what type of configuration your app allows. And it also contains the schema and the constraints. And all of that is put in there. And then you have Timoni ignore and so on, all these things that are here.

19:34 But the idea is that your module will contain schemas. For example, all the Kubernetes API schemas. So if you want to define a Kubernetes deployment, you'll be using the upstream Kubernetes deployment schema, which is very different from other templating engines. Why is it different? Because once you use a schema, then you know that template is type safe. You can you can add there a field, let's say replicas, and you make a typo and you don't type, last test. Running a Timoni vet, Timoni will tell you, hey, this field is not in deployment. You can't set this field.

20:21 So all those problems where you are just templating over a text, which is some kind of YAML, are quite gone. Are way safer using schemas because they are strict and all these fields are validated without you having to apply all the time to Kubernetes cluster and test if these things are okay or not. And Timoni goes even further than just offering type safe schemas and templates for Kubernetes native objects. You can also embed in your modules Kubernetes CRDs. And Timoni has a command where you give it a CRD definition, let's say, I don't know, certificate

21:13 from cert manager. And it will generate a schema for you. And when you define a certificate, it will have the same validation to know all the fields, all the possible values of those fields and so on. So this is like a major step in the direction where more and more controllers are there and now it's quite common to ship your application, not only with Kubernetes standard APIs, but also making use of all these extensions, all these custom resource definitions. So yeah, that's how a module looks like. Timoni comes with the whole set of commands for creating

22:01 a module from scratch, that's in it, verifying the module web. You can build it and see the final YAML or the final JSON. You can import Kubernetes schemas by just setting the Kubernetes version, and these are curated schemas that I've published on GitHub. And you can also, as I said, vendor any CRD you have there by just giving Timoni the YAML of that CRD or the URL of that CRD. And of course it comes with commands for you to actually run end to end tests like is my deployment. Okay, my deployment is valid. It has all these things, but what

22:43 about will it work on a cluster? So it has the supply command. You can, that works with local modules. You don't have to push them to a container registry and you can do all sorts of tests and create our own end to end test grid in CI and so on. For distributions, there are also dedicated commands like module push, module pull, module ls, which will query the container register, gives you all the versions, all their digests. It also comes with some helper functions like registry login and logout in case you don't have a Docker CLI or a Crane CLI locally.

23:29 And it also allows you to sign and verify modules when you do a push and when you do a pull. Nice. Any ideas, David, so far? How do you like it? I mean, it takes all the boxes for me. Right? Because let's go back to everything you've covered so far. I've not made it a secret on my YouTube channel. Right? I'm not a fan of Helm. And it's not because Helm does any one thing particularly bad. It's because it's YAML and because Go templating, in my opinion, just isn't good enough. It's a horrible syntax unless you've written a lot

24:11 of Go code before, which used to be true, right, in Kubernetes ecosystem. Everyone that was in the Kubernetes space had written a plenty of Go code, probably used the template and language. They were fine. But as more people came, I spent a lot of my time helping people understand Go templates rather than just and, hey. Use this or to, like, Ginger or handlebar syntax. Everybody's familiar with this now. Go templates resemble a little bit until it doesn't. Like, the special dot syntax and the way that, you know, trimming white spaces. There's loads of weird quirks. However, I'm not gonna bash about

24:42 Helm because Helm has also helped us get to where we are today. Of course. I I feel that there has to be something better. And the minute I seen Timoni and I saw that it is CUE and it was used in OCI, like it's just like stars in my eyes. I was like, this is this is what I wanted. This is this is like if I could build something and I had the the well to set and do it, this is exactly what I wanna see and what I want to see. So I was very

25:05 happy. I came into this knowing the problem space, knowing that I like CUE and knowing that OCI is where we need to be for the, you know, future facing get ups delivery pipelines. But let's assume there are people watching that are interested in Timoni, they haven't maybe got that experience with CUE. I would love to know from you, you being the person that made all of these decisions as one, why did you say that you had to write a new tool? And why did you just say that CUE would be the substrate or the, you know,

25:37 the the primitive for building that new tool? Yeah. I mean, I I got to queue out of frustration or writing Helm Chats, and I wrote a lot of them. And I still maintain a lot of them. Hell will not go away anytime soon. And I tried over the years several technologies that tried to enter this space and of failed. I think JSONED was the closest one. And there were different projects back in the day that are dead right now. I think the only one still alive and kicking is from Grafana, Tanga. Yeah. I JSON didn't stick with me.

26:37 I'm frank. I mean, it's powerful. It's nice. But I don't know. I couldn't deal with JSON schemas on top of JSON and other JSONs. And so, yeah. It was like I think what yeah. You're you're right. Like, I think one of the challenges with JSON at JSON, I don't even know how to pronounce it right. But it was familiar enough if you didn't do anything JSON at ish and you just talk to JSON. But the minute you wanted to start using their functions and the other features that made it as powerful as it it was,

27:08 it became so alien so quickly that you just really, oh, this this node doesn't feel like something I'm comfortable with anymore. And that was always one of my struggles been trying to work with it for sure. Yeah. For me it was like, okay, who uses it right now? And I I've seen that the Prometheus team is a big fan of of JSON. And I looked at JSON for deploying Prometheus operator and some Grafana dashboards. I yeah, I said, what? It wasn't for me. I don't know. I don't want to say it's not for everybody. Lots of people use

27:43 it and love it, but it didn't stick with me. Then, I don't know, years later, two years ago, I was at KubeCon Valencia. Someone came to me at the Flux booth and said, hey, we migrated all our charts and all our customized overlay to Keylang. I said, Okay, I knew about Keylang. I went to their website, I don't know, maybe two years, three years and a half ago. It looked very similar to Go with J. Wow, I do Go, I don't know, eight years now. But then when I looked at their docs, it felt like, I don't know, the docs

28:28 were written for language authors, for academic people. And I looked at that. And I yeah, I didn't continue. I almost forgot about it. Then this person that came to me, Varesha, we talked a lot. I think we talked about half an hour or something. And he really convinced me to go back home and give it another shot. And it took a while for me to really like CUE. And because that person was so convincing, like, was, like, so enthusiastic about it, Every time I said, Okay, I'm going to close this and go back to Flux doing

29:14 my customized controller or whatever, I said, Okay, but maybe I should try more. And yeah, after a while, really clicked at some point. And I started doing more and more things in CUE, understand better the language. I think for me, advantage was that Q itself, the language and the engine is written in Go. And if I couldn't quite figure out why is this happening, would I could just go in the source code and read it. And read their unit test, which was very, very helpful. Like even if the documentation wasn't great back then, they had like

29:56 really good test coverage. I could really understand what's going on. And I'm that type of person that likes to read code to understand the tool. I know not everybody should even think about that. It's not good for your health, but anyway. That's me, yeah. And what I really liked about CUE, and I'm very fond of it, is the immutability part, which I really, really dislike around how Helm, Customize and all these other templating languages are running. Like you can set a value in base, you set replicas one. Then in some other template you do an if and

30:41 you say, oh, replicas here are three. And some later on in your logic, at some point you remove replicas altogether, stuff like that. Which is fun because you have so much freedom, you can do whatever, but when you need to debug this stuff, it's a nightmare, right? Well, CUE is not like that. Once you set a field, that's it. If you try to remove it or set it to another value, it will say, hey, this field is already set. It has this value. So in a way, even if at first I felt really mad about it, like

31:21 why couldn't I just override this field? How can this be a template engine if I can't override things? And then I understood why this is good, right? Because it makes you for a Timoni module, right? You really need to define all the things in your options in the config file, and you can only set defaults to them. You can set replicas to interdeployment queue file, and then in the config set a default to five, and somewhere else, like if something, set it to seven and so And even if it's annoying at first, it forces you to create a good structure and expose

32:02 only the right things to the end users. And also for them, it's easier to configure your app because there is one way to do it, not 1,000,000 ways. And of course, customized patches, right? You have a base, you have an overlay, but then you can create another overlay on top of that overlay, and so on. Even if it's flexible, this type of hacking almost with all these objects becomes really, really hard to debug in the end. And with CUE, you have these constraints, so you need to understand the constraints. You need to be Okay with them. And afterwards,

32:45 things will look better. The end result will be better, if you want. So yeah, I'm not saying getting started with Timoni is hard, but it can be a challenge coming from customizing role because you'll try to apply the same principles, and in there it will not work. So yeah, it has a steep learning curve, but more about you need to change a little bit how you think about configuration and templates. Awesome. Yeah. That was a great description of CUE and some of its challenges. Right? You you said the word academic, which was was spot on. I remember the

33:30 first time I looked at the CUE documentation and I was like, isn't for me because don't understand all this terminology. The lexicon is just above my pay grade at some capacity. Like, I need to know what that disjunction is to understand the unification model of the CUE block. Like like, sorry. I just wanted to describe some Kubernetes resources. And I think they're getting better there. You know, it was a very academic project and I think now it's starting to find a bit more traction and they are trying to sand off these rough edges, make it easier

33:57 to consume and just use it as what it's used for without understanding what's going on under the hood. And I do completely agree with what you said. It forces you to understand where things become options or configuration for your CUE value object and you have to explicitly make these things that people can consume via definitions, private fields, whatever. And you never said this directly, but I think it's important and that kind of underpins everything that you said you liked about CUE. The schema definition live right next to the values themselves. That's where the unification comes in.

34:31 Like, the schema all gets merged together to give you some sort of concrete value. And I I think that is something that could throw people off at first, but then once you understand that dynamic, it becomes a bit of a superpower too because you can arbitrarily throw around definitions and expose them to people and then take an input objects and all this other wonderful stuff. So people will see when we get to the hands on bit for sure but Yeah. Yeah. I mean having schemas, constraints, and values all in one thing is clearly overwhelming for someone that starts today,

35:06 and for Timoni, you need to get into this as a module author when you write your app package as a Timoni module. But for end users, I'm not exposing them to this type of complexity. For end user, when you configure the app to deploy it, you don't write any schemas, you don't deal with defaults, you don't deal with constants, you just give Timoni concrete values, like I want replicas two, horizontal pod autoscaler max five and so on. And then Timoni injects the schemas, runs the constraints and it will say, hey, you are setting replicas two,

35:53 max replicas five and max replicas three. This is not okay. So I'm trying to not expose the end user to all this complexity because all this complexity is really needed on the other side when you compose your app definition, not when you use it. When you use it, it should be very, very simple. And yeah, we'll see how easy it is when you do the kickstart. Okay. Last two things about package management. So we see that, okay, Timoni allows you to define your app with Q templates. You can distribute those to container registries with OCR artifacts.

36:36 But how would end users, how would Kubernetes users, developers, you know, everybody that has to do something in Kubernetes, how are they going to consume it? And there are two major cracks here. One is app composition where you basically never deploy something alone. Maybe it has a dependency. Maybe your app needs a Redis server for the caching or the app that you are deploying is made out of multiple microservices. Some are optional, some are not, right? So with Helm, we have these umbrella charts and so on. I want to build that into Timoni at all.

37:21 In Timoni, there is a different approach on how you do app compositions and it's not about writing modules. And we'll see how that works. It's called Timoni bundle. But the idea is you can bundle multiple things to have your app as you want it. And another important feature of package management and what Timoni also does is lifecycle management. You can install it, you can upgrade it, you can roll it back, you can reconfigure it, you can run end to end tests after an upgrade or after an install, and all of these things have to be

38:03 like straightforward, easy to use, and we'll see how that goes. So how do you compose things? How do you, you as an end user, how you add configuration, how you specify modules and so on. This is done through a Timoni bundle definition, which is a CUE file. The same representation can be done with a YAML file or JSON. But yeah, I definitely prefer CUE also on this side because Timoni comes with some nice features when you use CUE for definitions. So what is a bundle? A bundle is a file where you list instances. What is an instance? You can think of

38:53 an instance as a for example. It's an instantiation of a module on your cluster. So a bundle is made out of one instance. If you deploy a single app with no dependency or nothing, or a bundle can be made out of multiple instances. And here the example is I'm deploying Podinfo, which is a toy app that I made a long time ago. And this app needs a cache server, which is optional, but you can also deploy to the cache server. So instead of having Redis as embedded in the Podinfo module, you would deploy Redis from its own module,

39:34 deploy pod info from its own module, and then you can have shared configuration between them. In this case, I want to pass the password from the runtime because I don't want to hard code here in CUE files, any kind of sensitive information. So I'm defining and you can define any type of input. Here is just a simple string password comes from the runtime. Then I'm using this in string interpolation to set up the Redis URL for the info. And I'm also using it in the Redis configuration where I explicitly set up the password. Here is the value

40:13 of it. And another thing to notice here, modules are referenced by an URL which is like a container image in Kubernetes, but has an OCI prefix. So it's clear that it's an address to a container registry. And you can reference a particular version. If you don't specify the version at all, Timoni will use latest. But as I said before, you can also refer to modules by upstream digest, so you ensure that no matter how, when you run this bundle, it will always use that particular version, even if tags are mutable in the registry, you can refer it

41:01 by digest. So yeah, this is how you would be using Timoni. You'll create these bundle files and you'll apply them and yeah, and Timoni will do all the deployment for you and so on. So Timoni has commands specifically for bundles. They start with Timoni bundle something. You can verify them, you can build them, you can apply them, you can do dry runs and diffs every time you do an upgrade. Maybe you are not sure what things will change on the cluster with that upgrade, so you should do a bundle apply with minus minus diff. I want

41:43 to show you a nice diff of the Kubernetes objects, only the fields that are changing, and you can query the status. Of course, you can delete it. And you can also distribute bundles, which are Q files and other things through a container registry. As a platform admin, maybe you want to create bundles for cluster add ons, and those look the same everywhere, maybe the password or whatever input is different, But if you want to share with other well known configurations, you can also push bundles and any type of Qt definition to a container registry. And there is this

42:31 Timoni artifact command, which has pull, push, list, and so on, like modules. But the main difference here is that if modules, for modules, you need to use SAMLER, and they are SAMLERD artifacts, it's free for all. You can do whatever you want here because these are snippets that you'll be sharing, So it's up to you if you want to use a mutable tag and so on. In any case, you can also do a pull using a digest and so on. But yeah, the artifacts commands are a way right now to work around the lack of

43:14 package management in the queue language itself. And I want to explain a little bit what this means. Timoni has, for example, helpers. There are some Q schemas, for example, for defining container images, for defining Kubernetes metadata with the right labels, with app Kubernetes labels. So you want to ensure some kind of good defaults and you want to have helpers so you don't have to type all this Q code on your own. So you build your little CUE library. Now, if someone makes a different bundle on their own repo, the only way right now with CUE is

44:02 they need to copy paste all the files or do some magic with git and symlinks. I don't want that. So this is where Timoni Artifact Push and Pull comes into place. You can share snippets of CUE code packages and so on. But the CUE team is actively working on shipping package management inside CUE for the language itself. And I'm very happy with it. They went with OCI as well. So at some point you'll, the future will not have to use the Timoni artifact push command. Will able to do this with Q alone and share code between

44:51 modules like that. But until then, this command that you can definitely use it today. So I'm curious. I know I I don't wanna ask too many questions before we get the terminal open and get hands on and stuff. But, you know, the CUE modules the the the CUE module stuff that the CUE team are working on would be to push arbitrary CUE to an OCI and then be able to pull that down and build, you know, queue values based on all of that stuff together. I mean, with the Timoni artifacts be different from that with different metadata, different

45:27 configuration or would it are you just gonna use directly what the CUE team build? Like, it feels like they're different, but maybe similar. So Timoni the Timoni artifacts command can be used to distribute reusable CUE packages. Right. Okay. And that part, I hope, can be totally replaced upstream using QGET, like GoGET, and that part will be will work with the CUE toolchain itself. But Timoni Artifact will still be useful if you want to distribute Timoni bundles and runtimes as well on configuration everywhere, because then you can pipe you can do Timoni artifact pool pipe Timoni bundle

46:21 apply from standard in, right? So it's a hacky way of using artifacts now to distribute things that CUE itself should distribute, but I made these commands for distributing well known configuration of bundles in the same way as you do with modules. The idea is you push your container image, the code that's running to the container registry. You push the module, which is how that app can be installed. And then you can push a bundle with secure first defaults. This is how it should be run, or examples of best known configuration. Got it. Thank you. Okay, and the last thing that

47:12 bundle supports are multi cluster deployments. And this is something that right now is supported as I'm presenting it here. But there is a proposal on the Timoni repo on how I'm seeing this in the future, how I want to improve it. The idea behind this is that a bundle can express all the different configurations of one app across your whole cluster, Philippe. And you don't have to have different bundle files or something like that. You can write a little program in CUE where all the different configurations, all the differences between how an app must be configured across environments can be embedded

48:04 in a single file. And with customize, for example, this type of customization means you'll need three overlays, the base one, staging one production. Oh, if you want to do customization based on cluster name, then you'll have other overlays for each cluster that import the production overlay and it gets to this, very messy file distribution. You need directories with files inside, other files, directories on top of directories. And this is what Flux does today with Customize. This is what everybody does with Customize. It works. It's great. It's pure declarative. You have all these directories on top of those.

48:59 You have the overlay system. But with Timoni, I want to give people a way to write this in a more compact form and have the expression of your app across environments in a single file. Of course, you can split this bundle into different files and Timoni will do a CUE unify of all these files so you can have these overlays if you want, but if you don't and if you want to have everything expressed in a single file, can do it because CUE allows it. You can write if conditions, you can with Timoni runtime environments, can take input from

49:44 outside, from other clusters, from your local environment. And based on that, you can create this little program that knows how to deploy your app everywhere. And what I'm trying to do next is, based on this, is allowing people to have in a different CUE file, which is called a runtime file, where you can list all your clusters along with the CUE context for each cluster. So you don't have to do Timoni apply bundle minus minus context production, because this is how it works today. And the problem with this is, if you mess up your context, you'll end

50:29 up applying the dev configuration on the production cluster or something like that. Right? There are many tools to switch context. You can put the context in a CUE environment variable and so on. And it's quite easy to mess it up. So I want to add to Timoni a way of you to declare all the clusters that you have and specify there the context. So you'll only do apply -f this file and Timoni will know, oh, these are all the clusters I need to execute them with this particular order, dev first, if that goes Okay, go to

51:08 staging, go to production finally. And do this progressive deployment of the bundle across the cluster fleet. And yeah, if anyone is interested in how this will shape out, there is an issue on the Timoni repo. Okay. Timoni versus Helm. We've touched a little bit around this. There is also a dedicated page on the Timoni website, Comparison Help, where it's quite a big list there of all the differences. I wanted here to just showcase some of the performance differences and how Timoni is, let's say, more performant, more efficient than Helm just because it uses the technology in a different way. For example,

52:01 Timoni does not store all your YAMLs in Kubernetes secrets. What Helm does every time you do a Helm upgrade, even if the upgrade modifies just a tiny bit of YAML, what Helm will do, it will get all your YAMLs in your chart, and it will store that in a new secret. And you end up with, I don't know, ten, twenty, one hundred secrets, whatever you can set the limit for those as well. But the idea is you always have the manifest twice. Once in ETCD and yet again in a secret. And if you version it, you do multiple upgrades, you'll end

52:49 up with all these things. And if you do a simple load test, let's say you do a Helm upgrade in a loop 100 times, you'll see gigabytes of data going between the CLI, the Helm CLI, and the Kube API. And you can see the amount of pressure Helm puts on the Kubernetes API. And yeah, my approach to this is I don't want to store all these things twice in the cluster, because the module definition is already in your repository, in your container repository. And what Timoni does when it creates an instance, it stores in the cluster your custom values, because those

53:42 really matter, and the digest and the URL to the module. So if you need to roll back, if you need to apply an earlier version, you can just reference that digest. Of course, it's not the same as does it, because even if you delete a chart, the YAMLs in that chart are already stored in a cluster. So you can do a restore without having to pull the chart. But if the chart goes away and the container images go away, you delete the container repo. Then the rollback will still fail because the images are gone. Right? So

54:27 why the hell doesn't even make a copy of all the images, right? That's a real rollback. You need to copy everything in a way. So my approach to this is Timoni also creates a secret as a storage. It doesn't store there. It doesn't store there YAML. It only stores the Q values that the end user created. And a list of references, which is kind, name, namespace, API version, to all the things that it manages. So this is like a tiny bit of data so Timoni can do all things. You can see what resources it manages,

55:08 what's their status. If you want to go back, restore the current version, if you do a kubectl editor or something, you have the module digest there so you can reapply it. So that's one main difference between how Timoni and Helm works around state persistence. Another difference in regards to performance is what I mentioned before. You have huge app definitions with hundreds thousands of manifests in there. And if you do an upgrade which only changes one deployment, Timoni will only apply that change. While on every upgrade, it will apply everything. And you can see how that can be

55:56 really impactful for your Kubernetes API. Okay. If you want to find out more, there are many, many differences. I try to keep this list up to date. And it's on comparison on the website. Some resources. How can you contribute Timoni? Well, Timoni is Apache two point zero licensed. It accepts contributions on GitHub with pull requests. But I think code contributions are great, but they are not the most valuable thing you can contribute. I think at this point, the most valuable thing is trying it out. Try to write your own modules and give feedback on that. And if

56:51 you create some modules, share it with others. They are like, I'm not going to translate all the 1,000,000 Helm charts out there. I'm not going to do that because Timoni is just a side project. I work on it ten nights and weekends. I don't have time to do all of that. And I know there are Timoni users right now which have fully migrated from the inherent charts to Timoni modules. But yeah, there are so many things out there that could be distributed as modules. And I think that's the most important thing. Feedback on that, how hard it is to

57:31 create a module. I'm quite sure there are so many UX improvements around authoring and helping people write CUE, generate all this code. CUE is quite challenging when you have schemas and all the things that we said. I'm also considering having dedicated docs in the Timoni website around modules, and maybe have a get started with CUE as well. Specifically for Kubernetes, not all the configuration out there, right? And I would really appreciate help from anyone who can build a get started with QNT money guide or something like that. I think it's quite more important than someone

58:23 hardcore code contributions, which are nice, but it's not the only way you can contribute. Okay. That was it, presentation. It took some time. Let's get the hands on. Exactly. Alright. So thank you so much for for walking through that. We have a couple of questions from YouTube. However, I think those questions will be best answered by us just getting the terminal open and actually showing it. So to Stefan and to YMO, your questions are great and I'd rather show you the answers as we play with Timoni. However, I'm gonna be really selfish and ask one of my own questions because, you

59:07 know, that's because you talked about valuable contributions that are gonna come from people writing Timoni modules and making it easier for other people to onboard to Timoni without having to write everything themselves. Right? The first question that pops into my head the minute you said that was, well, first, yes, let's get people doing that. But what's the discovery mechanism? Are you currently talking to anyone at like the the CNCF or Artifact Hub in order to get Timoni listed there? Has it already listed there and I missed it? Like how do people find these modules when they do

59:39 exist? Yeah. That's a a great question. I have an issue about so Artificial Hub is definitely a great way of being able to discover things without locking people into a unique registry or my own registry or something. I don't want to have Timoni as NPM. I don't want to be the person that runs the authorized registry and everybody has to push there and so on. The whole idea of distributing Timoni modules with OCI artifacts is the idea that you will be pushing your modules next to your app image. I don't want to host your app images, right?

1:00:29 And it also makes it very easy to replicate it. Like for example, AWS has public ECR, which mirrors a bunch of things from Docker Hub. When Docker Hub goes down, you can switch to that, right? This could, you could apply the same thing to Timoni modules. You push the Timoni modules to Docker Hub, and you can push it to ECR public, to GCR, and so on. I use GitHub Container Registry because it's so easy for me to publish there from GitHub itself. But it works everywhere. And the discovery part is quite challenging when you have all these modules everywhere distributed and

1:01:09 so on. And I will definitely need some help for someone which is intimate with, has some knowledge about Artifact Hub. I see that Tecton and other projects have registered their OCI type. It's no different than that. Timoni has its own OCI, artifact type, and layer type. Standard annotations, which are all from OCI standard annotations. I didn't come up with anything special there. So yeah, Artifact Hub is one way of doing it, and I will really need some help there. Another idea that I had is run a public instance, a different instance of Artify Hub only with Timoni

1:02:01 OCI artifact in there and use that UI to only search Timoni things. Who knows? But yeah, Artify Hub will be a first good step forward. Yeah. Alright. Well, I'm going to share my screen and let's start kicking the tires. Alright. You should be able to see my web browser now. This is the Timoni homepage. If you wanna check this out and play with it yourself, you go to tonomi.sh. As always, we first need to install the tool. So I'm just going to grab the brew command jump over to my terminal and get this kicked off.

1:02:50 But we're gonna do two things today if I remember correctly, Stefan. We are going to work through the quick start gate and then after we get through to the end, hopefully get people answers to all of their questions. We're gonna try and show more of production use case by making this work with like, you know, secrets and other things that become important when you stop playing with the tool and start deploying with the tool. Oh, I don't even realize the brookie man was right here in front of me too. Oh, well. Yeah. In the install page,

1:03:24 there are many other ways outside Brew. I discovered, for example, that Open SUSE team has published a Timoni module for an RPM package on their repository and so on. So, yeah, I've Nice. There is a mix package, a YI package, an arcade package, Alex, from Open Files added to arcade. Hopefully, at some point, someone will contribute it to the Windows ecosystem. It works on Windows, but you have to download the binary now. I know Windows has its own package management. I'm not sure how to do that. Yeah. Alright. And just because I see that there, I'll

1:04:15 even get my completions running. Bruce should have done that automatically. Alright. Okay. Cool. Alright. So the first thing we need to do is install a Timoni module. So let's just make sure I understand exactly what's gonna happen here and just kind of share this with the people that are watching. When we say install a module, what we're suggesting is that there's somebody out there, you in this case has written a Timoni module, thrown it to OCI, and we can just consume that without doing anything right off the bat. We could just say, Timoni apply, point it to it, and it's gonna throw

1:04:52 something into our Kubernetes cluster. Is that correct? Yeah. You also need to specify the namespace where that thing is. Where that's oh, sorry. No. Where that's going to deploy to in the cluster, right? Yes. Yes. Does that use the context namespace, or does it actually enforce that we specify at runtime? No. If you don't specify it, it goes to default, but if you have changed your default context, it will use that. So Timoni, CLI is built on top of the kubectl runtime package, the CLI runtime, I think. So it has all the flags from kubectl

1:05:36 and knows all things that kubectl knows. Default context, default namespace, and so on. Cool. Alright. Well, let's just do a deploy to the default namespace. And just because I'm always curious, right, if I remove dash dash version latest, does it default to latest? Sure. Let's see. Maybe to let her out and say, hey. You you really need to specify Adam. Think Why? But I don't see your console. Oh, I shut my whole window. What happened? Window. Oh, I shared window not screen. There we go. Alright. So yeah, I just did the Timoni apply, I removed the namespace, I removed the latest,

1:06:42 my key chain is asking for stuff and there we go. So it's applying this to my cluster. So we could run kubectl, get pods, watch, and it kinda beat me, but we do have a pod and full deployed 10 ago. Also, Timoni does a watch and when you apply it. And it waits for all the pods to be healthy. Yeah. So if I run this again, I'm assuming it's just gonna say there's nothing to do. Unchanged, unchanged, unchanged. Yeah. You also mentioned during your talk that we could run a diff. Gonna say unchanged. So that's doing a server

1:07:36 side apply dry run. Cool. Alright. I like it when things just work the way I expect them to work. It's always an absolute bonus. Okay. So yeah, that's just a Timoni module deployed to our cluster. So now we can use Timoni to inspect our cluster to work out what Timoni has deployed. Now you said during your presentation that this is not storing secrets. So is this using, No. That's not in cluster. This is actually inspecting the module. Okay. So it has the secret. It creates a secret. And it doesn't store inside the secret any Kubernetes resources. Ah, Okay. Okay. I misunderstood that.

1:08:20 So if we run get secrets or we do have a Timoni deployment secret. Yeah. And it's a custom secret type instance. Okay. And if I look at this, we just get the instance base 64 encoding. And this is, again, I'm super curious. I always wanna know. So I'm just gonna decode it. Did I copy too much? Again. Hey. Why is it doing that? Copy. Do it twice because that always fixes it. Right? Yeah. Oh, no. I think it's actually just decoding it in line, isn't it? There we go. Yeah. So this is a JSON instance, the CR. All right. Okay.

1:09:27 Got it. So this is not a custom resource deployed to my cluster because we haven't deployed anything to money specific to my cluster. This is just a CRD like object that Timoni understands for what it has deployed to the cluster. Yeah. Cool. Alright. Yeah. I was I was I I my first instinct was, I should have a CRD and CR instead of creating a secret. I should create a custom resource. But that would kind of beat the purpose of creating a CLI. A CLI should just work. So then if you would, if I would store this information in a

1:10:10 custom resource, then you have to initialize Timoni on every cluster, so register the CRDs and so on. So I reserve that for a time where there will be a Timoni controller serializing this as a JSON and put it in a secret for now, I think. It does the job, and the UX is better. You just don't apply and delete no initialization of versioning of CIDs and so on. But you can inspect what's inside the secret with with the inspect command. And it will it will show you different things from what serialized there. Alright. Let's take a look at that then.

1:10:54 I can't copy and paste today whatsoever. Alright. So let's remove the namespace just because I didn't use one to make my life more complicated. And we see the digest, the name, the repository, and the version that was deployed. Nice. Yeah. And if you do inspect values values for the info, you haven't set any values, and it gives you the default values. Alright. I have put in the module as default. And here is what characteristic of Timoni modules compared to hand charts and everything. It comes with a specification for container images. So Timoni is aware of what images are

1:11:51 in a module, what tags are there, and if you specify digest, it will use digest everywhere. And the vet command of a module will actually tell modulators, hey, you should set the digest. It's empty. And you can also do, for example, Timoni status for the info. And this will also list the images. So when you do a status, it's about, Okay, what things I have deployed? Is my deployment available? Whatever, whatever. This is very different from For Helm, a status of a release is stored after the upgrade or install is done. Timoni does not do that because I think

1:12:42 an app is alive on the cluster. It can fail after an upgrade. So what Timoni status does is it queries the live cluster and does a health check for everything at this moment. Even if the upgrade worked like one hour ago, maybe now you are out of capacity and the things that you thought are ready at upgrade time, now they are failing. So status is a live command that goes through every single resource and uses a thing from the Kubernetes project, which is called kstatus. We also use it in Flux, which knows how to help check

1:13:24 built in resources like deployment spots and so on, but also custom resources which have already status condition. Okay. So this tells us the deployment is available in the replicas as one. If I change the replicas on the deployment to two, does it just say deployment is available replicas two, or does it tell us that there's draft from what we applied? Let's configure now podinfo and make it change the configuration for it from the Oh, you want to change it here? Yeah, of course. It will just delete replicas too. Right? Status does not do a dry run or Alright. Okay. K.

1:14:09 That's what I was curious. Okay. Yeah. Yeah. Got it. Alright. Sweet. Okay. So where's our status command? We've already kind of done that now. So then we have the ability to begin to configure our module by providing our own queue value. So to do that, we have a queue blob like this, and then we're just running our apply command again. However, we can specify dash dash values and point it to a CUE file. So greater values on CUE. And we can see what's going on here. So this is configured under resource requests, CPU and memory,

1:14:54 and then setting the limits to be the exact same. I'm assuming we could also do replicas three like so. You have to look in the readme. I don't know if it's replicas. Is that am I jumping ahead of something that's already in this? No. No. No. No. Alright. Well, I mean, even if I break it, it's fine because Timoni will tell me, right? Yeah. So I can just say we wanna run a test, let's do a server side, apply with a test and specify the values to be my values. So we've got a value change. The metadata

1:15:34 generation obviously changed. The spec replicas changed just like I was hoping it would. And we get this as a resources here. I love this visual output as well. This is very clear exactly what is changing. And this is really difficult to get with other tools in this space. So that's very nice. Hey. Did you actually saved value skew? I did. Yeah. Why? What can what why were you curious? Okay. Yeah. I I didn't see save it. Okay. You can apply now. Yes. Worse that could happen. Oh, no. I forgot my my okay. I shouldn't have removed

1:16:27 It's myself there. Sweet. And now it's waiting for our three pods to become ready. Yep. And if you stop this and you do a status, the status should say progressing in rollout gives you more details basically about what's happening. Right. Let's do that. So I'll make this a bit more challenging for the cluster. We'll do the apply. I think get to this point and then run our status. Tounding enough because it did it already. Yeah. Put info is very fast. It's tiny. Plus it's a local Kubernetes customer where damage is already built. So really, we're not gonna

1:17:23 They need to scan the single node. Yeah. Alright. Well, because I always am perpetually curious, Let's do something else. Right? So I don't know if it's in the the documentation here. I don't want us to drift off course too much. Right? But if I want to overwrite the image that is used for pod info, how would I do that? Can I do that for the values? Of course. Alright. So walk me through that because I don't know. Do I just do containers? No. Alright. No. Let's look at the default values. Okay. We got that from

1:18:07 inspect values and specter. Right? Yeah. Okay. I gotcha. So we can say image repose in fact, let's change the image tag to be a value that doesn't exist. Right? So image tag v twelve point one point two point three. Okay, so it has to be I suspect here, let's do it at first, let's not suspect anything. Once I update the image and it's gonna downgrade our replicas back to two. And now we can do an apply, set the test. This is gonna say that the image can come, but I wanna see this from it is

1:19:01 an m progress pending termination. Can I do a status? Is there, like, a watch or something on this? No. The the apply would would have waited and report that it can move forward. And this is not Timoni's future. As I said, it's case status, which determines, oh, the progress deadline for Podinfo is sixty seconds. So after one need to say, I'm not waiting anymore because, you know, there isn't much to to wait for, it's it's gone. It can move forward. And it yeah, it basically depends on the progress deadline that you set on your deployments. Right? Also, Timoni apply has

1:19:50 a timeout flag. I think I set the default to five minutes. But if you are you're deploying stateful sets that are taking half hour, you should set timeout half hour and just wait for it. Right? It's Yeah. I guess what I was curious about is is this going to surface the image peel back off? But I guess not. Not because for now, it's because the replica set, the deployment has not yet given up on the replica set. Okay. Right? It's initial back off. So it does a back off for, I think yeah. It also depends on what your deadline

1:20:36 is set in a deployment. I think I've set it to sixty seconds or ten ten minutes for Podinfo. Yeah. It's okay. I'll stop taking this off course with my my sassy kudos. No. It's great. So I I set it back. I removed it. I removed my customization, and we're now running the standard version. So we can run through the rest of the tutorial, see if there's any questions from the audience, and then I can go roll, give you one, and say, okay. Oh, you can you can do something interesting here. Let's see with Helm. If you have Helm values,

1:21:10 you can set some random thing that doesn't exist in the chart, then Helm will apply it, right? Let's specify a random field in the values that's not in the module at all. All right. Okay. Let's do this. Ah, values Rawkode field not allowed. So what these things does is basically you as the owner of the app, the author of the module, when you define what things end users can fiddle with with your app, the options, also the users have this error in front of them. Hey, you are setting something that wasn't configured is not allowed because that type of configuration

1:22:14 is not something that the module knows about. So there is no point in going forward. Other tools like Customize or whatever, specify a patch to something that doesn't exist, to just move and do its thing. What I think when you distribute applications, you should you know, ensure correctness of the configuration, not let the user think it has set something and it has actually changed, while in fact there's not even allowed. And that saves you from typos, bad indentation, and all of that. This is the power of CUE in a way. It's Timoni just surface surfaces that as errors be applied.

1:22:58 Nice. So I noticed that nothing in my directory at this point in time, because we're working with an OCI artifact. Like, these you know, we we can use the status and the values command. Right? Inspect values. And we can see what these values are. It's it's cached on a disk somewhere, like the queue definitions that people can look at if they want to, or is that where the bundles come in and where things start to get vendored into package or etcetera? If you want to see the default values, you say? Well, yeah. If I wanted to

1:23:36 see the actual definition for this to see what is allowed. Yeah. Or you can do Timoni mod pool. And I'm gonna need that string. Yep. Oh, I put path. Yes there. It's a Oh, yeah. It's okay. Help. Did I set the output path? Output. Okay. It should work with minus one. Because the directory just has to exist. If you put Ah, there we go. Yeah. The directory just have to exist. Yeah. This actually someone opened a pull request today to fix this. You can create a directory if it doesn't exist. Right? So now you can see the source. And usually

1:24:52 what you should do is look at the Readme markdown. We don't need Readme's. Right? You don't need Readme's. There is a table there, like for hand charts with all the possible configurations and whatever. But let's say you don't want to look at it. It means you have two things where you can see how the values are. So the default values are in values. Queuing the root of the module. Yep. And here are just images normally. If you want to see the whole schema of the values, you go templates config dot queue. It's the first file. And here you see

1:25:38 all the things that you can set with some defaults, with types. Right? So you have affinity, you have pull secrets, all the things that you would expect from, I don't know, Kubernetes is most relevant. Right? You need to specify all things and understand all things about it. So yeah, I don't think this should actually be the way people should write modules. I think you should expose you should put in there good defaults, and they should just toggle some things, like auto scaling enabled. And all default should be good. I don't know. 99 CPU or whatever.

1:26:23 Would I think the fact that we ship app configurations without good defaults, people are now accustomed to do a deep dive there and set half of the fields in all the Kubernetes things, it's like we put too much pressure on the engine. We should at least have a default secure first configuration with all things in there or production configuration and so on. And only request people what they really need to add as inputs, passwords, certificates, all these things. But yeah, the podinfo Timoni module is a replica of the podinfo chart. And that's why it doesn't have all defaults.

1:27:14 But I wanted to have this exercise like, Okay, I have this M chart. It's been used by, don't know. It has like millions deployments. Let me just replicate it and see how I can do this with Timoni. Yeah. Yeah. So let me clarify my thinking then just to provide a bit more context about why I was asking this. There's two reasons I was asking this. Right? Because I think it really surfaces why Timoni is so powerful and so good. Right? Is one, the CUE team are working on a language server protocol, an LSP. So the author and

1:27:48 experience can become a lot better when you pull the module locally or vendor it into your bundle. Because then when you're in Versus Code, it can auto complete all the types based on the CUE definitions generated from the Timoni, I can't remember the command. I think it was just vendor CRD, right? Which is very nice. But also one of the superpowers now that we're invested into the CUE ecosystem programme community, whatever you wanna call that there is that as we start to build out Timoni bundles that consume multiple modules from across anywhere, right? Where we're in a position here.

1:28:26 We can now use CUE and say, well, our org pod security policy, maybe you just wanna make this value like this, Could be blah blah blah, not that important. And then as we consume other modules, we can then just say pod security policy. It's our org one, but when you need to override it, you can do that. So you don't have like one of the huge problems with Helm is trying to apply policy across whale amount of third party external charts for deploying Postgres, MariaDB, MongoDB, operators here, controllers here, whatever. It just requires so much YAML duplication and

1:29:08 pain, but we actually have with CUE the ability to tidy up and make things actually consumable. And I'm sure Timoni has other layers into that, but we've not even covered yet. So again, I don't wanna take us too far off where we're supposed to be going through because we are running way over where we wanted to be time wise. So I just wanna check that you're okay for us to continue through the quick start and finish. Are you okay for time? Yeah. Yeah. Okay. Cool. Sorry. Me going off road is always gonna be a challenge.

1:29:37 So this is really cool. I'm I'm loving what we're seeing. So let's let's let's finish the quick start and take a look at the secret stuff in the production application of Right? And then I've not looked at the questions in ages, I'm sorry, in live viewers, but I'll check it in just a moment. But we're let's do our delete. It's all gone. Come back to our tutorial and now we can start to play with bundling instances together. And I'm gonna copy this because we're going to do a bundle apply. So let's close all this bundle

1:30:11 here. What this is doing is saying we wanna bundle going to call this pod info V one alpha one, where we're going to deploy a Redis module and the same pod info bundle that we've already deployed. However, this time we are going to enable caching and point at this. And we got this. And call mains the same as yours because terrible person. There we go. So let's handle like one of the practical questions that I'm sure it's gonna come from audience and this even in my head. Right? It's like people watch this, but they're holy shit. This

1:31:02 is amazing. I'm convinced. I'm their team Timoni. That's it. Let's go. Are they working with Timoni modules? Are they always working with bundles? Are they always writing their own bundles that consume modules? Like, I am dual blogs developer. I turn up to work tomorrow. I say, hey team, this is what we do now. Do they start writing their own bundle right away and consume third party modules or do they just deploy modules? Like how do you see that working? Let me give you a concrete example because that's that's very abstract. Right? I'm Dropbox developer. I'm going to work tomorrow. I wanna deploy

1:31:36 MongoDB and a web application. What's the first thing I do? Well, there is no Timoni module for MongoDB. So you could spend a lot of time writing that from zero. Then you have to write the module for your own app, and you should definitely publish those to Container Registry and use bundles and not rely on best scripts that are calling setting values to this file and so on. You should definitely use bundles and do a simple apply of that and have in the bundle everything. I'm guessing the PVC is failing because maybe your cluster does not have it. We knew

1:32:18 that was gonna I assured you it would be okay, and now I feel bad. I don't have Let's look at the pod info namespace and see what's happening there. Hey. I didn't specify a namespace to the bundle. You did in the bundle. Mhmm. Ah, right. Gotcha. Okay. Yeah. Okay. Yeah. So we do have one though. Why is the claim pending? Because you're not using the default storage class. It's not you, but, you know, it's looking for a standard default class rather than the default default class. Yeah. Okay. So we can actually change it. Oh, I'm gonna can I pull a bundle?

1:33:21 What? I would pull the module. Right? I do a module pull. Pull the reddest module. I wanna I wanna see if I can do this. Okay. So Timoni module tool. I don't know what's buying my my copy and paste just hits me today. Right? Output, Redis. And we know that this now needs to exist first. So make sure is not module pool. Mod. Alright. So let's go back to Versus Code. Then I open something else. That's okay. Templates config. So now I know Oh, don't have to scroll much. Yeah. So I can just set persistence storage class.

1:34:18 So let's do this up. I can tell you no queue because most people wouldn't know to, you know, wrap it in values. I have done a lot with CUE. I actually have a couple of tools I built on top of CUE myself, so I'm very familiar with it. Now my storage class is called local path. Now I could just leave this as blank, which would pull the default, but we'll do it this way. No. It should be oh, that's that's the name. Okay. Yeah. That's the name. But I mean, we could just force it to blank, which would

1:35:01 tell the cluster to use the default storage class. Let's let's let's try it. Right? Yeah. So let's just pull that and then we'll do our 20 bundle apply. And if we just come down here, oh, it's immutable. Yeah. Let's do a force apply. I was going to just delete the claim, but I guess this also works. And then if we do a GET PVC. So Timoni does, it has code that detects immutable errors returned by the Kubernetes API. And when you set it to force or you can also add annotations for those objects, it will recreate them. But

1:35:57 force tells Timoni, hey, look for immutable errors. And if there are any immutables, only then recreate the object. Don't destroy objects which don't have immutable changes to immutable fields inside. Force is quite special. Yeah. Well, even when I set the storage class, it's still failing, so that's my bad. But it's okay. We planned for this. I planned for this. And my that's a lot of downloads. And my downloads folder, I'll check the name over here to spare everybody the mess of my downloads folder. I have a CUE config file, which we will nicely put here where

1:36:45 I will export CUE config equals EWTKC. And we will do that actually worked. Worked. And I was gonna go to my backup cube. So get PVC. So it's just a slow oh, we got it. Nice. Awesome. And it also has end to end tests. So if we look a little bit at what the output of the bundle applies, you'll see that it goes into stages. And the module creators, the software vendors, they have control over stages. And they can instruct Timoni how to apply things in stages, do health checks, run end to end tests, then move to another thing. For example,

1:37:52 orchestrating a Redis deployment, in my case, is first you create a masternode with a persistent claim. You wait for that to be healthy. Then you create a replica set, the Redis replicas. And at the end, you run an end to end test to make sure that the cluster is in good shape and only then move to the app that uses what it is. Ah, that was so subtle. I didn't notice that, but you're right. Look, we have service icon config map and register. Then it has the replicas. And then I I I just didn't even notice the job with

1:38:31 the registers. Nice. Yeah. So what does that look like in TimoniQ? Is that in the this here? Yeah. Ah, so you can have multiple apply steps like so. Then you have conditions. If the end user wants to run the test and the test is enabling values only then do the test jobs. And how Timoni treats tests is yet very different from what Helm does. For Timoni, tests are part of the desired state. They are not some afterthought that they can exist. There is no test command. You don't run tests out of nothing. How tests are part of the desired state,

1:39:22 and every time, if you run now again the apply, there will be no test run, because the desired state didn't change. But if you change a value, then Timoni will regenerate the job and run the test because you want to run tests only when things change. So that's also a big difference in how it does it. Oh yeah. It's unchanged, right? But if you change something in your Redis configuration, the computed checks some of the desired set changes. And that tells Timoni, hey, I have to wipe out the old job, clear it out also along with all the pods, and run

1:40:14 a new test. So yeah, tests are quite different. Here in TimoniQ, where the module authors have the whole power of defining how the apply works, you can also set here things like, I want to apply this only if an object does not exist on the cluster. And this is how you can do the same stuff as have pre installed hooks. Because Timoni has no hooks. It's always an apply. There is no install. There is no upgrade. It's apply, right? But you can have same Timoni, apply this only if it doesn't exist or apply it anytime or clean it

1:41:04 up after you applied it. So that's how we can do prerequisites for an installation or prerequisites only for upgrades and so on. Alright. Quick question then. This I'm just curious how you look over these stages. Right? Because there's no specific ordering other than the order that you've put them inside of this file. And so does CUE maintain that order? Well, yeah, of course. Uh-huh. I actually didn't know that. Nice. Okay. Very cool. So I'm using the AST to work the CUE files. So if the file is one and the other, why wouldn't I execute it in that

1:41:51 direction? Okay. So you're not actually just rendering out the concrete value. You're working with the queue value itself as part of Timoni. So Yeah. There are many, many things that Timoni does around queue files. It's that Some files get directly compiled for other files. I'm working DST from other files. I'm doing unifying. Yeah. It's it was quite an adventure, but I like it in a way. Yeah. I learned a lot a lot of queuing. Alright. Let's tackle the question and then we'll do the secret stuff and then we'll finish up for today's session. So Daniel says, this looks so good. Awesome. We

1:42:34 glad you like this, Daniel. Why I'm always asking if we can touch on and I'll say we, but I mean you, Stefan. Could you touch on what this looks like for working with two separate teams, like a platform team and a dev team working with bundles as in maybe two separate repos managed by the same app from two separate teams with different concerns. Let's see if I can put that sentence together in a way that I understand. Do you know what why I'm always asking there, Stefan? So from a platform team perspective and dev teams or operations team, right Yeah. There are

1:43:12 several approaches here. I imagine the platform team will want to design Timoni modules for generic modules for node GS apps that have the right, I don't know, NGINX, sidecar, that exposes metrics, the node GS app container limits, all things in there. Right? Then dev team, when they want to deploy their own NGINX app, they will use a bundle. They will set their own images. Maybe they will do some little configuration of that deployment, but they will use in the bundle those standard modules created by platform teams. That's one way to do it, and you don't need

1:44:00 the repo. You don't need to look in the repo. The platform team will publish all these modules to the internal Container Registry. Right? And app developers, app teams will just use those modules from there. That's one approach. And maybe it's too strict. Maybe app teams want to develop their own modules, so maybe the platform team will only create these CUE libraries, which abstract away a bunch of stuff in Kubernetes, and they will do Timoni artifact push of that particular CUE library. And when the app team creates their own modules, they will do artifact pool. Okay, I want to

1:44:42 use all these components and I'm writing my own module because I know best how to configure, how to expose my app configuration. Then the operation team will create a bundle using either modules created by the platform team or the app team or the app team themselves. They will have good bundles for everybody to deploy and they'll publish those and operations team can use those bundles. It's like, it really depends on your organization structure and how teams are collaborating between them. I think Timoni and and and CUE modules themselves offer a lot of flexibility here. So, yeah, I don't think there is one

1:45:32 way of doing it, but it's quite flexible. The idea is that at the end, someone will create a bundle that will use all these modules created by someone else or themselves. Yeah. I'm gonna touch on that with my own opinion just a little bit as well, which is very much in line with what you were saying, but just for why it was a question. Right? It's like, there's two things that when you're talking about working with platform teams and dev teams. Right? There's two different constraints that are important here. One, the platform team is gonna say to deploy to

1:46:02 our cluster, you must do this. Right? There's policies, there's enforcement. And that's always gonna happen in the cluster via admission controllers of some variety and they will enforce that there. However, what I think why I'm always also touching on is that you may be in a position where if you can get buy in where the dev teams and the platform teams are using Timoni, the platform team for cluster add ons and the dev teams for their applications. You might get to a point where maybe the platform teams are building some best practices that people can consume.

1:46:30 And I think those could be Timoni modules that are versioned and you can opt end to those defaulting behaviors for security context and policies and so forth. Even though there would be a higher level of mission control somewhere. So I think you you can just the platform team or even the dev teams maybe they do it, right? Maybe you're Amazon and there's 5,000 dev teams working on 10,000 microservices and you write out Timoni modules. This is a good deployment strategy for our customers and you can opt in to using that behavior if you want. And because everything's versions and

1:47:00 there's digests, yeah, you can opt into that. So from an opt in perspective, use the module system for any enforcement. I think it has to be admission control. I don't think Timoni would set in that pipeline whatsoever. Would you agree with that? Yeah. I no matter how much policy you you can put on a client side, Indian someone can do a KubeCatulated right on the cluster. So you actually need, unless you go full on GitOps, no one connects to the cluster at all. There is no way to work around having webhooks. And webhooks are like

1:47:42 evil. They will bring down your whole cluster, are single point of failure for Kubernetes API. The more webhooks you have, the slower everything gets. Dry runs are really, really crazy hard to do with webhooks and so on. But that's where we are. That's the reality. Right? Sadly, yeah, we need to run these webhooks, and when they go down, they take everything down. And good luck saving a cluster if your webhooks are are in the crash Yes. We've all been there. Why I wanted follow-up on our question. So they said they didn't think about sharing via

1:48:22 bundles, but I hope that is interesting path two. But they have said, how do we guarantee that the dev team always use the latest bundles from the platform team? Is that possible? The latest modules maybe. Yeah. The latest modules. So, you know, if the platform team does provide a module, is there any way for them to make sure the dev teams are always upgrading to the latest and greatest? No. No. Yeah. Unless you don't specify a version in the bundle. And every time you do an apply, Timoni will look which is the latest and and it will pull latest. But if

1:48:55 you use a digest or a version, you pin that module to a particular version. Of course, we can roll out with latest everywhere, every time you don't apply, it will be latest. But, yeah, I'm not sure is that the best approach. No. I I I don't I don't think that's a good pattern. I'll go back to what I said. Like, it's when you're using these Timoni module, I I feel like you opt in to the latest version because you're comfortable, you've tested it, you're happy with it. If any enforcement has to happen, it has to be in mission control.

1:49:27 So I wouldn't do that this way. But we hope that helps. Alright. We've got one more thing to show, I I believe, which is let's mildly productionize this bundle and do some sort of secret injection. So you did share with me a guest, which is here, which let's just copy and paste it first, and then we can run through on why or how this is different. And this actually also correlates to another question, the why I'm all asked at the start of this session. So we will tackle that too. So my bundle. Queue, that's in the right

1:50:03 one. Now here you were specifying that we have some value for a password using the queue attribute where we're saying that this password will come from a runtime string, which is assuming either a flag or an environment variable on the Timoni CLI. And this is then used just to augment the values. And it's also interpreted down here to the Redis URL. Is that the only changes in this file? Yeah. Yeah. I think there are more more things. Yeah. You can delete test enabled through. I don't think that was in the original one. Yeah. Delete that's the test for

1:50:45 Timoni for you could disable test for Redis. So we only Yeah. So what's important here is there's a secret value that you wanna inject into your Timoni application. We're saying let's get it from runtime. That's all that's important here. And then we use that to set the value on the Redis deploy and interpolate it into the connection string for the pod info. So let's copy the command just down here. And I think there are multiple ways to do this, but I'm I'm guessing right now. So let's check. Right? So if we paste this, what we're seeing

1:51:27 here is the reddest password is test, and we're doing a bundle that on bundle dot queue and run time from env. So are there multiple ways to provide runtime values beyond the environment? That's my first question. Yeah. You want to do yeah. That is for very dynamic. You want to do an apply instead of that. Yeah, so what we have here is typically for a CI system, right? You have some I claim. Claim. Yes. Yeah, so in a CI system, right, you can have these secret stores. I know, GitHub has one. GitLab, everybody has it. Right? And

1:52:24 what you can do is read from the CI secret source, set an environment variable from that thing, and then pass that through the environment variable to the Timoni apply command. And Timoni does not read environment variables by default, so you need to actually tell Timoni, hey, use the runtime from the environment, because this is something specific to local testing. Say, read this password equals something, Timoni apply, which that's just an environment variable setting, right? It's not something specific to me that's from the command line. Or we can do an export and so on. But if you are

1:53:14 applying from CI, you can pass in this way secrets through the environment and inject them in the bundles. Because the bundles usually where they are stored in the Git repo, you shouldn't be placing any sensitive information in there if there is a way around it. If you want to place sensitive information in there, there is documentation on Timoni website how you can use SOPS, the CLI, encrypt some JSON or YAML file. And then when you do a bundle apply besides the CUE file, you can also tell Timoni, hey, read from this YAML file that's decrypted there in place by

1:53:54 SOPS and then removed and goes through the memory and all the SOPS operation model. So this is one way of doing it, right? But when secrets are in CI. But usually with Kubernetes, you'd have some external secret controller, or secrets are arrived into NTCD from Vault, or stuff like that. So in most cases, you will have these secret values in some resource inside the cluster. How there is a way with Timoni runtimes to tell Timoni, hey, query the cluster for a Kubernetes secret, extract this field from here, then use this field as the runtime value

1:54:45 inside the bundle. Right? And we can actually set the the password now from from the cluster. So you should go back to the gist. You will have to create a Kubernetes secret. So this has created a secret in the default namespace. Now we have to create this other file, which is a runtime. Runtime definition. Why is this different? Because runtimes can be reused across bundles. If you have many bundles which have to connect to the same Redis cluster or whatever, use same thing. You can reuse the runtime to not write the query every time. Yeah. There is an error because diff does

1:56:05 not work on immutable things, but it has changed the password here. And also, if you go up, you will see the new password for the Redis replicas and the master things. Nice. And this basically has queried the cluster for the secret. It went to the secret and this is how you can specify that query. Yeah. This is one of my my favorite features. Like, coming back to that whole platform dev thing from the question earlier, you know, like the platform team goes on to hopefully puts out a lot of effort to provide in the platform with all of the

1:56:48 end cluster add ons that you need, the external secrets operator hooked up to KMSs or vaults, you know, DNS records, all those external bulbs that you need for your application to actually run successfully and being able to do this sort of dynamic lookup and inject this into this money runtime. I think it just simplifies a whole lot of what we're trying to do, especially when you start getting into get ups and you've got controllers and things like that there. Because I mean there is a Timoni controller right? That runs all of this in cluster for you. No. No, I thought there

1:57:22 was. Alright. That's next week's mission Stefan. All right, this is awesome. Very, very cool. So I think we're showing everything right? I'll stop sharing my screen. Right, cool. So if anyone has any last minute questions now is your time to get them in for Stefan before we wrap up. This session has gone on a little bit longer than we did anticipate, but I've had a whole lot of fun. So just, thank you so much Stefan for joining me and guiding us through this. If you're happy to, well, we just wait a few minutes for any last minute questions. Sure.

1:57:58 What what's next? Like, what's what's on your roadmap for for Timoni? So I have this proposal around multi cluster deployments where the runtime definition that you've played with, the size values will also have a list of clusters. So you can run a single apply command that will target all these clusters, and it will get values from each cluster, and so on. That's that's one of my things that I would like to do, next when I have the time to work on it. Yeah. And slowly, want to know it all depends in the end on the

1:58:45 amount of people using it and the feedback I'm getting for it. But I would be like, I don't know, Q1 next year to stabilize the API. Everything is V1 alpha one right now. I think I have a good I have some good concepts implemented there. Could just label it v one, beta one, but I I yeah. It took us three years to make Flux version two GA, and I'm not the type of person that rushes into labeling things GA or stable. So I think I will wait more before getting more and more users, more feedback, and

1:59:29 and stabilize the API. Only after the API is in stable shape can be a discussion around the Timoni controller for Flux, the Timoni controller on its own or whatever whatever the options are. Also, around controllers, I I really don't like the idea that you would be you'll have to write the bundle in a YAML file because KubeCTL apply or the API server does not understand CUE. It only understands YAML or JSON or Kubernetes objects. So maybe I'll write my Kubernetes API server extension that you can do on that. It actually understands CUE, and you don't need to

2:00:18 write YAML at all to receive the CUE and do the things there. I'm not sure. There are so many ideas floating around. But yeah, at some point there will be a controller. How it will look? What I know for sure is that I don't want for the custom resource to be a representation of an instance. I want the custom resource to be a representation of a bundle, because that's a proper app definition made out of many pieces with things, inputs from the cluster, and so on. So I think that would be, the API object that that Kubernetes will will

2:01:02 deal with. Yeah. I know not to go too far off track. Right? But going over to the Flex CT stage of things, there was a discussion, I think, some point last year about potentially allowing the customized controller to duct tape the sources. Is that something that's still being considered? Do you know what I mean by that or should I wouldn't worry Duct tape. The source. Duct tape. Sorry. It's my Scottish accent. So you know how the customized controller has got hard coded references to like the OCI repository, the GET repository, etcetera. Like if we could just say it accepts anything that has

2:01:37 these sets of fields on this on the status or spec and it doesn't have to be one of those, you know, duct typing. Right? It just it looks like a source provider, but it does it's not one that we know of. Would that simplify things for a Timoni controller where you could have a simple controller that monitored the OCI repository, run the Timoni command in the cluster and spy out a new source? Like, that that seems like it would be a nice approach. No. No? Definitely not. Alright. I'll show that. That's quite something that's about customized. It does patches,

2:02:08 which shouldn't be in any way something that you deal with Timoni. Right? Timoni has the stage apply, all of that. Right? It should be the Timoni reconciler that sets the cluster state, not customized controller, which does customized things. And also those plain Kubernetes YAML manifest was the part I was thinking about hooking into. Right? It's just have a Timoni controller that just spits out the the YAML for the customized controller to apply. That's already how you can do it. You do I can't do it in cluster, right? I would have to I would have to run

2:02:45 it locally, generate the YAML, and push it somewhere, right? Yeah. Yeah. You can run a registry in cluster. Very tiny. There are many registries out there that can run-in cluster. But I think a Timoni controller should be an applier before anything because it has its own mechanisms which are very different from Flux customized controller or Flux Helm controller. Yeah. Well, we can schedule a livestream for how long do you need to write that? Two weeks? Three weeks? Don't know. Yeah, maybe. The API is all set, which is the most important thing right now. And I also like to get

2:03:31 good feeling of is the user experience for authoring CUE modules good? Are going people are people going to write modules? If if they aren't and CUE is not appealing to them, why would I continue this project? I'll just archive it and move move forward. It's very tied to the language. Alright. Yeah. I'd be very sad to see that. I know the CUE team are actively working on their language server implementation. I know I spoke to Paul Jolley recently, and he said that that was his next job is to improve that authoring experience for people actually rating CUE. I know they're also handling

2:04:12 the module stuff, but you covered that earlier on in the session. So. Yeah. I I mean, for me, the fact that you can share, bits of CUE easily is not the top problem for me with CUE. My my my major issue with CUE is definitely a lack of a language server. I cannot do go to definition. I don't have auto completion. I don't have anything. It's all the things that are supposed to have because everything is type safe. Everything is so nice. Like when it comes to the editor, you are in V mode with nothing else,

2:04:46 right? Not even yeah, I don't know. Even writing YAML has auto completion now for Kubernetes in Versus Code and IntelliJ. CUE doesn't even have that. So, yeah, unless they get there, I don't see how people will enjoy writing CUE. Yeah. Also, Copilot does not understand CUE, which is kind of fun. It'll get there. These are all easy things to fix. I hope. Right? And I I hope these will be fixed quickly and in six months time at the next CUE con in Paris, can all Yeah. Have a laugh and see things are much better now.

2:05:25 But but but we'll see, you know. Yeah. I think we have any more questions from the audience. So I'll just say again, thank you so much. Like, I I think that's one of the most interesting projects in the CNCF landscape right now. I hope people, one, like what they're seeing and two, start going out there doing tutorials, blogs, videos and contributing wherever possible because you know, these open source projects need momentum to keep it interesting for you to be doing all the heavy lifting and hard work. And I hope that other people come along to help you with

2:05:55 that as well. Thank you again. Thank you very much, David for inviting me. It was like, this was my first ever talk on Timoni and I really enjoyed it. Awesome. Well, thank you all and feel free to leave comments after the fact. I'll do my best to route them to Stefan as they come in and we'll see you all next time. Have a wonderful weekend.