0:42 Introduction & Housekeeping

0:42 Hello, and welcome to today's episode of Rawkode live. I'm your host, Rawkode. Today, we're gonna be taking a look at CUE, a language for validating data amongst a whole bunch of other things that we're gonna dive into in today's session. Now before we do that, this is a little bit of housekeeping. First, if you're not already subscribed to the YouTube channel, you know, please feel free to do that now. Click the bell and you'll get notifications for all future episodes. We stream weekly and loads of various cloud native and Kubernetes content. So hopefully, we're for

1:13 you. If you're not watching live, I would encourage you to join our Discord server. There is lots of good conversation and chat around cloud native and Kubernetes in that channel, and we also talk and suggest new tools and technologies to cover on this show. And lastly, I wanna thank Equinix Medal. They are my employer, but they very kindly allow me the time and resources to invest in this show producing cloud native learning materials for us all to learn together. If you wanna try Equinix Metal, then you can use the code, the raw code. This will get you $200 of credit.

1:45 You can spend that very quickly on fifty hours of compute on a beefy instance or get up to four hundred hours of compute on something a little bit more modest. Check it out. Let me know how you get on. Now today to dive into CUE, I am joined by its creator and maintainer, Marcel Van de Hausen from the Google team. Hey there. How are you? Thank you for having me. Good. I'm doing good. Thank you for having me. It's always great to be able to, you know, demonstrate what I've been working on. So appreciate that.

1:54 Introducing Guest: Marcel van der Kouwen (Creator of CUE)

2:12 Yeah. I'm really excited. I'm a a big fan and almost advocate of CUE. I feel like I'm always telling people to go check it out, and I'm really looking forward to this, just kind of having a bit of fun today and showing people how cool of a a technology that is. Cool. Before we get started, why don't you just take a few minutes to introduce yourself, and then we'll we'll start and tell people what CUE is. Yep. So I'm Marcel van Lowhausen. I'm employed at Google. Been working there for a long time. Was on the founding team of Bork and

2:44 started out in the search engine team. And then after a few years, ended up on the GO team. Went hung out there for a while. And then a couple years ago, I started working on CUE and, you know, to sort of realize some of the vision that we had earlier, really, in the early Borg eras, but but never got to, if you will. So and that's how I ended up here, I guess. Nice. Awesome. So I believe you're gonna do a little bit of a a presentation just to introduce us to CUE. I would encourage

3:11 What is CUE? (Presentation Start)

3:17 if you wanna just get your screen ready for that, we can move into that. I would encourage everyone watching to, you know, get your questions and send them over to us. We'd happy to answer them as we go. And there will be a hands on section afterwards as well. Yep. You get to play Fender Window. Right? Yeah. I think I've I'm there already. Yeah. Yes. There we go. Perfect. Thank you. Yeah. So I hope alright. Great. Yeah. So I'll get right to it. So, first of all, what is CUE used for? So it's really an offspring of GCL.

3:50 History and Inspiration (GCL, Borg, Kubernetes)

3:50 So JSONET itself is is an offspring of GCL. But, rather than focusing on just data configuration, it really tries to cover the entire configuration space. Right? So things that has been used for is is, like, fact discovery, workflow definition, configuration, data templating, even cogeneration, policy checking, test generation even, and Istio uses it for open API generation, for example. So a little bit of the history behind it. So, as I mentioned already, I started on the you know, I was on part of the initial Borg team. And one of the reasons I ended up there is, because I worked in this little

4:34 project before Borg, which was sort of like Borg, a little project, little system that I created was like like Borg. And so already, that system had a lot of knowledge in it from how do you deal with configuration. So some of it was, experience from before that that Google had on, like, how do you you know, what should you not do, especially. But also from my previous employer, you know, we worked there with configurations. There were hundred thousands of lines. Managed and distributed teams, you know, purely declarative and was fine. Right? So that was a good example of of

5:11 how, you could do declarative configuration and how powerful it is. Right? And this was basically the, idea we wanted to put into into, Bork as well. So this is what, GCL, was originally. However, we made some errors with GCL, which made it, sort of not possible to realize that original, goal anymore. Right? So this is one of the things that's that prompted CUE, right, to to do to do it again right, right, to really get these promises from from, declarative configuration management. So, also, a lot of things changed in in this time. Right? So the Kubernetes

5:54 team, which is sort of an offspring from of Bork, they did their own research in configuration, and they drew their own conclusions. Somewhat different, but, you know, very valuable lessons to be learned. And I tried to also incorporate that in in the in the design of CUE. Right? And then, of course, also my Go experience, you know, how do you deal with with programming languages and languages in general, had a big influence on how it end up looking now. Definitely looks very different now than if I'd written it fifteen years ago. So some of the key takeaways from all

6:30 Key Takeaways & Design Philosophy

6:32 this knowledge that went into it, I think the by far, the biggest one is, like, never ever use overrides. Right? Just just don't do it. To qualify a little bit better, like, you can have one level of overrides. This was already the conclusion we had before this this very early system. But more than one layer, it just gets too complicated. Right? And it doesn't matter if you have file overlays or inheritance. Just don't do it. The other thing is you really wanna separate code from configuration. So this is another thing that went sour in GCL.

7:05 So there you see that that there is actually a lot of com computation in in GCL in the end. So the problem is here that we thought, you know, you could always separate it by pushing the computation into the binaries, but then that's kind of a, you know, too idealistic view. And in practice, that doesn't really work. So you have to deal with this reality that on the one hand, you do wanna have com computation at the configuration layer, but you still really wanna separate the two. Something that came in later, more from from Brian Grant and

7:38 folks from the Kubernetes land is the the whole idea that parameterization is is, even though abstractions can be good, generally, parameterization and configuration is bad. So CUE has a so I agree with that, but CUE takes a different approach in solving that problem. So one other thing to mention here is that if you look at configuration, how it develops within a a company is that, well, first of all, the best thing is to have no configuration whatsoever. Right? Like, in the ideal scenario, you know, you have maybe one parameter to tweak, and that's it. Right? Like, you you

8:00 The Configuration Problem Cycle

8:17 you you don't tweak anything. So in practice, what happens is that, you know, systems develop, you know, more and more things that you can configure. So at some point, people start pulling it out of the system, you know, then then and into individual configuration value. So that gets hard to manage, so people start to develop systems around it. Ultimately, that's it's still too inflexible, so people end up with DSLs. And then these DSLs end up being so complicated that people think, like, well, if it's it's so complicated anyway, why don't we put it in code?

8:50 And you get back to the top. Right? And the whole cycle repeats itself again. So if you look at what's, you know, this is a a a phrase taken from cloudgoogle.com describing, the whole notion of, you know, like like, configuration as data, or infrastructure as data, rather, is that, you do want to be able to, record everything in a single version control. So it's kind of nice if your configuration is text. Doing it declaratively has a lot of benefits, especially because it makes the contracts between all the boundaries of your system very clear. So if you think about this clock and,

9:32 like, considering all these options, like like, having something text based and declarative is is still the best option if you can do it well. Right? So the problem is, how do we avoid this, complexity? Right? And this is one of the main aims of CUE to make this, simple, right, and and do not get into the same trap as many other configuration languages where there's too much computation, too much, complexity in general. So, so let us take a big step back. Right? Like, what is configuration? So you have templates, data, validation rules, policy. Right? And and, really, if you think about

10:03 CUE's Approach: Unifying Configuration Aspects

10:13 it, all of this is spread, across many different components. Right? So you can have a person or or team that provides a system. Right, the system itself is configured, but then also, users of that system can provide additional configuration, right, which then has to be combined in a certain way. Then also, this can come from multiple customers. That system itself might use another system, all of which have config configuration that's somehow related to each other. Right? And if you try to combine this, it ends up being not that simple. Right? So, really, it looks a little bit like

10:49 this. Okay. So you have some some data that needs to be filled out in templates, and then you have validation rules, but these validation rules only work on concrete data. So you have to make sure that all the templates are filled out in the right order, and this can be across systems. And it just gets this really big mess. Right? Very complicated. So and this happens at all different levels. Right? So combining data. So what happens if you conflicting, values? Right? Like, how do you override it? Like, what's the order? Same thing with rules and data or, like,

11:11 Challenges with Traditional Configuration Management

11:24 what comes first? What data do we need to evaluate first before we can even apply the rules? What if you can't get the data at all? Right? Like, how do you deal with with incomplete data? You have different sources, JSON schema, open API, like, all this stuff. Right? And then to make matters worse, so configuration tends to be a cross cutting thing, right, where you really don't have a specific order in which things get combined. Right? It really comes in from every different angle, and there's very little, control you have over this. And in order

11:57 to manage that, then people tend to, like, dumb down their systems or or sort of restrict the configuration in a way that that reduces the functionality of the system. So how does CUE address this? Right? So the idea with CUE is that, okay. Well, let's get a grip on configuration and, you know, create some way of dealing with configuration that you actually can throw it all together on one big heap. So, you know, you don't have to worry about the order of application anymore. You just dump it in all one big bag, and CUE figures out,

12:08 CUE's Technical Foundation (Unification & Logic)

12:31 how to combine it and how to do all the validation and and data production. So how does CUE do this? So there's a few things that change. So to the user, now everything has to be done declaratively. Right? Like, a big part of CUE is that it unifies rules and data, and, like, all of this is the same thing. Right? So to CUE, all these different aspects of of configuration are unified into a single space, if you will. So I'll let that sink in for a moment. Right? And and I'll I'll come back later to what does that

13:06 exactly mean. But this is basically, what CUE does. And to the user, what it means is that CUE puts some restrictions on what you can do with overrides. Right? So you have basically default values, which you can see as an override. So that's one level. But after that, you're done. Right? Like, once you set something to a specific value, you can never override it. So Dareit limits you, right, like, more, more than other systems. On the other hand, it gives you a lot more flexibility and composition. Right? So where you wanna combine things from different angles, you now can,

13:38 whereas whereas previously, you had to specify a certain order. So this is to the user how things change. So for what the CUE, the way it handles this is basically defining a very rigorous model on what configuration is and then having a logic reasoning engine underneath it, you know, to to deal with it. And it sounds very complex. Right? But a good analogy about this and it is. It is actually very complex. But the idea is that this complexity is a complexity for the implementer, for the purse the people that write CUE, but it serves to make the life of

14:16 the users easier. Right? It's very much like a garbage collector. Garbage collector is very complex to implement or can be if you do it right. But the whole purpose of it is to make memory management much easier for the user. Right? So this is a little bit how you should think of this. Next slides. There we go. So another way to think of CUE is, so if you have a spreadsheet, right, like you have all these cells where you can put in formulas and you can refer to other cells, so CUE is really like a,

14:42 CUE as a Spreadsheet for JSON

14:51 a spreadsheet for JSON. Right? Rather than having a matrix, you have your JSON tree, and you take base basically individual notes and express these in terms of other notes. Right? And then there's this logical reasoning engine that that helps you execute that. Right? So that's really another way, to see what it is. So CUE is not just a language. It's a whole ecosystem. So there's the CUE command. There's all kinds of, higher level libraries that allows you to specify workflows. There there's all kinds of adapters to translate back and forth to different, formats, API generators,

15:11 The CUE Ecosystem

15:30 policy engines you can build on top of this. Right? So there's all kinds of tools for injection and how you combine different piece of data, different levels. There's refactoring tools, formatting. So CUE is very much designed for automated generation also. So the the language is, designed for machine generation and readability. So that's how CUE looks like. And when when you think of it as a language, right, like coming back to the DSL side of things, when you think of it as a language so what requirements do you have as a configuration language? Right? So people,

15:52 Configuration Language Requirements

16:08 might be aware of the difference between a scripting and a programming language. So a programming language, is much more focused on readability, can be a little bit more tedious to to, to write. Right? But really, the point is that it's correct and that people can read and understand what it what it does. Right? Because, a programming language is maintained by multiple people. Right? Like, sometimes it has to be, read again over a long period of time. Even the original author might have forget exactly what it does. So the same thing holds even more for configuration languages,

16:42 because now it's not only, you know, often it's written by the original team that maintains a piece of software, but very often, it is then maintained or changed by somebody outside of that team. Right? And the type of people that that work on that might not be the peep you know, might not be the same kind of group that is familiar with the original language that that product was was written in. Right? So so think of an SRE team. Right? You cannot expect them to be, you know, very familiar with every language that all of the systems they maintain is written

17:16 in. Right? So you need to to have something that's even more familiar, more readable, and and simpler to understand. And this is complicated. Right? So this is a little bit what they what they try to do with Go. Right? And the whole idea is that simplicity is complicated, and it is. But, ultimately, it makes life of everybody easier. So what does CUE look like? So I mentioned already before that CUE, unifies types and values and constraints, policy validation. It's all the same thing. So if you look at the bottom, you basically have, in this this Moscow,

17:42 CUE Syntax: Types, Values, Constraints

17:56 entry, you basically have JSON with syntactic sugar. Right? Like, if you squint your eyes a little bit, it's just JSON. You have key value pairs. So if you look at the top, it looks more like a ghost structs. Right? Like, you have a a the name of the structs, you have the fields, and then the types. Right? So in the middle, you see something that is smack in the middle of these two. So you have a large capital. So the name is, again, a string. A capital is now set to true, fixed to true, and

18:25 population is set to be greater than 5,000,000. Right? So here you see this whole spreadsheet, analogy coming back as well. And, really, you can mix and match, concrete values with types, with constraints, however much you want within CUE. Then CUE is also aspect oriented. So what that means so one way to view CUE as well so remember that I said it's just, sort of a spreadsheet for JSON. So if you generalize JSON or if you wanna see what JSON really is, right, like, it's you can see it as a tree of of data. But another way to view it is a

18:43 Aspect-Oriented CUE & Path Selectors

19:03 path, with some concrete value at the end of that path, right, and then a sequence of that. So you have it's basically a sequence of path value pairs. Right? And, basically, what you can do in CUE is the value doesn't have to be a concrete value. It can be any type. But, also, the path doesn't have to be a single path. It can be any group of paths. Right? So, for example, if you look at the this last name example here, so we can say last name is Jones, so you have the path and a concrete

19:33 value. I can say the last name is a string, which means it's of a certain type. I can also say it's of a type, you know, of a certain subset of strings, namely this any string that starts with a capital letter, which doesn't even work for my last name, but, you know, it's just an example. But what you can also say is that every field that ends with name, right, I wanted to start with a capital letter. So here you see a group of paths, combined into a, you know, assigned a single type. So now I'll so first of all,

20:10 Q&A on Presentation

20:11 let me ask if there's any questions or if this was clear. I'll stop sharing. Yeah. It was definitely very clear. I mean, you also gave me, you know, the fear that I've been doing configuration wrong my entire life. But, you know, I think there's a lot of, you know, wisdom in in those slides and and, you know, really, your articulated the problem that we have very well and how you hopefully CUE is gonna come in here and and kinda help us do this all better. So that was really great. Actually, even while you were talking, I was

20:40 tweeting, I loved that line that inheritance is questionable in programming, but it's fatal in configuration. Like, I just think that was Yeah. That was perfect. Yeah. Yeah. Yeah. Yeah. I mean, in Go, you see they're already starting moving away from this. I I mean, it has inheritance, just no subtype inheritance. Right? But, another way to see it with inheritance is so one of their critiques of CUE is that people say, you know, well, this is too complicated for for software engineers because they're used to inheritance. Just use inheritance. Right? So so maybe that's true. But for anybody that's not a software engineer,

21:13 inheritance itself is kinda crazy. Right? Because, basically, what you're saying is, okay. Let's say we have a cat. Right? And now we wanna create a dog. So, okay. Let's inherit from cats because they're kinda similar. They both have four legs and stuff. So we just override a cat and say the the nose is now wet instead of dry and it barks instead of meows. And now we have a dog. Right? So to any normal human, that would be absolutely insane. Right? And and and it kinda is. Right? So the way, so the way you solve it in CUE

21:42 is to say, like, okay. You cannot have a dog inherit from cats. You create an animal. Right? You can still have a hierarchy. Right? But you create a a mammal. Right? And then you have a a cat and dog, instance of that. Right? You make it always more specific. Everything you do is making everything more specific. And that's that's really the basis of it. Yeah. Awesome. I think one of your friends and maintainers has joined us. Paul Jolley says, CUE sounds like fun. Hey, Paul. Yeah. And yeah. So let's I know we have no questions right now. So why don't we dive

22:16 Hands-on Demo: Basic Concepts & Combining Data

22:16 into the demo? Again, if you're watching and you have questions that you want us to answer, please drop them into the chat, and we will get right on to that. I will pop your screen up with the code. Yep. There you go. Take it away. Yep. So yeah. So I just wanted to give a little feel here for what it means you can just dump things together, right, with CUE. So so here I have a little directory with nope. Wrong directory. Good thing I checked. So here I have a little directory with a bunch of files.

22:48 And so let's look at a n dot JSON and b dot YAML. So here you see two files. They both define a restaurant's, you know, top level restaurant map with, you know, name, you know, like like, basically, a name of a restaurant and then some structs describing the restaurants. Right? So if I do queue exports, a dot JSON pretty much get what I saw there. Right? Then I can, combine it with, with the YAML one and basically, well, let's do the output in YAML, a little bit shorter. Demo effects. Alright. It's my Go heritage. Still not used to

23:35 this. So, basically, what you see is it combined these fine into a single, single map. Right? Because there's basically no overlap, and everything is fine. So so this provides you a way of combining, you know, like JSON, YAML, whatever whatever you want. So now the next question is, if I look at, for example, I have another file here. So what if I combine these two files? So what would happen? Alright. So let me do that. A dot json, e dot json. So basically, happens is complaints complaints because, it's the same restaurant with the same key,

23:53 Demo: Introduction to Schema & `cue vet` (Validation)

24:20 and but they both define this cuisine field, right, with a different value. And CUE basically says you cannot do that. Right? Like, that's the the cat and the dog issue again. So so, basically, it says resolve this. Right? Like, I cannot resolve this. So another thing you can do so let me open so the schema here. So another thing you can do is apply schema. So here I have a JSON schema in written in YAML. So what I like to do with JSON schema because I find it kinda hard to read, I can basically, turn it into a CUE schema. So CUE

24:59 reads JSON schema natively. It uses this this schema definition and some other, clues to to figure that out. And this is essentially how the queue looks like. So I have a top level restaurant map, and then which is, you know, of type restaurant. There's a cuisine and a list of tables, which basically describes, you know, like, how many seats and whether it has a view or not. So very, very simple schema. So now I can, basically, I can use the schema to verify if, my my values were correct. Right? So I can do it with a dot schema. It's

25:39 a correct restaurant. B dot schema was correct. But now I have here, I have so let me open a verb first. So I have a code. Was it this one? I think so. So here I have another YAML thingy. So I can do a vet, so I don't wanna see the output. I just wanna validate if it's correct. So I can run it, and now I get a, you know, error, which is correct because if I look here, right, the maximum is 10. So here I have 12. If I fix that and run it again

26:16 so that's fixed, but now I have another problem. So this is what's called an incomplete problem. That's why I wasn't showing it immediately. So the other was in hard violation, but now it's something that could have been solved if I provided more information. So the problem here is if you see that seats is actually a required attribute, but it's not specified for the last item here. Right? And, really, the problem was that there was an extra dash. That's really what what it was going on here. So if I run it now, it's fine. Right? So similarly,

26:50 if I so open, for example, another schema here. Let's open the so let's open the already have that. Let's say I have these two schemas. Right? And, so one is an older version of the other. So let me let me, run this again. So here here, you see the queue is a little bit easier to read. So if I now combine that with the, the later scheme I had, so what you see is the the old schema basically sets the seats, you know, every table seats at least four people and at most 12. Their newest newer schema, you know, they'd like

27:43 to have a little bit more space. So it says, you know, it seats maximum 10, minimum two. So if I combine these schema, it takes the intersection of that, and it's greater than four, smaller than 10. Right? Like, you can use this to also check backwards compatibility of schema and all these kind of, kind of things. So let me move to to a next demo here. So open these files again. One thing that makes CUE work very well, and this was already a realization that twenty years ago in sort of the the ancestor formalism that CUE uses,

28:23 is that if you have a a schema validation oops. So I have codes. So, can I do two things at the same time? So if you if you have a, so if you have a schema for something that really also, at the same time as a template, right, that you can use to reduce boilerplate, really these two things. If you have to write formulas, and these two are the same things. So, so one common use case for for CUE is that you say, okay. We start out with this whole bunch of JSON or YAML, whatever you have.

28:31 Demo: Schema as Template & `cue trim` (Reducing Boilerplate)

28:59 Right? And now we first wanna check if it's actually correct. So what you do is you start with the schema to sort of that captures the rules and sort of your understanding of what all this code is. And then you start to, to run this against your your YAML and JSON and see if it's actually, you know, is the way you thought it was. And then very often, find tons of errors. Right? The first time I started applying this to my own JSON, I found and and YAML, I found tons of errors. And then once you've eliminated all these errors,

29:29 you can use the same schema to then say, like, well, you know, now I actually want this to be, you know, like, reducing boilerplates. Right? So it basically does two things at the same time. So let's see what's going on here. So what you see here is not Kubernetes. It's a simplified form of it, so it actually, you know, fits within within one window here. So but, you know, it's similar. So what we have is I can say I have a map of deployments, keys by their name. Right? I say the kind is always deployment.

30:00 Their name is always the same as the key. So replicas, I have a small service. So, usually, I only have one. Right? So that would be the default, but it can be any integer. So there must be an image, right, of string, and I have a few selectors where I say, you know, can be more than that, but at least the the the app should always be the same as the name. And then env, I now hardwire to prod. Could be, you know, one of a choice of a few things. Right? But for now, I hardwire it. So same

30:32 thing. We have a a service doing something very similar. And then here, this is an interesting one. So in this setup, let's assume that for every deployment, we have a single service. Right? And so what I'm basically saying here is that for every deployment and the way this this comprehension works is that this is like a yield. Right? Like, the final curly brace is a yield, and that just gets dumped in, at the configuration at that point. So, basically, what I'm saying is here is that for every deployment that we have, there exists a service with the same name,

31:09 which has the same selectors and then, you know, like, has a port, the same as the port that's defined in the deployment. We don't require this port here, but it's this this implicitly assumes it's there. So, well, so the first thing we can do now is we can say, well, let's see if this even works. Right? So, actually, because I use, a package here at the top, I don't have to, mention the files anymore, so let's let's not do that. So the package basically makes them combine altogether. So let's run that, and it gives two errors here.

31:44 And it basically says a trim line eight column 12, which is here. It says I I'm finding one, right, you just type string, which doesn't match integer. So I'm getting two errors. So if you look at the schema at the respective, places, the problem here is that, you know, replicas is an is an integer, but it has two values. Right? Like, integer, the type, and the concrete value. Could be simplified, I guess, by recognizing you know, focusing on the integer part first. But, you know, it found the error, so let's fix that error and then see if we're good now.

32:24 And, yes, we're good. So this is very typical. Right? Like, you might think you have, no errors in your configuration, but that's you'll be surprised. So now I I said, like, okay. Now I wanna use this as a template. Right? So how do that? How do you do that? Like, well, first of all, you know, I can do an export on both, right, as I did before, and you basically see I'm getting this full, configuration. So you see the ports are there. So, you know, it's exactly as I was expecting here. So now but I now I know that

32:59 I can remove a lot of these fields here. Right? So I see kinds, all these things. I wanna basically remove those and, get the same results. Well, if I wanna remove them, that's just simple logic. Right? So why would you do that by hand? So there's this command for it called q trim. So I'll run that now. And then what you see, it has removed here all the all the fields, of my configuration that can be inferred from my validation templates. Right? And then if I run q export again, I'm getting the same results. Right? Because, essentially,

33:34 it only removes the things that could be derived from this template. So, I'll have another little demo of where CUE is heading, in the future, but for now, I'll hand it back over to David. Thank you. That was awesome. I have not played with trim, but that's a a very cool tool there as well. And just a a nice good overview of some of the the basics basics of CUE. So we did get a couple of questions. So I'll try and throw them at you now, and then we'll dive into the the kind of live component as well. So Kevin

33:44 Q&A (OPA/Rego, Writing CUE, Code Generation)

34:14 asks, still not entirely clear on the trade off between CUE and OPA. Could we maybe discuss that a little bit? I don't know. There's just a follow-up there. This is actually probably it's CUE versus Regal. What are the differences? Yep. Yep. That's a very good question. So so, basically, OPA is or I don't know how you pronounce these. The OPA or is it OPA? According to the docs, it's OPA. Yeah. OPA. So that means grandfather in in Dutch, just so you know. So anyway, so so yeah. So OPA is more of a of a product, right, whereas CUE is more of

34:51 a technology. So there is no equivalent of OPA for CUE per se. Right? What you could do is instead of using Rego, right, like, basically, rig OPA to use CUE instead of Rego. Right? And and and and that would work. So Rego is more mature. Right? It's a further developed language. So so there are certain we did a lot of recently, a lot of research in in CUE into using it for policy in in several contexts, actually. And there's a few, you know, tiny features that are missing to make it really powerful for for for validation and and policy,

35:28 checking. So you can expect more than for normal data templating. You can expect more, rough edges with CUE using it. Right? On the other hand, so if you see where CUE is coming from, right, the the class of languages where CUE is coming up from versus RIGO, so there was this this well, I wouldn't call it battle, but, you know, it's the same people that wrote a prologue, basically, that that went to develop sort of the successor to it that that that fed into CUE. Right? But Regal is basically based on Datalog, which looks very much like Prologue. Right?

36:04 But when it came to configuration, it it turned out that that Prologue really didn't scale, and this is not from a performance perspective. It's just, hard to understand. Right? It was hard to combine, the schema. So so, basically, in the end, from for large scale engineering, management, right, like, it didn't really work for configuration. Right? And then they came up with this graph unification formalism, which was very successful instead. Right? And so so CUE is basically the same thing. Right? We I I believe it's it's much better suited for configuration than RIGO. It's actually designed so what you wanna do

36:43 in OPA is basically define constraints. Right? So CUE is optimized for just the constraint definition, whereas, RIGO is, in essence, a query language. Right? But, but definitely, you know, Rigo is is is more ahead. So so it's sort of like how much, you know, do you wanna get into the early bandwagon. I mean, we definitely you know, we're definitely making it a fully fledged policy language. Right? So another thing you can see is if you unify all these different domains together. So with Rigo, you really only can specify policy. Right? Like, you wouldn't wanna define data

37:21 or templating in in in Rigo. Right? So so with CUE, you get a solution that covers the entire spectrum. You have to learn less. But the other nice thing about it is because it functions at all levels, you also can go meta on it. Right? So it's you can define schema for your validation. Right? So you can define the space of what are valid validations. Right? And so the this is something you can make it arbitrary meta, basically. Yeah. I I I think you've answered that great. I'll just reiterate the bit that I think is really important is that Regal really

37:54 just does constraints Yeah. Validate you know, validation constraints. Does this document match my constraints? Whereas Yeah. He does much. I I love Prolog. Don't get me wrong. Right? But it's this is just purely practical engineering. Yeah. So so I wanna I see that Bart has a question. Or Yeah. I'm gonna put that up right now. So yeah. Bart Lomey has asked. Thank you for the explanation. As we learned, CUE is read but when you configure something, you're writing configuration. So there's lots of edit and write processes involved. Exactly. Yeah. So so so yeah. So definitely,

38:29 we wanted to make it read friendly even if that meant sometimes making it less write friendly. Right? And what we hear from SRE so far, which is almost the polar opposite of GCL. Right? Like, GCL is very easy to get going. So that would be sort of a a, you know, JSON equivalent. It's very easy to get going and and be very you know, generating stuff quickly, but it very quickly becomes complicated. Right? Like, you cannot see where values are coming from. Whereas in CUE, it's sort of the opposite. Right? Because you have these these restrictions also

39:02 might for very well be because you're not used to not having this inheritance and being able to override. Right? So because you have these restrictions, it put a lot of restrictions on how you can can organize things, which might be very hard in the beginning. Right? So writing it is indeed, potentially more complicated than than with, JSON or GCL or whatever. So we do think, it's a matter of getting used to. Right? But, even I have to get used to it, at times. Right? It's a but on the other hand, it you know, we we had

39:35 this bug in CUE a while back, and somebody gave us these hundreds of files of CUE. Right? Like like, thousands many thousands of lines generating. It was not that big, but generating a hundred thousand, hundred 20 thousand lines of of JSON, basically. But, you know, it was still complicated enough. And to prove whether it was a CUE bug or a bug in the configuration took about one minute. Right? We basically could very quickly narrow it down. You could find the two relevant lines in the in the queue, and you knew that they were just dumping

40:08 all files together. Right? So we just from these two lines, we just knew what the answer had to be. Right? Like, there's no inheritance change to follow. Nothing. Right? Like, it was literally one minute to figure that out. Right? So that's that's, yeah, that's something you just just can't forget with with GCL, basically. Awesome. Alright. We've got one more question, and then we'll get the my screen shared for some live stuff. So Kevin is asking, would or could I use CUE to start a code gen pipe thing? For example, CUE based service definition to generate gRPC code.

40:47 Yeah. So at the moment, all the at at least within the CUE framework, all the generation is custom implemented. Right? So gen you can generate open API, for example, which is part of the way there. But for anything custom, it's it's a a little bit tricky. So there are some third party, you know, Accenture usage of CUE where, basically, they use the the Go templating engine. So there's a Go templating engine built in, right, to to generate all kinds of stuff. Right? So I'm less familiar with that, to be honest. So it's but, you know, there's definitely other people that

41:27 have been working on that. So one of the ideas is that at some point, we also have a sort of custom generation packages that you can use to make this generation more easy more easily, right, or make it more easy. But for now, this is all hardcoded. But as I said, there are third party solutions for that that I haven't digged in too deeply, to be honest. There is the ability to generate protobufs, isn't there, I think? Or is that a fact of our buff? More the so for protobuf right now, it's the other way around. Alright. Okay. So we

42:02 we can take proto definitions and turn it into CUE. 0.4 alpha also generates, proto values. Right? But that's that's something else. But, yeah, generating gRPC is is very well possible. So the only thing it doesn't do yet, so so design suggestions are absolutely welcome, is, so the service service definitions. Right? Like, that's not really done natively in in CUE yet. I mean, of course, you can model it in data, but you wanna have some native representation for that in in CUE. There's some proposals for that, but narrowing it down to what people want would be

42:42 be nice. Cool. We have one comment. That's not a question, but I'll I'll pop it up anyway. Anna says, their docs were previously generated by CUE and it worked great. Awesome. Despite your initial suspicion, no suspicion. Come on, Anna. But the total owner just seemed young. Yeah. I guess that's fair. CUE CUE is young, but it's evolving quickly and, you know, it's it's doing great things. Right? Yeah. Yeah. Alright. Let's get my screen shared. This is the CUE homepage. So if anyone wants to, you know, dabble along and play with the tutorials themselves, you can find them

43:10 Exploring More CUE Concepts

43:19 at qlang.org. We will pop over to the documentation. And I guess we could take a look at a few things. So we cover some of the the language basics from here, and then maybe just try and go a little bit deeper before we finish on your slightly more advanced demonstration. Yep. Okay. Alright. Let's just pop into the tutorial. So I'll just try and trim this. Hey, that's a nice pun for today. I'll trim this based on what you've already kind of shown as well. So I don't think we need to pop into the introduction. And I think it probably covered most of

43:55 the types and your your starter demo. So I guess this may be worth mentioning on top and bottom. These are things that I get I think I tripped up on at the start of my my CUE journey. Do you wanna give us a little bit of introduction to those? Like Yeah. So so so the way CUE handles it, that it always can combine something, is it defines this this value lattice, if you will, which basically means that any two values you take, whether it's just a integer or an entire configuration, if you combine them, there's always

43:59 Concepts: Top & Bottom

44:29 a single unique, answer to this to that. And, you know, this is kind of mathematically, you don't really have to to know about it too much, but then, you know, like, the the the value that everything combines into ultimately is called bottom. And, really, this shouldn't have been put in the language, right, like this this sort of offensive symbol there. Really, we should have just called it error, make it a build in where you can pass a message as well. Right? And this is still something we wanna do, which is which is which is easier. Yeah.

45:02 Is there a specific thing you were tripping over or or this weird symbol? Yeah. I think it was probably just more the symbols, you know, especially as I started, you know, maybe really great in CUE documents and you'd see something, you know, like this or I don't know. To me, it just didn't it wasn't intuitive. Yeah. Oh, no. That's not it. Yeah. I can't even remember. Yeah. But I think it just it confused me at first. It just took a little bit of understanding and that when I give something to CUE that it that it

45:36 is trying to turn it into a concrete value. And then when I started to understand that that was happening and then validating it, I guess, by doing that, think things started to get a little bit easier. Yeah. Hello. I'm I'm still very much learning CUE. Every day I learn something new, which is great. We all do. Yeah. Alright. So numbers and strings, those are all covered. We talked about, you know, you mentioned concrete values and closed values and stuff. Should we cover just a little bit about this? Maybe we should create like a little document with a schema to

46:01 Concepts: Definitions, Concrete Values, Private Fields

46:10 actually explain these values. Yeah. So let me do a main dot queue. Okay. So you've already shown off like, we can do something like this, which has a default value. The default values are star based. So I could do this. Right? This is valid queue document. Let me get a terminal. You you don't need the outer curly braces, by the way? Just to say. I think that's just a nasty habit. I've I've actually Yeah. Well, it's not necessarily nasty. Yeah. I don't know why I do that. I think we're I don't know. So we can do an eval on main

46:45 dot queue, and then we see that the name is coming out with David with that. And I think one of the things that I also struggled with when I'm first starting with CUE, this is gonna just be my, I think, my eating session of all the things I learned the hard way, is that, you know, you can bring things in. I don't know why I picked John. I really should have picked Marcel. Is that I could bring oh, there we go. Is that we can bring in new values? And this this whole concept of the constraints

47:14 living next to the actual data that I've got just took a little for me to become more accustomed to. Like, to understand that these were unified to give something that I could export Took a little while for me. Yeah. The the default thing is a bit weird, but and the original formalism that I worked with so so Michael asked, have I ever wondered what this work with or exposed to description logic? So not specifically many types of logic, and the symbol definitely comes from from logic, absolutely. But yeah. So so so the the form systems I originally worked with did

47:52 not have these defaults. But for, like, cloud configuration and many types of configuration, you really do need this. And this is CUE's CUE's way of cheating. So, really, this is one level of of of, this is basically a override. Right? So this is an overlay, if you will. But you basically only have one level, plus it's very clearly marked that this value might change. Right? So if you see star David and string, you know there might be another value that overrides it. Or if you see Marcel, you know that that's it. Right? But, yeah, that is a bit

48:25 but this basically allows us to get away with inheritance and still be able to reduce a lot of boilerplate. Yeah. And that one level of override just means that that I can't then come back in again and change this value because it's it's already concrete. Is that the right terminology at this stage? Yep. Alright. Okay. Yeah. Have two conflicting values. Yeah. We had a comment from Paul, but I'm not sure if this was maybe related to the gRPC questions. I will just put it on the screen anyway. But Paul is suggesting in get pretty far using template and using

48:57 cue string interpolation, which has the added benefit over go text template. Yeah. Okay. Yep. Yeah. CUE has pretty nice multiline strings. So this is taken from Swift, actually, which is, yeah, pretty powerful. Is that these raw strings here? Yeah. I didn't know I actually already had a tutorial about it. It's like, yes. That's it. Yes. So we get the description with the one to I've never used this. This is a long string. Now you need to close it with two hashes also. Yeah. So you don't need the two hashes. The two hashes basically escape the escape character. Right?

49:49 So you can you'd now need with two hashes, you need backslash hash hash to to escape. Right. So I think the idea originally comes from Rust and then Swift modified it. Yeah. I think Rust does have Rust looks little bit different. Yeah. Exactly. It's it's the same principle, though. Okay. So we also were taking a look just there at this concept of, like, a closed struct and structures in general. So why don't we just define like a simple structure and then try and understand the dynamics of what does it mean to be closed or or not closed.

50:27 CUE uses the hash symbol or pound symbol, whatever you call it, to define structs like this. And then those can have any arbitrary properties in it. Again, I could do something like this. Now, what is special about a definition in CUE versus like a normal struct? Is there? It's well, it's it's just really like any queue value, but we conflated a few of the concepts there, which which generally is a very bad thing to do. But in this case, I think it was the right choice. So if it starts with a hash, that basically means so it's a definition that

51:11 basically means it's recursively closed. But, really, if you define an API, it really shouldn't be closed. Right? Like, you want APIs to be extensible. So, basically, the recursive closeness is just for the purpose of error checking. So you can compare it to proto buff. So if you, have a proto definition, right, that you, convert to Go and you have a unknown field, right, like, you you you cannot basically access an unknown field within this generated Go code. That's a compile error. And that's really how you should see structs within CUE. Right? Even though the proto, in theory, should

51:46 allow other fields as well, and it shouldn't be an error if you have an incoming message to get other fields, you do wanna have that type checking within your code. Right? And this is exactly what a definition does. Right? So it's open in principle but type checked. It's it's signals queue to type checkers for for, you know, like, in in a closed setting. That's really what this is. Does that make sense? Okay. Yes. Thank you. I I think one of the things that also tripped me up and oh, no, that didn't do it. Oh, it's

52:14 export. Right? Is it yeah. These kind of definitions don't come out the other side as well as like there's a concept of a private field as well, something like this, which also doesn't come out in the data. I think I guess what I'm saying is my biggest hurdle was just this these two things living by the side by side. And it just took a little while to get used to. So I guess I'm just trying to make that clear for the people that are watching that, you know, these are things that can trip you up, but they're very they work

52:43 pretty well when you understand that. Yeah. One obvious thing that's missing from the documentation is sort of an explanation of what all the prefixes mean. Right? So there's underscore and hash. So you also see dollar sign, which really doesn't mean anything to CUE. It's it's just, you know, JSON schema thing. So but, basically, just a short description, like, what what do these things mean and and, like, why are they there and how should you interpret it? Right? That would be very helpful just at the very start. Yeah. So and and your opinion then, like, you know,

53:14 people have configuration, you know, as a substrate for pretty much all of our applications. We need to be able to configure things regardless if we're deploying to Kubernetes or we're deploying using cloud formation or any of these Terraform style syntaxes and stuff. Like, is CUE something that people should adopt to define the constraints and validate those configurations? Or is the eventual goal just to have CUE being the actual config layer too? Does that make sense? Yes. Well, so so there's multiple things to it. I think when it comes to pure data specification, I'm not sure if you would wanna send

53:52 CUE over the wire per se. Right? Like, you wanted something, like, really dumb and simple like like JSON. Dumb and simple is good here. Right? But for yeah. And then and then also, a lot of people don't need to specify more than a few lines of configuration. Right? And then they really should do it in what they're whatever they're comfortable with. But anything sort of on top of that if you start defining constraints or even, you know, as as a specifying data to meet CUE is is much nicer than than YAML, for example. Right? Like, I have I have the auto formatting,

54:26 auto indent, which is kinda hard to do in YAML. Right? So so for me, it's it's it's much nicer to specify data in it, but it's that shouldn't be a requirement, right, to to do. But, yeah, policy validation, all that stuff. Yeah. Okay. I guess we'll we'll cover a few more basic things, and then we'll maybe we'll work our way through the Kubernetes tutorial. Tutorial. So once we have definitions like this, we can begin to to use these definitions like so. And that's the finds key constructs, which can have a list of any number of definitions

54:52 Concepts: Using Definitions & Validation (Structs & Lists)

55:04 of structs. And I'm already hitting myself for the per naming on this, but we can now say that this is a list which has an object. Hopefully I'm not gonna mess this up, which has age. And I'm gonna get this wrong on purpose just like you did in your demo. Zero. So I should, I believe, be able to run an eval on this and we should see an error saying that the first value, enter a list of structs is invalid because the age is the wrong type to find in the definition here. That look okay?

55:41 Yep. We'll find out. There we go. Cool. And if I set that to 21, we get our structure at the other end, but we if I export that, we can remove all of the private information and just get the raw data. Cool. I didn't mess it up too badly. Is there anything else you think is worth covering from this list before you do the Kubernetes one? Maybe, like, disjunctions or bounds. Yes. Okay. So shall we add a disjunctions? So we say that our really terribly named structure here could only have two values and because we are here,

56:07 Concepts: Disjunctions (`|`)

56:28 we'll do this. This is what we call a disjunction. I think of a calculator there kind of threw me at first. Do wanna just talk about that if you don't mind? Talk about what? So disjunction, is is this from the mathematical concept of the the thing here, or does that come from somewhere else? Yeah. It's it's basically being able to say something can be an integer or a string or one of two values. And because q it's it's a little bit hard to name this construct, right, because CUE unifies types and values. So if if it were defined on values,

57:01 you would call it an enum. Right? And if you if you if it's a, you know, a programming language type, you would call this a sum type. Right? So it really depends on what you use it for. Also, you can use it for if you're looking up in a map and you you you want that to succeed, you can say this is the default value or or, you know, this is the if everything goes well, I wanna take the map lookup. But if it fails, I wanna take a default value. So that's like no coalescing.

57:28 You can do that too. Right? So, So really because it unifies everything, it's this one operation for for many purposes, which makes it hard to name. Right? But disjunction is a very generic name for these kind of things. Yeah. It because I think all of the challenges I had with CUE was pretty much they're all my fault. It's that I probably didn't spend enough time going through the documentation, looking at the vocabulary, understanding the syntax. You know? I I pretty much just started writing CUE and using eval and export export loops manually, like like print lane debugging, I

58:01 guess, in a weird way. And it's not until I see the error, you know, if weekly set this to David two and run that eval. I was just oh, yeah. It's not until I seen them that, you know, the word disjunction here as, oh, And it just you know, if you've not used that language before, I guess it it threw me a little bit. But, again, my own fault for not spending enough time in the documentation. Yeah. It's what I would do. I mean, with the terminology, You know, maybe these RMS just could be made more earthly, but yeah.

58:34 Yeah. So we can we can fix that, and now we're we're back to have them working. So this this pipe symbols, the ability to specify that it can be one of multiple types, one of multiple values, it's a it's a junction. Cool. And you also mentioned bounds. Is that just like Yeah. Minimax It's simpler simpler thing, but it's it's an interesting concept, right, that you can say so you can compare you you have comparators, right, like in any other language. You can say, you know, a is greater than b. And then if there are two different kinds

58:50 Concepts: Bounds (`>`, `<`) & Conjunctions (`&`)

59:06 of numbers, it sort of, you know, converts it to the the common denominator. So that's allowed. Normally, it's pretty strict with types, but with comparators, it's it's a little bit more, permissive. But then you have this concept where you have this unary comparator. Right? And then it's really a little bit like a spreadsheet validation, fields, right, where you can basically say, this value needs to be greater than three, and you just write greater than three. Right? And then the end symbol is the the counterpart of this junction. So you can specify many validators. Right? So

59:42 you can say this must be greater than three, and it must be an integer, and it must be smaller than eight or less than eight. So you can combine many different validators there. Yeah. In fact, I'll use a a real example because the it shocked me and amazed me when I first did this. But I I I'm sure it's so trivial, but I was so happy at the time. But I was working on a a dev rel platform to track all of our events that we were going to, and we wanted to store it in YAML, but we

1:00:11 wanted to be able to validate it before we tried to build, like, the static site that came out the other end. And one of the things we had was this concept, like, of a start date, which I'm I'm just gonna make it an end right now because I'll forget how to actually bring in the date time package. But see, it's just an end. And it's one, two, three, etcetera. And we have an end date. It's like one of the things I wanted to be able to do is always validate that the end date actually came

1:00:36 after the start date. Yeah. I I was actually when I was doing that in CUE here is when I realized that it was actually possible to do that. And I'm sure I'll mess up the syntax here, but is it just that? Yeah. Yeah. So using CUE, we actually have the ability to reference other properties within, I guess, document. I'm not sure if there's another word I should use there. But Yeah. We can make references to other values and make sure that they're valid. So here we're saying that the start date the end date must be

1:01:06 after the start date, and we can verify that. So one thing that might trip users up, so if you go back to that oh, yeah. You can yeah. So don't forget the end percent. Right? Like, the end because otherwise, it's int is greater or equal to start date, which is, in theory a valid queue expression. So we try to catch these problems, right, because it can never be satisfied, right, because int will never be concrete. Right? But yeah. So so just don't forget the the ampersand. Right? Like, that's the Yeah. So we have our conditions here. We

1:01:39 have it actually failing. So I guess there. There we go. So our invalid value is zero out of bounds greater than or equal to one. Is that about to do? And and just, you know, it's not been particularly challenging that this kind of write this document. I feel that it reads really easy and I think that's really powerful for people as we're trying to validate this enormous amount of configuration that we have in all of our application. Cool. I hope that people are as excited by CUE as I am. I don't know if I'm just weird that way.

1:02:13 But it just, you know, takes all of these parts of our application that were really difficult or we had to write really cumbersome tests far before and just brings it across a very simple data plane or config substrate, whatever. Okay. So we're at 03:00. We got around thirty minutes left. How long do you need for your last demonstration? Roughly. The five minutes. Sorry. Okay. Cool. So let's try and use this now. I think we've we've shown off just, you know, queue constraints, junctions, disjunctions, etcetera. So we'll take a look at this Kubernetes tutorial here. So

1:02:51 Kubernetes Tutorial Demo: Importing YAML (`cue import`)

1:02:55 the goal of this tutorial is to show people how they can simplify the Kubernetes manifest. Would you say that's correct? Yeah. Everything, really. So convert, import, manage, validate, simplify, yeah, templatize. Yeah. Okay. So I'm gonna skim this and I'm sure I'm gonna miss bits. But let's see. So what do I need to get started here? I guess I need to run the CUE mod in it. Yeah. So you have to yeah. Well, yeah. You can just start in this original directory. Right? You don't necessarily have to copy it. But queue mod in it. Yeah. Okay. Do I need the example.com?

1:03:40 I'm just trying to You don't need to. No. That's yeah. It's not not mandatory. I'll just create an e b c directory. Again, my naming is always on point, but queue mode in it. Does this trying to so once you use queue import to grab some YAML files and convert them to CUE. So should I just create a very simple Kubernetes deployment? Yeah. So basically, this this is based on an existing dataset, right, that's being converted. So you would have to copy that dataset. Or do I need the whole thing, or can I just create a deployment?

1:04:13 Does it matter? You you Yeah. It would be easier to to copy the get the Git repo. Yeah. You're right. I shouldn't wing it. Bad things happen when I do that. Let's just clone it into here. I'm sure there's a flag to tell it to ignore that. I'll just do this. K. So what directory was that? Doc tutorial. So it's in yeah. You you go to docs, yeah, tutorial Kubernetes. Then original, you could do it there, I guess, if you don't mind modifying the repo. Okay. So queue mod and that from here. Will that work okay?

1:04:59 Yep. Because I feel like I've went off a weird path now, but it should be alright. Kubernetes original. Okay. So now we have our queue module stuff here. It's a simple we have this directory of services, and we want to then import all of our services. Like, so only that path is gonna be different. Yeah. I would go to you in services already? Or yeah. You don't have to, actually. It doesn't matter. That's Okay. So you just want me to import services like so? No. Just c d into services. You don't need to, but before

1:05:37 we forget later and, like, you know, command to assume that you that you are. Yeah. Yep. We're importing all of this, so it's just gonna be dot. Is that right? All the all the subdirectories too. So dot slash dot dot dot. Alright. Okay. It's like that. Yeah. Import everything. Yes. So what this says, basically so if you go back to the tutorial, we'll do a little shortcut so you can scroll a little bit down in the for the sake of time. So I think, yeah, if you take that command, I think that should do it.

1:06:02 Demo: Understanding Imported K8s Structure

1:06:15 This one here? Yep. So, basically, so all the Kubernetes definitions, they're all in this flat space. So I like to have these YAML files with the dash dash dash in between. Mhmm. And what we're doing with this import is that we're, putting it in a namespace. So there's some feature features plan to queue where you don't need to do that. But right now, we're basically assigning them each a unique name, basically, by grouping them by kind, right, like deployment or service and then a name. And then, so we're treating all of it in the same namespace,

1:06:47 basically. And that's what this command does. So it gives it derives the name from the kinds and the metadata name. And the and what the dash r does, that's interesting. So some of these config maps, so you can run this, for example. Yeah. That should be fine. And then if you look, if open the file in Mon, the Mon in the Mon so there's a subdirectory in services. There's Mon. Yeah. And then Prometheus and then config map YAML first. So if you look there, there's this, you know, sort of inconspicuous, like like, pipe symbol, right, which really means

1:07:34 that what follows is not YAML. It's a string. Right? So that basically means, like, most of the configuration in this file is really treated as just one big string. Right? Like, untyped, not even YAML structure. Right? So the dash r option searches for these, scenarios and then recursively converts that to queue as well. So if you go to the generated queue in config map dot queue, you will see that, you know, like, it also converted that into into structured queue. And right now, it's a let, but you can then manually assign it to a field if you wish, and

1:07:37 Demo: Handling Nested Configurations Recursively (`-r` flag)

1:08:11 maybe I'll make that an option so that it does that automatically. And then it has this call to to YAML Marshall. And because it's generated, it has these funky numbers to make it unique. It has this call to the YAML Marshall to to convert that back into the original YAML string. But, basically, it allows you to also recursively validate the YAML within that queue. Right? That's what that does. Yeah. That's almost magic. The fact that it's taken a YAML multilane string and has kind of interrogated it to see if it is in fact itself more configuration

1:08:49 and then just giving you that back as the queue option. And this is one of the problems in GCL, right, that it really had only one layer of of validation, whereas very often you have these nested configs and you wanna be able to to handle that as well. Right? Like, typing within typing. Well, I think you've made literally everybody's head explode. So Julian Pavotto has said, wow. Kevin actually said his brain has exploded, and Michael is agreeing. So there's a lot of magic happiness, I think, in the audience seeing that work. That was really cool.

1:09:25 And now we've got yeah. This is just cue representation of that entire thing. Yep. Yeah. That's very cool. So, basically, recursively imported for all the directories, put it in that directory. So the directory structure of CUE, so they all have the same package name. We might provide other modes. We're sort of looking into what modes people wanna have, but the very common pattern we saw is that, you know, you have this directory hierarchy of configuration where, basically, you wanna have the the schema and sort of the root config. It can be a subdirectory too. Right? But there, specify

1:10:02 the schema that applies to all the schema and the subdirectories. Right? So think of it as ACL. Right? So your validator that you specify there sort of should apply to everything in the subdirectory, very much like ACLs work or or whatever. Right? And this is basically how this is set up now. So now you have this one package which crosses an entire directory structure. And the CUE module basically is a single hermetic view on all your configuration. Right? So it's it's supposed to be completely closed and and fully defining everything what's going on within these directories.

1:10:36 So that's that's the significance of that. Cool. We had a a comment from Anna who's suggesting that CUE is a kind of data focused mix, which is a quite an interesting way to look at it. Are you familiar with NextLang as well? With Next, yeah, I am familiar with it. Yeah. So it's it's a nice language. Right? Pretty pretty neat language. So I think Next is more focused on package management. Right? It's a more generic the language itself is more generic, which you can see it has some elements of it where they are focusing on that.

1:11:11 It it so so a lot of the more needs configuration languages like Dial and Mix, they do focus more on functional, whereas CUE really is more logic language. Right? So I think ultimately for configuration, because you have this aspect orientedness, right, like, the you can really only solve that with with logic and and not so much with functional approaches. So it's it's you know, depends on your use case, but yeah. So CUE takes quite a different approach than DAO and and NICS, right, for that for that reason. Not not to say that there is other use cases where it can be quite

1:11:46 useful. Yeah. It looks like Kevin and Anna are having a conversation about potentially using CUE to validate next. I can't keep up with it, but interesting stuff. Okay. And there's a a nice joke from Kevin, actually. Your recursive flag is to de ugly things. That's a way to look at it. Yeah. Alright. So this is Let's see what this tutorial is is doing next. So we've imported Yeah. So that all looks better. Yeah. So that's so create a top level template. Yeah. You can you can do that. I I would skip the validation. It's not that interesting. You should

1:12:27 Demo: Creating Templates for K8s Manifests

1:12:35 but yeah. So one thing that's interesting thing to do, you scroll back up Oh, yeah. Sure. So this is a very good pattern. So I I showed you how to, given a template, reduce configuration. So theoretically, so within graph unification is a known algorithm. It's called anti unification. I could also do the opposite. So given all the, like, JSON and YAML we have now, like, compute the optimal templates, that after running a trim would, would give the the most compact configuration. Right? Like, that's not, implemented. So but you can it it's actually quite easy to do that by hand.

1:13:13 So what you would normally do is to copy one of the templates that's sort of representative, for what you wanna do, right, or what that you think this is sort of a typical example of my my configuration. Copy that to the roots, right, and then start stripping anything specific to it and and generalizing it. Right? If you don't wanna do that by hand, then you can do what this cat, know, block is doing. So this basically creates an initial, template, which is really a hand modified version of one of the the the templates. Right? So

1:13:50 we can do it by hands, but for the sake of of saving time, you can you can also just copy it. So that you don't need to do. Okay. You can either copy it and modify by hands, or you can do the cat thing, which already is the sort of the modified version of that. Okay. So let's grab this and just run this here. And this yeah. In the root directory. So in services. Don't know if you're in services. I was in services. Yes. We've got a CUE.CUE now. Yep. And then so, basically, you see this short

1:14:29 example that I gave. This is pretty much the same thing. Right? Like, you have replicas. You know? Like, there there's a few more labels, but it's basically the same thing. Just scroll all the way to the bottom. Oh, yeah. No. It's just these two things, so I don't generate services from deployments. But, yet, I don't do that here yet. But, basically, I do the same thing. Right? So I say, match any of the the the names in the service map. Right? And then I basically say, okay. That name should be the same as metadata name,

1:15:04 and that's, you know, all these little tricks like that. Right? So there's a few few of these. And yeah. That's the demo you did at the start, right, where you created a comprehension over one of these and then just added some extra constraints that locked the service and the deployment together. Right. Okay. And the square bracket basically means it's this this path selector. Right? So the square brackets basically mean match any path of of you know, like, that I'm writing here. And here, it's underscore, so it matches any path in deployment. But you can also

1:15:34 say, you know, anything that starts with a capital letter or anything that that is you know, ends with something or whatever. And we're working now on an extension where you can even say, you know, like, matching patterns within the value. Right? So you can really do very complicated pattern matching there to to select where you wanna apply this template. Yep. Awesome. Yep. Alright. So what's coming up next in this tutorial then? This eval is Yeah. You can run that or that, whatever, and then you run into errors. So this is the idea here that if

1:16:09 Demo: Validating K8s Manifests (Finding Errors)

1:16:15 you write down these validation rules, right, like, very often so so so you you might run into errors in your configuration, but you also might run into errors in the rules that you've written. Right? So I haven't run this demo in a very long time. So, yeah, so one of it is that component isn't defined everywhere. So if you go back to the tutorial. So it mentions that. Yep. So, basically, component isn't defined everywhere, so especially in the so so yeah. So so not all subdirectories, you know, not all files and all subdirectories have this defined. So here you see we

1:16:59 have a constraint. Right? Let's say we always want this component label, but it's actually not specified everywhere. So if you run the code below, we actually create a small cube file in every directory where we define this this component label. Right? So that's that's some shell wizardry that's not specifically CUE. Yeah. I think I I I got kinda confused when I copied this and just pasted it, but now I actually see what we're doing. We've we've created just, like, some constraints to check all of our services and deployments. Okay. And we're expecting them to have a

1:17:31 piece of metadata label that is component with a value. Yeah. And and not everybody not all of the files have that. So Okay. So this is just suggesting that we use set magic. Yeah. Know what happened there. Oh, yeah. It's Oh, no. It's about it. Yeah. So it's created all these little files. Yep. So, yeah, it's Prox? There seem to be something wrong. There should be proxy, but but either way. So it might look run into more problems. Is that just me failing at copying and pasting? That's what it is. Right? Yeah. Don't know what happened there.

1:18:27 I wonder if when I ran it here, if I potentially broke something. Oh, I think I definitely have. I can always do it by hands but See, the directory is called proxy, so I don't know why that's You can change the proxy. It's probably a good idea. Then I don't know if it got wrong. And I'm not even getting the star. Maybe it's a shell incompatibility. Who knows? Oh, yeah. I'm running that shell, which I guess should mostly work. It's not important. I I think we can have covered a lot of cool stuff there. And I'm worried that my failure to copy

1:19:11 and paste has done something very weird there, especially the one I control seed, which was halfway through, like I said. Yeah. Yeah. I'm not sure. Alright. Awesome. I will pop that away. And we'll jump over to your last demo and we'll we'll finish up. If anyone has any questions, feel free to jump them into the comment section, and we'll be finishing up in the next five to ten minutes. You looking for your window? Or have you frozen? Oh, no. You're there. I'm here. So shall I start Yeah. I think you said you had to just pick a five minute Yeah. Thing. So

1:20:00 Generating OpenAPI spec from CUE

1:20:00 we'll finish with that, and then we'll we'll do a quick recap, and we'll end it there. Yeah. Yeah. The Kubernetes demo is is quite long, actually, and it's it shows you a lot of things of that can go wrong. And and so the idea is that, basically, by the end of it, which is, like, twenty minutes, if you run through the commands, like, you see that you reduced the amount of configuration by half, roughly, or maybe slightly more, slightly less. I I I don't remember. But it's, you know, like, this whole repetitive trim thing, like like, really

1:20:30 reduces. And you can see we start with about 1,800 lines, and by the end, it's it's like considerably less. And that's just stripping down the boilerplate and, you know, making things reusable whenever possible. Just it cleans things up a lot, definitely. Okay. Julian says he has a a thousand questions. Well, we'll maybe try and answer a couple of them. Feel free to drop them. So so so I'll I'll give this this one demo first. So yep. It's a clean thing. So let me just run. So what I'm doing here so so this is basically to to show that, really, there's

1:21:08 Demo: Generating CUE from Go Types (`cue get go`)

1:21:13 a logic engine behind it. And and there you know, like, it really doesn't you can use CUE for things that have otherwise nothing to do with CUE. So here I have a piece of Go code, which has, types and validation code. And what I wanna do now is to, generate open API from this. Right? So, basically, the first thing I have the commands handy. So so, course, nothing. Okay. Fine. I should do it by hand. Gets go local. Do I have to yeah. Current directory, I guess. Let's see if that was the command. I think so.

1:22:00 So if I run get go, what it does, basically, it analyzes this Go code. So this already works. And then generates this restaurant gen Go gen queue. And what you see here is this nicely documented, you know, like like, queue representation of the Go types in this file. So this is nothing really too spectacular. Right? What you can see, though, is that I also have this validation code defined on table, which has constraints. Right? And I would like to have these constraints represented in queue as well. So I love, you know, like, think tinkering with

1:22:43 SSA. So so what I wrote is this little SSA analyzer. What it basically, what it does is first tries to identify a validation code. And the way it does that is is if a code has no side effects and returns an error, it consider its validation code. And then I can see, of course, it's like, well, this is a method on table that validates or takes table as a first argument either way. Right? So this must be something that validates table. And then, basically, what it runs what it does, it runs over all possible paths in

1:22:52 Demo: Extracting Constraints from Go Code (Static Analysis)

1:23:21 the SSA, you know, representation of that code and tries to find cases where it knows it will return an error and then sort of logically reasons backwards to get, to get facts, from that. Right? So let's do that now. And this has been, run also on the, Istio code base for which I did some analysis. And I was able to, so so just to give you an idea of the coverage, I was able to get a % of the validation code. Right? So the, the the accuracy, both recall and precision, there was a %. And about 95%

1:23:57 of all possible code paths, I was able to determine whether it would return an error or not. Right? Then the rest sort of was, the code sort of had to give up. So if we look at what this produced so this was, yes, s s a dot q. This produces something else, and it's, so this is only one constraint. Right? So this is quite simple. But if you if you have many constraints, it becomes quite barfy. Right? Like, you get all these weird because it's out of a logical negation, it gets this dump of of all kinds of facts.

1:24:35 And then, basically, if you wanna combine that into a single table, right, we now do CUE DEF, which means schema. Right? I don't wanna get to the concrete data. I wanna define a schema. So we can say CUE DEF, SSA, and what what's the other one? Restaurants, go. Now you see we have the same schema documented, right, with the constraints, inside now. And then, of course, I can also do out open API. Now we have that same schema in open API. So here you can see we have a open API, you know, like, definition of the original,

1:25:04 Demo: Exporting to OpenAPI

1:25:17 GoCodes without any further specification from the user. Right? So that's what that's what happened here. So this is kind of the thing you can do with this with this CUE ecosystem. Right? It's the idea. Damn. So I just wanna make sure I understood that right. That was just random not random, but, you know, that was just go code where using CUE get go, it was doing static analysis of that code to work out what the constraints were on it from the different functions that interact with the data and then generate in the queue constraints. Yep.

1:25:33 Conclusion & Final Q&A

1:25:54 You then merge the or unified, I guess, would be the correct or unified the two different queue files and then output it in open API schema. Document it. Yes. Yeah. Yeah. That's the, yeah, that's the idea. This is even though, you know, like, in my analysis so I I was able to extract these facts for 95% of the code paths. Right? Like, that that doesn't mean, it's already implemented. Right? Because now you have to translate that to to something concrete. And that's, that's doable, but you have, like, you know, hundreds or about a hundred or so, like, in

1:26:28 that order, like like, different cases you in edge cases you have to handle. Right? So it's kinda tedious to implement, but it's definitely possible. And and, you know, also, it didn't take too long. It runs, like, within a minute, right, on the on the Istio code, more like twenty seconds or so. So it's it's it's quite excessive to go for all possible code paths, but it's it's definitely doable. Yeah. There there's just so much magic there. There. I think I'm actually gonna need time to think that one through and understand it. But the I mean, the applications of that

1:26:58 across a whole variety of different code bases is a is a pretty cool thing. Very exciting. Yep. Alright. Sorry. Question. I think everyone is speechless. I don't think we have any more questions. There's not been anything since the the the magic static analysis stuff. So I think what we'll do is, I mean, all of that was great. Thank you so much for taking the time today to to join me, you know, not just introduce us to CUE, but show us some stuff that's coming next and and even just getting some of that experience out of your

1:27:30 head to understand the why as well. I think it's just been completely invaluable. So thank you very much. I'll let you get back to the rest of your day. If there are any more questions, people can ask them in the YouTube comments. I'll do my best to relay them on or join the Discord, and we'll take it from there. So, Marcel, thank you again. Have a great day. We'll speak to Thank you for having me. My pleasure. Yeah. If anybody has questions, feel free to contact me. GitHub discussions, there's you can ask questions there. That's our

1:27:57 main portal for asking questions now. So Yep. There are links to all of this and the description the video to the CUE repository to Marcel's Twitter, and I think there's even a link to the CUE community Slack channel as well. So lots of way for people to get involved, get in touch. Again, thank you. Have a good day. I'll speak to you soon. Well, thanks.