0:55 Introduction and Welcome

0:55 Hello, and welcome to today's episode of Rawkode Live at the Rawkode Academy. I am Rawkode. That is three of the Rawkodes too many. Today, we are taking a look at CueBlox, a actual personal project of mine and a friend and colleague, Brian Kettleson. We're gonna be guiding you through everything you need to know to do cool things with your data. Please remember to like, share, subscribe, comment to these videos, join the channel, and you'll get updates for all new episodes of Rawkode Live. We also have a course in flight on InfluxDB, so check out the options for joining the

1:28 academy. And there's also a rather active Discord server, over 500, nearly 600 people talking cloud native Kubernetes, Go, Rust, you know, all sorts of cool technology things. So come and join us. Say hello, and I look forward to meeting you. Okay. That's the housekeeping over the way today. Like I said, it's a personal project. Brian and I have been working on this for for months and months and months, and I we're both really excited to share it. Right, Brian? This is really exciting. We've been we've been iterating on this forever, I think, and it's starting to become something that feels right,

2:05 which is awesome. Yeah, I couldn't agree more. I think we talked about it for a while before we even started building it. I think our conversations around us maybe started towards the end of last year, maybe a little bit earlier, where we both just had these grand plans for how we wanted our data to work. And just, it's been really cool watching this project change a lot over the last seven or eight months. Yeah. I think my favorite thing about this particular project is the fact that we've been completely unafraid to evolve our implementations as we realize

2:41 the way things are working didn't really meet our visions. And and when when the the code takes you in the direction you don't wanna be, it's it's totally okay to throw that out and and start over. And we've definitely done that more than five times. Yeah. I hadn't I hadn't actually thought about that. But, yeah, we do we we experiment a lot. Right? We're definitely tankers always playing with new technologies and and new tools. And I think we we kind of take those learnings and experience and and bring them back to what we're actually doing. So yeah.

3:14 Introducing Brian Kettleson

3:14 Let let we kinda skip the step here. For anyone that's not aware of you, Brian, can you give us a little bit of an introduction and tell us a little bit about yourself, please? Sure. My name is Brian Kettleson, I'm let's see My nice little lower third there. I did text name off. That's it. You you don't deserve my purple name anymore. I'm a cloud developer advocate at Microsoft on Azure, and we focus on Linux and open source, all of the fun stuff. So, I really enjoy my job because I get to do open source y, Linux y things

3:47 all day long for Microsoft and surprise people. I guess it's it's maybe less surprising now than it used to be, but, you know, Microsoft has a ton of open source and we really enjoy contributing back to the community and and helping people with their open source projects running them anywhere, including Azure. And our our team is full of contributors to all sorts of different OSS projects. Yes. Definitely not the Microsoft of the nineties. I'm always super impressed with the work coming out of Microsoft over the last ten years. It's it's really refreshing and it's great to see.

4:27 Push, you're always hacking on weird things on Azure too. And in fact, there's been a couple of really cool experiments that you've done with CueBlox alone on Azure, which I've done really interesting as well. Like, you just seem to have this really cool way of being able to take stuff that should be run on, like, a real machine and making it serverless in some way, which is always exciting. You know, it's so funny because maybe three or four years ago, I was as anti serverless as one can be. Just why in the world would you want

4:47 Discussion on Serverless

4:56 to give up control of your code and how it's hosted? And now, I don't want to host anything of my own at all. It it better be in the cloud or I'm not gonna run it. That's just it's it's so weird how how quickly I changed my mind on the serverless concept. And, you know, I'm I'm I'm totally embracing the fact that I used to be anti serverless, and now I'm very pro serverless. It you know, it's it's okay to change your mind. It's okay to have new opinions and and learn. Yeah. I think the frameworks are also I

5:30 I mean, I don't wanna throw into a pro serverless conversation by any means, but serverless as a technology has improved so much as well over the last few years. Like, you can now take go by any reason or Java application, Python applications, whereas initially, was really just here's about JavaScript and we're gonna run it in some way for you on this platform. Yeah. Yeah. It would it's definitely evolved a lot and it's still evolving. I mean, most of the platforms that where you can run things that aren't JavaScript, you have to do them over some sort

6:00 of HTTP bridge, you know. So it's we're we're we're way back in the nineties doing CGI again, and that's okay. It works and it's fun. It will evolve more and that's fine. Sweet. Alright. So the topic of the day is CueBlox. Do you wanna give people the that Taylor thing is super cool. Do you wanna give people the TLDR? Have you have you got a patch ready for it? How does one do the TLDR of CueBlox? So CueBlox as a vision was started because we wanted a way to have a schema for our data that we could share in a group.

6:20 What is CueBlox?

6:44 So if if you and I were talking about an event and that event happened to be described in a YAML file or a JSON file, we wanted to be able to share those events in our teams and our groups and be able to reuse them across the team without any crazy translation of, you know, your field names and my field names. So, that was the beginning of the discussion that ended up with us writing thousands of lines of Go and learning Cue and, months and months and hours of work. And what we've ended up with is

7:23 way to define a schema for a record set and use that schema to validate your record set, including referential integrity between groups of records, and then publish it using all kinds of really interesting little tools that we've built along the way for publishing and, share that data among, just yourself or among a team with published group schemas. We've got a lot going on in in CueBlox. It's it's I don't know if there is a TLDR. There's certainly no elevator pitch. Yeah. I mean, I think, you know, if, you know, if if if anyone's watching this

8:05 and they go, I have YAML data or, you know, I have JSON data or you have marked down data with front matter. Like, if you want to be able to enforce a schema on that front matter or the YAML or the JSON, There are certain ways to do it right now, but none of them really provide, I feel, like the platform. I feel like CueBlox is is turned into a platform to work with that data. And if I I'm gonna pull up something that you created on the website that I just absolutely love. So The pyramid? The pyramids. There we go.

8:35 This is Maslow's hierarchy of data. I think this pyramid does a really good job of explaining why people should be interested in CueBlox. You know, we we start with markdown and YAML validation. I think that's one of the things that when you define the schema and you point CueBlox to it, you just get that by running a a blocks validate command and it just says, yeah, your data is good or no, your data is bad. Yeah. And at the basic level, that's a really great way to do things like validation for the front matter in your blog posts.

9:07 You know, something really simple, you're not gonna ship a blog post without a date or a title or an OG field, you know, whatever it is that you've got in your blog that is important to make your website look good. Yep. You you can use blocks to validate that they're all there and in the right format. Yeah. And the referential integrity, I think, kinda comes in as there too. I mean, I don't think I've seen anything else offer that for YAML data, and I think that that alone is a great reason for people to be curious or

9:40 at least their interest to be piqued. I I I won't describe that though. I think when I say reference on Checkatrade, I hope people understand what that means and we'll we'll definitely show it as a natural demo in a few moments. We then have the ability to convert or manipulate that that data. And you've done some really cool things there, I'll let you kinda tackle that one. Yeah. We we start with either YAML or or Markdown and we can convert it into all sorts of different things, JSON being the most obvious and basic of them.

10:13 But, we built a templating engine into CueBlox so we can use Go's templates to run any sort of template over either a record set or individual records. Mhmm. And I've used that for some ridiculously awesome stuff. I built a template that builds Bash scripts and runs the Bash scripts for each record and each record set. And it's just I mean, it's amazing how much power you have with, Turing complete language. It's a lot of fun. I can't remember if we talked about it or you actually you did it one day, but the Atom RSS feed, is that something did we just

10:55 talk about that? But another that would be a really good use case for this as well. Right? No. We we did that. There's we have an RSS feed. You name it, you can build it with the templating engine. It's really fun. Alright. Moving up our pyramid, we have the shared schemata. But one of the primitives, I guess, in CueBlox is that we we try to provide some functionality for you to pull in schemata that is stored in a central location so it can be shared with other people. And the way that we do that is just by defining metadata on all

11:29 of the on all of the schemata that gives it a namespace and a type. So we kind of explore that as we write our first schema during kind of today's session. And then a custom schemata repository, I think, is just really a follow on to that. So Yeah. If if I can expand on that though, the Oh, please do. This is this is really the core of of what brought us into CueBlox in building this was getting people on the same page and making sure that my definition of an event is the same as your definition of event.

12:01 And once we share the same definitions, it makes sense to be able to publish them either privately or publicly. And so now we have the idea of creating our own schemata repositories that we can use, and everybody points at that repository and our command line tools make that dead simple. You know, it's it's as simple as a go get is in Go, we can do the same thing for a schema in in blocks. Yeah. Definitely. And and some of the really cool work you've been doing this week actually moves some of this functionality out of core

12:35 and into plugins, which we'll get to have a really fun look at as well during today's session. And the last one is consistent aggregated team data. So, you know, something Brian and I have been trying to think about as we build out this tooling is that, you know, individual people may own a subset of data, but you may wanna aggregate that at a team or organization level. We wanna be able to make that as easy as possible. Have I missed anything that you wanna add to that? No. I think the core of where we ended up is the idea

13:11 that we have these federated independent datasets that share some properties in their schemata that allow us to aggregate pieces of them. I mean, my blog articles may not look anything like yours, but our event schema are the same and that means that we can publish an aggregated calendar for you and I if we were on the same team. And the data would would be perfectly in in good shape. We wouldn't have to do anything to that data because of that. And that's really fun. That that made this project well worthwhile, I think. Yeah. I still enjoy seeing Scamata just for

13:53 the record. I think it's a great word. It is a great word. And you know, what what what we haven't really talked about in this whole pyramid is is the the myriad of tools that we've built to consume this data and to publish it. You know, we've we've had a lot of fun along the way, building things that make REST APIs out of CUE data automatically and GraphQL APIs. And Yep. You know, all of that is is dead simple and one GitHub action away. So David, from minute one, has been insistent emphatically insistent that this is GitOps. You push to a Git

14:37 repository and things happen. And I think that's been a really solid, goal for us is to have Git be the source of truth for this data and then publishing things just happen afterwards. And that's that's that's where I've had most of my fun, think, is is taking that cue data and alright. How are we gonna make this into GraphQL serverless on for sale or on Azure, on Google Cloud, you know, and I've had a lot of fun doing that. Yes. 100%. Alright. Before we dive into the hands on bit and we start actually playing with CueBlox,

15:12 Real-World Use Case: Blogging

15:16 I think let's try and cement everything we've said so far with like a real world use case for what we're actually using CueBlox for. So I think we'll try and stick with what we have in common and then feel free to add on whatever you wish from your own project too. But I think as developer advocates, we we both have articles that we write that we wanna publish to our website. Those articles are typically marked down and we would now both have them living in a git directory with a git hub action that on every commit,

15:50 it will validate the front matter on every article. And if it passes, it will output a JSON blob that can be consumed by other tooling. That JSON blob is published as a release artifact on both of our GitHub repositories. And we have oh, how are you using your API right now? I'm using well, actually, I'm using it in two different ways in the same blog. I'm I'm I'm pulling it directly out of the file system in markdown and using MDX, and I'm also consuming it over the new API The surf command. The no. Not the surf command. The the static

16:37 CDN. Oh, nice. So that that's the plugin that I built last week, which I can show off. Plugins have been so much fun. Yeah. I've only been keeping up with what you've shared with me on Discord, but I'm loving what I'm seeing. So because we got a really good demo of that. So, yeah, your data lives in in Git. You can validate it in a GitHub action. You can publish it as an artifact, as JSON to consume it from other tooling. We also provide helpers to serve a GraphQL API, and all just kinda works really nicely. So

17:11 Initializing a Blox Repository

17:11 I'll share my screen again and we will jump over to a terminal and maybe we can just show how to get started with the tool itself. So our latest version, zero seven o, released, what, two hours ago, three years ago? At least some point this afternoon. More than that. Was it? Yeah. It was it was pretty early this morning, my time. Oh, okay. Maybe six or seven hours ago. Oh, we we don't have a version command as a flag. There we go. Nice. And the way that you would get started with blocks is first to run a blocks

17:49 init command, which is gonna set up a little bit of boilerplate that just tells you where to store your schema, assets, the data itself, and build directory for any intermediaries or outputs that we produce as part of the block project. Yeah. It's probably important to notice that this is opinionated but completely changeable. The blocks dot cue file has references to each of these directories, you can change them if it suits you. And that's something that I take advantage of heavily in my blog. For example, the static directory is actually the same public static directory as

18:28 my Next JS blog. So I share the the static directory between the blocks repository and the website because why not? Yes. So this is the blocks up here. All configurable and do whatever you want. But, you know, we'll we'll stick with the basics for today. My Versus code is crawling today. Hopefully, that'll be alright. This is what happens when you use your development machine and the streaming machine at the same time. So the first thing we need to do is define a schema. So I think we'll keep this relatively simple for today's exercise, and I'm

18:50 Defining Our First Schema

19:08 gonna cheat and just have a single schema dot queue file. And we need to raise some Q. So for anyone that's not familiar with Q, I think you're missing out, it's such an exceptional language for working with constraints across data and schema together with some really cool abilities to like query the document and do interpolation and stuff like that. I'm I'm really impressed with Cue, but it's not the focus of today's session. But this is a a Cue document and in order to in order for it to be a CueBlox schema, it requires one piece of metadata,

19:49 which is something called a schema. I think it's named. Right? And I'll call this Rawkode.demo.live. There we go. Demo. There we go. I forgot how to name space. And a name, this will be my article schema. I think we both know the article spec pretty well, so maybe we'll run through that and touch on some of the features as we go. So that's right away is valid. CueBlox would scan this directory, read this file, and say, hey, we have a namespace and we have schema for an article. However, we've not really declared any types, any definitions,

20:31 any entities, whatever you wanna call them from your own background. Cue denotes these through something called well, through the the hash or pound sign depending on where you're from, but this is called a definition in Cue. So here I can define a person and I can define an article, like so. Now in order for CueBlox to know that this definition should be consumed by CueBlox, we also need to provide oh, sorry. Data set tag like this. And this just requires I can't remember. Does it yeah. I think it is required, but you have to set

21:08 Want the plural? Yeah. And you can also provide a supported extensions, which we'll talk about right now, I guess. I think this does default to what we consider data that we can read, which is what YAML YAML and Markdown. And Markdown. Yeah. Perfect. Yeah. And those extensions are YML and MD. I changed mine to add MDX too. Alright. We'll just we'll add a few in just now, and we'll show how that works. So the plural of person is going to be people. And we'll copy this down here. Now, why do we need a plural? You wanna tackle that with that tape?

21:49 Yeah. Why do we need a plural? Because English is funny. Because we store the data in directories named after the plural. It's really that simple. Yeah. Yeah. We use it for the data directory's name. It ties the clue there still that you don't have purse persons would be a directory name. Or if you wanna expose it for GraphQL, you can do cool things there too. Alright. What does a person have? Let's do a really simple schema here. Name. Or name. Alright. Yeah. We'll just do name. We'll keep it simple. And age. You happy with that person?

22:33 No. No. I would never put an age in a data field. You could put a date of birth or a year of birth, but age is immediately invalid as soon as you store it. Because time carries on. You're 100% correct. I also used to be a DBA. Alright. So you're gonna hear that if I do a title too then. Right? And then maybe Okay. And then we have an article, which we're just gonna give a title and a body for right now. And then we'll have a concept of an author, which right now, I'm just going to do person ID.

23:14 So CueBlox has a little bit of magic logic. There's nicer ways to do this, but I think we'll start off with the magic and that anything with an underscore ID will automatically be inferred as a relationship to the person type, and we can use the referential integrity on that data to validate that it is correct. So let's run two really quick documents. So we've got the plural name here, so that's going to be people and we'll create another directory under data called articles. And I'll create one in each. I don't know if we ever changed

23:39 Hands-on: Adding Data and Validation

23:50 this, but the ID is the file name. Is that still correct? Yes. The what you would consider to be the slug, the the first part of the file name is the ID. Can you see my Versus code is telling me that the markdown extension does not understand backspace or on key enter. And I cannot delete or press return. Maybe you should kill that extension. Yeah. I think I will be disabled in the markdown extension. That is particularly weird. And I'm pretty sure that's a baked in one. That's not one I install. Right? Well, you really made Versus Code mad. I'm

24:42 not sure why. That Docker running. That's normally what happens. Now you've got some markdown stuff installed there. Oh, no. Those are No. Not the installed ones. You can do how'd you do the weird search? Is is installed colon true? That's really struggling. I don't know. I'm gonna use Vim. Okay. Data people Rawkode. Okay. Comment from Kevin in the chat. Did I break something again? It's not even Thursday yet. My computer has just been technically unhappy with me lately. I'm not unsure why, and this definitely hates me. So this is a markdown fail, so we just need to indicate that it's

25:36 front matter. In fact, I'll just change that to YAML because then we can show the body processing that works in CueBlox, which we haven't covered. I'm gonna modify our schemata slightly. In fact, yeah, doesn't matter because we support both. So we need to rename this to data people raw code dot yaml. Okay. And we'll add one more, which is going to be our article, which will be helloworld.md. I don't have any more than don't have any react stuff, so we'll just do that. Okay. And this requires a title. So hello world and a body, and the body will

26:19 automatically be inferred from what comes after the front matter and the document. Now as long as I haven't messed this up, but I'm sure Brian would be in there quick if I had, we should be we should be in a position now where where we can run block build on this directory. It'll load the schemata. It will validate the data, and it'll tell us it was good. And it it worked. Okay. So what actually happened there plugins. That's so sexy. The plugins are pretty great. Yeah. I'm really happy with that. Oh, we just just added those. We didn't add the

26:54 author. Did we okay. I've got a typo there, but hopefully this still works. Good. Okay. So I've now contrived the situation where the data is incorrect. Now the BlogsBelled by default does not handle referential integrity, but you can see from what I have in data people Rawkode and my ID, so what I've used to reference this entity as the filename of Rawkode. However, under articles and then hello world, we have person ID, which is loaded magically as a relationship should fail. And we can pass the dash I flag. Oh, it worked. Does it not do relationships without the relationship

27:44 tag? Did we remove that? I think I made it I made I think I made it before. Alright. We're just gonna pretend that that gave us another message and said that ID doesn't exist. Oh, add the relationship tag and show it works. Yeah. Yeah. Yeah. Brian and I weren't happy with that approach anyway. Because when you expose this data to other tooling, it also has to be aware that person ID is a reference to some entity called person. And in my GraphQL API, I actually want it just to say author, like so, and still work. Now the way that we

28:20 do this through CueBlox is by using an attribute called relationship on our data and telling it the entity to join the thing too. So I'm gonna go out on a limb and break this again and and and just hope that something that I've written here works. So now we have Just you can add that relationship tag in the data? I thought that has to go in the schema. You are correct. Thank you. Or should we run a block spells and see what happens anyway? Because now we have a field that shouldn't be there. Right? Correct.

29:02 Why is we not getting our messages? What's your schema look like? I'm going to set it to 23. Give me one error message. I've broken something. I'm just going to keep playing. Okay, let's modify my schemata now. And we're going to now say that we actually want this to be a relationship to a person and we don't want to use the magic IT thing. And we'll go with author. Okay. Oh, we're not getting any error messages at all. Dataset name and namespace. Pull up your schema again. There's gotta be something wrong with the schema. The builder effect's okay.

30:03 Oh, we're not getting the article. Okay. No. Something is wrong. My plurals are lowercase in all of my I don't know if that makes any difference, just looking at my data compared to okay. Build data schema static. Schema. Maybe supported extensions as required. Let's find out. It is because if I I can think of the line of code in my head if it checks to see if the extension isn't in support supported extensions, it's not gonna happen. And then what do we have in build? Okay. So it's not that. It's definitely me broken something. Maybe I should just jump to the dog

30:57 fit repository. And change the plurals to lowercase, see if that makes any difference. Maybe it won't, but Yeah. I don't think it it shouldn't matter. Data set, portal, ported extensions. Then data. Oh, no. I made it worse. Someone stop me typing. I mean, come on. Wait. Don't you need can you person. Alright. Okay. So now we don't have articles. So I've I've got it closer. Why are my articles not being loaded? I'm gonna copy this. I must be doing something really silly. Yeah. I didn't want the right extension. Right? Phew. Okay. So supported extensions was required. Then I spent then I spelled

32:18 it supported extension, and then I got it right, but doesn't include MD for it to load the file. I'm gonna go out on a limb here. Right? And I'm gonna I'm gonna go back to the magic first. All right. Let's see. Okay, so person ID lock, oh no, I need to update my data. I thought I could leave that. What'd you think my notes are here? Is it gonna work? I don't know. It does. Okay. So everything works as long as I am not an idiot. Okay. So this is our our let me go back into our schema because I I

33:05 know I typed a lot there very fast. Right now we have author with the relationship attribute commented out. It has no longer been listened to or adhered to by CueBlox. However, we have the person ID and the elephant underscore ID as a magic relationship. It will be inferred to the type where a person, lowercase version of the definition name here. And what you'll see in my data where we define the article is that we define person ID as Rawkode. And when we run the blocks build with the dash I, so the dash I just means verify

33:38 the referential integrity and we can ignore this. This is that field that shouldn't be allowed. It says that we have a referential integrity failure because data articles and then the hello world article, the person ID does not exist as a person, which is perfect. So now if we come in and we fix this and we say you are Rawkode and we build it again with referential integrity, things will work. However, we've now got a field error that we don't want. So we'll remove the magic. We'll go back to our schemata, and I'm gonna bring back the less magic,

34:13 but the one that is more flexible with regards to the naming of the field So we can use the at syntax in Cue to define a relationship where the entity or the definition is a person. Yeah. And just to be clear about that at syntax, that's just a tag in Cue, and we're using it as arbitrary metadata. So we're creating a tag in that definition and because the tag is relationship, we're treating that data a specific way. There is no there's no relationship thing in queue where we built that code. We did build that, which is probably why

34:52 I'm having so much problems now. So now it's telling me that my author doesn't exist. It's at relationship. Right? It's at relationship and then capital person. That's how mine work. Yeah. Try that. It told me to go away. What have I got wrong? Oh, that's fine. I actually do have a referential integrity. Oh. I am making an absolute new Every time. Again, when you do things right, they just work. Okay. So we have a schema. We are defining two types within the system and we can see how adding data to this works, validates integrity. And we could take a look at the

35:50 build output, which build, I need to cat it and then put Or I can get a second and put it. And we get this JSON blob at the other side. You'll see the body is automatically parsed through the markdown and the front matter. So that works together really nicely. The ID and the author are all joined there. So that's another JSON payload that you can work within your application to consume all of those types and you're got this guarantee that any references to other types within the system just work. That was a lot harder than it should have

36:30 been. Yeah. And this this for the first couple months, this was our our the the extent of our output. We we output this data dot JSON file, and that was what we use to build and and populate everything else everywhere else. But only recently have we exposed the Cue database, as an API so that we can use other tools to read that database once the validation has been completed. And that really changed a lot of things. That that's what allowed us to do GraphQL and HTTP servers without exporting the the data to JSON format, which saves

37:11 Hands-on: Serving Data via GraphQL

37:15 IO and time and processing power. Alright. I will show the surf command, and then I'll hand over to you for all the fun stuff, if that sounds good to you. Sure. Nuno has jumped by the chat, someone we we both know. Pass him by to see how amazing you both are. Thank you, Nuno. I appreciate that. Okay. I guess I could well, I run GraphQL and and Dogfood or rather than my my cellular data here. We have relationships and stuff there, don't we? Yes. There's lots of relationships in Dogfood. Okay. So we actually this is one of

37:20 Serving Data as GraphQL

37:52 Brian's really cool initiatives. Oh, there's some work I've been working on. Stash it. So Brian, you built the docs with the pyramid and you did that all for CueBlox, which I think is really cool. So I'm gonna run a blocks build here and you'll see we have a lot of different types. Do we stall your dog food, schemata. So we have a concept of an article. We have categories. We have images. We have pages, profile sections, and websites. Literally, our entire website is defined through CueBlox. Articles here, and we've got some really cool stuff, I guess, we could talk about. We

38:36 have the ability to template through attributes. So you can actually use the Block CLI to generate new YAML and markdown files and use the templates as initial values. Cue allows you to set default values. We have our relationships here. You can even have lists as well. One more that I think is maybe quite interesting is do profile. Profile has multiple types. So you can see you can define a Twitter account, a GitHub account, a miscellaneous social account. And then social accounts can actually have an arbitrary list of any of those types. And so really cool use of it kind

39:14 of enumeration types within queue. And we can actually generate the Twitter URL from other parameters within the queue value itself. And you can see we interpolate username into twitter.com. So the dog fit directory, if you just wanna learn what we're doing with CueBlox is probably one of the best ways to do it. Let's run Blocks serve. So what Blocks serve does is validate all of that data, generate a GraphQL schema, and then serve it for you over HTTP. So you'll see here, we have a server running on 8080, and that's that pop over here. And what's really cool is that if you

39:58 don't want to use the don't go to HTTPS. Just go to oh, it's redirecting. Is that Firefox been really annoying? Could be. That's one Firefox. If you use 127001, will it do the same? Oh, I think it's just because I'm not doing we serve on GraphQL. Yeah. But Firefox keeps putting that HTTPS in there because it's a word, not a number. Where does that GraphQL thing live again? It's configurable. I just can't remember. It's configurable. I should be more prepared. Positive story. Looking at the code. UI. The GraphQL UI is under slash UI. Thank you.

41:28 Which makes perfect sense. Right? I'm sure I did that. Well, there's more than one. There's this one, which is the kind of fancy UI, the playground one, and then there's the graphical one, which I can't remember where we made that live. However, we should be able to say, hey, I want all articles and then use my all complete. So you give me the title, the body, and let's get a nested attribute. So we want a profile and we want the first name and the last name. And then if we hit go, we get all our data back over GraphQL.

42:10 And I think this is just amazing. It really is. I gotta make a huge shout out to you. You plumbed so deeply into the depths of Cue to make this GraphQL handler work, and I still have zero clue how it works. This is all you, and it's some of the most amazing code I've seen put together in a long time. So kudos. This is really awesome. Thank you. I've I've never blushed on my own stream before, so there you go. You're responsible for that. Well, turning YAML into a a full GraphQL dataset, that's not a small feat.

42:43 Yeah. And we'll highlight some of your work with that because I I'm not I'm not gonna let you pull credit this way. But this also works with serverless, but not with the serve command. Right? Because users work with Azure, Versal, Netlify. I think there was one more where the there is a static GraphQL JS library that can serve the build output, the build artifact from a blocks build over GraphQL too. So, like, you don't want to run servers, you can still make this work. However, you will lose access to the relationship attribute, but Brian and I are working on that too.

43:19 Or you can use the block server command, which builds the schema on the fly for you. And it you can also run blocks serve in all of those serverless environments because all of the ones you mentioned support Go. So that's that's not hard at all. In fact, I have a repo somewhere. I don't know. I've got so many repos. I've got an example of that somewhere. We can dig it up when we're we're doing our show and tell. Alright. Sweet. Alright. Well, that is that's CueBlox as its basics. We can initialize in your repository. We can very quickly using

43:54 Cue define our schemata. We can define the types within that schema. We can start to generate data. There's data data. What was the command from that? We've got a lot of commands. Let's list them. And there's new, which allow you to template. So generate new data, YAML files or Martin files based on the schema itself. The remote commands are managing this schemata. The render one is really cool, which Brian's gonna maybe Yeah. Renders are small, so that off. And plugins, which I I don't know anything about. I'm gonna I'm gonna I'm gonna throw that over to you

44:28 to to do that justice. I snuck those in under the fence this last week and David approved them. I think you read the code though, let's be honest. Yeah. I think we're in a position where, you know, we were adding a lot and you you raised us a week ago, maybe two weeks ago that we we just got a lot of code and we need to find a way to make this to extract some of it and make other make it easier for other people to also contribute plugins. Yeah. If you wanna pop your screen off,

44:55 once that's there, I'll jump over to our other screen share mode and we can show off plugins and render. But if you have if you're watching, you have questions, you're curious about maybe you have a use case for CueBlox and you wanna know or you want us to confirm that, yes, that would be an awesome way to use it, jump into the comments. And if anything I have covered you want a bit more detail on, just drop that in the comments too. Is that screen share coming? It's coming. Is the computer behaving badly? Oh, Edge wants to control my screen or

45:41 share my screen, and I have to remember what my password is. It's then gonna ask you to restart Edge, which you don't have to do. You can just say no, and it still works. I do that a lot. Good. Later. Alright. So I'm gonna do a window, blocks. I cheated just a little bit, and I put ton of projects into one workspace so that we can show them off differently, separately. Your Versus code is now live, so feel free to to take it away. It's live? My Versus code works. So we've got that as a starting point here.

46:00 Blox Plugins

46:20 This is a working Versus code. And even if for some reason I could have the strange problem where backspace didn't work, I use Vim key bindings, so I could just d d and get rid of those files. Yeah. Show off. Yeah. Well, it's not so much a show off. It's it's muscle memory. And I blame Eric St. Martin for that. You know, twelve years ago, he's like, you need to learn VIM. And once you learn VIM, it's really hard to unlearn VIM. Alright. So, one of the things that I built very recently is the idea of plugins. We have

46:58 we have a ton of commands. Let's look in our internal command directory here. There's a ton of commands. And I was looking at the the code that manages the remote repository. So blocks build and blocks serve and blocks render, that's all use in your local repository. But then we have this whole concept of a a schema repository that you can create for your company or your group or your team. And there's a bunch of commands that manage that, and it feels to me like that code doesn't really belong in the same place. It doesn't belong polluting our

47:37 blocks help file. It it should be somewhere else. It should be a plug in. So I went down a total rabbit hole and found the HashiCorp plugin code. Let me see if I can show an example of that. And I'm really impressed with this go plug in package. It allows you to create an interface, to find an interface. And in the interface is just a contract between the caller and the thing that's called and it uses RPC or gRPC to talk between the the server and the client, and it it could not have worked any

48:19 easier. So I I I thought about it, and for the basic use cases, we really have kinda two things that we need to do. We need to have, things that happen before the build and things that happen after the build. So I created prebuild and postbuild as plug in interfaces. And then the GoPlugin package has this concept of a magic key and a magic cookie, the key and the value. And those are just handshakes that that let the, the client and the server know that they're both speaking the same protocol and and they should be,

48:58 talking to each other. And and like the comment says, this is not a security feature. This is more of a are we speaking the same language and and should we be talking to each other kind of feature. So we've got these, handshakes configured and the interface is configured. It made it really easy to build plugins. So let's take a look at, the first plugin I built is blocked images. And I am obsessed with serving the right size images for the screen that's on on display and being able to use the height and width fields in my image

49:36 tags. So I built this plugin that reads all of the files from the static directory and that's declared in your queue or your blocks dot queue file. And it goes through and uses this where is it? Imaging dot open. So it reads the reads the file. If it's if it's an image, it reads the file and then, modifies the YAML or actually creates YAML, excuse me, with a file name, the height and width and then potentially a CDN endpoint which would be a prefix for the URL. So because this is a pre, plugin, we're actually creating,

50:25 data that will then be compiled and built by CueBlox. So let's look at that in practice in my blog. I've got my blocks dot queue in the root directory of my blog. And you can see my data and schema, all these directories are a little bit different than the demo that David showed because I want to be able to share these directories among, both CueBlox and the blog itself. So specifically, the static directory is public static. And that means that when CueBlox runs, it's gonna be looking for static files in that public static directory, which is the same

50:31 Plugin Demo: Image Processing (Pre-build)

51:08 static directory that Next uses when it's serving images. So I've got all these images in my public static directory. If I open up a terminal sorry. I'm on a strange keyboard. Where's the terminal thing? Here we go. I put away my loud keyboard. Yeah. I can hear you typing there. That's that's different. I like it. Get rid of my custom built blog or blocks and make sure that I'm using the one that I brew installed. Oh, by the way, Blocks is available on Homebrew. You can use that on Linux or Mac OS. So if I do a Blocks build,

52:03 what it's going to do is run those plugins. Let's go back to my Blocks queue. There's a pre build section with an array of plugins and a post build section with an array of plugins. So I've got a plugin called Blocks images, and the executable is Blocks images. Let's make sure that exists. So if I type box build, it's modern. Oh, I need to be in the data directory. I do? It said that your data schemata directory didn't exist. How can that be? Where did it go? It it's really gone too. Git restore. How about that?

53:14 Uh-oh. Yeah. That's frightening. Where's the schemata directory? I bet it got moved. Sections. Look at that. See? It's schemata. It got moved. I do that all the time on my laptop somehow. I fat finger and move things. Alright. Let's see if it's back. Oh, it's back. Good news. Alright. Let's try that block build again. There we go. So sir. You see that it's running the plugins. It registered the blocks images plugin. It initialized it, and now it appears to be doing nothing. But what it's actually doing is blowing up because the post plugin isn't there, but that's

54:03 okay. The pre plugin actually ran. And if you look at the public static images directory, you can see this directory structure and a bunch of images. And now we go back up to my data directory, and you can see that my my prebuild plugin created YAML files for each of those images that has the height and width and the file name. And because it has those, now in Next. Js in the templates for my my website, I can specify the the image information directly from that slug or directly from that YAML file and, get the nice pretty image

54:50 tags from Next. Js that does the responsive image sizing. Very cool. I like that. Right. This is the most talking I've done in three months since I've been on paternity leave. Yeah. I've done a lot of nursery song singing, though. Yeah. You've only been back, what, less than a week, and I've already dragged you in front of the camera to do a stream. So, you know Just a couple days. Yeah. So that's the prebuild plugin. It creates all of these YAML files. And then if you look in the build directory here, you can see individual data files because that's

55:29 another thing I added recently. Instead of just a single data dot JSON file, which still exists, I also break them out into, both record set level data and then individual record data so that I can serve them from a CDN, and that's part of my post build process. So that's a great segue into plug in number two, which is static sync. Static sync takes the output either in the images or the data that's built and syncs it up to any CDN or cloud bucket that's supported by go go cloud. So in my case, it's using Azure's

56:00 Plugin Demo: Static Sync to CDN (Post-build)

56:23 blob storage with a CDN in front of it. Bucket missing. I must have Did you use DRAM when you went into that directory? M r c. Well, I just have to be careful not to give away keys here. Edit bucket. We should be there. Bucket missing. Oh, I might not be signed in to Azure on this machine. I do most of my development on a different laptop. So Uh-huh. Possible that I'm just not signed in. But what this plug in does is it's actually a good example of that interface, that plugin interface being used twice.

57:23 So in the main, I've got two different plugins defined in the same binary here. So I've got the static sync plugin and the data sync plugin, and those are each in two different Go files. So the data sync plugin reads the build directory, that output directory that has the data dot JSON file. And the static plugin reads the build directory for the stat I'm sorry, the static directory for our images and and other files. So by setting a couple environment variables and running this plug in, the output gets synced to two different buckets up in Azure storage, which

58:01 are fronted by a CDN. So you can go to images.brian.dev/images and get all of the images that, are in this static directory. But also the build data gets output as well. So instead of wasting any time running blocks serve or spinning up some sort of, Node JS server or anything else like that, I can now just call images.brian.dev/data/articles.json and get the latest articles as JSON file, and they're cached with an appropriate time to live, and the build, process automatically kills the, CDN cache when I rebuild it. So a really awesome and great way to take advantage

58:55 of cloud storage and run one less server. So those are the two plugins. I'm sorry. Go ahead. Yeah. I I think what you know, just to clarify is that CueBlox doesn't try to be too opinionated on what you do with your data, but instead open up pipelines for whatever you want to do with your data. Like, we have completely different ideas of how our data works on our our site, but CueBlox is still the critical bit of infrastructure in the middle that allows all of those pipelines to happen, which I think is just, you know, I think we

59:28 got the experience just right with a lot of that. Yeah. I agree. And and that was because of all of the meeting in the middle about how things should work. It's flexible enough that it handles both of our unique styles of of serving and and consuming data without getting in the way. And a great example of that, if you look at the GitHub workflows for the blocks project itself, in the main, this is the thing that builds CueBlox and runs GoReleaser, and it also runs our, where is it? Data. Here we go. So this is the

1:00:13 the thing that publishes the data and then also builds our websites. All of that works across the same set of data, but can push to any number of different endpoints. This particular GitHub action is pushing to, Azure static web apps, but we've also got some in here that push to, Vercel just as an example because we can. And it's it's really fun taking that same data, using it as many different ways as you want. Here's another example of that. Let's see here. I've got this particular dataset. This is my b k m l library, and it's actually served by a Go binary.

1:01:10 And this Go binary has the HTTP and GraphQL handlers that are built into CueBlox, and they're served as an endpoint, one for GraphQL and one for REST. But then I also added a image handling service that will resize images and perform, image manipulation if you want to. And then also an OG image creator. So you can pass in, text and it will automatically create OG images that look like this, for example, for Twitter and social media sharing. So completely different take on serving the data, and this particular one gets served by, Fly, fly dot I o, which is an

1:02:06 amazingly flexible service for running stuff. I'm having a lot of fun playing with Fly. Yeah. Fly's pretty neat. Really impressed with their service as well. Yeah. So that's that's a good example of taking that same data. So this is the same it's actually a copy of the data in my blog. So it's the same data, same articles, images, whatever, and it's being served in a completely different way, over fly. So that that lack of opinion in terms of how to serve the data, has been a lot of fun. Alright. Shall we take a look sorry.

1:02:48 Hands-on: Rendering with Templates

1:02:48 I need to ahead. No. It's okay. Just wanted to show off the templates real quick. So here's here's another repository. I've got a lot of these laying around with different datasets so that we can make sure we're not getting too opinionated about our own datasets and things work well with other use cases. So this one, we talked a little bit earlier about blocks render. This one has some templates in it, and the templates build shell scripts. So if you look at, the articles.text.tmpl, it will run for, the articles dataset. So you can see there's a range in

1:03:31 in Go's template format. So for each article, it's gonna call Azure OG, which will create OG image, and that's just a quick Go app that I wrote that does the same thing as the OG image we looked at just a second ago. So this creates a shell script that builds the OG images and then, executes it. So this this first one creates the shell scripts. The second one actually executes the shell scripts. And I think that's just so much fun. That's there's so much power in that, generating scripts and then executing them all based on the data.

1:04:11 And then we've got, that's something I think I was just playing with. Here's one that creates OG images for each of the pages in the website, and then here's a template for the RSS feed. That's it. There's there's almost nothing to it when you think about this. This is almost all boilerplate for RSS and just a tiny little bit of Go templating. And So cool. I I didn't have to write hundreds of lines of anything to make this happen. I just ran it through the template engine. That's fun. Yeah. Definitely. I didn't know the RSS one which is

1:04:49 sitting there in your repository. I'm I'm definitely gonna be stealing that at some point. Good. Yeah. But I think it's just really cool the way CueBlox gets out your way and just lets you manipulate the data in this way. And the way the render the render sub command with the templates works is just, you know, it's so simple and yet so powerful at the same time. And I think that's just the sweet spot. I really love it. Yeah. And this this repository, this is the b k API repository under GitHub, b kettleson. This one is

1:05:23 built specific to deploy to Vercel. So this one deploys a Vercel serverless API using, the JavaScript engines. And let's look at that real quick. Here's the API. This uses that JSON GraphQL server we were talking about just a bit earlier and then JSON server. These are two really generic services that we found just kinda screwing around on the Internet one day, and JSON GraphQL takes data that almost exactly happens to be shaped like the data that we build in blocks, and it serves it as a GraphQL dataset as long as it follows those same conventions

1:06:08 of using the underscore IDs for relationals. And then JSON server does the same thing except with REST. So this is, 20 odd lines of code in JavaScript that create a an express, endpoint with both REST and GraphQL and also image handling in Vercel. So lots of examples of very strongly opinionated ways to serve this data in very different ways, but it's all the same tool underneath. And that's that's really fun. Yeah. Definitely. Alright. Is that everything we we want to look at on your side? Will I pop us back over into, like, face mode? Or Sure.

1:06:58 There we go. So, I mean, we're just gonna we'll wrap this up though, but I think this has been a lot of fun. Like, we haven't really spoken about CueBlox much, like, publicly or on a stream. You know, we've just been kind of iterating on it. What I would like, you know, the audience that are watching is to to play with it and give it give us feedback. Like, if it doesn't fit your use case, we'd love to know why, and we're still iterating on it a lot. We wanna add more features. After what I've just seen from the plugins

1:07:01 Conclusion and Call to Action

1:07:28 that Brian's done, I think we're gonna be adding a lot more plugins over the next kinda weeks and months. So maybe we should try and get together and just do this regularly as an update to CueBlox, maybe every other month or like that. But I think That's a great idea. Yeah. Yeah. I've got literally a dozen plugins that I wanna build. I mean, now that the plugin engine is so easy to use, I I can think of just so many different plugins that I can add without polluting the Blocks code base, and that makes me so happy.

1:07:57 Yeah. One of the ones I think you were talking about, and I think I'm really keen on, I was getting to work sooner rather than later, is, like, you know, leveraging all of these cloud databases that have, like, a free tier. Like, know, being able to run a blocked build and have that first validate all my YAML, markdown, and JSON, etcetera, but then store it at a cloud database. And I don't need to worry about the block server, and I maybe just get access to the Contentful API or the FaunaDB API or the DataStax, AstroDB API.

1:08:26 It's just it's just there. Right? Like, I think that's so many, you know, planet scale. There's all of these different great cloud databases that have free developer tiers that would easily serve any traffic that my blog could could send it. And, you know, these plug ins make it really easy for us to build an adapter that that synchronizes our data. In fact, this morning, I saw Supabase, which is, yet another, Firebase clone that is hostable both for them and you can self host it. And that inspired me because it has real time eventing. So you can you can notify your

1:09:07 web clients when there's new data added and that big kind of fun plug in to build, sync my blocks data up to Supabase and and build my website so that when I post a new blog post, it gets up in your face and annoys you and tells you that there's a new post that you should be reading instead of the one that you were reading. Yeah. That's really interesting. Like, imagine we had like a pre build step that checks what's changed since last build, like using the get history, and then propagate that through to a post

1:09:36 step that actually fired off those only publish those changes to, like, super base or to FunnelDB, etcetera. I'm not sure how that would work, but I think we could make that work. I've got faith in you. Well, I was gonna say maybe it's not necessary though if we just upsert all of the data. I was thinking about deletes. Oh, yeah. We need to work at that. Good point. But we we for that our next session, we'll we'll have that working. I'm guaranteed about Yeah. We'll link that through. Alright. Awesome. Well, I hope people are are

1:10:06 interested in CueBlox. I think you like what you're seeing. Like I said, drop us comments. We're both really active on Twitter. Both of Twitter handles are on the screen, so reach out to us if you need any help. We're both in the Discord. You can ping us there. Brian's got these fancy titles. I'm gonna have to steal that. I don't know how I'm gonna make that work, but I want it. Any last words, Brian, before I let you go? No. Thanks for having me on. This has been great, And, this is truly one of the most fun projects that I've worked

1:10:30 on, in in forever. You know, we've we've had such a great time iterating slowly, throwing things away, and the and the debate we've had about how things should work has been really educational, and and I'm proud of of how this is coming out and how it's evolved. I couldn't agree more. It has been. If if folks check out CueBlox, please do let us know what your pain points are. We we tried to put a lot of energy into the documentation, but some of it might be outdated. You know, if things don't make sense or if you stumble somewhere, let us know where

1:11:07 so we can get those docs cleaned up and make it easier for people to jump in. You know, so much of this is locked into what we intend versus what we've written. And, we hate for people to not be able to enjoy CueBlox because didn't spend enough time on docs. Awesome. Well said. Alright, Brian. Thank you for joining me. Have a wonderful day. I'm sure and hopeful we will do this again soon. Well, thanks again. Thanks for having me. Bye all. Thank you for watching Rawkode Live.