0:53 Introduction & Housekeeping

0:53 Hello and welcome to the Rawkode Academy and today's episode of Rawkode live. Today, we are taking a look at Docker's BuildKit and BuildX. Now before we do that, there's a little bit of housekeeping. First and foremost, please subscribe to the channel, tick that bell and you will get notifications for all new episodes of Rawkode Live and Clustered and everything else in between. Also check out the membership options if you wanna take advantage of some of the courses that we have coming out on the academy. Currently, are working through the complete guide to InfluxDB two. Also, we have a very active Discord server

1:28 available at Rawkode.chat. There's over 500 of us in there talking Docker, Cloud Native, Kubernetes and everything in between. So come on, say hello, and I look forward to meeting you. Okay. Today's session, BuildKit and Buildx. I am joined by Thomas Tighe, a maintainer of BuildKit and an engineer at Docker. Hi there, Thomas. How are you? Hi, David. How are you? I'm doing alright. Thank you. Can you just for anyone who is not familiar with you, although I'm sure they have all seen your GitHub avatar on every repository in the Docker ecosystem. But just tell us who

1:42 Meet the Guest: Thomas Tighe

2:02 you are and a little bit about you, please. Yeah. So I work at Docker. I'm a I've been, like, a Docker maintainer for quite some long time. Think bunch of different stuff there in the early days, like like some some storage stuff and and d and the ones working integrations. And and the last years, I've been, like, more looking at the build side. So I started with with the multistage builds that's happened, like, a couple of years ago already. And then and then, like, kind of, like, boost up the BuildKit project and later on, like, BuildKit

2:43 project as well for And yeah. I got I guess that that that's where if if you've seen my amateur, then then these are probably probably the repositories there. Awesome. Well, thank you for that. So, yeah, multistage builds were just awesome. I think everybody makes so excited with that. And then the BuildKit projects just seems to be bringing more and more functionality to to what we need to build our images in an efficient and, you know, super cashed way. But for anyone who's not sorry. Let me go. Yeah. Hope they're going in in in the same

3:18 direction. Like, it's it's a it's just like I guess it takes a little bit time for for, like, the main mainstream places to catch up. So we are, like at the moment, like, very loud. If I if I look at the, like, the Dockerfile talks or something like this, then, like, every every talk is, like, multistage builds. So it's like but, like, actually, it's, like, happened, like, already, like, a three years ago. So, hopefully, like like like, in in a year or or two, we will see, like or 50 to have, like, a synchronous session.

3:50 Nice. Okay. So let's break down the two different components we're we're looking at today. We're gonna be looking at something called BuildKit and something called BuildX. Can we just explain what both of those are to the audience? Sure. Sure. So so, of course, like, Docker itself has always had, like, a mount well, sorry, like, a build endpoint. Right? So the there's Docker Docker build command. Everyone knows about it. It has been there for the beginning. So what BuildKit is is basically, like, rethinking the the background component of this from break, like, ground up, and it's done in a

3:51 BuildKit vs Buildx: What are they?

4:25 separate project so that so that not only Docker can use it and and and we can, like, basically, like, rethink, like, what it what it means to have, like, a cool builder in, like, a modern container ages. And and one thing for maybe or not about BuildKit is that might be a little bit confusing is that if you're just, like, building images, you don't necessarily need to care about BuildKit itself that much. Like, it's more like an internal tool. It's more like builder. But if you're, like, building your own builders, if you're, like, developing a language and want

4:59 to make sure that it builds fine in in containers. If you want to, like, contribute to BuildKit and things like that, then you then you would look at BuildKit. If you're just building images, you're probably just using a tool that's using BuildKit under the hood like like Docker or or or, like, there there's tons of other other ones as well. So although although, like, sometimes we, like, use BuildKit also as, like, an umbrella term to just, like, talk about, like, the it's like a modern stack or something like this. But but but yeah. Mostly, it's like a back end

5:33 back end component. It's very important for making all the builds efficient and all those things. Not necessarily something that, like, a user actually needs to care about. Alright. Awesome. And can we tell people what BuildKit is then? So, yeah, the BuildKit is more like a Docker side of of a component where we we like where we develop our, like, Docker UI for for the build. So these were, like, the Docker Docker user experiences. Like, on BuildKit side, we're, like, very generic. We just we don't want to be opinionated or anything like this. We just want to

6:08 provide the back end that that works for everyone, for for every use case. In build BuildKit side, we're taking, like, the Docker user experience that the the users know and just tries to, like, add add new stuff when it build on top of it. BuildKit itself is like a CLI plug in for Docker. So it comes with Docker. It's like a Docker command, but it's actually, like, a stand alone binary. So we can develop it separately and and in addition to to, like, just like a build command, there's also, like, a ton of other command required.

6:42 I'm sure we we get into there, but what what what's in there, but, like, build exit. But when the old build command is just, one build command, Then build exit, like, 10 commands or something like that. So it it, like, it provides a lot lots of other use cases that that weren't there initially. Nice. Well, we will be playing with both of these in just a minute so people will get to to see them hands on, which would be awesome. I think there's just one more thing I'd like to kinda talk about before we move into the hands on

7:09 component. And before I say that, if anyone has questions in the audience, please drop them into the chat. We'll keep an eye on it, and we'll do our best to answer them as we go. Okay. So my final question. BuildKit. Like, it just I don't I don't know if it's just me. I wonder if you feel the same from your side, but the adoption of BuildKit across other tooling seems to be the velocity of that just seems to be going really fast. Like, I see BuildKit being used as like a a cache or a build layer for so many different projects

7:20 BuildKit Community Adoption

7:38 from, like, Dagger to Airfleet. Is that something that you're seeing a lot of uptake on as well? Sure. And, well, like, from my side, I'm I'm glad that it's happened because, well, that that's what it was meant for. Like, it's it's it's kind of like the standard, like, when when we're, like, doing any kind of build flows and, like, including containers in there, there's definitely, like, this shared part, that's common part that's, like that's actually very hard to develop, very hard to get right. And, like and it doesn't make sense for, like, every tool out there to try

8:16 to invent their own. Like, that that's how, like, the, like even, like, the old Docker build, like, it did something, like, just for for their use case. So, like, didn't quite think of it maybe true in the in the first first version and, like, ended up something that's, like some some cases work for Docker, didn't really, like, allow further development, didn't in fact for everyone anyone else. So, like, so, yeah, that that's what BuildKit is for. Like, like, anytime you use some kind of, like, a container as build flow and you need caching support, you need

8:54 to need to be fast. Like like like, that that's what you should be using. Okay. Thank you. Alright. Well, let's get my screen shared and try and work out what we're going to do today. So I've got the BuildKit repository here, the BuildX repository here. I have a local directory with two projects that I'm familiar with, one in Go and one in Rust, and we have a shared Versus Code live session so that we can both type as we try to explore this. So I believe what we're gonna do is maybe start with a goal project, the blocks,

9:02 Hands-on: Improving a Go Dockerfile

9:31 and try and take a look at some of the new features that BuildKit exposes to make our build system cache efficient using multi stages and other things. I just tried to explore some of these features that BuildKit brings to the table. This thing good? Sure. Sure. Let's start with that one. Alright. So here's the fun bit. Oh, well, thing. And currently blocks doesn't build inside of a Docker file. We use GoReleaser and let's change that. So we're gonna attempt to build our Go application and a Docker file using a multistage builds and take advantage of as much caching stuff

10:00 Building a Go Project with Docker

10:07 as we can as possible. And I've heard a rumor as I've not been able to try it yet that we have the ability to build binaries out and put them on the local machine and stuff. So we're we're gonna try and explore this and and get your knowledge and experience along the way as we go to. Sure. Sounds sounds good. Yeah. Yeah. Like, this kind of Docker file definitely feels like that. The biggest thing we can do in in Docker files, like like like, if if if, like, anyone watching this learns one thing, then, like, don't don't, like, put binary or, like,

10:35 copy binary in in some type of time. Yeah. We could we could definitely improve this. So but Yeah. So so, yeah, like, I assume, like, this one, you you you call it from, like, a make file. Right? And and you you do, like, a local go build and and then and then you call Docker build after that there. Like, if you look at Yeah. We've got go build here. So yeah. Not not perfect. Right? We we let's get this into a container. So Yeah. And and what what's the other one? The the other one was generating

11:10 talks or something like that. Yeah. We have a command that generates the documentation. Yeah. Well, we'll see where where we get to. But we we can we can maybe do something for this one as well. Yeah. Alright. Awesome. We got some love for BuildKit speak in the chat. So we'll take a look at that as we go, I'm sure. And we have a couple of questions already. I don't know if you wanna tackle them now or not, but let's I'll let's mention them and then you can decide if we tackle them now or you bring them in

11:42 later. So Russell is asking, I'm hoping we can find out some juicy details. I've never heard juicy and Docker used together, but I like it, Russell. And something like cache being unprotected and a good target for injecting extras into images. I'm not sure I understand that question, Russell. Do you? The last part, maybe maybe not. I think we'll get to, like, yeah, like, the remote caching. Like, we we can we we we'll probably cover that as one as well. But first, I think we need to fix up this Docker file. Alright. Okay. I'll let's just do that first. You're right. We

12:17 should. So let's pull up our why can't I see? There we go. So I'm assuming we're gonna use a multistage build here and maybe start with, like, a a goal based image. Yeah. I think we can, like, just put, like, a goal and something in here. Right? Yeah. We're we can we can first do, like, a very simple one. So, like, that that's also not good, but, like, show that, like like, it's not that hard. And basically do the same same whatever build you had in here. Right? I don't know. Or it's something like this,

13:04 sir. Yeah. And let me just I think that was pretty much spot on for the command. I don't know if we do a Google build command box, but I'll double check. Yeah. I'll just copy this. I see MD. Okay. Yeah. So we'll need okay. So in theory, this should actually give us the exact same thing, but without a local build step. Yes. Well, like, you have Ubuntu in here. Like, I I just put Alpine because, like, I I promote my stuff. But but, yeah, like, that doesn't matter. Like, it could be it could be Ubuntu in here as

13:51 well. And yeah. And and, of course, like, the benefit of this one is that is that whenever someone, like, check checks out your repository, they don't need to have exact setup as as you have. Right? Like, don't need to do this part of the build on the on the host and part of it in the container. Like, everything is in a container. Like, everything is defined by by you as a project owner. You always get the get the same same versions everywhere and and things like that. Like, you can even, like, just be nice in here and definitely configure

14:29 about this. I don't know if you use code to one seventeen at there. Oh, yeah. Well, it's, one sixteen minimum because we do use go embed on this project. So one seventeen, that'll be alright. Okay. Yeah. I keep them even, like again, if you want to, you could make data, like, a best matrix, make this changeable and and things like that. Right? So but, of course, like, what's what's bad about this one, right, is that your source code is still in here. You have the whole cover on time in here, things like that. Like, you don't you

15:03 don't want those. Right? So so, yeah, let's let's make it on this stage right away as well. Yeah. So Alright. It's a bit stupid then. Point point. Yeah. And I I think that should be enough for the first run. Right? Do we want to try to build this on? Yeah. Be rude if it's not to try it. So okay. I'm just gonna go sorry? Yeah. We can just click. Maybe try BuildKit. So Docker BuildKit build or, like, let us foresee what what you do you have in stock in here in the in the system. Yeah.

15:48 First Build with `docker buildx build`

15:57 Normally, I would just do Docker build. I'll give it a fake tag for now and then pass in my context as dot. Like, you're telling me that we should evolve this now? Well, we we can let's let's let's let's try and see what it does. Okay. This one is a little bit messed up for some reason. For oh, it's just messed up for me. I can see that the other stream is correct. The in the in the in the live share, it's it's a little bit messed up. So, yeah, like, you can already see, like, the

16:27 the the BuildKit UI, like, that's a little bit different than than what you had before. Something is wrong because we copied to that root in here. We actually we got a a comment about that just a second ago from Chris in the chat. Good day, Chris. And still not right. What what did we do wrong? You don't need yeah. That seems up here. There you go. Okay. Yeah. And and yeah. So this should give you me, like, a quite smaller image, and you already see, like, the BuildKit UI in here. Let's let's cancel this one, and let's look

17:27 look at BuildKit right away. Otherwise, we're going to, like, switch around a little bit. So what I can do in here, it's alright that there's a Docker BuildKit command, hopefully, in your in your machine. Yes. And that it is. See what version this is. Okay. This is, a little bit older version, but I I guess I don't care. Or or or do we want to update this? I can do it. I I did tell it to download update earlier today, but I guess it doesn't restart it. So what I do You want me to Actually, like, I'm not

17:40 Exploring `docker buildx` Commands (`inspect`, `ls`)

17:59 sure if that's actually even even the like, if you want to update, though, we can we can get it just from GitHub, but but I I don't know it. Like, we probably won't need anything anything from the new version. Okay. I'm up to you. I don't know. Well, let let's see. Let's just continue. And then if we had a wall with something because there's a a feature that we're missing, we can talk about it, and then we'll do the upgrade. The only only thing that I can think of is that maybe we if you get more error

18:25 messages then, like, the newer version has my error messages. But but so what I want to show is in here as well that there's lots of command in here. Right? So there's a build command. Yeah. When I'm highlighting, of course, in my screen, it doesn't show. But but, yeah, there's there's a build command that's pretty much the same command that that you had there that was the old Docker build command, but there's bunch of other stuff in there. Some people were asking about bake. Like, you can see bake is in here. There's there are options to create new builders

19:03 in here that we would we'll probably get to if we get to the multiplatform things and and to manage those builders, manage build cache with with the disk usage and and things like that. Maybe first thing we can do is just look at the Docker build x inspect. That would just, like, show you what what's the current, like, the builder that BuildKit is or, like, the BuildKit is configured to use. So you can see that it's using the Docker drivers. So, basically, it's just using Docker like a regular, like, or a Docker daemon at the moment to do the builds.

19:41 We have the platform support because, like, those those come with Docker desktop. So, potentially, we can, like, target all those all those platforms. Then we can basically we can do the same command that you did before. What what did you name it? I just called it local. Yeah. A terrible name, but I'll do. So what's what's different between my build command and this BuildKit build? So what's what's different in here is is is is not very much because, like, by default, like, build BuildKit will try to give you, like, the exact same experience as as you had before. So it was

20:26 still, like, building with Docker daemon so that everything you build will be in your Docker images and things like that. But what you get is these extra commands. So, like, once we once we develop, then then there's there's going to be a point where, like, we have an option in here or we have a we have a command that's only in BuildKit, and you and you can't use the same one in the in the in the old command. So, basically, just like, if you if you stick stick to this flow, then then you will go go much further.

20:59 Okay. You got a from Crazy Max. I don't know if you know anyone called Crazy Max, but there you go. Yes. Indeed. Know. And just to look at those commands. One more extra comment about this, like, BuildKit versus or just Docker build is that if you would run, like, this command, Docker build install, then it will basically just, like, switch out your Docker build command. And and it like, you can you can just try Docker build, and it will come come come to BuildX. I don't know if you're if you're ready to do this or not, but we we

21:40 can we can skip this for now. Like, you can you can make your decision later. Yeah. I mean, is the is the goal for BuildKit to replace build eventually? Sure. Yeah. Sure. Okay. It's it's I think it makes makes sense for the user standpoint. Right? Like, sometimes you need to sometimes you're using you're building with Docker. Sometimes BuildKit also supports, like, building, like, in Kubernetes or inside a container and things like that. Makes sense for the maintenance wise as well. Like like, we we can develop build features without, like, waiting for, like, a whole Docker

22:17 engine release and stuff like that. Okay. Got it. So but but yeah. Like, your image was built. Like, if we're yeah. I guess that's that's your command. Right? Yes. That is indeed the blocks command there. There you go. Perfect. So how much time did it take? Do do remember? Like, almost almost a minute or something like that or, like, even maybe maybe more. So It took fifty two seconds. It's it's yeah. Let's try to make it faster, probably. One thing you can don't know if you noticed if you're if you're on this command or, like, maybe we could just make make

22:21 Comparing Build Speeds (Old vs New Dockerfile)

23:12 a simple change in here and just run it again is that it installed your your dependencies in here. Right? Yes. Right. Because you don't have a vendor directory. Correct. So yeah. Like, I'm I'm in the vendor camp, but, like well, for sure, like, it it doesn't make sense, for example, that when you when you were building your your process in in a host, right, then you get your audio caches and stuff like that. Like, if you're if you're on, for example, make a code change like I did now and rebuild it, it doesn't take you a minute anymore. Right? It only

23:26 Optimizing Dependency Caching (`go mod download`)

23:59 takes you a minute on the first time. So I think we're gonna first figure out the the dependencies thing just to make this, like, just to clarify a little bit better. Right. You mean, like, we could copy on the GoMod and the GoSum first, run a GoMod download, and then bring in the rest of the source? Exactly. That's what that's what I was what I was thinking exactly. So we can just put the go mod in here and then run, like, whatever it is. Go mod download there. Yeah. Go mod download, I believe, should be.

24:36 So so benefit of this one, right, is that we only download the dependency on the first time. As long as you don't modify your GoMod file, then this step will always be cached, and it will and next time you make a Go change, only only this one will run. It will already have dependencies. It will not start to pull them again. Right. Okay. So let's let's build that and get the cache in place, and then we should be able to confirm that that worked. Sure. Do I need to start this? Or Yeah. I wasn't sure if we were still

25:25 modifying the Docker file or not there. Okay. Yeah. Alright. So let's do the download. So I guess in the meanwhile, can we already start for the next steps? Or it's like a Yeah. I have a a question gonna be able yeah. Go ahead. I have a question that came in on on Twitter from Kieran. Kieran asked, can I please ask Thomas about how cache from and to are best used in a CI environment where different branches are all being built? With the so I think the question there is like, are the different bells from the different branches

26:08 all using the same cash tag or should you do something else? Yeah. So yeah. I'm not sure, like, if other people will follow this question right away. It's very specific. But but yeah. Like, my my recommendation at the moment is to use the different test tags for for for, like, give different tags for for different branches so that they won't override each other. If you get to that platform, like, there there's, like, many different ways how you can how you can actually, like, include the remote cache in our CI. Like, in some cases, for example, if you're using,

26:48 like, GitHub actions, I don't think, like, the image cache is, you know, like, the best option. Like, I I think, like, the there's, like, a local caches, and and we have some new GitHub caches there as well. Like, that might might be even, like, a better option there and might, like, solve this problem more elegantly. For example, like, the GitHub caching, GitHub already provides you a scoping for different branches, so you don't even need to worry about this aspect at all. So, yeah, like, this this one is now, like yeah. It's it's fully cashed, but, like,

27:00 Build caching with mounts

27:21 I can even make a change in here. Right? In the in the Docker file. And if I build it again, then the the half of the build was was cached. Right? So only, like, the binary is is needed. Let's cancel this. Yeah. But the but, of course, like, the binary is still, like binary stuff, they it takes too long time, right, compared to your local one because, like, the Go added some kind of, like, cool caching capability in the Google compiler. Right? Like, when when you're running the Google compiler, it will put some files in the in

27:57 the in some directory, in your cache directory, and that's why your next builds are faster. So we can actually take leverage of this one in Dockerfile as well. So for that one so this is not the multistage anymore. So this is only in build BuildKit now, but it's, like, every version of BuildKit. So run now allows some more track and Oh, I guess Thomas dropped. Oh, you're back. Did I drop out there? Yeah. I'm not sure what happened there. You just got it it disappeared. Yeah. My my my whole, like, tab crashed for some reason. But

28:07 Optimizing Compiler Cache (`--mount=type=cache`)

28:45 yeah. Alright. Okay. So So in BuildKit, you're saying we can use the dash dash mount tag, and then that's kinda where you dropped off. Yeah. Okay. So may maybe it's actually makes sense to look at the documentation first. Yeah. So so that so that we see something that we point to a particular link. Should have a link now. Yep. Got it. So this one for this document list, like, some, like, additional things we've added to. Like, there's there's some cool things in here, like how you can do how you can do mounts, how you can

29:40 do, like, secrets when you're cloning, like, private repositories, how to, like, forward your Recent version, we added here docs. So it's just like a document where where you will find, like, all all those described that are, like, the new Dockerfile additions. And one thing in here is that basically allows you to create mount points that are available when your when your process runs. So for example, like a point, you can use just to, like, point, basically, your build context to your to your process instead of making, like, a copy of it. Like, the process can just directly see the

30:20 files. In this case, like, a interesting one is the cache one. So this one just makes, like you can think of it as a as a volume, basically. It's it gives you a directory. You can write into that directory. The if you run the build again with the same cache mount, the the files from a previous build will basically be there. Uh-huh. So so you can use this as, a persistent anonymous storage where you can, like that you can use for for storing stuff for the for the cache purposes. Of course, like like, this thing

30:57 is, like, excluded from your instruction cash, so you need to be, like, careful about it just to put things in there that your process can actually handle. Like, basically, like, you can put you can put anything in there as long as your process still works with basically any content Because, like, like, cache cache volumes can can be deleted in arbitrary times and, like, at some other like, let's say you're you have another build and it writes the same cache volume, like like so you can only use it for certain things, but you can use it for the

31:26 things, for example, when when your compiler provides a cache directory, then you can just, like, use use this as a location. And then your compiler will write there. You're on the builder next time, and it will get the files there as well. Okay. That makes a lot of sense. And I can see there's there's a whole bunch of there's also a a temp files system. There's the the secret stores. I got it. Like, for for the secret one, for example, you can mount in secret files. So we make sure that they're, like, never stored in your image and and stuff

31:56 like that. And, like, on on as well. And and you can the other one is you can mount the SSH sockets in there. It's basically to to, like, forward your forward your SSH agents. Awesome. So I had to correlate that back to kind of what we're doing then with this this go build here is that we're doing a copy of the GoMod to go some the GoMod download. That's that's just basic image caching. Right? That's not using any mounts or local caching thingy. Yes. Now is is that okay there? Or is that something when you would probably just

32:35 encourage, you know, to remove these two lanes and then use a main type cache and pull in the the gold cache? Or does it not matter? Well, are you saying that, like, if we have a cache directory as well, then it doesn't really, like, give us anything to to pull the dependencies before? But it actually it still it still does because, actually, like, dependencies are not in in the in the same location. So what we can actually do is we can do we can put another cache mount in here. Think it's co p k g. Right?

33:12 So so for example, like, next time this command runs, then then it will pull the dependencies, but it will pull it on the cache mount. Yes. So do we even need to worry about the copy of the go mod and go to some anymore? Like, is that actually bringing us any value, is that just, like, the older way of doing things? I I guess well, no. It's it's still like in certain cases, it definitely brings us value because, like like, the difference in here is that if your go mode and go some have not changed, then

33:51 the builder will just keep over this command completely. Like, otherwise, it will it will still run this command and the goal itself will detect that. It will need to compute some hashes of the files, and then eventually, it will figure out that, hey. I already have those dependencies and stuff like that. Okay. Got it. So so, like, it will be it will be faster, but it won't won't be zero. Right? So so it it still makes sense to have them in in two. This case, though, like, I guess, theoretically, like, these lines in here don't buy us anything because, like,

34:25 this go build will also write everything in the in the code because g one is doing this first slow build. Right? Yeah. So so, yeah, like, I'm I'm not sure, like, if you want to keep this or not. I think it It's It's I like it. Questionable. Yeah. One thing it might come back is when we if we get to the multiplatform things, then then for example, let's say if we ever turn this into start to, like, cross compile or anything like this to multiple platforms, then this step in here that's pulling dependencies, that is the same for for all the

35:04 platforms. Like, if you're if you're if you're if you're, for example, building for three platforms, this command theoretically can run once while the compiler needs to run three times. So it might be like it might become useful there. Yeah. Good catch. Okay. So let's let's validate what you've just changed here. So we're now mounting in our our cache here for the dependencies and the actual Google build cache. So what what you're saying here is if we run this, we get a 52 build and then run it again as even the Google build step should be near

35:36 instant. Right? Yeah. Or at least fast. It will it be instant. It should be, like, pretty much the same that you get on host when you're when you're when you have all your, like, specific Go setup setup in in your host, like, everything is set up correctly, and you're and you're, like, have those persistent folders in those. One thing here, we can do an extra thing. I'm not sure. Like, this copy probably was super fast in here. So, again, it yeah. I can see that it took, like, zero point two seconds. So, like, I guess

36:00 Using `--mount=type=bind` for Source Code

36:08 it doesn't quite much make sense to optimize it. But, you saw, there was a pipe mount pipe bind there as well. Yeah. So, basically, we could also do just, like, mounting here, bind is the default to type. So if do this, then you don't actually need you may need a copy. So then it would just mount our contact into this into this working directory. And, yeah, like, we will we will we will not need to duplicate the data. We'll like, by default, it's it's read only as well that might be issue. In some cases, I

36:47 don't think it's issuing here. But, basically, that allows us to just, like, directly access those files. So does that mean on on the whole test. So I may sense early here, but I'm gonna go on some of my things that I think I know about Docker. Right? So when we do the the build, we pass in the build context, which is a dot. And as far as I believe in the I remember, this is literally a tars up of the entire context to the BuildKit daemon. Can that actually be a meted now if we're using the type mount targets, or is

36:56 Build Context Upload Optimizations

37:18 that relative still to the build context? So it's still relative to build context. But but so what you said was, like, half correct. So BuildKit does to, like, lots of optimizations. For example, like like, if we if we just, like, had a copy go mode in here and we didn't copy anything in our build, then BuildKit is smart enough. It doesn't it doesn't, like, tar everything up into giant turbo like the old builder did. It it actually, like it pulls the files when you need them. So the the builder side will tell you that, hey. I only

37:57 need to go mod, and it will basically adjust, like, fetch to go mod. Uh-huh. Nice. So so so and and the same thing works with the updates as well. Like, basically, like, when you're doing the second build, the reason why, like, some of this this is fast on the second time already is that we can already BuildKit can already see that that you uploaded the context one time. And the next time, it just, like, does, like, a com comparison and see what files are are changed and only copies the files that are changed. But it isn't directly, like, looking at your client

38:29 desktop. Like, we need to make a snapshot of the file. So so so, like, when it's asking that, for example, like, give me go mod, then we actually make a publication on the daemon side of of the file. Because, like, an otherwise, like, if we like, patching things would happen if you would allow this file to be mutable. So we need we need to make a snapshot of it. That that's the that's the only caveat. Like, otherwise, it's it's quite smart. Yeah. So what what did we get? What's what's the time of this one? 30 nine seconds

38:56 for the build. So so it's thirty nine for the download and thirty nine for the or thirty thirty Thirty point eight for the download and thirty nine point four for the build. Yeah. Okay. So so the next one, like, if you would just want to build it again, it should be, like, instantaneous. Right? Because, like, nothing has changed. You can try this one for a second. Alright. Let's do it. So we can just push up and run it. Yeah. Alright. Nice. It was was pretty fast. And all the all those like, it just needed to, like, validate that

39:31 some tags are still valid. Right? That your, like, Alpine has not updated and and things like that. Your build context was uploaded in a hundred milliseconds, but all the all the steps were cached. Right? So now if you make some kind of a change Yeah. For example, yeah, I have to adjust it. Whatever. Okay. So let's let's run this locally first and see how long it takes, and then we'll do the the container build. Sure. Sure. And time. That that running the local and then the container was just like a question that we had from Russell. He wanted to see if

40:13 we could compare the speeds. I expect them to be pretty close. Maybe not as close as I Well, okay. In here, of course, it depends on what, like, what your, like like like, your state was before. And and also, like like, it depends on things like like how many CPU cores you have enabled in Docker for Mac and stuff like that as well. But then that Yeah. There there Like, I I don't I don't know if if I I think by default, we don't enable, like like, your own machine or something like that. But but the other guy like that. Yeah. This

40:51 I can't remember what my configuration is. It it should be it should be similar. So that was fast. Almost as fast as the host. Yeah. So yeah. You're right. That's awesome. And then I I think it's yeah. Like like yeah. Yeah. It it it it's it's it's similar. It's, like yeah. It is like, I I would have hoped it would be, like, a little bit closer. Like, I don't know if it's, the CPU settings or or something like that or or, like like, it's yeah. It I like the default settings. Maybe it's an Apple magic a little bit,

41:28 but, yeah, like it. Yeah. I should get an m one, but I'm sure that will help. But Yeah. Yeah. For sure. For sure. Yeah. We went from seventy four seconds then to two point six, and then we were only five point seven seconds here. But you're right that, like, my Docker for Mac does not have access to all my cores. It doesn't have access to all my memory. It's constrained. So, you know, that's really close considering there are those constraints in place. So that's awesome. I really like The point the point is that yeah. Like, theoretically, there shouldn't be much overhead. Like,

41:57 if if, like like, like, at least, like, there shouldn't be overhead from, like, from the standpoint of, like, that that doing getting container is is less efficient because it can't access some files or or or or something like that that the host can access. So so the process itself should be should be very similar except, of course, like, everyone gets this like, you just need Docker to get the same flow all the time and you don't need any, like, whole setup or anything like that. Alright. Awesome. We have a question from Noel in the chat. He says, with BuildKit, it

42:32 Q&A: Dockerignore with BuildKit

42:34 doesn't copy over the whole context. Only thing it needs only thing that needs to be copied. Does this mean that the Docker ignore is not needed when using BuildKit? Pretty much yet. Like like unless you're doing, like, copy dot, of course, like, that you shouldn't be doing. But, like, if you're if you're just copying, like, the specific files that you that you need in Dockerfile anyway, then it already works out like a reverse Docker ignore. And, also, like like, you should never use Docker ignore for filtering your files anyway. Like, you should Docker ignore is same as ignore. Like, you you

43:00 Local Outputs

43:13 just put files there that are, like, that that are, like, true or false that you that you never want to be as part of your project. Like, if you want to, like, actually filter, like, some files from your project, then you use the copy rules for that. Alright. Awesome. So there was another feature that we kinda spoke about briefly. I mentioned briefly at the start, which was the ability to get a binary out of this so that I can you access it and run it locally on my machine. Could we maybe walk through the process

43:31 Exporting Binaries Locally (`--output=type=local`)

43:42 of doing that? Sure. And it's it's very simple. I again, like, if you maybe let let's look at the docs first. Yeah. Like, should go to the BuildKit docs. So to be the BuildKit tab, just scroll down a little bit for the documentation. You can just, like, write it from the read me as well. Yeah. So go to the build command in here. BuildKit build, and it should be the outputs. So that's just output. Yeah. That show. So so this is just click on the link for the for the flag. Yeah. So so the difference in in

44:26 as like a the the conceptual difference between the BuildKit builder and BuildKit is that the BuildKit are always built to Docker images. You always run when you run Docker build, it will give you an image back. BuildKit does not do that. BuildKit just builds it, and it's left to in your build cache. Like, the user basically configures what to do they want to do with the build result afterwards. So it might be that the build result is is just testing the build cache, might be the it's turning to an image, might be that maybe you want to get a

44:58 binary back, wanna get a back. There's, like, different options in here. So what you wanted is local binary, so it's the it's the local option in here. And but, yeah, you can see that there's other ones that probably can get the and and and things like that. Yeah. In your case, you you just want the local one. So, basically, you just want to do, like, dash show and the directory where you want to write it. Like like, I'll take the first example, basically. Or, like, you probably want to write it to a Bing directory or something like that.

45:30 Okay. So So Yeah. I'll do it from the Write write it, but don't present there yet because I'll I'll explain it why it's like a little bit wrong. Okay. Because I I think what a lot of people do over the years, and I see this in Stack Overflow a lot, people asking questions is that they use Docker to build their go application, their rest application. And then normally they run a container and copy the binary back onto those. This this removes that whole little hoop jump. Right? So Abs absolutely. Like like the if you go

46:01 back to the to the, like, the first Docker file you had where you copied the the binary. Right? Mhmm. Then, like, in a very old days where there was no multistage builds, then, like, how I built all my all my stuff, for example, was that I I needed to have two photographers. One was like this real one, and the other one was one that copied the binary, and then I needed to do, like actually, you couldn't copy out from an image. You need to create a container from an image and then copy the binary out from the

46:31 container, and then you could use it in a second build. So, like like, that that's all gone with multisage builds, that's and and in this case, where you want to get just the builder in your host because you want to run it, like, that will that's also, like, that that's fixed by the the show. So you don't need blocks in here. It's just like I guess you you want, like, local being direct for the first. And but the the problem with this one now would be that it will copy out this whole stage, so it will

46:56 Exporting Specific Binaries with a Scratch Stage

47:02 copy out Alpine as well. Can we can we copy a single binary, or is it just Yeah. Yeah. Of course, you can. So what you probably want to do in here is that you want to go from scratch. Scratch. So scratch is, a new object in Dockerfile. Right? Mhmm. So it like, zero layers, nothing. It's just just new. We can do we can do the we can just put the binary on on the scratch sheet. We can even, like or maybe I could better to do this is to paint this this stage binary, and then

47:41 if if we just do it by default, we want to build Alpine or, like, image based on Alpine, so we have, like, shell and stuff like that as well, then we can do from binary in here and in here. But so so, like, we have a stage that is from scratch that only contains the binary. And then at the last stage, we we copy all all the binaries that's copied from from bin. So the benefit of maybe this pattern is that, let's say, you have a second binary as well, like, in in well, whatever. Like, you're you're building three binaries

48:17 or something like that. So you can put them all in here, and then we don't need to copy paste all the copy steps in here for, like, all the three copy steps for every binary. You can just copy, like, all the binaries targets. Okay. Does that mean for our build x command with the output flag that we've got pending, do we have to target the the binary? Yes. Okay. I mean, it's just that it depends on what we choose as default target. Right? Of course. Like, if if you if I would, like, put the binary stage as a last stage

48:50 or or, like, just put, like, from binary in here, then you wouldn't need to do that. It just depends on you as a project developer, like, what do you want the, like, the default Docker build command to do. Oh, yeah. I think I dropped this and then make target anyway. Right? So but, really, I just wanna be able to build that layer with the scratch and the bin and then pull that down to here. So does that command look okay to you? Yeah. Sure. Press. Said with such confidence. Let's see. Alright. Okay. We were a little bit past it

49:22 now. Like, 3.8. So so now so and that yeah. Now you have the you have the binary. Right? I built it for Linux, and now I'm trying to run it on a Mac. Yeah. There we go. Exactly. We'll we'll get to that as well. Right? Okay. I like that. But so so but yeah. Like, this gave you maybe the final error rate so that you can add so one thing is, of course, like, if you want to, like, copy those binaries around, probably don't want it to be dynamic. Yeah. You find it on a static binary.

49:57 Ensuring Static Binaries (`CGO_ENABLED=0`)

50:09 The the actually, like, the recommendation I have for, like, all all time to build go is to just, like not not to always dis disable seago, but but to always define what do you expect the seago value to be. Because, like especially if you get to, like, building multi platform targets, then Go has this weird logic that, like, your native target is using seagull by default, and your other targets are not or other platforms are not using Sego by default. So you get, like, a completely different, like, dependencies depending on what platform you're building. So I would

50:47 just, like, always recommend you to just think about if your project is using Sego or not. I I'm not sure if you're project using using Seagull. No. I think it should be okay without Seagull. Yeah. Okay. Then then then this is the sim simple thing to do. That that that should take care of this one as well. Okay. So shall we move on to will we try and do multi platform? Sure. Will I be able to build a Mac version and BuildKit? We'll see. We'll see. Alright. If you don't run out of time, then then yes.

51:08 Introduction to Multi-Platform Builds

51:25 So oh, yeah. That because the SQL enabled is invalidated or build cache. So But yeah. That. I'm not sure even sure, like, what what does it do. Like, if it just invalidates all build cache or it it's just, like, because the positive packages has changed. But, yeah, like, now. It's It's static. Yeah. Now now you can copy it to anyone. So if we think about multi platform now, then, actually, it's like by default, it's it's very easy. So you have this command in here. Right? Mhmm. And I'm not making any changes to the Docker file at all.

51:59 Building Multiple Platforms with the `--platform` Flag (Issue Explained)

52:06 Let me just look at help for a second. So there's a platforms flag in here. Right? Yep. And and you can just put, like, the the basically, the value of what platform you you want. The value is the format of the value is, like, OS arch and, like, optionally, a variant thing here. Okay. And and the and the values are are easy if you come from Go. Otherwise, like, every every component in the world defines their own OS and arch values. So, like, you just need to look look up what the what the values are.

52:51 And theoretically, you want what you just to do is just put, like, platform in here, and let's say you will need around 64. And, basically, you're done. And this actually worked here and without BuildKit. And and but with BuildKit, what you can also do is okay. My my screen share doesn't handle this anymore because it went over some column limit. So, basically, you can do, like, yeah, like, you can share yeah. AMD sixty four in there as well. And, like, how do we want to do this? Can we do Darwin AMD 64? Not yet because, like, remember

53:38 when you when you we looked at the Docker BuildKit inspect? Oh, yeah. It had the the platforms, isn't it? It it had it had, like, bunch of platforms. Doesn't have doesn't have Darwin in here because, like like so these are the, like, the emulation support that comes with or, like, that that comes with the Docker desktop. So we have there's a machine, of course, supports one native architecture. But for other the binaries, we can just run them through an emulator, but with a performance penalty. So what maybe what we can do in here is just try to run this. You will get

54:16 an error, and then I will explain the error. Exactly. So so it would have worked if you just specified one platform. But but because you specified multiple, then we can't use the Docker driver anymore because, like, the the whole like, the Docker image store only handles images for a local platform. It doesn't handle multiple images, like, locally. So what you basically need to do is run run BuildKit so that BuildKit runs in a different back end and and either, like, in Kubernetes or Docker container. So, again, like, if you want to maybe look at the

55:00 Buildx Drivers (Container Driver)

55:02 docs first or or yeah. Like, basically, you just do Docker BuildKit's great to use. So you can you can look at the go go one step back and look at the create command. Alright. So back from the build build command. Oh, yeah. That's right. Yeah. So, yeah, I can grab the driver, for example. Like, this one explains a little bit about the like, what what okay. So I'll explain it here. Oh oh, well, I did may maybe, like, a little bit down. So, basically, it's a it's a three types of drivers that we support at the moment. There

55:49 was a Docker container, Kubernetes, and and the default that's just using your Docker daemon. So for the things like multi platform images and and actually for, like, some types of remote caching, like local remote cache and and things like that, you want to use the back end that's either container or or Docker one. So but, otherwise, it's like you can just run the command, and I will I will, like, show you what what it what it did in in the back end. So you can that that's it. The so you can use the Docker build XLS now.

56:19 Creating a Buildx Builder Instance (`docker buildx create`)

56:25 You don't have anything there yet. So now you will see that there's a there's a new new builder in here. It's in status inactive. The the previously, we're using the default builder with, like, a Docker driver. Now we're using the Docker container driver. So what you can do is Docker build, like, inspect and just put that's just bootstrap in there. That's that's you don't actually need to run this command to build stuff. You could just run build at the moment, but let me let's just show you what it what it how it comes up. So we would that's

56:55 just bootstrap. Yeah. So now we're just, like, telling to build that. Just just put it so don't build anything. Just just put itself. Okay. Whenever we use the Docker container driver, we get a different container with BuildKit for the architecture that you want to build for. Not not for the architecture you want to build in. Just just one container that but that that build that container now contains full BuildKit, so it has the full capabilities, for example, for the multi platform stuff. So this this one can can manage the multi platform images and things like that. So, yeah,

57:31 now you can see that it's running. You can run the LS again, for example. You can see that it still supports your platforms. Right? And and if you do that container LS or Docker PS, then you should see the the the container as well. Like, yeah, like, this is where the actual, like, the container is running. BuildKit is inside that inside that container. And and now when you run new builds, this is actually where your builds will happen. Okay. So now we can We can run run that on the same build build again with the two platforms.

57:59 Multi-Platform Build with Container Driver (QEMU)

58:10 Okay. So, you know, you're Yeah. You do a lot of Go development. Right? So, like, would you lean on BuildKit and multi platform, or would you lean on Go's kind of cross platform build? Like, is there a difference there? Is there a preference? What what do you mean? Do you ask for, like, a with with a with a lean on, like, the course cross compiling? Yeah. Would you use go build with cross compilation to different architectures, or do know do you think the build can Abs absolutely. Like, all our project use the cross compilation, and and

58:48 I would show you, like, how how to do this as well. But I just want to, like, show, like this is, the this is, like, the first way Okay. That basically to do this. Like like, remember, like, we didn't make any changes to a Docker file. Right? Like, the like, the Dockerfile was just, like, built for a single platform. And and we only thing we basically say is, like, yeah, like, okay. Like, now give me give me an image for a different platform or give me a multi platform image, and it does work out of the box.

59:21 What so at the moment, you can see, for example, that, yeah, it's it's running the, like, it's running the go build command at the moment, and you can see that it's actually running two times. Right? Yep. So one one one is running natively like you did before. You don't get previous cache at the moment for this build because you're in a container now. So you will get the co cache again for the for the next one because you because you switched the back end, then then you lost that cache. Yeah. That makes sense. So so this is

59:51 so this variant is super simple. Right? No no changes. Obviously, like, one of them is running in an emulator, so it has a performance penalty. And I was saying ARM 64 one's gonna take a little bit longer than the AMD sixty four. Yes. Yes. Like, I would maybe expecting here may maybe, like, three, four times longer. Alright. So so so especially, like, it depends on what your Docker file is doing. So if you're, like, installing packages, like, for example, you are from Alpine installing packages, absolutely no problem to run this in emulator. It will take, like, hundred milliseconds natively, and

1:00:35 it will take, like, hundred and fifty in emulator. Like, if there's no there's no CPU intensive things in there, Like, it's absolutely fine to run this in an emulator. Like, if you're doing compilation of of, like, big code bases, then you will really see, like, where this emulator thing will will start to, like, hurt you. And and but we'll see, like, given that it didn't require any any changes in the Docker file, like, you will not build this, like, every time you make a code change. Right? Every time you make a code change, you will you

1:01:11 will build that previous one that that that we had, like, running back in three seconds or something like that. Mhmm. Like, if this runs, like, one time in your CI, like, maybe it's even fine. Like, I I I would say, like, like, we don't use this one. I would I will next show you, like, what we use. But I think most people who use BuildKit today use this one because, like like, the the like, multiple platform build, the transient CI, yeah, it takes, like, three times longer. Like like, we like, the the it it doesn't slow them down

1:01:44 down as well. Like like, the those builds don't happen that often. Okay. I guess we've got about another minute before this one. I hope this finishes. Look. May maybe if you go to back to the browser, I can talk a little bit about so you got this for free because you're using Docker Desktop. Right? Yes. So so what that means is that when in your Docker Desktop VM, you already have the the emulators installed in the kernel, basically, of the of the of the VM. So every time we we make that we make a builder, it's just like you

1:02:01 QEMU Emulation Performance & Installation

1:02:26 can see that they support six platforms. Why it supports six platforms is because they're already like, the emulators are loading in in a kernel. If you're not in the in the same system, then you actually need to to one more step. I'm copying to the link. You need to actually load the emulators. So if you go to that link, you can you can actually see, like, how you would how you would set up those emulators if you don't have them in the system. And it's very simple. It's just a Docker image. So if you scroll down a bit,

1:03:08 like like, basically, you just run the run the install command in there, and you can just say that's just install all. And then and then it will it will load all those emulators from this image to your kernel, and you have the same experience every time. So for example, like, in the CI system or something like this, may maybe you want to run this beforehand to make sure that you're that that you have have those on. And then it will show you what what's all supported for your biosystem and what what the model does she have installed.

1:03:43 Okay. So, basically, she just uses QMU then, and you can just install your own platforms if required into the BuildKit container using this tool. And and, doc, this is the QMU. So so this is, like, the the QMU needs to be in kernel, then BuildKit can use it. Right. Okay. So so, basically, like, once you install use this command, it will install the emulate. It will enable QMU basically in your kernel. And then when you do, like, build x inspector, build x l s, you will see that BuildKit finds all all those emulators for you.

1:04:00 Bake with HCL

1:04:19 Right. Right. Okay. That makes sense. Nice. Okay. Did our build figure it out or not? It's still not. It's still thinking. But why don't we I mean, there were a few sub commands here. Right? And we've not covered all of them. Maybe we could what what does the bake subcommand do? So so bake well, bake task is basically, like, for example, like, when you the command you're running at the moment. Right? Like, the the binary like, you you're running a command that's doing a build. It sets a target to binary. It sets the output to local.

1:04:38 Docker Buildx Bake (Declarative Builds)

1:04:59 It sets the platforms to do platforms. Right? And you need to you need to remember it somewhere. Like, you need to, like, put it in a make file or something like that. So what bake is that it allows you to create this high level targets that already, like, contain your your specific configuration. Like, for for example, like, you can you can say that binary target for my project means that I'm I'm building a binary target. I'm putting in a test and pip inform format. I'm building it for two platforms. You can write this all in a big

1:05:33 file, and then you can can run Docker build ex big binary, and it will be all taken care of. It's like it feels like a little bit like Make. Right? Yeah. Like, that's where the where the name comes from. Right? But it can do, like, things that Make can't do. Right? Because, like like, in Make, you would you would need to if you're building, like, front multiple targets, for example, like, make and figure out that those targets can actually build in parallel and stuff like that that that BuildKit can figure out. If you just go back to the bin format project,

1:06:06 I believe bin format project uses uses bake Yeah. There was a And and bake like, this is HCL. Like, it doesn't need to be HCL. Like, it can be just a composed file or, like or even, like, the language is a little bit, like like, not not not that important in here. But, yeah, you can see, like, we we have some, like, nice yeah. Exactly. Like, the the binaries thing is exactly like like like explained here in here. So so in in this project, you could basically just do big binaries, and it will figure out what what to

1:06:44 do. And we have some, like, weird targets in here. Like like, some on my projects have, like, the linters and, like, for example, you have the tox generation and the, like, the vendor generation. Like, they can all be, like, defined as as, like, a big targets. Then in the end, you can have, like, a group that just say, like, pre commit or something like that. And that that just, like, runs all all of them in parallel, all the all the validation logic in in parallel. This one is a little bit complicated because, like, there's, like,

1:07:11 variants of it and and, like so there's, we have, like, a bunch of configuration options that go, like, build different versions of those emulators and stuff like that. So but that that's basically it. So finally, four hundred and fifty seconds. Yes. So take a bit of time, but we got there. To prove the point that that if you're running your peak compilation step, then maybe emulators is not the way to go. Unless you're running like that, build once a day, like, then whatever. But are they are those both cached now? Yes. Okay. So so first of all, like,

1:07:46 you can look at their bin directory now and you you have multiple binaries there. Oh, yeah. Look at that. So yeah. So you you should have the ARM 64 binary. You can just, like, test it in the file. Just show that it's there. Typing. But, yeah, we'll get there. Yeah. So r two q four. Very cool. So I we we can I could just copy that target like I did there and create a Docker dash bake.htl and then just run Docker bake target and that should still all be cached and just does the exact same command that we

1:08:11 Demo: Using a Docker Bake File (`docker-bake.hcl`)

1:08:24 ran? Absolutely. Absolutely. Alright. I just I wanna see it. Okay. You said there were multiple formats. So h e HCL is one format, and there's Docker Well, I want there's we can just do we can if you have it just like a composed file, you can just do, like, a, like, with a compose target as well. Okay. Let me replicate this. Copilot's ticket. There we go. And we want to target the binary target and our container and open as Ben. So I think that's the same defined in HCL as we have in the command line. Is that?

1:09:12 Yeah. It's pretty much it. So you you can you also do I'm not oh, the target no. Wait. That's the tag is local target. Yeah. Okay. That's the same, I think. You can also do group default. Yeah. I've seen that in in your files. Why don't I just Yeah. Yeah. Yeah. Exactly. Yeah. So And then just match this. So with this, instead of that big long Docker build x, we can do build x base. First of all, you you can do it like a big just to touch the print, for example, for for just like a

1:09:47 test. That's this will show you, like, what what is going to be built. Sorry. That's that's test? Print. Print. Print. Sorry. Yeah. What's wrong? What's the name of the file? Docker dash bit. But it's Oh, okay. Oh, no. I've it in the wrong Oh, you got it in the wrong directory, I think. There we go. So so, yeah, now now it will show you exactly what it's going to build. And, yeah, like, if you remove print, then it will just run on the on the build for you. And you can of course, like, the the command line takes the targets

1:10:29 as as arguments. So you can do, like, binary, full bar, and, like, every everything. Like, you can do, like, like, million targets together as well. Okay. So it used the build cache on our AMD sixty four. Hopefully, ARM 64 is just gonna be about thirty seconds. And it should still have cache. Like like like, GoCache should still do that. Like, you There we go. Okay. Okay. Didn't take that one. So it did Alright. Three times as long as as the as the making one. K. I like that big fail and just making all declarative and getting that. Like, yeah,

1:11:07 I could write a make target, but maybe just use it. You can do, like in the HCL syntax, you can do, like, weirdnesses, like like, define variables and, like, functions and, like, we can, like little bit, like, have, like, a macro style in there to have it, like, more configurable as well. Like like, this is, a a project level logic. It would maybe put in a big file that, like, defines, like, this is, like, a different configurations of how your project is supposed to be built and stuff like that. In in Dockerfile, maybe you have, like, more generic things like co version

1:11:38 and stuff like that that are, like, that are, like, universal for all the projects. Yeah. We got a comment from Crazy Max talking about some of the things that they're doing with big, so building boundaries, multi platform image, generate docs, lens, update, validate vendors, generate progress and test the app all in a big definition. So that's pretty cool. If you've got a link Crazy Max to one of your projects that does that, I'd love to see the HCL at some point. So please share. And no other comments saying that they love conditionals and bake. So cool. Lots of options there.

1:12:00 Cross-compilation with native toolchains

1:12:12 Alright. We've we've covered an awful lot here. Is there anything that we've missed before we finish up? Do you think would be really cool to share with people? Mean Put you on the spot. Like, what what have we forgotten? I mean, what what we've forgotten is is the, like, the cross compilation. I'm not sure if it if you can push through it with ten minutes or or not. Like like, that that's that's what was for your question of, like, what's the alternative for QMU and what's the and what's the like, how you can get a Darwin target?

1:12:28 Cross Compilation using Dockerfiles (`FROM --platform=build`)

1:12:46 Like, we can try to answer that if you if you if you can push through it. Well, yeah, why don't we make a start on it? If we feel that we're not gonna get through it in time, we'll, you know we can we can leave it. If we're almost there, we can always run a little bit over. We we can see it. Or if you want, we can leave it for another day. I'll I'll leave the decision in your hands, Todus. Okay. Let let's try it. Okay. So That's good. What we basically, what we want to do

1:13:16 in gross computation is if you look at this article file again, Yeah. Post your link just yeah. The link I posted to you is how I do gross compilation and how, like, Docker projects do the gross compilation. So that that one is, like, a little bit advanced. We will probably not go go to this one. But this one has, like a first section is is has, like, a introduction to cross compilation. So, yeah, like like this cross compilation primary. So, basically, what the best way to do this is that we have this certain automatic arguments in Docker files that we just

1:14:01 provide from the builder. Like, you can even click on the link to automatic platform arcs in GlobalScope. Yeah. So this is the Docker documentation. So, basically, all those arguments are available to you automatically inside a Dockerfile. We just define the arc, and we will figure out what the correct value is. So, basically, if you're building for, like, an like, a PowerPC, for example, we will put target platform equals PowerPC target OS equals Linux, Arch equals PowerPC. Okay. And and then you can use this one in to to have, like, some stages be native to your current platform,

1:14:40 some stages to be individual for your target platform. Right? So if you go back then now then so the easiest way to do this in a cross compilation thing is that in this first stage, we we say that, oh, there's a there's a massive mistake in this example. It's not supposed to be from Dash's from. It's supposed to be from Dash's platform. Ah, okay. So so Her request welcome. Yeah. So so it's, like, from and you say that this this stage is always for the build platform. So you say platform equals build platform, and then this stage will always be the

1:15:25 same. Even if you if you build for ARM and and you build for AMD, it will always be AMD in your case because you're I believe you were an x 86 machine. Right? Mhmm. Yep. So so it will always be the same. It will always be the same command. And then when you define a target platform, now the compile step is different because it the target platform value is different. So the so the Alpine image was built one time and then compiles command run two run two times with a different argument to the dashes target.

1:15:58 And so now we have two binaries, and now we have the second stage, Alpine, that doesn't have any dashes platform flag. So now this this stage is is separate again for for every platform, and we're just copying the binary to that stage. Okay. Make a bit sense. Yeah. I think that makes sense. Yeah. So, basically, in your case, like, I'm I'm I switch back to code now. Okay. So you're adding the from platform to our first part. Yeah. So, basically, like, in here, you could like, if you're stupid, then you can write something like this. Like but but, like, let's

1:16:37 not do that. Right? It will it will be the same value, but but, like, this one, then you can use you can cross compile from from ARM as well. So and in this case, like like, dependencies, we can only pull one time as well. Like, we don't care about pulling them twice. Right? Mhmm. And now in here, we can just do target OS, target arch. So just to define those arguments so that they're now available. Of course, the code supports cross completion super easily. Yeah. We can just do a target OS in here. The large

1:17:19 equals target arch, And I think that's it. Okay. Let's try. Alright. So So you can you can just run the run the same pay pay command again. Yeah. Just nothing changes. That's it. Alright. Now now you can already see, like, the the go image only one time in here. Previously, it was it was multiple times. And now now you understand why the one one miscached. Right? The other the other one isn't because, like, the the your any of the 64 notes did not have the RNCache yet. Okay. Right. Okay. It it make a little bit sense.

1:17:28 Demo: Cross Compilation for Multi-Platform

1:18:12 But but now, like like, if you if you run this if you make make a code change now and you run it run this again, Okay. So Okay. So so six seconds in total. Like, that's okay. Three for each pointer. So so now you have a you're building for multiple platforms with with the same speed that you had before. Yeah. So there there there was a change there. It only happened in the Dockerfile, and I I wanna make sure I understand this correctly. Sure. Right? As previously, before we added the platform to the from here and the

1:19:01 Cross Compilation Caching Explained

1:19:04 ARC for the target OS, we were using different container images with QEM emulation to compile the things within the different containers. But Docker or BuildKit seems to be smart enough to work out that if I have a platform from with these args things here that it can do that same compilation compilation in the same container. So But it's no longer using the the second QMU and the BuildKit. Is that right? So so the so the first thing to understand in here is that the whole, like, like, the BuildKit design is like deep application. Right? So you're building, like,

1:19:42 two targets in parallel. They share some parts. Like, BuildKit, we just, like, optimize them together. Like, it's it's like it's one of the, like, the key key principles of the design. So so, like, to to be efficient, the first step is to not run the code that's useless. Right? So it will always detect those things that if if your two builds share some some components, then it will combine them together. Now in this case, for example, let's say you're building for the for the ARM target. I this stage started before there was the Dashes platform flag in here. Mhmm. Though those

1:20:17 commands would have been different. One of those commands would have been get me a colon image for AMD sixty four, and the other one would have been get me colon image for ARM sixty four. Yeah. Now because because they're all both fixed to the build platform, now they're both saying, give me a colon image for AMD sixty four. And BuildKit will instantly, like, notice that, like, hey. Like, those two builds are they're using the same image, so they're the same command. And and and then it's just like, for example, once they go more download in here,

1:20:45 it still understands that those those two commands are the same command that runs on top of same image, so they're the same. Yes. It it's like that. Step but the compilation step is not is not same. Right? Because the compilation step has those two different environment variables, so that command is not equal anymore. Okay. So I'm just gonna make this special for other people in case because it it took a little bit moment for me. But when these were a metered, what we actually had were multiple steps like this. However, when we or when you made that

1:21:21 change to say use the build platform, we actually reduced the instructions that that was sent to BuildKit, which is why it can do those jobs as it did. Okay. That makes a lot of sense now. I got it. So so, yeah, like, let's say that if you were building this Alpine target in here, like, in line 17 still, like, then then previously, we would have pulled two Golang images and two Alpine images. Mhmm. Right? With now with this change, we're pulling one Golang image only. We run the compiler always on one Golang image, and then but we still have two

1:21:56 Golang images. Right? Because in the end, like, we want to push the e v 64 image based on e v 64 Alpine and and, like, the busy box in that image will be will will come from different images. Got it. Very nice. I like that. And we did it at time. Right? Yeah. So one one thing that was different thing well, first of all, let's check your Darmin. So just just put the run the same command, but put your that's just platform, put Darmin in 64 there. Because now there's no reason for the Darwin not to work.

1:22:28 Building for Different Architectures (Darwin Example)

1:22:33 Sorry. Can you say that again? I missed it. That's in the in the Dutch platform. You can So an HTML. Yeah. In here You want to add the Darwin? Yeah. Yeah. You can even, like, remove the other ones because, like, I guess for the default, like, you just want your local one. Right? Yeah. You can just try this one. Oh, okay. Okay. So this okay. Yeah. This works because of the build platform on the frontline, which always builds the AMD sixty four or whatever my build platform is, and then relies on the Google cross compilation

1:23:17 by setting the target OS and the target arch, which is why Exactly. I got it. I I I do but you want you understand why it didn't work before. Right? Yes. Because, a first step, like, it didn't even doesn't even reach to QMU. It's like the pillar would say, like, give me a Darwin in the 64 image for Golang. And, of course, there's no such thing. Like, there's no Darwin Darwin image there. Like, that that that's why it doesn't work. Finally. Yeah. I'm I'm obviously just picking this up a little bit slow, but I got

1:23:46 it. That's awesome. So now for languages like go and rust where cross compilation is available with the compiler tool chain, you can really build some pretty nice build pipelines with bake and HCL and BuildKit. I think that works really well. Cool. Yeah. And and and and so now you can try to run this one. Like, this one, just bin blocks should should run-in our machine. Right? Well Yeah. The same command. Oh, sorry. Your It's not gone. Yeah. Yeah. It's it's this one because you only built a single platform. It it's configurable, actually, like, if you if you don't want it.

1:24:27 Yeah. It works. And the last thing Yeah. What was the like, if you remember in in my project, there wasn't already named six four in there. There was just local. So local is just like a hack thing in the in the in the in the in in format projects when you where you copy the big file. Yeah. That one. Like, I I had local in here. Ah, so anyone that wants to build the binary, they can just use that local. So local is just like a magic platform value that is, like, if I'm copying if I'm running your big file,

1:24:31 Using the "local" Platform in Bake Files

1:25:00 I'm on ARM, then it will mean tarry in ARM 64 for me. If, like, Linux person is is running big on their machine, it will it will mean whatever, like, their system is. So if you want to just local binary, then you would use this one instead of instead of hard coding in the 64. It will be exactly the same binary in the end. Okay. So I can just Local. Yeah. It's okay. I guess that's all I'll go back, and that should be pretty instant. Right? So And yeah. And if you want to get, like, further into this, then this project, this

1:25:39 Advanced Cross Compilation (XX Project)

1:25:39 XX project is, like, what I use, like, for more complicated cross compilation flows. Like, for example, you can imagine, like, at the moment, we have the SQL enabled SQL zero in your in your build. Right? Like, let's say, we you need c go enabled equals one, then it's not so simple to do cross compilation anymore. Right? So like, the c cross compilation tool chain is way harder than than than go cross compilation tool chain. And this this project basically makes it easy. So, like, any anyone interested, like, can can look at just follow the examples in here. Like,

1:26:15 there's examples in here how to how to build c and, like and once you set up c, then then then you'll automatically have c go and and things like that. What was that? And, like, how how to install packages and, like, you you can, like, basically, like like I hope, like, for for most, like, c c and co project, like, they should this basically is like a almost like a drop in. So we just, like, drop drop this into your into your build stage and and basically the same simple things that you had in Go. Like, this works

1:26:49 for the different languages as well. Awesome. Alright. I'll include a link to this. I think people should check it out. Just see some of this. So there's the the Go example there. So Yeah. Alright. Cool. So this is what we actually use in all our projects. We just, like, drop this into our Docker file, and and then we have those those million exported targets. Like, if you look at, like like, BuildKit itself, like, it's all built from Dockerfile. It's built into, like, I don't know, 10 platforms or something like that, and it's and it's all through this.

1:27:29 Nice. Very, very cool. Alright. Anything else? Are you happy with that? Like Sure. Good question. Pig supports Compose. It like, yes. It does support support Compose. Okay. Yeah. So question, it would because of Pig could support Compose spec. YAML, is that different from Docker Compose? No. It's just like like you can you can use like, if you have a composed file, you can you can, of course, build it with the compose itself as well. You can also just directly build it with BuildKit. Can just do basically that. If you do BuildKit, you have a Compose file in the

1:27:37 Q&A and Conclusion

1:28:10 directory. It will just build all your Compose services with bake in parallel and everything. All all the goodness with that one. Or you can, like, choose a a composed service from your composed file same as your you had your binary target in here. Right. And we got a question from Noel asking, does XX use BIN format magic? So no. It so bin format is the QEM emulation. Right? Bin format is actually like the like the bin format misc is like the kernel feature that allows loading emulators. That's why it's called bin format. So bin format is the

1:28:47 emulation side if you want to if you want to add emulation, if you take the performance penalty, but what no changes in your Docker file, excess excess cross compilation. So that's that's that's for defining the if if you're if you're going through the cross compilation route, then XX makes it easy to build c projects and CMake and and all those things. Alright. Awesome. I don't think we have any more questions. I have learned an absolute metric ton today looking at BuildKit and BuildX. So thank you for walking us through this, Thomas. Really, really cool projects. I'm looking forward to

1:29:24 adopting more Docker files aren't just copying them a binary. Think we've cleaned up Thank I think we've cleaned up my the QBox project really nicely. I'm looking forward to sending that PR to Brian and getting him to take a look at it. So very, very cool. Yeah. A couple of thanks coming in the chat, I'll throw them up in the intro. Thomas, thank you so much. That was fantastic. So thank you for spending there your afternoon with us. Thank you. Alright. Have a great day. Bye all.