1:47 Introduction and Guest Introduction

1:47 Hello, and welcome back to the Rawkode Academy. I'm your host, David Flanagan, although you may know me from across the Internet as Rawkode. Today, we are taking a look at two brand new tools for building secure container images. That is apko and melange. Joining us today from ChainGuard, I'm gonna introduce us to these tools, show us how to get started, and explain to me why they're interesting because I'm not very smart. It's. Hello. How are you? Hey. Hey, everybody. No. How'd you go? Please feel free to introduce yourself. I'm Ariadne. I'm the security chair for the Alpine Linux distribution, and

2:37 I'm also a software engineer at. And, basically, I'm the author primary author of apko and melange. So, yeah, that's basically it. Awesome. Thank you very much. I mean, it's an absolute pleasure to have you here. I mean, you've kept me hanging for months, if not years, but I'm so glad that you've Yeah. Been able to come on and share some of that knowledge with us today. So for, you know, for me and for anyone else that isn't familiar or aware of what apko and melange are, can you give us, like, the the high level overview before we, you

3:11 What are apko & melange? Overview & Benefits (Scannability, SBOMs, Size)

3:19 know, jump to the screen share and try to understand what these things are? Yes. So apko is a image composition tool. It kind of acts like Docker in terms of you have a configuration file that defines a bunch of things, and then it spits out an image and publishes it somewhere. And then the key thing about apko is that instead of having, like, things like the run statement or copy statement or whatever, it only assembles images from APK packages. The reason why this is a big deal is because if you try to scan an image made

4:10 with Docker, generally speaking, the scanners can't really see all of the different things that have been put into the image. Like, it can only see, like, what's been added to the image through various package managers like APK or NPM or PyPI or whatever. So the idea behind apko is that if you have an assembly tool that only deals with packages, then you can guarantee that the scanners can scan everything that's in the image because it's all in the image, and it's all recorded in the package manager database. So that that's a pko in a nutshell.

4:57 But the other key thing about Docker is that you can use it to build things in sort of a cloud native way. But as previously mentioned, if you build things with Docker, then the scanners can't necessarily see them. So Yep. You need you need a second tool to take code and turn it into an artifact that can be processed by apko. And then when you combine these two tools together, you get something that can actually be scanned 100% by, like, Trivy or Snyke or, Encore, whatever scan tool you wanna use. It it the key thing is that they

5:49 have the visibility. So that that's basically what the tools do. I love that you said a % and then play it safe with, like, 99.9 or something like that. Like, it was just this is it's gotta make sure that trivia and all of that work well with us, which is really cool. So I'm gonna try and just make sure that I understood that correctly. Is that the combination of apko and melange mean that and my repository where I'm building my super million dollar application is that we could use these two tools to turn that code into

6:20 an APK package, which can then be turned into a container image and published. And it gets around this weird problem that I think we've all had at some point is that when we just copy random binaries and take container image, all these scanning things fall over. And does not fall over, but they have no idea what it is because they don't actually do binary introspection or anything like that. They they need to know where it came from and the metadata about versions, etcetera. These two pieces of software these two pieces of software together give us that

6:52 traceability, visibility, and so that's making sure that our container images can be scanned with these these tools, which is a really big deal. Like, this is really important step for understanding the security footprint of our applications when we are shipping them to production. Right. Exactly. And that's that's that's the point is making sure that the final stage in your image composition process is putting out something that is 100% visible to, the scan tools. And the other nice thing about this is since you have all of that visibility up front, you can actually, at that time,

7:36 build what's called an SBOM, which is a software build materials file. And if you have that up front, then you don't have to worry about whether or not a scanner is able to put it back together after the fact. So you can be confident in the results of your scan tools and the results of your SCA tools and so on. Because if you're doing everything after the fact, then you're missing things, basically. And the other nice benefit to all of this is Alpine images are already really small. But since we have these tools now, you can actually

8:24 build containers that have only the components of Alpine that you require to support your application. And so for an example, we built an NGINX Ingress controller, container for Kubernetes, just as a proof of concept, and it weighs in at 8.3 megabytes instead of the 34 megabyte NGINX Alpine image or the like, I think it's a 40 megabytes NGINX Docker images. And just kinda looking in the chat here, I noticed that somebody mentioned Chimera, and Chimera should already be fully supported by apko and melange if you want to use that as your base instead of Alpine.

9:22 Nice. So, I mean, Alpine images were already small enough and being widely adopted because of their size, and you've went, nah, not small enough. And now we're stripping even more stuff out of that. That's that's pretty wild, but I like it. So Yeah. The the the smallest proof of concept image that I made with a prototype of this tool last year, I built a simple hello world application that just contained Mucil, LibC, and hell and the hello world application, and that clocked in at around 400 kilobytes. So this tool enables extremely small images, like, much smaller than

10:17 the Alpine base image. Yeah. That's pretty cool. So, I mean, I guess what I so you're just 400 k. Right? So does that include a shell, or are we removing the shell when we talk about this? That removes the shell, and that's kind of a security feature because if you don't have a shell in there, then you know, like, say that you have, like, a WordPress container, and then somebody gets, like, r 57 shell p a not PHP installed on it. They can't then take that r 57 shell and then get that to get an actual

10:30 Image Size and Removing the Shell

11:01 shell and then root your container or whatever. You know? If there's no shell there, then it's already more secure just from not having that attack surface. Yeah. But there there are better approaches now for when you do need to be able to, you know, do debugs on production. You know, have to control debug, which can add a new container, web tooling, enter your pod. Like, there's really no reason anymore for shipping a shell inside of our application. Exactly. Awesome. Well, I'm really excited to play with this now. So I am gonna pop over to the screen share.

11:30 Preparing the Environment (Alpine Setup & Dependencies)

11:41 And all I have here is I'll pay Linux 3.15 or three fifteen. I'm not sure. First machine running on the node. I haven't done anything else. I know we've got a little bit of setup to run through, but we're we're starting from scratch, which is where you, the audience, would also be starting from. So we're gonna show you every step of the way how to to start playing with these two tools together. Cool. Alright. So I'm a first we're gonna have to update to Alpine Edge because some of the tools that we need are only available

12:20 in Alpine Edge right now. So if you pull up the repositories file and then you've replaced the v 3.15 with Edge for both of those repos, and then we'll have to add the testing repo as well. Alright? And then yep. And then, if you do APK upgrade dash capital u and then a and hit enter, it'll upgrade the distribution to Edge, which will take, like, two seconds. Well, maybe a little longer on It it's moving fast. Because there is most upgrades with this stuff. Well, that's that's one of the reasons why people enjoy using Alpine is because the

13:28 package manager is quite speedy. Yeah. Definitely. We wanna tackle a question just now. Are you happy with that? Like Yeah. I saw that Sam asked in the chat about comparison to distroless. And, actually, the apko and melange stack are intended to kinda be a spiritual successor to distroless. And don't worry. We're working on a GLIBC version of it, to support people who need GLIBC compatibility. So I'm just gonna give up trying to fix my I don't know why my comments are so huge and just taking over the whole thing. Yeah. So, basically, it's the same

13:38 Comparison to Distroless / Bazel

14:25 concept conceptually, and it's sort of meant to be a spiritual successor. Of course, Google's Distroless project uses Bazel to build all of the base images, and it's really a pain to deal with Bazel because it's just it it's Bazel is very verbose in terms of, like, how you actually use it to build the the legacy Distroless images. Like, you have to literally spell out each detail, like, go download this dot dev file with this checksum and then this one and this one and this one. And you have to actually solve all the dependencies yourself by hand. So

15:16 apko and melange, it's built on APK, and it uses the APK solver and all of that. So it makes everything a lot simpler in terms of actually, like, going from zero to having an image versus with Distroless where you're having to write, like, three to 400 lines of Bazel Starlark configuration, I believe. I believe the configuration language of Bazel is called Starlark. So you have to write, like, three to 400 lines of Starlark in order to actually, like, add new things to Distroless, and it's so we decided, you know, let's kind of rework that to

16:17 simplify it. And so that's how a p co a p k o and melange came up came about. Nice. Yeah. I think Bazel is a really phenomenal piece of software, but I just can't use it. I find it too difficult, too cumbersome. I'm not a fan of Starlark, because we're Python dialect. I don't want the JVM on my machines most of the time, so I just it's it's not a tool that I feel comfortable going through even though I appreciate what it does and why like, you know, companies like Google use it. Yeah. So it's nice that there's there are

16:54 more more alternatives to that coming up. We got PGSA is still faster. Why am I I give up the comments on this thing. Still faster than apt. Yep. We've got Nuno following along in WSL two. I'll move it one more time, but it it just stay. Right? And then we've got a question from John John Sun Junior. Any thoughts on how this was compared to Next Day? I think that's the the question. Are you familiar with Next Day? I have some passing familiarity. So if I'm right, I think Nixiri is the thing that builds images on demand with

17:38 Nick. Okay. Great. Yeah. We're considering doing something similar. There's a service called Contain.me, which builds a whole bunch of images on demand. And one thing that we're thinking about doing is contributing, like, an APKO version of that so that you can actually build APKO images on demand through Contain.me. The apko internals themselves are designed to allow one to completely programmatically drive, like, the apko image composition composition engine. So building something like that would not be that hard to do, and it is something that is on a road map for APKO. Alright. Sam says thank you as well for that

18:46 very detailed answer. Yeah. Definitely. Okay. So slight segue there, but we have our APK upgrade finished, like, a year ago. So we're we're we're good. We can move on now. What are the the next steps? Alright. So we're gonna have to install some packages. So we're gonna need Go. Obviously, the tools are built to Go. Buildbase, which is build dash base. We're gonna need git to download the source code, obviously, And then we're going to need bubble wrap, which is a lightweight container thing. And then we're gonna need fruit, which is p r o o t.

19:35 Yeah. I've not heard of Bubble Wrap. Bubble Wrap is from Red Hat. It's the container engine that's used by Flatpak. Alright. There we go. Learning lots of stuff today. That's great. Oh, we never got go1.eighteen, so we can't use genetics yet. Maybe next week. Well, the good news is that apko does not target go1.18 yet. So Yeah. I mean, that only dropped yesterday, right, I think? So Yeah. I see a lot of buzz around on Twitter. People seem to be very, very excited about being able to use generic. Yeah. It's gonna simplify quite a bit of both apko and melange

20:27 to be able to use it, I think. Pulumi too. Yeah. We we have to wrap all of our primitive types and Pulumi primitive type wrappers so that because there's a lack of genetics. So that'll be a a nice change for Pulumi in the future. Jason's saying, Dwight. I'm assuming that's an office joke. I I guess. Alright. We got we got all those packages. Alright. So, I mean, technically, we don't need fruit. Fruit's only if you want to do things rootless, but since we're keeping things simple, let's just not bother with doing things rootless today. I I I thought we're anyway, so we'll

21:12 just Yeah. So the next step is to well, which one do you wanna play with first? Because there's kind of apko is pretty useful on its own without having to deal with melange. So maybe it might be good to go with apko first and then go through melange and then show how they're combined at the end. So, I mean, based on my my understanding so far, which apko means we could right now declaratively build a container image by just point or just using some alpine packages. Right? Yeah. So if we just wanted to build an engine x container image, we could we

21:24 Demonstrating apko: Building an NGINX Image

22:02 could do that now? Yeah. Yeah. Cool. Yeah. Let's see it in action. That sounds great. Alright. So the next step then is going to be to download the apko source code, which is at, chain guard dev slash apko, chain guard dash dev. Oh, yep. Alright. And then you can go in there, and you can type in make, a p k o, and then hit enter, and it'll make it. And then it's gonna download a whole bunch of stuff here. We got a comment from the notes saying, not sure if this was intentional, but melange equals mix in French, and melanger

22:10 Download the Apko Source Code

22:57 is to mix. That that was intentional. The idea is that you can mix the artifacts into an image and then everything's scannable. So that that's why I called it that. Well, we're just waiting on some Go dependencies right now, which won't be long. It'll it'll be about thirty years because it's building Docker right now. Oh. Well, parts of Docker. Is it using BuildKit? Is it using other parts of Docker? I'm not even sure if I could name other parts of Docker now. The parts of Docker that are related to uploading images. Like, we use Go container registry to actually compose

23:54 the images and upload them, but that depends on Docker a little bit to do some of the stuff. So those are the parts of Docker that it builds. Yeah. Well, I'm glad that when I spun up the little node machine, I didn't cheap out and go for the smallest one. I did get, like, the four cores and eight gigs of RAM, I think. So that hopefully, that helps speed that up. Yeah. My my usual development environment is these days is a 32 gig RAM line node with 16 cores and Nice. Apko goes really fast on that. So it it

24:34 should go reasonably fast on this. Well, it has built, I'm assuming. Yeah. We got our apko thing here, and we have some sub sub commands. Yeah. So if we let's see. So if we look at, examples slash NGINX.YAML. Oh. I mean, I just said nginx randomly. I don't know if it's just gonna be sat there in the examples directory waiting on this. So that's serendipitous. I like that. Yeah. So this is what we used to build an NGINX based ingress controller as a proof of concept for Kubernetes, the one that I mentioned that was eight

25:32 megs earlier. Mhmm. This is it. So we can actually build that and import it into a local Docker instance if you want to set up Docker. We'll do that in a second. Now the interesting thing here is the service bundle entry point type, which is a new innovation for apko versus, like, Docker or something like that, a lot of people, they'll build, like, a container that contains, like, two or three tightly coupled services together, and they'll all be in the same container. And so what this does is it will automatically ensure that s six gets installed,

25:51 Service Bundle Entry Point Type

26:26 and it configures it. And it sets up the entry point where it just launches s six. So if you, like, build and run this image, it will run NGINX under a supervisor with s six running as PID one. And Nice. So it makes all of that a lot simpler than, like, dealing with s six overlay or something like that. It's just a nice, declarative configuration for the services that you want to supervise with s six. Okay. Does Docker not have something similar where you do dash in it and it runs everything under tiny, or is this different from that?

27:20 It does have the ability to run things under Tiny, but this is an actual, like, process supervisor, which ensures that, like, if NGINX were to crash or whatever, it would automatically restart it. So Alright. Okay. Okay. Gotcha. Plus, you could have multiple services as well with the s six approach Exactly. Which you can do with the container image. Not that people should go crazy and throw lots of different services into container images, but, you know, there are use cases here. Sure. Yeah. Okay. So I'm I'm just gonna assume that this might work. Close. I was I was making that up. But

28:08 Well, we have to go back into the other directory, or that'll work. Yeah. So it's apko build and then the config dot YAML file that you want to run. And then we can just call that, like, a p k o dash nginx colon latest or whatever you wanna call it. And then, like, a p k o nginx dot tar because that's the thing that you can import into Docker if you don't wanna, like, publish or whatever. And if you hit enter, it will build the image for the local architecture, and it's done. That's it. That was fast.

29:01 Troubleshooting Docker Load / Kernel Issues

29:01 And now if you, APK add Docker and then start the Docker service, Yeah. Service Docker start. And If you do Docker Load. It's import? Load. Right? I can never remember if it's or load. But And then you do the less than sign and then, yeah, apko nginx dot tar. The Docker daemon is not running. Why is it not running? You're looking there. I I think it was just there. Oh, no. It's check the VarLog docker log. It might be something related to c groups. Oh, we're gonna have to Okay. Go to ETC APK repositories. Open that up in

30:41 in your editor of choice. We need to bring back our five point 3.15 kernel. So if you do 3.15 and then you put a tag at the front of it, So a at the front of that line, if you put at 3.15 and then space Like that? At the front at the front of it. Alright. Okay. And then save it. APK update. Mhmm. And then APK add Linux dash vert at 3.15. And then Restart Docker? Uh-oh. Are you gonna say reboot? Yeah. We might we might have to reboot it because it looks like there was a kernel update in

31:51 since Linode built their Alpine image. So we're gonna have to reboot it, I think, in order to get the necessary modules for Docker to actually start up. Okay. Some Well, this is a a VM. 98 moments. It it may be quick. It usually Comment. We should have just added pod man. I I think that may have had the same issue, but maybe not. Let's see. Did we go oh, we got a few more questions there. So Nuno's looking for hoodie. I'll maybe run some swag giveaways soon, mate. Nuno also said crane from gold containers is

32:48 definitely one gem that not many people know about. I guess that would be as a alternative to using Docker's container registry package. Is that what Nuno's suggesting there? I'm not sure. Well, Crane Crane uses the same Docker components that apko uses because apko uses Go Container Registry underneath just like Crane does. Okay. There we go. Say I'm a bit late. Welcome. And that was fast as a good tagline for apko. Yeah. Feel free to quote me. Stick that on the website if you want. And Yeah. Fucking IP tables. It's always IP tables. Yeah. Fucking networking. It's always hard, isn't it?

33:40 We've got Devia, hey there. That's Welcome. It it's it's because I forgot that there was a security update for the kernel because of the dirty pipe thing. So Well, no. I don't remember that. He's bragging that he doesn't have to reboot with WSL too. So, you know, I'm glad you're still following along. I'm assuming that should be available now. Just assuming that the automatic reboot thing rebooted it. I wonder if we've lost our machine. Let's let's see. I can pull up the Yeah. Linode content. Sometimes Linode, it doesn't re reboot. It just, like, shuts it down.

34:36 Okay. I will reboot it from oh, is that I tried to reboot it from the console, and it told me that the the node is busy. Yeah. I think it's already trying to reboot it itself. Let's just set up some monitoring. This is how I do my monitoring for servers. I mean, that's how everybody does the monitoring servers. Yep. Just scrape that with Remetheus. Call it done. There we go. There we go. Finally, it's done rebooting. So Alright. We'll just wait for SSH to start, which will just be a moment. I'll grab the password. Yeah.

35:34 Alright. So if we Perfect. Start Docker now, it should work. Yeah. That's a happy go. Okay. Alright. Is that nothing So we're back then to the apko slash examples. We should have that a p k o a p k o nginx dot tar, and then we can load it in. And if you do docker, run apko nginx full latest, it should start the nginx thing up. Should I detach it or just keep it front and center? You can just go ahead and detach it if you want. And then if you do docker p s so and then if you do, like, p

36:36 s three or whatever, it'll show that it's all running under s six. You mean just on the host? Yeah. I thought e f would give me the tree. How would you get the tree? P s tree. Oh, really? It's just one word. Oh, how have I not seen that command before? Yeah. Is that an Alpine thing? Yeah. Alright. Oh. We're not not quite seeing what I think we wanted to see. Is that right? Or are we? Should all still be it should still be it should be running. Yeah. It looks it appears to be running.

37:45 Oh, there you go. Alright. So we do have That nginx run. Right? Particular that particular example container might be might not be correctly running anymore because it's kind of an older proof of concept, and we changed a few things. So but but as you can see, it has s six running in it, and there's NGINX inside it. And Yeah. Yeah. So Alright. We got a conference in Forest. Is that really a thing you can pass to p s? No. Probably on some of them. Yeah. Okay. Like, if you did APK add util dash Linux, it would

38:42 probably work. I'm I'm I'm not I'm not that bothered. It's okay. So let's recap that because this was interesting. We have about a dozen lines of YAML, probably less. We've got kind of two main components. First one is called content, which will allow us to specify Alpine repositories and Alpine packages that we want to get into a container image. So that's pretty nifty. We then got the ability to do an entry point. Now this is something new that you're bringing to the table here called service bundle, which will give us a supervisor inside of our container,

38:47 Recap

39:18 and we can specify the services of that we wanna start. Now Yep. Just in the name of why does it do that sometimes? And the name of completeness, I I have to really curious about whether what is fluffing my words because them is fluffing my text. What would a standard entry point look like here? Like, just Ben followed by a path, or can I can I still do this? You can do entry point and then command colon and then whatever command you want to run. So I could just do So, like I know that path isn't correct, but but that

39:38 Standard Entry Point

39:58 would that would just work as normal. Exactly. And would that also take list of args if they were required? Or would it all just be blah blah blah? It it would all be like that. Yeah. Okay. Cool. I like that. That's that's nice. But it has that that precondition that we need to have our applications bundled as APKs, which is what you're gonna show us the magic melange for now. Right? Yes. So Unless there was something else you wanted to show with apko first, and I I I don't wanna take it off. No. I mean

40:25 Introducing Melange and Recipe Structure

40:40 no. I mean, that's basically it. Apko is just the assembly tool, and so now now we get to see where we can actually, like, start building our own stuff to put into an APKO image. And so as a proof of concept, we have the GNU hello world program that we're going to run-in a container. And Okay. So I guess I have to make Milan by typing. Wow. Absolutely well. We have to build it first, and then we're are we working through one of the example directories? Yeah. So melange is a pipeline oriented builder. Like, if you're familiar with

41:47 GitHub actions or something like that, then this should all be fairly familiar to you. So if we pull up, like, a the GNU hello world examples, so that's an example slash GNU hello dot YAML. There's just two made three major components here. First is the package metadata that gets collected into the APK package metadata section. And that includes, like, you know, package name, version, what it does, and then, like, the copyright data and all of that. And then the second section here is the environment section, and that's literally just the description of an apko image

42:46 that you want to build in order to build the package. So, like, a base like, a base image? Yeah. You define your base image for the build environment. Okay. Cool. And then you have the pipeline, and that's literally just like a GitHub action style pipeline. You can see that there's a bunch of steps that use prebaked tasks, and we have inputs for those tasks like fetching. And then this is what the SHA two fifty six is supposed to be, and then we want to configure, make, and make install it, and it's autoconf based and all of that.

43:34 And then we have the sub packages section, and that lets you run steps in order to, like, split things out into sub packages. Like, if you want to have the main pages in a doc package instead of having it in the main package, that allows you to slim down your packages a little bit, which in turn allows you to slim down your images. And then each subpackage has its own pipeline and metadata and so on. So So where do these use these functions or targets come from? Those are bundled in the pipeline's directory. So if you get out of that

44:28 and we go back to the pipeline's directory, you can kinda see, like, the fetch dot YAML one. And So there's a base one, a primitive pipeline called run, which run arbitrary commands or scripts. Exactly. And we replace the expressions based on our inputs, and then that gets templated into a shell script, and then it runs the the fragment there in a shell. And it's it's basically what you see on the screen there. So what that does is it fetches the thing, and then it does the SHA two fifty six comparison, and then it extracts it. And if any of these

45:29 steps in this run fragment fail, the pipeline fails automatically. So and then the failures bubble up and all of that. So Cool. Why the double braces instead of just, like you know? We didn't want to conflict with normal shell scripting variable substitution. So we have our own syntax for doing variable substitution in the actual, like, configuration files so that there's no conflict, conflicts between, the shell and all of that. Cool. So Alright. I'm gonna look at one more just because I'm curious. I'm assuming they're all Yeah. Much the same. Cool. Is there any other parameters beyond runs, or

46:36 is are all the other pipelines constructed from our run statement? It's just constructed from a run statement. Okay. Cool. So if we go back to our examples, I'm assuming that we can run melange help, and it looks very similar to what we got with apko. Yeah. Except where you have some key gen stuff too. I don't think I noticed that on apko. Well, because apko is just a, image composition tool. But to, like, interact with the packaging system, then you need to be able to, like, sign packages and sign indexes and so on. So our first step

46:47 Melange Signing Keys

47:27 I would go back to the main directory. Mhmm. Our first step is we need to generate a key pair to, like, sign all of our things with. So if you do melange keygen, it'll actually generate the key pair that we're going to use. And if you look, you can see that there's now melange dot r s a and the melange dot r s a dot pub. Those are just standard x five zero nine key pairs, and there's nothing really that special about them. Now we can actually build something. So if you do melange, build examples slash g n u hello dot yaml

48:08 Building a Package with Melange

48:23 and then dash dash workspace d I r equals, like, work or something, and then do dash, and then do dash dash signing key equals melange dot r s a. It's signing dash key. Oh, right. Yep. It's the r s d. And then hit enter. It should build GNU hello and sign it. Oh. Oh, right. Anything that melange the user share? We have to do make install. That that's the part we were missing. Alright. Okay. Cool. Alright. So now it should work. Bad substitution. Bad substitution. That's weird. Go to go well, we can fix that real quick. Go

49:54 to open nano or open the user share melange pipelines autocomf configure dot yaml. Apparently, the part that doesn't delete the inputs dot ops thing is not working correctly for some reason. So if we just delete that, it should work fine. Kind of that was working earlier today, but oh, right. We had to delete the workspace to reset everything. So my work or something. Delete that. Yeah. Alright. If we run it again, now it should work. And here, it's doing the apko step. Nice. And there we go. Now it's working. This is awesome. It it should be noted

51:01 that melange is not released yet. So But I thank you for coming on to show us that, it's it's it's going very well. We have an APK. So so you now have, some signed APKs. And if we create a repository like, if you do if you copy the melange dot r s a dot pub to e t c apk keys, and then you do apk add and then the hello dot apk, it should install it. And then if you do hello, you'll get the GNU hello world program, which they managed to make a hello world program

51:14 Using the Custom Package in an apko Image

52:02 that compiles to a 500 kilobyte package. So yeah. So we have an APK here. Now the next step would be, I guess, to make a container that has that hello world package in it, which should answer a mean's question. He was they were asking about circular dependency. And the answer is no. There's no circular dependency because, melange only depends on the apko go package. It doesn't depend on the apko, like, command itself. So they're otherwise completely independent tools of each other. So if we go and make a directory somewhere, doesn't really matter where. Okay. And then we make a apko

52:31 Circular Dependency

53:21 dot yaml file. Was it packages? It's contents. Damn. Alright. And then we do repositories colon, and then we do, like, HTTP, you know, the usual Alpine one, d l c d n Alpine Linux Org. Copy it. Okay. Yeah. That that works. And then Do we need we replace either of these? The base We do need to keep, Alpine base layout. And if we do hello, and then if we go up to the first one, to the repository section, we'll make a local work we'll make a local repo. Mhmm. And we can have that, like, in

54:36 slash home slash or, like, wherever you're gonna put it, root slash repo. Sure. That works. You don't actually have to use the file colon slash slash. I would just recommend having it as slash root slash repository. Yeah. Okay. And then we can just do, like, entry point user bin hello. Like, the command. It it still needs to be defined. You were showing me some fancy Sorry. Hand there. User Not yet, but soon. Like so. Yeah. And then save it. Alright. So the next thing we have to do is actually set up a repository. So make DIR repository,

55:22 Set Up a Repository

55:28 and then go in there and then make DIR x eighty six sixty four. And then go in there. If we copy the APKs for the melange directory okay. Now if we do APK index dash dash output, apk index dot tar dot it has to be capitalized. Sorry. Oh, yeah. Like, a p that's right. Yeah. Dot tar.gz, space star, space star.a p k, and then hit enter. Okay. Now we have to sign that, which we're going to use melange to sign it. So if you do melange sign dash index and then dash dash signing key equals root melange dot melange dot r s

56:47 a dot pub or dot r s a rather, and then a p k index dot tar dot d dot tar dot g z, it'll sign it. Okay. Now we have a signed index. If you go back to the directory of our apko dot yaml file Jitter. Yep. We can now do we can now run that a p k o dot YAML using the a p k o build command. I don't think we may can stop that. So I'm gonna have to do a p k o a p k o build. Yeah. We didn't. And then it was just the failed. Yeah.

57:38 And then hello colon latest and then hello dot tar or something. Do I need to add my no. Oh, right. Right. Right. Right. Right. Right. You have to do, go go up and bring that or no. No. No. No. Go back to the new DIR. Bring that apko command back up because we have to actually add the Yep. Yeah. That one. We have to actually add the key to the trusted key ring. So it's a p k o build dash k. Is that for n secure? No. That's the add of the key. Alright. So it's dash k

58:38 and then the e t c a p k dot or e tcapk melange R p r s a dot pub. Yeah. That. It should work. There we go. Alright. So we now have a hello dot tar with the hello world package in it. And if you do docker load and if you execute it, it should say hello world in theory. Ta da. Nice. And that is a one megabyte container. Yeah. That's pretty trim. So That's cool. That's a p k that's APKO melange and then a how apko and melange are used together. Very, very cool.

59:39 Recap: The apko + melange Workflow

59:51 Obviously, like, in a enterprise, like, so called secure software factory, you would have, like, actual repository management instead of, like, using a local repo, ideally. But setting all of that up, it would not be fun for, like Yeah. A single episode of Rawkode Academy. So I feel a bit silly for asking you from Alpine Security working for chain guard, from apko, melange, and s bombs and all this stuff. If dash k was the same as curls and secure. I'll edit that out for sure. But that's really cool. I love that we can have this workflow

1:00:00 Production Workflow, CICD, and Key Management

1:00:43 where we build APKs, we build emojis, and we get the stain and verification all the way along. And we have a question about the about the s bomb, so I'll get to that in just a second. But this is just a really cool workflow. I'm assuming that, you know, given a bit more time, there'll be GitHub actions and that it's gonna make this really, really easy for people to get started. Is that the plan? So, yes, so for apko, we already have GitHub actions. Like, if you go to the distroless organization on GitHub, you can see that we're actually rebuilding the

1:01:00 Github Actions

1:01:21 Alpine base image based on whatever is an Alpine Edge every night. And that's fully done with GitHub actions. And the way that we've set this up is people should be able to go and reuse these actions without any problem. We have a template repo up, for example, that has everything that you need. And that's a good starting point. Thanks, obviously, to at for doing the legwork on the template repo. But if you use that template repo, you can have automatically refreshed images every night. And it'll go and it'll sign them with cosign, and, like, every single part of the pipeline

1:02:26 will be, like, ideal for the supply chain security aspect of it because you'll have images that can be scanned, and then they'll be always up to date. And it'll be yeah. It it'll be always up to date, and it'll be scannable. And as you can see here, it's not that complicated to do. Yeah. It feels to me, like, haven't walked through this with you now. There's only one part that I think can't really be ephemeral, and that would be the key gen step. Right? That would be something that we did. We do once and we store as a

1:03:14 GitHub action secret or in some other secure place, like like Vault or something. Right? Then that would be a pair of key. You you would use something like Vault ideally. Like, if you were going to use melange to, like, have automatically built APK repository for the public to use, You would use something like Vault and integrations for managing APK, assigning material if Vault already exist. Yeah. So And the other thing there would be the the APK repository that we created. I mean, that could be a femoral. That doesn't need to be maintained or long lived. It could

1:03:51 that can only exist during the CICD process. Is that is that fair? Correct. Right. Okay. Correct. But, I mean, if you want it to be permanent, there's stuff like JFrog, artifactory, and stuff like that that already supports managing APK repositories. So there's there's no shortage of options out there for all of that. And, obviously, we we would like to hear people's opinions on what's already out on the market for managing APK repositories in a corporate setting because I have opinions on it. But Okay. Let's tackle a few more questions that we've got today. So we got one here

1:04:19 Project Status and Future Vision

1:04:46 from g I could one. Do things that are fetched or added using pipeline stage Like, this get added or are they represented in the bill of materials? Not yet. Right now, the Alpine ecosystem is still using APK version two, which does not really support capturing all of that data yet. But APK version three does. And once everything switches over to APK version three ecosystem, all of that will be in the SBOM for free. That that's our intention anyway. Alright. Let's see. We got ta da. We got I agree. Oh, no. My comments are just coming in the middle again. Thank

1:05:45 you. Creating a local repo in Nortem just out of memory. Yeah. I was impressed by that too. The fact that you were just pulling these plucking these commands out of your head. But I guess you did this DND out. Right? This is Well, I mean, it it does help when you, like, made a lot of this stuff. We got a wave at yep. Converted all emojis to melange and apko. Yeah. I'm gonna be right behind you there. This is a really impressive tool chain. And it's so early. It's it's cool to be just seeing how this evolves as well.

1:06:21 Yeah. Especially if melange, there there's a lot that's still subject to change, but we're hoping to have, like, a release out pretty soon. There's a couple of other initiatives that are in the pipeline for my team at Shingard to kind of figure out first. Like like I said, we kinda see all of this as a spiritual successor to Google's distroless. So if if you think about Google's distroless as, like, distroless one, this is, like, distroless two that we're building, basically. And so we're kind of trying to come up with some ways to ease the transition away from Distroless one to

1:07:27 Distroless two. So at the moment, those are the things that are kind of getting ready to take priority. But the hope is that we'll have a early preview release of melange out later this week. Wow. Quick. And then apko 0.2, which we walked through using is also due out this week. So that's kind of the pipeline there. And then we're doing some similar tooling for Debian to kind of support the legacy distroless users as we try to evolve the Distroless ecosystem in this direction. And Is this you know, it's hard to tell with the with with the projects we're still

1:08:25 Contributing and Community

1:08:29 in this this kind of discovery phase and rapid evolution, but is this something that people can get involved with? Are you looking for contributors to come along and even provide feedback or write code? Yes. You are. Yeah. Absolutely. We've already been merging patches from external contributors. I think, PJ earlier in the chat, he's one of the external contributors that has contributed code to apko. And then there's another big contributor that's outside ChainGuard that's contributed a lot to both apko and melange named Cruskal, and we're trying to, like, always get more people involved in in the

1:09:17 project. We fully understand at that we need to do all of our work in a fully open source way. And so everybody's welcome to participate and Awesome. Contribute, like, ideas, bug reports, patches, whatever they want to do. Cool. We have a a couple more questions then, and then we'll we'll wrap this up. We got one from Moz saying, sorry. I joined late. I'm curious if melange can be used to send Docker images. Now I must have been melange is using cosign to to the same input, or is that wrong? Melange just uses the APK style RSA signatures to sign packages.

1:09:41 Signing Docker Images vs. APKs (Q&A)

1:10:05 Packages. To sign, like, a Docker image, you would use cosign, which that uses all the Sigstore infrastructure. And SigStore is kind of the modern way of doing it. One thing that I will say is that we are kind of looking at a way to integrate in the SigStore full CO style signatures into APKs so that you can have the keyless signing instead of having to deal with having, like, a key in in stored in, like, Vault or as a GitHub action secret or whatever. But that's gonna take some work on the APK side to do first. So

1:10:56 Cool. Alright. And is asking, would you mind explaining the intent behind the distro listing? Can we achieve the same thing by building images from scratch based images? Right. So the point behind Distro is is that you can take just the components out of a software distribution that you actually need to support an application and have it done automatically. So you can do things like build a statically linked binary and then copy it over into a scratch image. But the problem there is that if you do that, like, the tooling, like, Trivy and Snake and all of those scanners, they will

1:11:01 Explaining the Distroless Concept (Q&A)

1:11:51 not understand what's going on, and they won't know how to scan it. So by using this distroless technique and capturing everything into packages, you can have full transparency to the scanners, basically. And that's a powerful thing because right now, the scanners miss a lot of details because they can't see the app that you're trying to scan. Yeah. With the statically linked binary, you you don't know if you're using a version of open SSL for instance that has some current CVE Exactly. Problems with your application. That's that's the problem in the space that these tools are

1:12:36 are trying to solve and why they're so important. Alright. Well, we're all out of questions. Is there anything else that you would like to show or share before we wrap up for today? I think that's it. Awesome. Well, thank you so much for taking time out of your day and for coming on to kind of walk us through this. I really am really excited by this, and I can't wait to start kicking the tires on it. And it's just really chilling in a space that can hopefully pry us away from the the Docker file and all these

1:12:42 Conclusion and Thank You

1:13:10 other convoluted build tools. So very exciting. Yeah. Absolutely. And it's been good coming on the show, and hope to do it again sometime. Alright. Well, have a great day. I'll speak to you soon, and thank you again. Have a nice day. Alright. Bye.