1:35 Introductions

1:39 Hello, and thank you for joining us today. I am with Phil Estes, a distinguished engineer at IBM and also the maintainer of the containerd project. Hello, Phil. Thank you for joining us. How are you? Great. Yeah. Glad to be here. Thanks for having me on. No. This is awesome. Containerd is one of those projects where I am familiar with it exists. I am familiar that I'm using it in production, and I don't really know much more about it beyond that. So I'm looking forward today to getting into some of the details of how containerd works and why it exists.

2:16 Awesome. Yeah. Excellent. So I think let's just start with a viewer question because I think it is important, and I may already have made my first mistake of the stream. But this person asked on Reddit prior, how do I pronounce it? Is it containerd or is it containerd? Yeah. So the project, officially, we we call it containerd, but it's definitely been an inside joke or not so inside joke that we call each other containers. So, yeah, officially, the project name is definitely containerd, Thinking, you know, how systemd and other daemon like processes are are called. That's kind of the the nomenclature.

2:20 How do we pronounce containerd?

3:05 Right. See, that's sometimes you just don't think about that, but I think it's so prevalent, especially, you know, in the technology community. We have so many projects where you don't always know the right way to see it. You know, we always have these weird debates around, is it MySQL or MySQL and all this stuff. So I think it's good to be clear clear that up for sure. Yep. Alright. Do you wanna if you don't mind, just give us a little bit of a backstory then. How did containerd come to exist? Yeah. So, you know, I think the the easiest

3:30 How did containerd come to exist?

3:37 starting point is to think about, you know, Docker's rise as a developer tool, you know, that 2014, '20 '15 era. It was really easy to use and install because, essentially, that single binary, at least, you know, speaking in Linux terms, you know, there wasn't Docker desktop. There weren't all these other products around it. It was just Docker. And Docker, the tool, could easily be put on any Linux system. It was statically linked. You didn't have to worry about your distributor giving you the right version linked to the right libraries. And it was sort of an all in

4:18 one tool. It had registry support. It had ability to build containers. It ran containers. And, obviously, you know, over those next few years, there was an explosion of features, and, there was also, you know, some community tensions around when Swarm got added and Kubernetes was coming on the scene or effectively was already there, but rising in significance and usage. Out of all that, you know, you also had people with other ideas for what a container runtime should look like. And so out of that, the OCI was born, so the open container initiative. So most people,

5:00 you know, may know this, but at from about 2015 on, Docker or other container engines that are OCI compliant rarely are actually doing the execution of your contained process. That's left to a binary called run c of of open source project maintained by the OCI. And so as soon as run c sort of came on the scene and Docker started using run c, Michael Crosby and some others inside Docker built this kind of shim layer between the full Docker engine and run c, and that was the initial containerd code base. If you look back in our GitHub GitHub

5:44 history, you'll see a branch zero dot two dot x. That's the code base that effectively was that start of containerd back in that era. So containerd at the time was really a a runtime manager. So Docker said, you know, hey. Somebody called Docker run. Containerd, can you go start a run c process? Here's all the details. It builds the OCI spec, you know, which declares things like which binary am I running, which namespaces am I using, you know, any c groups. So all these configuration details that, if you're a developer, you think of those are flags like on Docker run, you

6:27 know, things I wanna change to this user or, you know, add this volume mount. All that ended up passing down through these layers and containerd effectively managed that run c layer. So, again, kinda skipping a lot of exciting history in the container ecosystem at that at that time. You know, you can go back and find tweets and blog posts. You know, Docker is getting too big. Swarm is competing with Kubernetes. And so, you know, cloud providers like IBM and others were talking to Docker about, you know, maybe it is time for a more kind of

7:04 stable core runtime. You know, Docker has storage plug ins and registry features and an enterprise product, you know, maybe we can segment out just that core runtime idea into its own into its own layer. And so containerd effectively moved from that zero dot two branch to the one o sort of release time time frame in late twenty sixteen. And that became the more full container runtime that if you install containerd today, if you do the latest one four o or or you're using Docker, which I think over the weekend updated to one three seven. You know, this this

7:49 version of containerd is now has, you know, full image support to push and pull images. It's not just a runtime manager, but it handles the content and the task management and metadata and all those things so that it can be a full run time underneath Kubernetes, which is probably the most popular use case for containerd. But there's still millions of Docker engines that use containerd under the covers, again, to manage that run c layer of, the actual processes being contained. So, yeah, that's the whirlwind tour. You can find some talks on YouTube about, you know,

8:29 what is containerd, where did it come from, and maybe a little more of that history. But that's kind of the the quick version. Excellent. You actually answered another viewer question there as part of that kind of summary there. There's another question from Reddit was why does containerd exist when Docker, you know, was a thing even though they understood that Docker now uses containerd? And I think the answer there as well, this was at least in the initial version was spun out of Docker to simplify that runtime aspect. Absolutely. Yeah. Okay. So, I mean, as far as containerd

9:00 What projects are using containerd?

9:05 goes and and as touch points with the container ecosystem, you you've mentioned Docker there. So Docker uses containerd to actually orchestrate or run the containers. Kubernetes now uses containerd as as the default runtime implementation. Are there is there anything missing there that we haven't covered? Is containerd used in any other large substantial projects that that's worth mentioning? Yeah. So the I I think yeah. There's two aspects that I I'm stumbling because I there there's a couple aspects to that. There's because containerd really has a a API layer that's been very attractive. There's a lot of sort of consuming of containerd

9:50 across a pretty large number of projects that aren't necessarily just using it as a runtime, but consuming it, in some other way, or expanding on its capabilities because many parts of the architecture are pluggable. So I think you know? So what what aspect of this whole thing is we could talk about all the cloud Kubernetes services that use it by default. We've already mentioned that. Linux kit, which a project that came out of Docker, kind of a containerized, you know, minimal secure Linux uses containerd as the core runtime. A lot of the sort of lightweight hypervisor

10:37 or other advanced isolation capabilities like Amazon's firecracker project or cata containers. They've built shims. When we look at the architecture, we'll we'll see what that means. But they've built shims so that you can, again, drive containerd through its API or from Kubernetes. And you can use these custom, you know, isolation engines, gVisor, again, another option there. And then Cloud Foundry used to have a run c management layer, and they've effectively switched to use containerd. So, again, Cloud Foundry is a pass like Heroku, and so that's another kind of major user. And then, you know, there's just a a

11:23 set of various vendor projects. So Alibaba Cloud built a tool called pouch container, which is almost like a a Swiss army knife, docker like container runtime that, again, embeds containerd and exposes all these features that Alibaba Cloud uses on their platform. And then Rancher did something similar when they built k three s. So this kind of minimize Kubernetes. Darren chose to build containerd directly into k three s. So you don't even use a separate containerd binary. Containerd is the run time for k three s, and it's built directly into that single binary. So and then, you know, I guess one more

12:09 that comes to mind, Alex Ellis built FASD. So, again, part of his open FASD hierarchy of projects. FASD gives you a way to you know, you could deploy open FASD as Kubernetes based deployment, but you can also run FASD and you're just effectively using the containerd runtime to run your functions as a service in containers. So yeah. That you know, I'm probably missing some, but that's you know, people seem to be happy to kind of explore how to embed containerd in other projects because of this simple kind of API layer. Actually, one one other project that's worth mentioning

12:52 because you may already be using it, the build kit, which again is an open source project out of Docker. If you're using a very recent Docker, I'm not sure if they've made the switch where build kit is default, but it will be soon. But, again, a a a container build engine that uses containerd's API under the covers and can build directly with containerd without Docker at all. So, yeah, those that's a handful of kinda containerd using or exploiting projects. That was a lot. There's a lot there. Is now a good time then where we pull up this architecture diagram and you can

13:30 Looking at the containerd architecture?

13:33 give us the the quick tour of some of those things you were talking about as well? Yeah. Yeah. Absolutely. Alright. There we go. Yeah. Look at all the boxes. So let I I think one reason why, you know, this particular image is useful is you can kinda see ways that various tools kinda insert themselves of various layers of the containerd architecture. One thing to note here that is important is that the containerd architecture, that core layer in the, I guess, greenish, light greenish color, is actually built as a unique or differentiated set of gRPC driven services.

14:24 So all these services could be used by themselves. And the reason that's important is that there are people who have written libraries or uses of containerd that replace, you know, a specific service so that you can use, you know, this construct even outside of using the containerd daemon altogether. So, you know, or as I don't maybe some listeners have heard of or as out of the desk labs team at Microsoft, but they liked the simplicity by which you could use the Go API for containerd to talk to a registry. But they didn't want the content service

15:04 to be writing or to writing to containerd's content store, which you would need the containerd daemon running. So they just were you can make these calls and say, actually, my content service is just this place in memory over here or this file system. And now you can write a library that uses a piece of containerd to do registry interaction, but you don't need the containerd daemon at all. So there's all kinds of uses where any one of these GRP services, so the content, diff, images, leases, namespaces, all these services are available. You should call them directly via gRPC,

15:41 or you could write your own implementation and and tell containerd, actually, no. Here's my implementation of content or containers or tasks. Most people aren't gonna wanna do that. They're just gonna use it. They're gonna go a layer up. So then containerd has an API where it simplifies talking and and meshing these services together. So you can think of, like, using it from Kubernetes through the CRI. So that's that left block of the containerd API. You're calling the it's using a containerd client go API to do things like run a container, pull an image, and all those services are used

16:22 by that API to do that. And, if you know about if you've ever configure configured the Docker engine with a specific graph driver, like, oh, I use overlay or I use device mapper, We call those snapshatters in containerd lingo. And so, again, we support a lot of the same ones you've seen before, but it's also pluggable. And that's interesting in that If you don't like any of these snapshatters, you can write your own, and there's some very interesting work. There's a talk from KubeCon a few weeks ago on the star g z snapshot, which does lazy image pull and anyway, it can

17:02 do major speed up in container startup time. So having the snapshot or be pluggable means you're not limited by the ones that we've thought up or implemented. And it allows things like, you know, Windows to use their own snapshot or their own implementations that Microsoft has written. Stepping to the right of that, when you actually run a container, I talked about shims. You know, obviously, the most natural shim is to call run c. I'm on Linux. I just want Linux kernel based containers. I just want run c to run my process with the isolation I specify.

17:39 But because our shim API is pluggable and you can extend it and configure your own, This is where you see Cata containers or Firecracker or gVisor or on my on Windows, the run HCS tool or or shim that Microsoft has built to run Linux containers on Windows as well as Windows containers on Windows. So, yeah, we've got so the shims are down there at the bottom. We've talked about Snapshotters, the pluggable content store, the array of gRPC services. And then, of course, above that, you know, you can have container engines consuming all of this by using the the Go

18:22 library or calling the API. And so Docker does that. The pouch container project I talked about does that, via the CRI API. Kubernetes does that. And then build kit, again, another consumer who also just says, I wanna use containerd as a client. I'm gonna use the Go API to use their features for pushing and pulling from registries, and so on. And then we have a simple administrative client called CTR on the far right there of the client. And that's, I think, the most important point for developers is we don't really expect that to be a replacement for the Docker command line.

19:04 The Docker command line, of course, has grown over the years to have lots and lots of features. We're not seeing containerd as a replacement for a direct kind of developer workflow. So CTR will probably feel clunky in some ways to to a developer who's used to the docker workflow. And there's a purpose to that because it's not really meant to be a replacement for your day to day use. It's more administrative debug so that, you know, you can test out containerd, But we expect you're probably using it via some other wrapping tool like a platform. So that top layer,

19:42 I'm using Azure's Kubernetes service. I'm using Docker. I'm using Rancher's k three s. You know, all the all these use cases don't require me to to use containerd as kind of my main development tool, but containerd is there providing, the platform, for these tools to then have a easy API for a container runtime. And so that that's kind of the end result of what we talked about with the the history is that containerd is kind of the stable layer that gives you all those features, but really is not meant to be a replacement for the developer workflow.

20:25 So, yeah, I I hope I did it justice. There's a lot a lot there, but that's that's the bulk of of how it's assembled and what the pieces are. Yes. You definitely did that justice. I think I'm just in complete not shocked, but maybe all. Just the size and breadth of the of the project, you know, and previously when I've always thought about containerd, I think of the cube blitz telling containerd to go run a container for me and it being such a small API surface. But, really, there's a lot of moving parts and a lot of things that can be

20:57 swapped in. I mean, just looking at the, you the containerdish shims there, moving between run c and maybe kinda containers or firecracker. Like, a lot of flexibility in this architecture that, I guess, is why people are adopting it for their own tooling. I you know, I think, you know, we've gotten a lot of positive comments about the API over the last couple years. You know, when Alex came and said, hey. Let's you know, can you help me figure out how I might use containerd underneath FASD? We've worked as used it underneath a few of their projects.

21:31 And, you know, there there's always a learning curve. It's like, oh, I didn't expect it to work this way. And and I think there's another question that you got about documentation, which we can talk a little bit more about. But, you know, it's hard to as a small team of maintainers to to cover all the use cases and make sure everybody has the information they need to consume containerd. But in general, you're absolutely right that people that have dug in a little bit say, hey. This is actually really easy to provide all the features I need

22:06 for, you know, embedding or utilizing a container runtime of my project, and it just works. It's stable. It it's simple. And and so, yeah, we're pretty happy with the comments we've gotten from people that that have ended up using it for their project. Excellent. I'm gonna quote you that when anyone ever asked me how difficult is it to integrate containerdames with a project. It's be Phil said this is easy. That's that's difficult. Do you see a future where more I I don't wanna say something's gonna try to replace Docker as, a developer tool, but do see more people using containerd as base

22:45 for developer tooling? You know, I know a lot of people are frustrated with, you know, local development tooling for Kubernetes. And do you see containerd be in the API that people start to adopt for things like that? Yeah. I I would hesitate thinking that containerd will will fully replace those kind of workflows. I I again, we see it as a more as an embeddable runtime. You know, I mean, a couple of comments like I think the there are a lot of people on Macs, for example. There there's people using Windows and WSL two. And so, you know, because containerd is a

23:30 Linux focused project, and, of course, we have Windows support as well, You know, you're you know, unless we find people who who really wanna dig in and build kind of another layer around it, you're not gonna get the ease of, like, Docker desktop. You know, where on my Mac, I've got this sort of streamlined. It feels like I'm running Docker commands on my, you know, Mac terminal prompt, and I'm pulling images. You know, all those all that ecosystem doesn't exist for containerd. So if you're a Linux only developer and you're, you know, building on a sort of Kubernetes

24:11 ecosystem layer, then CTR as a tool may be enough for, like, what you wanna do. Because, you know, quickly, you're gonna go from CTR to using kubectl, you know, for actually doing deployments or or your Kubernetes based work. So yeah. I'm I'm rambling a little, but I think, you know, I I don't see containerd really ever being that full kind of Docker ecosystem replacement. Because, again, Docker is more than just the container runtime these days anyway. You know, one thing we haven't said officially here is that there is no CTR build. So you're gonna either need build kit or

24:55 you're need Docker or some in cloud, you know, CICD build service to actually take docker files or whatever your input is to build containers. Containerd has no container assembly code build code in it. Alright. Got it. So there was a a talk you gave at KubeCon a couple of weeks ago, and there was a slightly different version of the of the architecture diagram. Yeah. Is there anything here we should we should cover? Yeah. It's it's a good question. I I'm gonna I'm getting old, so I'm gonna put on my glasses and make sure I can

25:35 read this. So the yeah. This is kind of a a it's almost the same view turned on its side, and it's trying to give a little more functional I mean, we're trying to make it clear, like, how the pieces actually connect together. And so the client has been expanded, and we've kinda taken out the consumer part of it. So you can see, like, the client now helps you see how do I actually access, you know, what's behind this. You know, how do I you know, so I configure my client with some options. How do I connect to the

26:13 gRPC socket? I subscribe to events. I you know, for containers themselves, I pass in options. I get an OCI spec. And then these proxies, that's that's the actual gRPC traffic across that boundary to the containerd one dot x API. And so that kinda you know, it it looks duplicated, but that's that helps you see that, like, on my client side, when I say CTR run, that task service will talk to the the task API and then use the actual implementation on the server side, which could use a you know, it's gonna use the images and

26:57 snapshots service to assemble the right root file system to actually call the task manager to start a shim if I'm using run c by default. So this view, you know, is easier to talk through the flow of, like, how does, you know, CTR run actually turn into a series of steps that gets me a running run c shim at the at the far right side. But there you know, as far as the API, the the great news for consumers is, like, we have not had to break that gRPC API for two years now. And so we hope we hope that this

27:36 API is extremely stable that if people need functional differences, they shouldn't be coming to us and say, hey. Please change the containerd API. They should be looking at the pluggability of this core and saying, oh, I could do that. You know, I need a different shim because I'm trying to do some fancy thing at the runtime level, or I need a plug in for snapshots because I want a new way to, like, represent the file system. Or I have this cool file system that you guys don't know about in my lab, and I want to utilize it with a

28:11 snapshot or so, yeah, the I think the main point of this is that actually, one more comment. The API layer, you see the CRI plug in. So one thing I didn't mention in the history is that containerd was initially built without an implementation for Kubernetes CRI, which is the container runtime interface. So that is a separate gRPC API defined by the Kubernetes community. If you're interested, SIGNode is the Kubernetes special interest group that deals with the CRI API, and it does evolve over time. So a set of sort of separate maintainers built a CRI plug in for

28:58 containerd, and so you can find two different GitHub projects. We brought them in to the containerd repository several years ago, but we're now actually for Kubernetes I'm sorry. For containerd one dot five, we plan on actually bringing their code into containerd, and so this plug in continues to get more tightly integrated. So if you're using containerd one two, one three, one four, that binary does have the CRI plug in built in. The code base change is just to continue that natural progression of we're all maintaining the same code. We're all working together, so the CRI doesn't need to

29:40 be a separate project anymore. So, again, that just by default, that gives us a makes us a Kubernetes runtime. The Kubelet can talk to the same gRPC socket and talk that CRI API directly to containerd. So yeah. Excellent. I think there's definitely a lot to unpack there with, you know, both architectural diagrams and the different components. And really cool that that kind of CRI layer and the API layer are maybe gonna consolidate. Does that mean that containerd then becomes a competitor with with cryo? Yeah. I I always try and be careful when I you know, I think a lot of a

30:00 Is containerd competing with cri-o?

30:28 lot of folks wanna see kinda sparks fly between containerd and creo. You know, creo is really had a similar vision to containerd and that Red Hat and Docker had, you know, some some competitive disagreements over Docker's kind of future, definitely around swarm and, you know, other issues around distributing Docker and Red Hat distributions of Linux. And so Creo had the same idea of kind of this core runtime built on some building blocks. And so if you go look on GitHub, you can see a project containers slash storage. That's actually a copy of the Docker graph

31:21 drivers implementation from, like, old Docker one twelve, one 13 era. And then Red Hat continued to develop it and, of of course, officially maintains it today as an open source project under that containers repo. So there's container storage and containers image. And what you'll see is that Red Hat has built a layer of tools on top of those two projects. So it's not just Creo, it's Podman and Builda and Scorpio. They all use the same layer, and that gives them the ability to say, hey. Our tools are actually not dependent on a daemon because they're all using the same container container

32:03 storage file store. They're all using this container's image library for talking to registries. And so all of this can happen, like, kind of in the tool itself. Creo is is not necessarily daemonless in the sense that the Kubelet has to have a CRI socket to talk to. So you have to have some listener somewhere. And so Creo does have, you know, these small daemons that listen for the CRI API and to do some monitoring of the processes. But, you know, it's it's really born out of the same time frame as containerd. And, you know, as a a a very personal

32:49 comment, not representing IBM or Red Hat or Docker, You know, there was definitely, personally, a hope that that we could rally around one or the other. And, of course, naturally, containerd was my intention to to convince them to work together on that. And the the cool thing is that we've shared code over the years. They import some of our sub packages and use containerd code for some things. And so, you know, at the maintainer level, you know, we've always had good relationship and work together. But to me, it's I hope it's becoming more like, you know, some people use Firefox,

33:31 some people use Chrome, you know, maybe on a Mac, you like Safari. You know, these aren't they are competitive in in a general sense that you have to make a choice about which kind of community you're gonna use or get involved in, but they're just options. They're just options that are out there. And, yes, there are some differences in kind of the design, but that that's where I see it today. Alright. Awesome. It's great context there just to understand how those two projects fit. And, you know, with the CRI there and the OCI stuff, like, the

34:05 compatibility is there for people to Exactly. Have preference. Yep. So we have one more question, and then I think what we'll do is we jump onto the hands on component of this session. And the question came from Reddit again, and it was, what are some best practices while running containerd via systemd? That's that seems like a way a wide scope, so I'll let you if you want to Yeah. Go back to that. Yeah. Hopefully, I'll answer the right question because there's not really like, I'm not sure if they're asking if there's kind of a strong integration with, like,

34:12 What are some best practices for running containerd with systems?

34:36 the way systemd manages processes because, you know, there are there have been methods. And and I know some people still use these where, you know, systemd is much more integrated in actually the spawning and management of container processes. That's not the case with containerd. So, you know, we have we have in our repo, and I believe it's delivered in our full release tarball, a systemd unit for containerd. And that just means that, you know, containerd will be started on boot, you know, with other system processes. At that point, that's that's really the last sort of interconnection

35:21 with system d. There's no other management other than the way system d manages, you know, units on the system. You can start it. You can check its status, etcetera. So I I don't know if that's the question being asked, but today, that's kind of our our interaction with systemd is if you're on a systemd controlled Linux distribution, then containerd's unit will make sure that containerd is started and and but containerd at that point will manage its own processes that are run via containerd. There won't be any integration with systemd as far as the spotting of containers

36:02 at that point. Okay. So just a question that popped in my head then as you were kinda talking about that. Are there any rules around restarting the containerd process about how that would affect running containers, or does it pretty similar? Yeah. It's yeah. There shouldn't be any issues with with restart similar to how over the years Docker matured to where it's not gonna crash all your containers. You're not gonna lose state. You know, containerd will restart and try and reconnect to the shims. So that's one of the reasons the shims exist. The shims are the parent

36:42 of your actual containerized process. And so if containerd goes away, the shims, thankfully, are still parented. I I mean, it's still parenting your container, so it's it's not gonna wipe out I don't know if people remember. In the early days of Docker, you crash your Docker daemon, you lose all your containers. That's not the case. So containerd can restart. It'll go find all the shims, and you should you know, unless there's a bug or some other complicated issue, you'll still have access and and management of the containers that we're already running. Nice. Alright. Let me share my screen then and

37:20 Getting hands on with containerd via microk8s

37:21 we'll kinda let's let's actually play with this. Right? So what I have in front of me is a Linux machine unpack it because I run a Mac and that's not gonna work. So I have got snap installed and I went ahead and ran snap install Michael Kates classic. I have done this your advice as you said that Michael Kates gives us access to the containerd tooling as well as access to containerd running as the CRI implementation for Kubernetes. So that's all I've done. I've not done anything else. What do I do now? Yeah. So yeah. In the in true Internet fashion, we're

38:07 flying by the seat of our pants because I haven't messed with microcates in a little bit. But but just to give people a quick flavor, this is Canonical's kind of like kind. People have used kind where you you kind of have a single node Kubernetes distribution all in one, which means, of course, you've got the Kubelet. You have your your single node is now everything. You're the master. You've got a Kubelet. You have a container runtime. And it just so happens that microcates uses containerd as their runtime. And so we should, at this point, have

38:46 a running containerd, and we should have the CTR tool available to us. So, yeah, for Lucky, this should show us yeah. So we can see the path there is inside this snap of microcates. It has its own, varlib containerd, and that seems to be running. And, yeah, we can actually see some shims because, again, this is a Kubernetes deployment. So there are actually some containers. As you know, if you start up Kubernetes, you have some some system cube system containers, pods that are already up and running. So I don't know. Yeah. Go ahead. So based

39:30 Using car to manage images and containers

39:33 on what I kinda picked up then on on the API surface of containerd is that I should although we're not recommending it as a, like, you know, daily driver, be able to pull an image and and run a container using CTR. Yeah. So let's see if CTR is in your path. I can't remember if you have to do microkates.ctr or Yeah. You you you do. But I do have access to CTR. So Okay. Yeah. Great. One of the things we should talk about at this point is that containerd, unlike the sort of default docker model, is that every

40:09 every bit of content and metadata is namespaced. And so maybe do a CTR space namespaces l s or or n s works as well. So you can see that there's already a kates dot I o, and this means that that micro kates deployment is actually whenever it runs containers through the CRI, they're getting placed in this kates.i0 namespace. It's kinda cool because now if we use CTR from the command line and we don't wanna mess Now, again, there's not this is really administrative. There's no security being applied here. There's no RBAC. That's all handled at higher

40:50 layers. But we can do we can basically create a new namespace. So we we can create one called default or Rawkode. I think it's just create and then a name. Yeah. I'm just assuming that that was gonna work. So Okay. So now if we do the l s, we have Rawkode. Now every other command takes a dash n or a dash dash namespace, which gets really annoying to type. So if you export the containerd underscore namespace equals Rawkode environment into the environment, then, basically now if we do a a micro k CTR containers or or just c by itself, l

41:41 s. We'll see now why do we see containers? I was about to say we shouldn't see anything because we've exported our namespace. Did I mess up the export? It looks right. Container d namespace. So let's try just bring back the CLS command and preface it with dash before the c. Just do dash n Rawkode. And what did I do wrong? Oh, connect two forms of the same flag. Does that mean it's reading the environment? Yeah. Yeah. That's what I'm kinda getting from. Yeah. Interesting. I I yeah. That's fun. I I can't think of why you're seeing.

42:36 Yeah. I think CLS should be namespaced. Oh, you know what? The I think we're having fun with micro case doing something default behind our backs. I think they're I think they're setting the namespace for us so that so that we get the microcates environment. Okay. That makes sense. Yeah. So that's kind of fun. I don't know if there's a way to to tell micro case not to do that, but that we can still play around. We don't Yeah. We'll just ignore it. Yeah. We'll ignore it. Yeah. So, yeah, that's the fun of using somebody's sort of

43:20 embodiment. So, yeah, these are the pods that are running, you know, via the microcades environment. So we can let's pull our own image. So So can I ask you a question? Yeah. So is the part a native concept in containerd rather than a container? No. So, again, we implement the CRI, which has some of those concepts, but it doesn't require the runtime itself to to have those concepts in its own. So there if if you go back and look at that architecture diagram, it just thinks in tasks and containers. There's nothing on here that says there's no

44:00 med data type called upon. And so it's really the CRI plug in that that that has the knowledge that, oh, these two containers are in the same pod, so I need to share the network namespace and UTS namespace, etcetera. So, yeah, containerd's core code base does not have the pod concept. Okay. Perfect. Yeah. So, I guess, just to show the poll and run scenario, we can do CTR images, and then it's pull. And the only difference between, like, the way you do it in Docker is you need to you need to fully specify, so there's no default to Docker Hub.

44:50 So you need to preface that with docker dot I o slash library because it's actually the core library slash n's in NGINX, and you have to add a tag. Like, we don't let you do the so cold latest is fine. So you fully specify the the exact image. It's gonna pull the layers, and it will unpack the one for your architecture if it's a multi platform image. So it's a okay. Your Linux AMD sixty four, and now you have that image locally. And so you can do a CTR images l s just to kinda see.

45:31 And it it's a little messy because it's showing a lot of detail around the architectures that are supported, the media type of the image. Again, you know A large font would be helping either. Yeah. Yeah. So but if you if, you know, it's hard to see in here, but if you've grabbed on your NGINX, you'd you'd see that, you know, we've officially pulled the NGINX latest. Yep. And that it was a multi arc image. It supports all those platforms, but it only unpacked the the Linux AMD sixty four image. So now we can run it.

46:14 And the way we do that is just a CTR. We actually have a simplified run command, but, yeah, container run works. Actually, yeah, run run, you don't need to specify kind of the the object type. So CTR run by itself without the container. Do I need to fill Yeah. No. You don't have to fill all that. Actually, yes. You do. I'm sorry. Let's just see if I break it then. Yeah. Yeah. Oh, I need a container ID. Yeah. So you have to name your container. We don't have the fancy fun, like, default name like Docker.

47:01 So I feel it would be remiss if I didn't give it a name and it's been, though. So let's call it angry Linus. I think yeah. So there's no it actually just takes positional parameters. So you the dash end isn't needed. So just give it I'm trying to think if there's an order here that I'm forgetting. So the image and then the there is a default command. You don't need to specify command in args. So the ID is the last. So what did you get an error? I was I missed it. You should just be able to specify a

47:44 name after the the image. Oh, but you'll need the you'll need the docker I o slash library in front of NGINX. Alright. But, again, it it there's no default to Docker Hub. Alright. You're get oh, you're a pseudo. Well, that's fun. You are as root. Still weird micro case thing, probably. Yeah. But I'm again, they may they may have made some changes in micro case to kind of keep you on the farm, like, not off running random images. So I yeah. I'm sorry. I I should've messed around with that too because I I I will take credit or

47:50 Shit, it’s not working. Lets install Docker

48:37 or the fall for recommending that we use microcades just as a simple way to get containerd. Alright. Let's just snap on install microcades. What's the numbers? Remove. Spell it right. We'll get there. And then I just we'll just do a curl get dot docker dot I o. Yeah. That that'll that'll give us a containerd instance that's our that should actually start it for us as well. No. It's not faster than moving things, is it? Yeah. It it actually has to stop the Kubelet and the container engine and stop a bunch of pods, and it tries to do

49:27 it. I guess I don't really need it to go away, so let's just yeah. I mean, the snap is actually not gonna get in the way of the standard containerd. Well, it's not it's not listening to my wonderful signal. So I will grab this IP address, and I'll just that's it. That's the same one. Course, it comes back. Right? Right. So curl, it should be at get dot docker dot I o. And we'll just okay. Let's just save it. Have I got that wrong? Get.DockerIO. I just copy it from here. Get off there. There we go. There you go.

50:41 Yeah. So that's interesting. I I was chatting with the microcades folks now and then and and seeing if I could use it for kind of as a standalone containerd, but that's it's been a while. So maybe something has changed with that. Alright. Yeah. I mean, I I've used micro kits a few times. It usually works pretty well. I'm sure it's just something silly. But we can quickly get back to where we were. So we've built the engine with latest image. And in theory, now I should be able to do run, and it was Docker IO library

51:20 engine next latest and then angry wireless. Ta da. Yep. There we go. So a couple things that are worth mentioning. So that can be disappointing for folks. You know, Docker defaulted or had a default implementation of an a core kind of networking implementation because containerd was built in the context of just, again, this core, like, run an OCI image, you know, take the spec, create the right namespaces. There's no default, like, IP forwarding, bridge, all those pieces that people kind of expect out of the box because Docker had it. Kubernetes doesn't need that, right, because you've got a

52:11 CNI plug in providing the that networking layer. So this NGINX we just built, there's no way to, like, talk to it because there's no IP address in this container you just started. There are some fun tools people have written. You can kinda add that on your own. You can mess around with that. Or, again, you can use it via another layer, and you get that to however that other layer provides it. So, you know, if people wanna play around with that, Michael Crosby has a a nifty tool called BOSS, b o s s, that wraps containerd,

52:54 add some management. I think he runs, like, his kind of personal network services and and system using BOSS. It has some nice restart and checkpoint features. So, again, containerd is really built as something you build on top of. And so there are things that just don't come out of the box such as networking by default. Yeah. I was gonna say, like, you know, you said earlier, and you've kind of we said it again there. This is not a tool for people sitting in front of a computer that wanna run containers. And I was thinking, but CTR run wasn't that bad, and

53:35 then that there's no networking. There's no volume system. Maybe not something I wanna do then. Yeah. Yeah. And then, again, with the CRI plug in as being one of our primary use cases, you know, you can think, well, if I put that in containerd, I'd be duplicating things that the Kubernetes layer already does for me. And so that's some of the thinking behind that is, you know, I don't need those features because the layer above me is gonna provide those. And, you know, it it it's a it's a good point, especially for that Kubernetes use case because

54:14 I know for a fact that, it becomes a security issue to some consumers of Kubernetes services when they find out, oh, you use Docker as the Kubernetes runtime. What do you do with the Docker default bridge? Because it's not it shouldn't be used, but it's there because Docker created it when it when the daemon started. And so people that use the Docker engine as a Kubernetes runtime, you may be asked in a security audit, you know, have you removed the Docker bridge because, you know, your containers are being connected potentially to two different networks, the Kubernetes CNI implementation you use,

54:54 but also the Docker default bridge. So, again, not having that in containerd is one less thing that you have to monkey with because there's no default kind of network implementation. Right. Got it. So we're approaching the the hour now, and we have a few more questions that were submitted via Reddit and Twitter, which I'll I'll run through if you're happy to, and then we'll we'll wrap up for for there for the day. I think that's great. So the first question I have now is how do I debug containerd issues? Which log files should I look in, and how do I

55:00 How do I debug containerd issues?

55:31 fix it? Yeah. So if you're in charge of running containerd, the daemon, there is a log level setting that if you do, you know, dash d on containerd, you go into deep debug mode and you get significant volumes of of information on events, the API, on registry interactions. So, obviously, if if you're able to to restart the daemon, reproduce your issue in debug mode, that's almost always a a valuable source of, you know, what happened? Why didn't this happen? What what was the error? Why did the daemon say my container couldn't start or the OCI runtime

56:21 So that that's kind of step one, run your daemon in debug mode. Gonna be a little trickier if you're a consumer of something that's using containerd, how do you get them to turn containerd into debug mode? So that that may be a secondary thing you have to figure out. If again, that that depending on your your setup, where that actually logs when you're in in debug mode. You know, as a as a developer of containerd, I'm just running containerd in a separate terminal, you know, either piping to a file. But if you're running via systemd,

56:58 you should be able to extract the logs using the journal, you know, commands for system d. If that's not producing the information you need, say you have a hang, you know, containerd seems to not be responding to the API. Similar to Docker and other tools written in Go, you can send a sig user one. And if there's actually a keystroke you you can do if if, again, if you're running containerd in a terminal. But if you send the signal and the cool thing is our shim's supported too. So remember that the shim is a separate process from the daemon.

57:40 You can send both of them a sig user one and get a full Golang thread dump process dump. And a lot of times, you know, we ask people reporting issues about hangs or unresponsive containerd to get hub, we ask them to send, you know, that signal and send us the the Golang routine output. And a lot of times, can see, oh, your root file system, like, isn't going away, like, when we're trying to clean up your container, and there's another the shim process is waiting for that to happen, and it won't return. So, again, the Golang

58:23 output can be really helpful for debugging containerd as it's useful for other go go based tools as well. So, yeah, that's probably I mean, those two things are gonna get you 99% of the way. People working at a much lower level, you know, you're building the next gVisor, kind of containers, and you're down in the bowels of kernel interactions. You know, there may be other you know, obviously, lots of cool things that are happening with eBPF and tools that, you know, track syscalls. You know, that if you're down at that layer, there there are probably a series of tools

59:00 Are there plans to improve the containerd documentation?

59:02 that could be valuable to figure out things like, oh, there's a seccomp profile blocking my container from using a certain Cisco. That's why it's not working. But that's gonna be more BPF layer related. Thanks. That was an awesome answer to a very big question. So the next one I'm gonna try and bridge it for you is is is quite lengthy. But this what I could the premise of the question is there seems to be some discrepancies between a v one and v two configuration schema. Now I'm not sure what that is, so maybe you could clarify that. Yep.

59:37 But the question is then, are there any plans to improve the documentation around that so that there's some those hurdles for people adopting containerd? Yeah. That's a good question. So I'm pretty sure they're referring to containerd's TOML configuration file. So like most daemons, you can run it with a default config. You don't have ever have to look at it. It probably is gonna do you know, when you just installed Docker and you got CTR and containerds running, you didn't have to create a config TOML file because Docker created one, containerd started, and you didn't have to worry

1:00:19 about those settings. Because containerd has this plug in model and there are a series of switches, you know, turn this on, turn this off. Over the years, the you know, we chose TOML. It's very simple, descriptive language, you know, has some similarities to YAML, of course. But over time, we found that as we were adding features to containerd, especially around, like, registry authorizer support, like, you know, pull through caches, all all these kind of fancy configuration options, and the fact that each plug in can have its own section in the TOML file. We found that what we had started with

1:01:12 was just gonna be too simplistic to handle some of the capabilities we needed. And so there's a new TOML format that the parser in containerd, if you have version equals two at the top of your containerd config, which is now the default, it will be read, obviously, as a version two config file instead of version one. And I think the feedback is right on that, like, maybe we haven't again, there's a there's a set of users who will never care about those things and never have to worry about it. But for the people that do,

1:01:52 I think it's absolutely correct that we have struggled to keep up all the documentation and all the way avenues that people come to containerd to be able to find the right documentation for that version two config. We do have we've been talking to the CNCF about and I think there were I don't know if it was part of this question that you had shared with me, but, you know, because there's so many use cases for containerd, there's the docker use case, the Kubernetes use case, there's it's embedded in this other vendor tool. It's been hard to kind of cover those

1:02:29 broadly from a documentation standpoint. And so we have been trying to get more help to raise the level of documentation that can start with how are you using containerd? Oh, you probably wanna start with API docs or, oh, you wanna see how to configure it in a certain way. And so, yeah, bear with us. We're we're trying to get there. It's an open source project. So as you know, it depends on the level of commitment from folks. We've had people say, hey. I wanna help with docs, and then they got busy with other stuff. So, yeah, we're still looking for people who'd

1:03:00 Can I proxy / pull through cache other registries with containerd?

1:03:07 love to help us improve that area of the project. Alright. Awesome answer again. We've got one final question. And I'll read this one out, because I'm not sure I understand this one. So someone has asked if it is possible to transparently use matters for all kinds of registries. Now they provide an example. They want to use a pull through cache for Docker IO, Quito IO, the GCR repository. They say that they have tried this in the past. However, they run into problems with certificates and authentication. Is that something that has changed or gotten better recently?

1:03:49 Yeah. I there has been I and I don't know which how long ago or which version of containerd they tried. You know, containerd one four just came out, which I think one four is the maybe the first version that has, like, a rework of how you configure registry connections, I guess, is one way to say it. But, you know, the the, you know, the certificates, the authentication, you know, which order to to visit registries in when looking for an image. I just found an issue from seventeen days ago, containerd issue four five three one, if people

1:04:39 wanna go look it up. But it it shows some of the plug in configuration for doing registry mirroring. And so, definitely, it's supported in containerd. It is a complicated area because, yeah, you're you're having to set up a series of registries and then all the authentication information. I wanna say, I hope it's a lot better in one four than it has been. It's been an area that we've focused on. It may be an area where we need. I I feel like I see a lot of issues around this. And it it also is a point in

1:05:14 time, so some registry implementations aren't really following the OCI spec well. So we've been we've been trying to resist adding tons of if else. Oh, well, this is artifactory, and they do a four zero three on this, and then you gotta respond with this. Or, oh, this is Docker Hub, and it it has it expects this series of of events to happen. So we've been you know, we've had people from, you know, GitHub registry come work with us in the community to make sure their implementation works with containerd, which they just got the GitHub container registry working well with

1:05:54 containerd in the last month. So it's a growing area, I guess, the simplest way to put it. If someone's really stuck, please come to the repo, open an issue because there are people smarter than me about kinda how the fallback and mirroring configs work. And then finally, hopefully, our documentation keeps improving that you can, you know, say, hey. Pull through cache. Where's the doc on that? And how do I get it set up? And, hopefully, we'll have a place that gives you the lowdown on on making that work. Awesome. Thank you very much. We are all

1:06:30 out of questions. I just wanna say thank you very much for taking the time out of your day to join me. This has been really helpful for me to help my understanding of containerd. I'm sure it's gonna be very helpful for loads of other people. So thank you very much, Phil. Yeah. Awesome. Thanks for having me. No problem. Alright. Have a nice day, and I'll speak to you soon. Thanks. Alright. Bye now.