# Rawkode Academy — Full Article Corpus > Hands-on cloud native education: videos, articles, courses, and live shows for engineers building and running production Kubernetes and infrastructure. # apko + melange: Distroless OCI Images Your Scanner Can Actually Read > A working tutorial for apko and melange. Package a binary with melange, assemble it into a signed distroless image with apko, and ship it with an SBOM Trivy can actually parse. - Canonical: https://rawkode.academy/read/apko-melange-tutorial - Published: 2026-05-19 - Authors: David Flanagan - Technologies: apko, melange, sigstore Your container scanner is lying to you. The scanner isn't at fault. Trivy, Grype, Snyk, every CVE feed on Earth, they all identify software the same way: they read a package manager's database. `apt`, `apk`, `dpkg`, `rpm`, the language-level lockfiles. If your image is built with a `Dockerfile` that does `COPY ./my-binary /usr/local/bin/` and then `RUN curl | sh` for the SDK, the scanner sees an Alpine base layer it can read, and then a pile of files it can't attribute to anything. The CVE report comes back clean. Nothing is clean. The scanner just doesn't have a name for what's in the image. This is the problem [apko](/technology/apko) and [melange](/technology/melange) were built to fix. Every byte in the final image arrives through a package manager, with a name, a version, a checksum, and a CVE feed entry. Distroless by construction. Reproducible by design. Signed by default. The whole pipeline is two YAML files and two CLI invocations, and you can run it on your laptop right now. ## See It For Yourself ```bash # install melange and apko (macOS shown; Linux ships in most package managers) brew install melange apko # generate the apk-signing keypair melange keygen # build a Wolfi-based "hello, world" apk and a 2 MB OCI image melange build melange.yaml --arch amd64 --signing-key melange.rsa --runner docker apko build apko.yaml hello:latest hello.tar --keyring-append ./melange.rsa.pub --arch amd64 docker load < hello.tar docker run --rm hello:latest # > Hello, world! ``` We'll build the two YAML files below. There are only forty lines between you and a signed, distroless, scanner-visible image. ## Why Dockerfiles Lie to Your Scanner A scanner finding zero CVEs in a `Dockerfile`-built image is, statistically, more alarming than finding fifty. Fifty means the scanner can read what's in there. Zero means it can't. `Dockerfile` lets you do anything, which is the problem. A `RUN curl https://example.com/install.sh | sh` step pulls a binary that the package manager database doesn't know about. The scanner sees the layer hash, opens it, finds a file, has no idea what it is. It can't tell you the file is a vulnerable `libcurl`, or a sketchy `kubectl` plugin, or a backdoored Helm chart. It moves on. The fix is to build images where every file came through a package manager that records what it installed and where. That means no `RUN curl | sh`. No `COPY` from anywhere except controlled sources. No `apt-get install` running against a moving target. The image format has to enforce these constraints, not the build script. That's what apko does. There is no `RUN` step in apko. No `COPY` from the host. No shell available during the build. apko takes a YAML config that lists `apk` packages and produces an OCI tarball. The set of installable packages is the set of packages in your declared repositories. The scanner sees the apk database in the final image and gets a 100% match against every byte. When you need software that isn't packaged yet, that's melange's job. ## Two Tools, One Job apko is the assembler. melange is the package builder. They split the work the way `apt` and `dpkg-buildpackage` split the Debian world, and for the same reasons. [apko](/technology/apko) takes a list of `apk` packages and produces a multi-arch OCI image with no shell, no package manager in the final layer, and a deterministic content hash. The whole tool is a thin shell over the `apk` resolver plus a tarball builder. It can't run arbitrary commands during the build because it never invokes a shell. It just resolves the dependency graph, fetches the packages, and writes the layers. [melange](/technology/melange) builds those `apk` packages from source. You give it a YAML that describes the source URL, the build steps, and the test command. melange runs those steps inside a clean sandbox (Docker, bubblewrap, QEMU, or Lima are all supported), produces a signed `.apk`, and puts it in `./packages//`. You then point apko at that local repository and apko treats your package the same as anything from Wolfi. The reason these are two binaries instead of one: every melange-built package is just an `apk`, indistinguishable from the thousands of packages in the [Wolfi](https://github.com/wolfi-dev/os) base distribution. You can publish your apks to a private apk repository, share them across projects, and forget which build system produced them. apko consumes apks. It doesn't care whose pipeline made them. ## Setting Up: Wolfi, Keys, Runners Install both tools: ```bash # macOS brew install melange apko # Linux with Go installed go install chainguard.dev/melange@latest go install chainguard.dev/apko@latest # Or pull pre-built containers docker run --rm cgr.dev/chainguard/melange version docker run --rm cgr.dev/chainguard/apko version ``` melange signs every `.apk` it builds. Generate a keypair once, store the private key somewhere safe (a CI secret manager for production, your dotfiles for laptop work): ```bash melange keygen # > generating keypair with a 4096 bit prime, please wait... # > wrote private key to melange.rsa # > wrote public key to melange.rsa.pub ``` melange runs the build steps inside a sandbox. Pick a runner based on what's available: - `--runner docker` is the easiest path on macOS and most laptops. - `--runner bubblewrap` is the lightest option on Linux CI. No daemon, no privileged container. - `--runner qemu` uses QEMU user-mode emulation to build arm64 packages natively on an amd64 host, without a cross-compiler. - `--runner lima` is the macOS-native Linux VM option if you don't want Docker Desktop. If you're not sure, start with `docker`. The build is hermetic regardless of the runner. ## Building a Package with melange Let's package GNU `hello`. It's small, it builds with autotools, and there's no point packaging anything you already use until you've seen the shape of a working config. Save this as `melange.yaml`: ```yaml package: name: hello version: 2.12.3 epoch: 0 description: "the GNU hello world program" copyright: - license: GPL-3.0-or-later environment: contents: repositories: - https://packages.wolfi.dev/os keyring: - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub packages: - build-base - busybox - ca-certificates-bundle pipeline: - uses: fetch with: uri: https://ftp.gnu.org/gnu/hello/hello-${{package.version}}.tar.gz expected-sha256: 0d5f60154382fee10b114a1c34e785d8b1f492073ae2d3a6f7b147687b366aa0 - uses: autoconf/configure - uses: autoconf/make - uses: autoconf/make-install - uses: strip test: pipeline: - runs: hello --version ``` Three blocks worth understanding: **`package`** is the metadata that ends up in the apk database. `name` and `version` are the identity. `epoch` is the build-of-build number, you bump it when the source is unchanged but the recipe changed. `copyright.license` is read by SBOM generators. **`environment`** is the build sandbox. apk packages, repositories, and keys go here. `build-base` is Wolfi's equivalent of Debian's `build-essential`. `busybox` gives us a usable shell inside the sandbox. `ca-certificates-bundle` lets the fetch step talk to HTTPS. **`pipeline`** is the sequence of build steps. Each step is a named "use" of a melange built-in pipeline plus its arguments. `fetch` downloads the tarball and checks the sha256. `autoconf/configure`, `autoconf/make`, `autoconf/make-install` run the conventional autotools chain. `strip` removes debug symbols from the output binaries. Build it: ```bash melange build melange.yaml \ --arch amd64,arm64 \ --signing-key melange.rsa \ --runner docker ``` If everything worked, you'll have `./packages/amd64/hello-2.12.1-r0.apk` and `./packages/arm64/hello-2.12.1-r0.apk`. melange also wrote an `APKINDEX.tar.gz` next to them, which is the metadata file apko's apk resolver needs to find your package. The `test:` block at the bottom is melange's smoke test. After the package is built, melange installs it into a fresh sandbox and runs your test pipeline. `hello --version` exits zero, the build is considered successful. If the binary is broken, the build fails before the apk is published. ## Assembling an Image with apko Now build the image. Save this as `apko.yaml`: ```yaml contents: keyring: - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - ./melange.rsa.pub repositories: - https://packages.wolfi.dev/os - "@local ./packages" packages: - wolfi-baselayout - hello@local entrypoint: command: /usr/bin/hello archs: - amd64 - arm64 annotations: org.opencontainers.image.source: https://github.com/your-org/hello-distroless ``` The interesting line is `hello@local`. apko's apk resolver supports tagged repositories: `@local ./packages` declares a local repository tagged `local`, and `hello@local` tells the solver to pull `hello` specifically from that repo. Without the tag, the solver would try to find `hello` in `packages.wolfi.dev/os` and fail. `wolfi-baselayout` is the absolute minimum filesystem skeleton: `/etc`, `/usr`, `/var`, the standard directories, plus `/etc/passwd` and `/etc/group` so the container has a `nobody` user. It's about 100 KB. Everything else in the image is `hello` and its runtime dependencies. `entrypoint.command` is what runs when the container starts. There is no shell to wrap it. The container is `hello` and nothing else. Build: ```bash apko build apko.yaml \ hello:latest hello.tar \ --keyring-append ./melange.rsa.pub \ --arch amd64,arm64 \ --sbom-path ./sboms ``` apko emits per-architecture OCI tarballs plus an OCI image index, all reproducible. Run the same command on a different machine with the same inputs and the layer digests match exactly. apko zeroes file timestamps and source-of-build metadata by default precisely to make this work. ## Loading, Inspecting, Verifying Load and run: ```bash docker load < hello.tar docker run --rm hello:latest-amd64 # > Hello, world! # Note: the multi-arch tarball loads `hello:latest-amd64` and # `hello:latest-arm64` as separate Docker tags (one per arch). If # you ran the single-arch quick-start instead, the tag is just # `hello:latest` (no suffix). # Image size: somewhere between 1 and 2 MB. docker image ls hello # > REPOSITORY TAG SIZE # > hello latest-amd64 1.43MB ``` Try to get a shell. You can't: ```bash docker run --rm --entrypoint /bin/sh hello:latest-amd64 # > docker: Error response from daemon: failed to create task: ... no such file or directory ``` There is no `/bin/sh`, no `/bin/bash`, no `busybox`, no nothing. An in-application RCE inside `hello` has nowhere to go. The classic post-exploitation playbook of "drop a webshell, spawn `bash`, pivot" relies on a shell existing in the container. apko removes that primitive. Inspect the SBOM: ```bash ls ./sboms/ # > sbom-amd64.spdx.json sbom-arm64.spdx.json jq '.packages[].name' ./sboms/sbom-amd64.spdx.json | sort -u # > "ca-certificates-bundle" # > "hello" # > "wolfi-baselayout" ``` That's the whole image, attributed. Trivy can read this. Grype can read this. Every byte in the layer has a name and a version. If `wolfi-baselayout` gets a CVE next week, your scanner will tell you before you push. ## Shipping It: Multi-Arch, Registry Publish, GitHub Actions For CI, switch from `apko build` (which writes a tarball) to `apko publish` (which pushes straight to a registry without a Docker daemon): ```bash apko publish apko.yaml ghcr.io/your-org/hello:latest \ --keyring-append ./melange.rsa.pub \ --arch amd64,arm64 \ --sbom-path ./sboms ``` `apko publish` writes both the per-arch images and an OCI index pointing at them, in one call. There's no `manifest create` dance. `--sbom-path` works the same way it does in `apko build`, so the SPDX SBOMs travel with the image whether you push locally or from CI. In GitHub Actions, [Chainguard publishes reusable workflows](https://github.com/chainguard-dev/actions) that wire melange and apko together with [Sigstore](/technology/sigstore) keyless OIDC signing: ```yaml jobs: build: runs-on: ubuntu-latest permissions: id-token: write # for cosign keyless packages: write # for ghcr.io steps: - uses: actions/checkout@v4 - uses: chainguard-dev/actions/melange-build@main with: config: melange.yaml archs: amd64,arm64 sign-with-temporary-key: true - uses: chainguard-images/actions/apko-publish@main with: config: apko.yaml tag: ghcr.io/${{ github.repository }}:${{ github.sha }} archs: amd64,arm64 keyring-append: melange.rsa.pub sbom-path: ./sboms - uses: sigstore/cosign-installer@v3 - run: cosign sign --yes ghcr.io/${{ github.repository }}:${{ github.sha }} ``` `sign-with-temporary-key: true` is the keyless equivalent for melange: it mints an ephemeral signing key per CI run, signs the apks with it, and discards the private key. The public key travels with the SBOM. `cosign sign --yes` over the resulting image gets a Fulcio-issued certificate tied to the workflow's OIDC identity. No long-lived key sits in a GitHub secret. You can verify the signature anywhere [cosign](https://github.com/sigstore/cosign) runs: ```bash cosign verify ghcr.io/your-org/hello:latest \ --certificate-identity-regexp 'https://github.com/your-org/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` If the binary on the registry was built by a workflow in your org, you get a green check. If anyone replaces it, the signature breaks. ## Where This Fits in 2026 This stack is the spiritual successor to Google's [Distroless](https://github.com/GoogleContainerTools/distroless) project. Distroless v1 required ~300 lines of Bazel per image, with every package, version, and dependency pinned by hand. apko delegates that work to the `apk` solver, so you write a dozen lines of YAML instead. The resulting image is smaller (Distroless `static` is ~2 MB; an apko `wolfi-baselayout` image with a small binary is ~1.5 MB), reproducible by default, and built with the same toolchain Chainguard uses to ship its own commercial image catalog. Reach for apko when: - You ship Go, Rust, or C++ binaries that are already statically linked or have a small set of runtime dependencies. - You have a security or compliance team that wants SBOMs and signed images, and you don't want to bolt those on after the fact. - Your scanner reports are currently full of "unknown" rows and you'd like that to stop. Don't reach for apko when: - Your build genuinely needs `RUN` steps (compile-time secrets, pre-baked database state, anything that won't fit a melange pipeline). Build a melange package instead and feed it to apko. - You ship Go-only services and don't need image-level package management. [ko](/technology/ko) is the lighter tool for that case: it skips the apk layer entirely and just packs a Go binary. - You're committed to a base image from a vendor who doesn't ship Wolfi packages and you can't move off it. ## Frequently asked questions ### Do I need to use Wolfi, or can I build apko images on Alpine? You can point apko at any apk repository, so plain Alpine still works for legacy or compatibility reasons. Chainguard moved its own images to Wolfi because Wolfi is glibc-based, ships only modern package versions, and is curated specifically for container use. For new projects in 2026, prefer Wolfi unless you have a hard musl or Alpine-tooling dependency. ### How is this different from a multi-stage Dockerfile with a scratch final image? A scratch image hides everything from your scanner. Trivy, Grype, and Snyk identify software by reading package manager databases, and a scratch image has none. apko produces an image whose entire contents come from apk packages, so every byte has a name, a version, and a CVE feed entry. You also get a signed SPDX SBOM at build time rather than reconstructing one after the fact. ### Can apko replace my Dockerfile entirely? Only if everything you need exists as an apk package. apko has no `RUN`, no `COPY` from host, no arbitrary shell. That's the point: it guarantees reproducibility and scanner visibility by refusing to do those things. If you need `RUN` steps, that's melange's job: melange builds the apk, apko composes the image, and the two halves cover what a Dockerfile used to do. ### How do I sign the final OCI image? melange signs the apk packages with an RSA keypair. To sign the resulting OCI image, use cosign, ideally with keyless OIDC signing in CI so no long-lived key sits in a GitHub secret. The `chainguard-dev` GitHub Actions wire this together: `melange-build` produces signed apks, `apko-publish` builds and pushes the image, and `cosign sign` attests it to the Sigstore Rekor transparency log. ## What to do now 1. Run the `hello` example end-to-end on your laptop. It takes ten minutes and the muscle memory matters. 2. Pick one service you ship today. Look at its current Dockerfile. Count the `RUN` lines. Each one is a place your scanner is currently guessing. 3. Try replacing it with an apko config. If you need a `RUN` step, that's a melange package. Build it. 4. Wire up cosign keyless signing in CI. The biggest security win isn't the distroless image, it's having a verifiable chain of custody from source commit to running container. The whole pipeline is two YAML files, four CLI commands, and a couple of hours of work to convert your first service. The payoff is an image that your scanner can actually read, end to end, every release. --- # cgroups: From Chaos to Control > Learn Linux cgroups v2 through Docker and Kubernetes examples: memory.max, pids.max, cgroup.subtree_control, OOM kills, and resource-limit debugging. - Canonical: https://rawkode.academy/read/cgroups-from-chaos-to-control - Published: 2026-05-18 - Authors: David Flanagan - Technologies: kubernetes Everything in Linux is a file ... and every Linux container's resource limits are no different. They're just files that live in a special directory. Inside that directory are files with names like `memory.max` and `cpu.weight`. When the kernel decides your container has used too much RAM, it isn't consulting a daemon or asking Kubernetes. It's reading a number out of a file you could have written yourself with `echo`. That's the whole mechanism. The story of how Linux got here, and the decade-long detour it took along the way, is the story of cgroups. --- ## See It For Yourself If you're on a Linux machine running a modern distribution, try this: ```bash ls /sys/fs/cgroup/ ``` You'll see something like: ``` cgroup.controllers cpu.stat memory.pressure cgroup.max.depth cpuset.cpus memory.stat cgroup.procs io.pressure pids.current cgroup.subtree_control memory.current pids.max cpu.pressure memory.max system.slice/ user.slice/ ``` This is a kernel pseudo-filesystem mounted under `/sys`. On a cgroupsv2 host, `/sys/fs/cgroup` itself is `cgroup2fs`, not ordinary disk-backed storage. Reading a file there asks the kernel a question. Writing to one issues a command. The kernel responds by changing its own state. There is no daemon in the middle. `memory.max` is the memory limit. `cpu.pressure` reports whether tasks are stalling on CPU. `system.slice/` and `user.slice/` are sub-folders: child groups, with their own limits inside. --- ## What a Cgroup Actually Is A cgroup (short for "control group") is a Linux kernel mechanism for organizing processes into named groups and applying resource constraints to those groups. That's the entire definition. Membership lives in a file: `cgroup.procs` contains the PIDs of processes in the group. To move a process in, you write its PID. To take it out, write that PID to another group's `cgroup.procs`. The kernel handles the bookkeeping. The mental model is this: **a cgroup is a directory, process membership is a file entry, and every resource limit is a knob you turn by writing to a file.** The kernel reads these files and enforces the constraints. If a process in a cgroup pushes memory usage toward `memory.max`, the kernel first tries to reclaim pages from that cgroup; if usage crosses `memory.high` along the way, allocations are throttled with backpressure; if reclaim can't keep up and usage hits `memory.max`, the kernel OOM-kills a process inside the cgroup. No daemon, no userspace agent. It's the kernel itself doing the enforcement. This design is elegant. What went wrong is what always goes wrong in systems that grow organically: nobody planned for containers. --- ## Try It Yourself The cleanest way to *feel* how cgroups work is to create one, cap it, and watch the kernel enforce the cap. Do this on a disposable Linux VM or lab host, not a production node. Any cgroupsv2 system will do. First, make sure the kernel will expose the controllers we want in our new cgroup. On v2, a child cgroup only sees a controller if its parent has enabled it in `cgroup.subtree_control`: ```bash # Should list at least: memory pids (and probably cpu, io, cpuset) cat /sys/fs/cgroup/cgroup.subtree_control # If memory or pids is missing, enable them at the root: echo "+memory +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control ``` Most modern distros enable these by default, but managed nodes and minimal images sometimes don't. With that out of the way: ```bash # Create the cgroup (it's literally a directory) sudo mkdir /sys/fs/cgroup/demo # Cap memory at 50 MiB, swap at 0 (otherwise overflow spills to swap # on hosts where swap is enabled, masking the kill) echo $((50 * 1024 * 1024)) | sudo tee /sys/fs/cgroup/demo/memory.max echo 0 | sudo tee /sys/fs/cgroup/demo/memory.swap.max # Run a memory hog inside the cgroup. The subshell joins the cgroup # by writing its PID to cgroup.procs, then execs python which tries # to allocate 200 MiB. sudo sh -c ' echo $$ > /sys/fs/cgroup/demo/cgroup.procs exec python3 -c "x = b\"A\" * (200 * 1024 * 1024); print(\"survived\")" ' ``` You'll see `Killed` rather than `survived`. The kill is scoped to the cgroup we just made. On hosts where you can read `dmesg`: ```bash sudo dmesg | tail -n 20 # ... Memory cgroup out of memory: Killed process 12345 (python3) ... # ... oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/demo,... ``` On hosts where `dmesg` is restricted (managed nodes, hardened distros), read the cgroup's own counter instead: ```bash cat /sys/fs/cgroup/demo/memory.events # low 0 # high 0 # max # times memory.max was hit # oom # times the cgroup went OOM # oom_kill # processes the kernel killed ``` No daemon was involved. The kernel read `memory.max` (50 MiB), watched `memory.current` climb past it, and pulled the trigger. Clean up the cgroup once it has no live processes (an empty cgroup is the only kind `rmdir` will remove): ```bash sudo rmdir /sys/fs/cgroup/demo ``` For real workloads on a systemd host, prefer `systemd-run` with unit properties (`--property=MemoryMax=50M`, etc.) over editing `/sys/fs/cgroup/` by hand. systemd owns the cgroup tree, and writing directly to it from outside breaks its accounting. The raw interface above is the right shape for *learning* what the kernel is doing; it's the wrong shape for production resource control. Three primitives. A directory, a file write, a PID. That's the kernel interface Linux container runtimes are built on. --- ## Now Block a Fork Bomb The memory demo killed a process after it allocated too much. The `pids` controller is more interesting: it refuses to *create* the process at all. Same files, same write-a-PID-and-go pattern, but the kernel says no at fork time rather than reclaim time. ```bash # sudo doesn't apply to shell redirection, so `sudo echo X > /sys/fs/...` # writes as your user and fails. Drop into a real root shell instead. sudo -i # Create the cgroup and cap it at 10 processes mkdir /sys/fs/cgroup/fork-demo echo 10 > /sys/fs/cgroup/fork-demo/pids.max # Move the current shell into it echo $$ > /sys/fs/cgroup/fork-demo/cgroup.procs cat /proc/self/cgroup # 0::/fork-demo ``` Now spawn background `sleep` processes in a loop and let the counter climb to the wall: ```bash i=0 while true; do sleep 60 & i=$((i+1)) echo "spawned #$i → PID $! (pids.current=$(cat /sys/fs/cgroup/fork-demo/pids.current))" sleep 0.2 done ``` ``` spawned #1 → PID 367 (pids.current=3) spawned #2 → PID 370 (pids.current=4) spawned #3 → PID 373 (pids.current=5) spawned #4 → PID 376 (pids.current=6) spawned #5 → PID 379 (pids.current=7) spawned #6 → PID 382 (pids.current=8) spawned #7 → PID 385 (pids.current=9) spawned #8 → PID 388 (pids.current=10) bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable bash: fork: retry: Resource temporarily unavailable ``` The eighth `sleep` pushes `pids.current` to exactly `pids.max=10`. (The shell counts as one process, and the `cat` inside the command substitution briefly shows up as another, which is why the visible count starts at 3.) Every subsequent `fork()` returns `EAGAIN`, bash prints the failure, and the cgroup is sealed. Ctrl+C to stop. The kernel keeps a running tally of denials: ```bash cat /sys/fs/cgroup/fork-demo/pids.events # max 4823 ``` That's how many `fork()` calls the kernel refused since the cgroup was created. Every denied `fork()` is one the rest of the system never had to deal with. Point a real fork bomb at this cgroup and the counter climbs into the millions while the host stays responsive, but I'm not going to print the bomb here, because copy-pasting one into the wrong shell is exactly how lab hosts become bricks. The `sleep` loop above already proves the mechanism: the kernel returns `EAGAIN` to `fork()` the moment `pids.current` hits `pids.max`, and it will do that whether the caller is a polite shell loop or a self-replicating function. Cleanup once the loop has exited. The root shell still belongs to `fork-demo`, and the kernel won't let us `rmdir` a cgroup with live members, so exit the shell first and remove the directory from the parent shell that spawned it: ```bash kill $(jobs -p) # reap the sleeps from inside the root shell exit # drop out of the root shell; its PID leaves the cgroup with it sudo rmdir /sys/fs/cgroup/fork-demo ``` Two cgroups, two different `.max` files, two completely different kinds of refusal: `memory.max` reclaims-then-kills, `pids.max` returns `EAGAIN` to `fork()`. The kernel exposes both behaviours through identical-looking interfaces, which is exactly the point of cgroups as an abstraction. --- ## Controllers: Who Fills the Folder A cgroup is a folder. But who fills it with files like `memory.max` and `cpu.weight`? The kernel does, but not as a single monolithic block. Each resource is governed by a **controller**: a piece of kernel code dedicated to one resource type. The `memory` controller watches RAM, exposes `memory.max`, `memory.current`, `memory.pressure`, and enforces the limits. The `cpu` controller weights and quota-caps CPU time. The `io` controller throttles block I/O. The `pids` controller caps the number of processes. Most resource-control files in a cgroup folder belong to one controller. The naming convention makes this explicit: `.`. `memory.current`, `cpu.stat`, `io.max`. Core cgroup files use the `cgroup.*` prefix instead. The set of controllers available on your system lives in one file: ```bash cat /sys/fs/cgroup/cgroup.controllers # cpuset cpu io memory pids ``` Five controllers on this system. Today. It used to be many controllers spread across many possible trees, and that's how the trouble started. --- ## Where This Came From In 2006, Paul Menage and Rohit Seth [proposed adding "process containers"](https://lkml.org/lkml/2006/9/14/370) to the Linux kernel: a way to group processes and attach resource controllers to those groups. The name was quickly changed to avoid confusion with the *other* kind of container that was starting to gain traction. Solaris Zones, Linux-VServer, and OpenVZ all already used "container" to mean a fully isolated user-space (own filesystem, network, process tree, users), and this new patch did none of that. It just grouped processes and counted their resource use. By the time the patch merged into [Linux 2.6.24](https://kernelnewbies.org/Linux_2_6_24) in January 2008, it was called **Task Control Groups** (TCG). The release notes describe it as a framework that can "track and group processes into arbitrary 'cgroups' and assign arbitrary state to those groups, in order to control its behaviour", with other subsystems hooking in to provide new attributes. The "Task" prefix quietly fell out of use over the next few years; everyone settled on the short form: control groups, or **cgroups**. From there, controllers were added one by one, year after year: By 2013, more than a dozen controllers existed. The original design had made a decision that seemed perfectly reasonable at the time: **controllers could live in independent trees.** CPU could get one tree. Memory could get another. Block I/O could get another. The promise was flexibility: you could organize processes differently for different resources. In practice, almost nobody used that flexibility. And the cost would dominate the next decade. --- ## Hierarchies: Cgroups Inside Cgroups Cgroups nest. A cgroup can contain child cgroups, and those children can have their own children. The result is a tree. The kernel calls it a **hierarchy**. Children inherit constraints from their parents. If `system.slice/` caps memory at 8 GiB, no combination of `nginx.service/` plus `postgres.service/` may exceed 8 GiB together, even if neither child sets a limit of its own. Limits cascade down; usage rolls up. One tree, one hierarchy. Hold on to that picture. In cgroupsv1, there wasn't *one* tree. There could be many. --- ## cgroupsv1: The Wild West You're debugging a container that keeps getting OOM killed. You SSH into the node, find the PID, and ask the kernel which cgroups the process belongs to. That information lives in a per-process file under `/proc`, the other kernel pseudo-filesystem, where each running process has its own directory: ```bash cat /proc/1234/cgroup ``` And you get... this: ``` 12:blkio:/docker/a1b2c3d4e5f6 11:devices:/docker/a1b2c3d4e5f6 10:memory:/docker/a1b2c3d4e5f6 9:cpuacct:/docker/a1b2c3d4e5f6 8:cpu:/docker/a1b2c3d4e5f6 7:cpuset:/docker/a1b2c3d4e5f6 6:freezer:/docker/a1b2c3d4e5f6 5:net_cls,net_prio:/docker/a1b2c3d4e5f6 4:perf_event:/docker/a1b2c3d4e5f6 3:pids:/docker/a1b2c3d4e5f6 2:hugetlb:/docker/a1b2c3d4e5f6 1:name=systemd:/docker/a1b2c3d4e5f6 ``` Twelve lines. The same container ID, repeated across many hierarchies. The exact list varied by distro and runtime because related controllers could be co-mounted, but the important part is the same: one process had multiple cgroup memberships. (A few of those names won't survive into v2 in the same form. `freezer`, for instance, was v1's way of pausing every process in a cgroup at once — useful for checkpoint/restore and as the prerequisite step to safely killing a process tree. v2 replaces it with `cgroup.freeze` and `cgroup.kill` on the unified hierarchy.) This is cgroupsv1. And it's a mess. The process is in the `memory` tree, *and* the `cpu` tree, *and* the `blkio` tree, *and* several others. Each hierarchy has its own mount point under `/sys/fs/cgroup/`: ```bash ls /sys/fs/cgroup/ # cpu/ cpuacct/ cpuset/ memory/ blkio/ devices/ # freezer/ net_cls/ net_prio/ perf_event/ hugetlb/ pids/ ``` Many directories, many mount points, many independent trees. Here's what that fragmentation actually looks like: With a v1 cgroupfs layout, when a container runtime creates a container, it has to: 1. Create a directory in `/sys/fs/cgroup/cpu/docker//` 2. Create a directory in `/sys/fs/cgroup/memory/docker//` 3. Create a directory in `/sys/fs/cgroup/blkio/docker//` 4. ... repeat for every mounted controller hierarchy 5. Write the process PID to `cgroup.procs` in each directory 6. Set the appropriate limits in each directory Compare that to cgroupsv2, where the same process gets one line: ``` 0::/kubepods.slice/kubepods-pod1a2b3c.slice/cri-containerd-a1b2c3d4e5f6.scope ``` It's worth pausing here, because the original v1 design was textbook good engineering. Independent trees per controller is exactly how a senior engineer would propose this from scratch today: small, decoupled subsystems, each responsible for one resource, free to evolve on its own timeline. Every Cloud Native principle we now take for granted points at that shape. The decoupling is a feature when the consumer is a sysadmin grouping arbitrary processes for arbitrary policies, limiting CPU on one grouping while throttling I/O on another. Containers turn that assumption inside out. A container is a *bundle* that wants every controller applied to the same set of processes, atomically, with one accounting view. The flexibility v1 offered became a tax: every controller wired up separately, every limit set separately, every monitoring query stitched back together by hand. v2 is the convergence that shape demanded. --- ## Why It Was a Mess The structural problem cascaded into operational pain. **Operations weren't atomic.** Moving a process between cgroups meant writing its PID multiple times, across multiple `cgroup.procs` files. There was no way to say "move this process across all controllers at once." If your runtime crashed between step 3 and step 4, the process might be limited on CPU and memory but not on block I/O. Container runtimes had to build their own retry-and-synchronization logic on top. **Interfaces were inconsistent.** CPU alone had two completely different paradigms. `cpu.shares` (values from 2 to 262144) gave you proportional weighting: a container with 1024 shares got twice the CPU time of one with 512, *but only when both were competing*. If you were alone, shares didn't matter. `cpu.cfs_quota_us` (paired with `cpu.cfs_period_us`) gave you absolute caps: quota 50000, period 100000 meant 50% of one core, hard. Two files, two mental models, same controller. **Accounting was a separate controller from control.** Want to know how much CPU your cgroup used? Check `cpuacct.usage`. A *different controller*, in a *different hierarchy*, with its own mount point. Want to limit it? Use `cpu.cfs_quota_us` over in the `cpu` controller. Nobody was happy about that decision. **Memory had two failed concepts.** `memory.soft_limit_in_bytes` was supposedly advisory, but the kernel's reclaim behavior around it was so unpredictable that most operators treated it as decorative. `memory.memsw.limit_in_bytes` capped memory plus swap *combined*. You couldn't set a swap limit independently, so you did arithmetic to figure out the effective swap allowance. **No unified view.** To answer "how much total resource is this container using?", monitoring tools had to read CPU from paths like `/sys/fs/cgroup/cpuacct/docker//cpuacct.usage`, memory from `/sys/fs/cgroup/memory/docker//memory.usage_in_bytes`, block I/O from `/sys/fs/cgroup/blkio/docker//blkio.throttle.io_service_bytes`, and stitch the answers together by correlating paths across hierarchies. cAdvisor, Prometheus node_exporter, and every other monitoring tool had to implement this stitching. It worked. It was fragile, slow, and ugly. (There were also smaller indignities. The `cpuset` controller's `cgroup.clone_children` flag, where child cgroups didn't inherit the parent's CPU and memory node assignments unless you flipped it on, broke expectations everywhere else. v2 dropped the flag entirely: children of a cpuset cgroup inherit their parent's `cpus` and `mems` by default, and that's that.) Tejun Heo saw the mess and [proposed a fix in early 2014](https://lkml.org/lkml/2014/3/13/503): a single unified hierarchy. That patchset would take two years to land and another four to stabilize. --- ## cgroupsv2: One Tree Tejun Heo's rewrite kept the conceptual simplicity of "cgroups are directories with resource knobs" and fixed the structural mistake: **there is exactly one hierarchy.** That decision ripples through every aspect of the design. ```bash # cgroupsv2: one mount point mount | grep cgroup # cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime) # One line in /proc/self/cgroup cat /proc/self/cgroup # 0::/user.slice/user-1000.slice/session-2.scope ``` Every controller lives on the same tree. When you move a process from one cgroup to another, it moves in all controllers atomically: one write to one `cgroup.procs` file. When you read resource usage, everything is in one place. ### The "No Internal Processes" Rule cgroupsv2 enforces a rule that confused people initially: **a non-root cgroup that distributes domain resources to child cgroups cannot also have processes directly in it.** Why? Resource accounting. If a parent cgroup has both processes and child cgroups, who gets charged for the parent's processes? Do they compete with the children? How do you account for their usage? In v1, the answer varied by controller and was often wrong. Exactly the inconsistent half-applied behavior we just saw. v2 eliminates the ambiguity where it matters. Once a non-root cgroup enables domain controllers in `cgroup.subtree_control`, processes need to live in leaf cgroups below it. A cgroup can still have both processes and children before it starts distributing those controllers, and the root cgroup is special-cased. Clean accounting, fewer controller-specific edge cases. ### Explicit Controller Delegation In v1, every controller was implicitly available everywhere. In v2, parents explicitly delegate controllers to their children: ```bash # See which controllers are available cat /sys/fs/cgroup/cgroup.controllers # cpuset cpu io memory pids # Enable cpu and memory for child cgroups echo "+cpu +memory" > /sys/fs/cgroup/cgroup.subtree_control # Now child cgroups can use cpu and memory controllers ``` This creates a clean permission model. A child cgroup can only use controllers its parent delegated. Container runtimes and systemd need exactly this when granting resource control to unprivileged processes (like rootless containers) without giving them access to controllers they shouldn't touch. ### Atomic Kill v2 added a feature that v1 never had a clean answer for: killing every process in a cgroup, atomically, with one write: ```bash echo 1 > /sys/fs/cgroup/some-group/cgroup.kill ``` The kernel walks the cgroup (and its descendants) and sends `SIGKILL` to every process in one pass. No racing the process tree, no chasing PIDs that fork while you're trying to enumerate them, no freezer-controller dance to pause the tree before you swing. Container runtimes use this to terminate a container cleanly; Kubernetes uses it indirectly through the runtime for pod shutdown. (Some workloads, like databases, JVMs, and game engines, need per-thread rather than per-process control. v2 supports this via `cgroup.type = threaded`, but it's a niche we won't dwell on here.) --- ## The Delta, Controller by Controller Here's exactly what changed at the file level. Skim these tables if you're migrating; the simulator below lets you feel how a v1 setting and its v2 equivalent translate to the actual file contents. ### CPU | cgroupsv1 | cgroupsv2 | What Changed | |-----------|-----------|--------------| | `cpu.shares` (2 to 262144) | `cpu.weight` (1 to 10000, default 100) | Rescaled range, saner defaults | | `cpu.cfs_quota_us` / `cpu.cfs_period_us` | `cpu.max` (`$QUOTA $PERIOD`) | One file instead of two | | `cpuacct.usage` (separate controller!) | `cpu.stat` | Accounting merged into cpu controller | | `cpuset.cpus` (separate hierarchy) | `cpuset.cpus` (same hierarchy) | No more separate mount point | The `cpu.weight` rescaling is a quality-of-life improvement. Instead of the bizarre 2 to 262144 range, you get 1 to 10000 with a default of 100. A container with weight 200 gets twice the CPU time of one with weight 100. Simple. `cpu.max` combines quota and period into a single file: `"50000 100000"` means 50 ms of CPU per 100 ms (50% of one core). Write `"max 100000"` for unlimited. No more reading two files and doing mental division to figure out the effective limit. ### Memory | cgroupsv1 | cgroupsv2 | What Changed | |-----------|-----------|--------------| | `memory.limit_in_bytes` | `memory.max` | Hard limit, cleaner name | | `memory.soft_limit_in_bytes` | `memory.high` | Actually works now; applies throttling backpressure | | `memory.memsw.limit_in_bytes` | `memory.swap.max` | Independent swap control | | `memory.usage_in_bytes` | `memory.current` | Cleaner name | | n/a | `memory.low` / `memory.min` | New: memory protection guarantees | | n/a | `memory.pressure` | New: PSI stall metrics | The big behavioural change is `memory.high`. In v1, `memory.soft_limit_in_bytes` was almost useless. The kernel's reclaim behaviour around it was unpredictable and slow. In v2, `memory.high` actively **throttles allocations and triggers direct reclaim** when the cgroup hits this threshold. Most workloads feel this as backpressure rather than a kill, which is exactly the point. It is not a kill-prevention guarantee, though: if a process allocates faster than the kernel can reclaim, `memory.max` (or the global OOM killer) is still in play. `memory.low` and `memory.min` are entirely new concepts: memory *protection*. `memory.min` is a hard reclaim barrier within its **effective** range: a child cannot protect more memory than its parents already protect, because the kernel caps the child's effective `memory.min` to whatever the ancestor chain allows. `memory.low` is best-effort protection on the same model; it shields the cgroup during system-wide reclaim, but under *local* cgroup reclaim (the cgroup is at its own `memory.high` or `memory.max`) the kernel will still reclaim from it. Together they let memory-sensitive workloads coexist with bursty ones on the same node without one starving the other. ### I/O | cgroupsv1 | cgroupsv2 | What Changed | |-----------|-----------|--------------| | `blkio.weight` | `io.weight` | Controller renamed entirely | | `blkio.throttle.read_bps_device` | `io.max` | Unified format per device | | n/a | `io.latency` | New: latency-targeted I/O control | | n/a | `io.pressure` | New: PSI stall metrics | The controller was renamed from `blkio` to `io`. The interface was cleaned up dramatically: instead of separate files for read bytes, write bytes, read IOPS, and write IOPS, `io.max` uses a unified per-device format: ``` 8:0 rbps=1048576 wbps=max riops=1000 wiops=max ``` Device `8:0`, read bandwidth limited to 1 MB/s, write bandwidth unlimited, read IOPS limited to 1000, write IOPS unlimited. One file, one line per device, all limits together. `io.latency` is new and powerful: instead of specifying throughput limits, you specify a **target latency** and the kernel throttles competing workloads to achieve it. This is much closer to what real applications actually care about. ### Cleaned-Up Naming v2 standardised the verbs across all controllers: - `max`: hard limits (exceed triggers enforcement) - `high`: soft limits (exceed triggers backpressure) - `low` / `min`: protection (guaranteed minimums) - `current` / `stat`: usage and metrics Memory controller values are in bytes, not pages, so no more multiplying by the kernel's page size to know what you wrote. `"max"` as a string means unlimited. No more `_in_bytes` suffixes everywhere. Consistent naming: `.` end to end. --- ## PSI: The Feature That Justifies the Rewrite If there's one feature that pays for cgroupsv2's existence on its own, it's PSI, or Pressure Stall Information. PSI itself isn't new to cgroupsv2. It landed in Linux 4.20 (2018) and has always exposed system-wide files under `/proc/pressure/cpu`, `/proc/pressure/memory`, and `/proc/pressure/io`. What v2 adds is the same signal **per cgroup**: every cgroupv2 directory gets its own `cpu.pressure`, `memory.pressure`, and `io.pressure` file, scoped to the processes inside it. That's the missing piece v1 simply couldn't produce. These tell you how much time tasks in the cgroup are **stalling**: waiting for a resource instead of doing useful work. ```bash cat /sys/fs/cgroup/kubepods.slice/memory.pressure # some avg10=2.50 avg60=1.30 avg300=0.85 total=48291738 # full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ``` `some` means at least one task is stalling. `full` means all tasks are stalling. The `avg10`, `avg60`, `avg300` numbers are percentages over 10-, 60-, and 300-second windows. This is transformative for monitoring. Instead of waiting for an OOM kill to tell you something went wrong (reactive), you can watch pressure metrics climb and take action before it happens (proactive). `some avg10=5.20` means tasks were stalled on memory 5.2% of the last ten seconds. A yellow flag. Scale horizontally before pressure hits 20%. Throttle ingestion before I/O pressure spikes. Combined with `memory.events` (which counts OOM kills, reclaim activity, and high-watermark hits), you get much better visibility into the kernel's reclaim, throttling, and OOM behaviour for a specific workload, without having to attribute system-wide signals back to a guilty cgroup yourself. `memory.events` isn't a one-off: most v2 controllers expose a `.events` file (`pids.events`, `cpu.stat` for usage and throttling counters, `io.stat`) on the same shape, so once you've learned to read one you've learned to read them all. --- ## Kubernetes Maps Policy Onto Cgroups Everything we've talked about so far is what Kubernetes builds on. When you set `resources.requests` and `resources.limits` in a pod spec, the kubelet and container runtime translate those settings into cgroup configuration. Nothing mystical. But before we look at the mapping, two pieces of vocabulary the kubelet uses heavily: **slices** and **scopes**. On most modern Kubernetes Linux nodes, systemd is the host's cgroup manager: it creates, names, and manages cgroup directories on the kernel's behalf, and everything else (the kubelet, the container runtime) integrates with it rather than writing to `/sys/fs/cgroup/` directly. `.slice` and `.scope` are simply systemd's naming conventions for those cgroups. A `.slice` is a cgroup used to *organise* other cgroups (an internal node in the tree). A `.scope` is a cgroup whose processes were started outside the unit itself (a leaf node containing externally-launched PIDs, like the ones a container runtime hands to systemd). That's why cgroup paths on nodes using the systemd cgroup driver look like systemd named them. Because systemd did. ### How Kubernetes Lays Out the Cgroup Tree On a node using the systemd cgroup driver with cgroupsv2, the Kubernetes cgroup layout looks like this: ``` /sys/fs/cgroup/ └── kubepods.slice/ ├── kubepods-pod.slice/ # Guaranteed pods │ └── cri-containerd-.scope ├── kubepods-burstable.slice/ # Burstable pods │ └── kubepods-burstable-pod.slice/ │ └── cri-containerd-.scope └── kubepods-besteffort.slice/ # BestEffort pods └── kubepods-besteffort-pod.slice/ └── cri-containerd-.scope ``` The kubelet manages the pod and QoS cgroups, the container runtime manages the container cgroups, and systemd is the host cgroup manager they integrate with. ### QoS Classes Shape Cgroups and OOM Priority Kubernetes Quality of Service classes shape the cgroup layout and the kernel OOM score adjustment. Node-pressure eviction is a little more nuanced: the kubelet sorts pods first by whether their usage exceeds their requests (over-request pods go before under-request ones), then by Pod priority, then by how far usage has climbed past the request. QoS is still the useful mental model: BestEffort tends to go first, Guaranteed tends to go last. **Guaranteed** pods have CPU and memory requests equal to limits for every container (or the equivalent pod-level resources, Kubernetes 1.32+). They get their own slice directly under `kubepods.slice`. The kubelet sets their OOM score adjustment to -997, making them the last to be killed by the kernel OOM killer. **Burstable** pods have at least one CPU or memory request or limit, but do not meet the Guaranteed rules. They go under `kubepods-burstable.slice`. Their OOM score is adjusted based on the ratio of memory request to node capacity: more request means lower score means killed later. **BestEffort** pods have no CPU or memory requests or limits at either container or pod level. They go under `kubepods-besteffort.slice` with an OOM score adjustment of 1000. When the node runs out of memory, these are the easiest targets. With the optional MemoryQoS feature on cgroupsv2 nodes, Kubernetes can also write `memory.min`, `memory.low`, and `memory.high`. As of Kubernetes 1.36, tiered memory reservation is still alpha and opt-in: with `memoryReservationPolicy: TieredReservation`, Guaranteed pods get `memory.min` protection and Burstable pods get `memory.low` protection. With the default reservation policy, requests do not automatically become `memory.min` or `memory.low`. ### From Pod Spec to Cgroup Files The default translation is: ```yaml resources: requests: cpu: "500m" # influences CPU shares / cpu.weight memory: "256Mi" # scheduler placement and eviction accounting limits: cpu: "500m" # becomes cpu.max = "50000 100000" (50% of one core) memory: "256Mi" # becomes memory.max = 268435456 (bytes) ``` CPU requests become proportional CPU scheduling weight. Kubernetes still starts from the v1-era `cpu.shares` calculation, and the OCI runtime maps those shares to cgroupsv2 `cpu.weight`. The exact number depends on the runtime version: | Request | Newer runc/crun `cpu.weight` | Older linear conversion | |---------|-----------------------------|-----------------------| | 100m | ~17 | ~4 | | 1000m | ~100 | ~39 | The Linux CPU scheduler uses that weight for proportional CPU time when cores are contended. CPU limits become `cpu.max`: 500m translates to a quota of 50000 µs per 100000 µs period, which is 50% of one core. Memory limits map directly to `memory.max` in bytes. Memory requests influence scheduling, eviction, and optional MemoryQoS protection when that feature is enabled and configured. ### The Cgroup Driver Choice The kubelet has two options for managing the cgroup tree. **cgroupfs driver:** the kubelet directly creates directories and writes to cgroup files. Simple, but it creates a split-brain situation: systemd (which manages system services and their cgroups) doesn't know about the kubelet's cgroups, and the kubelet doesn't know about systemd's. Under memory pressure this can lead to instability because systemd and the kubelet make independent, potentially conflicting resource decisions. **systemd driver:** the kubelet and runtime integrate with systemd, often through transient units, and systemd manages the cgroup tree. Single source of truth. Both the kubelet and system services are visible in the same hierarchy, managed by the same process. The paths look noticeably different. With the cgroupfs driver, the kubelet writes flat, kubelet-shaped directories: ``` /sys/fs/cgroup/kubepods/ ├── pod/ │ └── / ├── burstable/ │ └── pod/ │ └── / └── besteffort/ └── pod/ └── / ``` With the systemd driver, the same logical tree comes out with systemd's `.slice`/`.scope` suffixes baked into every directory name — that's the layout shown earlier under "How Kubernetes Lays Out the Cgroup Tree." Same structure, different names; tools that grep by path need to handle both. For kubeadm clusters, Kubernetes 1.22 made `systemd` the recommended driver and switched kubeadm's own default to it (we all remember this migration, and not fondly; right?). The kubelet's standalone default is still `cgroupfs`, but Kubernetes recommends `systemd` whenever systemd is the init system, and requires the kubelet and runtime to use the systemd cgroup driver for cgroupsv2 support. If you're still on cgroupfs with systemd, you're running against the grain. ### How Monitoring Reads Cgroups When you run `kubectl top pods`, here's the chain: 1. `kubectl top` queries the metrics API. 2. metrics-server scrapes the kubelet's `/metrics/resource` endpoint. 3. The kubelet reads cgroup stats such as `cpu.stat` and `memory.current` from each container's cgroup directory. 4. cAdvisor (embedded in the kubelet) continuously scrapes the cgroup filesystem for detailed metrics. On cgroupsv2, cAdvisor reads from a single unified hierarchy. On v1, it had to stitch together data from multiple mount points and correlate by container ID or cgroup path. The v2 path is faster, simpler, and has fewer failure modes. --- ## Operating It **Which version am I running?** ```bash # Linux only. Returns "cgroup2fs" for v2, "tmpfs" for v1. stat -fc %T /sys/fs/cgroup/ # Or look for the unified hierarchy marker test -f /sys/fs/cgroup/cgroup.controllers && echo "v2" || echo "v1" ``` As of 2026, cgroupsv2 is the default on the mainstream modern Linux node images: Ubuntu 22.04+ (21.10+ upstream), Fedora 31+, RHEL 9+, Debian 11+, Flatcar Container Linux, Talos Linux, and other current container-focused distributions. Managed Kubernetes has mostly moved too: GKE defaults to v2 for 1.26+ node pools, AKS for 1.25+, and new EKS managed node groups on Kubernetes 1.30+ default to AL2023, which uses cgroup v2. Existing clusters and pinned node images can still be on v1, so check before migrating workloads. ### Migration Gotchas - **Monitoring tools with hardcoded v1 paths.** Anything reading from `/sys/fs/cgroup/memory/docker/` is v1-specific and will break on v2. Most modern stacks handle both, but custom scripts and older agents may need updates. - **Container runtimes.** containerd 1.4+ and CRI-O 1.20+ have v2 support. Docker Engine 20.10+ works with v2 when using the systemd cgroup driver. runc 1.0.0+ has full v2 support; older versions are incomplete. - **JVM applications.** The JVM's cgroup-aware memory detection was rewritten for v2. JDKs older than 15, 11.0.16, or 8u372 do not detect v2 memory limits at all; they fall back to host memory and will quietly try to use far more memory than the cgroup allows. This is one of the most common "my Java app OOMs on new nodes" issues, and in 2026 there is no excuse for still being on those JVM versions — upgrade. - **Hybrid mode.** Booting with `systemd.unified_cgroup_hierarchy=0` (or older distro defaults) mounts the v2 unified hierarchy alongside the legacy v1 per-controller hierarchies, so processes show up in both trees at once. It works, but it's explicitly a transition mechanism, not a destination — accounting splits across two interfaces and tools have to guess which one is authoritative. Pick one and commit. ### PSI in Practice The PSI metrics from earlier aren't only useful for ad-hoc `cat` commands. Kubernetes added kubelet PSI metrics as an alpha feature in 1.33, promoted them to beta in 1.34, and made them stable and enabled by default in 1.36. On cgroupsv2 nodes with PSI enabled in the kernel, the kubelet exposes node-, pod-, and container-level pressure data through the Summary API and `/metrics/cadvisor`. ### eBPF One bonus from v2's unified hierarchy: BPF programs that attach to cgroup paths (for network filtering, socket operations, device policy) now have an unambiguous path to attach to. Cilium, Calico, and other CNI plugins that use eBPF for cgroup-aware policy get this for free. --- ## Where We're Going cgroupsv1 is legacy. The kernel's authoritative cgroup documentation is now the v2 interface, Kubernetes deprecated cgroup v1 in 1.35, and new feature work targets v2. v2's explicit controller delegation is also what makes rootless containers viable: systemd hands an unprivileged user their own subtree via a user-level `.slice`, with only the controllers the admin chose to delegate, and tools like Podman and rootless Docker drive resource limits inside it without ever touching root-owned cgroup files. What's ahead is incremental: better NUMA-aware memory placement, finer delegation primitives for those rootless trees. But the structural story is settled. Much like Kubernetes is YAML and controllers, all the way down; cgroups are directories and files on the surface, with kernel code doing the work of every write underneath. --- # Introducing the Technology Matrix > Your new compass for the Cloud Native landscape. Explore, filter, and learn about the technologies that power modern platforms. - Canonical: https://rawkode.academy/read/introducing-technology-matrix - Published: 2025-12-02 - Authors: David Flanagan - Technologies: kubernetes, docker I'm tired. Genuinely exhausted. Navigating the Cloud Native landscape has become absolutely maddening. We've all seen the Always Sunny meme, and honestly? It hits harder every year. The CNCF Landscape lists hundreds of projects. Hundreds. New tools pop up every week. Every conference brings another dozen "game-changers." And somehow, you're supposed to figure out what actually matters for your platform while also, you know, doing your actual job. I've spent years drowning in this chaos. Evaluating tools. Getting burned by hype. Watching projects I loved die slow deaths. And I kept thinking: there has to be a better way to track all this. To share what I've actually learned. So I built one. Welcome to the **Technology Matrix**. ## What is the Technology Matrix? The [Technology Matrix](/technology/matrix) is not another list. I'm so sick of lists. It's my **personal, opinionated tech radar**. A living map of how I actually navigate this mess of an ecosystem. Every technology in here has a story. When I first touched it. Why I rate it the way I do. Where I think it's going (and sometimes, where I think it deserves to go). Think of it as the CNCF Landscape if someone actually had opinions. If someone would just tell you "yeah, I used this in production and it was a nightmare" instead of neutral marketing speak. Today I'm releasing the first iteration. It's rough around the edges. I know. But the core is there, and I'll keep adding spicier takes and better features as I go. ## The Pipeline: A Journey Through Technology Here's the thing that frustrated me about every tech radar I've seen: they just label stuff "adopt" or "hold" like technology decisions are binary. They're not. My relationship with tools changes over time. Sometimes dramatically. So at the heart of the Matrix is what I call the **Pipeline**. Seven stages that represent where I actually am with each technology: | Stage | What It Means | |-------|---------------| | **Skip** | Not for me right now. Maybe it doesn't fit my use cases, maybe I tried it and walked away. No hard feelings. | | **Watch** | On my radar. I'm paying attention, reading release notes, lurking in the Discord. | | **Explore** | Worth a look. I'm actually trying this out, running tutorials, kicking the tires. | | **Learn** | I'm investing real time here. This technology earned my attention and I'm going deep. | | **Adopt** | Production-ready in my head. I'd confidently use this on a real project tomorrow. | | **Advocate** | I'm not just using it, I'm actively telling people about it. I believe in this one. | | **Graveyard** | The end of the road. Deprecated, abandoned, or it burned me badly enough that I'm done. Rest in peace. | Things move through this pipeline. Something in "Watch" today might be in "Adopt" next month. Or it might crater into the Graveyard. I've seen both happen more times than I can count. The Matrix captures that journey, and I'll annotate it with my thoughts as things evolve. ## The Story Behind Each Tech Click on any technology and you get the full picture. This is where it gets personal: - **Why**: My actual rationale for the rating. Not marketing copy. Real opinions from real experience, including the painful parts. - **Spicy Takes** 🌶️: The hot opinions. Things I might not say on a conference stage but you deserve to know. - **Makes Me Feel**: An emoji reaction. Sometimes you just need to know the vibe, you know? - **First Used / Last Used**: When did I actually start using this? Have I been running Kubernetes since 2016 or did I just discover this project last month? Context matters. This metadata turns a list into a story. You're not just seeing what I think. You're seeing *why* I think it, how I got here, and how that opinion has changed over the years. ## Advanced Explorer For the folks who want to really dig in, the [Advanced Explorer](/technology/matrix/advanced) is where things get interesting: - **Filter by Anything**: Pipeline stage, CNCF status, trajectory, confidence. Slice it however makes sense for you. - **Multiple Views**: See technologies as a grid by category, or explore other ways to organize the chaos. - **Preset Views**: Quick configs like "Pipeline View" (my journey) or "CNCF Maturity" (project lifecycle). - **Shareable URLs**: Found a useful filter combination? The URL updates as you explore. Share your exact view. The Advanced Explorer turns the Matrix from something you browse into something you interrogate. Ask it hard questions. ## Deep Content Integration The Matrix isn't a standalone database. It's wired into everything I create at Rawkode Academy. Every technology links to its dedicated page: - **Deep-dive Videos**: Tutorials, "Show & Tell" sessions, and "Klustered" episodes where I actually use the tool. - **Articles & Guides**: Written content to help you understand the concepts and trade-offs. - **Related Technologies**: See how tools fit together in the broader stack. If I've covered it, you'll find it. The Matrix is your index into the content library. ## What's Coming What you see today is Phase 1. Here's what's on my roadmap: ### Phase 2: New Perspectives - **Scatter Plot View**: A proper "Tech Radar" visualization. Plot technologies by stage vs trajectory to see where momentum is actually building. - **Timeline View**: See my journey chronologically. When did I adopt Kubernetes? When did I finally give up on Docker Swarm? The timeline tells the story. ### Phase 3: Deep Exploration - **Treemap View**: Drill down through categories hierarchically. See how the ecosystem is weighted at a glance. - **Sankey Diagrams**: Visualize the flow of technologies through pipeline stages over time. Watch the ecosystem evolve. The Matrix will grow as the ecosystem grows. New views, new filters, new ways to make sense of the noise. ## Why I Built It Look, I built this because I was frustrated. As someone who constantly evaluates tools, creates content, and advises teams on platform choices, I needed somewhere to put all of this. A structured way to capture what I know and share it without the corporate filter. The CNCF Landscape is comprehensive. I respect the work. But it's deliberately neutral. It won't tell you what's worth your time and what's a waste of it. I needed something opinionated. Something that says "I deployed this in production and here's the truth." My goal is simple: help you make better technology decisions by sharing my perspective honestly. Not as gospel. Not as the definitive answer. Just as one data point from someone who's been doing this for a while and has the scars to prove it. ## Start Exploring The Technology Matrix is live. Go poke around. - **[Visit the Matrix](/technology/matrix)**: See the full pipeline view with every technology I track. - **[Try the Advanced Explorer](/technology/matrix/advanced)**: Filter, slice, and interrogate the data your way. This is a living document. Technologies will move through the pipeline. New projects will appear. My opinions will change. That's the whole point. The Matrix captures the journey, not a frozen snapshot. Dive in. Disagree with me. Tell me I'm wrong. Tell me what I'm missing. Let's figure this out together. --- # Building a Rust Library for CUE: Architecture & Lessons > Inside cuengine, a Rust library wrapping Go's CUE evaluator via FFI. Memory safety, structured errors, and production-grade architecture. - Canonical: https://rawkode.academy/read/building-rust-cue-library - Published: 2025-11-27 - Authors: David Flanagan - Technologies: cue, rust I love shiny things, and when it comes to build tools, I'm always looking for a better way. That's what led me to create [cuenv](https://github.com/cuenv/cuenv), a modern toolchain built around CUE's powerful constraint-based system. There was just one problem: I'm building `cuenv` in Rust, and CUE's only mature evaluator is written in Go. So, what do you do when you want the best of both worlds? You build a bridge. This is the story of how I built **cuengine**, a high-performance, production-grade FFI library that lets Rust speak Go. We'll dive into the nitty-gritty of FFI, memory safety, and how to create a safe, ergonomic API on top of a completely different runtime. Let's get into it. ## The Core Problem: Two Runtimes, One Library If you haven't used it, CUE (Configure, Unify, Execute) is a fantastically powerful configuration language with a sophisticated type system. The official Go implementation is mature and battle-tested. But my build toolchain, `cuenv`, had to be in Rust. For me, the reasons were clear: - **Memory Safety**: Rust's ownership model wipes out entire classes of bugs. - **Performance**: I wanted zero-cost abstractions and predictable performance. - **Ecosystem**: The crate ecosystem for CLI tools and system integration is second to none. - **Cross-Compilation**: First-class support for targeting multiple platforms is a must. So the question was never *if* I should use Rust, but *how* to bring CUE along for the ride. ### Why Not Just Port CUE to Rust? Look, I'm not a masochist—at least not for this kind of punishment. Porting an entire language implementation is a multi-year odyssey of parser rewrites, type system reimplementation, and endless compatibility testing. Even if I pulled it off, I'd be chained to maintaining it against every upstream CUE change forever. No, thanks. I chose pragmatism: build a thin, safe FFI layer that uses the existing Go implementation while exposing a clean, native Rust API. ### What About libcue? The CUE team has been working on [libcue](https://github.com/cue-lang/libcue), an official C library for using CUE from C and C-like languages. It's a great initiative, but it didn't fit my needs for a few reasons: - **Low-level API**: libcue exposes primitives like `cue_compile_string` and `cue_unify` for working with CUE values directly. It's designed for fine-grained control, not high-level package evaluation. - **No package loading**: For `cuenv`, I need to load entire CUE packages from directories with full module resolution, imports, and registry support. libcue compiles strings and bytes—you'd have to build the package loading yourself. - **Still maturing**: The libcue README explicitly warns "expect constant churn and breakage." For a production tool, I needed something stable today. - **Different abstraction level**: I wanted a Rust-idiomatic API with caching, retry logic, structured errors, and tracing built in—not a thin C binding. That said, libcue is worth watching. As it matures and potentially adds higher-level features like package loading, cuengine could be reimplemented on top of it. For now, building my own FFI bridge gave me exactly the abstraction level I needed. ## My Solution: A Three-Layer Architecture I settled on an architecture with three carefully designed layers, each with a very specific job: ### Layer 1: The Go Bridge (`bridge.go`) Over on the Go side, I wrote a simple bridge that handles the CUE evaluation and hands back structured JSON responses. ```go type BridgeResponse struct { Version string `json:"version"` Ok *json.RawMessage `json:"ok,omitempty"` Error *BridgeError `json:"error,omitempty"` } //export cue_eval_package func cue_eval_package(dirPath *C.char, packageName *C.char) *C.char { // Evaluate CUE and return structured envelope } ``` A few key decisions made this work reliably: - **Structured Envelopes**: Every response is a predictable JSON object. You either get success data (`ok`) or a typed error (`error`). No guessing. - **Ordered JSON**: CUE's field order can be significant, so I couldn't rely on Go's randomized map ordering. I had to build the JSON string manually to keep it deterministic. - **Panic Recovery**: An FFI boundary should *never* panic. I wrapped the core logic in a recover block to catch any panics and turn them into proper errors. - **Typed Error Codes**: I created a set of error codes (and synced them with the Rust side) to let the caller know exactly what went wrong. ### Layer 2: The FFI Boundary (`lib.rs`) This is where the real magic (and danger) happens. FFI in Rust means `unsafe` code, and you have to treat it with respect. My approach was to contain it as much as possible inside a safe wrapper. ```rust pub struct CStringPtr { ptr: *mut c_char, _marker: PhantomData<*const ()>, // !Send + !Sync } impl Drop for CStringPtr { fn drop(&mut self) { if !self.ptr.is_null() { unsafe { cue_free_string(self.ptr); } } } } ``` I built this `CStringPtr` as a RAII wrapper. It's a smart pointer that: - Automatically calls the Go `cue_free_string` function when it goes out of scope. - Prevents use-after-free bugs by tying the memory to Rust's ownership system. - Uses `PhantomData` to mark the type as `!Send` and `!Sync`, which prevents it from being used across threads. This is critical because Go's runtime isn't thread-safe in the way Rust expects. ### Layer 3: The Safe Application Layer (`builder.rs`) This is the public-facing API. I designed it to completely hide the FFI complexity behind a friendly builder pattern. ```rust let evaluator = CueEvaluator::builder().build()?; let result = evaluator.evaluate(path, "cuenv")?; ``` Anyone using the `cuengine` crate never sees a raw pointer, a C string, or any `unsafe` code. They get a safe, ergonomic Rust API that provides: - Input validation before FFI calls. - Output size limits to prevent abuse. - Rich, structured error types. ## Tackling the Hard Problems Building this wasn't exactly a walk in the park. Here are some of the biggest hurdles I had to overcome. ### Hurdle #1: Taming Memory Across Runtimes **The Challenge**: Go has a garbage collector. Rust has ownership. When Go allocates memory and gives Rust a pointer, who is responsible for cleaning it up? **My Solution**: Explicit ownership transfer with my `CStringPtr` RAII wrapper. When the Go bridge allocates a C string, it hands ownership over to Rust. My `CStringPtr` wrapper takes that ownership, and its `Drop` implementation guarantees that the memory is freed exactly once, even if errors occur. ```rust let result_ptr = unsafe { cue_eval_package(c_dir.as_ptr(), c_package.as_ptr()) }; let result = unsafe { CStringPtr::new(result_ptr) }; // Takes ownership // result is automatically freed when it goes out of scope. Magic! ``` The `PhantomData` marker is the unsung hero here, preventing any accidental use of the pointer across threads that could lead to subtle, horrifying bugs. ### Hurdle #2: Don't Lose the Error Context! **The Challenge**: A simple string like `"CUE evaluation failed"` is a useless error. Go's errors have rich context, and I needed to preserve it across the FFI boundary. **My Solution**: Structured error envelopes with typed codes. Instead of just returning an error string, the Go bridge sends back a full JSON object for errors. ```json { "version": "bridge/1", "error": { "code": "LOAD_INSTANCE", "message": "Failed to load CUE instance: undefined field foo", "hint": "Check CUE syntax and import statements" } } ``` I synchronized the error codes (like `LOAD_INSTANCE`) as constants in both Go and Rust. This allows the Rust code to deserialize the error and map it into a proper, typed Rust `Error`. ```rust match bridge_error.code.as_str() { ERROR_CODE_INVALID_INPUT | ERROR_CODE_REGISTRY_INIT => Err(Error::configuration(full_message)), ERROR_CODE_LOAD_INSTANCE | ERROR_CODE_BUILD_VALUE => Err(Error::cue_parse(dir_path, full_message)), _ => Err(Error::ffi("cue_eval_package", full_message)), } ``` It's more work, but it makes debugging a thousand times easier. ### Hurdle #3: Making it Build... Everywhere **The Challenge**: My build process now needed a Go compiler, a C toolchain, and the Rust compiler, all playing nicely together. How could I make this work for other developers without a 30-step setup guide? **My Solution**: A smart `build.rs` script with fallbacks. I put a lot of effort into the `build.rs` script to make it as robust as possible: 1. First, it checks for pre-built artifacts from my Nix-based workflow. 2. If it can't find any, it falls back to building the Go bridge from source. 3. It then configures platform-specific linking, because macOS, Linux, and Windows all need different system libraries. This approach gives me fast, reproducible builds in my environment while still allowing anyone with `go` and `rustc` installed to build the crate from scratch. ## Safety and Observability What separates a quick hack from a production-grade library? For me, it's about getting the fundamentals right: memory safety, fail-fast validation, and comprehensive observability. Here's what makes `cuengine` solid: - **Comprehensive Validation**: The API validates all inputs *before* they hit the FFI boundary. Path traversal attempts, oversized inputs, and invalid names are all rejected early with clear error messages. - **Structured Tracing**: I instrumented every important operation with `tracing`, providing detailed observability for debugging and monitoring in production environments. ## My Testing Strategy I'm a firm believer in a multi-layered testing strategy. For `cuengine`, that meant: - **Unit Tests**: For testing the small stuff, like my `CStringPtr` wrapper and UTF-8 conversions. - **Integration Tests**: For end-to-end evaluation of real CUE files, making sure the whole pipeline works. - **Property Tests**: Using `proptest` to throw randomized inputs at my validation and parsing logic to shake out edge cases. - **Stress Tests**: Running the FFI calls in tight loops to check for memory leaks or race conditions. ## Performance Characteristics The FFI bridge obviously adds some overhead, but for evaluating configuration files, it's more than acceptable. I ran benchmarks using `criterion` to see what I was dealing with: | Configuration Size | Time | Notes | | ------------------ | -------- | ------------------------ | | 10 variables | ~3.7ms | Small config | | 100 variables | ~4.1ms | Minimal parsing overhead | | 1,000 variables | ~9.8ms | Medium config | | 10,000 variables | ~136ms | Large config | | 100,000 variables | ~11.3s | Extreme scale test | The benchmarks show performance scales linearly with the size of the configuration, which is exactly what I was hoping for. For a build tool, where you evaluate config once per run, this overhead is a tiny price to pay for the power of CUE. ## My Key Takeaways 1. **FFI is Unsafe, So Contain It.** The public API of `cuengine` is 100% safe. All the `unsafe` code is an internal detail, carefully documented and wrapped in safe abstractions. 2. **Good Errors are Worth the Effort.** Moving from simple error strings to structured JSON envelopes was extra work, but it made the library immensely more robust and easier to debug. 3. **Build Scripts are First-Class Code.** I spent almost as much time on `build.rs` as on some of the library features. A good build experience is a feature in itself. 4. **Fail Fast with Clear Errors.** Input validation and structured tracing aren't things you bolt on later. Building them in from the start makes debugging infinitely easier. ## What's Next `cuengine` is the solid foundation, but the real fun is the rest of the `cuenv` project I'm building on top of it: - **A full-featured CLI** for task execution and environment management. - **Nix Integration** for automatic software provisioning. - **Pluggable secret management** for 1Password, AWS Secrets Manager, and more. ## Try It Yourself `cuengine` is available on [crates.io](https://crates.io/crates/cuengine) with full [API documentation](https://docs.rs/crate/cuengine/latest) on docs.rs. The complete source is available at [github.com/cuenv/cuenv](https://github.com/cuenv/cuenv). To use it in your Rust project: ```toml [dependencies] cuengine = "0.4" ``` ```rust use cuengine::CueEvaluator; use std::path::Path; let evaluator = CueEvaluator::builder().build()?; let json = evaluator.evaluate(Path::new("./config"), "mypackage")?; ``` ## Conclusion Building this FFI bridge was a fascinating journey. It taught me that you don't have to choose between safety and pragmatism. By carefully containing `unsafe` code, designing for robust error handling, and thinking about production features from the start, it's possible to bring two very different ecosystems together and get the best of both. If you're building tools that need CUE in Rust, I hope `cuengine` gives you a solid foundation. And if you're tackling any FFI project, I hope my story helps you sidestep some of the pitfalls I stumbled into! The full source code, including all the safety invariants and production features I discussed, is available under the AGPL-3.0-or-later license. Contributions, feedback, and questions are always welcome. Don't be a stranger—come and join the chat on our [Zulip](https://chat.rawkode.academy) server! --- # Wassette: A New Era of Security for AI Agent Tooling > Wassette, a WebAssembly-based sandboxing technology for AI agent tools, and an analysis of why it represents a major step forward in security compared to traditional methods like Docker and direct execution. - Canonical: https://rawkode.academy/read/introducing-wassette - Published: 2025-08-07 - Authors: David Flanagan - Technologies: webassembly, docker AI agents are becoming more capable and more dangerous. As they gain the ability to write files, execute code, and make network requests through the Model Context Protocol (MCP), the risk of exploitation increases. That power must be controlled. **[Wassette](https://github.com/microsoft/wassette)** is a new open-source toolkit, announced by Microsoft's Azure Core Upstream team yesterday (August 6, 2025), that offers a path forward by combining WebAssembly's strong isolation with a capability-based security model. Here's why it matters and how it stacks up against Docker and bunx. ## What is Wassette? WebAssembly-Powered Isolation Wassette represents a new paradigm for MCP server security, leveraging WebAssembly's sandboxing capabilities to provide browser-grade isolation for untrusted code. ### How Wassette Locks Down Tooling Wassette's security is built on several foundations: - **Capability-Based Security**: Fine-grained permissions for specific operations. Tools must declare the capabilities they need (e.g., access to a specific directory or network host), and the user or host system must explicitly grant them. - **Deny-by-Default**: No permissions are granted unless explicitly allowed. A tool has no access to the file system, network, or environment variables by default. - **Memory Isolation**: Each WebAssembly module runs in its own linear memory space, completely isolated from the host process and other modules. It cannot read or write memory outside its sandbox. - **No Direct System Calls**: All system interactions are proxied through the host runtime via the WebAssembly System Interface (WASI). This allows the host to intercept and validate any attempt to interact with the underlying system. Unlike traditional execution models where processes inherit full user permissions, Wassette components must explicitly declare their capabilities through WebAssembly Interface Types (WIT), and users must grant each permission individually at runtime. ### Why WebAssembly Changes Everything - **Superior Isolation**: Wassette provides browser-grade sandboxing, a security model that has been battle-tested over years of web security research. The attack surface is minimal. - **Runtime Independence**: Components are self-contained with zero host dependencies (no Node.js, Python, or system libraries needed). This eliminates runtime supply chain attacks, though components must still be fetched from OCI registries where cryptographic signing becomes critical. - **Cross-Platform Consistency**: WebAssembly provides identical behavior across different operating systems and architectures. - **Fine-Grained Permissions**: The capability-based model allows for precise, auditable control over what a tool can and cannot do. - **Cryptographic Verification**: Wassette supports signed components via OCI distribution, ensuring authenticity and integrity. ### Runtime Requirements and Limitations Wassette is built on the Wasmtime runtime and written in Rust, installable as a standalone binary with zero runtime dependencies. Current limitations: - **Ecosystem Maturity**: Very few MCP servers are currently compiled to WebAssembly components - **WASI Constraints**: No support for threads, limited async I/O, and restricted system call access - **Developer Experience**: Requires learning new toolchains (wasm-tools, cargo-component) compared to familiar npm/pip workflows - **Performance Overhead**: Wasmtime adds measurable overhead for system calls and memory operations compared to native execution ## Why Existing Security Models Fall Short To understand Wassette's significance, we need to examine the security gaps in today's MCP server deployment methods. ### Approach 1: The Wild West with `bunx` (Direct Execution) The simplest way to run an MCP server is to execute it directly on the host system using a runtime like `bunx` or `npx`. **Security Reality**: When you run `bunx @modelcontextprotocol/server-filesystem`, the server process inherits the full permissions of your user account. It can read your SSH keys, access your documents, make arbitrary network connections, and spawn other processes. There is **no isolation**. This approach is convenient for development but poses a massive security risk. A single compromised dependency in the tool's supply chain could lead to a full system compromise. Unlike Wassette's OCI-distributed signed components, `bunx` fetches arbitrary npm packages at runtime with no cryptographic verification beyond optional package signatures. | `bunx` Pros | `bunx` Cons | | ----------------------------- | ------------------------ | | Simplicity & Performance | No Isolation | | Full Ecosystem Access | Major Supply Chain Risks | | Familiar Developer Experience | No Resource Limits | **Verdict**: Unacceptable for running untrusted or third-party tools. Use only for trusted, first-party tools in controlled development environments. ### Approach 2: The Walled Garden with Docker (Containerization) Docker improves upon direct execution by providing process isolation using Linux namespaces and cgroups. It’s a significant step up in security. **Security Reality**: A Docker container isolates the tool’s process, filesystem, and network. You can set resource limits (CPU, memory) and use security features like read-only filesystems and capability dropping. However, this security is not absolute: - **Kernel-Level Sharing**: All containers on a host share the same Linux kernel. A kernel vulnerability could lead to a container escape. - **Daemon Privileges**: The Docker daemon typically runs as root, representing a powerful target for attackers. - **Coarse-Grained Permissions**: Permissions are generally applied at the container level. For example, if a tool needs to read a single file, you must mount its entire parent directory as a volume. The container then has access to everything in that directory, violating the principle of least privilege. - **Unrestricted Network Access**: By default, a container can make outbound network connections to any destination on the internet. A malicious tool could easily exfiltrate sensitive data from a mounted volume to a remote server without any specific network permissions being granted. ```yaml # Docker gives us resource limits and some isolation services: mcp-server: image: mcp-server:latest read_only: true security_opt: - no-new-privileges:true volumes: - ./data:/data:ro # The container can read everything in ./data # By default, this container can send the contents of ./data anywhere on the internet. mem_limit: 512m ``` **Verdict**: A mature and robust solution suitable for production, but it lacks the fine-grained, deny-by-default security of WebAssembly. It’s a walled garden, but a determined attacker might still be able to climb the walls or send data over them. ## Comparative Analysis: Wassette vs. Docker vs. bunx Wassette's capability-based model is fundamentally more secure than the isolation models of Docker or the complete lack of isolation with bunx. ### Security Comparison Matrix | Aspect | bunx | Docker | Wassette | | -------------------------- | ---------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------- | | **Process Isolation** | None | Strong (namespaces/cgroups) [§](#resource-6) | Strongest (linear memory) [§](#resource-4) | | **File System Access** | Unrestricted | Volume-based | Capability-based [§](#resource-5) | | **Network Control** | None | Network policies | Per-domain permissions [§](#resource-1) | | **Resource Limits** | None | cgroups [§](#resource-6) | Built-in memory limits; CPU via host [§](#resource-5) | | **Supply Chain Security** | npm packages (runtime fetch) | Image scanning (opt-in signing) | OCI-distributed (signed components) [§](#resource-2) | | **Permission Granularity** | All or nothing | Container-level | Operation-level [§](#resource-4) | | **Escape Difficulty** | N/A | Moderate (CVEs exist) [§](#resource-7) | Very High [§](#resource-4) | | **Audit Capabilities** | External tools | Docker logs | Built-in manifest | | **Performance Overhead** | None | 5-15% | 10-55% [§](#resource-8) | | **Setup Complexity** | Minimal | Moderate | Low-Moderate | ## The Ecosystem Challenge: A Chicken-and-Egg Problem ### Sample Permission Policy Wassette uses YAML policy files to define what capabilities a component can access: ```yaml version: "1.0" description: "Permission policy for fetch component" permissions: network: allow: - host: "https://api.github.com/" - host: "https://opensource.microsoft.com/" filesystem: read: - path: "/tmp/cache" write: - path: "/tmp/output" environment: allow: - "GITHUB_TOKEN" ``` This declarative approach makes security auditable before execution, contrasting sharply with Docker's all-or-nothing volume mounts and bunx's complete lack of restrictions. ### The Adoption Problem Wassette faces a classic adoption challenge: users need components to run, but developers need users to justify building components. While Wassette offers superior security through WebAssembly's capability-based model, the ecosystem isn't ready for production use. ### Current Limitations - **Limited Component Availability**: Few MCP servers exist as WebAssembly components - **Tooling Maturity**: Creating Wasm components requires specialized knowledge - **Performance Overhead**: Some workloads may run slower than native execution ## What You Should Use Today (and Why) ### Today: Use Docker for Production Docker remains the recommended approach for production MCP servers. It's mature, well-understood, and provides solid security boundaries when properly configured: - Implement strict security policies and resource limits - Use read-only filesystems where possible - Regularly audit and update container images - Monitor your supply chain for vulnerabilities ### Tomorrow: Experiment with Wassette Start exploring Wassette in non-critical environments to prepare for the future: - Try simple tools first to understand the security model - Build WebAssembly versions of basic MCP servers - Share your experiences and patterns with the community - Consider hybrid approaches for gradual migration ### Never: Use bunx in Production Direct execution with `bunx` should be strictly limited to local development of trusted, first-party code where rapid iteration is essential. ## Beyond Security: Portability and Composability While security is Wassette's primary value proposition, WebAssembly brings additional benefits: - **True Portability**: Components run identically across Linux, macOS, Windows, and even edge environments - **Micro-tool Composition**: Build complex workflows from small, auditable components - **Agentic Workflows**: Wassette can serve as the default execution engine for autonomous agents, providing safe sandboxing without heavyweight solutions like gVisor or rootless Docker - **Edge Deployment**: WebAssembly's small footprint enables running MCP servers in resource-constrained environments ## Conclusion Wassette represents the future of secure AI agent tooling, but that future isn't here yet. Its WebAssembly-based, capability-driven security model is fundamentally superior to containerization, but the ecosystem needs time to mature. By being pragmatic today (using Docker) while investing in tomorrow (experimenting with Wassette), we can help build a more secure foundation for AI agents. The transition won't happen overnight, but it's a journey worth taking. I couldn't be more excited about Wassette for MCP, and I'll be exploring building my own components over the coming weeks. Check back for updates on my journey into WebAssembly-powered AI agent security. --- # News Digest for August 4th, 2025 > A dispatch from the cloud-native world, where the height of innovation is apparently pointlessly reinventing YAML while shaking down the community for container images like a common street mugger - Canonical: https://rawkode.academy/read/news-digest-250804 - Published: 2025-08-03 - Authors: David Flanagan - Technologies: kubernetes, docker The cloud-native landscape continues its rapid evolution, bringing significant shifts in core technologies, a new vision for AI-assisted development, and strategic realignments from key players. Grab a coffee, let's dive in. ## Kubernetes v1.34: Stability, Smart Tokens, and a New YAML Dialect ### The Facts Kubernetes v1.34 releases August 27, 2025, with zero removals or deprecations. Key graduations include: - **Dynamic Resource Allocation** hitting stable (`resource.k8s.io/v1`) - **ServiceAccount tokens for image pull authentication** reaching beta (enabled by default) - **Kubelet and API Server tracing** graduating to stable with OpenTelemetry integration - **Traffic distribution** gets beta `PreferSameNode` and `PreferSameZone` options - **HPA tolerance field** targeting beta for fine-grained scaling control Kubernetes is also introducing **KYAML** - a Kubernetes dialect of YAML - as a new kubectl output format. ### The Editorial Look, I get it. KYAML maintains backward compatibility while addressing YAML's notorious footguns. It's not forcing new APIs on anyone, and it works with existing tooling. It's a boring and sensible decision. But this still feels like classic NIH syndrome. **CUE** already exists and elegantly solves every problem KYAML is trying to address - yes, even if you need a PhD in type theory to write it effectively. The CNCF even has **KCL** sitting right there. Both are mature, battle-tested solutions. I wish this could be different, I wish the project could take more risks and bring in better solutions, but I emphasise with what they have to work with. maybe Kubernetes 2.0 can make the right decision. ## The Dawn of "Vibe Coding" and AI-Assisted Development ### The Facts Gene Kim declares AI coding "10 to 100 times bigger than DevOps." His book "Vibe Coding" (co-authored with Steve Yegge) was created using 70 million AI tokens. Their FAAFO framework promises five superpowers: - **Fast:** Ship features in minutes - **Ambitious:** Weekend achievement of long-term goals - **Autonomous:** Single developers functioning as teams - **Fun:** Eliminate tedious work - **Optionality:** Cheap parallel experimentation Adidas reports 2x increase in developer "happy time" across 700 developers using GitHub Copilot daily. ### The Editorial Whether you buy into the "vibe coding" hype or not, the data is compelling. Kim's prediction that this increases developer demand rather than decreasing it feels counterintuitive but tracks with historical automation trends. The question isn't if AI will transform development - it's whether "vibe coding" will be what we actually call it in five years. ## Building the "Internet of Agents" with AGNTCY ### The Facts Cisco donated AGNTCY to the Linux Foundation with backing from Dell, Google Cloud, Oracle, and Red Hat. The project aims to build open, interoperable, quantum-safe infrastructure for collaborative AI agents, including components for: - Agent discovery - Identity management - Messaging protocols - Observability standards ### The Editorial Smart move putting this under neutral governance. The agent ecosystem is currently a fragmented mess, and interoperability is make-or-break for this technology scaling beyond toy demos. The Linux Foundation's track record with Kubernetes gives some hope this won't devolve into vendor politics. Honestly, Cisco is embracing Open Source and the community and proving to be a good steward. Broadcom, take notes! ## KubeSphere Adjusts Open Source Project Focus ### The Facts KubeSphere announced changes effective July 31, 2025: - Discontinuing download links for open-source version - Ceasing free technical support - Code remains Apache 2.0 licensed (for now) - Pushing users toward commercial solutions ### The Editorial In news that will surely impact someone, somewhere, KubeSphere - the cloud-native platform you may or may not have heard of - is essentially abandoning its open-source community; assuming one exists... The "we need to concentrate resources" line is corporate speak for "this isn't making us money." At least they're keeping the code open source... for now. Real talk though: Open Source needs to be sustainable and frankly, it isn't. Again, I emphasise with these difficult decisions. These orgs place a huge amount of risk on Open Source being a viable marketing strategy and it continues to prove time and time again that it isn't. It's not the quick path to adoption, nor inbound sales. ## Bitnami Announces Major Catalog Changes ### The Facts Effective August 28, 2025, Bitnami implements massive catalog changes: - **Debian-based images:** Discontinued and moved to unsupported "Bitnami Legacy" repository - **Free tier:** Reduced to "latest" tag only, development use only - **All existing versions:** Moved to `docker.io/bitnamilegacy` with zero updates - **Production images:** Now require "Bitnami Secure Images" commercial subscription - **Timeline:** Users have until August 28 to update all CI/CD pipelines ### The Editorial **This is a disaster.** Broadcom has somehow managed to make Cloud Native *even worse* than VMware did - a title I didn't think could be claimed. This isn't just bad; it's catastrophically stupid. Millions of deployments depend on Bitnami images. Countless tutorials, documentation, and automation scripts reference them. And Broadcom's solution? Pay us or get fucked. The "legacy" repository is particularly insulting - it's not maintained, not supported, just a graveyard where your production dependencies go to die. They're not even pretending to care about the community impact. This is worse than Docker Hub's rate limiting fiasco. At least Docker gave people workable alternatives and time to adjust. Broadcom is essentially holding the community hostage: pay for enterprise licenses or scramble to rebuild your entire container strategy in less than a month. Google stepped in and provided a Docker Hub mirror, which is a lot easier to handle than maintaining thousands of Open Source charts. I'm not entirely sure any single entity can ease the burden of this disaster. The message is crystal clear: Broadcom views open source not as a community to nurture, but as a customer acquisition funnel to squeeze. They've taken one of the most trusted brands in cloud-native and turned it into a cautionary tale about what happens when private equity mindset meets open source infrastructure. **Start migrating now.** Seriously. August 28 will come faster than you think, and Broadcom has made it abundantly clear they don't give a single damn about your production workloads. --- # FluxCD: Why the GitOps Pioneer Remains Its Future > A definitive look at FluxCD's controller-first design and why its architectural alignment with Kubernetes offers superior security, efficiency, and operational maturity over ArgoCD. - Canonical: https://rawkode.academy/read/fluxcd-the-inevitable-choice - Published: 2025-07-15 - Authors: David Flanagan - Technologies: fluxcd, kubernetes **The Flux Advantage in 60 Seconds:** * **Architecture:** Flux is a set of discrete, single-purpose controllers that mirror Kubernetes' own design. Argo is a monolithic bundle of inter-dependent services. * **Operations:** Flux provides structural failure isolation and precise, per-component scaling. Argo's bundled design creates systemic failure modes and complex scaling logic. * **Security:** Flux’s compartmentalized controllers create a minimal, defensible attack surface. Argo's monolithic `repo-server` is a well-documented source of critical, system-wide CVEs. * **Extensibility:** Flux extends via composable, independent controllers. Argo relies on plugins that create tight operational coupling with its core `repo-server` process. --- FluxCD and ArgoCD both promise Git-driven, declarative delivery, but only one of them **embraces the DNA of Kubernetes instead of merely co-existing with it**. Flux's architecture, composed of single-purpose controllers, is a natural extension of the platform it runs on. If you prioritize operational clarity, security, and long-term cost of ownership, Flux isn't just another option - it's the inevitable choice. ### Let's Be Honest: The Appeal of ArgoCD is Real ArgoCD's popularity isn't an accident. Its all-in-one installation and polished, built-in user interface provide an outstanding out-of-the-box experience. For teams adopting GitOps for the first time, the `Application` CRD and the web UI offer a shallow learning curve and immediate visual feedback. These are significant advantages. However, this initial convenience comes with long-term architectural and operational costs - trade-offs that are often overlooked until a team is operating at scale, where they become impossible to ignore. ### Architectural Alignment: The Picture Is Clear Kubernetes succeeds because every task - scheduling, autoscaling, networking - is handled by a dedicated, single-purpose controller. Flux extends this contract seamlessly. | Flux Controller | Purpose | | ------------------- | --------------------------------------------- | | **Source** | Fetches Git/Helm/OCI artifacts | | **Kustomize** | Reconciles Kustomizations | | **Helm** | Manages `HelmRelease` objects declaratively | | **Notification** | Routes events to Slack, Teams, and webhooks | | **Image Automation**| Automatically updates image tags in Git | Each controller is its own Deployment, upgraded, scaled, and RBAC-scoped in isolation. This gives operators surgical precision. ArgoCD, by contrast, bundles its core logic into three main inter-dependent services - `repo-server`, `application-controller`, and `argocd-server` - and often relies on optional Redis and Dex components to handle caching and auth. ***Visualizing the Difference*** The diagram below illustrates the core architectural divide. On one side, Flux's controllers operate as independent agents, each with a direct line to the Kubernetes API. On the other, Argo's core services are bundled, interacting internally before presenting a unified front to the API. This distinction isn't merely academic; it's the root cause of the operational differences that follow. ![Architectural diagram comparing Flux's distributed controllers to Argo's monolithic services](./comparison.png) ### Day-Two Operations: Precision Beats "One Size Scales All" - **Resource Footprint:** Flux maintainers state that controllers use **~30 MiB of RAM each** after bootstrap, though actual usage can vary based on workload. In contrast, ArgoCD's documentation shows default resource requests of **256Mi for the** `repo-server` **and 1Gi for the** `application-controller` - before accounting for Redis. While these are defaults that can be tuned, the architectural difference is clear. - **Failure Isolation:** If Flux's `helm-controller` crashes, your Git and Kustomize reconciliations continue to flow. If Argo's `repo-server` experiences a failure, it can stall manifest rendering for *every single application* simultaneously. - **Scaling:** Need to handle more Helm repositories? Simply increase the `replicas` for Flux's `source-controller`. In ArgoCD, scaling often requires enabling and configuring more complex controller sharding logic, managed via environment variables and replica count adjustments. ### Security Posture: Smaller Binaries, Smaller Blast Radius Both Flux and ArgoCD achieved **CNCF Graduation in late 2022**, demonstrating their security, governance, and stability. Flux graduated on November 30, 2022, with ArgoCD following shortly after in December 2022. Flux's decomposed architecture means each binary has a limited scope and a smaller attack surface. Meanwhile, Argo's `repo-server` has experienced several vulnerabilities. For example, **CVE-2024-29893** allowed a malicious Helm registry to trigger a Denial-of-Service by causing the repo-server to run out of memory during unbounded data fetching. When a single component is responsible for fetching and rendering untrusted multi-tenant resources, its broad scope increases the potential blast radius. ### Extensibility: Rendering Philosophies Reveal Design Principles Flux's approach to configuration languages is straightforward: each gets its own dedicated controller. Need KCL support? Deploy the kcl-controller. Want CUE? Add the cue-controller. These controllers operate independently, following the same patterns as every other Flux component. ArgoCD takes a different path. While it can leverage the same operators Flux uses (like External Secrets Operator), its approach to configuration rendering reveals its monolithic nature. Support for languages like KCL or CUE requires either: Config Management Plugins (CMPs) - Scripts that run inside the repo-server, inheriting its permissions and failure modes Sidecar containers - Additional processes coupled to the repo-server's lifecycle This isn't just an implementation detail. When your KCL rendering logic crashes in a CMP, it can affect the repo-server's ability to process other applications. When a Flux kcl-controller crashes, your Helm and Kustomize deployments continue unaffected. The architectural divide is clear: Flux extends through composition of independent controllers, while ArgoCD extends through augmentation of its central rendering service. One scales horizontally with clear failure boundaries; the other scales vertically with shared failure modes. Of course, the elephant in the room is that KCL and CUE controllers may not exist yet. But the point is that Flux's architecture allows for their creation without introducing new monolithic components. This is a design principle that aligns with Kubernetes' own extensibility philosophy. ### But What About ApplicationSets? ArgoCD's `ApplicationSet` controller is an elegant solution for managing applications across a fleet of clusters. It is, deservedly, a killer feature. Flux achieves the same outcome, albeit in a more Kubernetes-native way, by composing its existing APIs. Using a combination of the **Flux `Provider` for Terraform** or by templating `Kustomization` and `HelmRelease` objects, a single source of truth can be used to stamp out configuration for dozens of clusters. While this may require a deeper understanding of the underlying tools, it avoids introducing another proprietary abstraction layer and keeps the management paradigm consistent with Kubernetes itself. ### The GUI: A Red Herring Argo's built-in dashboard is excellent, but this convenience isn't exclusive. The GitOps ecosystem provides purpose-built front-ends for Flux, such as **[Capacitor](https://github.com/gimlet-io/capacitor)** and the **[Headlamp Flux plugin](https://github.com/headlamp-k8s/plugins/tree/main/flux)**. This gives Flux users a choice of UIs **without forcing them to run and secure an always-on web server in every production cluster.** ### Conclusion: Choose the Tool That Aligns With Your Future ArgoCD is an excellent tool that provides a fantastic on-ramp to GitOps. Both projects have proven their production readiness through CNCF graduation. But production engineering isn't about on-ramps; it's about **mean-time-to-recover, total cost of ownership, and security posture**. On the metrics that matter for mature, at-scale operations - architecture, security, and efficiency - FluxCD offers compelling advantages. It is built not just to integrate with Kubernetes, but to reflect its core design principles. Stop paying a long-term operational tax for short-term convenience. Run GitOps the Kubernetes way. **Choose FluxCD.** --- # KEP-2831: Kubelet Tracing Finally Brings Node-Level Observability to Kubernetes > Kubernetes 1.34 will deliver distributed tracing in the kubelet, providing unprecedented visibility into node-level operations that have been a debugging black box until now. - Canonical: https://rawkode.academy/read/kep-2831-kubelet-tracing - Published: 2025-07-14 - Authors: David Flanagan - Technologies: kubernetes, opentelemetry Kubernetes 1.34, scheduled for release on August 27, 2025, will bring a significant enhancement to cluster observability. KEP-2831 introduces distributed tracing directly into the kubelet—Kubernetes' node-level workhorse that manages pods, pulls images, and orchestrates container lifecycles. While Kubernetes has provided structured JSON logs, metrics, and events for debugging node-level issues, correlating these signals to understand complex failure scenarios has remained time-consuming. When pods get stuck in Pending or ContainerCreating status, you might spend significant time piecing together logs and events to understand the sequence and timing of operations. KEP-2831 changes this by adding OpenTelemetry-based distributed tracing to the kubelet. This means you can now visualize the entire lifecycle of pod operations, see exact timings for each step, and quickly identify bottlenecks—all while correlating these traces with your existing application-level observability. ## What Kubelet Tracing Actually Does (And Why You Should Care) Think about the last time you had a pod stuck in Pending or ContainerCreating status. While kubectl describe and kubelet logs could tell you what happened, understanding the why often required correlating timestamps across multiple log entries and events. With KEP-2831's distributed tracing, you get structured visibility into: - Pod lifecycle operations: From scheduling to running state with precise timing - Container runtime interactions: Image pulls, container creation, and startup sequences with duration metrics - Volume operations: Mount/unmount operations and storage interactions with latency data - Resource management: CPU, memory, and storage allocation processes with performance insights The key advantage is seeing these operations as connected traces rather than isolated log entries. A single trace shows you that an image pull took 45 seconds, followed by a 10-second volume mount, making it immediately clear where your pod startup time is being spent. This is especially powerful when combined with your application traces, giving you true end-to-end observability from user request to container startup. ## The Technical Deep Dive: OpenTelemetry Integration KEP-2831 leverages OpenTelemetry for its tracing implementation, following the established pattern used throughout the Kubernetes ecosystem. Here's what this means for platform teams: **Vendor Neutrality: Your Tools, Your Choice.** By using OpenTelemetry, traces can be exported to any compatible backend—Jaeger, Zipkin, Grafana Tempo, or commercial solutions. This flexibility allows teams to choose the best tools for their needs without being locked into a specific vendor. **Ecosystem Compatibility: No More Silos.** The integration ensures seamless compatibility with existing observability stacks, making it easier for teams to incorporate kubelet tracing into their current workflows and correlate infrastructure traces with application traces. **Standards-Based Approach.** Using OpenTelemetry aligns with Kubernetes' broader observability strategy and ensures kubelet tracing will benefit from ongoing improvements to the OpenTelemetry project. With this integration, you can finally connect the dots between a slow API response in your application and a delayed volume mount on the node—providing the full-stack visibility that's been missing from Kubernetes debugging. ## Current Status: Beta and Production-Ready (With Caveats) Let's set the record straight on where KEP-2831 stands today. As of Kubernetes v1.27, kubelet tracing has reached Beta status, with General Availability targeted for v1.34 (releasing August 27, 2025). What Beta means for you: - ✅ Production use is supported (though not recommended for mission-critical workloads without careful monitoring). - ⚠️ API changes are possible but unlikely to be breaking. - 🔄 Feature flags may be required depending on your Kubernetes distribution. - 📈 Active development continues with community feedback driving improvements. For early adopters and platform teams looking to gain a competitive advantage in observability, now is an excellent time to start experimenting. Just ensure you have rollback plans and aren't betting critical operations on Beta features. ## Real-World Impact: Beyond Individual Clusters The implications of kubelet tracing extend far beyond solving individual debugging sessions. This capability represents a fundamental shift in how we approach Kubernetes observability at scale. ### For Platform Teams - Faster incident resolution: Trace-driven debugging reduces MTTR significantly. - Proactive optimization: Identify performance bottlenecks before they impact users. - Better capacity planning: Understand actual resource utilization patterns at the node level. ### For the Cloud-Native Ecosystem KEP-2831 positions Kubernetes as a leader in infrastructure observability. As organizations increasingly adopt cloud-native architectures, the ability to trace operations across the entire stack—from application code down to container runtime interactions—becomes a competitive advantage. This also aligns with the broader industry trend toward observability-driven development, where teams use telemetry data to inform architectural decisions and optimize system performance proactively. ## Getting Started: What You Need to Know Ready to experiment with kubelet tracing? Here's your roadmap: 1. Check your Kubernetes version: Ensure you're running v1.27 or later. 2. Configure OpenTelemetry: Set up your preferred tracing backend. 3. Enable the feature: Configure kubelet tracing through feature gates or configuration files. 4. Start small: Begin with non-production clusters to understand the data volume and performance impact. Keep an eye on the Kubernetes release notes and engage with the community through SIG Node discussions for the latest updates and best practices. ## The Future of Kubernetes Observability KEP-2831 isn't just about adding another observability tool—it's about completing the observability story for Kubernetes. By bridging the gap between application-level and infrastructure-level tracing, it enables a new level of system understanding that wasn't possible before. ### Kubernetes 1.34 Release Timeline The journey to GA is well underway with these key milestones: - **Enhancements Freeze**: June 20, 2025 - All KEPs locked in - **Code Freeze**: July 25, 2025 - Implementation complete - **GA Release**: August 27, 2025 - Kubelet tracing becomes production-ready As we approach the v1.34 GA release, expect to see: - Enhanced integration with popular observability platforms. - Performance optimizations to reduce overhead. - Additional instrumentation points based on community feedback. For DevOps engineers, platform engineers, and SREs, kubelet tracing represents an opportunity to level up your debugging capabilities and gain deeper insights into your Kubernetes infrastructure. The question isn't whether you should adopt it—it's how quickly you can start experimenting and learning. The future of Kubernetes observability is here, and it's more transparent than ever. --- # Lazyjournal: A Log Viewer for Cloud Native Environments > Lazyjournal is a TUI log viewer that aggregates logs from various sources, providing a unified interface for developers and system administrators. - Canonical: https://rawkode.academy/read/lazyjournal-log-viewer - Published: 2025-06-16 - Authors: David Flanagan - Technologies: kubernetes, docker Lazyjournal is a **terminal user interface (TUI) log viewer** that provides centralized access to logs from journald, auditd, file systems, Docker containers, Podman, and Kubernetes pods. Created by Alex Kup (Lifailon), this Go-based tool offers a modern alternative to traditional log viewing methods with advanced filtering, real-time streaming, and syntax coloring capabilities. ## What Problems Lazyjournal Solves Traditional log management in cloud-native environments requires juggling multiple tools and commands across different systems. Developers typically need to switch between `journalctl` for system logs, `docker logs` for containers, `kubectl logs` for Kubernetes pods, and various file viewers for application logs. This fragmentation creates friction during debugging and incident response. Lazyjournal addresses this by providing **a single unified interface** for all log sources. Instead of remembering different commands and syntaxes, developers can navigate through system services, containers, and pod logs using consistent keyboard shortcuts and filtering mechanisms. The tool's fuzzy search and regex capabilities make finding specific log entries significantly faster than traditional grep-based approaches. The real-time streaming feature with automatic reconnection eliminates the need to manually restart log tailing when containers restart or network connections drop. This is particularly valuable during active debugging sessions where maintaining continuous log visibility is crucial. ## Key Features & Capabilities **Multi-source log aggregation** stands as lazyjournal's core strength. The tool seamlessly integrates with journald for systemd logs, reads audit logs from auditd with human-readable interpretation, accesses file system logs from `/var/log` and user directories, streams Docker and Podman container logs, and connects to Kubernetes pods via kubectl. Support for archived logs in gz, xz, bz2, pcap, and asl formats extends its utility to historical analysis. The **filtering system** offers four distinct modes. Default mode provides case-sensitive exact matching, fuzzy search enables inexact case-insensitive matching similar to fzf, regex mode supports full regular expression patterns, and timestamp filtering allows date/time range selection. These modes can be switched on-the-fly during log viewing, enabling rapid refinement of search criteria. **Performance optimizations** include configurable line limits from 200 to 200,000 lines, intelligent switching between filesystem and API-based log reading for containers, color disabling for improved performance with large logs exceeding 100,000 lines, and background update intervals adjustable from 2 to 10 seconds. The tool maintains memory efficiency even when handling massive log volumes. The **interactive interface** features vim-style navigation with hjkl keys, mouse support with Alt+Shift text selection, real-time log output updates with visual loading indicators, and color-coded output for improved readability. The interface adapts to terminal size changes and provides contextual help with the ? key. ## Installation Want to install lazyjournal? The process is straightforward across various distributions and platforms. For **Linux**, you can use the following command to download the latest release directly from GitHub with [eget](https://github.com/zyedidia/eget) or via curl: ```shell eget lifailon/lazyjournal --to ~/.local/bin ``` ```bash arch=$( [ "$(uname -m)" = "aarch64" ] && echo "arm64" || echo "amd64" ) version=$(curl -L -sS -H 'Accept: application/json' https://github.com/Lifailon/lazyjournal/releases/latest | sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/') sudo curl -L -sS https://github.com/Lifailon/lazyjournal/releases/download/$version/lazyjournal-$version-linux-$arch -o lazyjournal ``` For developers preferring source installation, Go's package manager provides: `go install github.com/Lifailon/lazyjournal@latest`. **Package manager options** include: - Arch Linux (AUR) (`paru -S lazyjournal`) - Conda (`conda install conda-forge::lazyjournal`) - Homebrew (`brew install lazyjournal`) - Snap (`snap install lazyjournal`). Windows users can use PowerShell with `Invoke-RestMethod https://raw.githubusercontent.com/Lifailon/lazyjournal/main/install.ps1 | Invoke-Expression`. ## How it Works Technically Built with Go and the awesome-gocui library (a fork of gocui), lazyjournal employs an event-driven architecture that efficiently handles UI updates and log streaming. The single-threaded design keeps the binary small (~5MB) while maintaining responsiveness through intelligent buffering and update strategies. The tool implements **dual-mode container log reading**. When filesystem permissions allow, it directly parses JSON log files for better performance. Otherwise, it falls back to Docker/Podman API streaming. This adaptive approach ensures functionality across different permission scenarios without requiring elevated privileges in all cases. **Log source discovery** happens automatically on startup. The tool scans for available systemd units, Docker containers, Podman containers, Kubernetes pods, and filesystem logs. Sources are presented in categorized lists that users navigate with arrow keys or vim bindings. The centralized search feature (Ctrl+S) enables searching across all discovered sources simultaneously. **Encoding detection** supports UTF-8, UTF-16 with and without BOM, and Windows-1251, ensuring compatibility with logs from various applications and platforms. The tool automatically detects and converts encodings, preventing garbled output from non-standard log files. ## Usage Examples and Workflows For **basic system log viewing**, launch lazyjournal and navigate to journald logs. Select a service like nginx or ssh, then use fuzzy search (Ctrl+Z) to find specific patterns. The interface updates in real-time as new log entries arrive. **Container debugging workflows** benefit from the unified view. Select a Docker container from the list, toggle between stdout and stderr streams if needed, apply regex filtering to isolate error messages, and monitor multiple containers by quickly switching between them with navigation keys. **Kubernetes pod analysis** follows a similar pattern. Navigate to the Kubernetes section, select the target pod and container, enable timestamp filtering to focus on a specific incident timeframe, and use the streaming view to watch logs during deployment or debugging. **Advanced filtering scenarios** showcase the tool's power. For complex pattern matching, switch to regex mode and use patterns like `error|warning|critical`. Timestamp filtering helps isolate issues to specific time windows: `2024-06-15 14:00`. Combine multiple filters by first applying timestamp constraints, then adding pattern filters. ## Unique Advantages over Alternatives Compared to **lnav**, lazyjournal offers broader source integration including containers and Kubernetes, while lnav provides more sophisticated log analysis features like SQL queries. Against **stern** for Kubernetes, lazyjournal provides a unified interface for all log sources, not just pods. Unlike web-based tools like **Dozzle**, lazyjournal requires no additional infrastructure or exposed ports. The tool's **single binary deployment** eliminates dependency management compared to solutions requiring databases or web servers. The **interactive TUI** provides a more intuitive experience than command-line tools while maintaining terminal-native efficiency. **Cross-platform support** with native binaries for Linux, macOS, Windows, and BSD systems ensures broad compatibility. ## Value for Cloud Native Developers For teams working with **Kubernetes**, lazyjournal streamlines pod log access without complex kubectl commands. The ability to quickly switch between pods and containers accelerates debugging in microservices architectures. Integration with both Docker and Podman supports diverse container runtime environments. The tool's **lightweight footprint** makes it ideal for debugging directly on production nodes when necessary. The unified interface reduces context switching during incident response, where time is critical. Support for both streaming and historical logs enables both real-time monitoring and post-mortem analysis. While lazyjournal is written in Go rather than Rust, it effectively supports debugging **Rust applications** running in containers or writing to system logs. The tool can analyze WebAssembly application logs when they're routed through traditional logging mechanisms, though it lacks specific WASM-aware features. ## Community & Ecosystem With over 553 GitHub stars and inclusion in Awesome-Go, Awesome-TUIs, and Awesome-Docker lists, lazyjournal has gained recognition in the developer community. Active maintenance with regular releases ensures compatibility with evolving container and Kubernetes ecosystems. The project welcomes contributions, with several community members maintaining packages for various distributions. The tool's inspiration from LazyDocker and LazyGit is evident in its user-friendly approach to complex system interactions. As the author's first Go project, it demonstrates impressive functionality while maintaining clean, understandable code that encourages community contributions. ## Conclusion Lazyjournal represents a practical solution to the fragmented log viewing experience in modern cloud-native environments. By unifying access to diverse log sources through an intuitive terminal interface, it reduces operational friction for developers and system administrators. The combination of powerful filtering capabilities, real-time streaming, and broad platform support makes it particularly valuable for teams managing containerized applications and Kubernetes deployments. While specialized tools may offer deeper analysis features for specific use cases, lazyjournal's strength lies in providing fast, unified access to logs exactly when developers need it most. --- # Kyverno ValidatingPolicy with CEL: Working Examples (2026) > Replace ClusterPolicy with Kyverno ValidatingPolicy using CEL expressions. Real examples for image verification, label rules, and replica limits. - Canonical: https://rawkode.academy/read/kyverno-cel-validating-policy - Published: 2025-06-02 - Updated: 2025-06-04 - Authors: David Flanagan - Technologies: kyverno, kubernetes Kyverno 1.14 (released April 25, 2025) introduces dedicated ValidatingPolicy and ImageValidatingPolicy types that leverage CEL (Common Expression Language) - the same validation language Kubernetes now uses natively, creating a unified policy experience across your entire stack. For platform engineers and security teams implementing policy-as-code in Kubernetes, this convergence represents a fundamental shift toward standardization and performance. Let's explore why this matters and how to get started. ## The CEL Convergence Kubernetes' adoption of CEL for native validation creates an unprecedented opportunity for policy engines like Kyverno to standardize on the same language. This convergence delivers three critical benefits: **One expression language** to learn across Kubernetes and Kyverno means your team invests in skills that transfer between CRD validation rules, ValidatingAdmissionPolicies, and now Kyverno policies. **Better performance** - CEL expressions are parsed once and compiled to bytecode, providing significant speed improvements over traditional validation methods. **Reusable validation logic** between native Kubernetes and Kyverno policies eliminates the need to maintain separate validation implementations. Kubernetes now uses CEL in multiple contexts: - CRD validation rules (beta in 1.25, GA in 1.29) - ValidatingAdmissionPolicies (beta in 1.28, GA in 1.30) - Future features like ResourceQuota expressions and RBAC conditions This means mastering CEL for Kyverno directly translates to expertise across the entire Kubernetes ecosystem. ## Your First ValidatingPolicy Let's start with a practical example that demonstrates the power and simplicity of Kyverno's new ValidatingPolicy with CEL: ```yaml apiVersion: policies.kyverno.io/v1alpha1 kind: ValidatingPolicy metadata: name: require-pod-resources spec: matchConstraints: resourceRules: - apiGroups: [""] apiVersions: [v1] operations: [CREATE, UPDATE] resources: [pods] validations: - expression: | object.spec.containers.all(container, has(container.resources.limits) && has(container.resources.limits.memory) && has(container.resources.limits.cpu) ) message: "All containers must have CPU and memory limits defined" - expression: | object.spec.containers.all(container, container.resources.limits.memory.matches('^[0-9]+(Mi|Gi)$') && size(container.resources.limits.memory) <= size('8Gi') ) message: "Memory limits must not exceed 8Gi" ``` This policy enforces two critical requirements: 1. All containers must have CPU and memory limits defined 2. Memory limits cannot exceed 8Gi The CEL expressions use familiar functional programming patterns like `all()` to iterate over collections, making the logic both readable and powerful. ## Why This Matters: The Kubernetes CEL Native Story Understanding how CEL fits into the broader Kubernetes ecosystem helps appreciate why this standardization matters. Consider how the same validation logic can now work across different contexts. **CRD Validation Example:** ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: applications.platform.io spec: validation: openAPIV3Schema: properties: spec: properties: replicas: type: integer x-kubernetes-validations: - rule: "self <= 10" message: "replicas cannot exceed 10" ``` **Kyverno ValidatingPolicy Example:** ```yaml apiVersion: policies.kyverno.io/v1alpha1 kind: ValidatingPolicy metadata: name: replica-limit spec: matchConstraints: resourceRules: - operations: ["CREATE", "UPDATE"] apiGroups: ["apps"] apiVersions: ["v1"] resources: ["deployments"] validations: - expression: "object.spec.replicas <= 10" message: "replicas cannot exceed 10" ``` Notice how the core CEL expression `object.spec.replicas <= 10` remains identical across all three contexts. This consistency dramatically reduces the learning curve and enables teams to reuse validation logic. ## Advanced Pattern: Cross-Resource Validation CEL's power becomes evident when handling complex validation scenarios that require external data or cross-resource relationships: ```yaml apiVersion: policies.kyverno.io/v1alpha1 kind: ValidatingPolicy metadata: name: namespace-resource-quotas spec: matchConstraints: resourceRules: - operations: ["CREATE", "UPDATE"] apiGroups: ["apps"] apiVersions: ["v1"] resources: ["deployments", "statefulsets"] variables: - name: configs expression: > resource.Get("v1", "configmaps", "platform", "team-configs") validations: - expression: | has(object.metadata.labels.team) && object.metadata.labels.team in variables.configs.data && object.spec.replicas <= int(variables.configs.data[object.metadata.labels.team]) message: "Replica count exceeds team quota defined in platform/team-configs" - expression: | has(object.metadata.labels.environment) && object.metadata.labels.environment in ['dev', 'staging', 'prod'] message: "Must specify valid environment label: dev, staging, or prod" ``` This policy demonstrates advanced CEL patterns: - **External parameter references** using `paramKind` and `paramRef` - **Data validation** against external ConfigMap values - **Multi-expression validation** with distinct error messages - **Complex boolean logic** combining multiple conditions The referenced ConfigMap might look like: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: team-configs namespace: platform data: frontend: "5" backend: "10" data: "3" ``` ## ImageValidatingPolicy in Action Kyverno 1.14 also introduces ImageValidatingPolicy for specialized image security validation: ```yaml # https://github.com/kyverno/policies/blob/main/other/verify-image-ivpol/verify-image-ivpol.yaml apiVersion: policies.kyverno.io/v1alpha1 kind: ImageValidatingPolicy metadata: name: verify-image-ivpol annotations: policies.kyverno.io/title: Verify Image policies.kyverno.io/category: Software Supply Chain Security, EKS Best Practices policies.kyverno.io/severity: medium policies.kyverno.io/subject: Pod policies.kyverno.io/minversion: 1.14.0 policies.kyverno.io/description: >- Using the Cosign project, OCI images may be signed to ensure supply chain security is maintained. Those signatures can be verified before pulling into a cluster. This policy checks the signature of an image repo called ghcr.io/kyverno/test-verify-image to ensure it has been signed by verifying its signature against the provided public key. This policy serves as an illustration for how to configure a similar rule and will require replacing with your image(s) and keys. spec: webhookConfiguration: timeoutSeconds: 30 evaluation: background: enabled: false validationActions: [Deny] matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["pods"] matchImageReferences: - glob : "docker.io/mohdcode/kyverno*" attestors: - name: cosign cosign: key: data: | -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6QsNef3SKYhJVYSVj+ZfbPwJd0pv DLYNHXITZkhIzfE+apcxDjCCkDPcJ3A3zvhPATYOIsCxYPch7Q2JdJLsDQ== -----END PUBLIC KEY----- validations: - expression: >- images.containers.map(image, verifyImageSignatures(image, [attestors.cosign])).all(e ,e > 0) message: >- failed the verification ``` This policy combines signature verification with CEL-based attestation validation, ensuring: - Images are signed with the specified public key - Build provenance follows SLSA standards - Dependencies meet stability requirements ## Migration Guide: From ClusterPolicy to ValidatingPolicy Migrating existing ClusterPolicy resources to the new ValidatingPolicy type often simplifies your policy definitions. Here's a before/after comparison: **Before (ClusterPolicy):** ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-labels spec: rules: - name: require-labels match: any: - resources: kinds: - Deployment validate: failureAction: Enforce message: "Environment label must be dev, staging, or prod" anyPattern: - metadata: labels: environment: "dev" # Changed label key to 'environment' to match message - metadata: labels: environment: "staging" - metadata: labels: environment: "prod" ``` **After (ValidatingPolicy):** ```yaml apiVersion: policies.kyverno.io/v1alpha1 kind: ValidatingPolicy metadata: name: require-labels spec: matchConstraints: resourceRules: - operations: ["CREATE", "UPDATE"] apiGroups: ["apps"] apiVersions: ["v1"] resources: ["deployments"] validations: - expression: "has(object.metadata.labels.team) && object.metadata.labels.team != ''" message: "Deployments must have a team label" - expression: "has(object.metadata.labels.environment) && object.metadata.labels.environment in ['dev', 'staging', 'prod']" message: "Environment label must be dev, staging, or prod" ``` **Migration Benefits:** - **Reduced complexity**: Single rule instead of multiple rules - **Better performance**: CEL compilation vs pattern matching - **Clearer logic**: Explicit boolean expressions vs pattern inference - **Enhanced readability**: Self-documenting validation conditions ## Performance Implications CEL's compilation model provides significant performance advantages over traditional validation approaches. Kyverno maintainers report substantial improvements in policy evaluation speed, though specific benchmarks vary by workload: **ClusterPolicy Pattern Matching:** - Parse YAML patterns at runtime - Recursive object traversal for each validation - Variable performance depending on pattern complexity **ValidatingPolicy CEL:** - Compile expressions once at policy load - Direct object field access via compiled bytecode - Faster execution for complex validation logic **Real-world Impact:** - Measurably faster policy evaluation - Reduced CPU usage during peak admission loads - Better cluster responsiveness under policy-heavy workloads For clusters with hundreds of policies and high pod creation rates, these performance improvements directly translate to reduced latency and improved user experience. ## Getting Started Checklist Ready to adopt Kyverno 1.14's ValidatingPolicy? Follow this implementation path: 1. **Upgrade Kyverno** to version 1.14 or later 2. **Start with simple policies** using basic CEL expressions 3. **Gradually migrate** existing ClusterPolicy resources 4. **Leverage external parameters** for dynamic validation 5. **Implement ImageValidatingPolicy** for supply chain security 6. **Monitor performance** improvements in your cluster The convergence of CEL across Kubernetes and Kyverno represents more than just a technical upgrade - it's a fundamental shift toward a unified policy ecosystem. By adopting these new ValidatingPolicy types, you're investing in skills and patterns that will serve your platform engineering efforts across the entire Kubernetes landscape. Start experimenting with CEL today, and experience the power of unified validation logic across your cloud native stack. --- # Deploy to Cloudflare from GitLab CI with Dagger (Tutorial) > Build a portable GitLab CI pipeline with Dagger that deploys to Cloudflare Workers. Containerless caching, local-runnable, full source included. - Canonical: https://rawkode.academy/read/dagger-gitlab-cloudflare-deployment - Published: 2025-05-29 - Authors: David Flanagan - Technologies: dagger, gitlab, docker At Rawkode Academy, we've built a modern deployment pipeline that combines the power of Dagger for containerized builds, GitLab CI for orchestration, and Cloudflare Workers for hosting. This setup gives us fast, reliable deployments with automatic preview environments for every merge request. ## Why This Stack? At the Rawkode Academy, one of our values is "Future Facing". We strive to use the best tools available to ensure our projects are maintainable, scalable, and developer-friendly. Our latest deployment pipeline is a testament to this value, combining Dagger, GitLab CI, and Cloudflare Workers to create a robust and efficient workflow. ### Dagger **Dagger** leverages BuildKit's powerful caching mechanisms while providing an "as code" approach to CI/CD pipelines. Instead of writing YAML configurations, we can define our build and deployment logic using TypeScript and Dagger Shell, making pipelines more maintainable, testable, and reusable across different environments. While we use Dagger for CICD, it also can work as a local development tool, a task runner, and pretty much anything else you can think of. In due course, it will replace many tools: - Make / Just - Docker Compose - Tilt / DevSpace / Skaffold The Dagger team ship new features and improvements at an incredible pace, and the community is growing rapidly. It's a great time to get involved! ### GitLab **GitLab CI** offers robust pipeline orchestration with excellent integration features. It allows us to automate the entire deployment process, from code changes to production deployment. GitLab CI's powerful rules and triggers help us manage complex workflows efficiently. We love what GitHub are doing these days, but the Rawkode Academy team had an urge to host our own forge and integrate it into our entire SDLC and platform itself; and GitLab was a great choice. We explored Gitea and Forgejo too, both fantastic options, but GitLab's complete feature set, including security scanning, code quality checks, and built-in CI/CD, made it the best fit for our needs. ### Cloudflare Workers **Cloudflare Workers** delivers fast, global content delivery with generous free tiers. It provides a reliable and scalable hosting solution that ensures our content is accessible worldwide with minimal latency. It's widely supported by modern web frameworks and integrates seamlessly with our CI/CD pipeline. ## Architecture Overview Our deployment pipeline follows this flow: 1. **Code Changes**: Developers create merge requests in GitLab, triggering the CI pipeline. Nothing terribly exciting here. 2. **Build Stage**: GitLab CI uses Dagger to build the website, leveraging Bun. 3. **Deploy Stage**: Custom Dagger module deploys to Cloudflare Workers 4. **Preview/Production**: Automatic deployment to preview URLs or production ## Notes Ideally, the GitLab CI configuration would be a single job that calls `dagger ci`. However, there's a few challenges at the moment that we need to give time to mature: Dagger caching is not that great yet, we're hoping they release a managed service for this soon. The lack of caching means a `dagger ci` call would likely run everything, even though only a small portion of our repository has changed. ## Dagger Configuration ### Website Build Module Our website build is handled by a Dagger module that ensures consistent builds across environments. We currently duplicate this module across multiple projects, but this won't be required soon as Dagger has a working prototype of [Inline Modules](https://github.com/dagger/dagger/pull/10453) which will hopefully land soon. ```typescript title=".dagger/src/index.ts" import { argument, dag, Directory, func, object, } from "@dagger.io/dagger"; @object() export class Website { /** * Build the website and get the output directory. */ @func() async build( @argument({ defaultPath: ".", ignore: ["node_modules"] }) source: Directory, ): Promise { return dag.bun() .withCache() .install(source) .withMountedFile( "/usr/local/bin/d2", dag.container() .from("terrastruct/d2") .file("/usr/local/bin/d2"), ) .withExec([ "bun", "run", "build", ]).directory("dist"); } } ``` This module: - Uses Bun for fast package management and builds - Includes D2 for diagram generation in our content - Leverages Dagger's caching for faster subsequent builds - Returns a directory containing the built static assets ### Cloudflare Deployment Module We've created a reusable Dagger module for Cloudflare Workers deployments. This module can be used across all projects, it lives at the root of our monorepo. It can be added to projects with: ```shell dagger install ../../dagger/cloudflare ``` We provide two functions in this module: `deploy` for production deployments and `preview` for merge request previews. Both functions use the Cloudflare Wrangler CLI to handle deployments. ```typescript title="/dagger/cloudflare/src/index.ts" import { dag, Directory, File, func, object, Secret, } from "@dagger.io/dagger"; @object() export class Cloudflare { /** * Deploy a website to Cloudflare Workers. */ @func() async deploy( dist: Directory, wranglerConfig: File, cloudflareApiToken: Secret, ): Promise { const cloudflareAccountId = await dag.config().cloudflareAccountId(); const wranglerFilename = await wranglerConfig.name(); const deploymentResult = await dag.container() .from("node:22") .withWorkdir("/deploy") // Only doing this to cache the wrangler installation .withExec([ "npx", "wrangler", "--version", ]) .withMountedDirectory("/deploy/dist", dist) .withMountedFile( `/deploy/${wranglerFilename}`, wranglerConfig, ) .withEnvVariable("CLOUDFLARE_ACCOUNT_ID", cloudflareAccountId) .withSecretVariable("CLOUDFLARE_API_TOKEN", cloudflareApiToken) .withExec([ "npx", "wrangler", "deploy", ]); if (await deploymentResult.exitCode() !== 0) { throw new Error( "Deployment failed. Error: " + await deploymentResult.stdout() + await deploymentResult.stderr(), ); } return deploymentResult.stdout(); } /** * Deploy a preview website to Cloudflare Workers. */ @func() async preview( dist: Directory, wranglerConfig: File, cloudflareApiToken: Secret, gitlabApiToken: Secret, mergeRequestId: string, ): Promise { const cloudflareAccountId = await dag.config().cloudflareAccountId(); const wranglerFilename = await wranglerConfig.name(); const deploymentResult = await dag.container() .from("node:22") .withWorkdir("/deploy") // Only doing this to cache the wrangler installation .withExec([ "npx", "wrangler", "--version", ]) .withMountedDirectory("/deploy/dist", dist) .withMountedFile( `/deploy/${wranglerFilename}`, wranglerConfig, ) .withEnvVariable("CLOUDFLARE_ACCOUNT_ID", cloudflareAccountId) .withSecretVariable("CLOUDFLARE_API_TOKEN", cloudflareApiToken) .withExec([ "npx", "wrangler", "versions", "upload", ]); if (await deploymentResult.exitCode() !== 0) { throw new Error( "Deployment failed. Error: " + await deploymentResult.stdout() + await deploymentResult.stderr(), ); } const allOutput = await deploymentResult.stdout(); return dag.gitlab().postMergeRequestComment( gitlabApiToken, mergeRequestId, allOutput, ).stdout(); } } ``` Key features of this module: - **Branch-based deployments**: Supports both production and preview deployments based on the context (merge request or main branch). - **Wrangler integration**: Leverages Cloudflare's official CLI tool, `wrangler`, for robust deployments. - We use `wrangler deploy` for production deployments. - For preview deployments, we use `wrangler versions upload`, which allows us to deploy a version of the site without affecting the production environment. - **Automatic MR comments**: For preview deployments, it posts the deployment URL as a comment on the merge request, making it easy for reviewers to access the preview environment. ### The Config Module Pattern You might have noticed the line `dag.config().cloudflareAccountId()` in our deployment module. This demonstrates a powerful pattern in Dagger for managing configuration across multiple modules. The `dag.config()` pattern allows us to: 1. **Centralize configuration**: Store all environment-specific values in one place 2. **Avoid hardcoding**: Keep sensitive or environment-specific data out of our modules 3. **Enable reusability**: Share the same modules across different environments Here's how we implement our config module: ```typescript title="/dagger/config/src/index.ts" import { func, object } from "@dagger.io/dagger"; @object() export class Config { /** * Get the Cloudflare account ID for deployments */ @func() cloudflareAccountId(): string { // In production, this could read from environment variables // or fetch from a secret management system return "your-cloudflare-account-id"; } /** * Get the GitLab project ID for API calls */ @func() gitlabProjectId(): string { return "your-gitlab-project-id"; } } ``` This config module is installed in our repository root and can be accessed by any other module via `dag.config()`. This pattern is particularly useful when: - Managing different environments (dev, staging, production) - Sharing configuration across multiple deployment modules - Keeping sensitive data separate from your module logic To use this pattern in your own projects: 1. Create a config module at your repository root 2. Install it: `dagger install ./dagger/config` 3. Access it from any module: `await dag.config().yourConfigMethod()` ## GitLab CI Pipeline Our GitLab CI configuration orchestrates the entire deployment process. Firstly, we configure some base jobs that will be used by all other jobs. This includes setting up Docker-in-Docker for building images and installing Dagger. ```yaml title=".dagger/.gitlab-ci.yml" .docker: image: docker:latest services: - name: docker:${DOCKER_VERSION}-dind command: ["--registry-mirror", "https://mirror.gcr.io"] variables: DOCKER_HOST: tcp://docker:2376 DOCKER_DRIVER: overlay2 DOCKER_VERSION: "27.2.0" DOCKER_TLS_VERIFY: "1" DOCKER_TLS_CERTDIR: "/certs" DOCKER_CERT_PATH: "/certs/client" before_script: - until docker info > /dev/null 2>&1; do sleep 1; done .dagger: extends: [.docker] before_script: - apk add ca-certificates curl git - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh ``` Next, we define the deploy and preview jobs for each project. These are orchestrated with Dagger Shell, which is a really nice way to interact with Dagger without having to build custom modules. ```yaml title=".dagger/.gitlab-ci.yml" website-deploy-production: extends: [.dagger] stage: deploy environment: name: rawkode-academy-production url: https://rawkode.academy rules: - if: '$CI_COMMIT_BRANCH == "main"' changes: - projects/rawkode.academy/website/**/* script: - cd projects/rawkode.academy/website - | dagger <<. . | build | export dist ../../../dagger/cloudflare | deploy \ ./dist \ wrangler.jsonc \ env:CLOUDFLARE_WORKERS_TOKEN . website-deploy-merge-request: extends: [.dagger] stage: deploy rules: - if: "$CI_MERGE_REQUEST_ID" changes: - projects/rawkode.academy/website/**/* script: - cd projects/rawkode.academy/website - | dagger <<. . | build | export dist ../../../dagger/cloudflare | preview \ ./dist \ wrangler.jsonc \ env:CLOUDFLARE_WORKERS_TOKEN \ env:CICD_COMMENT_TOKEN \ $CI_MERGE_REQUEST_ID . ``` Dagger Shell has full access to modules and their dependencies, enabling us to build and deploy our website with a single command. The `build` function compiles the website which can be exported and stored within a variable; while the `deploy` and `preview` functions handle the deployment to Cloudflare Workers and take our export/variable as input. ## Local Development Developers can run the exact same build process locally: ```bash # Build the website locally # (Assuming you are in the 'projects/rawkode.academy/website' directory) dagger . build # Deploy to a preview environment (requires tokens) # This command also assumes you are in the 'projects/rawkode.academy/website' directory. # It uses 'dagger call' to invoke the 'deploy' function in the shared Cloudflare Dagger module. # The path '../../../dagger/cloudflare' points to this module relative to the current directory. dagger call -m ../../../dagger/cloudflare deploy \ --dist ./dist \ --wrangler-config wrangler.jsonc \ --cloudflare-api-token env:CLOUDFLARE_PAGES_TOKEN ``` ## Security Considerations - **Secret Management**: All sensitive tokens are managed through GitLab CI variables. This ensures that secrets are not hardcoded in the codebase, reducing the risk of accidental exposure. We use GitLab's built-in secret management to securely store and access tokens, API keys, and other sensitive information. - **Branch Protection**: Production deployments only happen from the main branch. This practice ensures that only thoroughly reviewed and tested code reaches production. We enforce branch protection rules to prevent direct pushes to the main branch, requiring all changes to go through a merge request process. - **Access Control**: Cloudflare API tokens are scoped to specific resources. This principle of least privilege ensures that each token has the minimum permissions necessary to perform its function, reducing the potential impact of a compromised token. We regularly review and rotate tokens to maintain security. - **Environment Isolation**: We use separate environments for development, staging, and production. This isolation helps prevent accidental changes to production systems and allows for thorough testing before deployment. - **Regular Audits**: We conduct regular security audits of our CI/CD pipeline and infrastructure. This includes reviewing access controls, secret management practices, and deployment processes to identify and address potential vulnerabilities. ## Performance Optimizations 1. **Docker Layer Caching**: The pipeline uses Docker-in-Docker with proper caching 2. **Dagger Caching**: Build dependencies are cached between runs 3. **Change Detection**: Deployments only trigger when relevant files change 4. **Parallel Execution**: Multiple projects can deploy simultaneously ## Conclusion This Dagger + GitLab + Cloudflare Workers setup provides us with a robust, fast, and developer-friendly deployment pipeline. The combination of containerized builds, automatic preview environments, and integrated notifications creates an excellent developer experience while maintaining production reliability. The modular approach with Dagger also means we can easily extend this pipeline to support additional deployment targets or build processes as our needs evolve. This flexibility ensures that our deployment pipeline can grow and adapt with our projects, making it a future-proof solution. If you're looking to modernize your deployment pipeline, this stack offers a great balance of simplicity, power, and developer experience. By adopting this setup, you can achieve faster, more reliable deployments, reduce manual overhead, and focus on what truly matters—building great software. We're excited about the possibilities this pipeline opens up and look forward to seeing how it can help other teams achieve similar results. The future of CI/CD is here, and it's more powerful and developer-friendly than ever before. --- # Federated GraphQL for Microservice Architecture > Learn how federated GraphQL simplifies data access across service boundaries in a microservice architecture. - Canonical: https://rawkode.academy/read/federated-graphql-microservices - Published: 2025-03-24 - Authors: David Flanagan - Technologies: wundergraph, dgraph, kubernetes In this article, we'll explore the design and implementation of our federated GraphQL API. This API serves as a single, unified interface to our microservice architecture, allowing clients to query and retrieve data from multiple services in a single request. If you're unfamiliar with GraphQL federation, it's an architectural pattern that combines multiple GraphQL schemas into a unified graph, enabling seamless data access across service boundaries. We'll cover the benefits of using a federated GraphQL API, the challenges we faced during implementation, and the tools and techniques we used to build and deploy our solution. You'll see practical examples of how clients can interact with our API and how it dramatically simplifies the process of querying data from across multiple microservices. By the end of this article, you'll understand how federated GraphQL APIs work and how they can create a more efficient and flexible API layer in a microservice architecture. ## Why Federated GraphQL? In a microservice architecture, multiple services are responsible for different domains of an application. Each service has its own database and API, which can lead to clients needing to make multiple requests to different services to retrieve all necessary data. A federated GraphQL API solves this problem by providing a single, unified interface to multiple services. Clients send a single GraphQL query to the federation layer, which handles fetching data from the appropriate services and aggregating it into a cohesive response. This approach offers several significant benefits: - **Reduced network requests**: Clients make just one request to the federated API instead of multiple requests to different services, reducing latency and bandwidth usage. - **Simplified client code**: Developers write a single GraphQL query to retrieve data across service boundaries, eliminating complex data-fetching orchestration code. - **Centralized data fetching logic**: The federation layer manages complex data retrieval patterns, including cross-service relationships, without burdening client applications. - **Enhanced caching**: The federation layer can implement intelligent caching strategies, reducing load on downstream services and improving response times. - **Incremental adoption**: Teams can add services to the federation incrementally, without disrupting existing functionality. ## Example To understand our API and its federation, it makes sense to first see an example query. ```graphql query { getLatestVideos(limit: 2) { id # videos-service title # videos-service creditsForRole(role: "host") { person { forename # people-service surname # people-service biography # people-biographies-service links { url # people-links-service } } } likes # video-likes-service } } ``` As you can see from the comments above, this single query is fetching data from multiple services. The `getLatestVideos` field is from the `videos-service`, and it's returning the `id` and `title` of the video. We're also fetching the `creditsForRole` from the `people-service`, the `biography` from the `people-biographies-service`, and the `links` from the `people-links-service`. Finally, we're fetching the `likes` from the `video-likes-service`. ### Adding Columns with a Service?! Yes! It's pretty common to need to add new fields to an existing entity. Now, you could modify your micro-service and add a new column, but this requires a database migration ... which is usually OK; right? 😅 What if we can avoid modifying a service, writing a database migration, and potentially invalditing all presently existing backups? 🤔 Well, we can. We can write a new service that persists the new column, all on its own and use some GraphQL magic to stitch it all together. This approach allows you to be more agile, take risks, and be experimental without impacting the stability of your existing services. Added a new column via a service and want to get rid of it? Easy done. Just remove the service and the column is gone. ## Our Federated GraphQL Stack Our federated GraphQL API is built using Apollo Federation's specification, but we've chosen to implement it using a fully open-source stack. ```d2 sketch API Client -> WunderGraph Cosmo WunderGraph Cosmo -> Service A WunderGraph Cosmo -> Service B Service A -> GraphQL Yoga A GraphQL Yoga A -> Drizzle ORM A Drizzle ORM A -> libSQL / Turso A Service B -> GraphQL Yoga B GraphQL Yoga B -> Drizzle ORM B Drizzle ORM B -> libSQL / Turso B ``` ### GraphQL Yoga [GraphQL Yoga](https://the-guild.dev/graphql/yoga-server) serves as our GraphQL server, chosen for its: - **Flexibility**: Runs anywhere JavaScript can run, including Cloudflare Workers and Deno (our preferred environment over Node.js) - **Simplicity**: Offers straightforward setup with sensible defaults - **Extensibility**: Supports a rich plugin ecosystem - **Federation support**: Natively implements Apollo Federation specifications ### Pothos [Pothos](https://pothos-graphql.dev/) is our TypeScript schema builder, selected after evaluating several alternatives. Key advantages include: - **Type safety**: Provides end-to-end type safety from database to GraphQL schema - **Code-first approach**: Enables us to define schemas in TypeScript rather than SDL - **Plugin architecture**: Offers modular functionality including direct Drizzle ORM integration - **Developer experience**: Excellent autocompletion and type inference ### WunderGraph Cosmo [WunderGraph Cosmo](https://wundergraph.com) manages our federated graph lifecycle, offering: - **Composition checks**: Validates schema changes against the complete federated graph - **Router**: Efficiently directs queries to the appropriate subgraphs - **Analytics**: Provides insights into API usage patterns - **Distributed tracing**: Helps identify performance bottlenecks across services - **Schema registry**: Maintains a history of schema changes and enables rollbacks - **Caching**: Implements intelligent caching strategies to improve performance, at the service, row, and even column level. ## Building Our Federated GraphQL API To demonstrate the power of our approach, let's walk through a real-world example: adding a new field to an existing entity without modifying the original service. We'll add thumbnail capabilities to videos by creating a dedicated thumbnails service. ### Defining the Database Schema We use [Drizzle ORM](https://orm.drizzle.team/) for all our database interactions, providing type-safe queries and automated migrations. Here's the existing schema from our `videos` service: ```typescript import { createId } from '@paralleldrive/cuid2'; import { integer, sqliteTable, text } from 'drizzle-orm/sqlite-core'; export const videosTable = sqliteTable('videos', { id: text('id').primaryKey().$default(createId), title: text('title').notNull(), subtitle: text('subtitle').notNull(), slug: text('slug').notNull().unique(), description: text('description').notNull(), duration: integer({ mode: 'number' }).notNull(), publishedAt: integer({ mode: 'timestamp' }).notNull(), }); ``` Now, we'll create a new service, `thumbnails`, with its own schema: ```typescript import { sqliteTable, text } from 'drizzle-orm/sqlite-core'; export const thumbnailsTable = sqliteTable('video-thumbnails', { // References the video ID from the videos service id: text('id').notNull().unique(), // URL to the thumbnail image url: text('url').notNull(), }); ``` This simple schema allows us to store thumbnail URLs associated with video IDs. Drizzle's `drizzle-kit` handles migration generation and application for us. Drizzle also has a wonderful plugin ecosystem, allowing us to convert out Drizzle types to many other formats, as well as defining DTOs for other APIs based on the database schema. ### Defining the GraphQL Schema Next, we transform our database schema into a GraphQL type using Pothos with its Drizzle plugin. The key to federation is how we reference and extend existing types. As our `Video` type is defined in the `videos` service, we need to reference it in our new service to be able to extend it with our new columns / fields. ```typescript import drizzlePlugin from '@pothos/plugin-drizzle'; import { eq } from 'drizzle-orm'; import * as dataSchema from './schema.ts'; // Create a GraphQL schema builder with Drizzle plugin const builder = new SchemaBuilder({ plugins: [drizzlePlugin()], }); // Reference and extend the Video type from another service builder.externalRef( 'Video', builder.selection<{ id: string }>('id'), ).implement({ // The 'id' field is defined in the original videos service externalFields: (t) => ({ id: t.string(), }), // Add our new thumbnail field to the Video type fields: (t) => ({ thumbnail: t.field({ type: 'String', nullable: true, resolve: async (video) => { // Use Drizzle to query the thumbnail for this video const result = await db.query.videoThumbnails.findFirst({ columns: { url: true, }, where: eq(dataSchema.videoThumbnails.id, video.id), }); return result?.url || ''; }, }), }), }); ``` The magic of federation happens in these few lines. We're: 1. Declaring that we know about an external `Video` type 2. Specifying that we need the `id` field to resolve references 3. Adding our new `thumbnail` field to the existing `Video` type 4. Providing a resolver that fetches the thumbnail URL from our database This approach follows Apollo Federation's [entity resolution pattern](https://www.apollographql.com/docs/federation/entities/), allowing the federation layer to "stitch together" data from multiple services. ### Publishing the Schema Changes After defining our schema, we need to register it with WunderGraph Cosmo. First, we generate an SDL representation of our schema: ```typescript import { printSchemaWithDirectives } from '@graphql-tools/utils'; import { lexicographicSortSchema } from 'graphql'; import { getSchema } from './schema.ts'; // Convert our code-first schema to SDL format const schemaAsString = printSchemaWithDirectives( lexicographicSortSchema(getSchema()), { pathToDirectivesInExtensions: [''], }, ); // Write the schema to a file Deno.writeFileSync( `${import.meta.dirname}/schema.gql`, new TextEncoder().encode(schemaAsString), ); ``` Then, we publish this schema to Cosmo using the WunderGraph CLI: ```bash bunx wgc subgraph publish ${SERVICE_NAME} \ --namespace production \ --schema ./read-model/schema.gql \ --routing-url https://${SERVICE_NAME}-read-458678766461.europe-west2.run.app ``` Cosmo then: 1. Validates our schema against the existing federation 2. Checks for breaking changes 3. Updates the router configuration 4. Notifies us of any issues or successful completion ## Why We Love This Approach This architecture has transformed how we build and expose our services: **Incremental evolution**: We can add fields and capabilities without modifying existing services, enabling parallel development by multiple teams. **Type safety throughout**: With Drizzle and Pothos, we have end-to-end type safety from database to API, catching errors at compile time rather than runtime. **Developer autonomy**: Teams can evolve their domains independently while maintaining a cohesive API for consumers. **Operational visibility**: WunderGraph Cosmo provides comprehensive monitoring, alerting, and tracing, making it easier to identify and resolve issues. **Read-write separation**: By focusing our federation on read operations, we've simplified our architecture while maintaining clear command responsibility in individual services. ## Challenges We'll be honest, we've not hit any huge problems yet with this architecture; it's allowed us to move fast and make mistakes without compromising the system. We will admit that the setup of a new service is currently a lot of copy and paste and duplicate code within our repository that could be easily generated. We're actively exploring the [Projen](https://projen.io/) project to help us build out a template for read-layer APIs that are generated when you run `bun` or `deno` install. This will allow us to remove the duplicated code and each read-layer service would be just the Drizzle objects and the GraphQL extension points; nice, right? ## Conclusion Federated GraphQL has proven to be a powerful solution for our microservice architecture, providing a unified API that simplifies client development while preserving service autonomy. By leveraging open-source tools like GraphQL Yoga, Pothos, and WunderGraph Cosmo, we've built a robust, scalable system that supports rapid development across multiple teams. If you're facing challenges with API integration in a microservice environment, we highly recommend considering a federated GraphQL approach. The initial investment in setting up the federation layer pays dividends in development speed, code quality, and system performance. Thanks for reading, we promise not to leave it as long next time for the next article. We'll be back with you right after KubeCon London, 2025. --- # The Rawkode Academy Architecture > A high-level overview of the Rawkode Academy platform architecture, exploring the innovative patterns and techniques that power our cloud-native platform. - Canonical: https://rawkode.academy/read/our-architecture - Published: 2024-12-13 - Authors: David Flanagan - Technologies: restate, rust, kubernetes Welcome to the Rawkode Academy! We're thrilled to start talking about our new platform, designed to provide developers with an immersive and engaging learning experience. In this article, we'll take you behind the scenes, exploring the innovative patterns and techniques we've employed to build a robust, scalable, and cloud-native platform. From CQRS and GraphQL Federation to Restate and our unique approach to service-oriented architecture, we'll delve into the key components that power our platform. It's my job to make content that helps developers learn stuff. That stuff has varied over the last ten years and I assume that will continue to be the case. Which appeals a lot for me, because I like shiny things. So when I decided to remove my dependency on YouTube and build my own platform, I decided to have some fun. Let me introduce you to the different patterns and techniques we've been using to build out the platform. This article will be a little high level, but stay tuned! Over the next few weeks we'll be going deeper into each pattern and technique, with code samples. Too impatient? Check out the [repository](https://github.com/RawkodeAcademy/RawkodeAcademy) today. --- ## Cloud Native Given that I founded the Rawkode Academy to help make Cloud Native, a vast and ever changing landscape, easier for developers; it's only fitting that whatever I build for the Rawkode Academy is built as Cloud Native as it can be. As such, the Rawkode Academy platform is built as a set of micro-services. Some are less micro and some are nano, so let's just settle on that this is a service-oriented architecture with it sometimes taking some liberties and sometimes going to extremes. We also adopted a hybrid-cloud approach, leveraging bare metal, managed Kubernetes, and serverless. Depending on what part of the platform you interact with; you'll be hitting either: - **Website:** Cloudflare Pages + Workers - **GraphQL API:** Cloud Run - **Video Encoding:** Bare Metal on Equinix Metal - **Auth:** GKE AutoPilot * We use GKE AutoPilot (with Spot) for consistently running services, such as [Zitadel](https://zitadel.com). I've taken a few liberties with the arrows on the diagram below (on the write path) in-order to have the diagram be more readable. All write requests, and some read (such as people) go through Zitadel for AuthN. ```d2 sketch # System architecture in D2 Website: Website Cosmo: Cosmo GraphQL Gateway Website -> Cosmo # GraphQL Read Services PeopleGraphQL: People Read ShowsGraphQL: Shows Read ShowHostsGraphQL: Show Hosts Read VideosGraphQL: Video Read TechnologiesGraphQL: Technologies Read Cosmo -> PeopleGraphQL Cosmo -> ShowsGraphQL Cosmo -> ShowHostsGraphQL Cosmo -> VideosGraphQL Cosmo -> TechnologiesGraphQL # Databases PeopleDB: People Database ShowsDB: Shows Database ShowHostsDB: ShowHosts Database VideosDB: Videos Database TechnologiesDB: Technologies Database PeopleGraphQL -> PeopleDB ShowsGraphQL -> ShowsDB ShowHostsGraphQL -> ShowHostsDB VideosGraphQL -> VideosDB TechnologiesGraphQL -> TechnologiesDB # Restate RPC Gateway Restate: Restate RPC Gateway # Restate Write Services PeopleRestate: People Write ShowsRestate: Shows Write ShowHostsRestate: Show Hosts Write VideosRestate: Video Write TechnologiesRestate: Technologies Write PeopleRestate <-> Restate ShowsRestate <-> Restate ShowHostsRestate <-> Restate VideosRestate <-> Restate TechnologiesRestate <-> Restate PeopleDB <-> PeopleRestate ShowsDB <-> ShowsRestate ShowHostsDB <-> ShowHostsRestate VideosDB <-> VideosRestate TechnologiesDB <-> TechnologiesRestate # Optional styling (uncomment and modify as needed) # direction: right # Website.style.fill: lightblue # Cosmo.style.fill: lightgreen # Restate.style.fill: lightyellow ``` ## Command Query Responsibility Segregation (CQRS) I've been a huge fan of the work of [Udi Dahan](https://udidahan.com/) and Greg Young for over ten years, even implementing my own CQRS & Event Sourcing libraries in a variety of languages (I actually used this as my project to learn new languages). Event Sourcing wasn't required for the platform, but CQRS is a fantastic pattern to implement in almost any project. It allows you to separate the read and write models for your application. In a strict CQRS fashion, these would likely actually have different views / data stores / projections; but we don't need to adopt this quite yet. Our read API is powered by GraphQL, with federation, and our write API is powered by [Restate](https://restate.dev). ## GraphQL Federation We use a single GraphQL Gateway, powered by [WunderGraph Cosmo](https://cosmo.wundergraph.com), which can build a query plan for your GraphQL query. This query plan is realised by individual GraphQL APIs, to the object or field level, provided by each service. This allows us to have a single endpoint for all our read operations, while still allowing each service to own its data and schema. This is a fantastic pattern for a service-oriented architecture. The icing on the cake for us with this pattern is that with GraphQL Federation we can actually extend the read model and its schemata with new fields, joins, and pretty much anything we need; all without dealing with database migrations. We'll be diving into this in great detail on our next article. ## Restate We use Restate as a HTTP RPC layer with built-in queues and durable execution which provides commands to write data to our, currently, shared data stores. Currently, each service shares a data store for both reads and writes. We prioritize write optimization through our RPC commands, as read optimization is not yet a concern. We'll dive into our RPC and Restate in a future article, but here's a quick example of how we use it. We used Restate, with its durable execution, as writing to our platform is a complex task. Due to the service-oriented architecture, scheduling a new live stream is a multi-part process; but we want a simple command to do so. Imagine the following write: ```shell curl -XPOST https://rpc.rawkode.academy/live-stream/new -d '{ "title": "Overview of the Rawkode Academy Architecture", "hosts": [ "rawkode" ], "guests": [ "icepuma" ], "when": "2024-12-23Z10:00:00:Z", "description": "BLAH", "technologies": ["Restate", "Rust", "TypeScript", "GraphQL", "CQRS"] }' ``` This is not a complete example, but to understand the complexity we need to understand this such a request is a [SAGA](https://microservices.io/patterns/data/saga.html). We need to write multiple entities to multiple services while also ensuring referential integrity across these boundaries. - Can we schedule a live stream for a date in the past? **No**. - Can we schedule an episode with host `rawkode` if that identifier doesn't exist in the `people` service: **No**. - Can we schedule an episode with guest `icepuma` if that identifier doesn't exist in the `people` service: **No**. - Same for technologies - and so forth We aim to prevent write failures by implementing checks and balances. While eventual consistency is expected in a distributed system, these measures protect the integrity of the read model. Again, stay tuned; there's lots of examples and code we can share to dive into this deeper. --- In this article, we've explored the architecture and design choices behind the Rawkode Academy's new platform. We highlighted our commitment to a cloud-native, service-oriented approach, leveraging technologies like Cloudflare Pages, Cloud Run, Equinix Metal, and GKE AutoPilot. We also shallow dived into our implementation of CQRS, GraphQL Federation, and Restate, explaining how these patterns contribute to a robust and scalable platform. Our use of Restate for durable execution and handling complex write operations, particularly in the context of SAGAs, ensures data integrity and consistency across our distributed services. We've also touched on the importance of eventual consistency in such an environment. This high-level overview provides a glimpse into the underlying technologies and principles driving the Rawkode Academy platform. Stay tuned for upcoming articles where we'll deep dive into each component with detailed explanations and code samples. Don't want to miss the new articles? Follow our [BlueSky](https://bsky.app/profile/rawkode.academy) account to keep up to date with the platform build out and our new articles. Until next time! --- # Zed: Show & Tell > Explore Zed, a fast and modern code editor written in Rust with collaborative features - Canonical: https://rawkode.academy/read/zed-show-and-tell - Published: 2024-07-05 - Authors: Stefan Ruzitschka - Technologies: rust A common habit of developers is searching for new tools on a regular basis. Visual Studio Code is such a tool which helped me over the years write some awesome code. But I always wanted to try something new. So I stumbled upon Zed. ## So what is Zed? Zed is a fairly new code editor written in Rust by the awesome folks which created tools like [Atom](https://github.com/atom/atom). It got open sourced in the beginning of 2024. Zed aims to be a fast, collaborative and hackable editor. But have a look for yourself [here](https://github.com/zed-industries/zed). I personally really like that it comes with a lot of [extensions](https://github.com/zed-industries/zed/tree/main/extensions) and ["vim mode"](https://zed.dev/docs/vim). ## Installation You could build Zed either from source or download a pre-compiled binary (only macOS is currently supported). I personally used `brew` to install Zed: `brew install zed` ## Let's open a project The first thing you see is an empty window. I typically use `⌘ + P` to open the file finder and navigate to my file of choice. Here you see the file finder in action. ## Working with Rust Zed is "language aware" and detected that I opened a Rust file and applies the correct syntax highlighting. The [Rust extension](https://zed.dev/docs/languages/rust) is shipped with Zed by default. The extension uses [rust-analyzer](https://github.com/rust-lang/rust-analyzer) for things like code completion, running tests and [tree-sitter-rust](https://github.com/tree-sitter/tree-sitter-rust) for syntax highlighting. It also added indicators for Rust tests, so that I can run them directly from the editor. Let's try it out :) After clicking the "Run" button (the little "play" icon next to the test function), the test output is displayed in the built-in terminal at the bottom of the window. Let's do some refactoring. I open the `editor: rename` option (via Command Palette) by pressing `⌘ + Shift + P`, change the name of the variable and see all changes in some kind of preview window. ## Conclusion Coming from Visual Studio Code, knowing its key bindings and what not, the transition to Zed was quite a smooth ride. I used it for 2 weeks to write some Rust and TypeScript and did not miss too much. I will definitely stick with Visual Studio Code for now, because some extensions e.g. [zed-sql](https://github.com/evrensen467/zed-sql) is still behind what the Visual Studio Code ecosystem offers. Things I haven't yet tried are the AI integration and the collaboration features, but I will definitely give them a try in the future. I'm very happy with the overall experience so far and I'm looking forward to a bright future for Zed. --- # How to ask for help > Master the art of asking technical questions that get helpful responses - Canonical: https://rawkode.academy/read/how-to-ask-for-help - Published: 2023-10-31 - Updated: 2023-11-10 - Authors: Russell Waite - Technologies: kubernetes When you are asking a bunch of strangers for help on the internet, there are tricks to get people to be more willing to help you. Unless you are paying for support, you need to make it as easy as possible for volunteers to help you. Here's some advice to help people help you. ## Make it Easy to Read Learn the system you're writing on so you can convey the information clearly. Having to read badly formatted text is not a nice introduction to your problem. Separate your question from the log outputs, most systems will have a way to enter code. Use that code format for logs, code, config, etc. Keep each type of sample separate, or at least spaced out well enough, with a comment to show what it is. ### Example ```plaintext title="dmesg log with error" [331337.438088] hub 4-0:1.0: 2 ports detected [331337.465300] ACPI: bus type thunderbolt registered [331337.612428] thunderbolt 0000:08:00.0: 0: DROM device_rom_revision 0x0 unknown [331337.613424] thunderbolt 0-0: ignoring unnecessary extra entries in DROM [331342.334285] ieee80211 phy0: brcmf_inetaddr_changed: fail to get arp ip table err:-52 [331357.956017] xhci_hcd 0000:3e:00.0: Unable to change power state from D3cold to D0, device inaccessible [331357.956034] xhci_hcd 0000:3e:00.0: Controller not ready at resume -19 [331357.956035] xhci_hcd 0000:3e:00.0: PCI post-resume error -19! [331357.956037] xhci_hcd 0000:3e:00.0: HC died; cleaning up # partial results from lspci 00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1) 00:1c.1 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #2 (rev f1) ``` ## Reproducible Issue Where possible create a minimal reproduction of the issue. This lets people try recreate the issue with your config and prevents you from leaking secrets that might need to stay secret. Unless you are certain the problem exists in just one file, make people's lives easier by giving as much info as possible. The best way to do that is a cut down, reproducible sample. ## Explain Environment Explain the environment you are running in, it may well be an environmental issue you are facing. Things like what hardware, software and config (where you can) it's running - i.e. Docker version 24.0.2, build cb74dfcd85 on a laptop running Debian 11.7 stable, an AWS VM `m7g.large` running Amazon Linux, etc. ## Make it Quick & Easy for People Be conscious of the amount of time people have to spend to get up to speed. This ties in with the reproducible sample. Don't post a link to an instruction you followed. If you do that and nothing else you are expecting people to read through that article which takes precious time. Even after reading it, unless there is an obvious mistake in the article, it may not assist you as chances are you've done something different to the article. Maybe you missed a step, maybe you made a typo, used different hosting options, used different versions of software. Feel free to add that you followed an article, but don't expect people helping you to have to read that article before getting started. ## Explain Your Steps so Far State what you have tried and what you think it might be, there is nothing less rewarding than helping someone who's tried nothing for themselves and given up. ## Provide Useful Information If someone is helping you, please reply with more than just "that doesn't work". Show how you followed their advice and what the result was. If you aren't paying in cash for the help, pay in effort, show people you've done as they asked and the exact result of what happened. ## Make Single Changes Change one thing at a time. If more than one person responds, don't apply more than one suggestion at a time. You need to see what each change does. The exception is when you are explicitly told to apply more than one fix as they can see you have more than one issue. Even then though, it's just good practice to make a single change and see what it does.