1:10 Introduction and Housekeeping

1:10 Hello, and welcome to another episode of Rawkode Live at the Rawkode Academy. My name is David Flanagan, and I am your host. You may know me as Rawkode across the Internet. First, we got a little bit of housekeeping before we dive into project sigstore supply chain for Kubernetes containers and other things. Now the housekeeping involves, first, subscribe into the channel. If you're not already subscribed to the Rawkode Academy, please do so now. Tick the bell, and you will get notifications and alerts for all new episodes as we explore the cloud native landscape together. If you wanna join and chat with other

1:15 Introductions

1:45 members of the Rawkode Academy about all things cloud native, Kubernetes, and everything in between, join us on the Discord at Rawkode.chat. And if you want to support this channel, please take a look at the membership options options available at the Rawkode Academy on the YouTube page. You can subscribe for as little as a single dollar a month and get loads more cool goodies. Nailed the intro. Thanks. I'm getting assaulted already before we even started. Thanks, man. I do my best. Alright. Today, we're taking a look at sigstore. I am not smart enough or wise enough

2:18 to guide us through this journey, so I am going to introduce Dan Lawrence. Hey, man. How's it going? Great. How are you? Well, I mean, I was better until I fluffed the intro a couple of times, but other than that, I'm doing pretty well. Can you, for the audience, if they're not familiar, just tell us a little bit about you, please? Sure. Yeah. I'm a software engineering lead at Google. I work on our open source security team. So my job for the past couple of years has been trying to make it easier to build open source software securely and make

2:21 Guest Introduction: Dan Lawrence

2:47 it easier for Google and all of our customers to use it securely and not have to worry about these nasty supply chain attacks that keep happening in there all over the news. Nice. So we're gonna be taking a look at sigstore and the multiple components of it today. Like so how would you describe the project to people? Yeah. It's a good question. So sigstore is a little complicated of a project. It's got a bunch of different components and a bunch of different entry points depending on exactly what type of software you're trying to build or consume or anything like that. So we're

3:06 What is sigstore? (Comparison to Let's Encrypt)

3:19 gonna start with all of these intros with the cosign tool. The cosign tool is spelled c o s I g n, and it's designed to make signing and verification of container images really, really easy. Here, we'll start there because this is a cloud native stream. Mean, containers and Kubernetes should be pretty familiar to most people. All the different components, we'll kind of explain as we get through them, going through some of the demos. But at a high level, the overall goal for sigstore is to kind of do what Let's Encrypt did for HTTPS and certificates,

3:49 but for open source code and signing instead of encryption and web traffic. So Let's Encrypt set out a few years ago to make it easy and free to get SSL certificates because before that, they were expensive and hard to get, and so people weren't doing it. So web traffic was insecure. They made it free, easy, and automatic, and now something like 98% of all web traffic is encrypted because people wanna do the right thing if it's easy and free. We see the same thing in open source code and software supply chains today where there are some

4:19 kind of existing code signing systems, but they're really expensive. It's like $500 to get a certificate to sign your code to ship it for different operating systems and platforms. So most people just don't do it, especially in large open source projects where, you know, there's not even a business that could be verified and that kind of thing behind these projects. So we've got a certificate authority that we're gonna be playing around with today to get some code signing certificates issued based on your email address. There's a bunch of different tools inside of cosign two to assign images with

4:45 YubiKeys and KMS systems and Kubernetes secrets and all that fun stuff. So depending on exactly how you're getting the image, who's producing it, who's signing it, we've got tools and flags and features for all of that here that we'll be walking through, and then we'll be explaining how kind of the sigstore infrastructure makes all of this possible. Awesome. Quite a lot there. We could probably dive into. But I think the the first thing I wanted to see, like, what the fuck the deal with prices of search ten years ago? I mean, I remember working for sorry. Don't mean to

5:15 swear. It's only afternoon here. But I remember working, you know, ten years ago, even longer, and you had to go buy all of these search, and they were, like, $500 plus if you went with a reputable company to get your website delivered over SSL as it was back then. But and now it's just something that if anyone new to development has less encrypt available, it's just an absolute game changer. So it's really cool. I won't complain about I I won't run about EV certs either. I'll leave that for another day. But And, actually, the EV certs are the super

5:38 The Problem: Software Supply Chain Attacks

5:46 expensive ones. That's kinda what you need to get for doing code signing today. So it's it's really bad. Yeah. But we are talking about supply chain. Right? And that's one of the I I think we've seen that a lot over the last twelve months. I mean, how many supply chain checks I mean, I only know what I've seen in the news, which is probably the more high profile one. But I'm assuming you may have more visibility into what's actually going on in the ecosystem right now. Like, this is a really prominent problem. Right? Yeah. They've they've been happening for years.

6:13 They've been accelerating, I think, for a couple reasons. Open source is getting more and more widely used, which is great for open source, but more companies are using it. Not all open source has a lot of people working on it, a lot of people looking at it, unfortunately. There's kind of this misconception that if it's on GitHub and the project has a bunch of stars, then there are gonna be no bugs inside of it. As maintainer of a bunch of projects with a bunch of GitHub stars, I can tell you that's incorrect. All of my code has

6:37 bugs. All of the code you're using has bugs in it. And so, yeah, the the supply chain attacks either through, you know, bugs that are put in unintentionally or the scarier ones where people kind of take over GitHub accounts. They steal credentials to container registries are are coming. They're coming quick. Thankfully, there are actually some huge ones over late last week over the weekend that thankfully were a researcher who reported them rather than an attacker. But somebody threw some misconfigured GitHub actions, managed to get write permissions to the entire PyPy package index. Oh, damn. Could've tampered with

7:11 change any of those packages in there. They're not really signed today, so nobody would have known. Thankfully, he reported it and got these issues fixed, got the GitHub actions locked down a little bit, but that would have been a really, really scary one. There's a similar one. I think it was even the same person actually with Microsoft Versus Code's GitHub repository. Got write permissions to branches there that would have triggered releases and pushed out, you know, malicious versions of Versus Code. Anybody that's kind of thinking about this and paying attention now would realize how terrifying and

7:40 scary that would be if somebody compromised the IDE that 99% of other developers use, you know, what kind of backdoor can they get put into all the code build? That's why I only had a code with said. So you know? I just write it on a piece of paper and hand it to somebody else. Nice. Alright. That was terrifying. There's now been, you know, executive orders in The United States to help start locking a lot of this stuff down and securing it because governments are getting hit with ransomware. A lot of scary things. Yeah. It is very scary.

8:09 And sigstore and its components help alleviate a lot of these concerns. Is that is that the best way to describe it? Good idea. Yeah. I think when we talk about open source security, there are kind of two high level problems that I think are orthogonal. They're they're different, and we've got to address both of them. The first one is that all the code is open source, which is great, but it means, you know, anybody's writing it, anybody's working on it. You might not have any idea who the maintainers are. That's pretty cool in some ways, but it's

8:16 How sigstore Helps

8:41 also scary because anybody that spent time on the Internet knows that not everybody on the Internet is a nice person. So you're taking code from, you know, potentially bad people and running it. Even worse, you don't even know where the code is coming from for the most part. If your Docker pulls some container, you have no idea where that came from. If the code even actually came from the repo, will it it's paired to the, you know, Docker registry or anything. And so that's kind of around transparency and not actually understanding what our supply chains are.

9:05 And that's the angle we're trying to address with sigstore. So if you've got a container, you can figure out exactly how it was built, what code went into it, what build steps were run, that kind of thing. That doesn't make it secure. It just means you understand the supply chain. That's kind of the first step in securing your supply chain. There could still be malware checked into that GitHub repo. But today, you can't even find the GitHub repo to look for malware in most of these cases. So the second half of open source software and security is that it's software, it has

9:34 bugs, some intentional, some unintentional. So sigstore isn't gonna secure everything completely. Right? There could still be bugs. There could still be CVEs, but at least we're hoping to give you the tools to understand what versions of code you're running, where it came from, that kind of thing. So you can start addressing that second problem. Okay. Let me try and phrase this in my own words so I kind of understand Sure. Where sigstore can fit. Right? But I like to think of it as the problem. Like, we all consume open source code in our projects. And what we can do is

10:00 we can audit the code that we use and never update it, which is usually the enterprise approach. Right? Or we can always be pulling in the latest one and trying to avoid all those security conflicts, and there's very little in the middle. Does sigstore help us bridge these two different approaches and have more confidence and frequently updating our packages and our dependencies? That's a really, really good way to put it. Yeah. And it that's controversial, it comes up a lot about, you know, that strategy of how often should I be updating and that kind of thing.

10:29 I'm on the the opinion that, yeah, you've gotta be updating frequently. It does put you at risk. In some cases, it's a lot of work. It's tough, but I think you just have to be doing these updates. Otherwise, you're behind. You're gonna get hit with these CVs that are found years later, and and you can get yourself into a position where you've got years of code to try to update all at the same time, which is really tough. It's it's really bad. It's embarrassing when you get hit with one of those two. Forgot to update struts that's been running here,

10:53 and now everybody's credit card data's been leaked, that kind of thing. So I think you've gotta get to a point where you can be updating frequently. You know, shift it left if you wanna use buzzwords. Update your dependencies as often as possible with stuff like Dependabot. And then the scary part there is, you know, the whole wall, it wasn't broken. Why would I fix it? Now I'm, you know, putting myself at a bigger risk, pulling in all these new changes constantly. And that's where since we're we're hoping to make that easier and safer and less scary

11:18 to do by making it you confident, basically, in all the code you're pulling in constantly, knowing where it came from. Nice. Well, easier and less scary sounds good to me. So why don't we dive in and get started then? Let me pop up my screen share. We're still here. We've got the sigstore website here. You suggested that we're gonna start by taking a look at the cosign project first to play around with some container image scanning. So we'll do that right away. To the people that are watching, thank you for all the comments so far. If you have any

11:37 Cosign: Container Signing Tool

11:48 questions for Dan, feel free to drop them in the chat. We'll do our best to tackle them as we go. Alright. Step one, cosign. This is container signing, verification, and storage, and an OCI registry. You wanna break that down for us a little bit? Sure. Yeah. So what Incosign lets you do is yeah. Just kinda stay right there. You can sign container images that you built. You can verify those, signatures, and everything for it is stored in an OCI registry. You don't need to run any extra services. This all just works with any Docker registry

12:23 you're gonna be playing around with today. There's a quick little demo that, you know, kinda scrolls there showing the simplest way, but there's about 37 different ways to kinda generate and manage the keys that we're gonna be using for signing stuff here. This is the simplest one. It just kinda throws one in a file on your disk. So that's probably the one we should start out with if you wanna give it a try. But we've got some other cool ones that I'll show you once you get cosign set up and get some stuff signing. Okay. So I guess my step one is

12:49 to install the cosign application. Yeah. We do have a we just got a brew release. Maybe it didn't make it into the docs yet. So if you've got go, you could go install. That might take a little while or you can just do brew install cosign. It should work. Yeah. I'm happy with the brew install approach. Let me just drag the bottom of this window. Oh, maybe it says it's on tap. Yeah. I'm just forgetting how this works. Here we go. Is there a tap repository on sigstore? Yeah. Cosign and wait. Tap. And I can always pull down the banner.

13:32 Right? But if we know the brew What is it called? Homebrew tap. There we go. Yep. Brew tap sigstore slash tap. Here we go. And then brew install goes in. You can do it all on one command. I learned this, like, last week, and I'm just obsessed with doing it on one command. Oh, nice. So, actually, there's a really cool little history I'll show while this is downloading. Yeah. Okay. So this just downloads directly from our repo. But at some point over the last year, Homebrew switched the packages that are stored in, you know, their default Homebrew

14:11 repo Yeah. From something I can't remember what it was, but they switched it over to GHCR. Yep. So the GitHub Container Registry. So Homebrew uses an OCI registry to store all of the binaries for Mac stuff just because it's free and it's everywhere and it's really easy to put stuff in it. So these aren't containers you're installing at Homebrew for the most part, but they're just kind of hijacking and using OCI registries anyway, which is cool. Yeah. I've I've noticed that with a lot of brew packages coming from g GHCR. It's really cool the way they're using the

14:42 OCI format there for that, definitely. But I think because this is your own tap, you would have to upload to that yourself rather than anything else. So we've got a couple of comments already. So Kevin the Fireflash saying it looks really similar to how RPM works. I'm not sure how RPM works myself, but I'll trust your take your judgment there. Kelly Jack was telling me to use Nix. I can't because I stupidly upgraded to the Mac OS beta and now nothing works except my streaming software. So stay tuned for lots more. Sorry. There's a next formula or whatever it's called

15:18 for cosine too. I got it running in a container, which is awesome. Sweet. And there's a a first question there for you. So is cosine admission controller going to be adopted into sext or or stay in your personal GitHub? Good question. Yeah. So it's probably referring to I have a little toy admission controller I was I wrote to just kinda learn how builder and the new admission controller stuff, the SDK is all put work together. I called it cosigned, c o s I g n e d, to check the make sure everything is resigned. I,

15:48 you know, didn't write tests. The code barely works. I I stuck it up there. Didn't wanna put it into the real sigstore thing. I wasn't sure if people would use it if it was worth spending more time on. But then people started filing bugs, of course, which happens with open source and running in production and all that scary stuff. I gotta figure out if I wanna take it down or actually move it over into the org. There's some other policy engines now that are adding support, so I don't know how much we're gonna keep needing mine.

16:12 Stuff like and gatekeeper and other support for cosign signatures. Yeah. It doesn't matter how much you put a warning in the read me saying, do not use this. People are quick because it's got to use a net to solve the problem. They wanna get it in the product and fix that. Yeah. Yeah. So it's there. I don't have a perfect answer. Sorry about that. Alright. So we got cosign CLI. Do you want me to jump back over to the repo and type in that command? Are we just gonna tap complete through it? Live from here. Yeah.

16:43 Yeah. So actually, real quick first. Do you have a container we can work with? Do have a container set up in some registry somewhere? It doesn't have to be super Hey. Yeah. I've I've I've got the clustered demo application, which we could sign, which runs which is stored on GHCR. So is that that work? Perfect. Yeah. As long as you get right permissions set up for that, that should be perfect. Alright. So, yeah, let's we'll start by generating a key pair. So you can just type generate hyphen key hyphen pair. Try out the hyphens right now. Should I

17:12 Demo: Generating a Cosign Key Pair

17:14 spend docker up just now, actually? Nope. Nothing docker. Okay. Alright. So generate key pair. Got a whole bunch of options. We're just doing it vanilla, so it's gonna generate a private key for us. Is that correct? Yep. And you can type you should type in a password. Should or must? I don't I don't remember what happens if you just hit enter, but hopefully, it'll match. Yeah. There we go. Cool. Alright. So now you've got two new files on disk here, which I'll explain quickly and we can show to people. This uses Cosign uses digital signatures,

17:59 which means you get two different keys. There's a public key and a private key. The public key is the one you publish. You distribute that one widely. Anybody who wants to check your images can check the signatures against that public key. So if you just wanna cap that, we can see what it looks like. Cool. Yep. So that this is the public key. That's it. It uses elliptic curve cryptography if you're into that and some details here. But, yeah, we picked some good defaults. There are no real flags here. We only support a couple different algorithm

18:31 combinations and stuff like that. That's the public key. It's pretty small. That's the thing you have to publish and send around to let people check against. Now let's take a look at the secret key too. So I said this is secret, but it's encrypted with that password. So it's actually okay to show as long as you don't tell us the password too. So says begin encrypted cosign private key. So nobody can do anything with this unless they also have your password. So hopefully, it's not just Rawkode one or something like that. No. It's a b c one two three.

18:57 Perfect. Yeah. Hunter one two or whatever it is. Hunter two. Yeah. That's the Yeah. Everybody type your type your passwords into the chat here. Yeah. So that's the that's the one we're gonna be using to sign stuff with. So we are ready to go now with these two files. Now you can sign your container image. Cool. So If you remember the URL, you just do cosign sign. Oh, it's saying. I was just making it out. Right? Okay. Oh, yep. That's right, though. Yeah. There's bunch of different flags here. As you can see at the top, we support,

19:17 Demo: Signing a Container Image with a Key

19:27 like, all the different cloud providers have a KMS system that you can use so you don't have to have the keys locally if you wanna use that stuff. HashiCorp Vault has that too. But if you've got the file, you can just do dash key and pass in the cosign dot key the secret key, and then the name of the image you wanna assign. So the g h c r clustered thing. We'll send we'll send you to it. Why not? You've gotta remember the password. So you're not authenticated. You might have to refresh or do something there.

20:07 I should have g h. Let's see. I'll log in. I'm already logged in. I wonder if I need to do something. Your g h c r. When in doubt, Google it. Login. Yeah. I don't think I've ever actually tried g h c r from my laptop. I know they give you those tokens automatically from inside of Git Hub actions. Yeah. I just assumed because I use the GitHub CLI that it would oh, you just use a personal access token. But how would that work with cosign? It'll it'll figure it all out. Yeah. You do the Docker login or whatever.

20:55 Alright. Okay. So let's And cosign, it'll just work once you do that. Okay. So there's Docker spinning up. I'm gonna create a personal access token and then I'll log in. I'll do that over here. Set up flashing credentials on this stream. Perfect. So I'm assuming you would not encourage people to use the the way that we generated the key and instead to use, like, a cloud KMS provider or something. Right? It depends what you have access to and what you're trying to convey, I guess. There are bunch of different, you know, reasons to sign things, and it's but

21:36 there's a lot of confusion around exactly what you're trying to express with the signature. Really, at the at the base level, all it means is that whoever had access to that private key signed the thing. It doesn't mean it's good. It doesn't mean it's safe. It just means that they use the private key to sign the thing. And so if you wanna keep it locally and you're sure you can protect it and not lose that password, that's a fine approach. You can also use something like a YubiKey. There are some commands for that if you've

22:01 got one instead, which means that people can't steal that from you unless they actually take the token, so you don't have to worry about losing it. The KMS stuff is great if you're in a cloud environment. It's better than storing the password somewhere else in the secret manager or that kind of thing. But there's some downsides there too, which is, like, you know, many times have people screwed up IAM on their AWS cluster or, you know, their GCP environment or whatever. You're kind of trading off not having to lose the key with making sure that you get the IAM and the

22:29 RBAC and all those roles right so that, you know, you don't make it too broad so people can start using the key. So there's some some trade offs either way. Alright. I'm just doing a Docker login. Okay. I'm logged in. I say that as a question rather than a statement. But Yeah. So now it should just work, hopefully. Well, that seems promising. There we go. Awesome. Yeah. So all the credentials that Docker uses are set up in this kinda global credential helper file. It's pretty well understood. So most tools know how to work with that once

23:16 you've done the Docker login stuff. Awesome. So that's signed. Let's well, naturally, we should probably just verify that first before we move on and jump in and show exactly what happened. Yeah. So let's try the other half there. You can type cosign verify, and then you'll see all the different options we have here. Yep. You now you pass in the public one. Step to private. I know I picked a different version there. I just wanted to see what happened. Alright. So it says so first of all, when I use the version that wasn't signed, we

23:27 Demo: Verifying a Signature with a Key

23:58 just get some sort of name unknown. So it doesn't know that something in this OCI registry exists a signature. I'm not sure how those signatures work with the registry, but maybe it's not important. When we use a tag that does exist, we get some checks performed. So there's closing claims, the signature matches the key, and that's the actual version. Right? Yep. So that's the actual signature that got stored there at the bottom. Do you happen to have the crane tool? There are couple different tools to work with, kinda debugging registries. If you have that, we

24:33 can show exactly how the signatures work and everything. The crane with a c, isn't it? Yeah. C r a n e, like for lifting containers up and down. Yeah. Familiar of the tool, but it's not something I keep at hand. Yeah. It's it's the easiest way to do some debugging here. Cool. This is working. Nice. And it came from no. It didn't. No. It did. Yep. It came from GHCR. Nice. Okay. So do we want to Yeah. So now we've got crane. We want let's take a look at what the signature is in here. So first, crane.

25:09 How Signatures are Stored in OCI Registries

25:19 Actually, there's a fundamental command first. We have to run do cosign triangulate. Oh, some trigonometry comes in here of the the image name we signed. So we hide these things inside of the registry, the signatures. And the way we do it is with a cool little naming convention, which should make sense in a second. So the image we signed was clustered colon v one sorry, v two. And the way a registry works is that there are tags and digests there for to an image. The tag is kinda mutable. It's just a string, but it points to a

25:51 fixed SHA two five six digest for that image that can't move around. The way we find the signature for an image is that we take that digest, the thing that can't move, and we turn that into a tag and then tack a little suffix there at the end of the dot sig. So if we wanna find the signature for clustered colon v two, we first resolve that to a digest, and that's what the cosine triangular command did. And then we turn that digest into a tag with a colon instead of the at sha two five six thing there.

26:21 And then we add the dot sig. So the signature itself, sorry, is stored in this image. It's not an image you can docker pull or run. But just like Homebrew stores other things in a registry, that's what we do here too. So if you do crane manifest and then that thing, this is gonna download the, you know, bytes that we've stored in the registry. Remove the chat 256. Right? No. So leave that. Just add the dot sig at the end, actually. Alright. Okay. Yeah. Cool. And then if you have j q, you could pipe it to that so it's more

26:53 readable. Cool. Perfect. So this looks like a normal Docker image or an OCI image. You can see it's got the layers, except instead of having a tarball that would unpack into your system, we just cram a little signature in there instead. Except that annotation is, the signature is just that base 64 encoded string. The MEQ thing, if you take that and look at it, it's just, you know, random bytes that were generated by the signing algorithm. And then the layer itself is a blob. You can see the digest for it. And that is the little payload we signed. So

27:28 when we did verify up above, we got that JSON that spit out on that last line. That is the thing we signed. So we store the signature and then the little thing we signed here in the registry. That's how that all works. So the verify step, when you typed in the wrong tag, gave you that error saying it couldn't find a signature. So that's what that was. It was looking in this location for an image. Couldn't find it. But then we did find it, downloaded this thing, checked all of it, and it was good. There are a couple other little features we

27:58 can show here with sign and verify if you wanna see. The first one is you can add in little random key value pairs when you sign, and you can check those too. So if you go back to your sign command, you can add in dash a and just make up a little probably have to put it before the image name because that's just a positional. Yeah. Perfect. You just do foo equals bar, whatever you wanna do. You could put in a git commit. You could put in a release version here. You could put in anything kind of more meaningful than just signing

28:02 Adding Annotations to Signatures

28:28 the image itself. You could say LGTM equals yes, you know, whatever you wanna do and just hit enter. So this is gonna show we can actually sign images multiple times. So we're gonna have two signatures on this image now. One doesn't have any of these annotations. This one will have the foo equals bar. Now if you verify again, we should see both signatures this time, one with it and one without. Cool. So you see these two JSON blobs get spit out, and the second one has that foo equals bar thing. Nice. When you verify now, pretend you wanna,

29:02 you know, only check things with foo equals bar, you can add that into the dash a, and it'll only show you ones with that matching thing. So people built up, you know, pretty good workflows with this to sign specific releases and then check specific releases, or you can put in container vulnerability scans or anything else you kinda wanna write up in your policy engine. I'm curious. Right? Like, this this is a random image that I pushed to a registry in the past, but, I mean, I have no idea really if that is even still my image. Like, what would be the actual

29:31 workflow that we would encourage people to use with cosignation? Should they build it, sign it, and then push it to the registry, or do they build it, push it to the registry, and then sign it? Like, what do you recommend there? Yeah. So there there's a couple different workflows that make that make sense here. The first one is trying to make sure that the image came from the source and was built the right way. And so the best thing to do there is to sign it in your build system, like, as close as possible to when it

29:43 Signing Workflows and Best Practices

29:59 was built. Your build system usually knows the digest of the image, and so you just have the build system sign in at the same time. So if your build system gets hacked or something like that, you're not really protected. But as soon as it leaves that build system, you at least know that that image is the same one that left the build system. There's a GitHub action that we have, the cosign installer that throws it into a GitHub action if you can use that. And then because GitHub actions gives you those cool personal access tokens, then you don't even need to

30:25 set up auth or anything like that if you're pushing and signing stuff in GHCR. Depending on your build system, you can plug in other stuff. Tekton CD is a Kubernetes native c I c d platform that actually has built in support now where you can just kinda say sign all the images that get built with this directly from the build system. So to kinda protect against that, making sure that the thing is actually what was built, signing it in the build system is the best way to do it. Some other companies in, you know, kinda regulated

30:54 industries and financial stuff and health care need to make sure that, you know, two people have approved each change that goes to prod. So it's kind of a different flow. And a lot of people there use, like, two different YubiKeys, and two different people have to sign the image as the very last step. So that's kind of like it's been built. It's been tested. It's been in our QA environment for a while. Now we have to have two out of these, you know, seven people sign it with their own Yuba keys as the last step. So you can kinda build

31:18 up workflows with, you know, different levels of signatures coming from different places. There's no real one size fits all approach. But I think for most people, they want that first one of just signing the stuff that left their build system. Alright. Perfect. We've got a few questions if you're happy for me to Yeah. Look for them now. Sure. So let's see. I think we may be covered the first one here. O six says, how is signing done in automation? Do you put no password, or is the position that real humans should sign releases? Yeah. So that's that's the one we kinda

31:31 Q&A: Signing Process and Storage

31:54 just talked about. I've got a little demo app called, like, sign container on GitHub. It shows how to do it in GitHub actions where you don't really need the password. You can kinda use it. And then there are even some different flows we'll get to next Okay. Using things like OIDC and different protocols to sign things inside of a build system. So, yeah, I think humans and systems should sign it, not one or the other, I guess. Okay. Next question is, does the token have permission or I guess, does the token that we use need to have permission to push to

32:25 GHCR? Yeah. That's another good question. With this flow, what we did is we stored the signatures right in the same repository as the image, so literally right next to them when we did the triangular command. So, yeah, you need that permissions to push to that one. That doesn't work for some people where they don't want their developers to have permissions to these registries, only their build systems. So there are some flags to kinda redirect and store them in a different one. So it still uses that same naming convention, except instead of, you know, g h c r slash raw

32:52 code, it would be something like g h c r slash raw code signatures. You would say, I'm gonna store all my signatures here so my devs can push to there, but not to the, you know, bucket where the containers themselves are stored. Okay. Am I correct in thinking that cosign executable needs to be built with the YubiKey feature enabled? Yeah. Another good question. The YubiKey stuff, depending on your OS and your platform, sometimes needs some native c libraries to actually work correctly with the drivers. On Mac and Windows, I believe it just kinda works. There are some go build tags

33:26 for Linux where you do need to have some special dependencies installed. It doesn't run by default on stuff like Alpine, so we publish two different builds for Linux. If you're not gonna use the EBAQ one, you can just do a native Go build with no CEO. But if you want that, then you've gotta use the CEO version, which is a little harder to cross compile. Alright. One more question, and then we'll carry on. Asks, what does triangulate do with multiple signatures? I'm guessing it provides them all. Yeah. So we actually store all the signatures in the same

33:55 blob. So, yeah, triangulate just points to the blob thing that has all the signatures. When we saw the if you do the crane manifest again command, we'll see now that there are gonna be two layers. The first time, there was just one layer. Yep. So all the layers just kind of get appended up into this list. So it shows you the address for all of those, and then we kinda go through and verify them against the key. We did both of these signatures using the same key, but you could imagine two different people with two different keys signing it, and you would

34:00 cosign: keyless mode

34:22 kinda see all of those appended up, and we'd filter through them as we do verify. Nice. Okay. So we have taken a container image. We have signed it. Is the next step that we want to restrict our runtime to only run images that are signed, or is there something in between? Yeah. Let's do a couple more things first. Also, I screwed up trying to get the policy engine set up this morning, so I'm stalling a bit too. But yeah. So let's let's let's show some of the cooler fancy stuff that'll actually kind of pivot and explain so that the rest of

34:55 sigstore and how that works. Let's do this is the cool experimental keyless mode that I like to call it. We just did some signing with keys, but now we're gonna be signing without keys. So we need to turn on the experimental flag first. I don't why it's experimental still in a minute and all that stuff. But it's an environment variable. So do export cosign underscore experimental equals one. Yep. All caps. Perfect. Alright. Now we're gonna do a sign again, except let's not pass in a key. So just do cosign sign. You can leave on the annotations if want.

35:29 Demo: Keyless Signing with OIDC

35:34 It doesn't matter. Yep. Just the image now. No keys. The same image or a different image? Up to you. Okay. That's it? Yep. Just hit enter. It's magic. There are no keys. We're gonna we're gonna see some logs, and now you're ah, yeah. You're only sharing your terminal. But what just happened here Sorry. There we go. Oh, perfect. Yeah. It popped open on browser window. So we're gonna go through here and do kind of the the OpenID dance. And if you see what was back in the terminal sorry to make you keep switching. But what we actually did here is we

36:06 generated a key in memory. This is generating ephemeral keys. And now this is the CA piece, the certificate authority thing we were talking about before. So we're gonna get a certificate from the sigstore CA tied to the email address, whichever email address you click on in that browser window. So if you're you're familiar with Let's Encrypt, the way they do it is they, you know, give you an automatic certificate by proving you own a domain name, and they do that with Acme challenge thing where they give you a little token you have to display on the

36:36 domain names, so then they're sure you own it. We do the same thing here except for an email address. There's already a bunch of protocols for that. OpenID Connect is the one we use here. So we approve you have that email address, and we give you a certificate for that email address instead of a domain name. So whatever the email is on the GitHub login you just did will be in a certificate. So instead of signing with a key now, we sign with an email address. So if we do verify now again with no key,

37:04 we will see the signature and the email address. Yeah. The magic keyless mode. David@rawkode.com. That's the subject of the certificate. Cool. You got some questions? Just that's super cool from Noel and Sean there. Okay. Cool. Yeah. So I'll explain kinda what happened there because there's some magic behind the scenes. It's kinda subtle to make all of that work, including that last line, the TLOG entry created with index line. So the way this works, if you've heard of Let's Encrypt and certificate transparency, we kinda copy that same thing. All of the certificates that we issue through

37:42 Rekor: The Transparency Log

37:53 the service get logged to a transparency log, which is a really cool data structure where it's append only and people can kinda verify that and check and monitor it. So the entry there is 33319. That'll only ever go up. Your certificate is in that log. The signature is in that log, and you can look at it, and it will never get tampered with. And it will be there forever the way the Merkle trees and fancy computer science works. And so what that lets you is it protects you against our service getting hacked. Right? If our service got compromised or hacked, we

38:25 could start issuing certificates for your email, that you didn't request. We could just kind of bypass the open ID thing, which proves you requested it. So this lets people monitor the log for any certificates issued for their email address. If they see something in there, they say, hey. I didn't sign a container that day. They know something bad happened. They also know the exact container that got signed too. So it's not just something bad happened. They know, hey, somebody stole my email account. They stole my password because I accidentally broadcast it live on the stream. And they started

38:54 assigning containers with my name. And you can figure out the exact ones that they signed and tell people not to use those. So the keys were generated at the top there in memory. They never touched disk. They were deleted right after. And so it'd be pretty hard for somebody to get those keys, but there's some other kind of fancy stuff we do there to protect against that too. We got another question? Just another. That's really cool. A lot of love for this approach, definitely. Yeah. So this is a really good way to do it as a user if you're trying

39:27 to sign with an email address. It's a little harder to set up for demos here today. But if you're familiar with projects like Spiffy and Spire, they use that same protocol, the OpenID Connect one, to do certificates and keys for machines. You set up, you know, nodes and pods in your cluster with those identities that instead of an email address, it looks like spiffy colon slash slash and then some name you make up for your domain name and then the machine and everything itself. So those services can go through the same flow too, where every time they wanna sign something,

39:56 they authenticate with their SPIFI ID, and we give that machine a certificate. So we can use that to sign stuff in that short time window. So for build systems and everything, that's if you don't have a fancy HSM or hardware security module or KMS system, then this approach works great too if you're fine with storing these certificates in a transparency log. So I've got a, I guess, then a couple of questions. So Yeah. The first one, experimental. Yeah. What is the status of this, and when is that likely to change and become generally available? Out that, you know, transparency logs and immutable

40:20 Q&A: Keyless Status and Log Auditing

40:34 ledgers and storing data in something that can't be deleted and bring out a lot of, you know, GDPR and legal and, you know, what happens if people sneak bad stuff in there that, you know, we're not allowed to host and that kind of thing, those risks. So it's it's good from a code and a stability standpoint. We're just trying to work through and get approvals from all of the lawyers and stuff like that to stand this up and let that data stay around and figure out what we would do if something really, really bad ended up in there. There's

40:58 nothing brand new to solve. Right? The certificate transparency people have been dealing with this for years. We just gotta get through that. So we're hoping in the next couple of months, we can take that off once we figure out the playbooks for what we do in some really bad situations. Okay. How do I like, I've got this index number. Yeah. Like, how do I check for entries with my email address and and and things like that? Yes. We're gonna have to get another tool that I don't think is in Brew yet, but we can download it and build it

41:00 Transparency Logs with rekor

41:28 and everything like that. But yeah. So you'll see when you did that check, cosigned, did the verification. Yeah. Existence of the claims and the transparency log is verified offline. So cosign did it automatically when we did the verify, but we can kinda walk through the steps to do it manually if you wanna get the Recore tool installed. See if we have I think we have proof for this yet. Your binary is published. It's in a bunch of different pack parameters, but not brew. Or you can clone the repo, build it, or grab the binary yourself, whichever you wanna

41:59 do. The Darwin Just get in. And the This stuff is you'll see these binaries are all signed too with cosign, so that's cool. Perfect. And I download. That's it. Alright. So let's let's scroll back up and find that index that it printed out, and we'll just grab that one right away. Yep. Just the number? Cool. Yep. I do Recore get dash dash log hyphen index, I think it is, and pass in that. Cool. And we'll get a bunch of hard to read data back out. So this is the entry for that specific signature we did in the transparency log.

42:29 Demo: Exploring the Transparency Log with Rekor CLI

42:53 We store the thing we signed, the hash of it. So that's the data hash value thingy. That's the hash of that JSON object we signed. Excuse me. There's that signature there, and then there's the public key. So that is gonna be the certificate. So you copy paste out that long thing. It's base 64 encoded. You can touch from the equal sign. So that's my trick. You should be able to see some more stuff. Don't think it copied right. It didn't work. Yeah. What's happening? Yeah. You're losing the equal signs at the end somehow. Right? It's losing much more than

43:34 that. So let's do that. Tap in difficult. Try again. There we go. Okay. So that is the certificate that we issued for you. If you've got open SSL, I can never remember the commands for this. Open SSL. You can type that to open SSL x five zero nine dash text dash maybe that's all you need. Let me see what happens. It's a million different OpenSSL versions. There's some tool that'll show you maybe no out. That's another one I have on here. Oh, sorry. Not dash x 509. Just x 509. Alright. Okay. There we go. Oh, and then do dash text maybe to

44:31 stick that on. We'll see So if you scroll up, you'll see the all the info in there where everything is stored, the algorithm, the key, your email address is there too at the bottom. That's the subject thing. There you go. Perfect. Yep. So if you're you can kinda tell this log there's new entries that come in. You can just keep grabbing the index and verify it. And the whole thing only works if people are doing that, which is nice because people are. So there are people that constantly verify our log and make sure that we haven't

44:58 tampered with stuff. In fact, if you do do recall log info, and now you're actually helping, and you'll be a verifier of our log to make sure stuff doesn't get tampered with. Log info. One word. Yeah. Cool. Yeah. So this is the first time you hit the log. You now stored stuff at that length. If you ever hit it again in, you know, a couple years from now, then it'll make sure that nothing in the log up until now was tampered with. So you're helping. You're doing your part. For searching, though, we can do this is kinda what

45:29 you might wanna do to look through it for things with your email address. There's a search command. Can search dash dash email. You can search on a couple different fields. That should work, hopefully. And that is just the one? Yep. Perfect. So nobody's been stealing your password, you know, before. Yeah. So that's how all that stuff works. There's a bunch of cool stuff with time stamps and everything to also help protect and limit the blast radius if you do lose your signatures. But it starts to get pretty complicated. Okay. So that was all the three parts

46:03 of sigstore. Just the plain keys, KMS systems, or you can start using transparency logs and certificate authorities. Okay. So we've got cosign, which is a tool for signing and verifying artifacts, one of which could be a container image. We've got rakor, which allows us to work with the transparency log, get information over to searches against it. And those are the two major components of sigstore. Yeah. There's the the third one, which is the one kinda behind the scenes, the one that actually issues those certificates, that's called Foolsio. You should never really need to interact with

46:12 Sigstore Components: Cosign, Rekor, Fulcio

46:37 it yourself. That's just the tools to get the certificates for you. We we got a a couple of questions there if you wanna tackle them, and and then we can Let's go for it. And then there's kinda one other thing we'll do now. We've got recourse set up after these questions. Sweet. Is okay. So CPP for life. Is there any info about what service was used to help Microsoft or something else to verify the email address? I'm assuming I added that last bound. Yeah. So that's a that's a tricky one we're trying to figure out too. We we store that.

46:52 Q&A: OIDC and Hosting Logs

47:08 Right? We have those three OIDC providers. That's GitHub, Microsoft, Google. Those are the only three we support. They're pretty big and trustworthy, we hope. But we do wanna figure out how to log that info and make it available just in case something bad does happen. The problem there is that the info they give you back usually has some long number in there, which is kinda fixed and can be a privacy concern. Also, if you change your email address on Google, that number usually stays the same, and you can start to peep and start to do some scary stuff like

47:36 correlate things across email addresses. So we've gotta figure out how to store it and make it available without leaking people's privacy. So we store it ourselves for a retention period just in case something goes wrong to do some incident response, but we don't publish that in the logs yet because it's a little bit scary. Alright. We got another question asking. So is keyless option using hosted by sigstoreorg and the transparency log handled by Rawkode? Yeah. So we run those two services, the CA and the transparency log kind of as a public good instance. So the sigstore community

48:08 hosts and manages those. Set up as a transparency log, you don't really have to trust it. You can audit it. You can't really do anything bad without it going on to that log in case something does happen. But it does require you to hit these external services. So there's some other modes you can set up inside of, like, an organization to host parts of that yourself so you don't need to be constantly going outside of your firewall to get certificates or to log data to the log. The most part, it's just hashes, so you're not really leaking

48:35 tons of proprietary information in a SHA two five six hash. But still, sometimes you don't wanna be even allowing outgoing network connections. You can run parts of it yourself too. Sweet. I'm curious. But you did as a signing episode with Pop on Cloud Native TV. Right? Was that for this hosted Brazil? Yeah. So that that was a that was our root key ceremony, basically. So, yeah, the full COCA and even the Recore log itself has to assign a bunch of things to make the proofs work and everything. And we need a bunch of our own

48:50 Key Management and The Update Framework (TUF)

49:07 keys in order to do that. And so keys are hard to manage. They're scary. People lose them. And so what we we did is we followed the protocol defined by the update framework or TUF, which is another graduated CNCF project. It's not really a code base. You can kinda just grab and start using. They do have some libraries and everything, but it's more of, a framework for managing keys in a, you know, distributed setup like this. And so there are five key holders as part of the sigstore project now, and we have, like, a rotation setup so people don't

49:35 have to do it forever. And we each have Yuba keys, and it takes three out of the five to actually, you know, sign things or delegate new roles or replace keys and that kind of thing. So if any one of us loses ours, the other four can fix it for them and, you know, sign a new key. So as long as we still have three of them at any given time, we're good. So that was that event we did. It was the the first one where we all had to kind of take them out of the box the first time

50:00 and establish that root of trust And now we can use the update framework to make changes to and iterate on going forward. Awesome. Alright. This is all very, very cool. What have we what have we got next? Alright. There's one more thing here. Right? I talked about the signatures. They're kinda they can just store basic information about the container, which is cool. If you sign in a build system, that's nice. But what we can actually do is store more than just the container name. We can store stuff about how it was built. So let me get you a quick

50:14 Attestations and Build Provenance (SBOMs)

50:32 command to run. And this works with our integration in the Tekton build system, but there's nothing really about Tekton that's special here. We have the Tecton build system generate payloads, basically, that explain exactly how something got built, like a container. So it's not just the container now. It has the exact commit that was fetched to do the build, the exact containers that were using the build steps, all of that cool stuff. And it writes it in a payload and stores that in a transparency log too. Let me find the easiest number to send you so

51:09 you can do this check quickly. It's not working here. I got a blog post where you could follow along. Yeah. Sure. There's a typo. Alright. Yeah. So the blog post is here. It's called the it's a the CD foundation blog, and the title is called the meta chain. If you wanna search for that, you should find it. Of course. Oh. For some reason, the command is not working. Meta chain CD This one? Yeah. Verifiable supply chain metadata for Tekton. Exactly. So we did a release of Tecton that was signed with Tecton itself. That's why it's called

51:55 Demo: Searching Attestations in Rekor

52:01 Meta. And then it stored all of that in the transparency log. Oh, sorry. That's actually the wrong one. There's a more recent one. But I gotta work it here. I'm gonna send you the number to grab. Oh, sorry. I've I've got it. July 29. Cool. So this is the last release we just did last week. So instead of just signing the containers that were part of this tech hunt release, you can see that we signed these giant they're called attestations. And this shows all the info about how the thing was built, and that goes into

52:31 the transparency log too. So if you start there at the top, if all you have is the container ID for the tech hunt release, that's the SHA 256 thing, Show you the next text box. Yeah. So if you if you got that running, you've got the SHA256 for a container. You can search that in the transparency log using that search command. So instead of an email now, we're just gonna look up everything we know about that SHA. Mhmm. We get these entries here, and then you can download one, and you can see all the information that's used, all the parameters,

53:01 the version information, the exact batch script that was used to build this thing. I'm assuming I could just run this on my own machine as well. Yeah. Yeah. I didn't wanna try to read that UUID out to you. Alright. So this is everything that we need to know about this build. Yeah. That's exactly how it was built. See the parameters that went into the text on task run. These are all the strings, the key value pairs, or the thing we fetch from in the Kubernetes cluster, that long bash script. It's native technology is based on bash and tarballs

53:36 and YAML. The exact container steps that were used, we had crane running in there. You can see the digest, all that stuff. So if something got compromised in any of these things, we we have all the tools to kinda trace all the way back. Cool. So this is all set up. You just install this in your Tecton cluster, you just kinda get this out of the box. Everything gets signed and logged. Nice. I like that. I'm curious. Like, this transparency log, you said, is append only. It's it's gonna continue to grow. Like, does that mean that the data

54:00 Q&A: Log Management, Integrations (Tekton, In-toto)

54:07 lives forever? Will it ever be trimmed? Do you deprecate or time to live older entries? Is that impossible because of the Merkle tree? Yes. So the way people normally do this is they kind of time bound stuff. If you look at, like, the Let's Encrypt Transparency logs, they have a different one every year. They keep the old ones for some amount of time. It's not a huge, huge, huge amount of data. Right? It's not, you know, petabytes of data that we've got in here. So it's it's nothing that's kinda gonna push the bounds of SQL technologies.

54:38 But, yeah, we do need to set up ways to kind of rotate and cut off the log and then start a new fresh one, but still let you kinda search back through the historical ones. Supply chain stuff is a little different because, yeah, you run code that's, know, maybe a couple years old. You might probably shouldn't be, but people do that. So we can't deprecate things as quickly as you can with certificates where once they expire, you don't care about them anymore. Yeah. It'll grow until we, you know, freeze it and stick them, publish the archives, that

55:00 Using Kyverno for Signed Image Policies

55:04 kind of thing. Should people ever run their own log? Yeah. A lot of organizations do or are trying to figure out how to or to to run them. It's not always the best fit. Right? Mirroring our log and verifying it is great. That's awesome because we can get kind of multiple copies of it that are independent around the world. But a lot of organizations trying to run them internally doesn't necessarily make the most sense because the log only works if you have, you know, people auditing it and checking you. And if you're all on the same team, you're kinda

55:34 checking yourself, and you're not really getting any of those benefits. It's easier to just use a plain database in that in that case because you do get a bunch of complexity that only makes sense if you do have people that are kinda being skeptical and making sure you're not screwed up. K. Awesome. Alright. I'll throw two quick questions at you, and then we'll we'll carry on. So we got a question saying, is the Tekton integration through chains or something else? Yeah. So it's done through the there's a little sub project called Tekton chains. So to

56:01 install next to your Tekton cluster, you just start getting this stuff signed. It can store stuff in, you know, any database you want to. It doesn't necessarily have to be the transparency logs, so you can throw it into BigQuery or DynamoDB or any of those document databases to do kind of more querying on later. Right. Is it similar to Intoto? Yeah. So that format is actually from Intoto. That's another CNCF project. The Intoto project defines a bunch of formats for kind of describing the supply chain step. If you think of a a build step as a bunch of inputs came in, did

56:34 a bunch of stuff on those inputs, and then a bunch of outputs came out, you know, those three things. And TOTO is a really good set of formats for describing those things in ways that you can link back together and check. Sweet. Thanks for that. Alright. What what do we what do we do next? Good question. I could not get my cluster and Kiverno and policy engine all set up. That was kinda what I wanted to do next. Sure. You can set up verification there. Yeah. Not quite sure what's going on. My cluster got all screwed up over the

56:52 Integration with Kubernetes Policy Engines

57:05 weekend, and I didn't get it working in time. But if you look at the maybe you could see the docs. I know they're working on a release that's supposed to go out any day now to explain how all of that works. Maybe we can find the stuff. But, yeah, basically, you just write a YAML file with so this should be under k y v e r n o there, GitHub. Oh, so this is the okay. So this is the Kibernal one? Yeah. This is what I was gonna get set up to show how you can block deployments unless the signatures

57:31 are matched with a little CRD they have set up so you can put the public key right there. Yeah. It's it's right in the main On the main one. Cavernote. Yeah. I don't actually know where the their docs forward are. I had a setup a while ago. It is in the release notes, though. We should be able to see it if you click on releases. Maybe somebody in chat will know where it is. Image verification. So if you scroll down yeah. There we go with cosign. So, yeah, if you get this installed and set up from ahead because I haven't quite done

58:10 the release yet, I think that's where I screwed up. I was installing an actual release instead of this draft. Then you can kinda define a new verific image verification CRD and put in some public keys, and it'll block you from deploying. If you click that Google Doc link, that's what I was going off of. Ah, okay. Cool. Yeah. So this shows what the CRD looks like. Install kabrino, and then you just kinda stick a public key into some YAML. That's just a normal cosign key, and then it would block all images that don't have that set.

58:47 I did have this working at one point. I even made a full GIF. Okay. So when I've got cosign running in my CI environment, sending all my images Yep. We can with Kivernal installed to a cluster, add a new cluster policy that monitors your pods and checks that there's checks the images that match a certain pattern assigned by a specific key. Yeah. And so then you can set it up to there's the enforce or alert, you know, failure action. So you might wanna block it. You might just wanna send an email to someone and warn them,

59:21 I just wanna log it. There's a couple different setups you can do there, but that because you can't accidentally you know, there have been a bunch of kind of typo attacks where you've fat finger and type the wrong name of an image from Docker Hub, and now you're grabbing something from somebody you don't know because you put two o's in the MongoDB container instead of one, and people set up coin mining and all sorts of terrible stuff like that. So Okay. A bunch of different reasons you might wanna do this type of policy. But if you get that set up in your

59:46 CI system, it's all signed, then you know that your prod environment is locked down to stuff that came from that CI system. Yeah. This was merged twenty five days ago. I'm assuming this maybe hasn't quite had a release yet. Yeah. It's r c three now. I saw it in the draft. So any day now, it'll it'll come out and I think yeah. I screwed up and didn't quite get the release candidate running. Well, it says that it's released. Right? One four two? It's still a draft though if you click on the Oh, yeah. 14 one's the latest.

1:00:15 So okay. Alright. So really soon. The latest is 141. Yeah. Anyway Definitely something cool to play with later. And I I like this. I wonder, though, like, this wouldn't technically work with the keyless approach. Right? Okay. Nope. So, yeah, they I don't know if we made it into this version, but you can see some discussion around it. You can put email or OITC accounts somewhere too and lock down to that rather than just the keys. And I think my little admission controller people were talking about earlier can do that too with a config map, or you can put

1:00:28 Q&A: Policy Engine Integrations & GoReleaser

1:00:51 in keys or email addresses, whatever kind of combination there you wanna verify against. Alright. Awesome. Very cool feature. I'm assuming that I think you mentioned Gatekeeper earlier, so the OPA projects are probably working on similar support. I'm assuming Cube Warden probably gonna have another policy and that works with all of this. It's nice to see that support coming from the broader cloud native ecosystem as well. Yeah. I've been trying to get it working in KubeBoard, and I thought it was gonna be so easy because this is all written in Go, and you can compile Go to

1:01:19 Wasm, and you can get that running in KubeBoard. But the the Go Wasm support isn't quite there yet, and you can't do a bunch of stuff you need in in that. So Yeah. It's soon it's not Kubarden. It's it's Go and WASM, unfortunately. So the the spec that we have for cosign explaining how this works is stable too, so you could go implement it in something else like Rust that does compile down to Wasm a little bit better. But I was disappointed. I thought it would just be adding some compiler flags, and then we'd have support.

1:01:46 Yeah. Just change the target to Wasm and hope for the best. Right? Exactly. Yeah. You just can't do reflection, which turns out is needed for JSON parsing. So doing any kind of well, I'm in the Kubernetes ecosystem is tough because it's all kind of reflection based. Yeah. I I think you should just jump on the Rust trend. It's a it's a it's a fun trend to be on, especially with the WebAssembly report. It's it's pretty sweet. Yeah. Alright. And is there anything else you wanna show, or will we is that a that is Yeah. I think that's a that's a good

1:02:18 overview, especially if you're working with containers. You can just grab cosign and then start to play with the rest of the transparency logs and the CA and all of those features. So, I mean, that was all pretty self explanatory. Like, running those commands, there was nothing too daunting there. It was all pretty nice, which just shows that it's a pretty great developer experience. Is this can I use cosign for like, it doesn't have to be container images? Right? I'm thinking of other open source projects I've got where we release binaries. Can I sign these binaries and publish to that log?

1:02:48 Yeah. Yeah. Don't know how we blanked on that. Yeah. The all the cosign stuff itself on the GitHub repo is signed with cosign. So instead of sign container, there's a sign block command where it just kinda spits out the signature. It's really easy to do that. The challenge is, like, how do where do you put those signatures? There's no standard great place for it. So for us, we just put them in, you know, that GitHub releases page, and you can check those with cosign too. Yeah. The cool part is a lot of things are moving to the container registries like we

1:03:15 talked about. You know, OPA lets you put bundles there. Envoy and Istio now lets you put Wasm modules into a container registry. So, yeah, you can sign binaries. You can sign all sorts of other packages, but it's kinda cool to see everything moving and converging on the container registries anyway at the same time. The other one, though, I think, you mentioned in your tweet today to chat about was the SBOMs, the software building materials and stuff. So there's a ton of different tools and formats for generating those and working with those. They're huge, you know, documents that you've gotta

1:03:43 upload, store, and sign somehow too. So cosign has some commands for working with those. And you can upload those. Works the same as our signatures. There's a just a funny little naming convention where you just put a dot s bomb at the end instead of a dot signature. And then it looks like an image, so you can sign that too, and you can point that at an image and download the s bombs for your image and that kind of thing. Alright. Okay. I'll ask you one more question. If anyone in the chat has anything they wanna

1:04:09 ask Dan before we finish up for today, it on the comments. You've got a couple of minutes. In fact, I'll I'll throw two questions at you then. So the first one would be, like, do you see this being something that would be integrated with a platform like Snyk? They do a lot of code scanning, dependency scanning, and it feels like they should be able to maybe pick up on these signatures and the s bombs and be able to give us more details as well. Is that something that already works or you know is in progress

1:04:34 or any info there? Yeah. That's I hadn't thought of it that way. Yeah. Once the stuff is stored, you know, things like sneaking, you know, show it and render it in the UI. Actually, the CNCF Harbor registry is working on that now, kinda rendering these and putting little lock green check marks next to things that are signed and aren't signed at the registry level. We didn't we just started working, I think, last week on kind of the reverse integration, though, which is how you might store these sneak or trivia scans in a registry for a container.

1:05:01 It's a little bit different than something like an SBOM because they go out of date. Right? If you just because something had no vulnerabilities today, it doesn't mean nine months from now, it has no vulnerabilities. These things get found over time. So there's kinda like a timestamp approach that you would put in there saying, it had no vulnerabilities today. And then you might have an admission controller that says, I wanna only allow stuff to run that I've checked within the last thirty days or something like that. So you can kinda combine those scanners with time stamps and

1:05:28 signatures to make sure that you're constantly checking them. You don't have to worry about forgetting an image that's sitting and running in your cluster. Cool. And what's what's coming next then? What's the the next major deliverables for the project? Yeah. The next one is really just getting rid of that experimental. That's scary thing. So there shouldn't be a huge change in the UX at all. You'll just you just won't have to set those flags, and that stuff will just be turned on and working by default. So you don't have to use keys if you don't want to. I mean, that's really

1:05:43 Future of Sigstore

1:05:57 just more making sure we have a playbook to do if something syncs its way into that log, and we do have to delete and move things forward for other reasons. Couple weeks ago, actually, there was a fun issue with certificate transparency where a cosmic ray flipped a bit and screwed up the Merkle tree. I don't if anybody saw that. Yep. Yeah. They traced it down. The right data went into the log, and one bit was flipped, and it kind of invalidated the whole Merkle tree. And so people pieced it together and figured it all out.

1:06:23 We gotta figure out what to do in those cases. Yeah. I'm glad other people are solving those problems. Alright. One question from the chat and a little from pop as well. Hey, pop. CPP for life is asking, is there any cosign integration with GoReleaser? No. Not yet. I don't think so. Nothing direct. Carlos has been working on a bunch of our release engineering stuff. Carlos is awesome. I don't know if he's listening today. Oh, actually, he's on vacation. But, yeah, GoReleaser has some scripting modes where you can kind of, you know, an escape hatch to do other

1:06:55 stuff. So he's got some of those demos working. But, yeah, it would be awesome to get some GoReleaser support, especially if people start storing random binaries in the GHCR registries rather than little releases page inside of GitHub. I think that would be a huge one for everybody. Sweet. Kinda Homer style. Alright. And of course, people, it's open source. You can all contribute. Go check out the organization on GitHub at github.com/techstore. Alright, Dan. Thanks for having me on. That was awesome. Thank you so much for taking time out of your day and and kinda guiding us through

1:07:16 Conclusion

1:07:29 sigstore, cosign, rakor, and keyless stuff is all very, very cool, all very exciting, and we can't wait to see what happens next. So have a a great day, and I'll hopefully speak to again soon. Yeah. Thanks for having me on. Bye. Bye all.