0:00 Introductions

0:00 Alright. Welcome to Cloud Native Compass. Thank you for joining me, Mark. For the people who are not familiar with you and your work, can you please tell us who you are, what you're up to, and and everything else you wish to share? Yeah. Thanks, David, for having me. So my name is Mark Borschin. I'm the CTO of Tremolo Security. We are a open source identity management company, so we focus on all things authentication, authorization, identity, automation. I've spent, you know, since 02/2015 working in the cloud native community on Kubernetes, whether it's writing documentation, helping folks out in the Slack channel, contributing

0:13 Guest & Company Introduction

0:34 to open source projects. So yeah. Awesome. Why don't you tell us a little bit about terminal security and what they do? So open source identity management. So that means a lot of things to a lot of people, especially in the cloud native world. The first part that you'll probably run into us is getting into your cluster via authentication. So probably our most popular tool is Open Unison. It lets you log in to your Kubernetes clusters whether they're on prem or up in the cloud, with whatever authentication system you have. So, you know, whether that's LDAP, AD, Okta,

0:38 Tremolo Security & Open Unison Overview

1:09 you know, Azure AD, doesn't really matter, makes it so that you can get access to all of your cluster applications. So, you know, there's there's koop control. Right? But your cluster is made up of more than koop control. Out of the box, we'll do secure access to the dashboard, which folks really enjoy. And then, you know, what are your other cluster management applications? You might have Prometheus and Argo. And, you know, all these different apps have their own SSO systems. So we integrate that and integrate it into your identity system. So that way, it all

1:38 just kinda works and users don't really have to worry too much about it. Nice. So that's cool. Open yours and opens our project that does identity management for Kubernetes. Now I don't wanna deviate too much from the questions that we kinda shared ahead of time, but I'm gonna dive straight into an open using question because I think it's interesting. Like, most people have the Kubernetes cluster. They get given a KubeConfig that lasts for a very long time. Identity problem solved. Yes or no. Right? I mean, it it depends on who else you wanna have access to your cluster.

1:50 The Problem with Identity and Kubernetes

2:09 You know, the the trick with that that kubectl configuration file is it's a certificate that Kubernetes doesn't know how to revoke. So in the certificate world, if you wanna make sure that somebody doesn't use a certificate that has already expired or you've revoked for some reason because, you know, maybe you emailed it to the wrong person, you accidentally checked it into a Git repo, There's a process called certificate revocation. Kubernetes doesn't know how to handle that. So and that's actually one of the the big audit findings every year, when they do the security audit of Kubernetes

2:44 is Kubernetes doesn't know how to do certificate revocation. And it's probably never gonna know how to do certificate revocation. Can I just confirm what you mean by that? Sorry to interrupt. Do you mind if I just confirm what you mean by that? Right? Like, what you're saying is if employee x works at our company, we give them a kube config. They have access to our production cluster by saying that Kubernetes doesn't support revocation of certificates. If that person leaves, their kube config is is valid forever. Right? It's not like we say, well, turn it over turn it over with your badge and

3:14 your your pass. Right? Like It's not forever. It's for however long that certificate is defined for. So most certificates that Kubernetes gives you are issued for a year. Well, anything that changes within that year, you know so let's say you're embedding groups into the certificate, whatever. Like, Kubernetes, you can't change that. That certificate is self contained. You know, certificate or Kubernetes doesn't have any sense of users. It's not like most enterprise systems where there's an object that you update that says, this is what my user looks like. Kubernetes doesn't do that. So everything comes in through authentication. In this case, the certificate.

3:18 Certificate Lifespan and Revocation Issues

3:54 So who you are, how long you're valid for, and what your groups are. And so when you get that certificate from Kubernetes, that's good for a year. If you get walked out the door or you, you know, even more likely, you accidentally leak it, you know, because we're humans. So it goes into the wrong email. It gets shared onto the wrong folder. It gets accidentally checked into a Git repo. There's nothing as a cluster operator you can do to make sure that that certificate and private key can't be used to access the cluster until it either expires

4:28 or you rekey the entire cluster. So that becomes where and I've got a, you know, I've got a blog post that gets into the details on why you don't wanna use certificates for anything besides break glass in case of emergency. I did talk about it at Kubernetes security days couple years ago or security con, whatever they're calling it these days. But, yeah, certificates like break glass in case of emergency. You know, you get your cluster, you know, that shiny new certificate. You go put it in a vault somewhere that nobody has access to unless everything's gone

4:30 Alternatives: OIDC and Impersonation

5:01 haywire. And then you use, you know, I my preference is OpenID Connect or impersonation to access your cluster. So that way, you're letting your identity system drive who has access rather than trying to own that all yourself as a cluster operator. Nice. I mean, I I could go into that, but I don't know if I would not to be the where we go in this survey, but maybe I'll ask the questions. And if we decide to strip it out later, we could do that. Sure. It sounds like there's definite there's maybe two trains of thought

5:27 Exploring KubeConfig Alternatives & Service Account Tokens

5:29 here for what people could do. Where they have this KubeConfig and someone leaves their company. This this to me feels like a real problem that we should probably strive. And there are solutions, of course, you know, open use and then other tools. But, you know, with the straightforward KubeConfig thing, are there any admission controllers out there that can start to block access based on that authentication model because we know that that person or that key has left? Or maybe a second approach is, like, can we change the default, time to live on service account tokens issued

5:58 by the Kubernetes cluster? I mean, you can change the time to live on service account tokens. Service account token expiration is still pretty new. It became the default in 01/2024, but if you give so it it depends on the way the service account is generated or the token is generated. If the token is generated using the token request API and what's called projected directly into your pod So that's the service account when your pod fires up. You go to, what, the far run Kubernetes secrets, I think. That token now has an expiration date. If you try to use

6:37 that token after the expiration date, you'll get a warning. They'll still be accepted by default unless the pod that it was projected into was deleted. So I think it's one hour is the default right now for service account tokens. But let's say you instead had it for, you know, I think ten minutes is the minimum, which is actually what I like to go with, to be honest. But I'm in security, and I'm paranoid. So let's say you have that ten minute service account token and that gets leaked, but the pod is still running, you can still use that

7:12 service account token against the cluster by default. Now that'll eventually change, but because expiring service account tokens And mind you also point out, you should never ever use a service account token when talking to your clusters. It was never designed that for that. It was always designed for pod identities. It was never designed to be able to hand to an external system the way you might a certificate. So that's a that's another part of that. But, you know, the the big issue with a lot of these questions around in cluster are the things that we can

7:47 do or features that can be added to Kubernetes in order to get around a lot of these issues are the fact that that would require the type of interaction and integration with things that you don't really want Kubernetes to be in charge of. So think about it from a standpoint of, well, why doesn't Kubernetes just have users? Right? And why can't we just attach a password to it? Okay. How do you get the password to the user securely? Are you gonna email it to them? Like, are you gonna have a human email it? Are you

7:51 Why Kubernetes Delegates User Authentication

8:17 gonna have an external system email it? Most passwords required to be reset periodically. And quite frankly, I don't want you using password anyway. I want you using MFA. So now is the how are you gonna get that second token? You know, or you can type it in. That's phishing susceptible. You wanna use a hardware token. Okay. Well, that means that that needs to be built into Kube control. You know, we haven't even gotten to the point. So folks who are used to using, for instance, smart cards to log in to infrastructure. So US federal government mandates it. I imagine there

8:52 are probably more governments as well that mandate it. But that's a a standard called p k c s 11, being able to talk to that smart card to use that. You can't do that with Kubernetes, and they won't implement it. Like, SIGAuth has said, you know, there have been a couple of caps open to do it, and it just it's something that's not because the the thing is that the CA that your cluster uses isn't designed to generate authentication certificates for users. It's meant to generate authentication certificates for kubelets and for other pieces of the infrastructure.

9:25 So we're we're trying to use this infrastructure designed for machines with humans, and that interaction is very, very different. And you don't want Kubernetes to have to start owning all those different integration points. Okay. So I think it could be interesting just to confirm a few things there for the audience and actually for myself because I made a bit of a full path there that you kinda corrected me on, but very politely, is that I have a straight to service account tokens. But, actually, if we're talking about couplets and API servers and schedulers, they communicate

9:55 over the API server with x five zero nine. Right. But I believe service account tokens are are JWTs that are projected onto pods. Yes. Okay. Perfect. So yeah. It it's complicated. I mean, there there's a lot of nuance in there that a lot of people don't you know, even experts, you know, don't always keep track of. Yeah. I mean, we're in 2023 and, you know, off end and off set are still things that are are difficult for one. And they're just I don't think they're ever really gonna get easier because, you know, hackers exist. People are always gonna try to exploit

10:33 whatever system we put in place, which means they're always gonna circumnavigate the protections we think we we have in place. And our posture is always gonna have to continue to evolve. So, yeah, fun fun times always. Yeah. I mean, the the key to me is making it easy. Like, it it's just gotta be easy. It's gotta be simple. Easy and safe? Sure. Why not? I mean, I've I'll be honest. I've got customers that I'll securely log in to my Kubernetes dashboard for them on this thing using a strong token. If I if I need to you know?

10:44 Making Kubernetes Access Easier & More Secure

11:05 But, you know, I can have a FIDO two hardware token on this, even better, on my iPad. You know? It's a hardware token. It's biometric. So I've gotta use my my fingerprint to log in. It's a combination of that and a strong credential, and I'm working on my dashboard. I don't need coop control. I don't need you know? Yeah. I'm not doing major deployment work, but if I've gotta check the status of some pods or I've gotta, you know, kick off a cron job or something, That's a really easy way to do that, and it's secure.

11:42 The the the tokens are only good for a minute. So if somebody gets a hold of that token, you know, it leaks in one of a bazillion different ways. By the time they have it, unless they're, like, sitting there waiting to get it, by the time they have it, it's useless. So there there are a lot of things we can do to make it a lot easier. You know, easy, that's relative. Right? But easier, you know, if if I can make it so that accessing clusters is second nature, which from a security standpoint, I I think that goes a long way

12:17 to be able to to make users more secure. Awesome. Well, you know, we're fifty minutes into this episode. We haven't covered any of the questions that we we planned and prepared. I think with Fidelity, it's a good taste of what we're sort of getting. Typical day at the bar. Exact exactly. Alright. I'm gonna jump around the questions a little bit just because there there are a few things I wanna cover and, you know, we we know what those are, but we've already talked about, you know, tokens, short lived tokens. You're talking about authentication. And this is a topic that I've seen

12:45 OIDC

12:50 you speak about recently. Right? When you were doing pipeline identity at QPuddle in Toronto just last week. Wish we were both there. It was great to kinda hang out in person. So maybe you could talk about what the this, you know, what's that talk about? Why are we looking at tools like OIDC? How do these improve our posture, and why should people start taking this technology a bit more seriously? Yeah. So pretty typical use case in the Kubernetes world, you say, well, I and this is one I hear a lot. I don't want users talking in my cluster. I want pipelines talking

13:11 Pipeline Identity: The Static Service Account Anti-Pattern

13:20 in my cluster. Okay. Great. So how is your pipeline authenticating to your cluster? Oh, I give it a service account token. Service account tokens don't expire by default. So you're you're kind of back to that certificate place. Like, you can delete a service account from the API server, but, you know, you have to have that process in place. You have to rotate. You have to get that. Service accounts were never designed to work that way. And so the point of the talk was, alright. If that's the anti pattern, how do I get my pipelines to talk

13:49 securely to my clusters? So we presented two approaches. The first was let's pretend we're a user and do it that way. So we demoed a GitHub actions pipeline, authenticating to an Okta identity provider using a service account for the folks not on YouTube. I'm using my cody fingers. You know, not a Kubernetes service account, but just a generic service account inside of Okta. I log in to open Unison. I get my tokens, and I interact with my cluster the way I would with, like, any other user. And so the nice thing about that approach is we're not using

14:28 certificates. We're using short lived tokens for interacting with the cluster. But the major downside is now I'm building this custom automation between OpenUnison and Okta. You know, it's 10 lines of Python, which, you know, big deal, but Okta changes its login process now that Python's broken. And, of course, I I mentioned MFA and how important MFA is before. Right? I I said that. So, you know, okay. Now I have a credential where I've disabled MFA. Like and there are ways you can still automate MFA. You can do it with, like, TOTP, which is the venerable Google off, you know,

15:04 your six digit code. That's relatively easy to automate, but still it's it's clunky. So it's better but ain't great. The font the what the last thing that we demoed was this idea of using your workflow identity. So just about every identity or workflow system, GitHub, GitLab, they all have their own sense of identity. And so what that usually means is a JWT that is injected into the workflow, usually is an environment variable that you can then use to identify yourself to your workflow API, but you can also use that to identify yourself to external APIs like a Kubernetes cluster.

15:42 Implementing Workflow Identity with Impersonation

15:42 So what we did was we set up a project that we built called CICD proxy. It's a it's a deployment of the Kubo IDC proxy with specialized Helm charts specifically for this use case where it sets up an impersonating proxy. So, in Kubernetes, what impersonation means is you authenticate however you want on the front to a reverse proxy like an NGINX or something like that. And then you there are special headers that get added to the request going down to, the API server. And when the API server sees it, it does it on behalf of

16:19 the impersonated user. So my workflow can be authorized to very specifically update the deployment as an example. And so the nice thing about that is now there is no shared secret. The CICD proxy is pulling in what's called the OIDC discovery document. So that says, you know, who's issuing these tokens and what public keys do I use to verify it. The token itself is short lived. I think they're good for power on GitHub by default, which as these things go is pretty short lived. And there's no shared secret. There's nothing to to to lose. And if GitHub

16:58 decides to rotate its secrets, we'll pick it up because we're just pulling it out of an OIDC discovery document. Yeah. I wanna make sure I understand how this works. I mean and I have I've configured this before, so I'm gonna try and apply this as an example. Right? What we're seeing here what you're seeing is that we have GitHub who has this authentication system. They understand identity. They know when you're logged in because you've logged into their system, hopefully, with MFA. So there's a sense of trust there. Right? Now GitHub published their public keys. And I

17:28 think what that means is what we're saying is when we get that job from GitHub actions, say, which is saying with the GitHub key and is identifying as a user, it's got a UID, a subject, whatever that is in the job. When we get that and verify it's saying with the public key that we know to be GitHub, we can make some assumptions that we trust as this user Rawkode from GitHub. And as such, we allow that user to map or impersonate another role within the other side of the wall. It's a little simpler than that. You can

17:59 do it that way. But so every GitHub action, every run has a a JWT that identifies the workflow. So your your username in JWT Parlance, it's called the sub attribute or the subclaim. And it's I forget what it is off top of my head, but it's like organization slash project slash repo branch, something like that. But it includes kind of that critical information you would think makes a a branch inside of a project unique. And so that JWT says it was issued by GitHub. It's audience and GitHub's unique in that or in that it lets you actually generate a correct audience specific

18:46 for external systems. So Alex Ellis had written some code that showed how to do this where you can get a custom audience. It wasn't particularly complicated, but, I wanted it to be a GitHub action. So I took what I learned from him and created a GitHub action that we published, in the marketplace so that this is just a one line call. But that actions generates a JWT that says, signed by GitHub for your cluster, whatever you said your cluster is called. It was generated at this time. It expires at this time. And, also, don't use it

19:24 before this time because, you know, clock skews. Right? And so now when I receive that token, the c the the Kubo IDC proxy says, alright. Was it signed by one of these keys from the issuer? Yes. Good. Is it for my audience? Is it for me? Because I don't wanna accept in a token for somebody else. Yes. Good. Has it expired? No. Perfect. So now I take those claims. And for workflows, it's simply just the sub. You don't generally get groups out of workflows. And that's the user ID that I now send to the API server. So

20:01 Auditing Impersonated Actions

20:01 when I generate a deployment patch and I send the request down, that patch HTTP command contains an authorization header, which includes my reverse proxy's identity because that's the thing doing the impersonation. And then it also includes an header called impersonate dash user for my workflow. So, yeah, Trello security slash some workflow ref main. The API server takes that and looks for RBAC rules that associate with my impersonated user to make sure that I'm allowed to do the things that I'm trying to do, authorizes it, and then does the work. Then when you look in your API server logs,

20:49 the event will have two identities. They'll have the identity of the reverse proxy, and you'll have the identity of the impersonated user. So that way you can track what work was done by who even though it was done on behalf of the workflow. It wasn't done using the workflow's identity directly to the API server. Nice. There's a lot going on there. There is. But you know what? It's understandable, especially when you have someone like yourself that can walk us through the process and how it works. And I feel like I've already learned quite a lot in the first twenty minutes

21:15 Enterprise Kubernetes

21:25 of this episode. So awesome. Alright. Let's actually talk about what we were gonna talk about now. So the last time we spoke, you actually dropped on this little nugget of knowledge that I didn't know before. And it's that you wrote a book on Kubernetes and the enterprise. And what I'd love to know is how you ended up writing a book. It's not an easy commitment to take on. Like, I've I don't know who in the right mind would go, yeah. Let's write a book. And not only write a book, write a book about Kubernetes. There's, like, two strikes. And then third, I'm

21:28 Writing "Kubernetes in the Enterprise"

21:55 gonna write a book about Kubernetes on enterprise. Like, three strikes. Like, what the hell were you thinking? I'm a glutton for punishment. So how I got into the book was my coauthor. So I coauthored the book with a gentleman named Scott Zorovich, and he is also a customer of mine. So he's a customer of Trimble Securities, and he was working on another book that he was asked to do a review of. And so he was doing a review of the book, and they really liked the feedback he was giving. And they're like, you know, you really know your stuff. Would you like

22:02 Motivation Behind the Enterprise Kubernetes Book

22:34 to write a book? And so he said, yeah. That sounds like fun. And so he started looking at, you know, what goes into enterprise Kubernetes. You know, he works at a big bank, and he wanted to you know, what are the things that he cares about day to day from running large scale Kubernetes clusters? Right. Yeah. We all talk about things like, you know, service mesh, all these real sexy things, but, you know, nobody's like, backups. Oh, I want that. And so, you know, it was what are those things that, you know, load balancers. Oh,

23:12 I really wanna see those. And so he wanted to get into, you know, some of that nitty gritty that, you know, the enterprise world's just a little bit different. And so we started chatting, and he's like, you know, I wanna you know, what do you think about this? What do you think about that? And then after about an hour conversation, he's like, oh, this is stupid. Do you wanna co write it with it? I was like, yeah. That actually sounds kinda fun. And so that's how I actually got signed on to coauthor the book. You know,

23:38 for any aspiring authors out there, I will tell you you're not gonna make a lot of money off of it. It basically fuels my cone of coffee habit, but that's about it. But, you know, it's it's it's fun. It is a lot of fun to do. And then, you know, coming from an enterprise background, you know, our customers are enterprises. My customers, you know, even back in my private consulting days were enterprises for the most part. So, you know, we kinda looked around and said, enterprises are a little bit different. They're siloed. Right? That that's a big thing. You've got

23:53 Unique Challenges of Enterprise Kubernetes

24:11 a service provider. You know, we always talk about these hyperscales or these big service providers like, you know, Uber and Spotify and Netflix and where everything driven is to this one thing. Right? It's delivering the video, delivering the food, delivering the person, whatever. Right? But, like, everything drives into that one thing. Whereas enterprises, you have thousands of silos that can be anywhere from a couple dozen users to a couple thousand users. And to the people in those silos, what they're working on is the most important thing and the most mission critical thing in the world because that's what they get paid

24:52 for. And so when you deploy something like Kubernetes, and I know the DevOps gods will strike me down for saying silos are a good thing, but that that's just the reality of enterprises. Everything's siloed because that's management structure is a silo, silos of silos of silos. And so we wanted to build a a Kubernetes book that addressed the technology from that perspective. So, it's not really a beginner's book. I will give Scott really big props. He wrote scripts to be able to use Kine, Kubernetes and Docker for folks that aren't aware, where we use Kine

25:22 Book Contents: Essential Enterprise Kubernetes Topics

25:36 for almost all of the workshops and the demos inside of the book and all the hands on stuff. And, like, I use it every day. Like, I'm like, okay. I need a cluster. I'm gonna go run Scott's script and boom. I have a kind cluster up and running. So load balancers, authentication, we get into using OPA, Gatekeeper. We dive into GitOps, backups, Falco. The last chapter, we do, we actually build out a namespace as a service with GitOps. So Argo, OpenUnison, Tekton, and GitLab, and building it out so that, you know, you can log in, request act

26:19 you know, request the namespace get created. It generates all the Git repos and everything. So it really dives into a lot of the nitty gritty, and and it's more than a cookbook, but it or it's more than a theory book, but it's not quite a cookbook. Like, we didn't wanna say here are specific recipes to use. We wanted to say, here are common use cases and common things that you see in an enterprise. Here's how we're tackling them and why, and here are the exercises that you can use to to kinda get your cause sharp on it. Nice.

26:52 I mean, based on all the things that you've just kind of listed there, but what's covered in the book, the topics. And, like, if we were to draw a Venn diagram for, like, shut this important for the enterprise Kubernetes customer and then shut this important for your regular Kubernetes, you know, technology company Kubernetes customer. Like, there a lot of that is right in the center. Like, it's still applicable to people on both side. You know, the obviously, the communication structures, the legacy procedures, backup and restore Doctor, it's gonna be slightly different on the enterprise side. It sounds like

27:20 there's a lot of value for just anyone who's in this Kubernetes space that that still wants to handle get ups and backups and load balancers and, you know, those are all nice words to me. I like those words. Yeah. I there is definitely a big center area. I think where the change where where the edges are, where where that overlap isn't, is a lot of it is around things like policy, you know, stuff that we, you know Yeah. I don't like that word. Yeah. People don't like that word. I'm a security guy, and I don't like that word.

27:30 Enterprise vs. General Kubernetes Authentication Differences

27:52 But policy, you know, checking the box, it it you know, we all joke about it, but we have to do it. We have no choice. Sometimes legally, we have to do it. And making it so that there's a a understanding control. So give me an example. A lot of folks will say to me, well, my cluster runs in pick your favorite cloud, and that cloud has its own IAM. Why wouldn't I just use that cloud's IAM for authentication? And the reason is or the question is, do you wanna be, as the cloud team, responsible for when something doesn't work that you don't

28:29 have control over? In this case, authentication and authorization. So if I'm running an AWS cluster as an example, EKS. Right? I get that little config map. So so for folks that don't use EKS or or aren't aware, EKS has a config map that it uses for RBAC authorization. So that that's where you define your IAM roles that are authorized and you map it to users and groups that then Kubernetes uses for our back. Well, do you wanna be do you, as the cloud team, wanna be responsible for that? Sometimes the answer is yes because of the

29:06 way the organization is. And I find that a lot of times in service providers, that answer is yes because of the way that silos have been streamlined. But in enterprise world, do you wanna be the one that gets called into a meeting every time there's a problem even if it's not your fault? But you still have to defend your team and your situation. So what I'll often find, and this is true of any centralized service, where the first implementation is this attempt to build this really stringent like, this is our vision. This is how it's going to work. It's a product. Again,

29:42 folks listening at home using my coding fingers. And then the problem becomes all these little corner cases that come up, and you're not able to handle it. And now you're going in front of the you know, it might be the CIO, my director, whatever. But they're saying, hey. This is your fault because I don't have access to be able to fix these things. And so you get a pendulum swing where it says, you know what? I don't want anything to do with this. Everybody gets cluster admin. Cluster admin for everybody. Right? And then security is your problem. And so now you get

30:13 this big other swing where it's like, okay. We're gonna leave everything completely open, and I'm sure it'll be fine. What's the worst that could happen? And so that's where, you know, authentication and being able to say, you know what? I'm gonna delegate to something that you own so that you can make your own authorization decisions. We'll still enforce the authorization decisions, but you're gonna be responsible for making those decisions. And so that's how you then split that particular baby in saying, I'm gonna continue to manage and own the cluster operations, but I'm gonna allow the people in charge

30:47 of my customer that's in charge, the users, the the app admins, to control who needs and has access and when. And that that to me is the big difference between an enterprise implementation and most service provider based implementations. Awesome. So, you know, we're talking about identity and security and all this fun stuff. Right? This is I mean, I've seen a lot of trends, and I'm sure you've seen them too over the last six or seven years with Kubernetes with service mesh being huge and then get ups being huge. And then there's a least uptake in open policy,

31:03 Kubernetes Security Landscape & Supply Chain Trends

31:20 but it kinda got railroaded by the security and supply chain, not movement, but huge rock rolling down the hill in the energy on sale to the entire Kubernetes community. You know, even at KubeCon, you said, you spoke at the SecurityCon. It's got its own day now, but not only did it have its own day, it had the security village, but it had even more talks throughout the conference. So it it's very hard to to get away from what's happening with supply chain security identity and policy. And I'm curious, you know, you've you've been on this a long time.

31:49 You've seen everything progress and hopefully improve. But how has the landscape evolved over the last twelve months, and and what needs to happen over the next twelve months to get us in to quote, unquote, a good position? Are we already in a good position? How do you wanna tackle that? We're not in a terrible position. We're probably not in a worse position. You know, automation's always gonna be a double edged sword. Right? Because it makes it much easier to do things, which means it makes it easier to do security, which you have to do the

32:18 security. Right? You know, supply chain specifically, you know, I I kinda look at supply chain in two different aspects. Like, there's two really important aspects of supply chain security. There's the security of the image. Right? How many CVEs? How many known vulnerabilities are in your images? Like, that's you talk to any security expert. The first thing they're gonna tell you is most breaches are because of unpatched known vulnerabilities. It's not some, you know, state actor who, you know, figured out how to break into your cluster because there's a team of a thousand hackers figuring out it's you

32:54 know, they bought a vulnerability off of someone that, you know, from three years ago that never got patched, someone said, oh, I can use this. And so that's one that is easy for most people to conceptualize. Right? I have an image. I need to keep it updated. How do I do that? And there are lots of great tools for for doing that. But the other side of it is it becomes an identity problem. So, Sigstore is a great example. This is a really interesting technology to me. So what the folks at Sigstore did, ChenGuard, the name of the company, created this

33:17 Supply Chain Security: Beyond Vulnerabilities to Identity

33:27 project where they said, alright. We're gonna do image signing. So that way, you know that the image you're running was the one that you built. That's great. Right? But what's really interesting about what they did was they said, okay. We're not gonna make it about keys anymore in the you know, because we've doing artifact signing for what? Thirty years? But, you know, if the signature was invalid, we just hit continue and kept going anyway. Right? I mean, it it it had no meaning. So what they did was they said, okay. We're gonna make the certificates

33:58 just a mechanism, but not the important part. So when you sign an image using cosign, you authenticate to the cosign server, the CA, which gives you a temporary certificate, a short lived or ephemeral, whatever you wanna call it. Think it's good for ten minutes, key pair that you then use to sign your container. You then also write that information into a transparency law. So what you're validating from a policy level and from a signature level, yes, you do the cryptographic stuff, but really what you're validating is the identity of the signer. It's no longer about the

34:36 key. The key is inconsequential. It's the identity of the signer that becomes important. So if the identity of the signer is important, then you now have to go back through your infrastructure and make sure that that identity is not valid. Right? So let's go back to the GitHub action. That GitHub action generates a JWT. Great. You can use that JWT to get a certificate and sign your image. Who kicked off that action? Was it kicked off by a logged in user? Did somebody commit some code that that was a part of? Did that person have correct

34:43 How Identity Management Solves Supply Chain Security

35:16 access to that repo? So that's an identity problem. That's not a a key problem. Now let's get back to the idea of, okay. I've generated this this cert. Think about it from a GitOps standpoint. You're gonna have usually two repos. Right? You're gonna have a code repo and a, infrastructure repo. Your code repo runs. You want it to update your infrastructure repo to say, okay. Here's the new version of my image. And then you've gotta get that infrastructure repo into your cluster. You're gonna use something like Argo or Flux. Right? But how do you know that all

35:54 those things that are talking together are doing it with a real identity, a unique identity, and that identity that you can trust? And so the supply chain problem now becomes an identity problem, not a key management problem, to solve or at least a big part of it. And then you're increasing the security of your environment now because now everything's talking with these short lived keys and specific identities. And so everything else comes for the ride. And so going back to the where we're talking earlier about, you know, my service account talks to my cluster. If you're using that static service account from

36:33 outside the cluster, the identities that are generated from anything that gets created by that static service account are now in question because you have it's much harder to verify. So I'm actually gonna be out at Identiverse. Well, it's next week now. I don't know when this is gonna publish. It's it's the June, and that's actually an identity management focused conference talking about this and showing a a doing a lab of how do you securely provision your environment to actually get out of Sigstore what you're hoping to get out of it. It it's it's kinda like

37:12 I I like to explain it as it wasn't the astronomers that solved cross Atlantic navigation. It was the watchmakers. It's not gonna be the cryptographers that signed supply chain security. It's gonna be the identity management experts because it it it's that lower level problem that then bubbles up to everything else. Awesome. I think that monologue is a pretty good way to end the episode. That was great. Thank you. Alright, Mark. It's been an absolute pleasure. Thank you so much for your time today and for sharing all of this wonderful security information with me and with

37:40 Shameless Plugs

37:49 the audience. Please take a moment now to give us more information on your company, your products, any shameless plugs, anything at all that you want to share before we finish up. Yeah. And thanks for having me, David. I always have a blast. So you wanna learn how to, you know, simplify access to your clusters? Go to openunison.github.io. All our stuff is open source. Everything's documented there. You can ask issues on GitHub. Happy to handle there. And, of course, support contracts, always available. And then if you're interested in using our CICD proxy for for managing access for your clusters from

38:27 your workflows, that is cicdproxy.github.io. Again, % open source and, yeah, helps you automate that as well. That'll do your ingress controllers. That'll do your RBAC rules that simplifies that entire deployment process for you. Alright. Thank you very much.