0:00 Introductions

0:00 Welcome to Cloud Native Compass, a podcast to help you navigate the vast landscape of the cloud native ecosystem. We're your hosts. I'm David Flanagan, a technology magpie that can't stop playing with new shiny things. I'm Laura Santa Maria, a forever learner who is constantly breaking production. EBPF is Turing complete and can be written in Rust. So, obviously, I'm sold. That's just two of the things we learned on today's episode. Want to learn more? Contrary to not so popular belief, eBPF is not some secret Berkeley green project that's out there to eat your refrigerator, but it is a gateway to becoming a

0:37 Linux kernel maintainer. If you're curious about what e p eBPF is, why it matters, how to pronounce it, and how badly you can break your kernel when trying to learn it, this is the episode for you as we talk with Liz Rice, the author of Learning eBPF and Chief Open Source Officer at ISOVAILENT. Alright. Thank you. Les, could you please say hello and tell us a little bit more about you? Hi. Yeah. So my name is Liz Rice. I am chief open source officer at ISOVAILENT, which is the company that originally created Cilium. And, a lot of people will have heard of

1:00 Guest Introduction: Liz Rice

1:14 Cilium being based on eBPF. And earlier this year, I published a book about eBPF called learning eBPF. I feel like I may have answered more than one question in one go there. Hey. No. That's fine. That's fine. Little bit of context. Just us perfect. Yeah. Good context. Because I'm doing we're about to talk about eBPF and go into a little bit more detail there. So yeah. Awesome. Alright. Well, let's just start. You know, not everyone is familiar with eBPF. So could you give us the the TLDR? What do people need to know to understand the rest of the conversation

1:46 What is eBPF?

1:48 today? So they don't need to know what it stands for. It stands for extended Berkeley packet filter. But, honestly, forget that. It doesn't really help because it does a lot more than packet filtering now. So we tend to say the acronym doesn't really mean anything anymore. And what it really is is the ability to run programs within the within the kernel, within the operating system kernel. So we can dynamically change the way that the kernel behaves by loading these eBPF programs. And I think when I say that, I have to make sure that people really know

1:49 eBPF Explained: Running Programs in the Kernel

2:26 what I mean when I say the kernel. The kernel is the part of the operating system that interfaces between our applications and the hardware that the, you know, processor and and its peripherals. So if you are writing to a file, you're doing anything over the network, writing anything to screen, even allocating memory, the kernel has to get involved. Your application can't do it directly. It has to ask the kernel for help. And the kernel is also coordinating all the different processes that might be running on the machine. And that means the kernel is involved whenever you're doing anything interesting,

3:05 really. So it's a really great place to write things like observability tools and security tools, and we can do that with eBPF. And we also get to customize the way that the kernel behaves for the things that it takes on, things like the networking stack. We can modify the way that behaves with eBPF. So it's really powerful and a really interesting way to instrument all of your different applications that are running on that one kernel. Nice. And here I was hoping you could I was gonna say, here I was hoping you could actually explain why it said Berkeley in

3:37 Origin of the Name "Berkeley"

3:43 the middle of all of it, but that's a whole another thing, guess, more than anything else. The original bucket filter paper was written by two people whose names I can't quite remember right now, but they were at Berkeley at the time. So it was Okay. You know, the Okay. That makes me recite now. That it all originates from says, like Lawrence Berkeley Lab or however you say that's called. Yeah. You know, just for fun for somebody like me who's like, why does it say Berkeley? Okay. Anyway. So I I'm curious then. Did you join eBPF because you were really interested

4:18 Liz's Journey with eBPF

4:23 in eBPF, or are you now really interested in eBPF because you've joined eBPF? Like, what came first? No. I really got interested in eBPF. The first time I heard of it, I saw Thomas Graff, who is the CTO of eBPF, talking about psyllium and eBPF at back in 2017. And at the time, I thought, well, that's pretty interesting technology. And at that point, it was really cutting edge in the kernel. You needed you know, it wasn't available to most people in production. It wasn't available to hardly anybody in the Linux distributions that they were using back

4:59 then. But I thought this is a really interesting technology, and, I kinda kept an eye on it. A couple of years later, I started working on a project that was, sort of using eBPF. And I was also as part of learning about it myself, I was going out and doing talks. I I find that, you know, the best way to make sure you really understand it is to try and explain something to somebody else. So, yeah, I've started doing talks about eBPF as well. And through that, actually got invited back to the eBPF summit in, I guess, 2020,

5:37 which, iSurveillance put on sort of on behalf of the eBPF community. And there was so much really cool stuff going on, you know, in the world of eBPF and particularly at iSurveillance. It turns out I hadn't realized before that summit just how much iSurveillance had been involved with eBPF right from the get go. So Daniel Borkman, who's one of the three maintainers of eBPF in the kernel, was, you know, one of the early engineers at ISOvalent, and he's still there. So, you know, we're so embedded in the way that eBPF has developed over the years. It's it's, you know, I I

6:15 really do get to work with the the people who created it and the people who've been using it and had the vision for building things like Cilium. You know? So, yeah, I joined ISA Bayland because it's just absolutely full of really cool people doing really fun things with eBPF. Awesome. Makes sense. So you started experimenting with eBPF in in 2017. We're now in the latter half of 2023, which just seems absurd to me now. But, like, over those years, you've seen the adoption grow as we all have, especially across the the industry and even the CNCF with projects

6:45 The Rise of eBPF

6:51 like PEXA and Falco and, of course, Cilium as well. Why has that adoption grown so quickly for a relatively niche? I don't know. Is that a niche technology? I think it is. Like, why are people point, I don't know. I don't know. It seems like it's kinda everywhere. So Yeah. It's one of those things that, I I guess, expertise in it is pretty niche, but a lot of people are using it without really even knowing that they're using it. I mean, there's probably people using Cilium who don't realize that it's based on eBPF. Certainly, a lot of people will be using

7:23 things like TCP dump and never really sort of thought about eBPF, but and that that's fine. That's you know, there's so many really powerful tools that have been built. I I think, you know, things that Brendan and Greg have popularized, you know, in the kind of 2017 period, you know, he was already out there talking about how Netflix were using eBPF for, observability purposes, for tracing, for, you know, diagnosing and then improving performance issues and, you know, really popularizing the power of eBPF. The reason why I think there was that kind of sudden upturn in adoption

8:08 Factors Driving Adoption (Kernel Support, Distros)

8:14 is the fact that the the level of eBPF support in the kernel had reached a a point, I think, around the 04/2018 kernel version around that kind of time frame is where you really start getting sufficient eBPF support to do really interesting things. The more modern your kernel is, the more additional capabilities in eBPF and probably lots of other areas of the kernel as well. And but there was this real turning point when, I would say, particularly when RHEL was probably the last of the distributions to, kind of it it's it's always, you know, relatively cautious

8:53 about upgrading to to new versions of the kernel. And at the point where really all of the distributions were using a modern enough kernel, that meant you could just deploy these eBPF based tools in production regardless of your distro, and I think that really made a huge difference to the adoption. Yeah. Awesome. So you mentioned the Berkeley packet filter. And for the people that are not aware, it's like a networking thing that allows you to do IP tables like stuff and okay. I I'm not trying gonna go into it in any more detail next. I'll I'll make an

9:13 Extending Beyond Packet Filtering

9:25 absolute best of it. But it does network unique stuff. Right? It blocks packets. It rewrites packets. It does all the stuff. But eBPF has kind of grown beyond that now. We're seeing it used for a whole variety of different technologies like Falco and PEXY. Is there, like what's the right way to phrase this question? Really, I should have it prepared, but why has it extended beyond this? Why does it have these new capabilities? What is it enabling within the kernel for people? Why is it interesting to you and to others? I know that's a very broad question.

9:40 Why is eBPF Interesting?

9:53 But Yeah. So so I think the the original idea of packet filtering was to be able to look at each incoming packet and say, you know, make decisions about what to do with it with that packet. And I think in the very first place, it was really just do am I interested in sort of seeing this packet? I maybe I want to filter, you know, packets that are going to a particular port so that I can count them or something like that. So it was making fairly simple decisions about, you know, what kind of what to do

9:55 How eBPF Expanded: VM, Hooks, Maps

10:26 with these with these packet filters. The extended part involved, think, a few different trains of thought. One was the idea that if you extended this sort of relatively small instruction set that could be used to to examine packets. If you turn that into something a bit more kind of like a virtual machine instruction set, you know, when if you look at BPF bytecode, it's very reminiscent of, like, machine code. You know, it's all about registers and loading values into registers and and comparing them and and jumping to other instructions. It's very, very similar to to machine code. So there

11:10 was this idea that having a a virtual machine in the kernel could allow you to do all sorts of interesting things. There was the idea that maybe you could attach these programs to other points in the kernel, not just to incoming packets, but you could make decisions or change the behavior at other points in the kernel. And I think the last major thing that distinguished extended from from its predecessors is what's called eBPF maps. And maps are these data structures that you can access from within an eBPF program, and you can share them between eBPF programs, and you can also

11:49 access them using system calls from user space. So it's it's a way of exchanging information between user space and eBPF programs or between multiple different eBPF programs. And all those things kinda combined has turned out to be really powerful to the extent that one of my colleagues recently did a talk at one of the kind of Linux kernel developer conferences where he showed that eBPF is now Turing complete, which is pretty cool. A question just on that. Right? Because, you know, the talks I've seen, you know, from yourself and others in this space when you

12:25 Ensuring Safety with the eBPF Verifier

12:29 talk about eBPF, one of the the things that's always mentioned is the fact that the eBPF sandbox, the virtual machine can and should never crash. Mhmm. Is that is that still true even with the ability for the eBPF programs to communicate with each other and with user space programs? Yeah. Exactly. So the reason we're able to make that claim is because of a thing in the kernel called the eBPF verifier. So as you load a program into the kernel, it goes through this verification process, which is really analyzing all the possible paths through the program

13:03 and ensuring that well, first of all, it will run to completion. So this a long time ago, that used to mean no loops at all. Now that's been, kind of improved and optimized and you can have loops. It's, checking for things like, there's only a limited set of what's called helper functions that you can make from an eBPF program, and the set that you can call depend on really the event that triggered it. So if you were being triggered because a network packet arrived, then you can call helper functions that are related to looking at that network packet.

13:42 But you can't, for example, ask for what's the user space process associated with this packet because there is no user space process associated at that time. Whereas if you were in an eBPF program attached to a user space program making a system call, then you absolutely could ask a helper function to give you the the process ID. So the verifier is checking that all this sort of contextual helper functions are being used appropriately and that memory access is safe, that if you're gonna dereference a pointer, you have to explicitly check that it's not nil before you do so because dereferencing a nil

14:25 pointer, if, you know, anybody has ever written a c program, they will have crashed their c program by dereferencing a nil pointer. I guarantee it. Mhmm. Yeah. So the the verify is really just checking that that program is safe to run. Safe in the sense that it can't crash the machine, that memory access is safe. Because it can't tell the difference between, you know, maybe I'm a legitimate networking packet that's, an legitimate eBPF program that's filtering network packets. You know, maybe I'm protecting against DDoS attacks, or maybe I'm a malicious eBPF program, and I am just

15:05 throwing away packets for fun. Yeah. The verifier can't tell the difference between those two things. But when we talk about being safe to run-in this context, we really mean it's not gonna crash or hang the machine. Okay. Cool. And I think worth pointing thinking face on this. Yeah. I've got my thinking face on. Sorry. Go ahead. Go ahead, Liz. No. Keep going. Keep going. Well, it's the the thing that really distinguishes eBPF from writing a kernel module. So you always could extend the kernel, always, for a very long time. You've been able to extend the kernel by writing kernel modules,

15:27 eBPF vs. Kernel Modules: Safety Comparison

15:41 but people are pretty reluctant to install kernel modules because it's if if there is some kind of bug in it, if it does crash, it's gonna bring down the whole machine and there's no kind of safety net like what the eBPF verifier is bringing. Makes sense. So out of curiosity, like, obviously, we're talking a lot more in-depth kernel kind of things. And, obviously, you referenced if you've ever developed a c program, you know what this is like in a machine code and things like that. Who would you say are probably the most common user that you encounter or, like,

16:01 Who Uses eBPF? (Tools vs. Programming)

16:18 the the people who are really using this the most? Who do you think this is the most relevant for, I guess, right now? So lots of people will use eBPF through tooling that's built on it. So, I mean, David mentioned, you know, there's various different projects in the CNCF. There's all the, BPF trace and and, the BCC family of tools that people can use on the command line to do observability. There's there's lots of different tools that have been built on eBPF as a platform. And I think for the vast majority of people, that's how they'll

16:56 really experience it. They'll use things like Cilium or, you know, Falco or Pixi, and they may be interested in the fact that it's using eBPF, but they don't actually have to get involved in the details, which turns out to be a really good thing because, I mean, I'm I'm the sort of person who really wants to understand, you know, like, how the sausage is made. I I I I wanna kinda get inside and get a feel for, like, how is this really working? You can learn about eBPF programming. You know, it's relatively easy to get, you

17:16 eBPF Programming Requires Kernel Knowledge

17:32 know, things like a hello world or or, you know, some basic networking capabilities running in eBPF programs that you've written yourself. But you do quite quickly start hitting the point where you're interacting with kernel data structures. And at that point, you kinda need to understand what those data structures represent and what the effect of you changing them might have. So it it does quite quickly turn into kernel programming. So I kinda say that on the one hand, most people just don't need to know anything about the details of it at all. But if they're interested, it's

18:09 relatively accessible for people who are comfortable with programming to to kind of dip your toe in. And then if you really wanna become an expert eBPF programmer, which I, by no means, consider myself at all, that really does start to require kind of kernel expertise. But, fortunately, I work with lots of people, I said, I don't who have that kernel expertise. So would you say this is the the gateway to actually becoming a Linux maintainer? Honestly, yes. Yes. I I've not done it, but it never crossed my mind that I would ever make a contribution to the Linux kernel. But now I've

18:35 eBPF as a Gateway to Kernel Contribution

18:51 kind of in that world, I sort of start to think, you know, if if I had another twenty five hours in the day, that would come of there. The temptation is high to go get involved. I understand. I've asked my one obligatory question, David. Now it's your turn. Thanks. No. I'd like I'd like that I like that discussion. Right? Because it's it's one of those questions. Like, I do a couple of talks on that touch on eBPF. Right? I don't go deep on it because I'm not that smart. But I always do the same demos.

19:20 eBPF for Developers

19:25 And this is iovisor bPFCC tools demo. Mhmm. Specifically, I show off exact snip and open snip because, you know, in the SRE platform, DevOps world, it's quite interesting and important from a security perspective and an automation perspective to be able to show when certain sensitive files are opened on a desk. And eBPF makes that really, really simple. And another really cool demo is just by using execsnoop, you can actually monitor for sudo and site UID bit one binaries on the machine. And so when people elevate their privileges, you can get notifications for that kind of stuff too.

20:00 And the question is always like, how much do I need to know about eBPF to then start doing tools similar to that? And then you show them the source code and there's like 20 lines of Python. It's not a lot to do these kind of things. And I think that's because I feel that people can start to build eBPF programs just in traces without going deep into it. In the same way that with containers, we can all run containers on our machine, but you don't really need to understand what a control group as our namespaces anymore.

20:26 I feel like eBPF may make that same transition probably already has made that same transition. So I'm gonna flip that around a little bit and throw the question to Liz. If people do see these demos, they listen to this and they're like, okay. EBPF is really cool. What are the languages and the SDKs that they can go and start to work with right away to experiment with the the new tech? Yeah. So you kinda have to answer that question in two parts. There's the actual eBPF program itself that's gonna run-in the kernel, and then there's the user space part

20:35 Languages and SDKs for eBPF Development

20:57 that might interact with it. There are some occasions where you don't even need a user space part. So, for example, if you're doing networking functions, sometimes they don't need any kind of user space interaction because you can just load them into the kernel and they can they can do what they need to do. But, usually, we're gonna have both these parts. For the kernel, the program's gonna be in ultimately, it's going to be in eBPF byte court byte code form talked about. There being these byte code instructions that look like machine code. You could just write the machine code by

21:39 hand. Apparently, there are people who do that. But, you know, for for for me, I I would rather write in a in a slightly higher language than that. And the languages are restricted by being they have to be able to compile down to bytecode. So, the compilers that support it right now are Clang, GCC with both of which can compile c and also the Rust compiler. I'm not aware of there being other programs that support BPF bytecode as a target. So, yeah, c or rust really become your choice there. There is a little bit of a caveat

22:23 in that. There is this project called Iovisor BCC. David mentioned the the tools and and things like OpenSnoop and ExecSnoop that come from that project. And BCC gives you some friendly kind of macros such that you can write your code in a sort of hybrid of Python and c, and it and it takes cares of a few takes care of a few things for you, from your from your c programs. But then there's this the user space side of things, and there you have a much wider choice. I mean, really, you're you're not restricted at all except that you probably

23:04 want some SDKs that will make system calls for you and make it easier for you to interact with the eBPF program through that syscall interface. There are there's a Go SDK. There's a Rust SDK. In fact, there's a couple of Go ones. And there's a c one, which is probably, I would say, today, the most widely used called LibBPF. It Cillium uses Go. We have a a Go eBPF library, but I think a lot of the projects outside outside of that probably directly using LibBPF. Yeah. You said one thing there that I completely disagree with, and it's you said you

23:46 have a choice between Rust and Sea. That's not a choice. I knew this was coming. I knew it. Of course, David's gonna talk about Rust. I'm gonna just break in before he gets going on it. It occurs to me there was the mention of when containers came around and things like that, things kind of changed. And as there are more and more languages that people are familiar with that you can compile down to the eBPF bytecode. How often do you find people getting into trouble like they used to do when containers first came around? Because they didn't quite know what they were

24:01 Troubleshooting eBPF Programs

24:25 doing but they kind of got it enough to get away with it. So how often do you find people getting in trouble and like, how do you get them out? Like, that's always my question. It's like, how do you troubleshoot this thing? Especially when this is like kernel level and you can really, really mess things up fast. Yeah. I I would say that there's probably two major ways that people get caught out. One of them is around the kind of the tool chain and installing things that are compatible with each other because Got it. You

24:42 Common Pitfalls: Toolchain & Verifier Errors

24:57 know, every kernel version has different eBPF support, and then you need, you know, maybe your user space libraries that maybe are or aren't compatible and different distributions of Linux might have different, you know, versions of different, either libBPF or or tools, things like the the BCC tools or, particularly, there's a thing called tool that you can use for managing BPF programs and making sure that you've got a compatible set of the packages, the source code, the kernel, the tools that you want can trip you up in numerous different ways. And the other thing that catches people out,

25:43 once they got everything installed and everything seems to be compatible and they've started compiling some code and then they go to load it into the kernel and they hit verification errors. And, I I would say over over the last few years, there's been a lot of improvement in the kind of information that the verifier gives you about why it's objecting to whatever it is it's objecting to. But, I have certainly there there's a blog post somewhere that describes the verifier as a fickle fickle beast, I think it is. Nice. Alright. So, you know, we've covered a lot about eBPF

26:20 The Future of eBPF: Beyond Infrastructure?

26:23 so far, and just kinda to understand the landscape right now. Like, it's heavily used for networking. Selium has gone all in on eBPF even towards the service mesh angle now. As a villain even though have Tetragon going for the security angle and trying to help people with that. And, you know, we're seeing, like, PEXA and Falco to do more security and monitoring automated observability, all of this stuff. Is there as I I don't know if, like, there's an eBPF maturity curve. Right? But as people start to do more of it and the skills become more aware or people are using

26:57 it more, will eBPF creep into, like, our day to day application code? Do you see people using eBPF to write, you know, their CMSs or their, you know, proprietary applications? Like, I I don't know what those use cases are. I don't know if they exist. But will eBPF become more than what it is today? Which is a bit of a forward thinking question, but maybe there's some thoughts. It's really interesting to sort of speculate about what you could do and also what would be useful. But I think one interesting parallel is the way that networking capabilities

27:17 Potential Evolution: Functionality Moving to the Kernel

27:29 that used to be in user space have migrated into the kernel. TCP stack, you know, I am old enough to remember when that was more commonly in user space, you know, that you'd you'd use a TCP library. And now we we just expect the kernel to take care of that. And I think what eBPF will allow us to do is to gradually move more of that kind of functionality into the kernel, but in a way that doesn't require everybody to take the the leap at the same time because we don't all have to be running the

28:05 same eBPF programs. So I think something like service mesh is a really great example where, Cilium as a as a CNI networking component for for Kubernetes is in this really great sort of position in the kernel to be able to, you know, pick up network packets and put them where they need to be and observe them and report on them and do kind of security related operations on them, all of which are very much the kind of things that we expect from a service mesh. Today, we can't do everything in the kernel. I I mean, theoretically,

28:43 I think we could, but in practice, I think all the kind of layer seven operations, we're using a user space proxy to do that. We're using Envoy to do that. I think over time, you know, I expect that in, you know, some number of years time, we all of that code will be in the kernel. But maybe the you know, Kubernetes will be in the kernel too. Who knows? Maybe that's kind of future Yeah. Why not? You know? So fancy rewriting Kubernetes. Perhaps we should all do it in Rust in in eBPF. David's looking like he might actually do that.

29:24 Yeah. No. He might. He might. I mean, I am not personally writing any Kubernetes components in Rust, but people are exploring that these days. So you never you never know. Right? I don't trust that statement that you're not doing it yet. No. I I I know. I don't have time. I'm too busy talking to you. Alright. Well, is there anything else, Liz, you would like us to throw at you to ask before we we wrap this up? And if it is just sitting on the top of your tongue waiting to be said. Sorry. Yeah. No.

29:51 Upcoming eBPF Summit

29:59 There is one thing I would like to mention, which is the upcoming eBPF summit. Because if people are interested in, you know, hearing more about what's going on in in the kind of eBPF community, learning more about how it's being used, seeing some of the kind of interesting directions that people are going with it, and learning more about the future of eBPF itself. EBPF summit, it's online. It's free. You know, it's a virtual conference. This is gonna be, I think, the fourth time that we've held it. It's on September. And, yeah, come come join in. If you'll

30:36 go to eBPF.io, you'll find a link to the summit there. And, yeah, it's it's always been a really nice kind of community feel event, so I'm really looking forward to it. Excellent. Alright. Have you want to shamelessly plug anything else yourself, anything else, feel free to mention it now or forever hold your peace. We'll make sure all the links end up in the show notes as well. I guess it would be remiss of me not to mention learning eBPF. There you go. Available either for you know, if you want the PDF version, you can download that from isavailant.com,

31:03 Learning eBPF Book Plug

31:14 or you can, you know, order it from your favorite local bookseller or get it from Amazon if that's your bag. Shop local. Exactly. Alright. Well, thank you so much for your time. Pleasure. Thank you, Liz. Thanks for joining us. If you wanna keep up with us, consider subscribing to the podcast on your favorite podcasting app or even go to cloudnativecompass.fm. And if you want us to talk with someone specific or cover a specific topic, reach out to us on any social media platform. Until next time when exploring the cloud native landscape on 3. On three. One,

31:27 Conclusion & Outro

31:52 two, three. Forget your compass. Forget your compass.