1:34 Introduction to Litestream

1:34 Hello, and welcome to today's episode of Rawkode live. I am your host Rawkode. Today, we are taking a look at Litestream. It is a replicated SQLite with a whole bunch of really cool use cases that we're gonna explore a little bit today with its creator and vendor. Before we get started, there is just a little bit of housekeeping. First, please subscribe to the channel and also remember to click the thumb on the video. This will do magic things with YouTube's algorithm, which will help other people find this content, which is the goal here. Right? We wanna provide cloud

1:53 Housekeeping & Community Info (Discord, Subscribe)

2:12 native learning resources for those all to learn this fast landscape together. Also, we've got a Discord server available at rockwood.chat. It's a really great place to come and ask questions, ask for new different episodes on new technologies, and generally just chat Kubernetes, cloud native, go, and a whole bunch of other things. So come and join us. Great place to hang out. That's it. Boling bet done. Now we can get started with this session. And for that, I am joined by Ben Johnson. Ben Johnson is the creator of Litestream. Hey, mate. How is it going? Hey. How are doing, man? Yeah. Very well.

2:45 Ben's Background and Motivation for Litestream

2:48 Thank you. Do you wanna just take a moment and give us the TLDR on Ben Johnson? Yeah. I am a good developer. I've been doing it for eight years now, and I kind of always work in the world of databases for that time. And I wrote BoltDB. That's kind of another open source project I've done in the past. Just kind of a key value store that got pretty popular. It's actually in Kubernetes itself as part of etcd. And then, I don't know, just played around with a bunch of different open source projects over the years. And then one thing that

3:23 I've always kind of loved was just like that, like, embedded databases. Like, they're just they're so easy to use, like, kinda, like, just be up and running. There's not a lot of dependency issues. They're easy to debug typically, but I wanted kinda, like, SQL support. But I also wanted to kinda make it kinda more robust and, like, production ready. So that's kinda how I came down the the Litestream path. So, yeah, I've been working on Litestream for the last it's kind of been going on since the last maybe year or two. Just trying to figure out the ins and outs of SQLite

3:52 and figuring out how you can get this all to kinda work in different iterations and just trying to simplify it as much as possible. Nice. Does it take a special kind of person to work in databases? Like, these now worked on numerous that I'm aware of. And, like, these must be, like, the hardest thing in the world to be working on. Like, do you just like pain? Is that it? Yeah. I think it's just masochism, really. It's weird it's kind of a weird thing. Like, I have people that I work with that work in databases.

4:00 Discussion: Working with Databases

4:20 They always say, like, I am going back to doing just, web apps. Like, I'm doing something simple next time, and we always kinda come back to databases. But, like, it's a you think it's really difficult, but it it's actually, like, it's like a smaller problem space. Like, typically, you only work at, you know, you work with bytes and, like, it's very low level. Whereas, you know, I used to work in, like, web apps and front ends, and that's all, like, I got four different browsers to work in and different, you know, kind of a things that, you know, CSS problems. Like,

4:47 I get in CSS now and I just I get lost. So it's one of those things that it seems really difficult from the outside, but it just depends on where you're coming from really. I guess it all comes down to where you wanna focus your expertise and your domains and like, I guess every other domain is gonna appear challenging. I suffer the same. Like, I tried to write CSS a whole bunch of times. I've rewritten my blog 14,000 times in the last month and CSS is always trips me up. It's hard. But Yeah. Yeah. So it's it's cool that

5:15 Why Choose SQLite?

5:15 you're always working on databases. Maybe you can kind of then tell us what drew you to to SQLite. Sure. Yeah. I think as far as like, SQLite's kinda known for just being super rock hard, stable. Like, they put it in airplanes. They put it in all kinds of devices, and it just kinda works everywhere. The team behind it is just super chill and laid back. They're not always out, you know, on social media, like, touting how amazing they are and whatever stuff. It's kinda like they just you know, I feel like, like, people that are just, like, really database

5:45 nerds, like or they don't do all the marketing fluff. I feel like I feel like that's kind of a a good sign. So as far as SQLite, like, it's it's one of the few embedded databases that just kind of is ubiquitous and is has SQL support, like, strong SQL support. So you can do a lot of things you can do in something like Postgres where you get JSON support, you get, like, CTEs, like the recursive queries you can do in there, windowing. You get all kinds of functionality. It's kinda surprising for just a single small embedded

6:17 database. That's kinda what drew me there. Yeah. I think it was maybe shortly after your livestream announcement. I it it kinda got me thinking, like, I've never considered and I think it's just my own ex the way that I've worked with SQLite over the years, it's always been used for testing environments and never never really considered chipping it to prod. And I threw the question to Twitter. I mean, like, when when I say SQLite, what do you think of? Like, production, testing, embeddings, and stuff like that. And there were a lot of production SQLite people that were

6:20 SQLite in Production Use Cases

6:48 like, yeah. Like, definitely. And I I think it was just my own mindset just wasn't there and now I'm having to reevaluate that. So is that something Yeah. That you do yourself or receive other people that SQLite has a strong contender for a production database. Yeah. Honestly, I'm always surprised by how many people out there actually use it. And just like and it's just running stuff all over the world. And, you know, there's there's kind of a range of different applications even before Lightstream that you could use SQLite for a production database. Like, if you can do, you know, hourly or

7:22 daily backups with your data and, like, that window of data loss isn't gonna tank your company or your organization or whatever you're doing, like, that's that's a great approach. Just a simple replication. There's simple simple backups. So I think there's a lot of ways you can use it and still get a lot out of it. And it just it reduces a lot of the ops overhead. So if you're not an ops person, like, I'm not a big ops person, I think it goes a long way for that too. Okay. So maybe we can get a pivot

7:48 What Problem Does Litestream Solve?

7:50 over to Litestream then. And, like, what does Litestream bring to the SQLite world? Sure. Yeah. So Litestream, it's a single file on your data on your disk. And if you're running your application and then it just dies suddenly, your disks crash, and you lose all your data, that sucks, and, you know, hopefully, you have a backup. But the problem with, you know, historical backup solutions for SQLite is they are kind of a big snapshot. So you have to run them every hour, every day, something like that. So you get this huge window of data loss that, you

8:23 know, doesn't work for a lot of people. So what LightStream does is it kind of takes the changes that are written to to SQLite and then batches them up every second and ship ships them up to s three or some other replica. So that means So it's a go ahead. I see where it goes. So does does that mean then if I'm running Litestream against my SQL database that in an ideal world, I'm only ever gonna lose one second in the event that my application crashes or the failed system has a corruption or everything like that.

8:56 Yeah. So, ideally, it's one second of data loss. You also get other benefits too where if you did a drop table on your or, you know, a bunch of drop tables on your database and lost a bunch of data, it will corrupt it somehow just through, like, easier error. You can actually replay the database back into a previous point in time so you can kind of recover from those scene or scenarios as well. Ah, I I So you get a multiple level. Oh, go ahead. I I had no idea that point in time was a was an option when you

9:25 hook up Litestream. I I guess that makes a lot of sense. Right? I think you've got some slides to talk about how this actually works under the hood. Do you wanna pop them up and we can Sure. Yeah. Can pop those up. Yeah. Yeah. Oh, there we go. Indeed. Cool. Can you see my screen now? Yeah. Yep. You are live. Go for a bit. Cool. Awesome. So you got Litestream. So you I'm sorry. Streaming, like, replication for SQLite. And just kinda this is how SQLite works internally mostly, and then you can kinda see how Litestream

9:45 Slides: SQLite Streaming Replication with Litestream

10:02 kind of attaches onto that. So SQLite and, honestly, like most databases honestly, it's kinda ridiculous when you hear comparisons of, like, SQLite to, like, maybe MySQL's faster or Postgres is faster. Like, they're all gonna be just b trees storage on your disk. Like, they're mostly kind of structured the same. They have a couple different features, but you're not gonna get, like, wildly different performance from SQLite versus Postgres or whatnot. So this is what a table kinda looks like on disk or kinda looks like logically. You have kind of a root node and then it branches out. And then at the

10:36 very bottom, you have leaf nodes, which is where your actual data is stored. But all these different parts of your data or chunks of data are all stored in these pages, which are usually about four kilobytes, and they're structured as a a b tree. So what happens this is kind of a busy slide here. So the on the right side, there's the logical rep sorry, representation where you have this tree of how your your data is stored on disk, but that actually gets stored as just a sequential list of pages, and they kind of reference

11:08 each other, but they're all these four kilobyte blocks. That's kind of the important thing to know, and they're all stored in this a single database file. So when you make a change to SQLite, it has a couple different modes it can work in. Lightstream only works when you're in a SQLite to wall mode, which is usually the kind of recommended one to use anyway. Instead of overwriting the page on disk, it'll actually write it it'll append it to another file called the write ahead log, and that write ahead log actually kind of just it references to the pages on the database

11:17 How Litestream Works: WAL Mode & Checkpointing

11:43 file. So in this example, we have a database file that has pages zero through eight. And then if we make a change to something on page three, it'll copy or it kinda make it'll write the whole page three to the wall file. And then we make a change to page one, and yes. You see now you can't see my cursor. So it makes a change to page one, it'll pin that on. And then if it makes another page change to page three, it'll pin that on. So it kinda has this this layering of the wall file on top

12:11 of the database file kind of while it's running. Not sure if that makes sense. That's kinda how a lot of databases work. So, eventually, you know, this wall file gets bigger and bigger, and it'll do something called checkpointing, where checkpointing moves all those pages back into the main database file and overwrites everything for all those change pages. And then it'll kinda clear out that those pages on the wall file, and it'll restart from the beginning. So it's it's basically kinda like a circular buffer where it'll kinda fill it up and then restart from the beginning and just kinda keep going

12:45 in a circle. So what, Litestream does is we kind of unroll those restarts of the wall file. So your first wall file gets written out, and when it restarts, Litestream basically is copying those wall pages to a kind of station area. And then once it restarts, it'll start a new wall file with those new pages. So it kind of incrementally keeps track of all new rights to the wall even though that the wall kind of gets restarted and reused. So then we have the staging area with Litestream, and we start shipping out those chunks of the wall out to either an

12:47 How Litestream Works: Unrolling the WAL & Replication

13:28 s three buckets. We can do Azure. We can do Google Cloud Storage. We can do you do NFS over, like, a mount. You can do SFTP and yeah. Yeah. All kinds of different little replica destinations you can go to. And, effectively, once you ship them out there, if you wanna recreate the state of your database, we just pull down a snapshot, which is kind of your whole database at a certain point in time, and then replay all the wall pages that occurred after that up to a certain point in time. So that's kind of the

13:59 the basics of how that works. Does that make sense? It does make sense. Yeah. So I know it gets a little technical and nerdy with the some of database pages and terminology, but kinda really, the underlying structure is just kind of this append only set of data that Litestream keeps track of. It's all those wall page changes. So it kinda remain I don't know if you're familiar with event sourcing, but it's a really cool architecture that I've experimented with over the years, and it kinda feels like that. I got I never really correlated that to

14:22 Analogy to Event Sourcing / Kafka

14:32 write ahead logs in general, but maybe that is a Yeah. I mean analogy? Yeah. That's a decent analogy. I haven't actually built anything with event sourcing, but I know what you're talking about. I kinda think of it also, like, Kafka is pretty similar where it has this kind of append only set of events. You can replay all those events and get the same final state. So, yeah, very similar. So but look. Based on what you just showed, is there a rig? I mean, Litestream is doing really what SQLite is doing as well, which is, you know,

14:55 Replication Retention Policy and Costs

15:04 s right into the wall and then doing the checkpoints and the snapshotting. Does Litestream just forgo the the checkpoints and always stores the different walls in the different iterations or do you actually merge them at some point? I'm wondering how far back in time does the Justin, the point in time recovery go with Litestream. So they should take care that one. Yeah. No. That's that's great. Yeah. So what Litestream does is it has kind of a retention policy on it. So by default, it's about twenty four hours, and then it'll do a new snapshot and kind of start

15:38 from there with the wall again. And you can configure that snapshot more often, but and then retain longer. So if you want to retain, you know, for seven days and maybe do a snapshot every day or every hour, you could do that. So your your point in time recovery is however long that retention period is. Okay. And, you know, if you have stuff like s three, like, three, the way that AWS really makes money is kind of when you download files, but for replication, you really just upload files. So it's actually super cheap to to replicate

16:10 up to s three, which is great. Yeah. Plus it supports is that any s three compatible API, or is that like a a subset that you've kind of just tested that against? So it's any s three compatible API. The only caveat I would say is that it did not work on Google Cloud Storage, which has a compatible API because it wasn't that compatible. But we had a native support on the last version. So it's works on there in Azure as well. Awesome. Alright. Alright. Your question is, can you say more about why Litestream unrolls the wall

16:38 Q&A: WAL Unrolling & Recovery Time

16:43 pages at checkpoint time? Wouldn't you just need the most recent wall pages plus a snapshot to replay or build the database? That makes sense to you? Well, I know think. So it's would you only need the most recent wall pages plus a snapshot to replay it? So it it doesn't keep yeah. You could keep a unique set of pages, but I think that'd be difficult to track over time. Kind of the doing the unroll and pushing up to s three makes everything immutable, which kinda simplifies a lot of pieces too. We're not trying to merge merge pages together.

17:18 If I understand your question correctly, you could technically merge those, but but, yeah, it could be for future feature. Alright. Nice. Thank you. So is it fair to say then that the recovery time of Litestream is is pretty fast depending on the size of your database, of course? I'm assuming that you're just pulling down the snapshot, the wall, replaying it, and then your application is is up and running again. Right? Yeah. So it definitely yeah. It definitely depends on kind of your your length of time between from your snapshot to where you wanna replay to and how many pages you have.

17:54 It does pull down the the individual files in parallel. So it tries to fill up the pipe, you know, your your bandwidth, getting those down. And then the actual replays of the walls is pretty fast depending on your disk, but it's it's literally just copying four k pages from one file to another. So it's nothing it's nothing too crazy. Nice. And I I like I hadn't really thought about the drop table scenario. I mean, I think we've all done it at some point in our careers, I would imagine. But been able to have that point in time

18:22 recovery to revert that would be very, cool. Almost the same performance. I wish wish I'd had that at various points in my career. So awesome. Yeah. Alright. Anything else you wanna share on Litestream before we get hands on? Did we did we touch on all the the major bits there? I would say that let's see. That's all the major bits. One thing I would say as far as some kind of some feature work that's kind of coming up, there's stuff we're looking at around doing live replications. You can have multiple read read replicas. So if you wanna start growing

18:35 Hands-on: Setting up a Local S3 Target (MinIO)

18:57 out your kind of a cluster of SQL like applications, you can have you can scale your reads for that. There's, you know, work being done around looking at encryption, so you could replicate up to an untrusted data server, replicate destination, things like that. So there's a lot of cool stuff coming. Awesome. Alright. I believe you've got a a Slack channel and stuff as well, so we can drop links to that in the chat and people can Oh, yeah. Get involved as well. Let me pull up my my screen share just now. So this is the Litestream homepage.

19:30 It is available at litestream.io. And today, we're just gonna kinda run through the getting started guide to give you all of idea of how you can start using Litestream for yourself. If you do have any questions, feel free to drop them into the comments and we will do our best to get them answered. Now you've already watched me do this once before. You said that was a lot of fun. Right? You did a great job. I was impressed. You got a q three s up. Yeah. It was good fun. I really enjoyed working on that. Working on that.

20:07 I'm gonna jump ahead first and just get a manual running and a Docker container. So this is the s g compatibility manual. We can just run locally without ever having to speak to Amazon s three, which I am a Mhmm. Big fan of. Really cool software. I was used to use the AWS s three for the getting started, and just it was very confusing to anyone that's never set up, you know, IAM accounts and AWS buckets and all that stuff. So It's definitely one of big challenges. I mean, I have set up IAM buckets and roles and groups and all this before,

20:41 but and it's still complicated. Just Yeah. Oh, yeah. You do one wrong thing and you just gotta Yeah. See what happened now. Figure out what you did. Super painful. Yeah. Yeah. I do not recommend it. Oh, Was this here last time? I don't remember a tap. Oh, yeah. Yep. Go tap on there now. Fancy. Alright. Let's split this terminal. I don't even I'm obviously just gonna be learning stuff today. I don't know you could do brew install and then the tap and then the package name. Does that automatically do the brew tap for me? Yeah. It automatically hooks it up. I can't

20:56 Hands-on: Installing Litestream

21:21 remember who told me about that, but, yeah, I used to have the whole instruction of, like, yeah, the tab and you do this and update through, but, yeah, it's great. Just one line. Awesome. Yeah. As long as I keep learning stuff with this, I'm a happy person. Oh, where is all? Is this all is this compiling it from source? Or is this just because I break my machine last week and it now wants me to do loads of silly stuff that I haven't done in a while? Let's find out. Yeah. It shouldn't be that's weird. It shouldn't

21:51 be building anything. It should just be pulling down the binary. Yeah. Oh, it's just my machine. I I said this to you prior, but I was Yeah. Share my stupidity with the audience. It's that I installed the Mac OS 12 really early developer beta last week and nothing on my machine worked and I had to revert everything back to the start. So, yeah, there's gonna be loads of these little things over the next couple of weeks, I would imagine. Yeah. I guess I could always just grab the from get With a fast connection at least. That's

22:23 good. I do. I got fiber with my ISP recently and it's Nice. Pretty solid. Yeah. I'll grab the binary just now. Yeah. That works. Actually, take out the dot io at the top. Yep. You're On the yeah. We got a bunch of different support for different operating systems and architectures too. So we have You feeling brave while I go with alpha? No. No. Don't do that. Or because some refactoring going on for dot four, but let's yeah. That's fine. I mean, I'm ready. For a lot. Yeah. We got, yeah, Mac, Linux. We got Windows support on there, OpenBSD.

23:13 We do a bunch of different ARM chip versions too. So you can go all the way down to what do they call it? Raspberry Pi zeros. Yeah. Those tiny little $5 ones. You can run Litestream on there if you want. That's pretty nice. Okay. Let's see. What did I grab there? Litestream, zip, Alright. There we go. I'll just let x code do its thing in the background. Okay. And I'll get the brew tap version another day. But it's nice that they exist. I don't know if I missed that or I've just forgotten that I used that. I'm pretty

23:51 sure I downloaded the binary last thing, but brew tap should definitely Yeah. Think it's been there for a little while. Especially with the Linux support now, which I haven't used a great deal because I'm an arch user, so I use arch for pretty much everything. But for real analysis, it's pretty cool. Alright. Litestream. We've got Menio. We've got our credentials without any IAM policies, Menio admin and Menio admin. But I think we are probably good to get started. Seven minutes now. That was a minute a second ago. Alright. Okay. So we should be able to browse to menu

24:08 Hands-on: Creating a Sample SQLite Database

24:25 on the web interface. I guess I'll just need to create a bucket. Yep. That's all I gotta do. I will say the interface always throws me off. I'm always looking everywhere but the lower right hand corner. Yes. It's done that to me a few times. Alright. I'll just call this bucket Litestream. There we go. We've a bucket. Yeah. We've got a comment from Russell in the chat saying, kill product database backup save careers. That indeed is true. Also, hi to Jack Lindsay and some way earlier. Oh, yeah. We gotta we gotta wait to okay. So we've created our bucket and now

25:00 Creating a SQLite Database

25:06 we wanna create a SQL lite database. Do you have any tips for the people watching on tooling at you guys for working with SQLite, or do just keep it pretty generic with the CLI? You know, I keep it pretty generic. We do have, like, a tips and caveat section inside of the Litestream website that gives some tips for just running it with different settings and how they can kind of affect and help things just from more of an application development standpoint more than anything with Litestream. But for the most part, it just kinda works, especially if you're if you're in wall

25:39 mode. That helps a ton. And you could create, like, concurrency, like, reconurrency with that. So I've gotten just crazy performance, honestly, with just writing I write everything Go, like web apps and everything. Yeah. Go web apps with SQLite back end. You just you have no latency, like, per query latency. So I typically you know, I can run several queries, render HTML, and you get it all back in, you know, fifty microseconds. So it's pretty snappy. Yeah. I guess because of it's in wall mode. Right? Ever does that mean everything runs in memory to a certain degree, you're never

26:16 actually really touching the desk until snapshot? Well, I mean, naturally get a lot of memory stuff anyway. It has a bit of a cache inside of SQLite. I think it uses a map as well, so you kinda get the OS page cache as well. But the wall stuff is it it basically has different versions of your data. You know, there's different pages, and I have different versions. So, you know, a transaction started, you know, at first or at one point in time has its own view of the database, and another one can start later on

26:46 before that other one finishes and get a different, you know, snapshot of the database. So it really helps with that. I guess if you also throw some NVMe discs on this, it's probably gonna give you a nice performance. That can't hurt. Yeah. Yeah. Alright. Let's export variables. Yeah. Also, also just to with the stuff you're doing right now Yeah. You know, we're just creating a database that has a list of fruits in there. And we basically wanna replicate this database out to Litestream, and you can kinda see how quickly it will give you the the

27:20 new version of that just kind of in the background. You don't have really even have to think about it, and then you can kind of restore that easily as well. Yeah. So we've just got a simple create table and set a couple of through it. I don't can I just drop out of this now? Yeah. Okay. Yeah. So we've got that. Alright. Actually And there's a DB. You see I I sensed that Yeah. In your voice. Oh, yeah. Sorry. Yeah. Stay in the the database there. One thing is you can I mean, it should be

27:52 fine, actually, but you're gonna have to have multiple windows running to kinda really see it? How do I use the database? I think you you hit command d. I think it'll exit you out. So can I just select Control? Sorry. Control d. Alright. Okay. Yeah. I just Oh, sorry. Was sorry. I see what you're saying. There's only one database SQLite. Because it's SQLite, and it's a fail. I'd alright. Okay. Mhmm. I will say one other nice thing about SQLite that people don't think about is you can get, like, actual tenant isolation. Like, you can put a tenant per database

28:29 file, and that works totally fine. So there's just there's a lot of cool little things you can do that you wouldn't really think of naturally, I would say. Alright. Let's get one more of these running and we copy oh, no. I did copy something, but I've lost it. Oh, yeah. The token. Okay. Alright. Yes. Yeah. Okay. So this is our Litestream. Let me zoom in on that a bit more. Our Litestream command. We just call replicating. L. You're missing the l at the beginning. There you Just wanna make sure you copy the whole thing. I I see. I think it's

28:52 Hands-on: Running Litestream Replicate

29:03 little copy all buttons on the docs. It's killing me. Those really handy. Definitely. Yeah. I always say that to everyone I have on this show. Every time there's a mistake, I guarantee you it is my fault for some reason or not. Oh, man. See, one of the other things that I really do on the show that I'm starting to hate myself for is that I use temp directories for everything and now I'm no longer in a temp directory. Now I can either download Litestream again or if very quickly that find it. Yeah. This is one of those moments where

29:44 I need to stop using temp directory. Right. There we go. Right. We still got our fruits. I should still have my export. I also use pair directory history, which I love and then hate on the stream as well. So Alright. I'm just gonna type this from memory because it's so intuitive and easy to use. But replicate through it dot dbs3 local host three So we're actually gonna put almost Litestream dot local host. So you're gonna put the bucket name before local host and dot. You sure? I don't think so. I could be wrong. I think you're wrong. I know you're the

30:28 author of the software, but I'm just gonna go out there and and say, I think you're wrong. Damn it. So is that something Yeah. So there's two parts. There's the what was that? Go ahead. I mean, I've never seen, like, wildcard DNS aliases in that way for local host. Is that a thing? Yeah. I mean, we're just using the URL kind of to pull all the data together, kind of all the the connection info together. It's really more of a connection string than anything. And, typically, it makes probably more sense when you have, like, an actual s three bucket

31:06 where the s three buckets are typically referenced as s three colon slash slash and then just the name of the bucket. So it's a little bit hacky for Mineo, to be honest. Well, that looks successful. Looks yeah. So it starts off I'm give you oh, go ahead. No. It's okay. Go for it, man. Yeah. So it starts off to saying it's replicating and kinda gives you the database or databases that you're replicating, where they're going to, and then how often it's gonna sync. So by default, it syncs every one second. And the whole there's some talk about generation in

31:44 here. A generation is just basically when you have a snapshot and then kind of a contiguous set of of wall pages after that. So you can get in a situation where, say, you you stop your Litestream process, your application keeps going and then you start it back up again, your your other process may have already checkpointed that data and kind of gotten rid of it and Litestream would have lost its place. And Litestream will just detect that automatically, and generations are just kind of where it can kind of kinda start over. It'll restart re snapshot and then just keep replicating from there.

32:17 Very cool. So let just go back to we've got our manual browser open. So I just wanna kinda see Mhmm. See the thing. There's the generations you were kinda talking about. And we've got our snapshot. Yeah. We've got our right ahead log. So Mhmm. I mean, we're we're pretty happy that this is working. Alright? So Yeah. So far, so good. If I insert more groups, we're gonna see a bit of activity on the Litestream side. Is that pushing more stuff? No. It's not it doesn't log excessively. It typically logs in kind of major events, you know,

32:53 like snapshots and then errors. Right. So it tries to keep the log relatively light. Otherwise, you probably get a lot of activity. So but and it probably would be worth adding an option to make it more verbose. Well, I am going to oh, my computer is not happy with me today at all. There we go. You jumping back? I should just if I could choose Linux for streaming, I totally would. Things just work on Linux. So I never thought I'd say that. But alright. Let's add. Everyone agrees with you on that. No. Definitely not. Peach. What color is a peach? Peach color?

33:32 Yeah. Yeah. That sounds good. Like peach peach. Okay. So what is the the best way, you know, for people that are playing with livestream? You've mentioned that there's not maybe a lot of logging coming out here. Like, is there any way for them to just check that the replication is is happy and working? Or if there's no errors, is that thumbs up? You're good to go. Go have a beer late. Yeah. So generally thumbs up if there's nothing going on in there. It's it's kind of come like a constant loop that's running. That's that's checking and kind of uploading stuff. One

34:04 Checking Replication Status (`litestream generations`)

34:04 thing you can do, there is a some stuff from the actually, if you jump in, I guess, another window, I if you got room for one more. Always got room for more terminals. There you go. And then kinda have to go back at the same temp directory that you were before. Yep. Alright. Cool. So, yeah, if you just type Litestream and hit enter, it'll show you some commands. So, yeah, if you you can so yeah. It's a little bit small, but so there's kind of three basic commands outside of so there's replicate and restore, which

34:30 Litestream Subcommands

34:43 is where you kind of push out your data, and then restore brings it back and kinda recreates it. The other ones are databases, generations, and snapshots. Databases will list the databases you have if you're using a config file or just kinda more the recommended format, you know, just kind of playing around with it. And then generations generation snapshots and wall are the other ones that kinda see the state of your actual replica information. So if you do Litestream generations and then put in the name of your database or just a local path there, Yeah. Hit enter.

35:20 Oh, shoot. It's the put in the Rawkode URL. So the s3colon slash slash. Alright. Relate stream. And then Litestream. Yeah. Local post 9,000. And slide. Yep. Like that. I don't know why it's taking so long. It usually is pretty snappy. Oh, that's because you don't have credentials in there. That's right. There you go. There we go. Okay. So that'll show the lag on there. So right now, it's two minutes and nineteen seconds, which is the last time you put that's when you put the peach in peach bro in there. But if you add another fruit in there and then

36:06 run your generations command again, then that lag should drop. I mean, I'm feeling pressure. Pineapple. There we go. And And, actually, if you before you do that, if you run it, you can run that generations command on a watch if you wanna see it return immediately. And you can even go pretty pretty low. Oh, there you go. Again. Always gonna happen. My fault for not For for breaking my computer. I'm gonna I'm gonna honestly, I'm gonna That's right. Day for the next month. I'm just always gonna think that one little thing that never worked. Yeah. No. I was happy with pineapple as

36:48 my choice there too. What's your thoughts on pineapple on a pizza, Ben? People wanna know. I know. That's sacrilege. You know, I do cheese pizza. Like, I'm vegetarian so I don't do meat, but I'm pretty simple, to be honest. Alright. Okay. Cheese pizza. Alright. So Do you what's that? Cheese pizza is oh, cheese. Alright. Yeah. So if you enter that in there, then pretty quickly, it should should lag that down. So now it's down to, you know, one second, two seconds. So that's it's grabbing the kind of the last timing or time out of the wall

37:25 file that's up there on menu. Alright. I'm gonna pick the color first and then try to think of it through. Okay. Red. Damn it. It's what I do. Cherry? Cherry. Good catch. It's probably fair. It's pretty quick. Like, I mean No. Yeah. It's snappy. Yeah. It's usually about a one second delay, so you'll usually see it sub steps. I'm curious. Is is that one second, you know is is that arbitrary? Is that something that you've picked as part of the code? I mean, could people get that lower if they wanted, or is that just a sensible default?

37:50 Sync Interval Configuration

38:01 Like Yeah. There's kind of a trade off between kind of the time to push out. It's not even a really performance issue for Litestream because it can usually check pretty quickly about the underlying data. You could drop it lower. Usually, the issue is that you get charged, I think, it's, $5.10 thousandths of a a cent or something like that every time you do a post up to s three Mhmm. Which is pretty cheap. And if you're running just a constant stream of rights to your database, then dropping it below one second, you'd start getting expensive.

38:36 It used to actually be ten seconds was the default, but people usually don't have a constant stream of of rights to the database. So it seemed like a sensible default dropping down to one. And is that configurable if I wanted that on a minute or even ten minutes or whatever? Yeah. Yeah. Yep. That all that totally works. Nice. And that'll actually kind of stop your it'll cut off your wall and kinda restart it automatically every minute so you can get that. Like, your your point in time recovery is kinda determined by that kind of how

39:05 often it restarts that wall just because you can only have a certain granularity to replay into. Okay. I also noticed If that makes any sense. Yeah. I think so. I'm getting curious, like, the SQLite commands that I executed were just there was nothing that told it to use the wall. So is that something you get for default? Or because you mentioned I think you said at the start, like, it had to be enabled that way. Can you maybe elaborate on that? Sure. Yeah. So, like, stream auto, think we'll put it into wall mode if it's not

39:13 SQLite WAL Mode Auto-Enablement

39:37 in the wall mode, which was a decision I was kinda back and forth on. But, honestly, most like, it's usually the recommended mode to run-in with a few exceptions. It's usually if you have like, lot of people will use lights or SQL lights as kind of a file format. So because it stays in a single file. If you really want it to just be that single file, then you can use their other journaling modes. But wall will have kind of two separate files that'll work with, but it just it provides all the concurrency support. It's way faster. There's a lot of

40:08 there's a lot of big benefits to it. So it just automatically triggers it into wall mode. And then that that persists across processes. Yeah. So that means if I have you know, an application that's been using SQLite for for years, I don't really need to compare or do anything. Like, when Litestream kicks in, it's just gonna go, hey. You should be in wall mode now, and it does the same thing. Yeah. Yeah. And I've actually seen I've seen a lot of people actually just put this onto all kinds of different app like, existing applications without that application knowing

40:35 Using Litestream with Existing Applications

40:39 anything about it. So I've seen, like, ghost blogs. I've seen a bunch of people do that. I think I just saw somebody do it with WordPress. You know, they can use the SQLite version, and it just kicks in. They've deployed in, like, a little Docker container and everything. So it's kinda cool. Some people actually run this. Like, I usually recommend running with a persistent disk just to get, you know, add additional durability guarantees. But some people run Litestream with no just an ephemeral disk, and they just, you know, go crazy, and it seems to work. So

41:11 I'm I'm not that brave. Yeah. Lots of stuff to experiment with there. I've already got a few ideas from my head. Like, oh, I could just, a a RAM disk and go a bit crazy with that. But No. Yeah. I mean, you can if you want. I mean, again, there's, kind of that window data loss where if you had a persistent disk, it would, you know, be fine. I mean, technically, Litestream will flush out changes, when it powers down or when it shuts down cleanly. Mhmm. So, generally, it should've worked actually to run on an ephemeral disk, but

41:39 I'm just super paranoid. So we got one more question. We've got a question on the chat. We'll tack on a second, but something else popped in my head and if I don't ask it now, I forget. But let's say I've got my application, it's using SQLite, I add Litestream, it does some stuff, I shut down Litestream, forget to start it, erase some data to my SQLite, and then restart Litestream. Is Litestream okay with that? Yeah. That'll be great. It'll just pick up where it it'll send it'll notice that the the walls changed, you know, especially if it's been running long

41:42 Q&A: Litestream with K3s (Kubernetes with SQLite)

42:14 enough and enough rights go through. Then it'll sense that, you know, that wall is kinda restarted and it needs to start a new generation. Now it'll just pick that up automatically. And it doesn't it cleans out the old generations based on retention. So you actually still have that old version of your data even though there's kind of that went that gap of time where Litestream wasn't replicating up. Nice. I like that a lot. Okay. Let's tackle Riel's question then, which I promise you I've not paid anyone to ask. But Rio is asking, would this work for SQL lite in case yes?

42:49 It was a fantastic question. It is a fantastic question. Do you wanna tackle that? I'll let you answer that one. Alright. Okay. No. I think I think you're the expert on this I actually Rio spent a good few hours experiment with just that. One of my fascinations at the moment is cluster API, which uses Kubernetes in a weird way, essentially like a database and KCS with SQLite and Kane. I thought it would be a really cool way to hook Litestream up to that replicate s three. So hopefully, the exact same use case that you're talking about. And the answer

43:19 is yes. It worked perfectly. I didn't even need to do anything other than read the documentation, run Litestream, and it just worked. I have had cluster API running for days. I've shut down the machines, spun it back up and things just worked. So that is a 100% yes on KTS and Litestream. Okay. Let's jump back over to our CLI. We don't have too long left, so we'll run through just a couple more things, I think. Sure. I I guess you can run through the restore. That probably makes sense. Yes. Do you wanna cover anything else on these

43:51 Litestream Subcommands Overview (`snapshots`, `wal`, `databases`)

43:56 subcommands? Because we were Yeah. Sure. Sure. Yeah. I mean, the snapshots in the wall are just gonna list out the snap like, the little snapshot up regularly, just the full database at a certain point in time. And then the wall, you know, you can see all the little individual files that have been pushed up. But, generally, you only need those for debugging if you have issues or just kinda wanna see some stats. But nine times out of 10, you're really looking at restore and replicate and then running generations if you just wanna see the lag of

44:26 replication. Okay. So let's do a restore. I'm assuming the the getting started guide is probably going there anyway. Right? So yeah. Yeah. Restore. So why don't we I can just close Litestream, delete the Fritz DB, and then trigger a restore. Just looking So you have to keep Lightstream what's that? Keep keep Lightstream running and then run the restore on a separate window. Is that what you're saying? No. I was gonna delete the first dot DB and then run our restore to pull it back down. Are you suggesting something else? I always see what you're saying. I was

44:32 Hands-on: Restoring the Database (`litestream restore`)

45:10 suggesting restoring it to a different location. Ah. Okay. Alright. Let's do it your way. So we're gonna do a restore oh, yeah. We've got this output flag, so we can say, fruits two dot d b. My very imaginative naming. And then stream local host. I mean, this is I'm confused. I always we we I need to work this. Litestream.local host. I know I'm coming back to this, but it's just it's not sitting right in my head. Is that something Litestream is doing to work out the bucket name or is that actually something that is a convention on computers? Because

45:59 I'm a little bit shocked. No. No. It's no. It's just it's kind of a hacky way of getting many o to work on the command line or, like, with a single URL. So the URL references is kind of just a a locator for words replicating to. Okay. So it part it it notices that there's local host reference in there, and then it extracts everything before that to get the bucket name. Okay. Got it. I could I'm that means I'm not gonna lose any sleep tonight now because I was definitely worried about losing sleep. No. Yeah. Because there's a couple of like,

46:31 other services like I think it's either Digital Ocean or Linode or Backblaze. Like, they use some convention where they use sub domains as your bucket. So with that kind of stuff, kinda follows that same convention, but here, it's really more PAC. So Okay. So we can wrap Actually Sorry. I need to go. Before you before you run that, I would add a dash v at the beginning. Yeah. Just you can get a little more information. Just it's just for robust logging. So let's see what's going on. Yeah. So runs pretty fast, but, yeah, basically, it pulls a snapshot down

47:17 and puts it in, like, a temp location next to the file where it's gonna go to. It'll download the wall files in parallel and then apply them in order as they come through. And then finally, once everything's applied and the database is in the state it's supposed to be in, then is when it finally renames it to the final destination so you don't get it's kinda like an atomic restore. So you're never gonna get, like, a partially restored database and your in the location you wanna go to or the the destination path. So that's what it's doing there.

47:48 Okay. So we should now have first d b two. Yeah. Yeah. First two. Yeah. So you can see the light into that now. Should be all there. And there it is. Yeah. Cherry Road. There you go. Software just shouldn't be this easy to use. I honestly, like, a big like, I really try to focus on just narrowing the the feature set to just, like, the simplest. Like, if there's anything moderately complex, I'd I'd just wait to try to implement that until I can find a simple way to do it. But I don't want people to have to read

48:09 Litestream's Focus on Simplicity

48:24 their, like, pages and pages of docs and, like, figure out weird concepts or debug things. It's mainly because I don't wanna debug issues. But, really, it's just, you know essentially, it's just a snapshot and there's a series of wallpages that it's saving. So right. If you think about it logically like that, it's a lot easier to think about. Awesome. No. I think that's that's pretty sweet. Like, hopefully, you know, the peoples that are watching, you can see just how easy it is to to connect Litestream to SQLite. The backup process, because less than a few minutes, the restore

48:46 Summary of Benefits and Production Use Cases

48:59 process was even faster. And then I guess at that point, you know, as long as you've got an s three compatible storage that, you know, you trust, like s three, GCS, something like that. Got pretty sweet SQLite production resilient database setup. What more can people ask for? Exactly. That's what I'm always saying. Alright. Yeah. No. I think it's a I think it's a honestly, a great way, you know, people can I think I think SQLite is really a great database? Like, I think people should definitely consider production use cases. And I think the way that I think

49:31 about it personally isn't, like, should I use SQLite, but more like, should I use I kinda default with SQLite as kind of the the database I wanna use. It's just a lot easier to develop with and test with. And then I kind of if there's a reason to use something different like Postgres, then I'll I'll use that, but I kinda default with SQLite personally. Yeah. I I definitely something that's a lot more appealing to me these days. You know, I'm quite deep into the cloud native Kubernetes ecosystem now and storage is still the the biggest pain point there because

50:00 Summary

50:05 you're having to either learn new database technologies that are built to be replicated by default or, you know, you're trying to shoehorn this kind of behavior and or really get that tight feedback loop with backup and restore and minimize that, you know, you talked about it, the checkpoint snapshot in time, but minimize that window of where really bad things can happen. And SQLite's a really nice way where I can just have a fail on a disk, use Litestream as a sidecar container, have it replicate and backing up to somewhere that I trust and I don't have

50:13 Future Possibilities (Read Replicas, Edge Databases, WASM)

50:35 to manage. I'm happy to pay whoever sends it as per meg for that or gig. Yeah. It's just really And then you talked about future stuff and you're like, oh, well, you should be able to have live read only replicas of this data as well. So not only if I got my application run-in with SQLite, but I've got scalable copy of it to a certain degree with the read replicas. Yeah. Yeah. And also on the, you know, the replication side. Oh, sorry. On the replication side, it's like it's not even just necessarily have a ton of rights, but I don't

51:07 if you ever look at, like, JAMstack where people will generate out all their pages instead of content and put them on CDNs. So you get super low latency at the edge, but it really works well as, an edge database if you wanna a server, you know, The US, but you also have another another replica in Europe and Asia, wherever you want, where those you know, those people are getting those super low latency queries as well. I just have an idea, and you're gonna hate it or maybe love it. I'm not sure now. But can Litestream compel to WASM?

51:38 I haven't tried it yet, but maybe. I'm not sure what the SQL support is and go for WASM, but we brought the shot probably. Might be able swap out the SQL lite or implementation for, SQL JS or something. Yeah. I wonder if you could compare this to WASM, have a web interface for modifying static data that actually replicates that in the browser to an s s three compatible store. I think that would be really nice. Why? Well, why not? Yeah. Exactly. Okay. We got a couple more comments from Rio there. So that was exactly my use case I was

52:11 Community & Getting Help (Slack, GitHub, Office Hours)

52:15 thinking of. Hopefully, you mean cluster API. If so, that's really cool. I'll share links to the videos where I actually got that working. And another one about the local hosting. Currently startup local host does work on Linux, at least, they say on Fedora. I could add an entries to host file for no reason then for years. So I'm I'm gonna be playing with that later. Right. And is there anything that we didn't cover from a documentation that you wanna run through before we we we wrap up today? No. I don't think so. The only thing

52:41 Conclusion and Wrap Up

52:50 I would say is, yeah, we have a a pretty active Slack community. If everybody has questions and needs help getting up and running, drop on there, and we're happy to help. Obviously, you know, GitHub if you wanna post issues. And then I do office hours every Friday too. So if anybody ever wants to chat live, I'm happy to chat. So just hit me up. You can find the link on the website as well. Alright. And I'm ben b johnson on Twitter, so you can usually find me there too. Yes. There is the Slack link here. Get involved

53:19 with this big button here too. Yeah. Go do that. I have the office hours. I know. I took out the the yeah. Anyway, well, just hit me up. I'll we got office hours. We'll join the Slack, and Ben will share the office hours link with you as well. I'm sure. There you go. There you go. Definitely. Your Twitter handle is on the screen. Definitely fun to follow you and see all the weird, silly things you're doing with database. I used to say silly. It's now it's borderline genius, think, to a certain degree. Like so Thank you. It's a fine line, I think.

53:48 Alright. Well, thank you very much for working on Litestream. I'm already seeing so many different applications for cluster APIs. The the one I'm really focused on right now, but I maybe I'll get some time to stream some Wasm stuff with it and see if we can do something stupid there too. But thank you so much for for joining me today, getting us through this and sharing your experience. Any last words? Yeah. Thanks for having me on. No. Thanks for having me on. It's been great. I'm glad everything that seemed to just work, so that's awesome.

54:14 And, yeah, happy to chat. Thanks. Awesome. Well, have a great day. I'll speak to you again soon, and thanks for watching everyone. Cool. Bye. Have a good one.