0:00 Introductions

0:00 Welcome to the show, Frank. Please take a moment to tell us a little bit more about you and what you've been up to lately. Cool. Thanks thanks, David, for having me on, first of all. Well, for everybody out there, I'm Frank. I'm a currently a staff engineer at Personio, which is a big HR company based out of Germany. I joined Personio about a year ago. Before that, I did a couple years in in fintech, which gave me a lot of appreciation for the sort of more consistent side of things. Like, losing a transaction is not something that fintech

0:06 Frank's Background and Experience

0:40 really really accepts. So especially in sort of larger distributed systems with many systems integrating often with events and messages, that was a really interesting experience. Before that, I did a lot of work for Circle, the airports here nearby. I did a fully event driven real time dashboard application for their internal operation with highly sensitive data. That was in TypeScript. And for the rest, I've done a lot of stuff in in PHP. My current day job is more with Kotlin. So for me, like, languages or whatever. And so I've been doing this for a while now. Think, like, I'm close

1:31 to fifteen years of of doing this stuff. So yeah. And I'm looking forward to doing it at least fifteen years more. Yeah. Awesome. Alright. Thank you for sharing. So, yeah, I I kinda you know, I've followed you for a while with Tati before in the past, and I knew you from the PHP ecosystem, but I did see that you were doing Java and Kotlin and TypeScript as well. Like, it's I think it's always interesting that the more we do this, the more the language becomes less important. Right? And it's more about all the things you'd learn

1:45 The Knowledge of Others

2:02 along the way. So, hopefully, we'll we'll get into that in more detail as we go through the episodes. Yeah. I, I totally agree. Right? So you see, all of things sort of lumping together and doesn't at the end, it doesn't really matter what tool we use anymore. But it also means that you're gonna encounter the same problems in just in in different flavors. Yeah. Yeah. Or rewrite the same library 14 times on other languages that you're gonna work with. Wow. Like, I have written or rewritten EventSauce, which is an open source library that I maintain.

2:32 Rewriting the EventSauce Library

2:39 I've rewritten that in a couple of languages now, which is always an interesting experience because while the majority of the things that you do sort of maintain or they are the same, but some things you really adjust towards what's kind of language mechanics you have. And that's always an interesting routes to explore when whenever I'm doing that in a new language. Yeah. I mean, I I I don't have the first question. I'll pan out, but I'm gonna skip it right now. It's go down this little path where we're talking because one of the things I found most important

3:02 Learning from Language Idiomatics

3:12 in my career, and I think you just kind of alluded to it there as well, is that when I wrote c, I learned how to write c code. When I did PHP, learned how to write PHP code. When I did Alexa, I learned how to write Alexa code and so forth and so forth. Right? Go and Rust, like I love experimenting with languages. But what I love more is learning from the idiomatics of each language. Like, concurrency and Go is very different from what I've done in PHP. Concurrency and Alexa is very different to Go

3:37 as well because it has this actor based process model. And, like, it's just through the combination of all of these different idiosyncrasies, I don't even know if that's the right word, of each of these languages and runtimes. But you build up a different picture of what software should look like in your head regardless of the language. Like, I don't think I would be the developer I am today if I hadn't worked in all of those languages and learned all of the different things they do different in their ecosystem. Does that make sense? Absolutely. Like, one of the first, conference talks

4:03 Applying Concepts Across Languages ("The Knowledge of Others" Talk)

4:07 that I ever did was at Lericon, I think, in, like, when was this? 02/2012 or something like that? And it was a talk called the knowledge of others. And it was basically a talk about looking at various ways of solving a problem. And during the talk, I sort of looked at different types of problems and then looked at what languages are most likely to solve that's that's that problem. So for example, like in in JavaScript, you have a lot of sort of in process events driven stuff. So I looked at like, hey, what's what does an event

4:45 emitter dispatcher look like in in JavaScript and how can we sort of get the same kind kind of capabilities in, for example, a a language like PHP and also what's missing or what's not fitting very well and how can you overcome that? Like, I think that's a very interesting way of looking at it. But you you look at these languages and how they frame it and the idiomatic solutions always provide a certain type of framing. Right? So for for Go, like you mentioned Go, I think the the model of concurrency there is is very interesting to to to look

5:22 at and it's very like or the way that I did it at least was very channel based. Right? This is what's go, the programming language where it really promotes and they've got a very strong sense of how you should route write code like that. And using stuff like that in like, trying to replicate that in in different languages is sort of an interesting exercise. It doesn't always play out, but you can definitely take some learnings from one idiomatic approach and apply it in a different different space. If anything, it's a lot of fun. Yes. I will 100% agree with that. And I

5:57 have taken a note about the knowledge of others LaraCon talk. So I'll make sure there's a link to that in the show notes so that I can find that online for people to to check it. I was like, I don't wanna watch it myself. So Well, I I don't think it's probably not my my best, talk ever. Like, I think it was one of the first talks that I ever did in public, and I actually don't remember a lot of it because, like, being on stage, especially that point was, like, such a sort of high intensity, high stress

6:24 moments that, you know, my brain blocked to mature. So mind you, the quality might be not what you expect it to be. Awesome. Alright. Let's go back to the planned first question then. So, obviously, we've discussed event sourcing. Before, you've already mentioned event source, your library, you've now written it in multiple languages, and it's a great way for people to get started with event sourcing. But what I want to understand is how did you even end up in this event driven architecture ecosystem? How what led you from writing, you know, your your PSP and just writing

6:35 A Need for Event-Driven Architectures

7:00 web apps or whatever it was you were doing back in the day to go, crap. Like, I want everything to be event driven. What was the path like for you there? Well, so, I was very fortunate enough to work on a couple of projects where this was just needed. Like, the only way to do things was an event driven way. The first project where it was really relevant was the website for Schiphol. So that was still in sort of the website y area, but it it got a little bit more complex with some some logic there. But

7:07 First Event-Driven Experience (Consuming Events)

7:33 effectively, the way that we had to ingest flight information needed to be as close to real time as possible. Right? So as soon as possible, view on what the planes were doing. As a customer, you wanted to know like, hey, I'm going on a KLM flight and I'm going to London. What's the status of the of the flight? I want to, in my web app, follow that flights, keep keep track of it. And whenever something new is up, I would want to see that. Now the only real way to do that for the scale that we were at, like, doing

8:08 that for all of the passengers that go through simple, you can't do that on a polling basis and then figure out the delta between all of the flights that are there everywhere. Right? So one, it's it's it's not efficient enough, but that's also not quick enough. Right? You want it as soon as it's as it happens. So we built a a whole event based integration there and that was really on the consuming side of an event stream. So I I would say that's a relatively comfortable way of starting out in the events driven space is to be a consumer of events.

8:42 Right? So you can sort of skip the whole how, when, and in what form should I be producing this and just see it as a stream of information that you can respond to, that you can create projections from and all that. So that was my sort of initial introduction into the event driven space, which was soon followed by going into the event sourced space. For this same airport, we created a way to book parking spots for Schiphol which for an airport is a huge revenue stream. And so it's being a huge revenue stream. You also want to get a high level

9:07 Transition to Event Sourcing (Producing Events)

9:20 of insight into what is going on and why things are going the way they are. So for example, if you're booking parking spots, you get multiple types of parking spots made available for you which are at different price ranges. So you also want to know as a vendor of these parking spots, like what does the positioning of those parking spots do for the general choice of the parking spots afterwards. Right? So you can do this with just an API call and record the response, then track where you positioned it, and then do a separate call afterwards to say, hey, which

9:58 one was picked? But it's very indirect. And a more direct way is to just record that all of as events and then make sure that you use those events to gain those insights along the way. There I was very fortunate to work with Marine Heisenfeldt who was doing event sourcing, like, I think maybe even before I started programming. So I had a very, like a veteran of event sourcing as a mentor and as somebody who guided that entire project. So I was very much in a following capacity when I was executing that and Marine was really leading that effort

10:34 there. Nice. Well, yeah. I I I got a couple of questions. Right? So you can imagine event driven and then event sourcing. And, you know, to the people that are listening, I'm gonna ask you very the same question twice, but for each of those. And you can just say yes or no. You can give more information if you want. Right? But would you say that most applications could benefit from adopting some event driven aspects? Yes. Alright. What about event sourcing? Is this what the question I'm really getting at here is what is the decision? What what thought

10:45 When to Event Source

11:07 process do people need to make before they say, I'm gonna bring event sourcing into my application? Can they do it for all applications or is it a very specific requirement need? How how do you make that decision? Yeah. For me, it needs to be the core core business that you're that you're modeling. And there needs to be a like, I like it when there is a temporal effect to it. So multiple things happening after each other. There's a sequence to it. And what happens is sometimes as interesting as the state that you end up with. So if you compare

11:17 Event Sourcing vs. CRUD (Domain Fit)

11:43 it to CRUD applications, right, CRUD applications, you have a state, something happens, you overwrite something, you have a new state, you overwrite the past effectively. This also means that you lose that state. Right? So you forget about the past, but you also forget the delta. With event sourcing, you store everything that happens as a general principle as events and then you just append those events. So this also means that you retain all of the information over time, but you also retain all of the delta. In fact, the thing that you retain is the delta. So you're always able to get from where

12:21 you were to where you are now. And there are a couple of benefits to that, which some problems are more prone to benefits from than others. So for example, if you have changing policies and those policies need to be applied even though something was vetted through the process before and was deemed okay. Now a new policy is added, things no longer valid. How do you deal with that? Right? Having that stream of events and not just the outcome of something is valid, yes or no, right, is useful in in such a case. If you're doing something that's purely crud, right,

12:57 I need to get an inventory or I need to have what is it? A user management, all of those things probably not useful because you're you're only interested in really the current state of things. Like how how things got there, you don't really care about it. Permission management, right, you don't care about how the permissions got to the state where they were, but you care about what your current permissions are and how to apply them. So not all problem areas really benefit from. You got to understand your domain to the point where you understand there's value and those missing pieces of information

13:33 that we purge with a standard current based operating system. I kind of understand that. Right? Yeah. But then by saying that we wouldn't adopt event sourcing for a current application, I think what the subtext is saying there is that there's a cost of adopting event sourcing within your organization. Yeah. And so it's it's not always just four lanes of code and I've got an event sourced application anymore. So I'll come at this from two different angles then. It's like, what what is the cost? Why is it challenging for people to bring event sourcing into an application to their team, to

13:37 The Cost of Adopting Event Sourcing

14:00 their organization? And part two of that as well, what is it our future where event sourcing becomes more of a commodity where it's easier to bring into a project without that cost? Yeah. So there's definitely a cost to it. I would argue that the cost is mostly misinterpreted or misassigned. The biggest cost is probably that on people. So if you're looking at experience for crowd based application development, it's there in abundance. If you're looking at the experience in building event driven architectures, there are still plenty, but it's a lot fewer than, like, finding people who can build good crop

14:12 Understanding the Real Cost (People & Infrastructure)

14:42 applications Where people usually attribute the cost of of event driven architectures is saying like, hey, now I need a queue. Now I need all of these background processes. Now I need all this orchestration. And I would just say like that's that's not inherent to event driven architecture. So you can do event sourcing, which is a very event centric approach of modeling domains. You can do that in a single threaded language like PHP without a queue. Right? You can do in process consumers where the events that you record are passed in the same web request to the consumers

15:27 that project their state, send their emails, send send out API calls to other kinds of services. That's all possible. Is it ideal? No. But it is a good starting point. Right? If you have a tiny application that needs to do need to do a couple of things, reliability is not yet a big concern, then you can really reduce the cost by turning down all of these levers. Right? You don't need fully false tolerance, everything at the start of a project. Right? So yeah. Okay. So let's pivot the question again one more degree. You've got this team of engineers.

16:03 Defaulting to Event Sourcing? (Experienced Teams)

16:06 You're on it. Right? You're you're kicking ass. You built an event driven system before. You've got event source and 2,000 languages. Anybody can build event source application nice and easy. Would you personally, just you, if I came across and said, I have this really simple CRUD based application or microservice that we need as part of a larger application. At that point, is it easier for you to do it with event sourcing, or is there still a cost for you to do it? Like, I'm wondering if, like, if I have the experience, if we have a team

16:33 of experienced people, like, would you default to event sourcing or would you go just credit just credit? No. So defaulting to any type of technology for me always sounds like a very dogmatic thing to do. So you need to balance it with, like, the benefits of implementing some something and the cost of it. And so especially when you have a large team around you, if you don't have any event driven experience within a a company, then you're saying like, hey, we're gonna change this core component and we're gonna switch it from a more crowd based to

16:40 Balancing Costs, Benefits & Organizational Learning

17:09 a events driven approach. I that has a serious expense to it. So it it's more that the expenses that you're gonna the organization will need to learn about event driven architectures and that means that they're gonna fail at at it for some time. And so you need to find a a mode within the company that they're able to fail. And sometimes the expense is is high, but even even then it can be okay enough to to to do it. So for example, in in some systems that I worked on, all of the adjacent business processes were taking direct database access,

17:49 to the columns that they needed, to get their information from. Well, this also meant that it was very difficult to change that system. So by introducing events, we allowed the second secondary business processes to decouple from those internals, giving the internals the freedom to change. Right? So because this was a very necessary change for the system, the rewards were higher than the cost. Right? And so in those cases, you can do that. But you need to have seen sort of the other side in order to make that assessment. And that's sometimes difficult. Right? If you

18:23 have decision makers within the same company who haven't yet seen that that benefits, then it's hard to sell them based on the on the claims and the merits of those those solutions. Yeah. I love that you mentioned the the failure aspect there. I think what a lot of people often forget is that people with experience or people that are good at rating Kotlin that doing Kubernetes about a microservices, doing event sourcing. They weren't magically good at it because they've done a lot. They're they're good at it because they've done it wrong more times than anyone else. Right? And it's

18:56 yeah. Teams have to build up that competency. And sadly, fastest way to do it is to get a lot of shit wrong. So Yeah. Yeah. Sort of, like, eat dirt, get up and continue. Alright. Look. We've chatted for, like, twenty minutes now, and we haven't really tackled for anyone who's new to event source and CQRS and event driven architectures, I kind of overview. Now we have covered a little bit of it through your conversation. You've talked about squashing or losing history and maintaining history and stuff like that. But maybe we could give people a quick five minute overview on

19:15 What is Event Sourcing

19:31 EDA event sourcing and CQRS and how those work together. Yeah. Alright. I I will start with with event sourcing. So event sourcing is really a way to model problem domains. We're using events as the the primitive to record what's happened with the store states. In event sourcing specifically, you also say that the states that you use in order to base new decisions upon are in essence sourced by the events that you record. And this usually means that you reconstitute your domain model by reapplying all of the events of the past, putting you in a condition

19:37 Event Sourcing Explained

20:13 to make a decision that will then again result into a newly recorded event. This is different from traditional entity based and state based modeling where you overwrite the past. A reason why this is was introduced is effectively for not losing the history that you have recorded along the way. It's very inspired by accounting. If you have a bank account somewhere and you deducted too much money somewhere, right, you don't get that row adjusted. You get a counter transaction that says like, hey, I'm gonna file a correction now. So for example, I overbooked some money, removed

21:05 120, but I meant to deduct 100. Now I'm gonna sort of reimburse you 20. We all know how this happened and how we got to the end state. We know how the correction happens. Well, for some business processes, this is really important to to have us as context. Now that sort of event sourcing in its in its essence. So sometimes it helps to relate back to other primitives to sort of see how it really works. I usually say like, hey, an aggregate which is what's or an aggregate root which is what you refer to in event sourcing terms

21:46 as as the entity is sort of the you have a stream of events and you sort of map reduce that those events into a state. You use this state in order to provide the context for your next decision and then afterwards are more events and that's what you get out. The benefit of this is that it becomes really deterministic. If you have state based application, applications, you need to set the state and whether that state is something that you would end up in or not. Right? That's that's out of the question. And then you need to

22:18 make a decision and then new state is what you actually assert on. In events driven systems, especially event sourced ones, you can have a really nice given when then structure which says like, given these events, when I do this action, now I ex expect these events to be recorded. And that's very deterministic. It's almost like mathematical in that sense. You can sort of see that as a proof of the system working without having to manipulate any inner parts of the system. So that's that's very nice. So I think does that sort of clarify the whole event sourcing

22:55 part? Yeah. I think that was spot on. I'm I'm gonna add one example that I found really useful. I can't remember who it was. It was either you, Tomi, and the pub, or maybe it was Greg going on one of his talks. But it was the ecommerce example where if you think about building an Amazon or someone adds something to a car and removes it from the car and they do that 5,000 times and then they buy one item, there's 5,000 items there that you can upsell them later if you track that they added it and removed it from the car. And

22:58 Example: Value of Event History (E-commerce Cart)

23:22 that that was the one and I still remember that example clear as data today. It's the one when I went, oh, there is a lot of value in some of that information that we've always parched in the past. Yeah. I think one of the situations that I think is often underutilized even in event sourcing is recording of failure. So if you if you, for example, want to pay for something but you have insufficient budget, right? In a entity based system or a state based system, right, you say, I'm I'm entering the methods. Do I have enough

23:34 Value of Recording Failure Events

23:55 balance? If no, throw exception. If yes, mutate the states. Only then it has effect. Within the event source system, you might say, into the to the methods, see if there is enough balance. If not, record an event and throw an exception and you always store any type of events that you record. So usually you put that in a try and then finally block where even if there is an exception, you store all of the recorded events. And all of the sudden, have insights into all of your unhappy parts. Right? So you can capture the context of

24:34 the failure and that's really like where you see the users have friction with your product or with your business process or anything in relation to that. Nice. Great example. Alright. Let's tackle the the next one then. So CQRS. Yeah. So CQRS like I'm I'm bigger on event sourcing than I am on CQRS. So CQRS I often see or when it's talked about in the community, it's talked about applying it in a very dogmatic way. And so you're you have to separate everything that's an instruction to the system so that that tells the system to do something.

24:50 What is CQRS

25:14 And the other part is really how to get information out. The reason they they want that is to allow for different models on how you query that data which are optimized for querying and the decision model is not necessarily the best model to also query it. So for example, if you have an order based system and you want to aggregate how many orders there were today and what their cumulative amount was, right, you can do a sort of SQL aggregation over that. But it might not be very efficient if you have a large amount of customers. Right? So using events

25:54 to then say like, hey, I'm gonna record all of the events of the purchases and then I've got a secondary model, which is the accumulation of all of the amounts that were stored for a day. You have a a sort of a live view of what that means across the day and querying it is just basically querying a column within a database. Right? Very efficient. And so there's definitely value in separating out these models, but always doing that. Right? I think lacks a bit of nuance in that sense. Right? For many of the the problems that you

26:30 have, the original decision model has enough data points that you can use for reads. And also always doing it, how do you say this? A lot of people applying CQRS also say like, hey, then you put it on a message bus. And so that means that the state that you want to query maybe in the same request is already eventual consistent. Right? So then you need to deal with that fact again. Well, people create all kinds of ways to to work around that problem. So one is like your aggregate has versioning and now your projections have versions as well.

27:07 You're just gonna pull the the read model until you get the right version if you want to get the response for that that version. Right? That's a possibility, but it's also very cumbersome. Right? So you get some dogmatic things like command buses shouldn't return any, values and all that shenanigan. Like those shenanigans, I think it's too dogmatic. So I think as a pattern, like being able to separate those things, fine. Always doing that, I don't really see the value in that. Okay. Let's touch on one more thing and then we'll move on with the questions.

27:40 Crossover of Microservices and Event-Driven

27:42 But you've covered event sourcing. Great. CQRS. Perfect. So if we have all of these events, we have to build projections. We have message buses. A lot of what we're seeing here, right, kinda maps to what I'm seeing in a different ecosystem, which is cloud native Kubernetes. Right? We're seeing micro service adoption, which is being built on event architectures where people are preferring to do publish events to Nats or Kafka, RabbitMQ, etcetera. And all of these service are independent. Right? The goal is to be loosely coupled. And it feels like there's actually a lot of crossover

28:18 and benefits to both of these architectures maybe working together. But I'm curious if this is something that you've seen a lot of from the event sourcing ecosystem. Are people doing microservices? Are projections of microservice? Like, how do these two different tangential things work together or correlate? Yeah. And so I think they're just the same thing at the different scale. Right? So if you've got a monolithic application that is recording messages that the that it's consuming itself in order to record state as a projection that it's surfing on a different rest end point. That's for me the same as two

28:34 ES & Microservices: Coupling, Volatility & Versioning

28:53 services which are which there is a dependency between which where you have a source system for information and a dependent system and it's using messages to be kept up to date. Right? So in essence, it's the same thing but then scaled up and maybe with a higher infrastructural cost associated with it. But in principle, it's it's just the same. The thing where where deviates is in the coupling and knowledge area. So if you have two different services, you know that the services have a different level of detail regarding a particular subject. So if you're in a monolithic situation

29:32 where you're the consumer of events, you tend to have a a high level of detail and a high level of volatility of the information that you're consuming. So if you're modeling your business processes with events, then innovating on that business model means changing events. If you're doing that for yourself, you're exposed to your own volatility. Right? So that means that you need to if there is a new version of something, you need to account for the backwards compatibility at least for the periods where you're exposed to those changing events. Now if you amplify this to the microservice

30:07 world, this volatility means usually that you're gonna be disrupting other services unless they accommodate for this change as well. And especially in larger organizations where those services are not maintained by the same people, this is where you get up into Shit Creek area. Right? So if you're changing the event stream that somebody else is consuming and you're not informing them and allowing them to accommodate this in in some way, shape, or form, you're gonna cause a disruption in production. And so that's more difficult. Now luckily, there are patterns to deal with it. But this usually like you asked before like,

30:47 hey, what's the cost of introducing event driven stuff into into projects? It's learning this and learning the ways to deal with it. So in event driven architectures, usually say like, hey, just expose everything. Right? So as a first iteration, I've got all my internal domain events. Well, now they're all external domain events and everybody can listen to everything. Now I want to change something. I've done the whole integration part before. Now I want to change anything. Now I need to change all of my integrations. Right? And so while event driven architecture is usually attributed that is great for decoupling,

31:28 unless you do that on the information level, you're still just very highly coupled from business process to business process and you still need to do sort of diligent information disclosure and deliberate information disclosure in order to prevent that. Right? So within object oriented stuff, we say, well, we've got an interface and then you can touch these public things, but you cannot touch these private things. Right? Within events or different architectures, especially if your implementations mature, you see the same effects. Right? You've got internal information streams, you've got external information streams, and you you act differently

32:09 With object oriented, you say like, hey, I'm maintaining sort of a backwards compatibility for a large degree and sometimes I'm creating APIs with a new version. Right? That same applies to event driven architectures. But there then you're accommodating that by versioning the payloads that you send and then putting anti corruption layers in place, which are either on the producing end or on the consuming end that smooth over the differences between these versions. So that means, so for example, if you are producing a new version of your event which has more fields or fields that are just formatted

32:49 a little bit different, something that's really backwards incompatible, then what you can do is have sort of a mechanism on the producing end that says like, hey, I'm gonna consume this stream and I'm gonna convert it to the older version and publish that as well. And so a mechanism like that gives other parties a lenience period where they can still rely on the old information stream, but then also have at their disposal an opt event stream, which they can consume in parallel and then try it out for a bit and then convert all of their integrations

33:31 without needing to do it as as a hard cut. Can I just confirm something there? Because that was really really interesting to me personally. Right? Because versioning is hard. And when we talk about challenges of this, I'm sure versioning is gonna be one of the biggest ones. So maybe we'll come back to it. But I never really thought of it publishing or having a producer who already has all the knowledge of the versions. Right? Because it's their domain. Do you publish them as separate events with the different versions or do you, like, wrap them in an envelope and publish them as

33:35 Event Versioning Strategies

33:58 one event with multiple versions as a payload? Like, how how does that work in practice? In practice, it's so if you need to couple them in the same message, they needed to be produced atomically. Right? So thinking of this a little bit longer, I think you you can do it like that even if the producer doesn't produce them both versions. But what would the benefit be? Well, I was thinking if the producer emits version three of an event Yeah. Has a function to downcast it to v one and v two, wraps them in an envelope and publishes it,

34:32 then the consumers can kinda cherry pick out the version that they understand the most or the latest version that they Yeah. React to and and handle it. But then, I guess, you you she also went downstream to be working through the latest version too. Yeah. Yeah. Tough problem. So that is possible. You would be publishing multiple versions of the same event inside of the same event stream. Right? So what you sometimes have is that upgrading event versions or entire streams, they're used as an opportunity also to reprovision all of the infrastructure that's behind it. So if you need more

35:08 topics on your Kafka stream, right, having a new stream is sort of an opportunity to say like, hey, I'm gonna over provision a little bit again. So I've got some breathing room for my concurrency, for my consumers. So usually you want to have that as a possibility. And now a consumer will rarely in one business process consume multiple versions of the same events. Right? So usually what you do as a consumer is you're gonna put side by side multiple consumers that do version a and version b side by side and then you can sort of compute. Like

35:42 if you want to do a parallel run and you can check for for differences afterwards. So they just have one thing. So you you can you can certainly do it. I've never seen it being done like that. So usually what you what you see and what I think is idiomatic in that space is that you have either on the same stream, you have that payloads of multiple versions. So you have a type and a version of the type and it's published over the same stream. Or if the entire stream is being upgraded from one version to another, which is usually

36:18 the case if the structure of the data or the payloads are fundamentally changing. So either there's something changing in the envelope, that's usually a case where everything got bumps up one version. Then you see more often that an entire new stream of events is being introduced for those games. Nice. Well, it feels only fitting that we're talking about some of the hard stuff then with event sourcing that we maybe see. Are there any other challenges that you've faced adopting event sourcing as capital and other places that you wanna share with people like, you know, assuming

36:45 Challenges of Event Sourcing

36:52 some masochist listening to this podcast is when, you know what? This sounds really cool. Hard, but really, really cool. Like, what else should they be looking at for? What else have you worked through? Yeah. So one of the areas where it's both a virtue and a and a curse to have event driven architectures in place is disaster recovery. So as soon as sort of metaphorical ship hits the fan and maybe you have catastrophic infrastructure failure and the consuming system loses all of their state, they need to rebuild that state from somewhere. So in more state based systems,

37:28 you either work off of maybe a database replica or something else. You sort of like ingest or restart that whole replication process with database native technologies. You're in in place relatively relatively quick. For event driven architectures, it can be more cumbersome and you need to design for it. So for example, if you need to if you've got a projection and you need to recreate all of your current state based on everything that's been recorded before, the question is always like, how far back do you need to go in order to figure out what to reconstitute from?

38:04 Are you gonna do that from the epoch of your business, right, from day one of your business? Or are you gonna have some intermittent states where you're gonna say, like, alright, I've got a snapshot of the of the state that it was at, and now I'm gonna reconstitute from from that point and out and reconsume. Now that is not something that is very trivial to design for. And you need to apply a couple of sort of supporting practices which help you, to do that. I think Kafka as a technology makes it easier to do some of those

38:38 things mostly because they have a incremental offset based storage strategy where you know where you left off and if you have a snapshot that's at a particular Kafka partition combination, you can take that state and know, alright, I'm I was exactly there and I can go on and reconsume the rest. If you don't have such a system and you have got your own sort of like delivery mechanism, if you're going through RabbitMQ or any type of other like sort of transient message queue implementation, Right? You need a way to inform the producer of those events to re flush whatever

39:17 they had. So reconstitution of these read models becomes a lot more involved. So it's it's not all cherries and rainbows in that sense. One of the things that you can actually do is use your partitioning strategy and your log compaction strategy on Kafka to always ensure that all of the leave notes, retain the latest bit of information. And if that fits your domain model, then you can use that as a way to quickly get to the latest state of everything. Because effectively on all of those partitions, you guarantee with like every key, the last message of that key contains

40:02 the last state that I'm interested in in order to get to the to the current state. But that's very much like in the details of of your implementation already. So there are opportunities there, but you really need to understand whether those solutions really match the problem space where you're applying it. And that's not always the case. Awesome. Alright. I'm not sure how many people still wanna build events or systems after hearing that, but it definitely is. It's fun. At least you're Well, so sometimes you don't have a choice. Right? So for example, if you have a

40:30 Trading Problems & Benefits (It Depends)

40:34 source system which is unreliable and you have a high value system that's dependent on it, right, are you gonna accept the downtime of this system as a limiter for your value creation stream? Like, no. Right? So you want to be kept up to date with the with the latest information. So, yeah, you're gonna swallow some of those costs and, and difficulties that are associated with event driven architectures. Usually, you're just trading one bag of problems for another where you hope that the the downsides of those problems will affect you less and that the upsides of having done that will reap you more

41:11 benefits. Exactly. The the classic answer, it depends. Right? Like, you got you got to pick your trade offs. You got to pick which level of complexity you need for your application. And event sourcing has a very interesting and applicable and useful and value add and some domains that you really need to go down the rabbit hole. But definitely not all. Yeah. Alright. Well, I'm curious then. What are you excited about these days? What new shiny things are you playing with, experiment, whatever, even just interested in you've not attempted to play with yet? Like, what what's in

41:40 Exciting Futures

41:48 the future for Frank? So I'm I'm curious to see if we like, on the sort of real time aspect of ingesting data, there is a lot of, like, experience that I have on implementing that. The one area where I have lots less experience is on the sort of on the controlling side of it. Right? So real time interactions with with system, systems which can also be message based from so imagine you have a react UI and that's a that's not a blog or another type of sort of simple information system, but more like a system that you really interact with hands

42:28 on to see like, hey, how can you interact with that system in real time and then get real time responses out. So I know that PHP is not really a language where that's super easy to do In the sort of TypeScript node worlds, there's definitely some precedence for that. And in the Java, like JVM ecosystem, there's also a bit of that. But I also want to learn more about Rust, so maybe I will just use Rust as my next language to experiment with and implement that. So, yeah, that's a general area, but it's still really

43:04 fuzzy for me in terms of like, hey, what does that really look like and what type of systems can I study in order to learn more about it? Sweet. Alright. Well, I'm looking forward to seeing Event Source and Rust then. I think that's gonna be a lot of fun for you to watch. I'm I'm I'm guessing it should probably like if I'm looking at Rust, it probably involves some kind of macros. So I mean, you could write some really sweet procedural macros in Rust to handle the replay logic within structs. I think that would be super cool,

43:39 and everything would work within an implementation block. I think the developer experience and the API you could provide for event sourcing in Rust would far surpass any other language. Alright. Because as really great macro support. Metaprogramming in Rust is good. Metaprogramming in Go Not so good. So much. No. No. Like, in in Go, there are some awkward bits if you need to do event sourcing there because, like, it's very difficult to do anything very dynamic and event stuff is usually usually benefits from some dynamic forms of programming. We'll we'll go to that generic support in one

44:15 eighteen or one nineteen. So I think that landscape is now wide open for some innovation, but certainly prior to that, it it's been tedious at best. So this has been awesome. Thank you so much for spending the last hour of your life sharing your event sourcing journey with me and lots of interesting information for people listening at home. And then I'll give you just a minute and say you wanna share your Twitter handle, plug your website, your company, open source projects, now would be the time to to plug away. Cool. So if you have any file system needs

44:25 Plugs & Conclusion

44:46 within the PHP ecosystem, check out Flysystem, which is the main PHP file system abstraction library. If you need to build anything event sourced also in PHP, check out EventSauce or check out EventSauce.io. Check out my employer. They're called Personio. If you're a company that needs to work with HR processes or any other type of HR tooling, that's what they provide. So do check them out. And when I have the time, I blog on my personal website, which is blog @frankdejonga.nl. Alright. Thank you very much.