0:59 Introduction and Guests

0:59 Hello and welcome to today's episode of Rawkode live. Before we get started, I wanna thank my employer Equinix Medal. They provide the time for me to invest and to produce in this material so that we can all learn together. If you want to try out the platform, please use the code Rawkode Live. This will give you $50 of credits. You can splurge that all in a few hours with some beefy hardware with 400 terabytes of RAM, 400 gigabytes of RAM, would be a clever trick and loads and loads of CPUs or you could use it more wisely. I prefer the

1:00 Introductions

1:27 former. We're gonna take a look at OpenTelemetry today. So I'm going to invite my wonderful guests now. Hello Amy Toby and Liz Fong Jones. How are you both today? Hello. Hello. I'm doing good. Good and alright. There we go. So let's start with you, Amy. I like alphabetical order lists. Can you just tell us who you are, who you work for, and then we'll move on to Les, and then we'll get started with today's episode. Alright. I'm Amy Toby. I've been in in SRE since before it was called that, about twenty years now. I work at Equinix Metal

2:02 with David, but in I am a principal SRE in our software engineering group. Okay. And Liz? Hi. I'm Liz Fong Jones. I'm a principal developer advocate at Honeycomb.io. We are a company that helps developers understand their software and how it's behaving in production, and I am a contributor to OpenTelemetry. Which is great because today, we're talking about OpenTelemetry. Could we maybe spend then the next, let's say, couple of minutes and just kinda cover what OpenTelemetry is and what problem it's trying to solve? Absolutely. So OpenTelemetry is a solution to the problem that previously there were a number of different ways that

2:30 What is OpenTelemetry?

2:42 you could use to try to get your data from your application into platforms that could help you understand what was going on. So you can think of everything from New Relic to Datadog to Splunk to kind of more upstart modern tools like Honeycomb. Right? Like, all these tools have their own kind of explorers, their own SDKs. And, therefore, there is a high barrier to kind of adding one to try it out, to switching, to experimenting. And each of us who are on the vendor side had to reimplement the same kind of SDKs to collect trace data, to kind

3:16 of to to instrument to instrument frameworks. So we all decided, basically, back in 2019 at at the end of 2018, actually, we've been on this journey two years now, to merge all of our open source efforts and to combine them together into something called OpenTelemetry. So you may have heard of OpenTracing or OpenCensus. Those are kind of the two predecessor projects to OpenTelemetry. But OpenTelemetry's goal is to kind of unify all of this that there's just one perspective on how to do it and not three. Awesome. Alright. So let's we have a couple of comments. So we

3:59 got Waleed saying hello. Hello, Waleed. Sayam, thank you for joining us again. Now the the premise of this show is generally that we try to do very little upfront. I like everything to happen in front of the viewer's eyes. And I think that's really important because, like, you know, Amy and I as principal engineers, you know, we like to demonstrate to people that, you know, hey. We as principal engineers sometimes struggle with syntax. We as principal engineers have our compilers, like, not work. Right? Like, those are things that happen to everyone. We're not, you know, perfect human beings just because

4:29 we're principal engineers. I got some inkling, so I did cheat a little bit and ran through some stuff. But I think David's starting with a a blank slate. So hope hopefully, we just eliminate the the the really tedious bits. Well, mean I think we're gonna up the ante a little bit with this episode because not only are we starting from mostly fresh slate and I'll cover them mostly in a second, but we're all working with a repository that we have no experience with essentially. And I don't think any of us are particularly familiar with the project we're gonna use. So

4:50 What was setup in advance

4:59 let's cover the context then. So this is the I opened the wrong tab. This is supposed to be tankerbell.org. Okay, the problem. Which is a provisioning platform for bare metal and Tinkerbell aims to solve the problem of I have a bare metal machine, I wanna spin it up, get an operating system on it and make it available for someone to use and provision their own software. Now there's quite a lot involved with this, there are many moving parts. So we're gonna focus on two aspects of it today hopefully, which is if I move over here

5:00 Setting the Stage: Tinkerbell Project & Setup

5:35 and pull up the read me and the tank CLI and the tank server. Now these are the smallest kind of surfaces that we can work on. The tank CLI gives us some commands that we can run that speaks to the tank server over gRPC. Hopefully all very easy for us to instrument with OpenTelemetry to a certain degree. But there's still a lot of complexity in this project as a whole. Now we got very lucky and that a colleague of Amy and I's Jason has worked on a tilt file which I'm using today. Tilt is amazing. I love it.

6:06 Yes. So this is the bit that has not merged into the main branch yet but it is available for everyone to use and the reason I did this upfront is that there's a whole bunch of dependencies that happen in the background lots of cache stuff that needs to be available in Docker images. I don't really wanna make everyone sit and watch this for twenty minutes while it's spun up. So I have done this upfront, I do apologize but we are in a position now where we can make changes to the CLI and the server components and the auto live reload. That is

6:33 all I've done and like this will be available on the main branch soon and to make sure that worked, I did what any senior developer would do and added a print to standard out called help me too. So I Hope I'm trapped in the server factory. And you can see hopefully if we've got this right, if I just remove the two then tell as amazing as it is detects that change, rebuilds image, redeploys it and we can see the help me is now available here. So that's all we have done up front. What that means is we have a

7:05 feedback loop for making changes to this, which I'm hoping is enough for you to to be able to guide us through the process of adding OpenTelemetry to this project. Does that all sound good? Great. Sounds perfect. We're all about short feedback loops here at HoneyGo. Awesome. Alright. Well, let's start then with what is the first step, you know, as someone working on a project has this great idea. I wanna add OpenTelemetry to it. What what is step one? What what do we do here? Step one would be, why don't we go through and look at a sample Go application,

7:36 Exploring OpenTelemetry Go Example

7:42 a sample Go server, that's a gRPC server, to understand how does OpenTelemetry instrumentation work. So if you go to github.com/open-telemetry/opentelemetry-go, without the dash this time. I know. We couldn't get OpenTelemetry, the GitHub name. Someone had squatted it. If you're out there and you hear that your organization owns Open dash or OpenTelemetry, one word, we want to hear from you. Okay. And now go to example. And let's go pull up the gRPC server. So it's under basic. Basic. And look at main.go. I think this is it. No. This one doesn't have the gRPC. There is oh, I know where we moved that

8:30 to. We moved that to OpenTelemetry Go contrib because it's we didn't want to make everyone importing OpenTelemetry import all of import all of gRPC. So now let's go to instrumentation gRPC. GitHub's run? Yep. Yep. Probably there. Yep. Example. Perfect. And server. Okay. So let's look at what a sample application that's a gRPC application instrumented with with OpenTelemetry looks like. So here's the standard, you know, say hello open open standard gRPC code for saying say hello. So what you'll notice is that we haven't actually changed the code very much for that. But what we have changed is that we've

9:23 changed the instantiation of the gRPC server. So if you scroll down, you should see that there is it's still all boilerplate boilerplate. Right there. That we've configured a stream interceptor that will automatically create trace spans anytime a request is received in the server. So that's what we're going to need to add to our gRPC instantiation inside of inside of Tinkerbell. Okay. So just so that I understand that correctly then is the OpenTelemetry, when we add it to our Go package as a dependency, it provides especially the gRPC stuff just provides hooks that we can just add to the server definition

10:11 that does all the instrumentation for us. Like, we don't have to manually go an instrument any of our gRPC functions or anything like that. It adds the basic skeleton. Right? You still have to add meat to the skeleton. But if you don't have a skeleton, right, like, your your body's gonna fall apart. Right? So we kind of have Halloween. I know. Day past Halloween, we can still be a little spooky. Right? Like, if you don't have a skeleton on the inside, like, everything's gonna fall apart. Right? So either you can add that skeleton by hand, or you can have

10:36 something helpful like hotel gRPC add the skeleton for you. So the skeleton here is skeleton. The skeleton here is RPC calls coming in and leaving. Right? So when you issue an RPC call, you want to have that instrumented. When you receive an RPC call, you want to have that instrumented as, hey. This is a trace scan. This is your network. What's not done automatically for you is adding data on kind of some of the attributes of what's happening. For instance, things like what is the request what server did you request? Right? What size of server did you request?

11:08 Or where is the server located? Right? Like, those are parameters that the generic framework cannot possibly know about, so you have to manually add them. Similarly, in the same way to how we kind of added the help help, right, like, into the into the console output, you can choose to add kind of logging messages or up and attach them to trace spans. Those are all things that you can do, but you have to have a trace span that's already instantiated to add those attributes or or log events to. Okay. So now we have to do this over here.

11:41 Implementing gRPC Instrumentation

11:41 So let me to see this is the server.main.go. Now this is it? Yeah. This is. If I had to guess, it's under a gRPC server. G r p ah. Okay. So we should see something that looks like. Yep. Right there. Set up gRPC. Right. GRPC new server. Right. Okay. So do we have we need to then start. I'm assuming we have to add something to our goal dot mod that that tells it to bring or just add the import to the top of the file to tell it to bring in the OpenTelemetry code. That's correct. So if you go back

12:20 over to the exam, you can see what the import is for hotel gRPC. Yep. Perfect. That'll do it. I'm gonna not save that because I'm pretty sure if I do, VSCool is gonna remove it. It's thinking about it. So helpful. So helpful. Okay. And then let's add the interceptor. Again, you can basically you can basically cut paste that. Oh, just do that. Okay. So we want to can I just copy all of that? So you'll notice that when it says gRPC new server, the interceptors are passed as parameters to the new server, whereas you're programmatically

13:07 constructing your list of server arguments in your application. So you'll copy the gRPC lines here, the stream interceptor and unary interceptor. But where you paste them is not going to be into gRPC new server because do you see, like, params..dot? So you need to add to that params list. Ah. Yep. So are we gonna call You already have a from me? I think it should coexist. I think you can have more than one interceptor. Does the order matter? So so this is gonna go, like, be in the call stack. Right? Like, when the request comes in, gRPC is

13:45 gonna go through this list and call each of these interceptors and pass it some data. Mhmm. So we probably want it after that TLS setup thing or before. Like See, it works fine with permissive. It's it'll work fine here. Right? Like, the credentials are are automatic. Right? Or sorry. The credentials are done on on init. Right? The credentials are not per request. That should be fine. Okay. Okay. So now if you save this and then comment it oh, in fact, the s code is detected and added it already. Perfect. Okay. So that's great. But now in addition, what

14:21 we need to configure is we need to configure a exporter. We need to configure an exporter so that we can see where this output is going. Versus code is is Well, Versus code has gone a little rampant. What? Okay. We're good. We're good? Yeah. I didn't touch it. Okay. So first, let's check, make sure that it compiles as is. It's currently building. Oh, we have a restarting field container. I think that was just that weird moment where we had that weird paste three times and I think we're Oh. They just back up again. Your database schema is out of date.

15:25 If that's just a a warning, we can ignore that one. I've already ignored that one. Okay. Interesting. We have a small problem. Boots and oh, team server. I mean, that's gonna show me the same that I got anyway, but I'll just double check. Make the same change locally and see what it does. Live debugging. Live debugging is excellent. Nothing ever works the first time. No. Of course not. So I just remove it. We'll make sure that it goes back to a healthy state, which ah, I have insufficient CPU potential. No. I think I've found our one of our

15:55 Debugging Initial Instrumentation

16:15 problems. Oh, it's serving now? Yeah. But I removed the gRPC hotel stuff. So So here's the other thing that we're missing. There are some common init stuff that we did in the hotel con in the hotel config in the example app that you have not yet done here. So if you go back to GitHub, and do you see where it says config.init? That is doing some boilerplate that we need to also add. So let's go look where that comes from. So that comes from it's common init code shared between the client and server. So, yep,

16:20 Adding the OpenTelemetry initialisation code and stdout exporter

16:50 that may be able to take you directly there if GitHub let you. Perfect. So this is the stuff that we also need to put in, except for this. This stuff is not specific to gRPC. So instead of putting it in the gRPC code, you should put this in your main function. So you can literally just sleep all of that in it, and I would, like, call a function called o Telenet or something. That way it's, like, nicely modular and put it in your main function. Okay. So we go to our tank server main. We're gonna drop in this edit function.

17:25 We're gonna give it a slightly better name. Yep. And let's see if that says. Okay. So maybe missing a few things. Yep. You're probably missing every single one of those imports. And even if you aren't, even if Odell was not or even if the ESCO is not able to figure it out, it'll resolve the duplicates, so you'll be fine. Yeah. I'm hoping when it saves, it'll just yep. Yep. There you go. Yeah. Wasn't able to figure out SDK trace. So, unfortunately, we have two packages named trace in OpenTelemetry. One of them is SDK trace and the

18:02 other one is API trace. So, therefore, the name trace is not necessarily an ambiguous. That's why we have SDK trace. So this is actually a great opportunity to talk about the difference between the API and the SDK in hotel. The API is what you use when you're instrumenting your code. Right? So let's just say, you know, this is a span, this is a child span, whereas the SDK is something that you configure once per process initialization. And the SDK is the thing that keeps track of where am I writing out data. Right? Like, what are my configuration options? Right?

18:31 So your application code doesn't need to worry about the SDK except for in the one place where you're where you're initializing your system and configuring it. And this means that libraries don't need to have a a OpenTelemetry SDK. They just need to call the API. So that's something I want to get you to talk a little bit Liz, was so if what you just said means we can start to lay in OpenTelemetry into this open source software, and it can lay dormant until somebody wants to turn it on or hook it up to Honeycomb or

19:07 LightStep or whatever. That's correct. That's cool. Okay. So does anyone know why my fatal is showing me Probably because your log package is not necessarily the same log package package as as we're as we're expecting. Just see what autocomplete tells you that logger can Oh, try logger.fatal. You have a logger here, so let's let's use your logger. But that's initialized too. I guess we can pass it into the hotel in it. Right? So Sure. Yep. I don't go very often. Does someone wanna take over for this little bit? What file? I think Amy just volunteered. Go ahead. Which

19:50 file is that? Main go. Main dot go. Yeah. So I guess we'd have to say log and then put in a type or You don't want log. That's going to cause a type collision. Yeah. I'll grab it in just a second. Where's the interface? Logger. Pointer? Let's go pointer. So while you're doing that, we did have a question Yep. It'll work. A question if we can tackle that right now. Wait. Is it logger.logger, or is it log.logger? Pretty sure the package name is Logger is loaded. Let me fix it. No. But the the package name for log for the capital l

20:31 logger is log. The package name is log. Is it? Oh, I see. I see. Right. We have a question saying if you have the Prometheus interceptors not not enough, can it feed OpenTelemetry? Yes and no. Prometheus can feed its metrics into the OpenTelemetry collector to produce metrics, But this should bring us to an excellent occasion to talk about the difference between a metric and a trace. A metric is a pre aggregated point of data. Right? Like, it's like a gauge that's, you know, amount of CPU used or, you know, amount of disk space used. A or it's a counter, like number of

21:11 requests received. Whereas tracing is a lot more granular. The Prometheus exporter, by definition, is not capable of producing that fine grade data about the execution of every request. I can tell you how many requests you got, but, like, not how each request executed. So Prometheus usage is not incompatible with OpenTelemetry. Right? Like, you can feed Prometheus metrics into the hotel collector onto your metrics platform of choice. It's just that, you know, you you would rather part of why we're going through this exercise is to get that higher granularity data. Excellent. And I think Amy's code has now got the hotel in

21:49 it happy. So I'm assuming we can after our logger here, let's just Yeah. With the help me. We don't need Yeah. That verification anymore as that we can just pass in logger. Yep. I think that should do it. Oh, I think you want I don't know if it needs to be a pointer, but it'll do for now. It's not Sure. Why not? Anymore, so I'm not gonna complain. Okay. So that looks like Versus code is happy, which is usually a good indicator. I'm happy when it's happy. So What does still think? Let's pop over to tilt at

22:28 Tilt's always so critical. We have a restart and failed container. Did did it just build? So it's currently in this orange. Okay. Yeah. So it may oh, wait. Wait. What was it? K. That might be it. Let's see. Alright. So what's it complaining about? It's saying new config file found, but that's info level. Do we have the pod logs? Do the pod logs tell us that there's, like, some kind of a crash or something? Okay. Let's take a look at that. So we can do let's get pods. Let's grab the most recent one. I think

23:17 that should match with what we get there so we can try describe and see if there's any other information. Yeah. It looks like it is just crashing. There is a Well, there are logs and metrics. This would be so easy. So we're not really getting anything useful. Now if we take out and it does well, if we take out the the g r p c stuff, it does work. So I'm trying to work out It it does with the build it out there. Okay. So we were able to run OTELLONIT, but not the gRPC interceptor.

23:54 Can we can we try that to verify? We can try that. So let's Yes. Let's take out the gRPC interceptors. Oops. You commented out one too many lines. Yep. There we go. Okay. So let's try that to see whether we're at least able to initialize the hotel boilerplate. Yep. It's green. It's health it's healthy. Interesting. Okay. So there's something about the interceptor that is not happy. So let's try the theory that maybe trying to have two different interceptors is not working. So let's comment out that and then comment this and see whether that works. I clearly don't know enough about gRPC interceptors.

24:18 Verifying Basic Instrumentation

24:38 This is a research thing for me to learn about more later. It's green. I think it works. Did it recompile already? Yeah. Okay. So the answer is you cannot have more than one interceptor. GRPC will crash the server without an error message if you have more than one interceptor. Okay. Great. So now let's fire up the CLI, and let's establish a connection. Okay. So I I made a small just fail that hits the server. So that is called a hardware list. So we should have some something somewhere. Okay. Great. Let's find out whether we got

24:45 Triggering our first trace

25:17 a gRPC let's let's find out whether we got a gRPC call recorded to our terminal. So let's go over to the That's file. Yep. Uh-huh. Yes. We did. Yay. Okay. Could could could we cover that then? So we are not actually writing that trace anywhere. So by default, the interceptor just drops that into standard out? The interceptor does not, but the boilerplate, if you go back to the boilerplate in the In Yep. Main. In main dot go, what you'll see is that it's configured to write to it's it's configured to write to write to standard out.

26:04 We can change that. We can send this output to Honeycomb instead. Amy, do you have the Honeycomb API key? I do. Let's Okay. Why don't you research why don't we go find the Honeycomb API key? Got it. I'll send it to David so he can, like, not try not to put it on the screen. Yep. That that's that's alright. We can work I can make that work. I'll I'll Slack it to you in the background. Okay. So what are the first steps then to sending this to to Honeycomb? Okay. So, again, like, we like to walk through the journey

26:10 Adding the Honeycomb exporter

26:37 as if we were a user. So I happen to know this, but let's go ahead to GitHub Honeycomb v o OpenTelemetry export or go. I I don't currently have admin, so that is a API key I found. So if we accidentally expose it, we're gonna have to go bother people to clean it up. So You can create a new you can create a new API key. I can. Just create a new write only API key, and then it's fine if it appears on stream. It's k. Alright. So just set that up. I must be a team owner to create and

27:11 edit API keys. Okay. Why don't I sneak Let me let me just give me a second. I might go get this one real quick. I know people. Although he's offline. So Alright. I'm just gonna start copying the basic code and see what happens. So Excellent. That's how things are done these days. Yeah. I'm okay with that bit. Yep. It's completely normal for principal engineers to do. So you copy the case to me, but then you kinda hammer on it for a while, make it look okay. Does this code go in our hotel in it then and not in the gRPC stuff?

27:49 Yep. I'm assuming we maybe replace this part here. Let's mark this out for now. Yep. Similarly, your dataset. Yep. Don't turn on with debug enabled. We already have we already have, whatchamacallit, standard out output. So we don't we don't also need this, and it's a handy demonstration. Okay. But I do need the bracket. What's your team name? Is it Equinix? Yeah. Great. Okay. So now I need to understand what that honeycomb is. So I'm just gonna copy this URL and hope that that magically does it. And paste this and call this Yep. Honeycomb. Yep. I I also Oh, no. It's slash honeycomb.

28:44 It's slash honeycomb. Okay. Okay. We took care of that for you. Yep. Codependencies have always confused me. I try my best. Okay. Finding an owner, suing in as them. He he's actually poking at it right now too. Oh. So Okay. Great. In my case, I will leave leave pseudo mode. Okay. And you want to change your dataset to be a descriptive dataset like OpenTelemetry Tinkerbell or similar. Yep. Or, like, dev Tinkerbell maybe since we're using this for dev right now. You'll in a production environment, you want to change this to be configurable with a environment variable or

29:31 similar. But and our service name is Tinkerbell. But would would we wanna include things like the environment name and stuff in that, or would that just be the app and then we could use other dimensions for, like You probably want to have one dataset for environment, so you would want to programmatically construct your dataset name. Okay. Can we kinda just break down some of these terms, Jennifer? Well, for me, that's not familiar. So when we say dataset, based on the conversation that you two are having, is that like a table, a database, something that groups It's essentially

30:00 a database. Yes. It's a kind of unit of of being able to look at your data at least within Honeycomb. So this concept does not necessarily have to exist in OpenTelemetry, which is why it's specific to the exporter. Okay. And the service name, should that reflect that this is the Tink server? Yes. Right. Okay. So we have our dev tanker bell database. We're tagging these traces that they come from a service called tank server. Yep. We still have a little bit of red squiggles. So let me see if I can work that one out just now. So the

30:32 import seems to be okay, but this is complaining. Okay. So the dataset Oh, it's because you already instantiated a another exporter called standard out. So I would I would, like, change exporter there just above to say standard to say, like yeah. So this is, yep. And then And then again there. And then now you want to pass another STD trace with syncR, Line 66. Yep. And pass in addition to passing the standard out exporter, you want to yep. With thinker, and then we pass in our honeycomb. Right? Yep. That's correct. There you go. That should roughly do it, and now we

31:27 just need that API key. And as long as it's configured to be a write only API key, it's safe to expose it relatively straight to expose it on stream, and you can just revoke it afterwards. Worst case, someone writes, a hundred events into your Honeycomb dataset. Whatever. Okay. So the Honeycomb exporter doesn't seem to have a close. Can I just comment a sec? It should have a oh, that's out of data documentation. Let me see what that that ought to be. I believe that might be let me check that. One second. It does not have a close. It does

32:11 have a shutdown. Okay. Shut down. Okay. And I think this is what we actually grabbed from the honeycomb doc. So I'm gonna pull down the other one, which I I think I just got an update reference to. Oh, yeah. Yeah. Yep. So that needs to go down. Yep. And You want that error nil above. Because, otherwise, if creating the honeycomb exporter fails, you want to have it fail out before you Sorry. Can you say that again? The line 65 to 67, move that above the instantiation of the trace provider. Because, otherwise, if the honeycomb order instantiation

32:59 fails, you want to know about it before you Yeah. You have you have it stubbed to underscore on the That means the context. Yep. Yep. Background context is fine. Why is it not You don't have it loaded yet. Or no. It's context. Like, context dot. Okay. There we go. Oh, that's not completed Do we have the API key? Yeah. David has it. You you can paste that one. I'm gonna add Did you send that? So I can clean it up too. Yeah. Just I just slapped it to you. Okay. Do you want me to just paste

33:39 it in? I'll just move this over here. And this is a write only one, I can share it, or do you want me to It is no. It's fine for now. Yeah. I'll I'll If there is a problem with it, we'll forgive it on the back end. Don't worry about it. Alright. So I've dropped it in, and I have closed that file. So we can't open that file anymore. And if we do, we'll just deprecate the key later. So it's okay. Yep. Okay. Great. So now let's go to tilt and having hit the having hit the endpoint.

34:11 Debugging Exporter Setup

34:19 So Yep. Okay. So we're serving. We successfully in the Honeycomb Explorer. It hasn't failed. So let's call the endpoint. Oh. Oh, try again, I think. Yeah. There we go. Better. Okay. Where is my email? Gonna try to add David real quick to an email. So if I understand what's just happened there, we've added the Honeycomb exporter to the gRPC plug in, and it's now writing the trace data when we ever whenever we had the tank server gRPC API and sending that to Honeycomb. That is correct. Okay. Unfortunately, I am not yet seeing your data on

34:58 the back end, so let's see whether there's any errors or anything. Oh, that doesn't look good. That sounds like we have some kind of fatal error. Oh, yep. Here we go. So our tell is telling us that something went wrong. Oh, your defer is bad. Oh. I see the problem. So what happened it out for now. Right? Like, it's not the end of the world. Yeah. You want to move that stuff into your shutdown sequence. Right? The problem is that it deferred at the end of your main function, whereas your main or, like, your does that make sense? So the The defer

35:40 runs at the end of the function it's in. Yeah. And you want you want it on the to to run at the end of main instead. Yep. Okay. Do you wanna you wanna move that to fair, and I'll just leave my screen away from it because we have the token. Have you got to print out? Let's do this. I mean, you don't have to run it. You do not have to run it. However, if you have a a graceful shutdown, right, the idea is you don't want to lose your traces that are in flight. Okay.

36:09 Where are we at? We're in hotel and So Yeah. You may need to make otal on that return the honeycomb object or return, like, a shutdown hook. I was just gonna paste it in here for now. So root command execute. Right. That should be fine for now. Right? It's not oh, oh, okay. I see. Okay. Now I understand. Yes. Because we we put it in the yeah. Okay. Fun. Fun. Fun. Fun. Oh, my key key repeat still isn't working. What's the type on that anyway? It should be a might be a. It's a Honeycomb dot exporter.

37:23 The honey Honeycomb dot exporter. So what was the problem when we moved the defer line down? It sounds like it got a little bit more complicated. Because the defer was was wait. Oh, right. There there we go. That should do it. So the defer was inside of our hotel init function, but the hotel init function returns control to the main function immediately. Okay. The defer needs to make sure that happens as part of cleanup after the rest of main finishes. So we needed to hoist the the close function out. Okay. So now we've done that. I think

38:05 that should work. Alright. Well, our tank server Is running. Sorry for Elite. We're not showing the codes just because the API token was there, but I will push this code to my fork with the API key removed so you can see that hotel in it function. Don't worry. Now the tank server says it was restarted two minutes ago. So have we definitely saved that file and triggered a rebuild or is it just tell is I mean, I did it colon w. You may need to trigger, like, an innocuous change to some other file, or you yeah. You can just press the

38:43 I'll just force a rebuild. We'll see what happens here. So let's give that a couple seconds. It's now running. So if I run this again, and it's pretty consistently now Now it's reliable. Okay. I see data. We have data. Okay. Yep. We have data. David, I sent you an invite. It should only take you a second if you want to log in to Honeycomb. It should be in your Equinix email. Okay. No problem. Let's look at some of these traces. So earlier, we were looking at the traces in, like, a text format, like, not super informative.

39:03 Viewing Traces in Honeycomb

39:28 So the goal here of using a visual tool like Honeycomb or if you want pure open source, Yeager, etcetera, right, the beauty of this is you, you know, can use the same instrumentation with multiple different back ends. Then you can see in a little bit more detail what's going on rather than just having the things printed out to standard out. Okay. So while I'm just quickly signing up for the honeycomb thing here, right now, we're instrumenting the gRPC server. Mhmm. Now we if we wanted to track that trace from the CLI, is that something that would

40:05 be just as trivial to to kind of add? Yes. That's essentially correct. So, again, we're basically following the hotel gRPC example except for you want to look at the client directory instead. Okay. This is Honeycomb. You should be able to see that now. Let me see if I can just zoom in a little bit. So how do I see this trace data that we Scroll down to the bottom. It'll show you, like, a list of recent traces, so you can just open one of them. And you may want to collapse the left sidebar so it's a little bit

40:41 Oh, there we go. Down at the bottom. There we go. Ah. So yeah. So you can see here that we have that the name of the gRPC call being made. You can see how long it took. You can see how long it took in comparison to other traces of that same name or other spans of that same name. If you scroll down on the right hand side by fields, you should be able to see a couple of other fields indicating, you know, hey. I'm a tank server. My status code was 0, and so so on and so forth.

41:14 So as you can see, it doesn't have very many fields. Right? It's only what gRPC was able to automatically infer. So that's where the thing that I mentioned about, you know, providing additional information about the client or about the request being made could be helpful here. Okay. So as a how how would I add extra context to that trace? Great. That's a wonderful question. So let's pull up the RPC handler. Okay. So here. Is this what you meant? Sorry. No. The RPC handler for that get all call. Alright. Okay. Gotcha. Okay. So we wanna jump

41:32 Adding Custom Attributes (Manual Instrumentation)

41:55 into the hardware, and it is this push call here is what we triggered, I believe. Oh, no. We did a list, didn't we? No. You yeah. You did you did a list. Oh, there it is. Yep. Okay. Interesting. You do have a database, it looks? Yes. So that speaks to Postgres database. It speaks to Postgres. Great. So one thing that we'll do, maybe if we have a little bit of extra time, I don't want to do the CLI instrumentation today. I don't think that's going to be as useful for your viewers. I think that the interesting thing would be

42:32 to instrument all handler, and then also if we can instrument the database calls underneath it so you can see out of the total length of time that this is taking, how much is in the database and how much is in your opcode. Okay. Great. So if we pull up the documentation for OpenTelemetry so if you go to in your browser to go.OpenTelemetry.i0 and go to the hotel, which is the API documentation, and go to subdirectories trip API trace. API trace. Yep. There we go. Perfect. So now you can see all of the things that are valid to do.

43:20 Now the first thing that you want to do, remember that gRPC automatically set up for us the the tracing of my request. So that means that what we have to do we don't have to instantiate any new spans, but we have to find the span, the hotel span that was created for us. So for that, we need span from context. So let's go look at the Go documentation for span from context. That one. Perfect. Yep. So you want to call go to OpenTelemetry.i0/hotel API trace, and you want API and you want trace dot span from context and run it on

44:03 your current context that gRPC has passed you, and it will give you an hotel span object, which has been helpfully shoved in the context. So let's see what that might look like in action. So at the top of your function, is your context object? I'm assuming it may be on the server. Yeah. It's probably Get workflow front? No. That won't be able to. This is where my gRPC knowledge fades off a little bit. I think server is it was in we were just looking at it. Yeah. It's in gRPCserver.com, and it does not have a context object.

44:53 So where does that all get called? I presume the No. We we previously removed the Prometheus thing from this, and I do see Prometheus code here. So would that help us understand where the context is if we look at this Prometheus code, or is that No. The Prometheus package may have its own way of passing the the info. So, normally, like, I'm looking at at some example code, and it looks like if you need a context, you should just make context the first the the the first method the first thing that you take in your signature. Yeah.

45:40 That's what the I was just looking the rest of the functions are doing, so I can add that. Yep. Okay. So we just add this context here and g r p c c t x space context c context dot. Yeah. Okay. Context. Okay. So now you have your context. So at the very top of your function, before you do anything else, you probably want to get the tray the tracer. So so remember, it's it's trace dot trace dot trace from connex. Not not it's not a method of the of the GoConnex. It's a method of o

46:17 of hotel trace. Oh, the hotel one. Okay. Alright. Yep. So hotel.trace. Sorry. It should just be trace dot. Yep. Trace from contact or spam from context. No. That's all I have access to. Yeah. It's I think it's using the wrong trace. Okay. Just just start typing span from hot next or yeah. There oh, I see. There's runtime trace. Oh, I I delete that. It's it's it picked up the wrong import. That's why. Okay. Okay. So in your imports, let's let's just do this now. In your imports at the top. Okay. So let's import go.opentelemetry.i0/hotel/API/trace. Great. Okay. Now we have the correct trace.

47:23 Hopefully, it doesn't delete it. I've not solved it. So Okay. Good. You're mistyping. Let's see if we get there we go. Yay. Okay. Yep. Check the API docs. I think it may potentially error. Yeah. I'm just pulling that back up. It should be Oh, no. It just returns spam. Great. Yeah. Yeah. Okay. We've sent it using. So now we can add attributes. What are some useful properties of this that may be helpful? Let's let's add something saying the number of results. That seems like a useful thing to do. K. So we have the So at the

48:09 bottom Yep. And then we do hardware. We make a list. So right after that error, we can go Yep. So here, spin set attribute, I believe. Attributes? Yep. Set attributes. And this takes a OpenTelemetry key value pair list. So OpenTelemetry has it, like, you know, because that was silly. OpenTelemetry has its own key value pair kind of yep. Is it I think it's just API. I think it's in the API package. Oh, okay. Do we need to import that or let us Yeah. You'll need to import it. Just one level up from trace. Just l

48:58 tell slash API? It's slash label. Oh, somebody deleted me. Yeah. I think just don't save the file, and it should hopefully stay. In no. It's not even under API. It's just hotel slash label. It did it again. It's second underscore. I know. It's the worst. It's the utter worst. Alright. Well, maybe underscore once we've got some coding. Oh, it's label. Okay. I'll remove it. Okay. You should be good. So I'm importing from label dot? Yep. And then label dot int. So capital I int. Yep. And then now you pass the name of the key, which is going to be

49:54 probably, like, num results or something. People in Honeycomb usually standards upon either camel case or underscores. It doesn't really matter which you pick. Right. And then you want to pass the length of your list. But we also had a comment from one of our colleagues, Manny, who said that we could have got the context from stream dot context in the old method rather than adding a new a new variable to the signature. It's Lin, not Link. Dirt. Do I need to derep that? It should be fine with the pointer, shouldn't it? Oh, that's a it's not a list. Dirt.

50:37 That that's a that's a proto. Yeah. But am I coming then? So we do stream that sent. Oh, it doesn't actually store them. It just streams them straight out of the database, so we'll just add a little counter. Yep. So you may need to yep. Great. Cool. Okay. Great. So now if you call it, you should start seeing results with You should start seeing in Honeycomb, you should start seeing, hey. There are zero results, I guess, until you add hardware to the list. Okay. Well, the tank server seems happy now, so let's jump back over here.

51:34 And it's all crashing. That's awesome. Okay. Now let's go to Honeycomb. And Oh, there it is. Go to Honeycomb homepage. It's easier. Upper left hand corner. There. And let's go look at one of the most recent requests, which is a few seconds ago. Yep. So now you can see on that span that there are scroll down to the bottom of your list there. Yep. You should see oh. They did not show? We do not see here the the new attribute that we added. Oh, is that code even actually called if there are no results? Good call.

52:27 Well, actually wait. No. Error get get all. Oh, wait. It's calling a function, so it's sending a closure over count. So is it actually gonna count? It's gonna still count zero. So Should still count zero. Still be sending it. Right? Because get all takes that function and then calls it for each result. Should we just confirm that we actually this this is the function that's being called? Yeah. That might be useful to do as well because we don't know for sure. Yeah. Let me just see what else we have here. But I I, yeah, I think

53:02 Debugging Attribute Visibility

53:07 that all is definite hopefully, is what we're we're calling. And we're run Well, I can tell you that the OpenTelemetry side is fine. So it's just a matter of are we calling the calling it appropriately. And, also, I guess, are we using the correct context object? But I'm pretty sure the one that Manny said. Because we had a comment from our colleague who says we should be using equals stream dot context. So maybe that's just the fix there. Oh, yeah. Okay. Let's try that. Okay. So let's just rebuild that with till it doesn't seem to notice the change in

53:40 this file. So it might just need to retweak to the setup. And that should be healthy again. Yep. Alright. Let's call it. Couple of calls. Jump back over here. Home page. Oh, wait. That's too much. Nope. You want the one from a few seconds ago. Yep. This one here. Yep. That'll be let's see. No. Direct. At hardware service slash all. Yeah. It it is calling hardware service slash all. Not are we in the first file? Just something silly at the top of the You could add a logger there to make sure that it's actually calling this code.

54:33 Oh, like a span dot it's like span dot log? I'm just saying about regular log. It doesn't depend upon the OTO subsystem. That way we know for sure. Do we have a log global logger handle? Or no. You gotta grab it. Okay. Got it. Do you normally wrap up your stream after an hour, like, on the dot, or do you let it go long sometimes? No. No. There's There's been some that I've planned for an hour that have lasted two and a half or three. So, like, as long as you both have time, I'm happy to Yeah. I have time. Let's let's

55:11 keep let's keep going. Let's while we debug this, why don't we add some database instrumentation too since we know that the automatic instrumentation of gRPC at least is working? Okay. So let's go to once Amy is done here, we'll go to the hotel contrib repo and have a look at the documentation for you're using the standard database classes in in GoA? Let's see. Why is it complaining about label now? Did we we did use label. Right? Yeah. Yeah. It's complaining that actually. Oh. So maybe that's our problem. Maybe that's why is that is that it never picked up our

55:56 change because it never compiled. But here's we're using it. So It says no package for import. Go OpenTelemetry.io. Can I see your go dot mod? Actually, I'll go look at it myself. What's your go mod say? GoMod says where is your GoMod? Wait. That's weird. Why am I not? I may need to refresh my my share. I think it's a little confused. See. Version 13. Yeah. That's correct. Are you looking at it different way for me? Because I don't see it. It's I stopped complaining there, if that helps. Oh. I think the s code may just

56:58 be problematic day. Yeah. But it's not complaining about label anymore. We let's see. Did we add a logger? We did add a logger to this function. So I'm going to trigger a Yep. Let's try it and see what happens. Tilt. And we'll see if we see what was the log message called hardware all. Alright. So it's running. Alright. Let's hit the endpoint. And I do not see her message. So it called Interesting. The service button. No. Okay. Now we know we've been instrumenting the wrong function. What the hell Alright. What else do we have then? Let's see.

58:05 So if we come to the command root, we've got the RPC servers here. I mean, it looks like we're definitely modifying the correct one. RPC servers tanked g r p c, g rpc, hardware Go. I mean, I I don't see how we've been getting getting that wrong. Yeah. That seems this seems very surprising. So what can we do? Now our tell also stopped noticing changes, which I'm gonna assume is maybe potentially related. So I'm just gonna add a comment to this file. Yeah. That's not triggering. Yeah. I think that's a little concerning because I'm curious whether Tilt is even building the

58:55 right. Like, is Tilt even building the correct files. Right? But we are seeing the hotel stuff from the hotel in it. So That's correct, but it broke a little while. Right? Like, it broke a little while after that. So Alright. Let's let's tilt up again just in case. So Alright. Do you need to tilt stop first or not? I just killed it. Okay. Great. Great. Great. If I reload this here, we can see there's no bringing everything back up. We'll need to wait till all of this screen. Okay. Great. While we do that, why don't we go ahead and look up

59:00 Further Debugging Build Issues

59:31 how to instrument databases? And we So you said coworkers that I'm missing stand up because this is more important. We're we're missing an all hands too. So You can watch the tape of all hands afterwards. It's fine. I'm just gonna poke around here and see that I can see then. What kind of keys are those, Liz? This is the terrible keyboard that I'm getting rid of soon. I do not recommend this keyboard. It is a Rawkat keyboard, and it unfortunately has a tendency to have key presses duplicate, which pisses the heck out of me. So

59:40 Adding extra spans / instrumenting database calls

1:00:14 I actually have a set of cherry drums and on a on a keyboard coming soon from from. Okay. So we want the SQL documentation. I found this import. Yep. I haven't found any documentation, but can I just browse to that? You should be able to find oh, I think that is a temporarily missing feature gap. Actually, I'm not sure that we have the SQL stuff instrument. So Well, I will We can just go the the database classes are pretty straightforward. It's not a Oh, yeah. Let's let's demonstrate how to add new spans. Perfect. Let let's do that

1:01:10 then. We automatic information is fun, but let's let's do manual instrumentation. Okay. In our DB So I'm assuming I pulled back up the hardware stuff, and we're looking for the DB calls. Right? Yep. Yeah. There's Where it says DB get all? Yep. So let's pop over there. Let's Which driver that instrumented? Hardware here, hopefully. Yep. There we go. So there's all. Okay. So here, it's, like, running a query, right, with the context, which which is fine. But my question is, where is it instance dot query? Where's the code for in for for instance query? Let's find it.

1:01:56 And then Like, what type is instance? It can we hook in a little bit lower down? Yeah. Yeah. Alright. Here. This what you wanted? Yep. But this looks like it's in go it this is in Go's library. Oh, it is. Yeah. Okay. Okay. We have a tank DB struct with an instance on it. Oh, that that's doable. That's doable. Okay. So if you have a tank DB, that means that if someone runs query right? Right. If if someone is running a query on it, then they're going to call the internal methods. Right? So we can create a span every time

1:02:47 the query method is called on TinkDB because TinkDB implements SQL DB, and that's what's being called. Right? Okay. That makes sense. Okay. So where is where? I I don't see it. They're all kinda open coded. Right? Like, they're if you go into the hardware.go Yeah. There's this here. It's They're just open written queries. So Right. No. No. No. I'm I'm seeing the query function. Where is the query function implemented on TinkDB? If we go back to TinkDB and search queries directly on TinkDB. Is is a struct that just has a SQL pointer. Instances just it's just a SQL pointer. So

1:03:42 it's calling the Golang package. Oh, calling the Golang package underneath immediately. Yep. So I guess naively we could start by instrumenting it directly here, but ideally, that we should be adding another layer instead of tank d b to proxy the queries so the instrumentation could be in a single place. Is that Yep. That's correct. Although, like, as I said, once we get the your future request for your future request is noted to get SQL DB in auto instrumentation ported over. We have not yet done that from OpenTelemetry slash OpenTelemetry, but we will. Okay. So see here,

1:04:17 where is our context? The problem is we don't have a context object here. Right? Like, that's that's the thing that worries me. Yeah. We kind of need that context. Some of them have it. Just that one that one call doesn't. Why does just get all not have it? But, like, we can just change it. Right? Like Yep. We can change it. I'm gonna change it in hardware to test the context and then Okay. I was able to once have it. Okay. Yep. Yeah. Let's let's make it always always take the context. So let me follow.

1:04:58 Oh, fairness info on that. It was did he pass this by around his pointers or just Nope. Nope. It's always passed by Volley. K. So there's context context. It should be good now. Okay. So we have a context. Did I totally break the build? I'm sure we'll find out. It looks okay. We're good. Nope. Build failed expecting Oh, yeah. N 51. Oh, why did it not change orange here? Never mind. Okay. Yeah. So it did fail. Did I guess I'm running the context passing? On line 51 gRPC server. What? Looks like we accidentally one of us

1:05:49 bumped the key here. Yep. That would do it. Okay. Back over to TinkDB. Oh, now we've got more interesting. Cannot use DB as type. TinkDB does not implement. Oh. Wait. What? Yeah. So we do have a a Versus code that is showing us an error. Do that instance dot query. Yeah. See here, we've got it doesn't accept with DB here. Oh, I see. When it's building us Oh, okay. Back in here. Let's do a quick diff on this file. I bet we accidentally deleted something. I'm not trying to do this quickly in the s code.

1:07:03 David, I think we should just look at a quick diff of what we've changed all we've changed in gRPC server, and then we'll quickly see what happened. I've got too much going on here. That's fair. Yeah. There we go. Hey. Hey, Liz. There's a question about how would we make sure logs events do we already read that one? Are are are during this request are correlated with traces? Ah, the log correlation? Yes. So instead of using your logger, what you want to use is something called a span event. So you can just call span.ads add span event and then pass your your

1:08:02 message. So it's just a change from using your standard logger to using a span event, and it'll automatically make it a child of the trace. Db.database. So d b comes through from in root.co, and it is passed in. Well, I think this should be d What's your type definition where where where where where databases is defined? Maybe that's d b dot instance. Yeah. What what does server take? Oh, it's private. You shouldn't have to change that. I think it's the interface definition for DB. Right? Like yeah. So I I don't get why it broke. I understand why it broke because you added

1:09:01 context. Right? You added context and function signature. There are some other interface definition presumably for testability where you need to make it take context now. Oh, because it it was passed in or or did we In all the other in all the other methods, it was. But you need to so we can you search your your code base for get all? Like, search your code base for get all. There we go. Yep. There you go. It's right there. Nice. That should do it. Messaging return. Oh, no. Wait. Let's see. Is that it? No. Force to recompile,

1:09:49 but also look at main go line 74. Yeah. As you can see, a lot of the fiddly stuff is not actually a hotel. It's just like Yeah. We we kind of expected a little bit of the this code to fight us a little bit. A little bit of spaghetti. Yeah. Oh, propagators. It's a return. Oh, honeycomb export. Alright. Hey. Now the auto build is working again, at least. We're getting there. It's happy. Up backup restarting. So I just made up what I added to here. Maybe we should just undo this a few times. It looks like maybe

1:10:54 this was the changes that you both made off screen. Is that right? Have I made that That that's that's correct. That's correct. Right? Like, because we had to pass in the honeycomb Exporter to be able to You don't wanna return star Honeycomb Exporter, please. Yeah. You want to return regular Honeycomb Exporter. It's in pink server main. What's the Oh, we need to we need to return Alright. This this function signature needs to return a pointer, not a not a concrete honeycomb exporter. So go to the function signature here. Alright. We we do need to report return a pointer

1:11:36 to the x. Yep. Yep. Right. That's exactly correct. And then down here, we're gonna go return we have a honeycomb Ampersand honeycomb explorer, I bet. Yeah. We'll need to delete that API key afterwards too. Yep. And I'll do that as soon as we hang enough. Oh, what? That's just missing there. Yeah. What's complaining about? How are these things all getting, like, randomly It's a it's already a pointer. Okay. Yeah. Because we're probably Type it over each other, I think. It's okay. Alright. Let's see what happens now. Running starts. Let's see. Can I no? Server's not running.

1:12:33 The only way of tracing. This would be so much easier. Back off restarting failed container. First to rebuild? Yep. Yeah. So it's compelling, but it's just not running. Did we accidentally revert the change with the interceptors? Is that is that what's happening here? Very possibly. GRPC? Yep. It would be in gRPC. There. Including commenting out the Prometheus. But something did go weird with that code at one just a few minutes ago, so I don't remember what we did. Yeah. The Prometheus Okay. That's still commented out. Okay. That's good. There was one there was one more line

1:13:39 in here. That's right. That set up that that it did something I thought was important. I don't remember what it was. Can you can you do a diff on this section real quick? Let's fail this one. Oh, the creds is missing. Oh, yep. The creds are missing. That would do it. Oh my goodness. That looks important. Yeah. It looks important. I have no idea how that lane disappeared, but then the The v yeah. The the curly brace disappeared at some point too, so must be all these sharing and everything overlapping. Yeah. Okay. Grumble grumble. So it's Alright. Let's rebuild and see what

1:14:38 happens. And then we can get back to the gap we are we are shaving, which was which was instrumenting the database call. Still getting the Still failing? Why? Let's diff can we diff everything we've done so far? Curious what that global diff looks like. Alright. So that's just the tank fail. Main Go add it at OpenTelemetry. Yeah. That looks fine. Okay. Yeah. That differs in the main, so that's okay. Yep. Yep. That looks fine. Yep. So that all looks okay. Added the context. Added the Yep. And that's go mod. That's very big. Yep. Let's go back up here.

1:15:46 Well, Prometheus gone. Hotel n. Yeah. Logger. Oh, you remove it you removed a logger equals logline. That might be important. That looks important. Wrong way. Okay. So Oh, yeah. That was there before. Right? I think so. So that's in the gRPC main. Right? Okay. Yeah. Yeah. Just probably right up that brace. Okay. That'll probably get the build build going. I'm wondering why they chose that in here. Anyway. Let's just keep blaming Versus code. Okay. Alright. It's running. Alright. Let's make sure that this works. Yay. Hey. I'm annoyed that it fails silently if you don't do that. You know? Like, that that

1:16:40 that really bugs me. But okay. Great. Back to where we were. Our result count shows up now. Oh, it does? Yeah. Okay. So the problem was that it never compiled after logger equals log disappeared. So none of our other changes were making it through. So now we can probably delete that extra logline because we Well, let's not delete anything until we get through this next step. Yeah. Yeah. Okay. Let's go back to the manual spans on the DB cluster. Let's go to create manual span from the DB. Okay. So which let's see. Where is that function?

1:16:45 Verifying Custom Attributes and Spans

1:17:19 No. That was in hardware, wasn't it? Under the DB. Okay. So this is where we wanna add our span. Yep. So first, we need a trace from context. Yep. And remember, it's trace dot span from context. And c t x. Okay. Yep. And now we need to set or we need to instantiate a new span. So we're going to if you look at the documentation for for the trace API go docs again. Yep. So look at the type methods on type span. Yep. So to answer the question about what people are saying earlier about trace logs, right, like,

1:18:11 add event. Right there. Add event. That's where you would have added a, added your your contextual logs. But instead of adding event, what we're doing sorry. It's answering the previous question of how you add a lot logs. Here, though, we need to create a new span. So there should be a I should read the docs instead of using Read read the docs. I'll I'll be right back. Yeah. No problem. Oh, why am I failing to remember how to how to do this? So when we call trace span from context, that's returning a new span, is it not? Or if I mess

1:18:56 up No. Spanner context returns our current span. Ah, I remembered the deal. We don't need we don't actually need to run span from con or we will need to run span from context later. Potentially, no. We don't. Sorry. The thing that we want is to get our our trace provider or trace dot start. Ah, that's it. Trace dot start. So something like let me paste it over. Actually, why don't I can directly edit this, can't I? Go for it. Where are we? We're in database? Hardware.com. Hardware.go. In database. Yeah. Great. Where are we? Lane 156. Ah,

1:19:37 yes. Perfect. K. So we want something like this. So something like this. Yep. That should do it. And I hope it automatically inferred. No. It did not. So we need to fix this or if it's not happy about this. I think the Versus Code web interface has some sort of operational transformation bug because I'm seeing lots of duplications. So I'm just Oh, no. I see it too. Yeah. And it's gonna clean this up. I think that is where we wanna be. So I've saved that. Yep. Right? Yeah. Yep. I'm gonna close my browser then to prevent

1:20:41 oh, no. It it did more of that. Okay. I'm going to close my browser so it stops do stops messing this up. Yeah. I think it's okay. I think yeah. We're we're good now. Okay. I'll look at the top. Let's make sure my imports are correct. Nope. That's bad too, and we need API trace as well. Okay. So API trace. I think it might be just r slash hotel slash global, not API trace. Okay. Hotel global and API trace. I wanna find out what feature that is so I can make sure I always turn it off.

1:21:25 Yeah. Delete stuff. It cleans up unused import. So you have to add this underscore at the start until you have code that you need. It is API global. Sorry. And it is API global. Okay. Okay. So now let's go ahead and add with the trace. Let's add add in some attributes to the span. Oh, actually, we don't need the trace library, I think, because we got the span automatically. Okay. So what would be useful to remember from here? What do we need? Length of the query, maybe the input parameters, like the hardware ID. But this is get all. So get all.

1:22:08 Okay. So maybe there's just no no useful info about the span to report. Great. Perfect. Okay. Let's solder on and see whether we get the get all span now in in Honeycomb. Should we just add something random so we can see that coming through? I mean, make it No. This this should make it come through. Just it'll create just a span named tink d b dot get all. Right? Okay. I gotcha. This is creating the shell span. So Let's check till it's still happy. So it'll still get us, like, how long the query took. Right?

1:22:37 Yeah. It'll still get us how long the query took. So deployed thirty seconds ago. I can still hit hardware less. So now we wanna go to Honeycomb, and we should see this getting a little bit more complicated. Right? We should have Yep. Multiple. That is correct. Oh, look. Two two colors. The database is only half of the time. That's cool. Well, I guess there's nothing been returned. So it's we don't have any hardware. You have some files ready to add add some. Right? Yeah. Should we should we should we instrument add hardware first, or do you

1:23:15 want to just run add hardware? Let's just add run. We're running a little short on time. Let's let's let's just insert some things into the database and now make it a little bit slower. I mean, this is per when I was looking through this code last night, this is way further than I thought we would get. So Honestly, Versus code slowed us down the most out of anything here. Yeah. The share thing is really cool, though. Well, I hope they keep working on it. Yeah. I hope they keep working on it. Okay. So we've just got to run this

1:23:48 command and, yep. That's the one. The hardware. What was it? Push fail. Hard. I think it went in. Oh, okay. I know. Ta da. Okay. So the next biblical hardware list. Okay. So now let's see what that looks like in Honeycomb. It looks the same. Well, it looks the same, and we should now see some attributes that are being set. Like, we should be able to see the Result count one. So we we we got a record. Do we have a result? Oh, we do. Wait. Yeah. Yeah. And the database is now the fastest part. So

1:24:44 it's weird. Okay. So that does beg the question. Right? Like, why is it that the that's why is it doing so much stuff in the forty milliseconds before it actually talks to the database? Right? This is part of right. Like, instrumentation for the sake of instrumentation is pointless. Right? Like, the reason why we do this is to and hover performance things or things that are like, That's not doing what I expected it to. Right? And then you dive in and debug. Cool. Well, I think this is probably a good place to leave leave off and probably

1:25:10 Conclusion and Q&A

1:25:12 wrap it up. Yeah? Yeah. Definitely. That was super valuable to me and up over to everyone else that was watching. I think Amy and I can have some fun adding instrumentation to this and try and work out with that forty milliseconds is going. But that was awesome. Thank you both very much for joining me. That was fun. Alright. Well Any audience questions? Oh yeah. If you have questions, you have a few minutes to get them in and we'll try our best to tackle them. Didn't think was very happy. Think. In my name. Oh, so much for So we've got a thank

1:25:48 you, that was really great. Thank you Jacob. Another thank you from Mike. Thank you, Mike. And we did answer the question earlier, didn't we? Yes we did. I think we're great. Great. Awesome thanks for tuning in. Easier. That was a lot easier from the point of view of importing OpenTelemetry and getting started. The challenges we had were completely unrelated which was frustrating but we got there and then so thank you both for joining me. Have an absolutely great day and that was a pleasure. Thanks.