0:00 Local Dev & Test without Dagger

0:00 Hello and welcome back to the Rawkode Academy. I'm your host David Flanagan. Today we're taking a look at Dagger dot I o. Dagger is a tool for building CICD pipelines. Go check out Dagger.i0 to learn more and of course keep watching this video. I'm going to show you how to use Dagger in a model repository context with secrets, composable builds that you can run anywhere including local host. As we see here on the Dagger website, you can run your Dagger pipelines anywhere. Why? Well, because they're just orchestrated containers. So as long as you can run a

0:05 Introduction to Dagger

0:35 container on your local machine, in GitHub actions, CircleCI or even in production or on a Kubernetes cluster, you can run your Dagger pipelines. One of the main selling points or what makes Dagger unique is that Dagger pipelines don't require you to write any YAML. You can define your Dagger pipelines as code. Today, we will be using Go, but you can also use Python, JavaScript, TypeScript and realistically any programming language. While these are the main ones supported by Dagger, as we can see on the homepage here, GraphQL is what makes all of this work. So provided you can take a GraphQL schema, you

1:11 could probably generate some sort of SDK or you could just write GraphQL and use your favorite programming language to execute it. That's up to you and maybe something we'll play around with on another video. But today, we're gonna focus on Go, where I'm gonna show you how to build your classic three tier application. The back end service written in Go, the front end service running in JavaScript, a database which could be either SQLite or Postgres and consumption of secrets. So let's dive in. Alright. So before we go into the code, before I show you Dagger, let's see what our

1:40 Local Development Experience (Without Dagger)

1:45 local dev experience looks like without Dagger. That is pretty good and local dev experiences usually are. However, they're not repeatable and other machines and other infrastructure and especially not in production. So I can go into the back end surface where I can do go run main. Go. That's a pretty hard dev experience to be. You know, they're using native tooling on a native machine and go makes it pretty easy. We can then jump down here where we do a curl to local host on eighty eighty or we can have the ping endpoint and we get pong back.

2:18 This API also has an ask endpoint where we can provide a post request with curl that means just providing a JSON body or anybody where we can provide a JSON request where we pass in a question. The question could be translate hello from English to German and hit return. Now we get an error and that's because we need a secret value. However, I've got that covered. Inside my top level directory, I have an ENVRC. Inside of this, I have the OpenAI token. However, I have it set to this funny looking string. This funny looking string is a one password

3:00 Handling Secrets Locally

3:02 secret reference. Now I can use this here because I have one password on my machine. Whatever your secrets management is, use that. One password is a good choice. You can run the operator and production and still consume secrets in the exact same way. Other notable options are Doppler from doppler.com. So that's up to you. Locally, I can do op run, go run main, where it will pop up on my watch and approve the request. And if I run our command again, we get the answer. Hello and dosh is hello. So now we have a pretty good

3:38 dev experience. We can now go into our front end application where we do a PMPM run dev. I can pop this open in my browser where I get my wonderful developer focused UI, which just means black text on a white background, where I can ask questions of Dagger AI. So let's see if we can translate hello to Polish. And voila. I'm gonna trust ChatGPT. So that's pretty good. But we're having to manage the back end, run the front end, and we're consuming secrets in a pretty good way, but still a little bit too much work.

4:19 Introducing the Test Suite (Hurl)

4:19 There's one other thing we haven't done yet and that is run our test suite. So we're gonna come out of the front end where we go into build test. Here, I have a test dot huddl file. If you're not familiar with huddle, go to huddle dot dev. That is h u r l dot dev. Gives you a text based format to describe HTTP requests where you can build assertions against their response. Here, I do a get on back end eighty eighty ping where I expect a 200. I also do a post to back end eighty eighty ask, sending in my JSON payload

4:49 with a question and asserting that I get the German word for hello back end response. We can now do hurl, test dot hurl and we see that it fails because we need to speak to something called back end. Now I could use local host here but that's not gonna work repeatably in other machines and other environments. So for this, I have created a just target where we can run just test and our tests pass. If we take a look at our just fail, we're using a nifty feature of hurl where we can pass dash dash connect to

4:57 Running Tests Locally (Hurl)

5:22 where we swap out the back end eighty eighty tuple for local host eighty eighty tuple. This is a decent developer environment at the moment. We have a just failed to handle the edge case where we can run the tests and where there could be a container based environment. But we wanna pull more automation into this. We want to be able to run it anywhere and we wanna be able to expand on this. If you started down a service oriented architecture or you're using a mono repository, it doesn't stop at three services or two services. It's potentially going to grow

5:27 Why Use Dagger for Automation?

5:51 and using composition of our CICD system, being able to reference other modules and actions and chain them together is gonna be paramount to your success. So let's see how we can take this three tier application, automate it with Dagger with a pipeline we can run anywhere that's extensible, cacheable and performant. So let's take a look at some code. Let's go through the back end first. This is a very simple Go application using the Gen web framework. As you can see here, we rely on an environment variable called open ai underscore token. This is how we provide that secret to

6:15 The applications code

6:30 the back end to speak to the chat GPT service. There's one other environment detail that is required for this application to work. When it's submitted, it's going to use an in memory SQLite database. Great for dev, but doesn't resemble production. So we wanna be able to provide this database URI with the same production environment that we use when we run our CI system. For today's example, we're going to use Postgres. I'm also a big believer that when you write code, the code that you use to orchestrate it, manage it, build it and run it should live right next

7:03 to it. We shouldn't be shipping this stuff away to another repository or another directory. It just doesn't make sense. So you'll see here, I have a build directory with a build dot go. This contains the Dagger code and function that are specific to this individual service. As you'll see here, we pull in the Dagger API or the Dagger SDK and we expose two functions. One called build container image, which is consumed by the export container image. If you've been writing code for one year, five years, ten years, twenty years, or longer, you have accumulated experience.

7:10 The Dagger Build Pipeline

7:40 You've learned how to write more code in a better fashion. Why do we throw this away when it comes to CICD when we just write YAML? Let's take all of that experience and build composable, extensible, maintainable build pipelines. Now this is a contrived example right now. I'm consuming a function from another function, but still is just the tip of the iceberg. So our export container function calls build and then publishes it to GHCR and that's it. Our build function because this is just Go code can use just Go code. We can just use format dot print to

8:09 Multi-Stage Build with Dagger

8:17 stick something into the console or logs. It's just that simple. Now because this is a mono repository, you're gonna see some funny little bits of lines of code here where I use runtime caller to get the current directory of this specific source file so that I can pluck out small parts of the mono repository to consume them in other parts of the system. I'm gonna skip over it for now but if you want to learn more, ask questions or just find the code, the links are in the show notes below. Once we have the source directory

8:44 for this individual service, we can use the Dagger client to get a container. We're gonna start with Golang latest where we mount our source directory to slash source. We set the working directory to slash source and look, we run a go build and that's it. Now this could be done with a Dockerfile. Sure. And can Dagger use a Dockerfile? Yes, it can. However, when we do it as code, it just becomes a little bit more maintainable and composable. We can reuse these step definitions. I can actually stop here, assign this to a variable like build output

9:21 and then do something else with it multiple times. Or maybe I run go build here for one architecture but then run it again for a different. I'm not gonna cover multi architecture right now because Dagger cover that really well in the documentation. So remember, dagger.io. Check out the docs, check out the cookbooks and the guides. Now, we all know that multistage builds are how we should be shipping things to production. We don't want to ship the entire Golang tool chain to a production image. So let's get a new container from Ubuntu 22 where we grab a file from the previous

9:56 build. You can see here we're seeing web fail store to slash entry point and what we're gonna store there? Well, it's the output or the fail output from another build, notably our build output where the fail is slash source slash back end which is the fail we built. We can then set the entry point and wipe out the default args. This is now an image that we can publish as we do with the export container image function. Voila. There's no other Dagger code here other than these two helper functions next to the back end. That's just for today's example and not

10:32 necessarily something you would do in a real application, but we'll cover that in another video. So for today's example, all of the code or the build pipeline that we're going to work with lives inside the build directory. If we open our test package, we have a main dot go. This just means that I can run main dot go here to test the service oriented architecture, which I'm pushing with that definition. This three tier application with a single endpoint. Consider this like an acceptance test. Okay. So the first thing we have in here is our main function. This just allows us to

10:37 The Main Dagger Pipeline (Test Harness)

11:06 run go run on this file and run the entire test harness. That is just handing off to a build function which creates a Dagger client. Now we know that our back end service has a few inputs that are required for it to run successfully. Notably, it needs an OpenAI token so that it can speak to the chat GPT API. Here we have a dev tool dot get secret. This is a helper function that I provide in a dev tool library that can be used anywhere in their Dagger pipelines. It takes a secret reference which is a

11:20 Using Secrets

11:40 string and returns a string or an error. The error being if the secret reference could not be resolved. It's important to note that we do not pass the Dagger client into this function. Why is that important? I'll get to in just a moment. So let's pop in and take a look at get secret. It takes the string and returns a string. We can see that it's actually just executing a command on my host. That's why it's important that there's no Dagger client. Why is that even more important when it refers to secrets? Because we execute this command on the host

12:15 as me. I am ambiently authenticated as me to speak to one password. In fact, when the op read hits my host, you've seen that it popped up on my watch for me to approve. It's important that this is not happening in a container context because the minute it does, we have that potential that our container or CI cache could have sensitive information and we want to minimize that risk whenever possible. So our get secret speaks to one password on the host, returns a string. So of course, once we have this string here and assuming there's no error,

12:52 the first thing we want to do is to sanitize this in a way that we can use it in our container context without it being leaked into the build cache or even out in the logs. So you'll see here that we call client a Dagger client dot set secret where we say that we want to name the secret OpenAI token and the value is our value from one password. This now means that we have enough guarantees that this will not end up in our logs and we'll do a demo to prove that before the end of this video. Next,

13:23 Setting up the Database Service Container

13:23 we call dev tool dot get database. This is a function that is going to return our production like configured postgres instance so that we're not using SQLite in our test harness. Again, trying to minimize the variance between production and test. That get database looks like this. This does take a Dagger client, so we know there's a container involved and in fact we get a container, we start with Postgres 15 as a base, we set the environment variables that we know need to exist in order for this container to run healthily, which is the user, the

13:59 password and a database name and because we're going to use this in our Dagger context, we're just gonna say that this exposes port five four three two so that when we run this as a dependent service, the networking will work. Now it's important to know that this is code. Right? Dagger pipelines are code which means we can do code like things. It may be that you don't always want Postgres 15. It may be that you don't always want to just support Postgres. Perhaps as an organization, you're starting to test MySQL, CockroachDB, MongoDB, your toolchain, you decide. But you still want

14:14 Composability & Extensibility in Dagger Go Code

14:36 to provide helpers that allow people to test their applications with good velocity. So obviously, the simple approach would be just to say, you know, get 14 and then we have a video. Right? But there's a lot of shared code between this and again, because we're using code for our pipelines, we can do code like things. So instead of just duplicating the code, why don't we do struct config DB version is string. Now I could have called that post credit version and I was about to, but we could prepare for the future. We may support multiple drivers at one point. We

15:12 can then say that this 14 is actually get database with a config where we have a config. That now means that this Postgres can become a formatted string like so. And we're formatting this with config dot db version and then we save that to resolve the format import. Now we still have a lot of duplication and again, you've got experience of writing code, I've got experience with writing code. We have patterns that we use for composition. We have patterns that we use for maintainable code. Those all apply to your Dagger pipelines. But we still have a lot of duplication.

15:50 So what we can actually do is remove that and just say that actually, we're just going to call get database of config with a default config where that database version is 15. Now this is backwards compatible, it's maintainable, and it's extensible. We can now modify and expand our config, making sure the default case is always what was the default case, but have a function that can return whatever we need. And that is pretty cool. So now that we have our database, we can start to build up the services that we need to run our test harness. You can see

16:18 Configuring Test Harness Services (Backend & DB)

16:24 here that we're pulling on something called backend builder dot build container image. And if you remember looking at the Dagger code for each service, the backend service, it had a function called build container image, which we're now using. We're then enriching it with a database URI and we're setting this to postgres postgres postgres database five four three two postgres. We're just setting up based on the information that we have from here. Now this could get smarter yet. Again, it's code and you can do code like things. What if the return type here was actually a database

16:25 Step Composition

17:00 credentials which had a host, a username, a password and a port. We can now start to consume all of these details here and remove the hard coded strings. Now I'm not gonna do all of this today, but hopefully I'm showing you that again, Dagger, code, maintainability, extensibility and composability all give you a lot of power to treat your Dagger pipelines like real software projects, improve them, maintain them and make them faster. Once we've configured the database UI, we pass in our OpenAI token. Now because this is a secret, we're passing this in as web secret variable.

17:40 Why are we doing this? Because we never want this to leak. I'm feeling bold enough that I'm gonna go to our back end service and paste in a printf. The prints are token to the log output. Now because we mount this end to the container as a secret reference, it should be redacted in the log output. We then say that the port is eighty eighty and we expose this port to be consumed by the test harness container and of course our backend has a dependent service. So we're going to do web service binding under the DNS host name database to

18:14 the database container. We're just connecting these two together so that our back end can make requests over the DNS name database to the Postgres instance. And then we set a web exec as empty which just means trigger that entry point that we configured production image. Now I am using a little bit of runtime caller just to get again a specific subset of our mono repository. In this case, I'm grabbing the test dot hull directory or the directory that contains the test dot hull and I'm mounting that into a container which is using the hull three point o image.

18:16 Setting up the Test Harness Container (Hurl Runner)

18:50 This has a back end, this has a dependent service too. The hull test suite needs to be able to speak to the back end over the DNS name backend. We've seen this in the hull file itself and the workaround and the just file for local execution. We mount the directory, we set the entry point and we pass exec which is the command in the Dockerfile or the arguments to the entry point. Now because of these can pass or fail, we grab the exit code and we grab standard error. I could grab standard error but I don't need it in

19:15 Running the Tests and Reporting Results

19:25 this particular instance. As the exit code is not zero, this means that a failure happened, failure occurred. So let's print out a red goblin or devil, whatever that is. With test fails, we'll print the exit code and we'll print the error output. And if things worked successfully, green tick and passed. So we're going to run this again. What we expect to see is a redacted secret inside of our log output and our test passing using two dependent services powered by Dagger. Now because this is Go code, I can and we will in a moment run the test main dot go as a

19:43 Running Dagger Pipelines

20:02 straight up file. However, only feedback you're going to get from Dagger is the format dot print lines that you do unless you configure the debug on the Dagger connection. However, there's an experimental UI which you may remember as set inside the ENVRC. To use that experimental UI, we can do Dagger run, go run build test main. When I run this, we're going to see the complete graph of all the steps that Dagger is going to run and whether they're successful, whether they're in progress or whether they have failed. So the UI should pop up and right

20:44 away because one of the first things that we do is a host call to get my password. We can actually see the only step in the DAG right now is the go run. There are no container executions at all. Let's approve that and now we're pulling Golang latest and Postgres. A lot of this will be cash because I've run this demo many many times. However, we did make a modification to the go binary which may trigger the build. So we're gonna give that just a few seconds to finish. Okay. So our test suite has now

21:15 Verifying Secret Redaction in Logs

21:15 finished. If we go to the very top, we'll see that the build has passed. But what we're interested in seeing is a redacted secret and the dependent service output. So let's scroll down to the dependent service where we have the entry point here. And as you'll see, we have our output of API token is here where it's just replaced with three stars. Sweet. Now the Dagger UI is great, you can use that as much as you want. It's great for debugging observability into what's actually happening now with part of the pipeline. But of course, because this is

21:40 Running the Pipeline via Go CLI

21:49 all just go code, we can also just do Go run build test where build test is the module name of where my test runner is and the main dot go. I run this like this. When it finishes, all we're going to see is any print statements that I have such as build a container image And the last print statement will either be test pass or test fail. Enter test pass. Of course, never trust a test suite that passes twice in a row, especially mine. Let's come on to our test dot huddle. Now I'm not going to change any Go code

22:12 Demonstrating a Test Failure (Fast Feedback)

22:20 or cache within the build system should still be pretty good. Meaning even if I change the test suite, it should run against the cached and built artifacts. Meaning, it should be really fast. So I'm pretty sure ChatGPT is not going to see Rawkode in response to our question. So let's run this one more time and see the tests fail. And at nine seconds as opposed to over a minute, we see that we do not get Rawkode as part of our response. Nice. So that's it for this video. Hopefully, you have a good taste for the things you

22:54 Conclusion & Key Takeaways

22:58 can do with Dagger now. The key takeaways are one, Dagger is awesome. You can write complete CICD pipelines and whatever programming language you want. Taking advantage of all the years of experience you have of writing code and applying them to your build system. We can pull in secrets. We can have dependent services. We can have a CI system that is truly composable. And that is worth its weight in gold. So go check out Dagger now. Remember to check out dagger.i0 for all the latest news and announcements as well as some great documentation. I'll see you next time. Have a great

23:31 day.