0:00 Introduction and Course Overview

0:00 Hello, and welcome to my latest course at the Rawkode Academy. This is Portainer in production. I want to say thank you to Neil and his team at Portainer for kindly sponsoring my time to produce this course. So what will this course entail? Over the next few weeks, we will be diving in to how to run Portainer in production to manage real production workloads. We'll be taking a look at some interesting features of Portainer, such as fleet management and GitOps, ditching Kubernetes, for simpler orchestrators like Nomad and Swarm, multi cluster management, securing your workloads, and even a live hacking experiment where I

0:52 attempt to add OCI registry support to enrich the GitOps capabilities of Portainer. I hope you are as excited as I am for this Portainer and production course. Today, we start with our first tutorial, which is to take an existing Portainer installation and look at the different ways to manage the x five zero nine TLS certificates. Let's have some fun. So when you install Portainer for the first time, you're presented with this screen. This is a vanilla e c two box running on AWS with a docker installation of Portainer. By default, TLS runs on nine four four

1:26 Default Portainer TLS and Browser Warnings

1:43 three, and Portainer is very kind. It generates some self signed certificates so that you can get started right away in a secure fashion. But, of course, as we all know with self signed certificates, our browsers can't verify or trust the route. So you'll often be presented with this screen which says your connection is not private. Now, of course, we can click on advanced and say proceed, and this is enough for most. Of course, we can improve the situation. In 2022, we have access to let's have free TLS certificates all day long. So in this session, we're gonna take a

2:27 Exploring Alternative TLS Methods (Certbot, Caddy)

2:31 look at how we can use cert bot on the host to provision the x five zero nines and have Portainer mount them in to the Docker container to replace the self signed certificates. But that still requires a little bit of manual work. So we're gonna take a look at a second option too, which is to run a reverse proxy in front of Portainer, which can handle the let's encrypt negotiation, the let's encrypt dance for you. And for this, we'll be taking a look at CATI. There's no right or wrong answer when it comes to using

3:07 either of these methods. If you are happy and comfortable with Portainer's cell sign certificates, then go nuts. If you don't mind doing a little bit of plumbing and running cert bot on your host, fine by me. And if you wanna take the extra step and configure a reverse proxy that manages the Acme dance for you, then so be it. So let's take a look at the different ways to manage our x five zero nines for Portainer. So the first thing we're gonna take a look at is the settings page. From the Portainer homepage, if you click on

3:33 Portainer's Built-in TLS Settings

3:41 settings and scroll down, you will see the SSL certificate settings. If you really want an extended validation certificate, an EV cert, and you wanna go and pay $500, this is where you upload it. You select file for the private key and the certificate, and you're good to go. Regardless of which TLS method you use with your Portainer, I'm gonna recommend that you tick the force HTTPS only box and apply. You don't really want to expose any web service in this day and age over regular HTTP. So let's practice good hygiene and secure it. Perfect. Okay.

4:27 Now that we have Portainer running with the self signed certificate and we are enforcing HTTPS mode, we can look at alternative ways to get our own x five zero nine certificates into the setup. The first way that I want to show you is using cert bot. If you're not familiar with cert bot, it is a piece of software from the team at Let's Encrypt. This speaks to ACME protocol to the Let's encrypt servers and issues certificates. To run certbot on Ubuntu is relatively easy. It's delivered as a snap application, and snap is available on all Ubuntu machines.

5:01 Certbot Installation Steps

5:07 So first, we run snap install core. We then do a snap refresh core. This will update the snap package list, allowing us to install the latest version of certbot. Next, we run snap install certbot. Only we're gonna pass the dash dash classic flag. Dash dash classic just means that we're going to run certbot with access to the host and no c groups or namespace constraints. We then make sure that cert bot is available as a system command on the regular path by moving it or creating a symbolic link to user bin. Next, we run cert bot cert only standalone.

5:50 What does this mean? Well, it means it's going to run its own web server on the host because the way the ACME protocol works. If you want a certificate for a domain, it must be able to reach that server on that domain on port 80, where it will respond with a key to confirm that you actually have access and the ability to provision certificates for that domain. It's a proof of ownership or at least administration. This is an interactive command, and when we run it, it's going to ask us for our email address and the domain that we

6:18 Automating Certbot Renewal with systemd

6:26 wish to provision the certificate for. Now we wanna be able to automate the renewal of these certificates, and we don't wanna have to have any manual intervention. So we're gonna use system d timers with a system d service to make sure that once a month, the certificate is checked to see if it's going to expire. Here, we create a certbot timer, which is going to run the certbot service on the first day of every month, a quarter past three in the morning. Then we have a cert bot service, which runs cert bot renew. Simple. We do a daemon reload and enable our

7:03 timer. Now don't worry about pausing and copying or typing everything that you've seen in that file there. The code is available at github.com/rawcodeacademy/portainer-in-production. The link is there. So let's run our start bot script. First, it asks for email address. I'll enter david@rawkode.com. Yes. I agree to the terms. Yes. I'll share my email address, and now it's asking for the domain that we wish to secure. I'm going to enter p.Rawkode.academy because that is the DNS that I have configured for this machine running our Portainer instance. Let's encrypt is now running a web server on this machine on Port 80, sending a

7:22 Certbot Certificate Provisioning Demo

7:54 request looking for a file available over HTTP that confirms that I have administration's right to this domain. As you can see, it successfully created our certificate in the slash ETC slash lesson crypt slash live folder, meaning this is a real production lesson crypt certificate that is now available for Portainer to use. Now if you remember in our system d timer, we have a command configured called certbot renew. I'm got not going to run renew now, so I'm going to pass the dry run flag. But this is the command that will check for the expiration date on your certificates and

8:32 make sure that as they're close to being expired, they will be renewed for another three months. Also, if you're looking at the commands I've typed and you don't want to run this script in an interactive environment, all the arguments that I passed, my email address, those remain and a green to the terms can be configured through flags passed to the certbot command. If you run certbot cert only dash dash help, everything you need to know is there. So let's configure Portainer to use these new let's encrypt certificates. Also available on the GitHub as a script.

9:06 Configuring Portainer with Certbot Certificates

9:17 So here we have the Docker commands with the correct volume mounts to take less encrypt certificates from the host into your Portainer instance. The two mounts that we worried about or the two mounts that we are concerned about are these two here. The ones with etcetera, let's encrypt live and etcetera, let's encrypt archive. You'll see that we have the term your domain. This is in the middle and end of each of these mount lines. So you can use search and replace to find your domain and replace it with the domain you're running Portainer on. We save,

9:54 Now we can run our p dot l e script, which will run Portainer over TLS with let's encrypt production certificate. So if we pop over to the browser, hit refresh, click on unlock, you will see our certificate is issued by r three, the let's encrypt authority. Awesome. Now there's not a lot of maintenance with that approach, and you can be very successful with Portainer's own self signed certificates or let's encrypt running on a host. But I think my preferred way is to use KADI as a reverse proxy, which does automatic TLS for you. It speaks ACME to let's

10:21 Method 2: Caddy Reverse Proxy

10:40 encrypt and handles all the renewals for you. We don't even need to disable the Portainer cell sign certificates, and we can still enforce TLS at all times. It just means that you, as the user with a browser, speak TLS to Caddy via the less than 10 production certificate. But it also means that Caddy speaks to Portainer over TLS using the self signed certificate. So we have end to end TLS encrypted web traffic. So I created a small script also available on the GitHub page, p-rp.sh. Here, you'll see the complete setup for this Portainer configuration. First, I remove all the containers from the

11:10 Setting up Portainer and Caddy Containers

11:26 host. Maybe don't copy that line. We then create the volume for Portainer, which I've already done as part of configuration, so it's commented out. We then run Portainer as always. Only this time, there are no volume mounts for let's encrypt certificates. For CADI, we also need two volumes, CADI data and CADI config. From there, we can run CADI, making sure that we bind ports four four three and eighty from the container to the host. Remember, for let's encrypt ACME negotiations, we must have physical port 80 available on the host. We also need to link Portainer

12:02 Understanding the Caddyfile Configuration

12:09 with the name Portainer. This means that DNS queries from the CADI container can query Portainer as a DNS name and reach the Portainer container. We then have to mount in our caddy file, which I'll show you in just a moment to slash e t c slash caddy slash caddy file. We then mount in the two volumes we created, CADI data and CADI config. As you'll see, I'm using the official CADI two dash Alpine image. If we take a look at the CADI file, we'll see, one, we have to configure our email address. This is because emails are

12:45 required for admin negotiations with Let's Encrypt. We then set up a virtual host of p.Rawkode.academy. We have a reverse proxy block which says, hey. Any request we get on this domain, we're gonna proxy to another server. We specify the two, which is going to our Portainer on port 993, the encrypted self signed certificate Portainer. Because it's a self signed certificate, we do have to tell CADI not to verify the route. So we need a transport block for HTTP where we see a TLS is enabled. However, skip the verification step. And that is it. We can run PRP,

13:27 Caddy Method Demonstration

13:30 which will delete any existing containers, start up the two new containers, and in fact, we can run Docker container l s. We'll see that we have caddy and Portainer both running. And now if we do a curl with a dash dash v to p.Rawkode.Academy, we get a whole bunch of text back. This is a good indication that things are working. Let's pop back over to the browser and refresh, and you'll see that we still have a happy lock with a certificate issued by r three from Let's Encrypt. So that is how you enable and secure

14:08 Summary of TLS Management Methods

14:10 your Portainer instance three ways. One, using their self signed certificates. It works as long as you don't mind clicking accept or advanced on that little warning page, you're good to go. Two, cert bot on the host with a system d timer to handle renewals. And three, reverse proxy with Caddy. They're all production grade. They're all good to go. Enjoy. We'll see you next time.