1:01 Introduction and Episode Goal

1:01 Hello. And welcome back to the Rawkode Academy. Today, we are following on with our Fermion Spin course. And in particular, we're following on from the last video where I showed you how to deploy the Fermion platform so that you can actually run your own spin applications on your own cloud provider or hardware provider of choice. Today will be a a quick episode where we take a look at an evolution of that architecture. So in the first one, we jumped through the Fermion installer repository where we used the Civo cloud provider, so I contributed by SIAM, to spin up

1:41 some Civo virtual machines and deploy everything with cloud in it. Well, I've actually contributed or at least it's in a PR just now to the installer repository with some code that allows you to do this with Equinix metal. Now why would I do this with Equinix metal? Well, one bare metal is more fun. I always like working with loads of loads of cores and loads of RAM and super chunky gigabit network cards. But they also expose BGP to you, which means we can do some really sophisticated routing at the network level when we have more

1:58 Why Equinix Metal? (BGP)

2:21 than one family on platform. Now you've probably seen me chat about BGP before. I'm a huge fan. I think it opens up infinite scale. Well, you know, that's a scary word to use, but it opens up certainly different scaling avenues where you can provide low latency applications to your customers. So we're gonna take a look at that today in the context of Fairmain's platform and web assembly. So let's share my screen. Hopefully, here we go. And this read me, and you can find this and currently a pull request at github.com/firmion/installer. This is the Equinix metal directory, which has

2:48 Equinix Metal Prerequisites & Setup

3:11 a read me. It tells you everything that you need to know. Essentially, you need an Equinix metal account, and you need a user level API key. We need a user level API key because project level API keys cannot change the BGP configuration. So you all have to use a user level key. And you must have BGP enabled for your project. So what does that look like? Well, if you come to project settings, nope. If you go to networking BGP and I'll zoom in. You can see here this is already enabled like so. If you don't see this, what

3:48 you're actually going to see is a big blue button that says, hey, do you want to enable BGP on this project? So as long as you see something that looks like this, you are good to go. Now you need to export your API key as metal off token as we can see here. This uses a bunch of defaults as part of the automation. One is gonna create your instances and call them fermion at least as a prefix dependent on the metro that we deploy to and it's going to use a c d small x 86 instance.

4:20 This is gonna cost you about $1 per hour. So remember to shut it down if you're only playing around and not deploying this to production. We also allocate one global IP address which allows us to use BGP. Next, all we need to do is CDN to the Terraform directory, run Terraform and Terraform apply. Now, let's see if it's that easy. As you can see, I'm in the Equinix manual slash Terraform directory and I'm going to run terraform edit. This has been done plenty of times while I was testing those pull request. So this should be nice and fast. If

4:35 Running Terraform Init and Apply

4:58 I just finished, I'm a meeting a little bit longer for you as it pulls down the providers. And I see Russell has joined us in the chat. Hey, Russell. I was a player. So now we can run Terraform apply. And well, spins up. We'll quickly take a look at the code and what's actually happening. I thought to be honest, it's really, really easy. Well, easy is a bold word, but it really is. From here we need an EquinixMatter project ID to get this. Just go to the console and this massive UUID that you have after projects is your project

5:24 Terraform Code Explanation (BGP Setup)

5:34 ID. You can also get this from project settings where you can copy it like so. We paste this in and before I had this, let me change one more thing. So and variables, we have the metros that we're going to deploy to and I'm gonna make this a bit bigger. This is smaller because I was on my laptop earlier. Here we go. This is our metros and you can see here I'm going to deploy it to D E which is Dallas and AM which is Amsterdam. And I'm actually gonna turn off Amsterdam for now Because I wanna see this perform

6:16 not slowly, but I wanna see the latency kind of well from The UK to Dallas, which is a bit more than from The UK to Amsterdam. And then we'll deploy that second Metro. The BGP advertisement will then start written or traffic and we'll get faster fairway on platform, which is the goal for today's session. So we can hit return on this ID. TerraForm is gonna go bloop bloop bloop. Oh, it's in changes rather than no. It's okay. It's ads. I know it's let's run that again, but now that I've saved the change. Hopefully, we don't get

6:50 alright. This is in a currently broken state. So that's gonna be absolutely perfect, doesn't it? Let's run a terraform refresh. Really, did not clean this up at some point. I'm gonna try again. I'm just gonna let it go. This may fail. We're gonna find out. So what happened is I think before I had COVID last week, so I was off all week, which is why the stream moved to today instead of last week. But I think during my testing, I've left up, realized that while I was sick, went to the Equities Federal Council, shut down all the instances,

7:27 and now we're in this position that we see today. But I don't think that's a big deal because we can see that we are getting the Dallas instance spun up and hopefully TerraForm will just clean up the state fail for their missing resources. Well, that spins up. It does take around one to two minutes. We can see what this code does. Now this is really standard terraform that grabs the reserved IP address from the Equinix middle API. We're just doing a little bit of juggling with the values so that we can get a global IP and a global IP with

8:01 the cider. This is needed for some configuration aspects, not important right now. We have the IP attachment which we do for each device and each metro and then we build our cloud and a configuration. So let's see. This bit here is already provided for you by Fermion. This is what all of the cloud providers use. However, you'll see that I've kind of thrown in this bootstrap component which just sets up all the stuff that we need for this platform to run with BGP integrated. And that just means installing and configuring Bert on the on the Linux hosts so that

8:44 we can get that BGP advertisement. Nothing else in here is particularly different from any other provider. This is just standard fair mail platform Terraform. The bet that we've added to make this different is to set up BGP script. Now we just make sure we have JQ installed because we are doing a little bit of JSON juggling and the fact that we pulled down the Equinix MERL metadata which is a JSON blob that our Terraform throws into it for us. We sleep because sometimes this metadata isn't there until the BGP is turned on for the device, which is why we then repeat

9:23 until we have access to a key called BGP neighbors. Once we have the BGP neighbor information, it means that we can configure BART and even add the BGP address to the loopback device on the host. Now if you're not familiar with Linux and Linux networking stuff, don't worry about it. We're just really making sure that this IP address will always route to one of our Linux machines and that that machine can actually respond for the IP address. We configure some default routing for the BGP peers and then we install BERD. Now again, I'm not really doing anything manually here. The

10:03 team Equinix metal formerly packet host, which is why this is packet host repository wrote a Python script published as network helpers and as long as you can just kind of pep the install and then run configure, it will actually speak to the Equinix metal API, get all the peer information, configure the well, just configure the entire bird instance so that you don't need to do anything. And in fact, the only thing we do do is just restart it, which is nice. From there, I did have to modify the standard fermion platform nomad jobs. So you'll see here, I am waiting for

10:34 Modifying Nomad Jobs (Patching)

10:42 the file provisioner, which throws all of the nomad stuff, haplo, family on platform, bundle, blah onto the root directory. And once we have that, we're actually patching it. So family on platform isn't really designed to lessen on like multiple not multiple IP addresses, but you know, each Linux machine has its IPv4 address and then the global IP address is different from it. And we actually want some stuff to listen on the machines IPv4 and not the global IPv4. So we have to modify the traffic instance, which is a proxy set in front of all of the fair mail platform components

11:26 and just say, hey, actually, we have the domain that we're configuring on a host level, but also we need to add on our BGP IP address to. Yeah, that kind of makes sense. Yeah. So instead of just saying, hey, listen on this, we need to add in the HIPAA host and we do the same for bundle actually, which is done here. Let me see here. And this is just so that we can when you're creating the spin applications deployed to the platform, you could use the BGP address and host name, and it doesn't matter which one you hit. But if you want

12:06 to get to the hyper UI, then you you really wanna have control over that. So we're specifically saying I wanna be able to request this on the local IPP for address. And we do the same for bundle and that's it. That's the only two modifications that had to be made to the entire stack, which is just as time for this to pop over here and hopefully be healthy. Well, let's go back to our read me. We're not fussed about the TLS configuration or anything like that right now. What I actually want is did they not put this in the read

12:25 Retrieving Platform Access (Hepco)

12:56 me? No big deal. What we actually want is the Hepco password. So we can run Terraform output, which will spit out all that stuff. But what we want to do is request a specific value, which is our heppo admin password dot value. No. We can do that with JQ. That pulls out a JSON blob. This is then JQ value and we wanna raw or put quotes though. Let's try again. What does that give me? Oh, it just gives me a password. Never mind. Let's copy this. And then we can come up here and copy the hippo address.

13:53 Accessing Hepco UI (Initial Attempt)

14:08 So Please work. Maybe it's not ready yet. Alright. When in doubt, we do debugging. So the other thing that we have access to is the SSH private key. So let's do Terraform, I put private key and write this to prev and then we open prev. Make sure we only have the private key. Change the permissions on it to be four hundred. And then we want to get to the IP address of the machine. And then the fastest way to get that is probably go to here and see manage servers. This is the IP address for the machine and

14:27 Debugging the Deployment (SSH & Logs)

15:05 not the BGP address. And we can connect to the instance assuming well, all good. And then tail bar log clouds and output. So you can see it's looping because it's not got the BGP information. And I don't know why. Let's take a look at our host. Go to BGP. Actually looks good. Let's run this manually and see what the problem is. Okay. Let's see. Interesting. This resource here is our BGP session for the machine. And I'm going to assume that maybe it wasn't added, although it does appear to be added. Oh, I think that's it because I have

15:24 Identifying Cloud-Init Loop (BGP Metadata Issue)

17:15 everything in such a weird state. Damn myself. When in doubt, we're on a terraform destroyer. And a tear form apply. That does mean that we will have to wait approximately a minute and half to two minutes for this machine to come up. Oh, no. What have I broken now? I really hate having the debug stuff live. I think my project's not found. Oh, wait. What's this? Schedule maintenance. Yeah. That's not gonna bother me. Right? Well, my project ID ends on 132. Oh, this is not the project it's using. What was I pasting? I have no idea.

18:14 Correcting Project ID and Reapplying

19:19 It's always me though, isn't it? Alright, so that's just gonna take a minute or two now. And what I'll do is as soon as this is spun up, as we'll just go straight into that HEPL UI, deploy our spin application, and then we'll quickly add that second one to the Amsterdam region where we'll see the latency drop by hopefully around three hundred milliseconds. Good evening. Welcome to the stream. So we can see the failed provisioner is now running. This is the one that puts Fermion installer no more jobs on to the machine. All that fun stuff.

19:26 Waiting for Redeployment & Plan

20:17 Let's grab IP address here BSSH. And we'll tail a cloud and an output where we can see currently it's on the g q install step, which is good. It's now trying to get the metadata. It's already got it and now it's proceeding to provision correctly. No rustle. No worry rules. Hopefully, no more breaks. So recovering from the COVID. Be nice. Be gentle. Happy testing negative for four days. That's Friday. Three days. I'm still tired. I'm very tired. Alright. We're almost there. Now it's all a nomad console vault. These are all the components used to give

21:27 us a fair mail platform. Bundle, which is where it distributes and shares or spin applications. Hepoo which is the kind of orchestrator component. Nomad which runs all of our jobs. Sweet. This is looking pretty good. So I'm gonna pop back off and we're gonna do a terraform output. We want the hippo admin password. We also want the URL, which is here. So I wish the Terraformer could go to the BGP one, but I want you to bypass it and just go directly to this machine because Bert may still be advertiser may not have actually advertised the address yet. But you

22:28 Accessing Hepco UI (Success)

22:39 can see that we did get the thing. So now we can grab the password and we can do admin password. Perfect. Now we have a platform where we could deploy an application, but we're not gonna do it this way. We'll do it from the CLI. So we need a new tab I'll approve this. And we need a spin application. Where did I put my spin application? This is what happens when you disappear. Listen demo. Alright. So now we can do a spin log in and we provide a URL, which is HTTPS, hippo. Password. Let's copy this.

23:03 Spin Login to Fermion Platform

24:03 Like so. Then to log and suspend application, Russell's like, woah, ASP dot NET Core. Yeah. Hippo is a days lab project, which I believe is mostly dot net. So we're not gonna send them a gap because we haven't configured it as to do any gap application client nonsense. We're going to use a new password for we see the heppool username admin. The password is that one that I've now lost all access to. So let's scroll back up. Like, so the bundle URL is the same IP address that we use for the hyper URL, but with bundle start.

24:46 So that's 1477535151 and it does need v one on the end. And there's no authentication and now we're logged in to our bare main platform. In fact, you can just spend deploy. Why? Not trusted. Can I skip that? Oh, wait. Why is that HTTP? Try again. I wonder if it's supposed to be HTTP and I've got myself into some weird mesh mesh mode. Yeah. So it's bit not. I'm just going to buy by it, bypass it. Oh, yeah. Because why would it be HTTPS because I'm using a slip. We're not using a real domain name.

26:13 Debugging Spin Deploy Failure (Curl & Patch)

26:25 Alright. It should endo one four seven five. Should really just be copying this from now on, but hopefully it's the last time. Spin, play. What did I get wrong? Let's see what happens if we curl this. 404. That happens if our patch doesn't work. I am never going to do a demo after being elegant. Let's jump onto this machine. We can run nomad status. Bundle is running. We'll check our logs to see if there were any errors. Oh, it looks perfect. We can even use this to see console. Yeah. Bundle's running. But it's not got both the hosts.

27:43 SSH Debugging: Patch Failure Confirmed (Dallas)

28:29 So our patch job. So if you've ever seen dot reg dot reject fails before, this is what happens when a patch doesn't work. So we're gonna do it normally. We're gonna do it manually. Manually, normally. Yeah. And why it didn't work? I have no idea. I'm gonna fix that later. Let's just copy it from here. Get rid of you. And we don't need to double dollar sign. That's just to get past the terraform interpolation. But that looks good now. And the last thing we need to do comes from comes from the Fermion installer where we have

28:50 Manual Patching of Nomad Job (Dallas)

29:43 to actually tell it to run the job. And in order for this, this is all getting very complicated now and you shouldn't have to worry about this. But I feel like I'm committed now. If we go to var var log cloud and output, scroll to the bottom. We need to copy this, export them to the environment, goes to the Fermion installer, grab the share Terraform VM assets, run servers and then here there's a nomad command to run bundle. Like so. So we need to run this to tell to recreate the bundle job. And I'm in the job directory. So let's

30:36 just pop up. Oh, it's looking for that list encrypt. We can just set that to false. Now restarting our bundle job, which is done. Alright. Thank you for bearing with me. My curl command is on this tab. And now we get a four zero four. So that was gonna be it. Let's find. Let's Let's see what we see and our services because we wanna see yeah. So anything on bendle. Should resolve to the bendle server. Maybe that 404 and the v one is fine. Maybe this Maybe that's a prefix. Not entirely sure. Let's just be bold.

32:18 Spin Deploy Success (After Manual Fixes)

32:31 Yeah. Russell only one of the patches failed. So Russell was asking there's two changes. Your setup script, you only made one in the thing on the terminal. And because it was only a thought a wrench and dot rejection for the bundle patch. I'm not sure why this failed to patch. Normally happens if you get spaces and tabs wrong, but all these dots here to suggest I'm using spaces which is fine. I will fix that. Something silly but I only had to apply the one. But as you can see now, it did deploy. We can call our application and we get

33:12 Testing Deployed Application (BGP Address)

33:14 hello fermion. This is good. We could benchmark this if we want. I don't have drill installed. Doesn't matter. Yeah. Throwing me for six. If you refresh console, let's use our application here, the random UUID. If we pop this out, go to here, change the path so it's not apt new. We now have our demo application. So we're now deploying to the fermion platform. We could run an MTR. Why have I not got any cool commands? This is just also cool commands. MTRs like a tracer drill is a HTTP benchmarking tool. We'll give them just a wee second.

33:16 Benchmarking Latency Attempts (Drill & MTR)

34:29 So we can run drill our IP address. Maybe I was wrong. If you're on drill again. Oh, you need to fail, don't you? I've forgotten how to drill rust drill. It should be. That's a bit more complicated than I was hoping for. It should be a request latency. Test my I don't test my latency. I have forgotten how to computer. This should be latency test. Solar runs. I don't know if I trust you. I want to test my ping. Oh, whatever. Let's see. Russell, how would you test the speed of an h two p request? Come on. Tell

36:08 me. HTTP test speed response. I mean, how hard can this be? Alright. Test. Cool. Tested from Europe. London's close enough. Well, that does its thing. What is m t r, the BGP address? So the BGP address is from our terraform output. This one here. If I go ahead to MGR. No. MGR host name. Oh, why am I gonna protocol on there? Oh, I hit me again. MTR requires to do. Finally, I swear I'm good at this stuff. Let's let them try running. Because you hear the latency for our from London to our application, three hundred milliseconds. I'm gonna let the MTR

36:32 Measuring Initial Latency (Dallas)

37:58 finish before I lose my mind. We're going to add that second region very, very quickly. And I'll need the project ID. And yes. Cool. So we've seen the latency from London to Dallas at three hundred milliseconds. Our MTR is working on it. We never actually tested the BGP route, did we? Maybe this is going to go into the great unknown. That if we copy this. No. This demo. There's our application. Yeah. That's the BGP one. Nice. Cool. So I don't know why the MTR is taking so long actually. Normally that'd be showing us the route.

38:57 Confirming BGP Address Access via Browser

39:19 Let register the IP address this time. The reason it failed was I didn't do it the first time. That's the other thing. Now our second instance is now up, which is good. What we can do, which is the regional IP output here to go on to Amsterdam. We can follow the progress of that deploy. It's working on it. MTR is working on it. The Carl's fine. Things are happening not necessarily all going well. Let's see. So once the Nomad jobs for the Amsterdam One deployed, when I do a curl on an MTR, it should rip to the Amsterdam

39:31 Applying Terraform with Second Metro

40:06 instance instead of the Dallas instance. We should see a latency, hopefully no more than about fifty milliseconds. And this was supposed to be a twenty minute stream and we're forty minutes in, but you know sometimes shit happens. As Russell knows all too well. Alright. Bundles on its way. No HEPL. What's going over that MTR? It's usually super quick. Alright. Almost there. Now I do need to log in and deploy the application to this instance. Now the BGP advertisement will be working. And so we probably see a photo for now if we were to curl. Nope.

41:07 Demonstrating BGP Routing (Dallas vs Amsterdam)

41:07 So you can see here, we got hold fermion and there we got a four zero four. So you still have to deploy your application and that means you have to log in to your application. And for each instance, we can drop it first from terraform terraform. Output. We want that regional IP again where we can log in .sslip.i0 to admin. We need the password. No. Don't get the course this time. We need the bundle URL. 14540. No. 754012SSlipIOB1. I wonder where that's been login goes. Can I change that? No idea. Like not to mess it up this time.

41:34 Spin Login Attempt (Amsterdam IP)

42:43 Bundle. Empty hours gone. Don't know what's happening. Let's get rid of it. When that's deploy finishes, we'll do the Carl. Hopefully it's not $4.00 4. Although the bundle configuration is not gonna work on this machine either, is it? And I just use the BGP address. We'll get there. Any minute now. Two admin password. Bindle, not BGP. We want this. Course that doesn't work. And I'm gonna type this and I'm gonna have to go into that patch again, aren't I? And it's gonna feel five. Resource not found. Easy job. We have a rejected fail. End of up normad.

44:46 Manual Patching of Amsterdam Nomad Job

44:46 Copy. Improve the extra quote dollar thing. Words. Cat bar log cloud in it. Get all of this. Go back to here. I closed it. This one. Change the list of crypt to false. Course, that's not gonna work. Good to uninstall my computer. Just never fit on it. I think it's time for it to go away. So this panel will work. The job will be replaced. It has been replaced. Do a deploy. Twenty minutes stream, my ass. Anyway. Just sort of giggles or on the pseudo mtr. Don't know why those are just therapeutic. It's during that Carl command back.

46:27 Testing Latency Again (Both Metros)

46:49 And we get hello from you. The MTR is probably not gonna ever work. So we're gonna go back to this wonderful random website that we found. And I wonder if I can just refresh the page. No. Run now. Now let's see if it's any faster. Give it up an empty hour. Well, this should not be a fast test, a slow test. So now it's a hundred milliseconds instead of three hundred. I should expect it to be faster, but I am not going to try and push this hodgepodge of automation any further because clearly the odds against anything working today.

48:10 Summary and Conclusion

48:20 So what is the take home from today's session? Things go wrong, granted. Firming on platform is open source. Right? There's nothing particularly special about it. It uses Nomad, Vault, and Console as your building blocks for a distributed orchestration system against multiple hosts. I took the easy approach today and deployed unique ones to each metro. Of course, there are more advanced architecture there. Right? Nomads can scale horizontally. We could have used the same Nomad cluster on multiple machines. However, that's a session for another day. Once we have that substrate to build our platform, we're deploying Hippo and Bundle from data

49:04 slabs, which allows us to run web assembly applications. And we also get that nice Hippo UI where you can see I could log in. I could see my applications, it gave me the URL, it gave me the logs, it gave me the environment details, all the components that you've come to know and love with Fermion Cloud. But we ran it ourselves. And using Equinix medal, we were able to be able to distribute this geographically. I only use two metros, Dallas in The US and Amsterdam in Europe, but you could do that to wherever your customers are to deliver them via BGP written

49:36 low latency applications. And that all coming together, I think is really cool. Now clearly the automation needs a little bit of work. I hope that by me debugging this on the go. You cannot understand a little bit more about what was actually happening. But of course as always, if you have any questions or you wanna do something slightly bespoke, just jump straight into the comment section. Ask away. I'll do my best to ask this. What's Russell's last comment before I sign off for today? If you make changes on the fly, does it become get ignore ops?

50:14 Responding to Chat & Final Remarks

50:14 This is why I've been doing more pre recorded video lately because then you never see my mistakes, but then it's nice to have fun and say hello to y'all while I do this. So go check out family on platform, Check out the pull request if you wanna play it with an equest metal. However, there are there is TerraForm code to deploy to AWS, GCP, SQL cloud, and many others. So go check it out. Thank you all for joining me today. Thank you for persisting with me if you're still here, I'll see you all on the next session.

50:43 End

50:43 Have a great day.