1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:15,040 --> 00:00:18,240 Okay. So, get excited for our next block 4 00:00:18,240 --> 00:00:21,840 of talks. We'll have Luke and I'm very 5 00:00:21,840 --> 00:00:24,320 excited for their talk. Please welcome 6 00:00:24,320 --> 00:00:27,510 on premises Luke. 7 00:00:27,510 --> 00:00:29,530 [applause] 8 00:00:29,760 --> 00:00:31,359 Hello, thank you very much for that 9 00:00:31,359 --> 00:00:33,280 introduction and yes, I'm going to talk 10 00:00:33,280 --> 00:00:36,160 to you about from on-prem to the cloud 11 00:00:36,160 --> 00:00:38,719 and back again or self-hosting for fun 12 00:00:38,719 --> 00:00:42,000 and profit. Um, what are we doing today? 13 00:00:42,000 --> 00:00:43,920 I'm going to give you some introduction 14 00:00:43,920 --> 00:00:45,680 about what we're talking about. I'm 15 00:00:45,680 --> 00:00:48,000 going to say what it takes to build a 16 00:00:48,000 --> 00:00:50,399 platform. I'm going to do a comparison 17 00:00:50,399 --> 00:00:53,360 of cloud and on-prem. And I want to give 18 00:00:53,360 --> 00:00:55,280 you some concrete advice for 19 00:00:55,280 --> 00:00:57,280 self-hosting. And I'm hoping to leave 20 00:00:57,280 --> 00:00:59,280 lots of room at the end for Q&A because 21 00:00:59,280 --> 00:01:02,239 I'd rather I'd rather answer the 22 00:01:02,239 --> 00:01:03,600 questions that you actually have rather 23 00:01:03,600 --> 00:01:06,080 than what I've imagined you might have. 24 00:01:06,080 --> 00:01:08,159 Um, who is this presentation for? 25 00:01:08,159 --> 00:01:09,840 Everyone who's here for the platform 26 00:01:09,840 --> 00:01:11,360 engineering track, no matter your 27 00:01:11,360 --> 00:01:15,680 experience level. Um, if the title has 28 00:01:15,680 --> 00:01:17,520 made you think this is about migrating 29 00:01:17,520 --> 00:01:20,320 between cloud and on-prem, that's not 30 00:01:20,320 --> 00:01:23,600 what it is. It's about my journey, um, 31 00:01:23,600 --> 00:01:26,240 which I'm going to be talking about. Um, 32 00:01:26,240 --> 00:01:29,280 and but if you love the cloud, hate the 33 00:01:29,280 --> 00:01:32,159 cloud, are curious, want to know how to 34 00:01:32,159 --> 00:01:34,880 host the building blocks of a serious 35 00:01:34,880 --> 00:01:37,280 platform on prem, that's what we're here 36 00:01:37,280 --> 00:01:40,320 for. Some definitions to start off with. 37 00:01:40,320 --> 00:01:43,360 A platform is the substrate of an 38 00:01:43,360 --> 00:01:46,640 application system that goes in between 39 00:01:46,640 --> 00:01:48,320 the infrastructure, the service at the 40 00:01:48,320 --> 00:01:50,320 bottom, and the business logic, the 41 00:01:50,320 --> 00:01:52,399 actual product that in most cases you're 42 00:01:52,399 --> 00:01:55,040 selling to a customer. And so you've got 43 00:01:55,040 --> 00:01:56,560 the stuff in between that makes it all 44 00:01:56,560 --> 00:01:59,600 work. The cloud of called someone else's 45 00:01:59,600 --> 00:02:01,759 computer where I'm talking about your 46 00:02:01,759 --> 00:02:04,560 traditional AWS Asia what's now called 47 00:02:04,560 --> 00:02:06,960 the hyperscalers but also things like 48 00:02:06,960 --> 00:02:10,479 Oracle cloud digital ocean and onrem of 49 00:02:10,479 --> 00:02:12,720 course is when you're hosting your own 50 00:02:12,720 --> 00:02:15,440 servers in a rented data center also 51 00:02:15,440 --> 00:02:18,319 known as collocation. 52 00:02:18,319 --> 00:02:24,000 Who am I? I'm Luke. Um I'm a queer 53 00:02:24,000 --> 00:02:26,800 rainbow person and it is so good to be 54 00:02:26,800 --> 00:02:28,400 at a conference that is queer and 55 00:02:28,400 --> 00:02:31,440 diversity forward. Um so nice to be 56 00:02:31,440 --> 00:02:33,440 here. I've been a platform engineer 57 00:02:33,440 --> 00:02:37,120 since around the year 2015 and as I said 58 00:02:37,120 --> 00:02:40,080 my personal journey I did five years of 59 00:02:40,080 --> 00:02:42,480 on-prem work. Then I was a cloud 60 00:02:42,480 --> 00:02:44,720 solution engineer for 5 years and now I 61 00:02:44,720 --> 00:02:48,080 am back to doing on-prem. I run Arch 62 00:02:48,080 --> 00:02:50,800 Linux BTW and I have way too many 63 00:02:50,800 --> 00:02:52,480 hobbies such as amateur radio, 64 00:02:52,480 --> 00:02:56,400 electronics, running and cycling which I 65 00:02:56,400 --> 00:02:59,440 do whenever I get to it. Okay, what is 66 00:02:59,440 --> 00:03:01,519 the platform that I am talking about? 67 00:03:01,519 --> 00:03:03,840 This is an image from Wikipedia about 68 00:03:03,840 --> 00:03:07,200 how email works or rather how one tiny 69 00:03:07,200 --> 00:03:10,080 little slice of email works. Um this is 70 00:03:10,080 --> 00:03:13,440 how to send an email via SMTP. 71 00:03:13,440 --> 00:03:16,239 Um, and of course, if you are running an 72 00:03:16,239 --> 00:03:18,800 email platform, there's a whole bunch 73 00:03:18,800 --> 00:03:22,159 more to that. Um, my company, Fastmail, 74 00:03:22,159 --> 00:03:24,000 is what is called in the industry a 75 00:03:24,000 --> 00:03:26,400 premium mailbox provider, which means 76 00:03:26,400 --> 00:03:29,840 that people have their email address and 77 00:03:29,840 --> 00:03:31,840 everything that goes with that. Um, with 78 00:03:31,840 --> 00:03:36,560 us, we do this with about 250 servers um 79 00:03:36,560 --> 00:03:38,640 that were hosting in three data center 80 00:03:38,640 --> 00:03:42,400 locations across two continents. Um, and 81 00:03:42,400 --> 00:03:45,200 I had a look at how much user data we 82 00:03:45,200 --> 00:03:48,000 storing and that's about 5,000 83 00:03:48,000 --> 00:03:51,120 terabytes. So that's five pabytes which 84 00:03:51,120 --> 00:03:53,040 for some big data people might not be a 85 00:03:53,040 --> 00:03:54,640 lot for others might be more than you 86 00:03:54,640 --> 00:03:57,200 can imagine. It's it's some amount of 87 00:03:57,200 --> 00:04:00,720 data to manage. Um, I am giving myself a 88 00:04:00,720 --> 00:04:02,879 little bit of help by also counting our 89 00:04:02,879 --> 00:04:05,120 replication and our backups in that. Um 90 00:04:05,120 --> 00:04:07,280 but it's a lot of data and we do that 91 00:04:07,280 --> 00:04:11,120 for about 200,000 uh paying customers. 92 00:04:11,120 --> 00:04:15,280 Actual users is is more than that. Um we 93 00:04:15,280 --> 00:04:19,519 run about 100 individual services and 94 00:04:19,519 --> 00:04:21,759 are actually split up roughly 50/50 95 00:04:21,759 --> 00:04:24,320 between platforms between the platform 96 00:04:24,320 --> 00:04:25,919 or support services and the actual 97 00:04:25,919 --> 00:04:29,199 business services. We do five to 10 prod 98 00:04:29,199 --> 00:04:31,600 deployments every day. Um and that 99 00:04:31,600 --> 00:04:34,160 causes one to five prod incidents every 100 00:04:34,160 --> 00:04:37,840 week and we do that with four full-time 101 00:04:37,840 --> 00:04:39,600 platform staff and about five other 102 00:04:39,600 --> 00:04:41,919 people in the company who contribute. Um 103 00:04:41,919 --> 00:04:45,919 it's quite a small company. Um but um 104 00:04:45,919 --> 00:04:47,759 that's the people we have to manage all 105 00:04:47,759 --> 00:04:51,680 this. Um 106 00:04:51,680 --> 00:04:54,160 the fastmail itself has been running 107 00:04:54,160 --> 00:04:56,160 since 1999. 108 00:04:56,160 --> 00:04:59,600 Um, I've been here since since 2023 and 109 00:04:59,600 --> 00:05:03,120 we are sort of on the upper end of small 110 00:05:03,120 --> 00:05:05,919 medium business scale or the lower end 111 00:05:05,919 --> 00:05:09,440 of enterprise scale. Um, 112 00:05:09,440 --> 00:05:12,240 little bit too small to run tuniper 113 00:05:12,240 --> 00:05:15,520 hardware, a little bit too big um to run 114 00:05:15,520 --> 00:05:17,520 microic hardware, but we use microic 115 00:05:17,520 --> 00:05:19,840 anyway. It works great. 116 00:05:19,840 --> 00:05:22,960 Um, all right, one more time. what I'm 117 00:05:22,960 --> 00:05:24,560 going to do, what does it take to build 118 00:05:24,560 --> 00:05:27,360 a platform, why building a platform in 119 00:05:27,360 --> 00:05:29,600 the cloud is terrible, um what 120 00:05:29,600 --> 00:05:31,199 components you actually need for your 121 00:05:31,199 --> 00:05:33,120 production scale platform, and some 122 00:05:33,120 --> 00:05:35,120 actionable advice if you want to do 123 00:05:35,120 --> 00:05:38,160 that. All the information from here on 124 00:05:38,160 --> 00:05:40,960 after is personal experience and wibes. 125 00:05:40,960 --> 00:05:43,280 Um it's inspired by what I do at work, 126 00:05:43,280 --> 00:05:46,240 but of course [clears throat] all of the 127 00:05:46,240 --> 00:05:48,400 opinions are my own. Um and it's not 128 00:05:48,400 --> 00:05:50,720 based on a particular architecture or 129 00:05:50,720 --> 00:05:53,520 theoretical model. It's just what I've 130 00:05:53,520 --> 00:05:57,520 learned um doing this for 15 years now. 131 00:05:57,520 --> 00:06:01,120 What is a platform? Um first you have to 132 00:06:01,120 --> 00:06:02,479 start off with what's under the 133 00:06:02,479 --> 00:06:05,199 platform. The substrate um that's your 134 00:06:05,199 --> 00:06:06,880 hardware and physical layer. That's 135 00:06:06,880 --> 00:06:10,639 power cooling connectivity and then the 136 00:06:10,639 --> 00:06:13,199 actual servers in your server rack. You 137 00:06:13,199 --> 00:06:15,600 have those great all cabled up. Now you 138 00:06:15,600 --> 00:06:17,600 got to run something on there. And 139 00:06:17,600 --> 00:06:20,400 before you can run your platform, you 140 00:06:20,400 --> 00:06:22,880 need those servers to actually be 141 00:06:22,880 --> 00:06:25,039 functional. So for that, you need things 142 00:06:25,039 --> 00:06:27,039 like network configuration, network 143 00:06:27,039 --> 00:06:29,360 security, configuration management, 144 00:06:29,360 --> 00:06:31,759 identity management, 145 00:06:31,759 --> 00:06:34,160 and um you've got a little bit of a of a 146 00:06:34,160 --> 00:06:39,199 stack here um on on the right. Um, so 147 00:06:39,199 --> 00:06:42,400 all the stuff um that's in pink here, 148 00:06:42,400 --> 00:06:44,560 that's your your operate what I'm 149 00:06:44,560 --> 00:06:47,039 calling your operator operating system 150 00:06:47,039 --> 00:06:50,560 layer. Um, and then let's actually get 151 00:06:50,560 --> 00:06:53,440 to platform services. Um, you're 152 00:06:53,440 --> 00:06:55,120 starting off with your basic network 153 00:06:55,120 --> 00:06:59,280 services. Um, then you have data. Um, 154 00:06:59,280 --> 00:07:00,720 you got to manage that. You got to store 155 00:07:00,720 --> 00:07:04,639 it somewhere. Um and that actually takes 156 00:07:04,639 --> 00:07:10,080 uh some amount of of services to do. Um 157 00:07:10,080 --> 00:07:12,639 then you need to manage the management 158 00:07:12,639 --> 00:07:14,720 of all the services. So starting 159 00:07:14,720 --> 00:07:16,639 stopping services, updating 160 00:07:16,639 --> 00:07:19,120 configuration, all that stuff. And then 161 00:07:19,120 --> 00:07:22,000 you also need a software pipeline. Um 162 00:07:22,000 --> 00:07:25,039 that is often if you talk about DevOps, 163 00:07:25,039 --> 00:07:27,280 um that's the main thing that you're 164 00:07:27,280 --> 00:07:29,840 often concerned with. Um and it is of 165 00:07:29,840 --> 00:07:31,360 course a very important thing and 166 00:07:31,360 --> 00:07:33,120 something that takes up a lot of my time 167 00:07:33,120 --> 00:07:36,400 to to manage and keep running. And then 168 00:07:36,400 --> 00:07:38,560 some other stuff that you need um just 169 00:07:38,560 --> 00:07:40,160 so that you actually know what's going 170 00:07:40,160 --> 00:07:42,960 on in your system. So logs, monitoring, 171 00:07:42,960 --> 00:07:44,720 replication and backups, failover and 172 00:07:44,720 --> 00:07:47,520 recovery so that if something goes wrong 173 00:07:47,520 --> 00:07:52,080 um you're not just on a sinking ship. 174 00:07:52,080 --> 00:07:55,280 Now cloud providers are going to tell 175 00:07:55,280 --> 00:07:57,599 you we can help you with that. we can do 176 00:07:57,599 --> 00:08:00,639 all of this annoying stuff for you. Um, 177 00:08:00,639 --> 00:08:03,840 so we're back to this stack model. And 178 00:08:03,840 --> 00:08:05,520 um, if you go from left to right, you go 179 00:08:05,520 --> 00:08:08,160 from on premise to infrastructure as a 180 00:08:08,160 --> 00:08:09,919 service to platform as a service to 181 00:08:09,919 --> 00:08:12,400 software as a service. On the left, the 182 00:08:12,400 --> 00:08:14,960 leftmost one is your on premise and all 183 00:08:14,960 --> 00:08:17,520 the other stuff is cloud. So if you're 184 00:08:17,520 --> 00:08:19,360 doing infrastructure as a service, that 185 00:08:19,360 --> 00:08:22,639 means you've got cloud uh, you've got 186 00:08:22,639 --> 00:08:24,560 your VMs in the cloud and you get a 187 00:08:24,560 --> 00:08:26,319 bunch of management around that. If 188 00:08:26,319 --> 00:08:28,800 you're doing platform as a service, then 189 00:08:28,800 --> 00:08:31,199 most almost all of the platform service 190 00:08:31,199 --> 00:08:33,680 stuff is being taken care of for you. 191 00:08:33,680 --> 00:08:35,279 And then the last thing is software as a 192 00:08:35,279 --> 00:08:37,200 service where it's turnkey software that 193 00:08:37,200 --> 00:08:40,240 just implements an actual business um 194 00:08:40,240 --> 00:08:43,360 problem um an actual business product 195 00:08:43,360 --> 00:08:45,200 for you. 196 00:08:45,200 --> 00:08:48,480 Now what the cloud also says you get 197 00:08:48,480 --> 00:08:50,959 with this is you get on demand 198 00:08:50,959 --> 00:08:53,760 resources. You don't have to 199 00:08:53,760 --> 00:08:55,360 pay a million dollars to get some 200 00:08:55,360 --> 00:08:57,680 servers which unfortunately costs a 201 00:08:57,680 --> 00:09:00,800 million dollars now and then has um 3 202 00:09:00,800 --> 00:09:02,480 months of lead time until you get them 203 00:09:02,480 --> 00:09:04,160 which is unfortunately also how things 204 00:09:04,160 --> 00:09:07,920 are now. Um just one or two years ago it 205 00:09:07,920 --> 00:09:12,160 was much better. Um we all know why. And 206 00:09:12,160 --> 00:09:14,560 then for those on demand resources you 207 00:09:14,560 --> 00:09:16,320 only have to pay what you need, pay what 208 00:09:16,320 --> 00:09:18,000 you use and you get flexible right 209 00:09:18,000 --> 00:09:20,480 sizing. You can make those VMs as big or 210 00:09:20,480 --> 00:09:22,640 as small as you need at any time 211 00:09:22,640 --> 00:09:25,120 whenever you want. And then all of the 212 00:09:25,120 --> 00:09:27,760 platform bits like your identity 213 00:09:27,760 --> 00:09:29,360 management, your monitoring, your 214 00:09:29,360 --> 00:09:32,320 backups, your failover, you get all of 215 00:09:32,320 --> 00:09:35,360 that out of the box. The cloud has a 216 00:09:35,360 --> 00:09:37,760 service for that and that lets you shift 217 00:09:37,760 --> 00:09:39,120 the responsibility for your 218 00:09:39,120 --> 00:09:41,040 availability, resilience, durability, 219 00:09:41,040 --> 00:09:43,519 and maintenance all to the cloud 220 00:09:43,519 --> 00:09:46,800 provider. Wonderful. Um, how does this 221 00:09:46,800 --> 00:09:48,880 what does this actually look like? The 222 00:09:48,880 --> 00:09:51,680 virtual machine is in a failed st state. 223 00:09:51,680 --> 00:09:54,560 VM status blob is found but not yet 224 00:09:54,560 --> 00:09:56,560 populated. None of those words are in 225 00:09:56,560 --> 00:09:58,480 the Bible. 226 00:09:58,480 --> 00:10:00,320 So, what actually happens is that you 227 00:10:00,320 --> 00:10:04,480 pay a 100% to 500% premium to start off 228 00:10:04,480 --> 00:10:07,120 with. Um, 229 00:10:07,120 --> 00:10:09,519 buying your servers versus renting them 230 00:10:09,519 --> 00:10:11,040 from the cloud are two different 231 00:10:11,040 --> 00:10:14,720 propositions. Um, capex versus opex. But 232 00:10:14,720 --> 00:10:17,120 at the end of the day, cloud a lot more 233 00:10:17,120 --> 00:10:20,079 expensive. Now then all of these great 234 00:10:20,079 --> 00:10:21,839 abstractions that I told you the cloud 235 00:10:21,839 --> 00:10:24,160 gives you, they are actually built for 236 00:10:24,160 --> 00:10:26,880 the cloud's convenience and not yours. 237 00:10:26,880 --> 00:10:30,240 Um they have a specific use case in mind 238 00:10:30,240 --> 00:10:33,200 and they also have managing get keeping 239 00:10:33,200 --> 00:10:35,920 their own stuff running in mind and 240 00:10:35,920 --> 00:10:37,519 whether that aligns with what you 241 00:10:37,519 --> 00:10:39,519 actually need is a little bit luck 242 00:10:39,519 --> 00:10:41,200 based. 243 00:10:41,200 --> 00:10:43,279 And another thing that they don't want 244 00:10:43,279 --> 00:10:45,440 you to know is that all of this stuff is 245 00:10:45,440 --> 00:10:47,279 held together with string bubble gums 246 00:10:47,279 --> 00:10:49,519 and lots of manual operator invention. 247 00:10:49,519 --> 00:10:51,920 Um there's a couple of good blog posts 248 00:10:51,920 --> 00:10:54,640 that tell you um things that went down 249 00:10:54,640 --> 00:11:00,640 at AWS, at Google, at Azure where the 250 00:11:00,640 --> 00:11:02,800 whole thing where you just have an API 251 00:11:02,800 --> 00:11:05,440 and when you make an API call, it goes 252 00:11:05,440 --> 00:11:08,079 away and it does things to real servers 253 00:11:08,079 --> 00:11:11,360 somewhere and that all works. 254 00:11:11,360 --> 00:11:13,279 Maybe most of the time, but definitely 255 00:11:13,279 --> 00:11:17,760 not all of the time. um some of the core 256 00:11:17,760 --> 00:11:20,320 Azure services until they were migrated 257 00:11:20,320 --> 00:11:22,720 were running in 258 00:11:22,720 --> 00:11:27,519 the Xbox Live um system. 259 00:11:27,519 --> 00:11:30,880 Just an interesting tidbit. Um 260 00:11:30,880 --> 00:11:33,040 it's also impossible to implement 261 00:11:33,040 --> 00:11:36,079 resilience like actual redundancy um 262 00:11:36,079 --> 00:11:39,760 geographic um distribution and so on 263 00:11:39,760 --> 00:11:42,480 with a reasonable overhead. Um if you 264 00:11:42,480 --> 00:11:45,279 look at what the cloud their blueprints 265 00:11:45,279 --> 00:11:47,920 tell you that you should do 266 00:11:47,920 --> 00:11:52,000 um instead of paying this 100 to 500% 267 00:11:52,000 --> 00:11:53,920 premium you pay it twice or three times 268 00:11:53,920 --> 00:11:57,760 or four times and um there's pretty much 269 00:11:57,760 --> 00:12:00,079 no efficiencies to be gained there. And 270 00:12:00,079 --> 00:12:02,000 then those great platform services that 271 00:12:02,000 --> 00:12:04,640 they say they can provide for you, you 272 00:12:04,640 --> 00:12:08,000 get nickel and dime for them as well. Um 273 00:12:08,000 --> 00:12:11,680 like for example if you have your logs 274 00:12:11,680 --> 00:12:15,040 in AWS and you want to search those logs 275 00:12:15,040 --> 00:12:18,560 you got to open your checkbook. Um and 276 00:12:18,560 --> 00:12:21,839 then the last bit the responsibility 277 00:12:21,839 --> 00:12:26,000 you cannot shift responsibility. Um your 278 00:12:26,000 --> 00:12:27,760 business stakeholders are looking to you 279 00:12:27,760 --> 00:12:29,600 to keep things running and if things 280 00:12:29,600 --> 00:12:30,880 aren't running and you say it's the 281 00:12:30,880 --> 00:12:33,279 cloud's fault that doesn't help you very 282 00:12:33,279 --> 00:12:35,920 much. Um, even if you pay for premium 283 00:12:35,920 --> 00:12:39,279 support and you can get somebody on the 284 00:12:39,279 --> 00:12:41,839 phone, um, you can get a cloud solutions 285 00:12:41,839 --> 00:12:44,720 team from Azure on the phone. Um, 286 00:12:44,720 --> 00:12:47,040 they'll try to help you, but if the 287 00:12:47,040 --> 00:12:49,760 cloud is just broken, like virtual 288 00:12:49,760 --> 00:12:52,320 machine in a failed state, then you're 289 00:12:52,320 --> 00:12:56,079 probably in trouble anyway. 290 00:12:56,079 --> 00:12:59,279 Um if you if you own your own stuff then 291 00:12:59,279 --> 00:13:01,680 at least you have got full access to it 292 00:13:01,680 --> 00:13:04,560 and whatever needs fixing you can do it 293 00:13:04,560 --> 00:13:09,279 um if it's possible. Now there are some 294 00:13:09,279 --> 00:13:11,200 disadvantages when you're doing things 295 00:13:11,200 --> 00:13:13,920 on prem um and some things are actually 296 00:13:13,920 --> 00:13:16,880 easier in the cloud. Um server hardware 297 00:13:16,880 --> 00:13:19,760 and management is time that you're going 298 00:13:19,760 --> 00:13:22,800 to have to spend. So you when hardware 299 00:13:22,800 --> 00:13:25,839 breaks on prem, you got to get it fixed, 300 00:13:25,839 --> 00:13:27,920 you got to get it replaced, you got to 301 00:13:27,920 --> 00:13:30,399 go to the data center and do it. And 302 00:13:30,399 --> 00:13:33,200 there's a drag. Sometimes it's fun. Um, 303 00:13:33,200 --> 00:13:35,040 but if you just need things to be 304 00:13:35,040 --> 00:13:39,360 working, it is a bit of a drag. 305 00:13:39,360 --> 00:13:41,519 In the cloud, the ability to just spin 306 00:13:41,519 --> 00:13:44,800 up another VM or another instance is 307 00:13:44,800 --> 00:13:47,279 really nice. Um, you don't have those 308 00:13:47,279 --> 00:13:51,120 lead times. um you don't generally run 309 00:13:51,120 --> 00:13:52,959 out of capacity the same way that you 310 00:13:52,959 --> 00:13:55,760 just hard run out of capacity um in your 311 00:13:55,760 --> 00:13:58,240 data center if you haven't planned to to 312 00:13:58,240 --> 00:14:01,680 expand your capacity. Um 313 00:14:01,680 --> 00:14:03,839 except that clouds also have capacity 314 00:14:03,839 --> 00:14:07,519 issues all the time. Um it's usually not 315 00:14:07,519 --> 00:14:10,639 a real showstopper. Um usually it's a 316 00:14:10,639 --> 00:14:13,440 specific data center that has run out of 317 00:14:13,440 --> 00:14:16,399 uh capacity. Um, and if you've 318 00:14:16,399 --> 00:14:17,839 architected your stuff right, you can 319 00:14:17,839 --> 00:14:19,279 just spin up capacity in a different 320 00:14:19,279 --> 00:14:22,399 data center and things can at least limp 321 00:14:22,399 --> 00:14:24,000 on. 322 00:14:24,000 --> 00:14:27,279 Um, one of the things in the cloud that 323 00:14:27,279 --> 00:14:29,600 is really nice is manage containers or 324 00:14:29,600 --> 00:14:33,040 Kubernetes. Um, I am a Kubernetes hater. 325 00:14:33,040 --> 00:14:35,120 Um, I think that if you're running a 326 00:14:35,120 --> 00:14:37,440 Kubernetes, you have become a Kubernetes 327 00:14:37,440 --> 00:14:39,279 operator. you have become an expert in 328 00:14:39,279 --> 00:14:43,199 Kubernetes and 80% of your effort is 329 00:14:43,199 --> 00:14:45,279 going to go into keeping your Kubernetes 330 00:14:45,279 --> 00:14:48,079 cluster running. Um, I've done it. Um, 331 00:14:48,079 --> 00:14:51,360 that's my experience. However, 332 00:14:51,360 --> 00:14:53,440 manage Kubernetes or managed containers 333 00:14:53,440 --> 00:14:57,760 in the cloud, they actually do they have 334 00:14:57,760 --> 00:15:01,040 done all of the work and suffered all of 335 00:15:01,040 --> 00:15:04,240 the pain to keep those things running. 336 00:15:04,240 --> 00:15:07,920 Um and so you pay a boatload of money 337 00:15:07,920 --> 00:15:12,000 for that. Um but you do get working 338 00:15:12,000 --> 00:15:15,680 Kubernetes clusters um that you can use. 339 00:15:15,680 --> 00:15:17,920 There's another thing in the cloud which 340 00:15:17,920 --> 00:15:21,440 is that if you are a little gremlin and 341 00:15:21,440 --> 00:15:23,519 you want to abuse cheap basic cloud 342 00:15:23,519 --> 00:15:26,560 components to build something that is a 343 00:15:26,560 --> 00:15:28,399 RP Goldberg machine that implements your 344 00:15:28,399 --> 00:15:30,480 requirements for way cheaper than the 345 00:15:30,480 --> 00:15:32,399 recommended solution is a fun challenge 346 00:15:32,399 --> 00:15:34,800 for you. There's lots of fun challenges 347 00:15:34,800 --> 00:15:39,839 to be had. Like for example, um hosting 348 00:15:39,839 --> 00:15:43,120 videos inside a cache layer for a 349 00:15:43,120 --> 00:15:46,399 container system um that was never 350 00:15:46,399 --> 00:15:48,320 intended for you to put arbitrary things 351 00:15:48,320 --> 00:15:50,240 into that cache. Um and then figuring 352 00:15:50,240 --> 00:15:52,320 out how to get signed URLs so that you 353 00:15:52,320 --> 00:15:53,920 can then download those and look and and 354 00:15:53,920 --> 00:15:56,800 play um those videos um without paying 355 00:15:56,800 --> 00:15:59,040 AWS single cent for all of the data 356 00:15:59,040 --> 00:16:02,720 egress that you have. Um, I think they 357 00:16:02,720 --> 00:16:05,040 fixed that at some point. And this isn't 358 00:16:05,040 --> 00:16:06,880 something that I've done. Um, I stand on 359 00:16:06,880 --> 00:16:10,399 the shoulders of giants. Um, if you know 360 00:16:10,399 --> 00:16:13,839 uh a lovely person called XSS Fox, um, 361 00:16:13,839 --> 00:16:16,639 she does things like that. 362 00:16:16,639 --> 00:16:19,440 Now, how do you get into this 363 00:16:19,440 --> 00:16:22,560 self-hosting thing if you want to? Um, 364 00:16:22,560 --> 00:16:25,040 here's a picture of my home lab. Um, 365 00:16:25,040 --> 00:16:27,279 figure A, a small form factor 366 00:16:27,279 --> 00:16:30,160 workstation that runs Proxmox, which is 367 00:16:30,160 --> 00:16:33,680 a virtual machine um, hosting system um, 368 00:16:33,680 --> 00:16:37,199 that runs a YOS router VM on it with 369 00:16:37,199 --> 00:16:41,040 DHCP and DNS and wire guard and all the 370 00:16:41,040 --> 00:16:44,399 other good stuff. Um, and so the router 371 00:16:44,399 --> 00:16:48,800 for my home network is a 372 00:16:48,800 --> 00:16:52,399 um, virtual machine. Um, 373 00:16:52,399 --> 00:16:54,800 if you know something about networking 374 00:16:54,800 --> 00:16:56,560 and and bootstrapping problems, you 375 00:16:56,560 --> 00:16:58,720 might think, "How does that work?" Um, 376 00:16:58,720 --> 00:17:02,639 it works. Don't don't ask questions. Um, 377 00:17:02,639 --> 00:17:06,480 figure B is a even smaller form factor 378 00:17:06,480 --> 00:17:11,280 workstation um that runs Pi Hole. Um, Pi 379 00:17:11,280 --> 00:17:14,240 Hole is just a add blocking DNS um, 380 00:17:14,240 --> 00:17:18,400 recursor. Um, and that's only running 381 00:17:18,400 --> 00:17:20,959 right on there. um because I haven't 382 00:17:20,959 --> 00:17:23,839 managed to migrate it into Proxmox 383 00:17:23,839 --> 00:17:25,839 um which I should have done years ago 384 00:17:25,839 --> 00:17:28,480 but hasn't happened. Um, figure C is a 385 00:17:28,480 --> 00:17:33,120 NAS server. Um, that's a an old micro um 386 00:17:33,120 --> 00:17:36,720 NAS server um that I got for 200 uh 387 00:17:36,720 --> 00:17:40,240 euros and now has approximately $15,000 388 00:17:40,240 --> 00:17:43,120 of um hard drives in it. Um, which is 389 00:17:43,120 --> 00:17:46,080 also um they are enterprise hard drives 390 00:17:46,080 --> 00:17:50,640 um which I have for some reason um and 391 00:17:50,640 --> 00:17:53,919 they are now um worth uh their weight in 392 00:17:53,919 --> 00:17:56,240 gold I suppose. I haven't calculated 393 00:17:56,240 --> 00:17:58,000 that out, but it's a lot of money. 394 00:17:58,000 --> 00:18:02,559 Anyway, um figure D is just a UPS. 395 00:18:02,559 --> 00:18:08,080 Um figure E are three USB hard drives um 396 00:18:08,080 --> 00:18:09,840 that are suspiciously just hanging 397 00:18:09,840 --> 00:18:11,760 around. Um also something that I 398 00:18:11,760 --> 00:18:12,960 shouldn't have because I now have this 399 00:18:12,960 --> 00:18:16,559 great NAS server. Um figure F is that's 400 00:18:16,559 --> 00:18:22,360 just Mr. Skelly. Um he hangs out. Now 401 00:18:23,200 --> 00:18:26,960 you can learn a lot about on-prem 402 00:18:26,960 --> 00:18:29,360 hosting and platform engineering with a 403 00:18:29,360 --> 00:18:33,600 home lab. Um I personally was privileged 404 00:18:33,600 --> 00:18:35,360 to get into platform engineering by 405 00:18:35,360 --> 00:18:37,200 joining strong teams that were already 406 00:18:37,200 --> 00:18:39,760 doing it um and learning by doing um 407 00:18:39,760 --> 00:18:43,520 along with them. Um and that is a great 408 00:18:43,520 --> 00:18:47,280 opportunity and if you aren't able if 409 00:18:47,280 --> 00:18:48,640 you don't have access to that you do 410 00:18:48,640 --> 00:18:50,640 have an uphill battle in front of you. 411 00:18:50,640 --> 00:18:55,440 Um so that is why I'm saying maybe a 412 00:18:55,440 --> 00:18:57,280 home lab is something you can you can 413 00:18:57,280 --> 00:18:59,440 put together relatively cheaply and and 414 00:18:59,440 --> 00:19:01,360 you can learn a lot of things like for 415 00:19:01,360 --> 00:19:03,679 example about um hosting VMs with 416 00:19:03,679 --> 00:19:06,480 Proxmox. Um those are you will 417 00:19:06,480 --> 00:19:08,960 definitely learn skills there both in 418 00:19:08,960 --> 00:19:10,559 the actual architecture and in 419 00:19:10,559 --> 00:19:12,880 troubleshooting that will make you a 420 00:19:12,880 --> 00:19:14,720 better systems administration 421 00:19:14,720 --> 00:19:16,880 administrator a better platform 422 00:19:16,880 --> 00:19:18,480 engineer. 423 00:19:18,480 --> 00:19:20,799 Um 424 00:19:20,799 --> 00:19:23,600 you can also buy microic routers. Um, 425 00:19:23,600 --> 00:19:27,760 Microic has a product range that goes 426 00:19:27,760 --> 00:19:31,280 from very tiny to enterprisegrade um, 427 00:19:31,280 --> 00:19:33,120 router and they all have a common 428 00:19:33,120 --> 00:19:35,600 platform, a common architecture and so 429 00:19:35,600 --> 00:19:37,600 you can buy one that's the right size 430 00:19:37,600 --> 00:19:42,080 for your home network. Um, and then um, 431 00:19:42,080 --> 00:19:44,320 your skills from there will translate 432 00:19:44,320 --> 00:19:48,640 into um, into professional network 433 00:19:48,640 --> 00:19:50,240 engineering if that's if that's the 434 00:19:50,240 --> 00:19:54,160 direction you want to go. Um, 435 00:19:54,160 --> 00:19:57,919 if you're working in the cloud, um, you 436 00:19:57,919 --> 00:19:59,919 can just deploy cloud VMs and you can do 437 00:19:59,919 --> 00:20:01,280 whatever you want on them. They're just 438 00:20:01,280 --> 00:20:04,880 computers. Um, you can get free cloud 439 00:20:04,880 --> 00:20:09,200 accounts, um, trials, um, always free 440 00:20:09,200 --> 00:20:12,880 sort of um, things for trying it out. 441 00:20:12,880 --> 00:20:15,120 Um, try to get some learning budget from 442 00:20:15,120 --> 00:20:18,799 your employer if that's possible. Um or 443 00:20:18,799 --> 00:20:21,039 you can look at the software that you 444 00:20:21,039 --> 00:20:23,600 already work on and see what is that 445 00:20:23,600 --> 00:20:26,320 running on. Um what are the admins, the 446 00:20:26,320 --> 00:20:28,000 platform engineers actually doing there? 447 00:20:28,000 --> 00:20:30,720 What are the supporting services? Um and 448 00:20:30,720 --> 00:20:33,679 knowing more about that is going to be 449 00:20:33,679 --> 00:20:35,360 good for you if you're a software 450 00:20:35,360 --> 00:20:38,159 engineer. Um because then there you will 451 00:20:38,159 --> 00:20:40,320 maybe get less push back from the 452 00:20:40,320 --> 00:20:42,080 platform engineers or the firewall team 453 00:20:42,080 --> 00:20:44,799 saying no, we can't do this. um because 454 00:20:44,799 --> 00:20:48,720 you might already know um how to do 455 00:20:48,720 --> 00:20:51,200 things um the way they're thinking about 456 00:20:51,200 --> 00:20:53,039 things. 457 00:20:53,039 --> 00:20:57,520 Now, what to do and what to avoid when 458 00:20:57,520 --> 00:21:00,799 you're building a platform. Um and 459 00:21:00,799 --> 00:21:05,039 that's all about having standards and 460 00:21:05,039 --> 00:21:06,559 consistent 461 00:21:06,559 --> 00:21:08,880 um processes. 462 00:21:08,880 --> 00:21:11,360 Um, and the top line is automate and 463 00:21:11,360 --> 00:21:13,200 standardize and be as boring as 464 00:21:13,200 --> 00:21:15,679 possible. Um, choose boring technology 465 00:21:15,679 --> 00:21:18,480 that is well known and has a community 466 00:21:18,480 --> 00:21:21,360 and act as a force multiplier for you. 467 00:21:21,360 --> 00:21:23,679 Automate your server bring up. Um, 468 00:21:23,679 --> 00:21:27,840 that's something that maybe um 469 00:21:27,840 --> 00:21:29,840 maybe some people don't know. You can 470 00:21:29,840 --> 00:21:31,679 your on-pre servers the same thing you 471 00:21:31,679 --> 00:21:33,919 can do with Terraform in the cloud. Um, 472 00:21:33,919 --> 00:21:36,880 you can do on premises with net booting. 473 00:21:36,880 --> 00:21:39,280 Um there's something called FII, fully 474 00:21:39,280 --> 00:21:41,360 automated install, um which I'll mention 475 00:21:41,360 --> 00:21:43,919 again later. Um which allows you to just 476 00:21:43,919 --> 00:21:47,440 turn a server on um and 20 minutes later 477 00:21:47,440 --> 00:21:50,559 it is fully installed. Um and you can 478 00:21:50,559 --> 00:21:52,799 just turn on the platform services on 479 00:21:52,799 --> 00:21:54,080 that server because they've already been 480 00:21:54,080 --> 00:21:58,240 installed. Um we do that um at work. Um 481 00:21:58,240 --> 00:22:00,960 we can shut the server down, reimage it, 482 00:22:00,960 --> 00:22:03,679 and 25 minutes later it's back in full 483 00:22:03,679 --> 00:22:06,720 production service. 484 00:22:06,720 --> 00:22:10,000 automate your deployments obviously and 485 00:22:10,000 --> 00:22:13,200 also automate your service management. 486 00:22:13,200 --> 00:22:16,159 So um your your fleet management for 487 00:22:16,159 --> 00:22:20,159 your servers and build tools. Um you 488 00:22:20,159 --> 00:22:22,080 want to have lots of tools. You want to 489 00:22:22,080 --> 00:22:25,440 be doing everything through tools 490 00:22:25,440 --> 00:22:28,640 and you want to automate that and you 491 00:22:28,640 --> 00:22:30,880 want your automation and your tools to 492 00:22:30,880 --> 00:22:34,000 work. You want them to be one and the 493 00:22:34,000 --> 00:22:37,520 same thing. So um 494 00:22:37,520 --> 00:22:39,120 whether you whether you're whether 495 00:22:39,120 --> 00:22:40,720 you're running a script on the command 496 00:22:40,720 --> 00:22:41,919 line or whether you're running it 497 00:22:41,919 --> 00:22:43,360 through the automation should be the 498 00:22:43,360 --> 00:22:47,280 same thing. Um that way you you haven't 499 00:22:47,280 --> 00:22:50,000 got any drift between uh those two those 500 00:22:50,000 --> 00:22:52,799 two ways. Um everything you do manually 501 00:22:52,799 --> 00:22:54,720 is the same that you're doing uh through 502 00:22:54,720 --> 00:22:56,240 automation. 503 00:22:56,240 --> 00:22:58,080 And that just makes things more 504 00:22:58,080 --> 00:23:01,280 inspectable and makes it um means that 505 00:23:01,280 --> 00:23:03,200 after the fact you know you know what 506 00:23:03,200 --> 00:23:06,720 what happened. Um always keep iterating 507 00:23:06,720 --> 00:23:09,600 towards that standardization have fewer 508 00:23:09,600 --> 00:23:12,720 better ways um to do things. 509 00:23:12,720 --> 00:23:14,320 Um 510 00:23:14,320 --> 00:23:16,400 and there are some things you might be 511 00:23:16,400 --> 00:23:18,159 tempted to use for automations but that 512 00:23:18,159 --> 00:23:20,960 are actually traps. Um anything that has 513 00:23:20,960 --> 00:23:23,919 distributed state is the devil. um 514 00:23:23,919 --> 00:23:25,600 distributed state that you don't 515 00:23:25,600 --> 00:23:29,760 understand whether that's uh replication 516 00:23:29,760 --> 00:23:33,280 um some sort of some some sort of thing 517 00:23:33,280 --> 00:23:35,919 called consensus algorithms um if you 518 00:23:35,919 --> 00:23:38,559 don't understand the failure modes um 519 00:23:38,559 --> 00:23:42,159 you will have a bad time um the other 520 00:23:42,159 --> 00:23:43,520 thing that I already mentioned is 521 00:23:43,520 --> 00:23:46,640 Kubernetes um like I already said I'm a 522 00:23:46,640 --> 00:23:51,280 Kubernetes hater um another thing bot um 523 00:23:51,280 --> 00:23:53,520 let's encrypt is an amazing great 524 00:23:53,520 --> 00:23:57,120 service. Um, Sbot is I want to say 525 00:23:57,120 --> 00:23:58,960 another word here. It's not good. Um, 526 00:23:58,960 --> 00:24:00,799 just use a different Acme client. Um, 527 00:24:00,799 --> 00:24:02,320 there's lots of them. Um, there's also 528 00:24:02,320 --> 00:24:04,240 web servers that already come with an 529 00:24:04,240 --> 00:24:06,880 Acme client. That's the protocol um 530 00:24:06,880 --> 00:24:10,080 that's used for for Let's Encrypt um or 531 00:24:10,080 --> 00:24:14,080 other um ACME services um that give you 532 00:24:14,080 --> 00:24:16,799 SSL certificates. Um there's lots of 533 00:24:16,799 --> 00:24:20,080 others. Use those. Um and Docker 534 00:24:20,080 --> 00:24:22,480 containers are amazing. Um Docker was 535 00:24:22,480 --> 00:24:24,400 the thing the first thing that really 536 00:24:24,400 --> 00:24:28,159 brought uh containers into into the 537 00:24:28,159 --> 00:24:31,840 limelight. Um but as a first mover um 538 00:24:31,840 --> 00:24:34,960 it's not it's got a lot of issues. Um 539 00:24:34,960 --> 00:24:36,799 for one of the alternatives that are a 540 00:24:36,799 --> 00:24:41,360 lot better is Podman. Um so when in 541 00:24:41,360 --> 00:24:43,279 doubt if you're deciding whether to pick 542 00:24:43,279 --> 00:24:46,880 Docker or Podman, use Podman. 543 00:24:46,880 --> 00:24:49,279 There's some things that you're going to 544 00:24:49,279 --> 00:24:52,080 do that you want to stop doing early. 545 00:24:52,080 --> 00:24:54,000 Um, there's lots of choices to make. You 546 00:24:54,000 --> 00:24:55,679 always have lots of choices to make. 547 00:24:55,679 --> 00:24:57,440 Some of them will be bad ones. You won't 548 00:24:57,440 --> 00:25:00,159 get it right the first time. Just be 549 00:25:00,159 --> 00:25:02,640 ready to outgrow 550 00:25:02,640 --> 00:25:04,880 um the choices that you made previously. 551 00:25:04,880 --> 00:25:07,279 Do not paint yourself into a corner. 552 00:25:07,279 --> 00:25:09,039 Take your bad choices and replace them 553 00:25:09,039 --> 00:25:11,200 with better ones. For example, as sage 554 00:25:11,200 --> 00:25:13,200 as orchestration, 555 00:25:13,200 --> 00:25:15,520 um, you're going to have a bad time. a 556 00:25:15,520 --> 00:25:18,480 sis admin oriented orchestration tool 557 00:25:18,480 --> 00:25:21,200 like rune that's too cumbersome if 558 00:25:21,200 --> 00:25:24,559 you're running a complex platform um 559 00:25:24,559 --> 00:25:27,120 build your own um you will not regret 560 00:25:27,120 --> 00:25:30,000 building your own orchestration 561 00:25:30,000 --> 00:25:33,120 um I mean you will have regrets but it 562 00:25:33,120 --> 00:25:35,440 will serve you better 563 00:25:35,440 --> 00:25:37,679 dependencies on third party repos and 564 00:25:37,679 --> 00:25:39,520 sources that you don't have cache that 565 00:25:39,520 --> 00:25:41,360 is sort of a trap that people just fall 566 00:25:41,360 --> 00:25:44,240 into um you can't depend on being able 567 00:25:44,240 --> 00:25:46,640 to pull things from the internet 568 00:25:46,640 --> 00:25:48,320 anywhere in your stack whether it's in 569 00:25:48,320 --> 00:25:50,000 your production system in your build 570 00:25:50,000 --> 00:25:52,400 pipeline um you got to have mirrors and 571 00:25:52,400 --> 00:25:54,880 and caches and container registries that 572 00:25:54,880 --> 00:25:58,799 you run and so that if um Docker Hub 573 00:25:58,799 --> 00:26:01,360 decides to rate limit you um your world 574 00:26:01,360 --> 00:26:04,240 doesn't just stop 575 00:26:04,240 --> 00:26:06,320 good practices 576 00:26:06,320 --> 00:26:09,200 um 577 00:26:09,200 --> 00:26:10,880 oh 578 00:26:10,880 --> 00:26:14,159 I I missed the slide there um good 579 00:26:14,159 --> 00:26:16,080 practice practices are all about change 580 00:26:16,080 --> 00:26:19,799 management and a 581 00:26:20,400 --> 00:26:23,679 okay um the monitor just stopped there. 582 00:26:23,679 --> 00:26:25,279 Um 583 00:26:25,279 --> 00:26:28,159 yeah, good practices and change 584 00:26:28,159 --> 00:26:31,360 management and avoiding surprises. Um 585 00:26:31,360 --> 00:26:33,120 there's already been great talks about 586 00:26:33,120 --> 00:26:36,000 this today which were amazing. Um so I 587 00:26:36,000 --> 00:26:37,679 don't actually have much to to to say 588 00:26:37,679 --> 00:26:39,039 about that that hasn't already been 589 00:26:39,039 --> 00:26:41,840 said. Um, you want to consider whether a 590 00:26:41,840 --> 00:26:43,760 change that you're making is going to 591 00:26:43,760 --> 00:26:45,600 screw up someone else's day. Um, we 592 00:26:45,600 --> 00:26:47,360 called it whether it's moving someone's 593 00:26:47,360 --> 00:26:51,200 cheese. Um, you want to make rollout 594 00:26:51,200 --> 00:26:53,840 plans for changes. You want to have 595 00:26:53,840 --> 00:26:56,640 stage rollout environments and you want 596 00:26:56,640 --> 00:26:58,720 to spend more effort than feels 597 00:26:58,720 --> 00:27:01,760 justifiable to you at the time on alert 598 00:27:01,760 --> 00:27:04,799 hygiene because alert fatigue 599 00:27:04,799 --> 00:27:11,279 is a killer. Um if you yeah um 600 00:27:11,279 --> 00:27:13,360 just spend more effort on cleaning up 601 00:27:13,360 --> 00:27:15,440 your alerts, fixing the things that are 602 00:27:15,440 --> 00:27:18,240 broken. 603 00:27:18,240 --> 00:27:20,880 Uh quick thing about recommended 604 00:27:20,880 --> 00:27:22,960 software that you might not know about 605 00:27:22,960 --> 00:27:26,720 um PXE boot and FAI Linux 606 00:27:26,720 --> 00:27:29,840 um which I just mentioned to just fully 607 00:27:29,840 --> 00:27:32,159 install your servers from scratch. Um 608 00:27:32,159 --> 00:27:34,240 service discovery and distributed KV 609 00:27:34,240 --> 00:27:36,159 store. This is one of those things that 610 00:27:36,159 --> 00:27:38,480 gives you distributed state um and you 611 00:27:38,480 --> 00:27:40,320 got to be very careful with but console 612 00:27:40,320 --> 00:27:42,960 is a solid product. Um you do have to 613 00:27:42,960 --> 00:27:45,760 spend time on understanding it and and 614 00:27:45,760 --> 00:27:48,159 figuring out its failure cases but once 615 00:27:48,159 --> 00:27:51,360 you do it it it really helps. Um it 616 00:27:51,360 --> 00:27:56,799 takes a lot of work off of your back. Um 617 00:27:56,799 --> 00:27:59,919 DNS unbound and canot DNS um those are 618 00:27:59,919 --> 00:28:02,159 sort of standard software really. Um, 619 00:28:02,159 --> 00:28:05,120 code hosting, GitLab is okay. Today, if 620 00:28:05,120 --> 00:28:07,600 I had to set up a new code hosting 621 00:28:07,600 --> 00:28:10,000 system for your code and your CI/CD, I 622 00:28:10,000 --> 00:28:11,919 wouldn't choose GitLab. Um, I've been 623 00:28:11,919 --> 00:28:15,200 using it for 15 years. Um, there's newer 624 00:28:15,200 --> 00:28:17,360 things that are better. Um, wire God. 625 00:28:17,360 --> 00:28:22,240 Wire God is amazing. Um, yeah, 626 00:28:22,240 --> 00:28:25,440 setting up VPNs in the bad old times, 627 00:28:25,440 --> 00:28:28,159 uh, used to be terrible. Um, no good for 628 00:28:28,159 --> 00:28:30,399 anyone. Now, Wire God exists. use wire 629 00:28:30,399 --> 00:28:35,840 god. And that was my talk. Um, you can 630 00:28:35,840 --> 00:28:38,000 find me at the lobbycon, birds of a 631 00:28:38,000 --> 00:28:39,919 feather. I've got fed words. I've got an 632 00:28:39,919 --> 00:28:43,679 email address. And I want to just um 633 00:28:43,679 --> 00:28:45,200 advertise the company blog there. 634 00:28:45,200 --> 00:28:46,720 There's a lot of marketing stuff on 635 00:28:46,720 --> 00:28:49,120 there, but there's also a whole bunch of 636 00:28:49,120 --> 00:28:51,200 technical deep dives. Um, we do a lot of 637 00:28:51,200 --> 00:28:54,240 our building in the open um at Fastmail 638 00:28:54,240 --> 00:28:58,000 and we blog about our technical changes. 639 00:28:58,000 --> 00:29:00,080 And that's it from me. Thank you so 640 00:29:00,080 --> 00:29:03,030 much. [applause] 641 00:29:03,520 --> 00:29:05,679 Thank you so much. 642 00:29:05,679 --> 00:29:07,440 Of course, we have our little mug of 643 00:29:07,440 --> 00:29:08,480 appreciation. 644 00:29:08,480 --> 00:29:09,200 Thank you so much. 645 00:29:09,200 --> 00:29:13,039 And we have time for one quick question. 646 00:29:13,039 --> 00:29:15,360 Good question. 647 00:29:15,360 --> 00:29:19,080 Yes, at the back. 648 00:29:23,600 --> 00:29:25,200 Um, yeah. Hey, thank you for your talk. 649 00:29:25,200 --> 00:29:27,520 That was really interesting. Um, I'll 650 00:29:27,520 --> 00:29:29,760 preface my question by being really 651 00:29:29,760 --> 00:29:32,399 candid. I'm not a platform person at 652 00:29:32,399 --> 00:29:34,799 all. I'm a data analyst. So, apologies 653 00:29:34,799 --> 00:29:38,399 if I say anything silly. Um, but so I 654 00:29:38,399 --> 00:29:40,640 work with a lot of non-technical users 655 00:29:40,640 --> 00:29:43,520 and we run into a lot of problems. Um, 656 00:29:43,520 --> 00:29:44,960 where they're doing a lot of repetitive 657 00:29:44,960 --> 00:29:46,480 things and people kind of get stuck in 658 00:29:46,480 --> 00:29:48,799 their ways. Um, take simple things like 659 00:29:48,799 --> 00:29:52,080 a financial tracker for being run in a 660 00:29:52,080 --> 00:29:54,559 giant Excel workbook. Um, and it's 661 00:29:54,559 --> 00:29:56,480 terrible. it's really hard to reconcile 662 00:29:56,480 --> 00:29:58,000 between different people and things like 663 00:29:58,000 --> 00:29:59,600 that. So, something I've been playing 664 00:29:59,600 --> 00:30:01,440 around with a lot lately is building 665 00:30:01,440 --> 00:30:03,760 what I kind of call like micro apps. And 666 00:30:03,760 --> 00:30:06,000 so, essentially, they're kind of adverse 667 00:30:06,000 --> 00:30:08,720 to using Python to automate some of the 668 00:30:08,720 --> 00:30:10,640 more, you know, manual repetitive bits 669 00:30:10,640 --> 00:30:13,120 of things. And so, what I've been doing 670 00:30:13,120 --> 00:30:16,159 is building, I guess, like zip files of 671 00:30:16,159 --> 00:30:18,080 Python code that will launch like an 672 00:30:18,080 --> 00:30:21,039 HTML kind of um front end. So they can 673 00:30:21,039 --> 00:30:22,799 enter the data in there and then a lot 674 00:30:22,799 --> 00:30:24,240 of the processing is happening 675 00:30:24,240 --> 00:30:26,159 automatically for them. And for the 676 00:30:26,159 --> 00:30:28,240 non-technical user, they don't even 677 00:30:28,240 --> 00:30:30,399 really recognize the technical stuff 678 00:30:30,399 --> 00:30:33,520 happening underneath it. Um I guess what 679 00:30:33,520 --> 00:30:35,120 I was wondering listening to your talk I 680 00:30:35,120 --> 00:30:37,279 guess about self-hosting. 681 00:30:37,279 --> 00:30:38,720 Well, the way I've been trying to do it 682 00:30:38,720 --> 00:30:40,880 is just make it as easy as possible for 683 00:30:40,880 --> 00:30:43,039 people. So it's all run like local host 684 00:30:43,039 --> 00:30:45,200 from their own machine. Um they just 685 00:30:45,200 --> 00:30:47,360 download a folder, you know, they run a 686 00:30:47,360 --> 00:30:50,240 little setup package and away they go. 687 00:30:50,240 --> 00:30:52,720 Um, no idea if that's good practice or 688 00:30:52,720 --> 00:30:54,559 not, but the business outcome has been 689 00:30:54,559 --> 00:30:56,080 great because people have been a lot 690 00:30:56,080 --> 00:30:57,279 faster. There's a lot more 691 00:30:57,279 --> 00:30:59,520 standardization and being able to roll 692 00:30:59,520 --> 00:31:01,520 things up to like more dashboard views 693 00:31:01,520 --> 00:31:03,120 or executive views has become a lot 694 00:31:03,120 --> 00:31:05,360 easier. So my question is like do you 695 00:31:05,360 --> 00:31:07,279 have any advice for someone in that 696 00:31:07,279 --> 00:31:09,520 situation? Um, and just I guess yeah 697 00:31:09,520 --> 00:31:10,960 best practices and things. 698 00:31:10,960 --> 00:31:12,640 If that if that works for you, it's 699 00:31:12,640 --> 00:31:15,360 great. Um, but obviously there's there's 700 00:31:15,360 --> 00:31:17,279 a lot of ways for that to go wrong just 701 00:31:17,279 --> 00:31:19,840 because you're just giving people 702 00:31:19,840 --> 00:31:22,960 folders and um if they're using if 703 00:31:22,960 --> 00:31:25,440 they're using the version from last week 704 00:31:25,440 --> 00:31:27,120 um is that actually still going to work 705 00:31:27,120 --> 00:31:29,039 today and for that I think what I would 706 00:31:29,039 --> 00:31:32,559 recommend is creating what like in the 707 00:31:32,559 --> 00:31:34,960 platform world that's called an admin UI 708 00:31:34,960 --> 00:31:37,519 um where you just have a web app um that 709 00:31:37,519 --> 00:31:40,000 does all of these things um and that web 710 00:31:40,000 --> 00:31:41,600 app is something that you can you can 711 00:31:41,600 --> 00:31:44,480 version that you have a a um a 712 00:31:44,480 --> 00:31:46,960 development process for um that you have 713 00:31:46,960 --> 00:31:50,399 change management for. Um and yeah, 714 00:31:50,399 --> 00:31:52,960 that's how I would do that. Um because 715 00:31:52,960 --> 00:31:55,360 that yeah, just makes it all a lot more 716 00:31:55,360 --> 00:31:57,600 centralized and and I think should be 717 00:31:57,600 --> 00:31:59,600 easier to manage. 718 00:31:59,600 --> 00:32:01,840 Cool. Thank you very much. [applause] 719 00:32:01,840 --> 00:32:05,000 Thank you.