1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:15,720 --> 00:00:16,880 [cheering] 4 00:00:16,880 --> 00:00:20,240 Hello, welcome back. I am excited to 5 00:00:20,240 --> 00:00:23,199 introduce Alistair and I'm sure you 6 00:00:23,199 --> 00:00:29,080 won't regret their talk. Give it up. 7 00:00:34,399 --> 00:00:36,640 Right. Uh, thanks everyone for choosing 8 00:00:36,640 --> 00:00:38,640 to attend this session. Um, before we 9 00:00:38,640 --> 00:00:40,000 begin, I'd like to acknowledge the 10 00:00:40,000 --> 00:00:42,000 Turble and Yagura people as the 11 00:00:42,000 --> 00:00:43,440 traditional custodians of the land on 12 00:00:43,440 --> 00:00:45,280 which we are meeting today. And I pay my 13 00:00:45,280 --> 00:00:47,039 respects to their elders past, present, 14 00:00:47,039 --> 00:00:49,200 and in her. 15 00:00:49,200 --> 00:00:51,200 Uh, my name is Alistair. Uh, we have a 16 00:00:51,200 --> 00:00:52,879 lot of content to get through today. Uh, 17 00:00:52,879 --> 00:00:56,320 so strap in. Um, while this is a 18 00:00:56,320 --> 00:00:57,760 sponsored talk, uh, the views 19 00:00:57,760 --> 00:00:59,199 represented it are my own and not 20 00:00:59,199 --> 00:01:00,640 representative necessarily of my 21 00:01:00,640 --> 00:01:02,879 employer's views. I'm going to talk fast 22 00:01:02,879 --> 00:01:04,479 probably. Uh, so it may feel like you're 23 00:01:04,479 --> 00:01:05,920 watching a YouTube video on double 24 00:01:05,920 --> 00:01:08,240 speed. 25 00:01:08,240 --> 00:01:10,159 AI has been used in the creation of this 26 00:01:10,159 --> 00:01:11,840 talk. Uh, however, this talk has not 27 00:01:11,840 --> 00:01:13,920 been written by a large language model. 28 00:01:13,920 --> 00:01:16,080 I like to use AI for repetitive leg work 29 00:01:16,080 --> 00:01:18,000 so I can focus on more interesting 30 00:01:18,000 --> 00:01:20,080 things. 31 00:01:20,080 --> 00:01:22,000 Uh, first up, some context on the 32 00:01:22,000 --> 00:01:23,840 background of this talk. Don't worry, 33 00:01:23,840 --> 00:01:26,240 this isn't a sales pitch. Uh, over the 34 00:01:26,240 --> 00:01:27,840 last 5 years or so, we've built a 35 00:01:27,840 --> 00:01:29,759 platform that we call SECDev, uh, on 36 00:01:29,759 --> 00:01:31,600 which we host things for our customers. 37 00:01:31,600 --> 00:01:33,200 We care about stuff like clustered 38 00:01:33,200 --> 00:01:35,200 services, high availability deployments, 39 00:01:35,200 --> 00:01:37,759 and zero downtime patching. Uh, we also 40 00:01:37,759 --> 00:01:39,280 run our engineering team for the 41 00:01:39,280 --> 00:01:40,799 platform pretty lean, although we do 42 00:01:40,799 --> 00:01:42,240 have engineers from other areas of the 43 00:01:42,240 --> 00:01:46,240 company make cameos from time to time. 44 00:01:46,240 --> 00:01:48,079 Let's talk about infrastructure as 45 00:01:48,079 --> 00:01:50,159 software. I expect many people here will 46 00:01:50,159 --> 00:01:52,240 be familiar with the concept. Uh you've 47 00:01:52,240 --> 00:01:54,240 probably heard of infrastructure as code 48 00:01:54,240 --> 00:01:56,399 and written some declarative domain 49 00:01:56,399 --> 00:01:59,040 specific languages uh like terraform. Uh 50 00:01:59,040 --> 00:02:01,680 Palumi is similar uh but supports a full 51 00:02:01,680 --> 00:02:04,000 infrastructure as software approach. 52 00:02:04,000 --> 00:02:06,399 Infrastructure as software is using a 53 00:02:06,399 --> 00:02:08,239 program that you run to declare the 54 00:02:08,239 --> 00:02:10,319 graph of resources you want to provision 55 00:02:10,319 --> 00:02:12,080 written in a generalpurpose programming 56 00:02:12,080 --> 00:02:14,480 language. Because we're working with a 57 00:02:14,480 --> 00:02:16,239 general purpose programming language, we 58 00:02:16,239 --> 00:02:20,160 can get really clever. 59 00:02:20,160 --> 00:02:21,840 Starting out, things don't necessarily 60 00:02:21,840 --> 00:02:24,239 look too dissimilar to Terraform. Here, 61 00:02:24,239 --> 00:02:26,560 we create an S3 bucket using our 62 00:02:26,560 --> 00:02:28,640 programming language. If only it were 63 00:02:28,640 --> 00:02:31,120 ever that simple. If you've ever needed 64 00:02:31,120 --> 00:02:32,720 to actually build something that uses 65 00:02:32,720 --> 00:02:34,400 S3, you've probably needed to do a 66 00:02:34,400 --> 00:02:35,920 little bit more to the bucket than just 67 00:02:35,920 --> 00:02:38,239 create it. Because we're in a 68 00:02:38,239 --> 00:02:39,840 programming language, we can wrap all of 69 00:02:39,840 --> 00:02:42,560 this up into a class, which means it can 70 00:02:42,560 --> 00:02:47,480 become really, really easy to reuse. 71 00:02:49,680 --> 00:02:51,599 We've just had an example of creating an 72 00:02:51,599 --> 00:02:53,760 S3 bucket. Uh but actions that have 73 00:02:53,760 --> 00:02:56,319 outcomes can be represented too. For 74 00:02:56,319 --> 00:02:59,440 example, there's no such thing as an AMI 75 00:02:59,440 --> 00:03:02,080 copy resource in AWS. The result is just 76 00:03:02,080 --> 00:03:04,239 an AMI that has been copied from 77 00:03:04,239 --> 00:03:06,879 somewhere else. Once again, because 78 00:03:06,879 --> 00:03:08,640 we're using a programming language, we 79 00:03:08,640 --> 00:03:11,120 can do fun things like declare resources 80 00:03:11,120 --> 00:03:15,720 per and perform actions in loops. 81 00:03:16,560 --> 00:03:18,159 There are a few more layers here to 82 00:03:18,159 --> 00:03:19,680 understand which are probably familiar 83 00:03:19,680 --> 00:03:21,040 to anyone that's touched terraform a 84 00:03:21,040 --> 00:03:23,120 whole lot. When we write and run that 85 00:03:23,120 --> 00:03:25,120 code, we are creating a model of what we 86 00:03:25,120 --> 00:03:27,120 want which we kind of call our desired 87 00:03:27,120 --> 00:03:28,720 state. But here I'm referring to it as a 88 00:03:28,720 --> 00:03:31,840 code model. That desired state is then 89 00:03:31,840 --> 00:03:34,799 applied uh by representing that from the 90 00:03:34,799 --> 00:03:36,319 state model by our tooling. In this 91 00:03:36,319 --> 00:03:38,159 case, Palumi, sometimes Terraform, a 92 00:03:38,159 --> 00:03:40,000 couple of other options available, which 93 00:03:40,000 --> 00:03:42,239 results in real resources or the actions 94 00:03:42,239 --> 00:03:43,680 that you want to be performed against 95 00:03:43,680 --> 00:03:45,519 your cloud 96 00:03:45,519 --> 00:03:47,680 because nothing can ever be that simple. 97 00:03:47,680 --> 00:03:49,280 Uh you then need to keep in mind that 98 00:03:49,280 --> 00:03:51,120 what exists in any of these categories 99 00:03:51,120 --> 00:03:52,879 can be changed out of band. Someone 100 00:03:52,879 --> 00:03:54,560 might modify your resources that exist 101 00:03:54,560 --> 00:03:57,599 in AWS. You also need to know that you 102 00:03:57,599 --> 00:03:59,360 can modify your state directly as well. 103 00:03:59,360 --> 00:04:01,280 So when you take when you change any one 104 00:04:01,280 --> 00:04:02,959 of these, you need to know the impact 105 00:04:02,959 --> 00:04:04,640 it's it's going to have on the others 106 00:04:04,640 --> 00:04:06,400 and on your production environment and 107 00:04:06,400 --> 00:04:09,959 thus your users. 108 00:04:10,000 --> 00:04:12,239 Automation first is the philosophy that 109 00:04:12,239 --> 00:04:13,920 if you need to change something or you 110 00:04:13,920 --> 00:04:15,360 need to do something, you will always do 111 00:04:15,360 --> 00:04:17,519 it with automation from the beginning. 112 00:04:17,519 --> 00:04:20,239 No click ops here. Uh you might choose 113 00:04:20,239 --> 00:04:21,840 this approach for many reasons. 114 00:04:21,840 --> 00:04:23,280 Personally, I love it when my 115 00:04:23,280 --> 00:04:25,280 infrastructure problems become software 116 00:04:25,280 --> 00:04:28,560 problems. Uh we bought into this pretty 117 00:04:28,560 --> 00:04:31,040 hard outside of incident response. We 118 00:04:31,040 --> 00:04:32,800 don't allow manual changes to our 119 00:04:32,800 --> 00:04:34,960 environments. Uh and entire replicas of 120 00:04:34,960 --> 00:04:36,400 our production environment are brought 121 00:04:36,400 --> 00:04:39,120 up and down daily for our development CI 122 00:04:39,120 --> 00:04:41,680 and staging stacks which really helps 123 00:04:41,680 --> 00:04:44,560 with our AWS bill. We currently draw the 124 00:04:44,560 --> 00:04:46,240 line though at automating the creation 125 00:04:46,240 --> 00:04:48,320 and the deletion of AWS accounts 126 00:04:48,320 --> 00:04:51,040 themselves and we do have a lot of AWS 127 00:04:51,040 --> 00:04:53,199 accounts. 128 00:04:53,199 --> 00:04:55,680 That brings us to the early days of 129 00:04:55,680 --> 00:04:58,000 building the platform. 130 00:04:58,000 --> 00:05:00,479 So our initial problem was how do we 131 00:05:00,479 --> 00:05:02,400 start an approach of deploying any of 132 00:05:02,400 --> 00:05:05,520 this at all? Um and to avoid complexity, 133 00:05:05,520 --> 00:05:07,280 our deployment pipeline, we decided on 134 00:05:07,280 --> 00:05:09,039 something pretty straightforward. Uh 135 00:05:09,039 --> 00:05:10,960 after building our code, we'd run a 136 00:05:10,960 --> 00:05:12,479 preview of the resources that it was 137 00:05:12,479 --> 00:05:13,840 going to change in our environment and 138 00:05:13,840 --> 00:05:16,240 then run a deploy. Uh and that would all 139 00:05:16,240 --> 00:05:19,600 be in one stack and all in one codebase. 140 00:05:19,600 --> 00:05:22,080 uh in retrospect that didn't actually go 141 00:05:22,080 --> 00:05:25,600 so well for us. 142 00:05:25,600 --> 00:05:27,520 We also needed to bring some sort of 143 00:05:27,520 --> 00:05:29,520 structure to our code that matched our 144 00:05:29,520 --> 00:05:32,320 conceptual model of the platform. 145 00:05:32,320 --> 00:05:34,720 Conceptually, we divide our platform up 146 00:05:34,720 --> 00:05:37,039 into things we call cells where each 147 00:05:37,039 --> 00:05:39,759 cell is more or less an AWS account and 148 00:05:39,759 --> 00:05:41,520 some other resources that handle a 149 00:05:41,520 --> 00:05:44,320 particular function. A few examples are 150 00:05:44,320 --> 00:05:47,360 our security log archive, bakery uh and 151 00:05:47,360 --> 00:05:48,720 tenant cell where our customer 152 00:05:48,720 --> 00:05:50,880 infrastructure is is held. Um, 153 00:05:50,880 --> 00:05:53,440 annoyingly our conceptual model has 154 00:05:53,440 --> 00:05:56,479 circular dependencies and thus so does 155 00:05:56,479 --> 00:05:58,400 our code. 156 00:05:58,400 --> 00:06:00,560 One example of this uh is in our log 157 00:06:00,560 --> 00:06:02,960 archival mechanisms. All of our cells 158 00:06:02,960 --> 00:06:05,680 send logs to a cell for log archival 159 00:06:05,680 --> 00:06:07,680 which goes into a bucket in that. But 160 00:06:07,680 --> 00:06:09,759 the log archive cell uses an encryption 161 00:06:09,759 --> 00:06:11,840 key and we centralize some encryption 162 00:06:11,840 --> 00:06:14,400 material in our security cell and it it 163 00:06:14,400 --> 00:06:16,240 is that case for the encryption key for 164 00:06:16,240 --> 00:06:18,960 our log archival which is kind of a 165 00:06:18,960 --> 00:06:22,080 problem. Um but then it gets worse 166 00:06:22,080 --> 00:06:24,160 because this is super simplified and 167 00:06:24,160 --> 00:06:26,240 this is still a simplified model of the 168 00:06:26,240 --> 00:06:28,479 interdependencies our environment has. 169 00:06:28,479 --> 00:06:30,000 So gives you a bit of an idea of how 170 00:06:30,000 --> 00:06:32,800 much worse it can get. 171 00:06:32,800 --> 00:06:34,639 It will come as no surprise to anyone 172 00:06:34,639 --> 00:06:36,080 here that when you deliberately create 173 00:06:36,080 --> 00:06:38,080 some classes with circular dependencies 174 00:06:38,080 --> 00:06:40,000 uh between their class attributes, you 175 00:06:40,000 --> 00:06:45,080 get an attribute error. 176 00:06:46,080 --> 00:06:49,759 Where have I gone on things? Sorry. 177 00:06:49,759 --> 00:06:51,360 Uh and of course there's no way to 178 00:06:51,360 --> 00:06:53,280 actually make this work. Even if I 179 00:06:53,280 --> 00:06:55,440 switch the order of execution, it's 180 00:06:55,440 --> 00:06:58,880 still going to be broken. 181 00:06:58,880 --> 00:07:01,360 We of course immediately decided to 182 00:07:01,360 --> 00:07:02,479 design support for circular 183 00:07:02,479 --> 00:07:06,319 dependencies. I wish I was joking. 184 00:07:06,319 --> 00:07:08,319 The resource graph we need we we need to 185 00:07:08,319 --> 00:07:10,880 declare for Palumi to interpret is quite 186 00:07:10,880 --> 00:07:13,919 sensibly not cyclical but it does use 187 00:07:13,919 --> 00:07:16,319 async Python. So we need to build a 188 00:07:16,319 --> 00:07:18,800 layer on top of that. You may recall at 189 00:07:18,800 --> 00:07:20,639 this point that I said we did not manage 190 00:07:20,639 --> 00:07:24,120 to avoid complexity. 191 00:07:24,560 --> 00:07:26,080 At this point, we need to take a slight 192 00:07:26,080 --> 00:07:28,560 detour and talk about async Python. Uh 193 00:07:28,560 --> 00:07:30,560 we're going to build uh classes 194 00:07:30,560 --> 00:07:32,400 representing a graph which is then going 195 00:07:32,400 --> 00:07:35,039 to be executed. It's all in pure black 196 00:07:35,039 --> 00:07:36,880 Python here. In the real world, Palumi 197 00:07:36,880 --> 00:07:38,160 does this, but you know, I love to 198 00:07:38,160 --> 00:07:40,639 reinvent the wheel. 199 00:07:40,639 --> 00:07:42,400 First, we have output, which is an 200 00:07:42,400 --> 00:07:44,720 awaitable. It represents a value that 201 00:07:44,720 --> 00:07:46,639 will be known at some point uh in the 202 00:07:46,639 --> 00:07:48,479 future, but we don't know it yet. Most 203 00:07:48,479 --> 00:07:50,560 of the work here is actually uh done by 204 00:07:50,560 --> 00:07:52,080 async.io. You can see we're inheriting 205 00:07:52,080 --> 00:07:54,400 from it here. uh and that provides 206 00:07:54,400 --> 00:07:56,479 methods like set result to set the real 207 00:07:56,479 --> 00:07:58,639 value on this object once it is ready. 208 00:07:58,639 --> 00:08:00,240 I'm just going to load this into the 209 00:08:00,240 --> 00:08:03,440 interpreter. Uh we all we also exam 210 00:08:03,440 --> 00:08:05,120 implement a few helpers on this and a 211 00:08:05,120 --> 00:08:07,199 similar manner to Palumi um with 212 00:08:07,199 --> 00:08:08,960 recursive attribute lifting and 213 00:08:08,960 --> 00:08:11,199 unwrapping when we await the result of 214 00:08:11,199 --> 00:08:13,759 this output. I don't expect anyone to 215 00:08:13,759 --> 00:08:15,360 like completely digest this code. We're 216 00:08:15,360 --> 00:08:16,639 just building a framework which we're 217 00:08:16,639 --> 00:08:18,879 then going to use later. All you really 218 00:08:18,879 --> 00:08:20,479 need to remember from this demo is that 219 00:08:20,479 --> 00:08:22,479 an output can be pending or resolved and 220 00:08:22,479 --> 00:08:24,000 it's only resolved once we actually know 221 00:08:24,000 --> 00:08:27,560 the real value. 222 00:08:29,120 --> 00:08:31,919 Uh next we have resource. Uh this is 223 00:08:31,919 --> 00:08:34,000 standing in for the multitude of classes 224 00:08:34,000 --> 00:08:35,599 that Palumi's libraries provide for 225 00:08:35,599 --> 00:08:37,440 resources like EC2 instances and S3 226 00:08:37,440 --> 00:08:38,959 buckets. We're oversimplifying our model 227 00:08:38,959 --> 00:08:41,360 to just adding a single one. Uh on 228 00:08:41,360 --> 00:08:43,360 construction resources take references 229 00:08:43,360 --> 00:08:45,040 to the inputs that they will need so 230 00:08:45,040 --> 00:08:46,560 that they can be created and those 231 00:08:46,560 --> 00:08:49,519 inputs can be yet to be resolved outputs 232 00:08:49,519 --> 00:08:51,920 from other resources. 233 00:08:51,920 --> 00:08:54,240 Load this one too. 234 00:08:54,240 --> 00:08:55,760 They also register themselves with a 235 00:08:55,760 --> 00:08:57,040 deployment class which we can see 236 00:08:57,040 --> 00:08:59,120 happening here. Uh and that deployment 237 00:08:59,120 --> 00:09:00,240 class which is what we're going to do on 238 00:09:00,240 --> 00:09:02,160 the next slide. Uh we'll be making sure 239 00:09:02,160 --> 00:09:04,959 that the create method is called. When 240 00:09:04,959 --> 00:09:07,200 the async me async create method is 241 00:09:07,200 --> 00:09:09,440 called, a resource waits until all of 242 00:09:09,440 --> 00:09:11,200 its inputs have been resolved and then 243 00:09:11,200 --> 00:09:13,839 it assigns a real value to its output ID 244 00:09:13,839 --> 00:09:16,399 and prints uh to its output output ID 245 00:09:16,399 --> 00:09:17,920 attribute which we can see defined here 246 00:09:17,920 --> 00:09:21,341 and then set here. Uh and 247 00:09:21,341 --> 00:09:22,640 [clears throat] it will also print some 248 00:09:22,640 --> 00:09:24,080 log messages so we can see the creation 249 00:09:24,080 --> 00:09:27,440 ordering of things. 250 00:09:27,440 --> 00:09:29,839 The last piece of our graph machinery uh 251 00:09:29,839 --> 00:09:32,160 is the executor of the graph. This 252 00:09:32,160 --> 00:09:33,839 singleton collects a list of resources 253 00:09:33,839 --> 00:09:35,440 as they're declared and then when we 254 00:09:35,440 --> 00:09:37,519 call execute schedules a call of all the 255 00:09:37,519 --> 00:09:39,120 create methods and waits for all of them 256 00:09:39,120 --> 00:09:42,600 to be created. 257 00:09:44,880 --> 00:09:47,279 Putting our new machinery to use, we can 258 00:09:47,279 --> 00:09:49,360 create a simple graph for execution of 259 00:09:49,360 --> 00:09:50,880 the three resources that need to know 260 00:09:50,880 --> 00:09:52,560 the ID of the previous resource before 261 00:09:52,560 --> 00:09:56,040 they can be created. 262 00:09:57,120 --> 00:09:58,959 We can see in our logs that we declared 263 00:09:58,959 --> 00:10:02,320 three resources. 1 2 3. And then we 264 00:10:02,320 --> 00:10:04,720 created those three three resources. 1 2 265 00:10:04,720 --> 00:10:07,600 3. 266 00:10:07,600 --> 00:10:09,519 At this point, I may appear a little bit 267 00:10:09,519 --> 00:10:12,480 like this. Um, but please bear with me. 268 00:10:12,480 --> 00:10:15,120 We're getting there. 269 00:10:15,120 --> 00:10:17,360 Returning to where we were before. Let's 270 00:10:17,360 --> 00:10:19,680 update our example code to use the new 271 00:10:19,680 --> 00:10:21,200 classes. 272 00:10:21,200 --> 00:10:23,680 Here's an example I prepared earlier. So 273 00:10:23,680 --> 00:10:27,240 when we run it, 274 00:10:32,640 --> 00:10:34,720 well, turns out we're not quite done 275 00:10:34,720 --> 00:10:36,240 yet. 276 00:10:36,240 --> 00:10:38,320 We need to introduce functionality in a 277 00:10:38,320 --> 00:10:41,040 parent class for our cell classes. Here 278 00:10:41,040 --> 00:10:43,440 at the top, we have a tpple that will be 279 00:10:43,440 --> 00:10:45,120 the names of attributes that we will 280 00:10:45,120 --> 00:10:48,800 export. Then at construction time, we 281 00:10:48,800 --> 00:10:51,600 make output instances for each of them. 282 00:10:51,600 --> 00:10:53,360 uh and we store those on our internal 283 00:10:53,360 --> 00:10:55,839 object dictionary. We also hook into the 284 00:10:55,839 --> 00:10:57,920 magic set atra method so that when we 285 00:10:57,920 --> 00:10:59,440 actually go to set the real value of 286 00:10:59,440 --> 00:11:01,360 those exported outputs instead of 287 00:11:01,360 --> 00:11:03,440 assigning the attribute to our class we 288 00:11:03,440 --> 00:11:04,959 set the result on the corresponding 289 00:11:04,959 --> 00:11:09,880 output for output object for it. 290 00:11:11,120 --> 00:11:13,680 With that now in place our cell classes 291 00:11:13,680 --> 00:11:15,519 need to be defined knowing the attribute 292 00:11:15,519 --> 00:11:18,079 names to export and now inheriting from 293 00:11:18,079 --> 00:11:21,600 the parent cell class. 294 00:11:21,600 --> 00:11:23,279 which we can see we've got our exports 295 00:11:23,279 --> 00:11:25,519 named uh and that's really the only 296 00:11:25,519 --> 00:11:26,959 change export name and inheritance from 297 00:11:26,959 --> 00:11:29,920 cell. 298 00:11:29,920 --> 00:11:32,160 Okay, now everyone take a deep breath, 299 00:11:32,160 --> 00:11:35,640 cross your fingers. 300 00:11:37,040 --> 00:11:40,320 We got it. Three resources declared, 301 00:11:40,320 --> 00:11:42,560 three resources created. If we look at 302 00:11:42,560 --> 00:11:44,160 the ordering though, we can see that the 303 00:11:44,160 --> 00:11:45,760 order they're declared in is different 304 00:11:45,760 --> 00:11:48,160 to the order they're created. See, 305 00:11:48,160 --> 00:11:50,240 create the declare the KMS key, declare 306 00:11:50,240 --> 00:11:52,000 the logger, declare the bucket, create 307 00:11:52,000 --> 00:11:54,480 the KMS key, create the bucket, create 308 00:11:54,480 --> 00:11:57,360 the logger. So, what on earth just 309 00:11:57,360 --> 00:12:00,640 happened? Well, the resources for 310 00:12:00,640 --> 00:12:02,640 creation still have a strict ordering 311 00:12:02,640 --> 00:12:04,959 that they need to be created in. This is 312 00:12:04,959 --> 00:12:07,680 inherently a cyclic. You have to do one 313 00:12:07,680 --> 00:12:09,279 added like one, but before you can do 314 00:12:09,279 --> 00:12:10,880 the next before you can do the next. All 315 00:12:10,880 --> 00:12:12,800 that we've done is build a layer on top 316 00:12:12,800 --> 00:12:15,760 of that that allows us to represent our 317 00:12:15,760 --> 00:12:17,920 cyclical model between our cell classes 318 00:12:17,920 --> 00:12:20,160 but still create an asyclic model 319 00:12:20,160 --> 00:12:23,440 between our resources. 320 00:12:23,440 --> 00:12:25,920 Hopefully you like me are beginning to 321 00:12:25,920 --> 00:12:29,360 see how cool this can be. 322 00:12:29,360 --> 00:12:32,720 Too bad that's not what we actually did. 323 00:12:32,720 --> 00:12:34,240 Due to constraints at the time we built 324 00:12:34,240 --> 00:12:36,079 this. The real approach we to talk took 325 00:12:36,079 --> 00:12:37,920 looks a little bit different is a little 326 00:12:37,920 --> 00:12:40,000 less approachable. uh and we track 327 00:12:40,000 --> 00:12:42,160 outputs more centrally. That said, it's 328 00:12:42,160 --> 00:12:44,240 still only about 100 lines and we 329 00:12:44,240 --> 00:12:45,920 haven't really needed to change it in 330 00:12:45,920 --> 00:12:48,480 about 5 years. But while we might lament 331 00:12:48,480 --> 00:12:50,399 what could have been, we're not going to 332 00:12:50,399 --> 00:12:53,839 dwell on the past too much. 333 00:12:53,839 --> 00:12:55,920 Now, let's talk about what happens when 334 00:12:55,920 --> 00:12:58,880 things get bigger. 335 00:12:58,880 --> 00:13:00,000 You might have thought we traded 336 00:13:00,000 --> 00:13:02,240 complexity in the deployment for 337 00:13:02,240 --> 00:13:04,639 complexity in the code model. Surprise! 338 00:13:04,639 --> 00:13:07,360 We ended up needing complexity in both. 339 00:13:07,360 --> 00:13:09,279 As customer numbers and features 340 00:13:09,279 --> 00:13:11,120 features we offer increased, so did the 341 00:13:11,120 --> 00:13:12,880 execution time to preview and deploy our 342 00:13:12,880 --> 00:13:14,480 stacks, which can become a real 343 00:13:14,480 --> 00:13:16,720 operational problem. It's also more 344 00:13:16,720 --> 00:13:18,560 tricky to gate certain features and do 345 00:13:18,560 --> 00:13:20,240 progressive rollouts when everything is 346 00:13:20,240 --> 00:13:22,880 in just one big giant stack. So, we 347 00:13:22,880 --> 00:13:24,720 decided to split it up into smaller 348 00:13:24,720 --> 00:13:26,240 stacks, 349 00:13:26,240 --> 00:13:30,000 which introduced more complexity. 350 00:13:30,000 --> 00:13:31,920 We split things up into what we call a 351 00:13:31,920 --> 00:13:34,000 backplane stack and multiple module 352 00:13:34,000 --> 00:13:36,399 stacks. To keep things simple, famous 353 00:13:36,399 --> 00:13:38,720 last words, the corresponding code 354 00:13:38,720 --> 00:13:40,800 changes just shuffled some resource 355 00:13:40,800 --> 00:13:42,800 declarations around but kept them in the 356 00:13:42,800 --> 00:13:45,440 same class. 357 00:13:45,440 --> 00:13:47,680 We then did some fancy Palumi state 358 00:13:47,680 --> 00:13:50,800 manipulation to move uh the resources we 359 00:13:50,800 --> 00:13:52,720 needed to a new Palumi stack with the 360 00:13:52,720 --> 00:13:54,800 execution of backplane and module stacks 361 00:13:54,800 --> 00:13:56,720 being in completely different pipeline 362 00:13:56,720 --> 00:13:58,800 stages. 363 00:13:58,800 --> 00:14:01,440 However, the fun didn't end there. 364 00:14:01,440 --> 00:14:02,720 Because they're executing in different 365 00:14:02,720 --> 00:14:04,560 stacks, of course, you can't access 366 00:14:04,560 --> 00:14:06,079 attributes that are being set by a 367 00:14:06,079 --> 00:14:07,279 method that isn't called in the 368 00:14:07,279 --> 00:14:09,680 interpreter that you're running that in. 369 00:14:09,680 --> 00:14:11,760 So, we ended up using an AWS SSM 370 00:14:11,760 --> 00:14:13,600 parameter that we call a pin out to 371 00:14:13,600 --> 00:14:15,440 propagate necessary information between 372 00:14:15,440 --> 00:14:16,800 the backplane stack and the module 373 00:14:16,800 --> 00:14:19,920 stacks. Having gifted ourselves this 374 00:14:19,920 --> 00:14:22,480 amazing level of complexity that brings 375 00:14:22,480 --> 00:14:26,600 us to the present 376 00:14:27,600 --> 00:14:29,440 with all of this, the codebase can be 377 00:14:29,440 --> 00:14:31,920 pretty confusing uh and au pretty 378 00:14:31,920 --> 00:14:33,920 confusing jungle to navigate and we 379 00:14:33,920 --> 00:14:36,399 would like to fix that. Unfortunately, 380 00:14:36,399 --> 00:14:38,480 we have active development on all of the 381 00:14:38,480 --> 00:14:40,000 areas of the codebase that we want to 382 00:14:40,000 --> 00:14:42,399 restructure. So, we need to work out how 383 00:14:42,399 --> 00:14:43,920 to do it peacemeal without blocking 384 00:14:43,920 --> 00:14:45,680 day-to-day operations. And remember 385 00:14:45,680 --> 00:14:47,120 earlier, if you change one of the 386 00:14:47,120 --> 00:14:48,399 models, you have to think about how it 387 00:14:48,399 --> 00:14:49,839 changes the other and have a migration 388 00:14:49,839 --> 00:14:52,320 path between. 389 00:14:52,320 --> 00:14:54,320 I'd have to say I've had far more fun 390 00:14:54,320 --> 00:14:55,760 creating the technical debt than 391 00:14:55,760 --> 00:14:59,160 cleaning it up 392 00:14:59,279 --> 00:15:01,360 in our codebase. Currently, the cell 393 00:15:01,360 --> 00:15:02,800 classes depend on resources that are 394 00:15:02,800 --> 00:15:04,880 created in their parent too. So in this 395 00:15:04,880 --> 00:15:06,240 example, you might notice we have a 396 00:15:06,240 --> 00:15:07,760 certificate that gets created in a 397 00:15:07,760 --> 00:15:09,680 parent cell class that's part of the uh 398 00:15:09,680 --> 00:15:11,920 module. And then that is used in a child 399 00:15:11,920 --> 00:15:14,079 tenant class that also happens to deploy 400 00:15:14,079 --> 00:15:15,360 GitLab and a load balance of it. It 401 00:15:15,360 --> 00:15:17,360 needs a certificate to do that. It's 402 00:15:17,360 --> 00:15:19,120 contrived example from our codebase. Not 403 00:15:19,120 --> 00:15:20,560 oversimplified, but hopefully you get 404 00:15:20,560 --> 00:15:24,160 the point. We'd like to end up here with 405 00:15:24,160 --> 00:15:25,360 a structure that's far more 406 00:15:25,360 --> 00:15:27,120 representative of how we conceptually 407 00:15:27,120 --> 00:15:29,279 model the cells in our way we think 408 00:15:29,279 --> 00:15:31,839 about the environment. Separate cell 409 00:15:31,839 --> 00:15:33,279 separate classes for cells that are 410 00:15:33,279 --> 00:15:34,880 modules and cells that are backplanes 411 00:15:34,880 --> 00:15:36,560 means we can split these over a file 412 00:15:36,560 --> 00:15:38,560 each. So maybe we can start to avoid 413 00:15:38,560 --> 00:15:42,880 files that are over 5,000 lines long. 414 00:15:42,880 --> 00:15:45,040 We have to find a way to unify these 415 00:15:45,040 --> 00:15:47,120 unrelated class hierarchies so we can 416 00:15:47,120 --> 00:15:49,120 start moving things between them a bit 417 00:15:49,120 --> 00:15:51,759 at a time. Wouldn't it be cool if we 418 00:15:51,759 --> 00:15:53,279 could just inherit from both of these 419 00:15:53,279 --> 00:15:56,800 class hierarchies at once? You can. You 420 00:15:56,800 --> 00:15:58,480 can list multiple class hier like 421 00:15:58,480 --> 00:16:00,079 multiple classes to inherit from when 422 00:16:00,079 --> 00:16:02,240 you declare a class and you get all of 423 00:16:02,240 --> 00:16:04,720 their functionality which is great. See 424 00:16:04,720 --> 00:16:06,720 if we run this we will get all of the 425 00:16:06,720 --> 00:16:08,959 attributes defined in all of these run 426 00:16:08,959 --> 00:16:11,040 methods across our class A, class B and 427 00:16:11,040 --> 00:16:13,519 class C 428 00:16:13,519 --> 00:16:17,320 if the code cooperates. 429 00:16:19,279 --> 00:16:21,279 Like people, these classes need to 430 00:16:21,279 --> 00:16:23,360 cooperate in order for this to be a 431 00:16:23,360 --> 00:16:26,399 success. Class A needs to know how to 432 00:16:26,399 --> 00:16:28,800 behave when cooperative inheritance is 433 00:16:28,800 --> 00:16:31,839 required. And class C needs to let class 434 00:16:31,839 --> 00:16:34,560 A know that cooperation is in fact 435 00:16:34,560 --> 00:16:36,800 required. What this actually looks like 436 00:16:36,800 --> 00:16:39,279 is class A knowing that it needs to call 437 00:16:39,279 --> 00:16:42,320 the run method on its superass when 438 00:16:42,320 --> 00:16:44,240 operating under a coop cooperative 439 00:16:44,240 --> 00:16:46,160 inheritance model. Even though that 440 00:16:46,160 --> 00:16:48,000 would usually be completely invalid 441 00:16:48,000 --> 00:16:49,839 because class class A doesn't actually 442 00:16:49,839 --> 00:16:51,519 inherit from anything and its superass 443 00:16:51,519 --> 00:16:55,600 of object doesn't have a run method. 444 00:16:56,720 --> 00:16:58,880 In this example, the signal from class C 445 00:16:58,880 --> 00:17:01,120 to class A that cooperative inheritance 446 00:17:01,120 --> 00:17:03,120 is in fact required is setting the class 447 00:17:03,120 --> 00:17:05,679 attribute cooperative to true where we 448 00:17:05,679 --> 00:17:07,679 are building class C which is then read 449 00:17:07,679 --> 00:17:09,520 in the run method on class A so it knows 450 00:17:09,520 --> 00:17:11,280 to dispatch to the previously declared 451 00:17:11,280 --> 00:17:15,640 on a previous slide class B. 452 00:17:16,720 --> 00:17:18,959 All right, conceptually we're here. 453 00:17:18,959 --> 00:17:21,199 Let's begin refactoring. I've got an 454 00:17:21,199 --> 00:17:24,000 example of our module, our various 455 00:17:24,000 --> 00:17:26,480 module sort of cell structure here. 456 00:17:26,480 --> 00:17:28,640 First, we can move the resources at the 457 00:17:28,640 --> 00:17:31,760 end. So, we can move one from the old 458 00:17:31,760 --> 00:17:33,280 structure to the new structure. Uh, and 459 00:17:33,280 --> 00:17:34,960 in between all of these moves, a bunch 460 00:17:34,960 --> 00:17:36,880 of other work might be going on in the 461 00:17:36,880 --> 00:17:38,960 surrounding codebase. Once you've done 462 00:17:38,960 --> 00:17:41,280 that, we can move the load balancer. 463 00:17:41,280 --> 00:17:43,520 That leaves our old tenant class empty. 464 00:17:43,520 --> 00:17:45,280 And now because that class is empty and 465 00:17:45,280 --> 00:17:46,960 there's nothing in it still that is 466 00:17:46,960 --> 00:17:48,720 necessary, we can start looking at its 467 00:17:48,720 --> 00:17:50,320 parent and we can move the certificate 468 00:17:50,320 --> 00:17:53,520 too over in the other file. 469 00:17:53,520 --> 00:17:56,832 Then we can clean up after ourselves. 470 00:17:56,832 --> 00:17:58,080 [clears throat] First the old tenant 471 00:17:58,080 --> 00:18:00,480 class and the uh migration class both 472 00:18:00,480 --> 00:18:02,640 go. Then the old cell class and the 473 00:18:02,640 --> 00:18:04,960 cooperative flag goes. And now we're 474 00:18:04,960 --> 00:18:07,960 done. 475 00:18:08,000 --> 00:18:12,240 It's never never that simple. 476 00:18:12,240 --> 00:18:14,720 This one really does a number on type 477 00:18:14,720 --> 00:18:16,320 checkers. 478 00:18:16,320 --> 00:18:17,919 They really hate when you access 479 00:18:17,919 --> 00:18:20,799 attributes on self that don't exist at 480 00:18:20,799 --> 00:18:23,440 static analysis time. We've also 481 00:18:23,440 --> 00:18:25,120 oversimplified our examples for this 482 00:18:25,120 --> 00:18:27,280 talk because we skipped refactoring the 483 00:18:27,280 --> 00:18:29,919 back pain backplane class. We have uh in 484 00:18:29,919 --> 00:18:32,640 the real world we also have a bunch more 485 00:18:32,640 --> 00:18:37,440 methods, a bunch more classes. We use K 486 00:18:37,440 --> 00:18:40,320 properties and a few things like that. 487 00:18:40,320 --> 00:18:44,480 But that's a regret for future me. 488 00:18:44,480 --> 00:18:46,240 I think that's all we have time for 489 00:18:46,240 --> 00:18:47,679 today or all I have for to give you 490 00:18:47,679 --> 00:18:49,360 today. Uh and I hope your brains have 491 00:18:49,360 --> 00:18:51,200 been sufficiently scratched by this 492 00:18:51,200 --> 00:18:53,120 talk. I'll be around the conference uh 493 00:18:53,120 --> 00:18:54,960 and at the XRD booth uh if you want to 494 00:18:54,960 --> 00:18:56,400 ask some questions but we might also 495 00:18:56,400 --> 00:18:57,840 actually have time for questions in this 496 00:18:57,840 --> 00:19:00,320 session as well. 497 00:19:00,320 --> 00:19:03,043 Thank you. [applause] 498 00:19:03,919 --> 00:19:06,960 Thank you very much. Please take our mug 499 00:19:06,960 --> 00:19:09,280 if it's open appreciation. And yes, we 500 00:19:09,280 --> 00:19:11,600 have time for questions. I think about 5 501 00:19:11,600 --> 00:19:14,160 minutes. Does anyone have a 502 00:19:14,160 --> 00:19:18,760 great question to ask please? Yes. 503 00:19:19,520 --> 00:19:23,200 Hello. Thank you. Very good talk. Um how 504 00:19:23,200 --> 00:19:25,520 do you see this kind of refactoring 505 00:19:25,520 --> 00:19:27,840 fitting in with you know the new way of 506 00:19:27,840 --> 00:19:32,559 working with AI and agentic workflows 507 00:19:32,559 --> 00:19:34,240 in doing this which like we're in the 508 00:19:34,240 --> 00:19:35,200 middle of it. That's why it's the 509 00:19:35,200 --> 00:19:37,600 present. Um, I've actually found that 510 00:19:37,600 --> 00:19:39,520 using tools like Claude Code and some of 511 00:19:39,520 --> 00:19:42,160 the more powerful models, uh, you do 512 00:19:42,160 --> 00:19:43,919 have to put a lot of effort into making 513 00:19:43,919 --> 00:19:46,559 it understand uh, what you're trying to 514 00:19:46,559 --> 00:19:48,880 do. Um, but once you have a a good 515 00:19:48,880 --> 00:19:50,559 description and some minimal 516 00:19:50,559 --> 00:19:52,000 representations of what you're trying it 517 00:19:52,000 --> 00:19:53,679 to do, it's actually not too bad at 518 00:19:53,679 --> 00:19:55,280 expanding that to a larger scale and 519 00:19:55,280 --> 00:19:58,559 giving you suggestions. Um, perhaps this 520 00:19:58,559 --> 00:20:00,080 is actually a relatively useful problem 521 00:20:00,080 --> 00:20:01,760 for it because we're not asking it to do 522 00:20:01,760 --> 00:20:02,880 something particularly giant and 523 00:20:02,880 --> 00:20:04,480 complicated. we have a good example of 524 00:20:04,480 --> 00:20:05,840 what we want to do. We want to move a 525 00:20:05,840 --> 00:20:07,280 very small thing to over here. And 526 00:20:07,280 --> 00:20:08,640 that's a pull request on its own. We're 527 00:20:08,640 --> 00:20:10,559 not doing them all at once. Uh and so 528 00:20:10,559 --> 00:20:13,200 the actual code changes each time are 529 00:20:13,200 --> 00:20:16,080 relatively minor. Um and so a good thing 530 00:20:16,080 --> 00:20:17,840 in reducing the tech debt is I find that 531 00:20:17,840 --> 00:20:21,200 I can have 10 minutes, 30 minutes, and I 532 00:20:21,200 --> 00:20:22,559 can go and I can be like, all right, 533 00:20:22,559 --> 00:20:24,160 what's the next attribute in this 534 00:20:24,160 --> 00:20:25,600 particular class that I care about that 535 00:20:25,600 --> 00:20:28,000 needs to move over? Within 30 minutes, I 536 00:20:28,000 --> 00:20:29,679 have a pull request ready to do that. 537 00:20:29,679 --> 00:20:31,039 Maybe even 5 minutes. I have a pull 538 00:20:31,039 --> 00:20:33,200 request ready up to do that. and I'm 539 00:20:33,200 --> 00:20:34,799 like progressively moving the code base 540 00:20:34,799 --> 00:20:36,640 into the state we want to be while 541 00:20:36,640 --> 00:20:38,480 everyone else is still and I'm still 542 00:20:38,480 --> 00:20:40,559 developing other resources on top of it. 543 00:20:40,559 --> 00:20:43,360 So I found it super helpful um because 544 00:20:43,360 --> 00:20:45,360 otherwise I would just never bother uh 545 00:20:45,360 --> 00:20:46,880 doing these these tiny bits cuz I'm 546 00:20:46,880 --> 00:20:50,400 always working on larger problems. 547 00:20:50,400 --> 00:20:52,640 Another question. 548 00:20:52,640 --> 00:20:54,799 Hello. Um I remember at the very 549 00:20:54,799 --> 00:20:57,280 beginning you said that you destroy your 550 00:20:57,280 --> 00:20:59,760 um staging dev and CI environments uh 551 00:20:59,760 --> 00:21:02,159 every day. Um, but then I'm also 552 00:21:02,159 --> 00:21:04,240 noticing how you're refactoring how 553 00:21:04,240 --> 00:21:07,200 you're deploying to customers. Um, how 554 00:21:07,200 --> 00:21:08,799 if you're destroying the staging 555 00:21:08,799 --> 00:21:10,799 environments before every time, how are 556 00:21:10,799 --> 00:21:12,320 you testing that what you're going to 557 00:21:12,320 --> 00:21:14,400 deploy after you refactor actually 558 00:21:14,400 --> 00:21:16,159 deploys to a production customer which 559 00:21:16,159 --> 00:21:19,200 is pre-existing, not newly created? Yep. 560 00:21:19,200 --> 00:21:20,799 So all of those environments before we 561 00:21:20,799 --> 00:21:22,720 merge any changes, we bring them up in 562 00:21:22,720 --> 00:21:24,799 the old state. 563 00:21:24,799 --> 00:21:28,320 So uh, hit 6 p.m. all my stacks come 564 00:21:28,320 --> 00:21:30,880 down. hit 6 a.m. all my stacks come up 565 00:21:30,880 --> 00:21:32,799 using the same code that was used to 566 00:21:32,799 --> 00:21:34,480 deploy them yesterday at the end of 567 00:21:34,480 --> 00:21:37,039 yesterday and then I'm allowed to merge 568 00:21:37,039 --> 00:21:39,039 my my next merge request that introduces 569 00:21:39,039 --> 00:21:41,200 a change and the CI pipeline and the 570 00:21:41,200 --> 00:21:43,760 staging stack will test the migration to 571 00:21:43,760 --> 00:21:46,400 the new state. Um because of the way our 572 00:21:46,400 --> 00:21:48,799 CI pipelines are structured, if our CI 573 00:21:48,799 --> 00:21:50,720 stack isn't up to test that migration 574 00:21:50,720 --> 00:21:52,159 pathway, it just completely blocks the 575 00:21:52,159 --> 00:21:55,520 merge in the first place. So we it's a 576 00:21:55,520 --> 00:21:57,360 little bit of a trade-off like um as a 577 00:21:57,360 --> 00:21:58,799 cost saving of only having the ads 578 00:21:58,799 --> 00:22:01,120 stacks up like half the time big cost 579 00:22:01,120 --> 00:22:03,760 saving. Um we just sort of go you're 580 00:22:03,760 --> 00:22:04,960 only really going to be landing the 581 00:22:04,960 --> 00:22:07,919 merge requests inside ours. Um but 582 00:22:07,919 --> 00:22:09,440 that's also not a particularly big deal 583 00:22:09,440 --> 00:22:10,720 either because working with 584 00:22:10,720 --> 00:22:12,799 infrastructure as code you tend to have 585 00:22:12,799 --> 00:22:15,039 sort of a little bit of a slower moving 586 00:22:15,039 --> 00:22:17,679 uh on slower moving cadence of landing 587 00:22:17,679 --> 00:22:19,520 your pull requests. Um and we also have 588 00:22:19,520 --> 00:22:21,200 a relatively smaller team that is 589 00:22:21,200 --> 00:22:24,840 managing anything complicated. 590 00:22:27,039 --> 00:22:29,039 Another question down here. 591 00:22:29,039 --> 00:22:32,480 Keep think uh thank you for the talk. Um 592 00:22:32,480 --> 00:22:33,919 there I was wondering a little bit when 593 00:22:33,919 --> 00:22:37,520 the my in the refactoring stuff like my 594 00:22:37,520 --> 00:22:39,039 understanding is that the end of all 595 00:22:39,039 --> 00:22:41,200 this you sort of have your declarative 596 00:22:41,200 --> 00:22:42,640 um system of like here's the new state 597 00:22:42,640 --> 00:22:45,280 of the AR architecture and therefore 598 00:22:45,280 --> 00:22:47,440 like I suppose for example you know hey 599 00:22:47,440 --> 00:22:49,440 this resource used to exist now long no 600 00:22:49,440 --> 00:22:51,679 longer exists therefore like destroy it. 601 00:22:51,679 --> 00:22:54,960 Um, have you experienced any pain around 602 00:22:54,960 --> 00:22:56,799 like like [clears throat] doing a 603 00:22:56,799 --> 00:22:58,159 refactor and then like for example 604 00:22:58,159 --> 00:22:59,520 accidentally getting rid of a resource 605 00:22:59,520 --> 00:23:00,799 because you were trying to move it? Like 606 00:23:00,799 --> 00:23:02,720 how do you express that notion of moving 607 00:23:02,720 --> 00:23:04,320 in this model? 608 00:23:04,320 --> 00:23:07,760 So the the interesting thing there just 609 00:23:07,760 --> 00:23:10,400 swap the slide back. 610 00:23:10,400 --> 00:23:12,799 You're probably talking about around 611 00:23:12,799 --> 00:23:15,520 this chunk where we start doing these 612 00:23:15,520 --> 00:23:17,919 sort of refactors. So the reason we're 613 00:23:17,919 --> 00:23:19,760 doing it this way is because this 614 00:23:19,760 --> 00:23:21,360 doesn't actually result in a resource 615 00:23:21,360 --> 00:23:22,640 destroy and a resource create in 616 00:23:22,640 --> 00:23:24,559 production because it's all running in 617 00:23:24,559 --> 00:23:27,280 the same codebase. We do need to do some 618 00:23:27,280 --> 00:23:29,600 sort of messing with aliases uh which 619 00:23:29,600 --> 00:23:31,039 you might be familiar from Terraform and 620 00:23:31,039 --> 00:23:32,640 and Palumi has a similar concept 621 00:23:32,640 --> 00:23:34,799 sometimes. Um but often times the 622 00:23:34,799 --> 00:23:37,200 resource identifier uh is actually the 623 00:23:37,200 --> 00:23:39,039 same. It's just where that resource has 624 00:23:39,039 --> 00:23:40,640 been declared in the codebase has moved 625 00:23:40,640 --> 00:23:42,720 which is why this is really nice. So 626 00:23:42,720 --> 00:23:44,080 each of the times that I've landed a 627 00:23:44,080 --> 00:23:45,919 pull request related to one of these, 628 00:23:45,919 --> 00:23:47,760 the preview has actually had no diff to 629 00:23:47,760 --> 00:23:49,360 resources created. It's just my code 630 00:23:49,360 --> 00:23:52,600 that's changed. 631 00:23:53,760 --> 00:23:55,679 Yeah. Yeah. So like preview for one of 632 00:23:55,679 --> 00:23:58,320 these should should and does show no 633 00:23:58,320 --> 00:24:00,000 resources needing change. It's just our 634 00:24:00,000 --> 00:24:02,000 code's cleaned up. Uh and if if it does 635 00:24:02,000 --> 00:24:03,200 show that something's changed, it means 636 00:24:03,200 --> 00:24:07,000 that I've done something wrong. 637 00:24:09,760 --> 00:24:14,679 We have another question. 638 00:24:17,600 --> 00:24:19,679 Um, you mentioned you have a pretty 639 00:24:19,679 --> 00:24:21,360 lightweight developer team. Do you think 640 00:24:21,360 --> 00:24:23,200 this sort of refactor would be more 641 00:24:23,200 --> 00:24:27,400 challenging with a larger team? 642 00:24:27,919 --> 00:24:32,159 Maybe. Um it it it's hard to say because 643 00:24:32,159 --> 00:24:34,159 the nice way the nice thing about doing 644 00:24:34,159 --> 00:24:35,520 this refactoring is that each of the 645 00:24:35,520 --> 00:24:37,919 individual changes are quite small. Um 646 00:24:37,919 --> 00:24:40,799 and so maybe with more developers 647 00:24:40,799 --> 00:24:42,400 actively doing it, we'd get more sort of 648 00:24:42,400 --> 00:24:45,200 incidental small changes. Um but we're 649 00:24:45,200 --> 00:24:47,039 also continuously deploying things in 650 00:24:47,039 --> 00:24:49,679 between. Uh and our re the point of the 651 00:24:49,679 --> 00:24:51,120 refactoring is that we don't actually 652 00:24:51,120 --> 00:24:52,720 change deployed resources for each of 653 00:24:52,720 --> 00:24:55,360 these refactors. So it's kind of easy to 654 00:24:55,360 --> 00:24:57,360 validate that it's uh it's not changing 655 00:24:57,360 --> 00:25:00,320 which is why we're doing it this way. Um 656 00:25:00,320 --> 00:25:03,679 so it might confuse more developers but 657 00:25:03,679 --> 00:25:05,120 overall I think it would still mean the 658 00:25:05,120 --> 00:25:06,799 refactor is progressing at the same 659 00:25:06,799 --> 00:25:09,799 rate. 660 00:25:15,679 --> 00:25:17,279 Believe we have time for one more 661 00:25:17,279 --> 00:25:19,440 question. 662 00:25:19,440 --> 00:25:22,440 Yes. 663 00:25:24,400 --> 00:25:29,600 So, um hindsight being 2020 and um with 664 00:25:29,600 --> 00:25:32,240 the understanding that you probably 665 00:25:32,240 --> 00:25:34,480 chose this approach initially because 666 00:25:34,480 --> 00:25:36,720 more established approaches didn't meet 667 00:25:36,720 --> 00:25:38,240 your needs or what you perceived your 668 00:25:38,240 --> 00:25:40,960 needs to be. If you were to go back and 669 00:25:40,960 --> 00:25:43,039 start this all over again knowing what 670 00:25:43,039 --> 00:25:45,520 you know now, what kind of an approach 671 00:25:45,520 --> 00:25:48,400 would you take? 672 00:25:48,400 --> 00:25:51,520 I think I would end up somewhere pretty 673 00:25:51,520 --> 00:25:54,559 similar. I might have just I might just 674 00:25:54,559 --> 00:25:56,799 skip all of the like hurdles along the 675 00:25:56,799 --> 00:25:59,919 way. Um the real sort of learning that I 676 00:25:59,919 --> 00:26:02,000 got out of actually making this talk is 677 00:26:02,000 --> 00:26:04,000 that a lot of the problems we've faced 678 00:26:04,000 --> 00:26:05,840 along the way are the result of needing 679 00:26:05,840 --> 00:26:08,080 to fix something now and dealing with 680 00:26:08,080 --> 00:26:10,960 the consequences of fixing it later. 681 00:26:10,960 --> 00:26:12,880 I'm not entirely sure whether or not we 682 00:26:12,880 --> 00:26:14,400 would have been able to foresee every 683 00:26:14,400 --> 00:26:15,919 single facet of all the potential 684 00:26:15,919 --> 00:26:17,279 consequences of making every single 685 00:26:17,279 --> 00:26:18,559 choice because it's not like we were 686 00:26:18,559 --> 00:26:21,039 deliberately trying to do something that 687 00:26:21,039 --> 00:26:23,200 resulted in technical debt at any time. 688 00:26:23,200 --> 00:26:24,799 Um, 689 00:26:24,799 --> 00:26:26,400 yeah, if I knew everything that I know 690 00:26:26,400 --> 00:26:28,159 now and I was building it from scratch, 691 00:26:28,159 --> 00:26:29,760 I'd probably just short circuit a bunch 692 00:26:29,760 --> 00:26:31,039 of things and still get to the current 693 00:26:31,039 --> 00:26:35,159 state that we're we're trying to get to. 694 00:26:36,080 --> 00:26:39,840 Wonderful. I believe that is time. Thank 695 00:26:39,840 --> 00:26:41,279 you. Thank you so much Alistair for 696 00:26:41,279 --> 00:26:43,760 giving this awesome talk and please come 697 00:26:43,760 --> 00:26:46,880 back in 698 00:26:46,880 --> 00:26:49,840 7 minutes for our next amazing talk. 699 00:26:49,840 --> 00:26:53,799 Thank you so much. Thank you.