1 00:00:00,000 --> 00:00:03,000 Our 2 00:00:05,239 --> 00:00:07,259 [music] 3 00:00:09,825 --> 00:00:11,845 [music] 4 00:00:15,360 --> 00:00:17,600 next speaker is going to reliably close 5 00:00:17,600 --> 00:00:19,279 out the track for us. Please welcome 6 00:00:19,279 --> 00:00:23,080 Shan to the stage. 7 00:00:24,235 --> 00:00:26,255 [applause] 8 00:00:26,480 --> 00:00:28,320 Good day everybody. 9 00:00:28,320 --> 00:00:30,640 I am positively busting for a wee. So, 10 00:00:30,640 --> 00:00:31,840 this talk's going to have a really 11 00:00:31,840 --> 00:00:34,880 distinctive energy to it. [laughter] 12 00:00:34,880 --> 00:00:36,160 We're getting to the end of the day. Is 13 00:00:36,160 --> 00:00:40,879 everybody feeling a little bit lazy? 14 00:00:40,879 --> 00:00:42,160 All right, cool. Let's get into that 15 00:00:42,160 --> 00:00:43,440 mindset cuz that's what this talk's 16 00:00:43,440 --> 00:00:45,440 going to be about. Uh, let's go. Good 17 00:00:45,440 --> 00:00:49,360 day, everybody. My name is Sean O'Keefe. 18 00:00:49,360 --> 00:00:51,760 I am the founding member at Almadori, a 19 00:00:51,760 --> 00:00:53,600 software reliability cooperative working 20 00:00:53,600 --> 00:00:56,399 with startups and scaleups. I am a proud 21 00:00:56,399 --> 00:00:58,399 father of two beautiful kids, which 22 00:00:58,399 --> 00:01:01,440 means I am very tired. 23 00:01:01,440 --> 00:01:03,280 I'm a humanist, which means I approve 24 00:01:03,280 --> 00:01:04,720 PRs, even if they have spelling 25 00:01:04,720 --> 00:01:07,680 mistakes. I am a digital minimalist, 26 00:01:07,680 --> 00:01:09,360 which doesn't mean I am missing fingers. 27 00:01:09,360 --> 00:01:10,960 It means that I believe each additional 28 00:01:10,960 --> 00:01:13,840 technology I add to my day is a way in 29 00:01:13,840 --> 00:01:17,040 which my day can be ruined. I'm 43 years 30 00:01:17,040 --> 00:01:20,320 of age, which means I am old, 31 00:01:20,320 --> 00:01:22,240 but hopefully still relevant because if 32 00:01:22,240 --> 00:01:24,400 not, I have made a huge mistake coming 33 00:01:24,400 --> 00:01:27,280 to talk to you all about reliability 34 00:01:27,280 --> 00:01:29,920 or more specifically to tell you about 35 00:01:29,920 --> 00:01:32,159 how I think you as a startup or scaleup 36 00:01:32,159 --> 00:01:33,680 should be aiming to have the least 37 00:01:33,680 --> 00:01:36,400 reliability possible. My goal today is 38 00:01:36,400 --> 00:01:38,079 to convince you to only do the 39 00:01:38,079 --> 00:01:39,759 reliability work that stands between 40 00:01:39,759 --> 00:01:41,680 your product's success and failure and 41 00:01:41,680 --> 00:01:43,840 to not invest a single dollar or minute 42 00:01:43,840 --> 00:01:46,159 more. I want to convince you the best 43 00:01:46,159 --> 00:01:47,520 way to do this is by deeply 44 00:01:47,520 --> 00:01:48,960 understanding which parts of your 45 00:01:48,960 --> 00:01:50,880 product are important to the business 46 00:01:50,880 --> 00:01:52,799 and then doing the boring work of making 47 00:01:52,799 --> 00:01:54,640 sure they stay up no matter what. But 48 00:01:54,640 --> 00:01:56,079 before we get into the heavy stuff, 49 00:01:56,079 --> 00:01:59,520 let's warm up. Let's do a quiz. 50 00:01:59,520 --> 00:02:01,040 First one's an easy one. You have two 51 00:02:01,040 --> 00:02:03,280 paths. Option one on the left is a 52 00:02:03,280 --> 00:02:05,840 straight line. Option two on the right 53 00:02:05,840 --> 00:02:07,439 has a bunch of bends. And the question 54 00:02:07,439 --> 00:02:10,080 is, which is the shortest path to take 55 00:02:10,080 --> 00:02:12,720 to get from A to B? Option one or option 56 00:02:12,720 --> 00:02:15,040 two. If everybody can scan the QR code 57 00:02:15,040 --> 00:02:16,720 bottom right and vote for the shortest 58 00:02:16,720 --> 00:02:18,160 path, option one or two. Is everybody 59 00:02:18,160 --> 00:02:20,319 scanned that? Great. Let's jump to the 60 00:02:20,319 --> 00:02:22,800 live results. And yeah, votes coming in 61 00:02:22,800 --> 00:02:24,480 for option one as we'd expect. Things 62 00:02:24,480 --> 00:02:27,200 are taking off. The room's surprisingly 63 00:02:27,200 --> 00:02:29,520 big. And there's our staff engineer 64 00:02:29,520 --> 00:02:30,879 coming in at the very end there with 65 00:02:30,879 --> 00:02:33,440 option two. Congratulations, guys. We 66 00:02:33,440 --> 00:02:35,200 made it. The line straight line is 67 00:02:35,200 --> 00:02:37,680 shorter than the curvy line. 68 00:02:37,680 --> 00:02:38,720 You've done well. And I'm going to mix 69 00:02:38,720 --> 00:02:39,519 it up a little bit. 70 00:02:39,519 --> 00:02:43,440 Is there an elevation change? [laughter] 71 00:02:43,440 --> 00:02:44,800 Whoa. Now you are thinking about this 72 00:02:44,800 --> 00:02:47,360 way harder than I am. [laughter] 73 00:02:47,360 --> 00:02:48,879 We're going to mix it up a little bit. 74 00:02:48,879 --> 00:02:51,200 Similar to op before, option one is a 75 00:02:51,200 --> 00:02:53,920 straight line. [laughter] 76 00:02:53,920 --> 00:02:57,280 Option two is a curvy line. 77 00:02:57,280 --> 00:02:59,120 Only this time along option two, you'll 78 00:02:59,120 --> 00:03:00,800 adopt a number of technologies which 79 00:03:00,800 --> 00:03:02,640 have been featured on tech radars, 80 00:03:02,640 --> 00:03:06,080 hacker news, and in CNCF road maps. Same 81 00:03:06,080 --> 00:03:08,080 as before, which is the shortest way 82 00:03:08,080 --> 00:03:09,680 between points A and B. Has everyone 83 00:03:09,680 --> 00:03:13,280 scanned the QR code? Good. 84 00:03:13,680 --> 00:03:16,000 coming in both sides. No, no, this time 85 00:03:16,000 --> 00:03:18,239 option two is pulling ahead. Okay, 86 00:03:18,239 --> 00:03:19,519 interesting. The room's leading towards 87 00:03:19,519 --> 00:03:21,200 option two, the curvy path. Anyone else 88 00:03:21,200 --> 00:03:23,680 want to put in a vote? 89 00:03:23,680 --> 00:03:26,400 Staff engineer. 90 00:03:26,400 --> 00:03:27,920 It's the time it took him to say it 91 00:03:27,920 --> 00:03:30,720 depends. Now, this time we lean towards 92 00:03:30,720 --> 00:03:32,480 option two, the curvy path with the many 93 00:03:32,480 --> 00:03:34,080 technical detours as being the quickest 94 00:03:34,080 --> 00:03:36,400 path between two points. Now, what would 95 00:03:36,400 --> 00:03:38,239 you say if I told you that the green 96 00:03:38,239 --> 00:03:40,000 lines in the first question and the 97 00:03:40,000 --> 00:03:43,360 second question were in fact the same 98 00:03:43,360 --> 00:03:46,360 length? 99 00:03:46,400 --> 00:03:48,640 You may be tempted to agree with me that 100 00:03:48,640 --> 00:03:52,000 in software we have a problem. The 101 00:03:52,000 --> 00:03:53,360 problem is that we seem to be attracted 102 00:03:53,360 --> 00:03:55,120 to complicated technologies and 103 00:03:55,120 --> 00:03:56,799 practices that may not always be making 104 00:03:56,799 --> 00:03:59,360 our lives easier. We take technology 105 00:03:59,360 --> 00:04:00,879 that's built to solve the problem of 106 00:04:00,879 --> 00:04:02,799 companies that are hundreds of times our 107 00:04:02,799 --> 00:04:04,319 size and use it in our threeperson 108 00:04:04,319 --> 00:04:06,959 startup. And we fail to understand that 109 00:04:06,959 --> 00:04:09,040 many of these technologies are 110 00:04:09,040 --> 00:04:11,680 speculative. They're a bet taken by 111 00:04:11,680 --> 00:04:13,200 companies that are already winning. 112 00:04:13,200 --> 00:04:14,959 They're adopted at considerable expense 113 00:04:14,959 --> 00:04:17,680 often whole engineering teams on the off 114 00:04:17,680 --> 00:04:19,040 chance that they help them further 115 00:04:19,040 --> 00:04:22,479 consolidate on a lead. Now startups and 116 00:04:22,479 --> 00:04:24,880 scaleups aren't consolidating, they're 117 00:04:24,880 --> 00:04:27,280 surviving. And you win in startups by 118 00:04:27,280 --> 00:04:28,800 understanding what's needed to be 119 00:04:28,800 --> 00:04:31,199 successful and doing as little work as 120 00:04:31,199 --> 00:04:33,280 possible to achieve this. And these 121 00:04:33,280 --> 00:04:35,520 solutions, they do the exact opposite. 122 00:04:35,520 --> 00:04:37,040 Essentially, 123 00:04:37,040 --> 00:04:40,080 our problem is solutions. 124 00:04:40,080 --> 00:04:41,840 We allow ourselves to let our defined 125 00:04:41,840 --> 00:04:43,040 solutions define the problems we're 126 00:04:43,040 --> 00:04:44,080 solving. Do we all like that? That's 127 00:04:44,080 --> 00:04:46,160 pretty good, isn't it? Yeah. [snorts] 128 00:04:46,160 --> 00:04:49,960 Our problem is solutions. 129 00:04:54,320 --> 00:04:55,759 I'm going to pop up on LinkedIn and do 130 00:04:55,759 --> 00:04:57,680 some numbers. 131 00:04:57,680 --> 00:04:59,759 Our problem is solutions. We find a 132 00:04:59,759 --> 00:05:01,280 solution that a well-known business has 133 00:05:01,280 --> 00:05:03,520 chosen. Then we look to see if we have a 134 00:05:03,520 --> 00:05:06,000 problem that can be solved by it. And 135 00:05:06,000 --> 00:05:07,520 sometimes when we can't find a problem 136 00:05:07,520 --> 00:05:09,600 to solve, we choose a solution anyway so 137 00:05:09,600 --> 00:05:11,199 that it will give us fun problems to 138 00:05:11,199 --> 00:05:16,280 work on. Why do we do this? 139 00:05:18,240 --> 00:05:20,639 Do we do it because we're [ __ ] 140 00:05:20,639 --> 00:05:22,080 I've worked as a manager for many years 141 00:05:22,080 --> 00:05:23,759 and this was a popular opinion amongst 142 00:05:23,759 --> 00:05:26,240 my peers. But I can say with some 143 00:05:26,240 --> 00:05:27,840 confidence that no, we did not do this 144 00:05:27,840 --> 00:05:30,800 because we are [ __ ] 145 00:05:30,800 --> 00:05:34,479 To be clear, some of us are [ __ ] 146 00:05:34,479 --> 00:05:36,320 But we don't do it because we are. Or at 147 00:05:36,320 --> 00:05:37,600 least I've not been able to find any 148 00:05:37,600 --> 00:05:38,960 evidence in my studies that makes that 149 00:05:38,960 --> 00:05:41,360 causal link. So why do we do it? Why do 150 00:05:41,360 --> 00:05:44,160 we complicate our lives like this? 151 00:05:44,160 --> 00:05:46,000 Sometimes we do it because we are by and 152 00:05:46,000 --> 00:05:48,000 large quite a clever group of people. 153 00:05:48,000 --> 00:05:49,520 Software engineers invest a ton of 154 00:05:49,520 --> 00:05:51,199 effort in developing the skills they 155 00:05:51,199 --> 00:05:53,759 need to understand complicated systems. 156 00:05:53,759 --> 00:05:55,199 And when we're put in an environment 157 00:05:55,199 --> 00:05:57,120 where complexity isn't the key 158 00:05:57,120 --> 00:05:59,280 challenge, i.e. startups, we may create 159 00:05:59,280 --> 00:06:01,440 complexity to challenge ourselves. Or 160 00:06:01,440 --> 00:06:03,600 sometimes smart all smarts also means 161 00:06:03,600 --> 00:06:05,520 that it's hard to say yes to a solution 162 00:06:05,520 --> 00:06:07,120 that isn't perfect and is just going to 163 00:06:07,120 --> 00:06:08,720 be good enough to last us the next 12 164 00:06:08,720 --> 00:06:10,240 months. 165 00:06:10,240 --> 00:06:13,759 Sometimes it's fashion. 166 00:06:13,759 --> 00:06:15,520 Engineers spend a lot of time staying up 167 00:06:15,520 --> 00:06:17,759 to date. We read a lot about technology 168 00:06:17,759 --> 00:06:19,680 and in the process we see that big names 169 00:06:19,680 --> 00:06:22,319 in tech are doing to handle 30 billion 170 00:06:22,319 --> 00:06:23,840 concurrent users. We adopt their 171 00:06:23,840 --> 00:06:26,160 technologies hoping to emulate their 172 00:06:26,160 --> 00:06:29,039 success and we merily fly our straw 173 00:06:29,039 --> 00:06:30,960 airplanes off the end of our startup's 174 00:06:30,960 --> 00:06:33,039 runway. 175 00:06:33,039 --> 00:06:34,639 But I think the main reason we failed to 176 00:06:34,639 --> 00:06:35,919 make the right things for the business 177 00:06:35,919 --> 00:06:39,039 is it's hard. Understanding what the 178 00:06:39,039 --> 00:06:42,479 business needs is hard. And fighting the 179 00:06:42,479 --> 00:06:44,400 temptation to chase squirrels is is hard 180 00:06:44,400 --> 00:06:45,759 when you don't know what the business 181 00:06:45,759 --> 00:06:47,600 wants. And it's it's hard even if we 182 00:06:47,600 --> 00:06:49,039 take engineers in general as our 183 00:06:49,039 --> 00:06:52,800 baseline. For product engineers, sorry, 184 00:06:52,800 --> 00:06:54,639 for product engineers, they work on 185 00:06:54,639 --> 00:06:56,240 parts of products that make intuitive 186 00:06:56,240 --> 00:06:58,240 sense to regular users and can freely 187 00:06:58,240 --> 00:07:00,720 collect feedback. Product engineers have 188 00:07:00,720 --> 00:07:02,880 user journeys, personas, road maps. They 189 00:07:02,880 --> 00:07:05,120 have product product managers to help 190 00:07:05,120 --> 00:07:07,840 them collect user feedback. someone who 191 00:07:07,840 --> 00:07:09,759 has strategic remit over creating a 192 00:07:09,759 --> 00:07:12,400 story about what the business needs. And 193 00:07:12,400 --> 00:07:15,199 even the C CEO, sorry, the CEO is 194 00:07:15,199 --> 00:07:18,479 usually crouched a few feet away for a 195 00:07:18,479 --> 00:07:19,680 chance to tell them what the product 196 00:07:19,680 --> 00:07:22,000 needs to do. They have all the tools you 197 00:07:22,000 --> 00:07:23,759 could want to figure out the right thing 198 00:07:23,759 --> 00:07:26,479 to build. And yet, we all know how often 199 00:07:26,479 --> 00:07:28,400 product teams still build the wrong 200 00:07:28,400 --> 00:07:30,720 thing cuz it's hard to build the right 201 00:07:30,720 --> 00:07:35,919 thing. And it's especially hard for SR. 202 00:07:35,919 --> 00:07:37,360 And when I say SRE, I don't mean the 203 00:07:37,360 --> 00:07:39,039 fancy Google job title. I mean all of us 204 00:07:39,039 --> 00:07:40,560 in this room, everyone who has taken 205 00:07:40,560 --> 00:07:42,960 reliability into their heart. It's 206 00:07:42,960 --> 00:07:44,800 incredibly hard for us. And it's hard 207 00:07:44,800 --> 00:07:47,759 because one, we're a specialtity. Much 208 00:07:47,759 --> 00:07:49,360 of the business doesn't understand 209 00:07:49,360 --> 00:07:51,680 reliability work. They know they want 210 00:07:51,680 --> 00:07:53,759 the website to be reliable. But you will 211 00:07:53,759 --> 00:07:56,319 not find many people people who have say 212 00:07:56,319 --> 00:07:57,919 given thought to what parts of the site 213 00:07:57,919 --> 00:08:00,560 are more important than others. Few 214 00:08:00,560 --> 00:08:02,639 people and even engineers will talk in 215 00:08:02,639 --> 00:08:04,560 terms of reliability trade-offs and 216 00:08:04,560 --> 00:08:07,199 genuine need. [snorts] Even PMs whose 217 00:08:07,199 --> 00:08:09,599 entire job is convincing others that 218 00:08:09,599 --> 00:08:11,039 some parts of the product are more 219 00:08:11,039 --> 00:08:13,440 important than others will wonder why S 220 00:08:13,440 --> 00:08:14,879 just doesn't make the whole site 221 00:08:14,879 --> 00:08:17,879 reliable. 222 00:08:18,080 --> 00:08:19,759 Uh the problem is trust. When the 223 00:08:19,759 --> 00:08:21,520 business needs specialists in a domain 224 00:08:21,520 --> 00:08:23,120 that isn't well understood, they will 225 00:08:23,120 --> 00:08:25,360 normally end up at one of two extremes. 226 00:08:25,360 --> 00:08:27,440 suspicion. We don't understand what the 227 00:08:27,440 --> 00:08:29,280 SR team does and we can't understand if 228 00:08:29,280 --> 00:08:30,960 they're succeeding based on signals we 229 00:08:30,960 --> 00:08:33,599 understand. So we apply heavy oversight 230 00:08:33,599 --> 00:08:35,680 and scrutinize decisions. Business 231 00:08:35,680 --> 00:08:37,440 involvement is adversarial and so 232 00:08:37,440 --> 00:08:39,200 reliability engineers disengage and 233 00:08:39,200 --> 00:08:40,719 subvert. We go off and make our own 234 00:08:40,719 --> 00:08:42,800 things and don't engage with the 235 00:08:42,800 --> 00:08:45,360 business. Alternatively, benign neglect 236 00:08:45,360 --> 00:08:47,040 is the other approach. We don't 237 00:08:47,040 --> 00:08:48,880 understand what SR team does, so we'll 238 00:08:48,880 --> 00:08:50,720 just trust them. the business doesn't 239 00:08:50,720 --> 00:08:52,320 support the reliability by giving them 240 00:08:52,320 --> 00:08:54,640 good feedback on their ideas and 241 00:08:54,640 --> 00:08:56,160 reliability engineers go off and do 242 00:08:56,160 --> 00:08:58,320 whatever they want. 243 00:08:58,320 --> 00:09:00,160 Lastly, visibility. Good reliability 244 00:09:00,160 --> 00:09:02,880 work is almost always invisible. Product 245 00:09:02,880 --> 00:09:04,800 can always be showcased in a way that is 246 00:09:04,800 --> 00:09:06,720 compelling to the folks upstairs at the 247 00:09:06,720 --> 00:09:09,279 company. The boring kind of reliability 248 00:09:09,279 --> 00:09:11,360 work that gets you up high up time often 249 00:09:11,360 --> 00:09:14,560 is not. Incentives often push SRRES to 250 00:09:14,560 --> 00:09:16,399 favor large changes, including big 251 00:09:16,399 --> 00:09:17,839 technology bets that make for great 252 00:09:17,839 --> 00:09:19,440 announcements, but only incremental 253 00:09:19,440 --> 00:09:22,160 reliability improvements. In short, 254 00:09:22,160 --> 00:09:23,760 engineers working in reliability have 255 00:09:23,760 --> 00:09:26,000 the same problems as other engineers. We 256 00:09:26,000 --> 00:09:27,920 have some extra problems, and we have 257 00:09:27,920 --> 00:09:29,519 none of the tools that other engineers 258 00:09:29,519 --> 00:09:31,519 have to fix them. Guys, my name's Sean 259 00:09:31,519 --> 00:09:35,480 O'Keefe. That's my talk. Thanks. 260 00:09:35,680 --> 00:09:39,200 Wouldn't that be bad if I did that? 261 00:09:39,200 --> 00:09:41,600 How do we fix this? How do we get the 262 00:09:41,600 --> 00:09:43,600 tools? 263 00:09:43,600 --> 00:09:45,440 How do we get access to the same tools 264 00:09:45,440 --> 00:09:48,480 that product have? Where can where we 265 00:09:48,480 --> 00:09:51,360 can have a clear idea of what works what 266 00:09:51,360 --> 00:09:53,519 work the business needs from us most? 267 00:09:53,519 --> 00:09:55,360 How can we point to concrete measurable 268 00:09:55,360 --> 00:09:56,711 impact that the business cares about and 269 00:09:56,711 --> 00:09:57,519 [clears throat] how do we get peers who 270 00:09:57,519 --> 00:09:59,279 are deeply invested in helping us 271 00:09:59,279 --> 00:10:00,720 understand what needs to be more 272 00:10:00,720 --> 00:10:02,800 reliable? 273 00:10:02,800 --> 00:10:04,080 I reckon the answer is critical 274 00:10:04,080 --> 00:10:05,200 journeys. And I'm sorry about that 275 00:10:05,200 --> 00:10:06,720 sudden light that must have just blinded 276 00:10:06,720 --> 00:10:10,720 you all. Um, CUJS. What is a CUJ? 277 00:10:10,720 --> 00:10:14,200 This is a CJ. 278 00:10:18,720 --> 00:10:20,880 This is a CUJ. 279 00:10:20,880 --> 00:10:22,640 It's a description of an important flow 280 00:10:22,640 --> 00:10:24,880 through your product and unambiguous 281 00:10:24,880 --> 00:10:27,680 list of steps. Expresses users intent, 282 00:10:27,680 --> 00:10:29,519 their goal, and most importantly, it's 283 00:10:29,519 --> 00:10:31,360 in language that can be used across the 284 00:10:31,360 --> 00:10:33,680 org. It's close enough to language that 285 00:10:33,680 --> 00:10:35,760 product PM close enough to to the 286 00:10:35,760 --> 00:10:37,920 product that PMs can use it. And it's 287 00:10:37,920 --> 00:10:39,360 specific enough that there's no 288 00:10:39,360 --> 00:10:41,279 ambiguity about which parts of the 289 00:10:41,279 --> 00:10:44,160 product it touches. 290 00:10:44,160 --> 00:10:46,160 So what it's it's a user story, right? 291 00:10:46,160 --> 00:10:48,160 We've seen user stories and this sounds 292 00:10:48,160 --> 00:10:52,240 a lot like product work. 293 00:10:52,240 --> 00:10:53,680 And why are we doing product work? We've 294 00:10:53,680 --> 00:10:55,662 all got computer science degrees. We're 295 00:10:55,662 --> 00:10:57,360 [snorts] sres where are the SLOs's? 296 00:10:57,360 --> 00:10:58,480 Where's console? Where's Kubernetes? 297 00:10:58,480 --> 00:11:01,120 Where's CFKA? Why do we care? 298 00:11:01,120 --> 00:11:03,040 We care because every startup has this 299 00:11:03,040 --> 00:11:04,560 alert. 300 00:11:04,560 --> 00:11:05,600 And then for those who don't speak 301 00:11:05,600 --> 00:11:07,519 Prometheus, it says when the error rate 302 00:11:07,519 --> 00:11:10,640 for all HTTP requests across the entire 303 00:11:10,640 --> 00:11:12,480 site goes above 3% for more than 5 304 00:11:12,480 --> 00:11:15,920 minutes, page on call. If this sounds 305 00:11:15,920 --> 00:11:17,279 good to you, please come and talk to me 306 00:11:17,279 --> 00:11:19,279 after. Um, 307 00:11:19,279 --> 00:11:20,480 we'll come back to this alert. Let's 308 00:11:20,480 --> 00:11:22,560 flesh out our CUJ. We can take the 309 00:11:22,560 --> 00:11:25,200 description from earlier and turn that 310 00:11:25,200 --> 00:11:27,839 description into something like this. Up 311 00:11:27,839 --> 00:11:30,640 top is the flow name with concrete steps 312 00:11:30,640 --> 00:11:33,200 listed below. And each step is mapped to 313 00:11:33,200 --> 00:11:36,320 the list of endpoints that underpin it. 314 00:11:36,320 --> 00:11:38,800 We take user language and map it to 315 00:11:38,800 --> 00:11:41,760 pages and endpoints. Now, this doesn't 316 00:11:41,760 --> 00:11:43,680 seem profound, but it's huge. With just 317 00:11:43,680 --> 00:11:44,880 the effort of a quick scan to the 318 00:11:44,880 --> 00:11:46,880 codebase, which is getting easy easier 319 00:11:46,880 --> 00:11:49,200 every day now with our clankers, we have 320 00:11:49,200 --> 00:11:50,800 the beginnings of a sort of Rosetta 321 00:11:50,800 --> 00:11:52,640 Stone for the business and reliability. 322 00:11:52,640 --> 00:11:55,440 We can map what we do as SRRES, keeping 323 00:11:55,440 --> 00:12:00,160 endpoints up directly to business value. 324 00:12:00,160 --> 00:12:02,160 And that's just one CUJ. Your product 325 00:12:02,160 --> 00:12:04,079 team probably already has their flows 326 00:12:04,079 --> 00:12:06,240 documented in one format or another. And 327 00:12:06,240 --> 00:12:07,680 if they don't, then even better. You can 328 00:12:07,680 --> 00:12:09,519 work directly with them to develop their 329 00:12:09,519 --> 00:12:12,240 definitions. 330 00:12:12,240 --> 00:12:13,440 Talk to them and you can start to 331 00:12:13,440 --> 00:12:15,120 develop a list of these journeys and 332 00:12:15,120 --> 00:12:16,800 repeat the exercise of breaking them 333 00:12:16,800 --> 00:12:19,200 down into steps and endpoints. Here we 334 00:12:19,200 --> 00:12:20,959 start to develop an idea of what the or 335 00:12:20,959 --> 00:12:24,160 believes [snorts] our product does. 336 00:12:24,160 --> 00:12:27,880 Then you talk to your PMs 337 00:12:28,079 --> 00:12:29,760 and you can come to understand the 338 00:12:29,760 --> 00:12:31,519 relative criticality of each flow to the 339 00:12:31,519 --> 00:12:33,839 business. You'll see some are critical. 340 00:12:33,839 --> 00:12:35,760 They must work. Each time they don't, it 341 00:12:35,760 --> 00:12:38,079 hurts the business. Others are medium to 342 00:12:38,079 --> 00:12:39,519 low criticality. These are optional 343 00:12:39,519 --> 00:12:41,600 features. UI sugar things that users can 344 00:12:41,600 --> 00:12:43,120 live without. It's nice if they're up, 345 00:12:43,120 --> 00:12:45,760 but the business may not want you to put 346 00:12:45,760 --> 00:12:47,440 fixing them ahead of fixing critical 347 00:12:47,440 --> 00:12:49,040 features. 348 00:12:49,040 --> 00:12:51,519 And then you talk to your CEO 349 00:12:51,519 --> 00:12:53,279 and you find out there are actually 300 350 00:12:53,279 --> 00:12:55,839 user journeys and every single one of 351 00:12:55,839 --> 00:12:58,160 them is critical. 352 00:12:58,160 --> 00:13:00,639 So let's not talk to our CEO for now. 353 00:13:00,639 --> 00:13:04,399 Let's go back to our PM. 354 00:13:04,399 --> 00:13:06,160 We can talk to the PM and understand not 355 00:13:06,160 --> 00:13:07,839 just what the flows are, but they're 356 00:13:07,839 --> 00:13:09,600 relative important to the business. And 357 00:13:09,600 --> 00:13:11,920 once we know that this flow is important 358 00:13:11,920 --> 00:13:13,200 to the business, then we can start to 359 00:13:13,200 --> 00:13:14,720 say that these end points are our key 360 00:13:14,720 --> 00:13:16,880 mission. There's no more everything's 361 00:13:16,880 --> 00:13:18,639 important. we can start to make the case 362 00:13:18,639 --> 00:13:20,480 that you know at startups in an 363 00:13:20,480 --> 00:13:21,839 environment where engineering effort is 364 00:13:21,839 --> 00:13:23,920 at a premium we focus our reliability 365 00:13:23,920 --> 00:13:26,720 work on keeping these endpoints up and 366 00:13:26,720 --> 00:13:28,639 conversely if an endpoint doesn't 367 00:13:28,639 --> 00:13:31,279 underpin a key flow we have to ask 368 00:13:31,279 --> 00:13:32,800 questions about why we'd get people out 369 00:13:32,800 --> 00:13:34,880 of bed if its endpoints start erroring 370 00:13:34,880 --> 00:13:36,880 we can ask should we spend scramble on 371 00:13:36,880 --> 00:13:38,800 getting issues fixed with it and you can 372 00:13:38,800 --> 00:13:40,079 all see that Claude's really done me 373 00:13:40,079 --> 00:13:42,240 dirty here by putting the orth endpoint 374 00:13:42,240 --> 00:13:45,309 on the non-critical endpoints flow 375 00:13:45,309 --> 00:13:46,959 [laughter] 376 00:13:46,959 --> 00:13:49,279 All right. So, it feels like we've gone 377 00:13:49,279 --> 00:13:50,959 through a lot of effort to get something 378 00:13:50,959 --> 00:13:52,480 we kind of already know. You know, 379 00:13:52,480 --> 00:13:55,600 important end points are important. 380 00:13:55,600 --> 00:13:58,399 It's obvious, but in the way that take 381 00:13:58,399 --> 00:14:00,639 driving home is obvious that the road 382 00:14:00,639 --> 00:14:02,320 home is obvious. And have you ever done 383 00:14:02,320 --> 00:14:04,160 the thing where you jump in your car, 384 00:14:04,160 --> 00:14:05,360 you start to drive home, and then 385 00:14:05,360 --> 00:14:07,680 suddenly you're in your driveway? 386 00:14:07,680 --> 00:14:09,199 You know where home is, but if you were 387 00:14:09,199 --> 00:14:10,880 asked to actually list out each turn you 388 00:14:10,880 --> 00:14:12,320 made along the way, you may struggle. 389 00:14:12,320 --> 00:14:14,399 And these are the implicit details that 390 00:14:14,399 --> 00:14:17,040 live in our mind but aren't easily 391 00:14:17,040 --> 00:14:19,519 shared. Writing out the steps, the end 392 00:14:19,519 --> 00:14:21,199 points will invite you and your peers to 393 00:14:21,199 --> 00:14:23,199 remove disagreement about what it is 394 00:14:23,199 --> 00:14:25,600 your product actually does. I used to 395 00:14:25,600 --> 00:14:27,040 work at a startup that made digital 396 00:14:27,040 --> 00:14:29,904 business cards. [snorts] 397 00:14:30,160 --> 00:14:33,120 Someone would say share a contact and 20 398 00:14:33,120 --> 00:14:34,399 different engineers would think of 20 399 00:14:34,399 --> 00:14:36,959 different workflows. And very likely 400 00:14:36,959 --> 00:14:38,560 when we each said we were going home at 401 00:14:38,560 --> 00:14:39,920 the end of the day, we we probably all 402 00:14:39,920 --> 00:14:43,600 assumed we were going to the same place. 403 00:14:47,920 --> 00:14:49,600 And the thing here is it's the actual 404 00:14:49,600 --> 00:14:51,519 artifact isn't the thing that's 405 00:14:51,519 --> 00:14:54,880 valuable. Writing up your CUJS isn't the 406 00:14:54,880 --> 00:14:56,240 hard part. You could sit down, write 407 00:14:56,240 --> 00:14:58,399 them all out one night by yourself and 408 00:14:58,399 --> 00:15:00,320 not get anything out of them because the 409 00:15:00,320 --> 00:15:03,279 value is the exercise. You've sat down 410 00:15:03,279 --> 00:15:05,519 with your product engineers and made 411 00:15:05,519 --> 00:15:06,880 sure you both agree on what the product 412 00:15:06,880 --> 00:15:09,440 does and how it does it. You've sat down 413 00:15:09,440 --> 00:15:11,199 with your PM and hopefully made sure you 414 00:15:11,199 --> 00:15:13,279 both agree what's critical for the 415 00:15:13,279 --> 00:15:16,240 business and what's not. 416 00:15:16,240 --> 00:15:18,000 And having built it together 417 00:15:18,000 --> 00:15:20,160 collaboratively, your list of CJ serves 418 00:15:20,160 --> 00:15:22,320 as a reference and as a contract. You 419 00:15:22,320 --> 00:15:23,680 can talk about a feature being high 420 00:15:23,680 --> 00:15:25,360 criticality and everyone agrees what 421 00:15:25,360 --> 00:15:27,360 this means. You can refer to user 422 00:15:27,360 --> 00:15:28,880 journeys and you can refer to each of 423 00:15:28,880 --> 00:15:31,360 them by name and mean the same thing. 424 00:15:31,360 --> 00:15:33,279 And implicitly you've planted the idea 425 00:15:33,279 --> 00:15:35,360 that we can think in terms of the 426 00:15:35,360 --> 00:15:37,519 relative reliability requirements of 427 00:15:37,519 --> 00:15:39,440 each feature which is not where everyone 428 00:15:39,440 --> 00:15:42,000 in your org started. 429 00:15:42,000 --> 00:15:43,519 And we need this all because they are 430 00:15:43,519 --> 00:15:45,600 going to underpin the discussion about 431 00:15:45,600 --> 00:15:49,519 removing this alert we saw earlier. 432 00:15:49,519 --> 00:15:51,759 This is the alert that you show to the 433 00:15:51,759 --> 00:15:55,120 business to show that you're serious. 434 00:15:55,120 --> 00:15:56,800 It's there because someone got tired of 435 00:15:56,800 --> 00:15:58,480 customers catching outages before your 436 00:15:58,480 --> 00:16:00,880 engineers did, which is fair, but it's 437 00:16:00,880 --> 00:16:02,079 also the alert that means you'll be 438 00:16:02,079 --> 00:16:03,680 woken up each night fixing things that 439 00:16:03,680 --> 00:16:05,680 don't matter. And it's the alert that 440 00:16:05,680 --> 00:16:07,199 means you'll spend your time fixing the 441 00:16:07,199 --> 00:16:09,680 FAQ or the dark mode rather than what 442 00:16:09,680 --> 00:16:13,320 matters for the business. 443 00:16:13,360 --> 00:16:14,800 Now, what changes for our engineering 444 00:16:14,800 --> 00:16:16,320 org if we instead start alerting on 445 00:16:16,320 --> 00:16:17,680 this? And again, for anyone who doesn't 446 00:16:17,680 --> 00:16:19,839 read Prometheus, we are now only 447 00:16:19,839 --> 00:16:22,959 alerting on our critical user flows, not 448 00:16:22,959 --> 00:16:24,639 on everything, on the parts that we've 449 00:16:24,639 --> 00:16:27,199 decided really matter. 450 00:16:27,199 --> 00:16:28,480 What if we say we're only going to get 451 00:16:28,480 --> 00:16:29,920 engineers out of bed when a critical 452 00:16:29,920 --> 00:16:32,720 flow breaks? What if only these flows 453 00:16:32,720 --> 00:16:34,720 result in action items that we scramble 454 00:16:34,720 --> 00:16:36,800 for? 455 00:16:36,800 --> 00:16:39,199 Hold that thought while we remind 456 00:16:39,199 --> 00:16:41,440 ourselves of the problem we're solving. 457 00:16:41,440 --> 00:16:43,920 We set out to fix avoiding the work that 458 00:16:43,920 --> 00:16:46,240 isn't going to help the business. 459 00:16:46,240 --> 00:16:49,380 It's avoiding the work. Sorry. And we 460 00:16:49,380 --> 00:16:50,720 [snorts] now have a sorry, it's avoiding 461 00:16:50,720 --> 00:16:52,399 the work that doesn't matter. And we now 462 00:16:52,399 --> 00:16:54,720 have our definitions of what matters. 463 00:16:54,720 --> 00:16:56,000 But we've not solved how we actually 464 00:16:56,000 --> 00:16:57,920 change how we work to use those 465 00:16:57,920 --> 00:16:59,920 definitions to choose better tasks. So 466 00:16:59,920 --> 00:17:01,120 let's do that. But first, we're going to 467 00:17:01,120 --> 00:17:04,720 pay a visit to this guy, 468 00:17:04,720 --> 00:17:07,959 our CEO. 469 00:17:08,000 --> 00:17:10,326 Yes, it is increasing. 470 00:17:10,326 --> 00:17:12,240 [laughter] 471 00:17:12,240 --> 00:17:14,079 to the fellow who when asked which user 472 00:17:14,079 --> 00:17:15,600 journey is the most important says all 473 00:17:15,600 --> 00:17:17,760 of them. And as much as I want to score 474 00:17:17,760 --> 00:17:19,360 points and say that they're wrong, that 475 00:17:19,360 --> 00:17:21,199 that's not true. At least there is no 476 00:17:21,199 --> 00:17:22,559 reason in the world that they would give 477 00:17:22,559 --> 00:17:25,520 you any answer other than this. Your 478 00:17:25,520 --> 00:17:28,011 CEO, 479 00:17:28,011 --> 00:17:28,640 [laughter] 480 00:17:28,640 --> 00:17:30,240 your CEO actually has a hard job. And 481 00:17:30,240 --> 00:17:31,919 and I'm here scoring points making fun 482 00:17:31,919 --> 00:17:33,760 of them with my little pictures, but it 483 00:17:33,760 --> 00:17:35,440 it is true. They've worked harder than a 484 00:17:35,440 --> 00:17:36,640 lot of people in the company. They have 485 00:17:36,640 --> 00:17:38,080 more skin in the game. They're the ones 486 00:17:38,080 --> 00:17:40,480 who have painted a picture of your 487 00:17:40,480 --> 00:17:43,120 product that has gotten us all jobs and 488 00:17:43,120 --> 00:17:44,480 they've spent the past two or three 489 00:17:44,480 --> 00:17:46,480 years having people say no to them and 490 00:17:46,480 --> 00:17:47,679 they've gotten this far by pushing 491 00:17:47,679 --> 00:17:50,000 through no. So it would be irresponsible 492 00:17:50,000 --> 00:17:52,480 of them to answer our question in any 493 00:17:52,480 --> 00:17:54,400 way any other way given the way we 494 00:17:54,400 --> 00:17:56,160 framed it. 495 00:17:56,160 --> 00:17:59,039 The onus is on us to show that there is 496 00:17:59,039 --> 00:18:01,200 a constraint that we do need to 497 00:18:01,200 --> 00:18:02,880 prioritize different parts of the 498 00:18:02,880 --> 00:18:04,559 product and we need to show them that if 499 00:18:04,559 --> 00:18:07,679 we reduce what we cover 500 00:18:07,679 --> 00:18:09,440 reduce what we consider urgent and 501 00:18:09,440 --> 00:18:11,440 important then we have a plan to come 502 00:18:11,440 --> 00:18:13,440 back. 503 00:18:13,440 --> 00:18:15,520 So let's talk about that plan. How do we 504 00:18:15,520 --> 00:18:17,679 roll out CUJS as the tool we'll use to 505 00:18:17,679 --> 00:18:20,160 make prioritization decisions with? How 506 00:18:20,160 --> 00:18:22,080 do we reduce the scope of what we cover 507 00:18:22,080 --> 00:18:23,840 so we can offer worldclass reliability 508 00:18:23,840 --> 00:18:26,000 on the journeys we do cover? but also 509 00:18:26,000 --> 00:18:27,360 build a process so that we can 510 00:18:27,360 --> 00:18:28,880 continually improve the quality of our 511 00:18:28,880 --> 00:18:31,200 coverage. 512 00:18:31,200 --> 00:18:33,039 We've done the important part. We built 513 00:18:33,039 --> 00:18:35,520 the artifact together. 514 00:18:35,520 --> 00:18:37,039 We have an engineering group who 515 00:18:37,039 --> 00:18:39,520 understand the idea. 516 00:18:39,520 --> 00:18:40,720 And next, we're going to do something 517 00:18:40,720 --> 00:18:43,440 called the ratchet. 518 00:18:43,440 --> 00:18:46,320 First, you start small. 519 00:18:46,320 --> 00:18:48,400 You choose a number of CUJs that simply 520 00:18:48,400 --> 00:18:50,000 cannot fail. Now if you're a small 521 00:18:50,000 --> 00:18:51,520 startup, if you are struggling with on 522 00:18:51,520 --> 00:18:52,960 call and it's started to get a bit of, 523 00:18:52,960 --> 00:18:54,080 you know, profile and we're thinking 524 00:18:54,080 --> 00:18:56,160 about what we can do, you could start 525 00:18:56,160 --> 00:18:58,720 with one a single CJ, the most important 526 00:18:58,720 --> 00:19:00,240 thing that your business needs and build 527 00:19:00,240 --> 00:19:02,720 up from there. The name of the game here 528 00:19:02,720 --> 00:19:05,280 is healthy, constructive, visionary 529 00:19:05,280 --> 00:19:06,799 myopia. 530 00:19:06,799 --> 00:19:08,640 We are going to ignore things for a bit 531 00:19:08,640 --> 00:19:10,320 and get comfortable with ignoring things 532 00:19:10,320 --> 00:19:12,960 for a bit. 533 00:19:12,960 --> 00:19:16,080 We pick our CJs. Next, we descope our 534 00:19:16,080 --> 00:19:18,559 alerts. [snorts] Before you saw that we 535 00:19:18,559 --> 00:19:20,400 were, you know, the power of us limiting 536 00:19:20,400 --> 00:19:24,559 our alerting to all of our CUJS, 537 00:19:24,559 --> 00:19:25,840 we're going to limit to just a couple of 538 00:19:25,840 --> 00:19:28,799 CUJs. And by that I mean we are only 539 00:19:28,799 --> 00:19:31,440 going to get people out of bed when our 540 00:19:31,440 --> 00:19:33,760 CJs that we've chosen alert. Everything 541 00:19:33,760 --> 00:19:39,720 else dark mode goes to Slack. 542 00:19:40,400 --> 00:19:42,000 Then we establish a process to check on 543 00:19:42,000 --> 00:19:44,960 the health of our chosen CJs. 544 00:19:44,960 --> 00:19:46,799 Once a week, once a Fortnite, we get 545 00:19:46,799 --> 00:19:48,400 together as a group and see how each of 546 00:19:48,400 --> 00:19:50,720 our selected CUJS are doing. We leave 547 00:19:50,720 --> 00:19:52,559 that meeting with a plan to fix any that 548 00:19:52,559 --> 00:19:55,600 are outside of grain. Now, these fixes 549 00:19:55,600 --> 00:19:58,160 should be boring and timely. No galaxy 550 00:19:58,160 --> 00:20:00,160 bone stuff, just the least amount of 551 00:20:00,160 --> 00:20:02,559 work we can do to get in the grain. Pay 552 00:20:02,559 --> 00:20:05,200 down longing errors, [snorts] add retry, 553 00:20:05,200 --> 00:20:08,000 play to pray to your deity of choice. 554 00:20:08,000 --> 00:20:11,440 Just don't adopt CFKA. 555 00:20:11,440 --> 00:20:12,799 They should issue larger, more 556 00:20:12,799 --> 00:20:14,320 comprehensive solutions until it's 557 00:20:14,320 --> 00:20:16,240 proven that boring stuff won't do the 558 00:20:16,240 --> 00:20:18,960 job. 559 00:20:18,960 --> 00:20:20,720 Instant review, which we're all doing 560 00:20:20,720 --> 00:20:22,240 here, 561 00:20:22,240 --> 00:20:24,160 follows a similar pattern. While you can 562 00:20:24,160 --> 00:20:25,760 and should continue to review all 563 00:20:25,760 --> 00:20:28,240 incidents for learning, our action items 564 00:20:28,240 --> 00:20:29,760 that come out of that instant review are 565 00:20:29,760 --> 00:20:32,240 now how we apply our ratchet. AIS that 566 00:20:32,240 --> 00:20:34,080 touch our CUJS are fixed promptly, 567 00:20:34,080 --> 00:20:36,080 ideally within the week, and any 568 00:20:36,080 --> 00:20:38,720 outstanding CUJ AIS should be reviewed 569 00:20:38,720 --> 00:20:40,640 in our weekly ops meeting. And most 570 00:20:40,640 --> 00:20:43,840 importantly, our action items are small. 571 00:20:43,840 --> 00:20:46,400 Again, simple, myopic, the smallest 572 00:20:46,400 --> 00:20:48,240 amount of work that will get our CUJ 573 00:20:48,240 --> 00:20:51,240 grain. 574 00:20:51,440 --> 00:20:53,039 Probably warrant saying that we should 575 00:20:53,039 --> 00:20:54,559 be communicating widely within the 576 00:20:54,559 --> 00:20:56,480 business that fixes that don't fit our 577 00:20:56,480 --> 00:21:01,320 criteria may need to wait a while. 578 00:21:01,679 --> 00:21:04,400 Last of all, your risk register. Look 579 00:21:04,400 --> 00:21:05,840 how excited everyone got when I said 580 00:21:05,840 --> 00:21:09,120 risk register. 581 00:21:09,120 --> 00:21:10,799 The risk register is where you record 582 00:21:10,799 --> 00:21:13,520 the things we've chosen not to fix. It's 583 00:21:13,520 --> 00:21:15,200 first of all how you communicate to the 584 00:21:15,200 --> 00:21:16,720 business what will not be improving 585 00:21:16,720 --> 00:21:18,640 while we go through this process. It's 586 00:21:18,640 --> 00:21:20,240 also where you record the more ambitious 587 00:21:20,240 --> 00:21:23,120 work we've estued for simple fixes. 588 00:21:23,120 --> 00:21:25,440 We come back to the reg register when 589 00:21:25,440 --> 00:21:27,039 the simple stuff hasn't worked and we 590 00:21:27,039 --> 00:21:28,799 need to be more ambitious. Now this 591 00:21:28,799 --> 00:21:30,799 sounds boring and for sure it's a 592 00:21:30,799 --> 00:21:33,679 spreadsheet. It's not fashionable but 593 00:21:33,679 --> 00:21:36,159 it's going to work. It's the thing that 594 00:21:36,159 --> 00:21:38,320 stops the ratchet from being full yolo 595 00:21:38,320 --> 00:21:39,840 mode and actually makes it into big kit 596 00:21:39,840 --> 00:21:41,840 engineering. And most powerfully, this 597 00:21:41,840 --> 00:21:43,840 is where you make the case for your big 598 00:21:43,840 --> 00:21:45,760 technological bets, for your CFKA or 599 00:21:45,760 --> 00:21:47,840 whatever. It you can bring this evidence 600 00:21:47,840 --> 00:21:49,280 to your leadership and say, "We've tried 601 00:21:49,280 --> 00:21:50,640 the simple stuff. We've tried the 602 00:21:50,640 --> 00:21:52,400 basics. This has been an ongoing problem 603 00:21:52,400 --> 00:21:54,240 and we need something more complicated 604 00:21:54,240 --> 00:21:56,960 to actually solve our problems." 605 00:21:56,960 --> 00:21:58,880 And eventually, if you work through this 606 00:21:58,880 --> 00:22:00,159 process, you reach the point where all 607 00:22:00,159 --> 00:22:02,720 of our CUJs are green. This happens. 608 00:22:02,720 --> 00:22:05,360 What do we do? 609 00:22:05,360 --> 00:22:09,360 add more CUJs and repeat the process. We 610 00:22:09,360 --> 00:22:11,919 go back to our list, add a couple more 611 00:22:11,919 --> 00:22:14,559 CUJS, we put them under alerting, we add 612 00:22:14,559 --> 00:22:15,919 them to our ops review and instant 613 00:22:15,919 --> 00:22:18,080 review and repeat the process. 614 00:22:18,080 --> 00:22:19,679 Eventually, we hit the point where our 615 00:22:19,679 --> 00:22:22,400 org can't absorb more critical user 616 00:22:22,400 --> 00:22:25,120 journeys. We add a CJ and no amount of 617 00:22:25,120 --> 00:22:27,200 simple work will get us back in the 618 00:22:27,200 --> 00:22:29,360 grain. And from there, we have a simple 619 00:22:29,360 --> 00:22:31,919 choice for the business. Do we get more 620 00:22:31,919 --> 00:22:33,679 engineers to expand the amount of things 621 00:22:33,679 --> 00:22:36,080 that we can support? Do we lower the 622 00:22:36,080 --> 00:22:37,520 criticality of some of the things that 623 00:22:37,520 --> 00:22:39,280 we simply cannot find the manpower to 624 00:22:39,280 --> 00:22:42,080 support? Or do we then go off and do the 625 00:22:42,080 --> 00:22:44,480 ambitious work to scale further with the 626 00:22:44,480 --> 00:22:45,919 same number of engineers but with better 627 00:22:45,919 --> 00:22:49,039 technology and bring all the evidence we 628 00:22:49,039 --> 00:22:50,640 brought from this process to bear on 629 00:22:50,640 --> 00:22:52,480 this to show leadership that actually 630 00:22:52,480 --> 00:22:54,320 this work has to happen for us to get 631 00:22:54,320 --> 00:22:57,120 better. Now I am desperately hoping this 632 00:22:57,120 --> 00:22:59,039 gives you all an idea of how we tackle 633 00:22:59,039 --> 00:23:00,480 the problem we set out to fix. how we 634 00:23:00,480 --> 00:23:02,400 give engineers a process to stare at 635 00:23:02,400 --> 00:23:04,960 daily at what's important to the 636 00:23:04,960 --> 00:23:06,559 business and to use all their creativity 637 00:23:06,559 --> 00:23:07,919 to defend it. Because as much as I make 638 00:23:07,919 --> 00:23:09,360 fun of us, I believe fundamentally that 639 00:23:09,360 --> 00:23:11,200 engineers are good people and we do aim 640 00:23:11,200 --> 00:23:13,440 to do good for the people we work for. 641 00:23:13,440 --> 00:23:14,960 From there, it's simply a matter of 642 00:23:14,960 --> 00:23:17,679 giving us clear goals, 643 00:23:17,679 --> 00:23:19,200 combine them with those good engineers, 644 00:23:19,200 --> 00:23:20,480 and you'll find it's very unlikely 645 00:23:20,480 --> 00:23:21,760 you're going to end up with CAFTA in 646 00:23:21,760 --> 00:23:24,240 your stack. Uh guys, I'm Sean. I'm from 647 00:23:24,240 --> 00:23:25,679 Almadori, and I want to thank you very 648 00:23:25,679 --> 00:23:28,921 much for coming and listening to me. 649 00:23:28,921 --> 00:23:30,941 [applause] 650 00:23:34,400 --> 00:23:36,799 Thank you so much Sean. Uh we wanted to 651 00:23:36,799 --> 00:23:38,559 present you with this uh special mug 652 00:23:38,559 --> 00:23:39,520 from Pyon AU. 653 00:23:39,520 --> 00:23:40,640 Hell yeah. Thank you. 654 00:23:40,640 --> 00:23:41,679 On [clears throat] behalf of all of us, 655 00:23:41,679 --> 00:23:43,120 thank you so much for sharing your 656 00:23:43,120 --> 00:23:43,919 knowledge with us. 657 00:23:43,919 --> 00:23:45,039 Thanks very much, man. 658 00:23:45,039 --> 00:23:47,520 Um we've got a few minutes for talks, 659 00:23:47,520 --> 00:23:49,200 but we're going to finish a little bit 660 00:23:49,200 --> 00:23:52,000 early to wrap up the uh track. Would you 661 00:23:52,000 --> 00:23:53,280 like to take questions? 662 00:23:53,280 --> 00:23:54,400 Absolutely. Yeah. 663 00:23:54,400 --> 00:23:57,720 Any questions? 664 00:23:58,000 --> 00:23:59,600 Hello. 665 00:23:59,600 --> 00:24:01,120 How do you recommend dealing with the 666 00:24:01,120 --> 00:24:03,600 scenario of one out of one queries goes 667 00:24:03,600 --> 00:24:06,159 bad at 1:00 a.m. uh in that 5m minute 668 00:24:06,159 --> 00:24:08,480 window from 1 to 105 a.m.? 669 00:24:08,480 --> 00:24:10,720 One out of one queries was bad. 670 00:24:10,720 --> 00:24:12,080 That's more than five. That's more than 671 00:24:12,080 --> 00:24:14,320 3%. [laughter] 672 00:24:14,320 --> 00:24:16,640 I I was furiously arguing with Claude 673 00:24:16,640 --> 00:24:18,240 until about 1:00 a.m. about all the 674 00:24:18,240 --> 00:24:19,840 corner cases of that alert that I put 675 00:24:19,840 --> 00:24:21,279 up. He's like, "It's fine. No one's 676 00:24:21,279 --> 00:24:22,559 going to like pick at it." And sure 677 00:24:22,559 --> 00:24:24,559 enough, yeah, [laughter] 678 00:24:24,559 --> 00:24:28,000 you're absolutely right. 679 00:24:28,000 --> 00:24:29,760 Uh, I I don't really have a clear take 680 00:24:29,760 --> 00:24:31,279 on that one. I'm sorry. 681 00:24:31,279 --> 00:24:33,200 The line is you're right to call me on 682 00:24:33,200 --> 00:24:35,120 that. [laughter] 683 00:24:35,120 --> 00:24:39,039 It depends. Yeah. 684 00:24:39,039 --> 00:24:42,360 More questions. 685 00:24:43,039 --> 00:24:45,440 This your one and only chance. 686 00:24:45,440 --> 00:24:51,880 Last chance to take Scaly home. Yes. 687 00:24:54,960 --> 00:24:56,559 Hey, I really love the humor, by the 688 00:24:56,559 --> 00:24:58,240 way. It was fantastic little watching 689 00:24:58,240 --> 00:25:00,400 it. But I was just thinking, it reminded 690 00:25:00,400 --> 00:25:01,919 me of like the duct tape kind of 691 00:25:01,919 --> 00:25:04,320 solutions that you have, which they 692 00:25:04,320 --> 00:25:06,720 might not be perfect perfection, but 693 00:25:06,720 --> 00:25:08,080 they work and then you kind of forget 694 00:25:08,080 --> 00:25:10,159 about it and then they're in prod. I was 695 00:25:10,159 --> 00:25:11,760 just curious what you think of something 696 00:25:11,760 --> 00:25:13,360 like that. Is that the minimum viable 697 00:25:13,360 --> 00:25:15,600 you need for reliability or would you 698 00:25:15,600 --> 00:25:18,320 try and go back and fix them? I look 699 00:25:18,320 --> 00:25:20,559 this is a very sort of like dogmatic in 700 00:25:20,559 --> 00:25:22,559 my it's borderline sort of like my 701 00:25:22,559 --> 00:25:25,360 religion. Um I feel very strongly that 702 00:25:25,360 --> 00:25:27,679 we are always smarter later. We have 703 00:25:27,679 --> 00:25:29,679 more information. We've learned more. We 704 00:25:29,679 --> 00:25:31,600 know more about our product. I think 705 00:25:31,600 --> 00:25:34,000 there is a huge power in fixing exactly 706 00:25:34,000 --> 00:25:35,679 what needs to be fixed and let it be 707 00:25:35,679 --> 00:25:37,279 creaking. Let it be miserable. Be 708 00:25:37,279 --> 00:25:39,840 unhappy because you know the the the the 709 00:25:39,840 --> 00:25:41,520 human experience is suffering. We should 710 00:25:41,520 --> 00:25:44,400 just lean into it. Um, but but I always 711 00:25:44,400 --> 00:25:45,360 feel like, you know, we kind of 712 00:25:45,360 --> 00:25:47,919 overstate how hard it is to fix things 713 00:25:47,919 --> 00:25:49,600 afterwards. I don't think it's often 714 00:25:49,600 --> 00:25:51,600 hard to pay down what what everyone 715 00:25:51,600 --> 00:25:53,120 calls tech deck. I think you need to 716 00:25:53,120 --> 00:25:55,200 have a compelling argument. Few people 717 00:25:55,200 --> 00:25:57,200 disputed when I said that. Um, I think 718 00:25:57,200 --> 00:25:58,799 you have a compelling argument about 719 00:25:58,799 --> 00:26:00,880 what's wrong, you can bring people back. 720 00:26:00,880 --> 00:26:02,080 And I think as long as you are 721 00:26:02,080 --> 00:26:03,760 establishing with the folks that you 722 00:26:03,760 --> 00:26:06,480 know are making the decisions that these 723 00:26:06,480 --> 00:26:08,720 gains now are possibly going to come 724 00:26:08,720 --> 00:26:11,679 back and be later on be more work, 725 00:26:11,679 --> 00:26:13,039 you're on your way. And a lot of the 726 00:26:13,039 --> 00:26:14,159 tools that we talked about in there, 727 00:26:14,159 --> 00:26:15,679 especially the risk register, is your 728 00:26:15,679 --> 00:26:17,360 way of highlighting. This is the thing 729 00:26:17,360 --> 00:26:19,760 that we've taken a loan out on. This is 730 00:26:19,760 --> 00:26:21,919 coming. This is in the post. And when we 731 00:26:21,919 --> 00:26:23,919 come back to you in 12 months, we need 732 00:26:23,919 --> 00:26:25,200 to talk seriously about fixing it 733 00:26:25,200 --> 00:26:26,640 because we've got the empirical evidence 734 00:26:26,640 --> 00:26:28,240 to show nothing else is going to fix the 735 00:26:28,240 --> 00:26:31,039 problem. 736 00:26:31,039 --> 00:26:35,039 Wonderful. We have one more question. 737 00:26:35,039 --> 00:26:37,200 Hey, I I really liked your talk. Thank 738 00:26:37,200 --> 00:26:37,520 you. 739 00:26:37,520 --> 00:26:40,400 Thank you. Um, do you think that uh 740 00:26:40,400 --> 00:26:42,159 critical user journeys need to be 741 00:26:42,159 --> 00:26:45,600 company or orwide or is there value in 742 00:26:45,600 --> 00:26:48,080 defining critical user journeys for your 743 00:26:48,080 --> 00:26:50,720 particular team or the the platform or 744 00:26:50,720 --> 00:26:52,799 product that you provide to the rest of 745 00:26:52,799 --> 00:26:55,039 the business and tracking those as well 746 00:26:55,039 --> 00:26:57,600 or separately or even as like a starting 747 00:26:57,600 --> 00:26:59,919 point to get the whole process launched? 748 00:26:59,919 --> 00:27:01,520 I think they it's that's a killer 749 00:27:01,520 --> 00:27:03,039 question. Um, I think they work really 750 00:27:03,039 --> 00:27:04,880 really well within smaller groups. um 751 00:27:04,880 --> 00:27:06,799 they they aren't tremendous at exposing 752 00:27:06,799 --> 00:27:08,159 things across the entire org but I think 753 00:27:08,159 --> 00:27:10,799 what they do do really well I said do um 754 00:27:10,799 --> 00:27:12,640 is between two or three teams they can 755 00:27:12,640 --> 00:27:14,720 be a really who work closely together on 756 00:27:14,720 --> 00:27:17,279 similar integrated features they can be 757 00:27:17,279 --> 00:27:19,360 really good way to find where the 758 00:27:19,360 --> 00:27:21,600 friction between your services are if if 759 00:27:21,600 --> 00:27:25,200 your three teams find that your CJ share 760 00:27:25,200 --> 00:27:27,279 endpoints it kind of prompts really good 761 00:27:27,279 --> 00:27:28,960 discussions around who owns that who's 762 00:27:28,960 --> 00:27:31,200 accountable for it I haven't really 763 00:27:31,200 --> 00:27:33,279 found and I must uh confess I don't 764 00:27:33,279 --> 00:27:34,960 usually work in bigger companies. Uh 765 00:27:34,960 --> 00:27:36,720 there is something in my constitution 766 00:27:36,720 --> 00:27:39,039 which doesn't agree with them. Um but I 767 00:27:39,039 --> 00:27:40,480 have found that the best place they 768 00:27:40,480 --> 00:27:42,080 really shine is in that smaller setting 769 00:27:42,080 --> 00:27:45,039 of two to three teams. 770 00:27:45,039 --> 00:27:48,279 Thank you.