1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:14,960 --> 00:00:16,560 I'm glad you are all here. This is 4 00:00:16,560 --> 00:00:18,480 ballroom 2 and we have a wonderful 5 00:00:18,480 --> 00:00:21,119 lineup of speakers for this afternoon. 6 00:00:21,119 --> 00:00:23,279 So without any further ado, please 7 00:00:23,279 --> 00:00:26,855 welcome with a warm applause Noah 8 00:00:26,855 --> 00:00:28,875 [applause] 9 00:00:33,200 --> 00:00:35,520 Excellent. Uh, hi there. I'm Noah 10 00:00:35,520 --> 00:00:37,680 Canros. I'm an SR at Geomagical Labs. We 11 00:00:37,680 --> 00:00:39,280 do computer vision, augmented reality 12 00:00:39,280 --> 00:00:41,280 stuff for IKEA, but I'm not here to talk 13 00:00:41,280 --> 00:00:44,000 about that. I am a squishy human. I have 14 00:00:44,000 --> 00:00:45,760 made mistakes in the past. I am certain 15 00:00:45,760 --> 00:00:48,000 to make us mistakes again in the future. 16 00:00:48,000 --> 00:00:49,680 I like to plan for the future, and if 17 00:00:49,680 --> 00:00:51,280 those plans are going to survive contact 18 00:00:51,280 --> 00:00:52,960 with reality, I need to make sure that I 19 00:00:52,960 --> 00:00:55,199 account for future mistakes. 20 00:00:55,199 --> 00:00:56,719 This is a talk about building resilient 21 00:00:56,719 --> 00:00:58,160 systems and that can mean a lot of 22 00:00:58,160 --> 00:00:59,840 things. We had a lovely track on 23 00:00:59,840 --> 00:01:01,280 Thursday dedicated to technical 24 00:01:01,280 --> 00:01:03,199 resilience. Building systems that can 25 00:01:03,199 --> 00:01:05,360 survive all manner of computer failures 26 00:01:05,360 --> 00:01:07,040 and yesterday we did a security track on 27 00:01:07,040 --> 00:01:08,880 how to be resilient in the face of bad 28 00:01:08,880 --> 00:01:10,799 people trying to do bad things. But I 29 00:01:10,799 --> 00:01:12,000 want to talk about a different kind of 30 00:01:12,000 --> 00:01:14,799 resilience about humans and teams. At 31 00:01:14,799 --> 00:01:16,720 time of writing, every team I've been on 32 00:01:16,720 --> 00:01:18,560 has been staffed by humans working 33 00:01:18,560 --> 00:01:20,479 together to build a thing. And so those 34 00:01:20,479 --> 00:01:22,880 humans make mistakes. 35 00:01:22,880 --> 00:01:24,560 One of the major advantages to humans 36 00:01:24,560 --> 00:01:26,240 teaming up on a challenge is that we can 37 00:01:26,240 --> 00:01:28,000 design systems that are better than any 38 00:01:28,000 --> 00:01:29,920 one person. But as the software 39 00:01:29,920 --> 00:01:31,439 engineering landscape is reshaping 40 00:01:31,439 --> 00:01:33,360 itself yet again, the challenges we need 41 00:01:33,360 --> 00:01:35,680 to build for are shifting too. Elephant 42 00:01:35,680 --> 00:01:37,360 in the room, AI tools are a big change 43 00:01:37,360 --> 00:01:40,240 for a lot of teams, mine included. Uh, 44 00:01:40,240 --> 00:01:42,560 and in large part, this talk is a scream 45 00:01:42,560 --> 00:01:44,320 into the void from deep in my soul about 46 00:01:44,320 --> 00:01:46,479 the stressors that AI is creating. But 47 00:01:46,479 --> 00:01:48,399 this isn't really a talk about AI. And 48 00:01:48,399 --> 00:01:50,240 the drive for teams to work faster and 49 00:01:50,240 --> 00:01:53,040 faster stretches way before AI existed, 50 00:01:53,040 --> 00:01:55,040 probably forever. This is talk about 51 00:01:55,040 --> 00:01:56,880 humans and patterns and how humans 52 00:01:56,880 --> 00:01:59,360 collaborate for better or worse. Our 53 00:01:59,360 --> 00:02:01,439 road map. First, we'll talk about how 54 00:02:01,439 --> 00:02:03,439 humans can fail at things, then 55 00:02:03,439 --> 00:02:05,119 strategies and techniques for improving 56 00:02:05,119 --> 00:02:07,600 those. And finally, how the struggles 57 00:02:07,600 --> 00:02:09,280 can affect the people and what to watch 58 00:02:09,280 --> 00:02:11,280 out for. 59 00:02:11,280 --> 00:02:13,040 While every human is unique, the ways 60 00:02:13,040 --> 00:02:14,640 that our minds break down are consistent 61 00:02:14,640 --> 00:02:16,400 in the broad strokes. I'm going to list 62 00:02:16,400 --> 00:02:18,239 off the most common failure modes, but 63 00:02:18,239 --> 00:02:21,120 these are lenses to view the world, not 64 00:02:21,120 --> 00:02:23,120 we're not trying to label every possible 65 00:02:23,120 --> 00:02:24,800 bad time that a human can have. We're 66 00:02:24,800 --> 00:02:26,160 trying to build a framework that we can 67 00:02:26,160 --> 00:02:28,000 plan around. 68 00:02:28,000 --> 00:02:29,840 Confirmation bias is a big topic, but we 69 00:02:29,840 --> 00:02:31,360 have to dive in somewhere. We like to 70 00:02:31,360 --> 00:02:33,120 think of ourselves as objective 71 00:02:33,120 --> 00:02:35,120 observers of the world around us. But of 72 00:02:35,120 --> 00:02:36,720 course, we also know that that isn't 73 00:02:36,720 --> 00:02:39,120 true. We pick and choose on many layers, 74 00:02:39,120 --> 00:02:40,640 and we're always primed to be more 75 00:02:40,640 --> 00:02:42,160 accepting of information that confirms 76 00:02:42,160 --> 00:02:43,280 our current understandings and 77 00:02:43,280 --> 00:02:45,360 assumptions. Seven of the most powerful 78 00:02:45,360 --> 00:02:47,280 words in any situation are you are 79 00:02:47,280 --> 00:02:50,160 already doing the right thing. To phrase 80 00:02:50,160 --> 00:02:52,239 it as a syllogism, confirmation bias 81 00:02:52,239 --> 00:02:54,000 often roots in our ego, our natural 82 00:02:54,000 --> 00:02:55,680 desire to see ourselves as capable and 83 00:02:55,680 --> 00:02:57,680 powerful so we are good at things. Then 84 00:02:57,680 --> 00:02:59,599 we add the industry-wide belief that 85 00:02:59,599 --> 00:03:01,599 smart people do good things and we end 86 00:03:01,599 --> 00:03:03,280 up with the logical conclusion that 87 00:03:03,280 --> 00:03:06,480 whatever we are doing must be correct. 88 00:03:06,480 --> 00:03:08,720 Confirmation bias can also manifest as 89 00:03:08,720 --> 00:03:11,120 deference to authority, over pri uh 90 00:03:11,120 --> 00:03:12,879 prioritizing things seen as accepted 91 00:03:12,879 --> 00:03:15,920 wisdom or highly hierarchical thinking. 92 00:03:15,920 --> 00:03:17,599 In the same way as we try to center our 93 00:03:17,599 --> 00:03:19,440 own skills in agency, we can also be 94 00:03:19,440 --> 00:03:20,800 biased towards fitting in with our 95 00:03:20,800 --> 00:03:22,560 teams, giving people excessive benefit 96 00:03:22,560 --> 00:03:24,319 of the doubt. Both of these forms 97 00:03:24,319 --> 00:03:25,840 highlight how these biases aren't always 98 00:03:25,840 --> 00:03:27,519 in the wrong. They're cognitive 99 00:03:27,519 --> 00:03:29,120 shortcuts, and we use them all the time 100 00:03:29,120 --> 00:03:30,879 because they often save time and brain 101 00:03:30,879 --> 00:03:33,680 cycles in complex social situations. But 102 00:03:33,680 --> 00:03:35,120 they also create these repeating failure 103 00:03:35,120 --> 00:03:36,959 modes when those shortcuts produce bad 104 00:03:36,959 --> 00:03:38,720 outputs. 105 00:03:38,720 --> 00:03:40,560 Moving on to a few classics. Intentional 106 00:03:40,560 --> 00:03:42,239 blindness isn't quite a bias, but it 107 00:03:42,239 --> 00:03:43,760 functions similarly to one. It's another 108 00:03:43,760 --> 00:03:46,159 mental shortcut. Eliding data that 109 00:03:46,159 --> 00:03:48,560 didn't seem important at the time, but 110 00:03:48,560 --> 00:03:50,480 then we think back on the memory of that 111 00:03:50,480 --> 00:03:52,560 situation, we feel like we have the full 112 00:03:52,560 --> 00:03:54,560 picture until someone points out the man 113 00:03:54,560 --> 00:03:56,560 in the gorilla suit that you missed. As 114 00:03:56,560 --> 00:03:58,640 applied to tech, the same can thing can 115 00:03:58,640 --> 00:04:00,000 happen when we try to remember back to 116 00:04:00,000 --> 00:04:02,000 something like a requirements meeting or 117 00:04:02,000 --> 00:04:03,680 a code review session or a pairing 118 00:04:03,680 --> 00:04:05,840 session where we feel like we have the 119 00:04:05,840 --> 00:04:07,439 full memory of what happened when in 120 00:04:07,439 --> 00:04:08,879 fact if somebody goes back and shows the 121 00:04:08,879 --> 00:04:12,480 transcript, we forgot huge chunks of it. 122 00:04:12,480 --> 00:04:15,200 Recency bias is a sizing outsized 123 00:04:15,200 --> 00:04:17,519 outsized importance to temporally nearby 124 00:04:17,519 --> 00:04:19,759 events when claiming to be using a more 125 00:04:19,759 --> 00:04:22,240 objective measure. How often is last 126 00:04:22,240 --> 00:04:24,400 week's low priority ticket going to 127 00:04:24,400 --> 00:04:27,919 usurp this week's high priority? 128 00:04:27,919 --> 00:04:29,520 Similar to inattentional blindness, 129 00:04:29,520 --> 00:04:31,120 repetition blindness is the tendency of 130 00:04:31,120 --> 00:04:32,560 humans to become desensitized to 131 00:04:32,560 --> 00:04:34,880 repeated stimuli. Reading code dials 132 00:04:34,880 --> 00:04:36,800 this one up to 11 because it's way more 133 00:04:36,800 --> 00:04:39,680 repetitive than normal pros. 134 00:04:39,680 --> 00:04:41,520 And then the big one, overconfidence 135 00:04:41,520 --> 00:04:43,919 bias pervades our industry. Software 136 00:04:43,919 --> 00:04:45,520 people have big egos, especially the 137 00:04:45,520 --> 00:04:47,600 ones who look like me. We collectively 138 00:04:47,600 --> 00:04:49,040 like to assume that we are better than 139 00:04:49,040 --> 00:04:51,520 average, that we've built the best team, 140 00:04:51,520 --> 00:04:53,040 and that through our excellence, we can 141 00:04:53,040 --> 00:04:55,520 beat any odds. But that's not how odds 142 00:04:55,520 --> 00:04:57,520 work. Asking engineers to do time 143 00:04:57,520 --> 00:04:59,040 estimates is such a joke, I don't even 144 00:04:59,040 --> 00:05:00,960 need a punchline, and yet we all do it 145 00:05:00,960 --> 00:05:02,880 anyway. How many of you have shown 146 00:05:02,880 --> 00:05:04,720 someone clear evidence that a project is 147 00:05:04,720 --> 00:05:06,560 on the wrong track and been told, "We 148 00:05:06,560 --> 00:05:07,680 just need to double down and work 149 00:05:07,680 --> 00:05:09,840 harder." How many of you have heard that 150 00:05:09,840 --> 00:05:12,080 a lot more in the last year or two? And 151 00:05:12,080 --> 00:05:14,160 as a quick aside, uh, Dunning Krueger is 152 00:05:14,160 --> 00:05:15,759 bad science. Please stop quoting it when 153 00:05:15,759 --> 00:05:16,880 you mean just a generalized 154 00:05:16,880 --> 00:05:19,199 overconfidence. 155 00:05:19,199 --> 00:05:21,520 Enough problems. Let's talk solutions. 156 00:05:21,520 --> 00:05:22,880 We don't have to do this alone. We have 157 00:05:22,880 --> 00:05:24,479 a few thousand years of experience to 158 00:05:24,479 --> 00:05:26,880 learn from. 159 00:05:26,880 --> 00:05:28,240 If you take one thing away from this 160 00:05:28,240 --> 00:05:30,720 talk, make more checklists. In days long 161 00:05:30,720 --> 00:05:32,400 past, I did a lot of scuba diving. And 162 00:05:32,400 --> 00:05:34,000 there is a saying drilled into every 163 00:05:34,000 --> 00:05:36,080 fresh student. Plan your dive and dive 164 00:05:36,080 --> 00:05:37,919 your plan. When you know you're going 165 00:05:37,919 --> 00:05:39,440 into a stressful situation that requires 166 00:05:39,440 --> 00:05:41,039 a lot of judgment, you want to make sure 167 00:05:41,039 --> 00:05:43,120 you're not burning mental energy unless 168 00:05:43,120 --> 00:05:45,039 you absolutely need to. So, you write 169 00:05:45,039 --> 00:05:46,880 the plan out ahead of time so that in 170 00:05:46,880 --> 00:05:48,320 the moment you can more easily go 171 00:05:48,320 --> 00:05:50,479 through all of the steps. 172 00:05:50,479 --> 00:05:51,600 Checklists can help with a lot of 173 00:05:51,600 --> 00:05:53,039 different situations, but the three most 174 00:05:53,039 --> 00:05:55,520 common that I make are the steps that a 175 00:05:55,520 --> 00:05:57,280 code reviewer should take, the steps for 176 00:05:57,280 --> 00:05:59,199 a code deployment, and the steps for 177 00:05:59,199 --> 00:06:01,919 initial outage response. More generally 178 00:06:01,919 --> 00:06:03,919 though, we can define leading indicators 179 00:06:03,919 --> 00:06:06,080 of where a checklist would help. So the 180 00:06:06,080 --> 00:06:08,400 first is it's a task that happens a lot. 181 00:06:08,400 --> 00:06:10,639 Second, the task is very repetitive or 182 00:06:10,639 --> 00:06:12,800 fairly repetitive. We go back to our 183 00:06:12,800 --> 00:06:14,400 biases. Remember repetition blindness. 184 00:06:14,400 --> 00:06:16,639 So this helps work around that. Third, 185 00:06:16,639 --> 00:06:18,240 the task will be done in a high stress 186 00:06:18,240 --> 00:06:19,919 environment. Often it's going to be 187 00:06:19,919 --> 00:06:21,600 timesensitive. So there's just inherent 188 00:06:21,600 --> 00:06:24,080 pressure to move quickly. And fourth, 189 00:06:24,080 --> 00:06:25,360 making a mistake will have a large 190 00:06:25,360 --> 00:06:27,120 impact, especially when those mistakes 191 00:06:27,120 --> 00:06:29,039 will be difficult to reverse. The more 192 00:06:29,039 --> 00:06:30,479 of these that you have in the problem, 193 00:06:30,479 --> 00:06:32,000 the more checklist will probably help 194 00:06:32,000 --> 00:06:34,720 with it. For the checklist to work, the 195 00:06:34,720 --> 00:06:35,919 most important thing is that it has to 196 00:06:35,919 --> 00:06:37,440 function as a checklist on a literal 197 00:06:37,440 --> 00:06:38,960 level. A human needs to be able to 198 00:06:38,960 --> 00:06:41,520 follow the steps in one at a time in the 199 00:06:41,520 --> 00:06:43,120 order presented. There should not be a 200 00:06:43,120 --> 00:06:44,240 large number of steps that you're 201 00:06:44,240 --> 00:06:45,520 expected to skip and there should be 202 00:06:45,520 --> 00:06:47,600 absolutely no steps that are impossible. 203 00:06:47,600 --> 00:06:49,199 You don't write a checklist with never 204 00:06:49,199 --> 00:06:51,680 write a bug as a step. If the checklist 205 00:06:51,680 --> 00:06:54,240 isn't followable, people will ignore it. 206 00:06:54,240 --> 00:06:55,840 And then need to practice them. I said 207 00:06:55,840 --> 00:06:57,360 before that checklists are good fit for 208 00:06:57,360 --> 00:06:58,960 high pressure situations and no one 209 00:06:58,960 --> 00:07:00,479 wants to learn a new thing while prod is 210 00:07:00,479 --> 00:07:02,400 burning. You go through them ahead of 211 00:07:02,400 --> 00:07:04,240 time. so you have the muscle memory when 212 00:07:04,240 --> 00:07:06,160 you need it. Do pairing sessions, do 213 00:07:06,160 --> 00:07:08,000 disaster simulations, whatever you need 214 00:07:08,000 --> 00:07:09,360 to do to make sure that everyone on the 215 00:07:09,360 --> 00:07:12,080 team can do all of the steps. 216 00:07:12,080 --> 00:07:13,599 And finally, if someone follows a 217 00:07:13,599 --> 00:07:15,199 checklist and gets a bad result, that is 218 00:07:15,199 --> 00:07:17,280 the fault of the system, not the person. 219 00:07:17,280 --> 00:07:18,560 I think more or less everything should 220 00:07:18,560 --> 00:07:20,800 be blameless, but it bears underscoring 221 00:07:20,800 --> 00:07:23,199 here. When building a team process, we 222 00:07:23,199 --> 00:07:24,720 have to assume good faith. That means 223 00:07:24,720 --> 00:07:26,240 that everyone is going to be taking 224 00:07:26,240 --> 00:07:28,800 whatever action next they think is best 225 00:07:28,800 --> 00:07:31,120 to advance the team's goals. If they 226 00:07:31,120 --> 00:07:32,800 pick wrong, that's not a personal 227 00:07:32,800 --> 00:07:34,080 failing. It's a sign they didn't have 228 00:07:34,080 --> 00:07:35,360 enough information to make a good 229 00:07:35,360 --> 00:07:37,759 choice. Checklists. Fill that in. Here's 230 00:07:37,759 --> 00:07:39,360 the information you need, or here's 231 00:07:39,360 --> 00:07:41,039 where to find the information you need, 232 00:07:41,039 --> 00:07:44,720 and here's the correct next action. 233 00:07:44,720 --> 00:07:46,400 As we continue down the path of reducing 234 00:07:46,400 --> 00:07:48,479 the impact of human error, a strong tool 235 00:07:48,479 --> 00:07:50,479 is to rely on objective measurements. 236 00:07:50,479 --> 00:07:52,240 Any place where we can use objective 237 00:07:52,240 --> 00:07:54,160 numbers is one less place that we have 238 00:07:54,160 --> 00:07:58,160 to decide in the moment where to look. 239 00:07:58,160 --> 00:08:00,240 Unit tests. These are a great example of 240 00:08:00,240 --> 00:08:03,759 objectivity in your CI. If CI says the 241 00:08:03,759 --> 00:08:05,440 tests are broken, that's bad. Otherwise, 242 00:08:05,440 --> 00:08:07,440 it's good. Except that's not even 243 00:08:07,440 --> 00:08:09,919 remotely true. Uh tests are a step 244 00:08:09,919 --> 00:08:12,000 towards objective measurement. But how 245 00:08:12,000 --> 00:08:13,199 do we know that we're testing the right 246 00:08:13,199 --> 00:08:15,680 things? Assert true. It's a completely 247 00:08:15,680 --> 00:08:17,919 valid unit test. It will pass. It will 248 00:08:17,919 --> 00:08:19,680 go green in CI and we have learned 249 00:08:19,680 --> 00:08:22,560 nothing about the state of our system. 250 00:08:22,560 --> 00:08:23,680 A lot of you have probably already said 251 00:08:23,680 --> 00:08:24,960 to yourself, well, that's why we check 252 00:08:24,960 --> 00:08:26,879 line coverage during testing. And that 253 00:08:26,879 --> 00:08:29,599 can indeed help. Uh with coverage tools, 254 00:08:29,599 --> 00:08:31,120 we can see a report of which lines and 255 00:08:31,120 --> 00:08:32,640 expressions were run during the 256 00:08:32,640 --> 00:08:34,640 evaluation of our tests. But we're still 257 00:08:34,640 --> 00:08:36,640 left with the same subjective problem. 258 00:08:36,640 --> 00:08:39,360 What do we need to test? 259 00:08:39,360 --> 00:08:41,279 Some teams solve that problem by going 260 00:08:41,279 --> 00:08:43,760 with the 100% coverage model. Every line 261 00:08:43,760 --> 00:08:45,839 must be hit in a test. Personally, I 262 00:08:45,839 --> 00:08:47,519 find that leads to incredibly brittle 263 00:08:47,519 --> 00:08:48,959 projects and incredibly brittle test 264 00:08:48,959 --> 00:08:51,920 suites. it it more or less leads to the 265 00:08:51,920 --> 00:08:53,920 commitment towards objectivity becoming 266 00:08:53,920 --> 00:08:55,920 a numberchasing miniame rather than a 267 00:08:55,920 --> 00:08:58,080 guiding force. 268 00:08:58,080 --> 00:08:59,920 Making a more direct recommendation, I 269 00:08:59,920 --> 00:09:01,760 like to focus on patch coverage in the 270 00:09:01,760 --> 00:09:03,360 context of a single merge request. 271 00:09:03,360 --> 00:09:04,959 There's a bunch of tools for this. Code 272 00:09:04,959 --> 00:09:06,880 cover by Sentry is great. It's easy to 273 00:09:06,880 --> 00:09:08,720 self-host. Uh I would recommend checking 274 00:09:08,720 --> 00:09:10,080 it out if you don't have a tool for 275 00:09:10,080 --> 00:09:12,080 this. 276 00:09:12,080 --> 00:09:13,839 If I can give you one sort of vibe check 277 00:09:13,839 --> 00:09:15,680 for tests and code coverage. You didn't 278 00:09:15,680 --> 00:09:18,320 write enough tests. Less helpful mood. 279 00:09:18,320 --> 00:09:19,680 Have you considered how to test this 280 00:09:19,680 --> 00:09:21,519 line? Or I think this line is important 281 00:09:21,519 --> 00:09:23,200 and it's not currently covered. That's 282 00:09:23,200 --> 00:09:25,200 more helpful. 283 00:09:25,200 --> 00:09:26,720 Static typing is another mostly 284 00:09:26,720 --> 00:09:28,880 objective tool we can lean on. It's only 285 00:09:28,880 --> 00:09:31,360 as useful as the annotations, but if 286 00:09:31,360 --> 00:09:32,800 it's used thoroughly, it can catch a lot 287 00:09:32,800 --> 00:09:34,320 of errors that are frustratingly 288 00:09:34,320 --> 00:09:36,560 difficult for a human to notice. Let the 289 00:09:36,560 --> 00:09:38,880 machines do what they do best. This kind 290 00:09:38,880 --> 00:09:41,279 of thing requires exhaustive analysis. 291 00:09:41,279 --> 00:09:43,600 Robots are great at that. 292 00:09:43,600 --> 00:09:45,279 A classic programmer joke is that a QA 293 00:09:45,279 --> 00:09:47,200 engineer walks into a bar, they order 1, 294 00:09:47,200 --> 00:09:50,399 10, 0, 1 million, negative 1, and so on. 295 00:09:50,399 --> 00:09:52,720 Static typing solves this problem by 296 00:09:52,720 --> 00:09:55,040 reducing the input space to only the 297 00:09:55,040 --> 00:09:56,399 types we've allowed. We don't have to 298 00:09:56,399 --> 00:09:58,000 write a million unit tests for every 299 00:09:58,000 --> 00:09:59,680 single function to figure out what would 300 00:09:59,680 --> 00:10:01,519 happen if we pass in every possible 301 00:10:01,519 --> 00:10:03,920 invalid type. The type checker just 302 00:10:03,920 --> 00:10:06,240 asserts that can't happen. Now, Python's 303 00:10:06,240 --> 00:10:07,839 still relatively early early in its 304 00:10:07,839 --> 00:10:10,080 static typing journey. So you can't 305 00:10:10,080 --> 00:10:11,839 actually do something like positive 306 00:10:11,839 --> 00:10:14,079 integer as a type but every little bit 307 00:10:14,079 --> 00:10:16,800 does help. 308 00:10:16,800 --> 00:10:18,880 Code liners and formatterers provide uh 309 00:10:18,880 --> 00:10:20,399 not just objectivity but also 310 00:10:20,399 --> 00:10:22,240 automation. If I never have to have 311 00:10:22,240 --> 00:10:24,079 another argument about quote types or 312 00:10:24,079 --> 00:10:25,920 spaces around operators, it'll still be 313 00:10:25,920 --> 00:10:27,920 too soon. This continues our running 314 00:10:27,920 --> 00:10:29,680 pattern of offloading the tasks that 315 00:10:29,680 --> 00:10:31,360 machines can do well and let the squishy 316 00:10:31,360 --> 00:10:34,000 humans focus on the rest. 317 00:10:34,000 --> 00:10:36,240 Consistency isn't just for fun. It's a 318 00:10:36,240 --> 00:10:38,000 key piece towards making a codebase more 319 00:10:38,000 --> 00:10:39,760 maintainable by a team rather than 320 00:10:39,760 --> 00:10:41,200 matching the bug bears of a single 321 00:10:41,200 --> 00:10:43,680 author. 322 00:10:43,680 --> 00:10:45,120 Always be on the lookout for cultural 323 00:10:45,120 --> 00:10:46,640 rules that you can promote into a 324 00:10:46,640 --> 00:10:48,880 plug-in somewhere. As with everything 325 00:10:48,880 --> 00:10:50,560 we've discussed, not every check is 326 00:10:50,560 --> 00:10:53,680 going to be easy to just yeet into code. 327 00:10:53,680 --> 00:10:55,839 But when you can shift that mental load 328 00:10:55,839 --> 00:10:58,959 into a script into a flake plugin, it's 329 00:10:58,959 --> 00:11:00,800 a good trade. And for the record, null 330 00:11:00,800 --> 00:11:04,000 goes before blank. 331 00:11:04,000 --> 00:11:05,360 Even after we've automated everything 332 00:11:05,360 --> 00:11:06,720 that we can apply this objective 333 00:11:06,720 --> 00:11:08,720 analysis to, we're still going to need 334 00:11:08,720 --> 00:11:10,480 human judgment to decide if we've 335 00:11:10,480 --> 00:11:12,640 written the right thing or not, almost 336 00:11:12,640 --> 00:11:14,160 always there's going to be a first 337 00:11:14,160 --> 00:11:15,600 person who can provide some judgment, 338 00:11:15,600 --> 00:11:17,279 the person who wrote the change. And if 339 00:11:17,279 --> 00:11:18,880 you do pair programming, well, you've 340 00:11:18,880 --> 00:11:20,399 got two pairs of eyes at that point in 341 00:11:20,399 --> 00:11:22,560 the process. But industry standard 342 00:11:22,560 --> 00:11:25,040 workflows are that we have a later step 343 00:11:25,040 --> 00:11:27,120 in the workflow where one or more people 344 00:11:27,120 --> 00:11:29,279 provide feedback on correctness and 345 00:11:29,279 --> 00:11:31,839 completeness. 346 00:11:31,839 --> 00:11:33,360 Code review is the most common place we 347 00:11:33,360 --> 00:11:34,640 check each other's work, but the same 348 00:11:34,640 --> 00:11:36,800 principles apply in many cases. Many of 349 00:11:36,800 --> 00:11:38,880 you flew here and heard some variation 350 00:11:38,880 --> 00:11:40,640 of arm doors and cross check shortly 351 00:11:40,640 --> 00:11:42,959 before takeoff. Every industry where 352 00:11:42,959 --> 00:11:44,640 safety is critical has some form of 353 00:11:44,640 --> 00:11:46,880 multi-person review. I'm very glad my 354 00:11:46,880 --> 00:11:48,720 code isn't as critical as an airplane 355 00:11:48,720 --> 00:11:50,160 door seal, but I can still learn from 356 00:11:50,160 --> 00:11:52,240 their solutions. 357 00:11:52,240 --> 00:11:53,760 To know how to build a good code review 358 00:11:53,760 --> 00:11:55,760 process, specifically we need to look at 359 00:11:55,760 --> 00:11:58,640 what we want out of it. I usually divide 360 00:11:58,640 --> 00:12:00,320 things into four pillars. The most 361 00:12:00,320 --> 00:12:02,399 obvious and the place where most teams 362 00:12:02,399 --> 00:12:04,720 stop is finding and fixing errors in the 363 00:12:04,720 --> 00:12:07,200 code before it advances closer to users. 364 00:12:07,200 --> 00:12:09,360 But in my opinion, no less important is 365 00:12:09,360 --> 00:12:11,360 that team members can learn a lot by 366 00:12:11,360 --> 00:12:13,120 reviewing code, especially code they 367 00:12:13,120 --> 00:12:14,639 haven't worked in, text acts they're not 368 00:12:14,639 --> 00:12:16,800 familiar with, etc. They might not find 369 00:12:16,800 --> 00:12:18,399 as many bugs at first, but never 370 00:12:18,399 --> 00:12:20,480 underestimate the value of, hey, I don't 371 00:12:20,480 --> 00:12:22,079 understand this. Could you explain it to 372 00:12:22,079 --> 00:12:24,720 me? That finds a lot of obvious in 373 00:12:24,720 --> 00:12:27,760 retrospect errors. Uh code review can 374 00:12:27,760 --> 00:12:29,120 also serve as a place to discuss 375 00:12:29,120 --> 00:12:30,880 architectural consistency or pull in 376 00:12:30,880 --> 00:12:33,040 subject matter experts. However, that 377 00:12:33,040 --> 00:12:34,959 often leads to frustration as people 378 00:12:34,959 --> 00:12:36,560 late in the process have to either 379 00:12:36,560 --> 00:12:39,279 rework or discard code. Try and do that 380 00:12:39,279 --> 00:12:41,920 much earlier. [snorts] 381 00:12:41,920 --> 00:12:43,360 I said the main reason that most teams 382 00:12:43,360 --> 00:12:45,200 do code review is to reduce defects. And 383 00:12:45,200 --> 00:12:47,360 sadly, I find it kind of terrible for 384 00:12:47,360 --> 00:12:49,519 that. Every one of us has reviewed a 385 00:12:49,519 --> 00:12:51,680 change that later was found to have an 386 00:12:51,680 --> 00:12:54,480 extremely glaringly obvious bug. it 387 00:12:54,480 --> 00:12:56,399 sailed right through the process and 388 00:12:56,399 --> 00:12:58,480 nobody noticed until the bug was hit by 389 00:12:58,480 --> 00:13:01,440 a user. Code reviewed processes can also 390 00:13:01,440 --> 00:13:02,880 create a lot of friction points in a 391 00:13:02,880 --> 00:13:04,959 team. People can feel like my code isn't 392 00:13:04,959 --> 00:13:06,720 getting reviewed fast enough. My agency 393 00:13:06,720 --> 00:13:08,480 has been reduced and I'm not seeing a 394 00:13:08,480 --> 00:13:10,800 benefit. And it's also draining. We'll 395 00:13:10,800 --> 00:13:12,079 talk more about the mental burden in a 396 00:13:12,079 --> 00:13:14,240 moment. But that load is real and it's 397 00:13:14,240 --> 00:13:16,880 hard to work around. 398 00:13:16,880 --> 00:13:18,800 Getting on my soap box, I think part of 399 00:13:18,800 --> 00:13:20,399 the problem is that there's two very 400 00:13:20,399 --> 00:13:22,560 different tasks we lump together as code 401 00:13:22,560 --> 00:13:25,200 review. Micro review is examining single 402 00:13:25,200 --> 00:13:27,120 lines or a small chunk on its own. You 403 00:13:27,120 --> 00:13:28,800 might have some context already. You 404 00:13:28,800 --> 00:13:30,399 know how the codebase is laid out, but 405 00:13:30,399 --> 00:13:32,240 the general idea of a micro review is 406 00:13:32,240 --> 00:13:35,440 it's as low context as possible. This is 407 00:13:35,440 --> 00:13:36,560 where you get a lot of your usual 408 00:13:36,560 --> 00:13:38,560 nitpicking and bike shedding or this 409 00:13:38,560 --> 00:13:40,959 doesn't match best practice reviews. In 410 00:13:40,959 --> 00:13:42,880 general, micro reviews can work okay to 411 00:13:42,880 --> 00:13:45,360 find very narrowly scoped issues, but a 412 00:13:45,360 --> 00:13:47,120 lot of things get missed. If the problem 413 00:13:47,120 --> 00:13:49,040 only manifests on more than a dozen 414 00:13:49,040 --> 00:13:50,880 lines, uh you're probably not going to 415 00:13:50,880 --> 00:13:52,800 notice it in this kind of micro review. 416 00:13:52,800 --> 00:13:54,240 Macro reviews are the big picture. 417 00:13:54,240 --> 00:13:56,000 You're looking over the entire diff. 418 00:13:56,000 --> 00:13:57,279 You're trying to figure out what are the 419 00:13:57,279 --> 00:13:59,040 architectural concerns, where will we 420 00:13:59,040 --> 00:14:00,880 expand this in the future, stuff like 421 00:14:00,880 --> 00:14:03,279 that. I don't think macro code reviews 422 00:14:03,279 --> 00:14:05,519 work very well. 423 00:14:05,519 --> 00:14:07,279 Like we have these solutions for the 424 00:14:07,279 --> 00:14:09,360 small nits in the bike sheds. We can use 425 00:14:09,360 --> 00:14:12,160 these small peeppole code reviews. We 426 00:14:12,160 --> 00:14:13,839 can use lynching and formatting tools. 427 00:14:13,839 --> 00:14:15,760 We can ride our own flake 8 plugins. 428 00:14:15,760 --> 00:14:17,440 It's not great, but we can usually find 429 00:14:17,440 --> 00:14:20,160 things and we can maybe sometimes catch 430 00:14:20,160 --> 00:14:22,079 poorly laid out code. Very often those 431 00:14:22,079 --> 00:14:23,600 kinds of problems are only obvious 432 00:14:23,600 --> 00:14:25,360 either in retrospect or 6 months later 433 00:14:25,360 --> 00:14:26,720 when you go to try to add a new feature 434 00:14:26,720 --> 00:14:28,240 and you say who wrote this and the 435 00:14:28,240 --> 00:14:31,440 person was you. Uh but we don't really 436 00:14:31,440 --> 00:14:33,279 have a great answer for the big 437 00:14:33,279 --> 00:14:36,240 problems. Things like well in this diff 438 00:14:36,240 --> 00:14:38,240 they didn't remember to put in 439 00:14:38,240 --> 00:14:39,760 permissions checks. There's just 440 00:14:39,760 --> 00:14:41,600 something missing from the code. It's 441 00:14:41,600 --> 00:14:43,760 really hard for a human to notice 442 00:14:43,760 --> 00:14:45,440 something that isn't there. Now, 443 00:14:45,440 --> 00:14:47,199 checklist could in theory help with 444 00:14:47,199 --> 00:14:49,519 this, but if you write a checklist with 445 00:14:49,519 --> 00:14:50,880 everything that somebody should check 446 00:14:50,880 --> 00:14:52,480 for, it's going to be a thousand items 447 00:14:52,480 --> 00:14:54,480 long, and that doesn't help. So, we end 448 00:14:54,480 --> 00:14:56,079 up just rolling the dice a lot of the 449 00:14:56,079 --> 00:14:59,199 time. To get an even soapier box, the 450 00:14:59,199 --> 00:15:00,800 hot take that made me want to write this 451 00:15:00,800 --> 00:15:02,480 talk, I think we are collectively 452 00:15:02,480 --> 00:15:04,240 deluding ourselves into thinking that 453 00:15:04,240 --> 00:15:06,320 macros scale code reviews work, that we 454 00:15:06,320 --> 00:15:08,480 are simply the best engineers, and so we 455 00:15:08,480 --> 00:15:10,399 can see all of the problems. I think 456 00:15:10,399 --> 00:15:12,399 instinctively most overwhelmed engineers 457 00:15:12,399 --> 00:15:14,160 adopt a process of not really reviewing 458 00:15:14,160 --> 00:15:16,000 the code itself, but reviewing the 459 00:15:16,000 --> 00:15:18,000 mental state of the author at the time 460 00:15:18,000 --> 00:15:19,920 of writing with the code as a window 461 00:15:19,920 --> 00:15:22,480 into that mental state. To explain what 462 00:15:22,480 --> 00:15:24,160 I mean by that, imagine someone on your 463 00:15:24,160 --> 00:15:26,320 team writes a new API feature. It has 464 00:15:26,320 --> 00:15:28,560 three very similar endpoints. You're 465 00:15:28,560 --> 00:15:29,839 probably going to check the first one 466 00:15:29,839 --> 00:15:30,959 very thoroughly. You're going to read 467 00:15:30,959 --> 00:15:32,639 through the diff. You're going to try to 468 00:15:32,639 --> 00:15:34,399 unpack all of the context into your 469 00:15:34,399 --> 00:15:36,399 brain. The second one, you're going to 470 00:15:36,399 --> 00:15:37,519 look through, but maybe not as 471 00:15:37,519 --> 00:15:38,560 thoroughly. And by the time you get to 472 00:15:38,560 --> 00:15:39,760 the third one, you're just going to see, 473 00:15:39,760 --> 00:15:41,360 well, does it kind of match the first 474 00:15:41,360 --> 00:15:44,720 two. This works, okay? Because if they 475 00:15:44,720 --> 00:15:46,320 did the first one correctly, they 476 00:15:46,320 --> 00:15:48,079 probably understood what was going on. 477 00:15:48,079 --> 00:15:49,440 And by the time you get to the second 478 00:15:49,440 --> 00:15:51,120 and third, they still understood what 479 00:15:51,120 --> 00:15:53,040 was going on. And so, well, this isn't 480 00:15:53,040 --> 00:15:55,600 perfect. It does correlate highly with 481 00:15:55,600 --> 00:15:57,680 producing the correct code. Not a 482 00:15:57,680 --> 00:15:59,519 perfect heristic, but it works okay in a 483 00:15:59,519 --> 00:16:01,279 lot of situations. 484 00:16:01,279 --> 00:16:02,639 And I know I said this wasn't really a 485 00:16:02,639 --> 00:16:03,839 talk about AI, but I do want to 486 00:16:03,839 --> 00:16:05,360 highlight this as one of my biggest 487 00:16:05,360 --> 00:16:07,360 concerns about any kind of heavily 488 00:16:07,360 --> 00:16:10,160 automated code workflow. This kind of 489 00:16:10,160 --> 00:16:12,959 vibe based review, it doesn't work very 490 00:16:12,959 --> 00:16:16,160 well with AI. Pios of linear algebra do 491 00:16:16,160 --> 00:16:17,680 not have a mental, let alone a mental 492 00:16:17,680 --> 00:16:19,279 state. Because of their stochcastic 493 00:16:19,279 --> 00:16:20,800 nature, they are equally likely to 494 00:16:20,800 --> 00:16:22,639 produce incorrect code anywhere in the 495 00:16:22,639 --> 00:16:24,399 process. So if you exhaustively review 496 00:16:24,399 --> 00:16:26,240 the first of those three endpoints and 497 00:16:26,240 --> 00:16:28,399 it's perfect, it tells you basically 498 00:16:28,399 --> 00:16:30,399 nothing about the second and third. This 499 00:16:30,399 --> 00:16:32,639 means that reviewing AI generated code 500 00:16:32,639 --> 00:16:34,560 is much much harder than the equivalent 501 00:16:34,560 --> 00:16:36,800 volume from a human. The same shortcuts 502 00:16:36,800 --> 00:16:39,199 no longer apply. 503 00:16:39,199 --> 00:16:40,959 Before I sent negative, code review is 504 00:16:40,959 --> 00:16:42,639 good and you should keep doing it. My 505 00:16:42,639 --> 00:16:44,399 problems with review practices are when 506 00:16:44,399 --> 00:16:46,000 you put too much faith in them and thus 507 00:16:46,000 --> 00:16:47,680 too much pressure on people to just code 508 00:16:47,680 --> 00:16:50,079 review even harder. 509 00:16:50,079 --> 00:16:51,600 If someone comes to you with a new tool, 510 00:16:51,600 --> 00:16:53,199 maybe it's an AI agent, maybe it's a 511 00:16:53,199 --> 00:16:54,959 boilerplate generator, whatever. They 512 00:16:54,959 --> 00:16:56,880 come to you and they say, "Yeah, this 513 00:16:56,880 --> 00:16:58,399 tool is brand new. It's experimental, 514 00:16:58,399 --> 00:17:01,040 but it's fine. We'll notice if the tool 515 00:17:01,040 --> 00:17:02,560 does anything wrong, and we'll fix it. 516 00:17:02,560 --> 00:17:04,880 They're proposing a very dangerous game. 517 00:17:04,880 --> 00:17:06,880 As we've discussed, humans can fail in a 518 00:17:06,880 --> 00:17:09,120 lot of ways, and even our best efforts 519 00:17:09,120 --> 00:17:11,600 boil down to basically, I promise to 520 00:17:11,600 --> 00:17:14,240 never ever make a mistake. That doesn't 521 00:17:14,240 --> 00:17:17,039 work. Hypervigilance is a defense 522 00:17:17,039 --> 00:17:19,439 mechanism, a state of constant awareness 523 00:17:19,439 --> 00:17:21,039 of everything around you that could go 524 00:17:21,039 --> 00:17:22,959 wrong and everything around you that 525 00:17:22,959 --> 00:17:25,679 might hurt you. I find myself locked 526 00:17:25,679 --> 00:17:27,760 into this more and more at work. And I 527 00:17:27,760 --> 00:17:29,679 think I'm not the only one. As we're all 528 00:17:29,679 --> 00:17:31,280 expected to move faster and faster, it 529 00:17:31,280 --> 00:17:32,799 very often feels like the expectations 530 00:17:32,799 --> 00:17:35,360 on me are exactly that. Never ever make 531 00:17:35,360 --> 00:17:37,679 a mistake. The plan that doesn't work. 532 00:17:37,679 --> 00:17:39,679 Hypervigilance leads directly to anxiety 533 00:17:39,679 --> 00:17:41,760 and exhaustion, which only tightens the 534 00:17:41,760 --> 00:17:43,520 spiral downwards. And at the bottom is 535 00:17:43,520 --> 00:17:45,919 burnout. I am not a mental health 536 00:17:45,919 --> 00:17:47,760 professional, but many have written or 537 00:17:47,760 --> 00:17:49,280 spoken at length about burnout and how 538 00:17:49,280 --> 00:17:51,440 it plagues our industry like few others. 539 00:17:51,440 --> 00:17:53,039 I'd like to highlight a talk from Pyon 540 00:17:53,039 --> 00:17:55,919 Australia 2016. Oh god, I feel old as a 541 00:17:55,919 --> 00:17:57,280 great place to start. If any of these 542 00:17:57,280 --> 00:17:58,880 words about hypervigilance are landing 543 00:17:58,880 --> 00:18:00,880 extra hard for you, just know that you 544 00:18:00,880 --> 00:18:02,880 are not alone. This isn't your fault. 545 00:18:02,880 --> 00:18:05,039 We've somehow herded our entire industry 546 00:18:05,039 --> 00:18:06,960 on a path that pushes every possible 547 00:18:06,960 --> 00:18:09,840 stress button in all of us. 548 00:18:09,840 --> 00:18:12,160 Computers have RAM. You can only load so 549 00:18:12,160 --> 00:18:14,000 many programs into memory at once. 550 00:18:14,000 --> 00:18:16,080 Cognitive load is similar. Each person 551 00:18:16,080 --> 00:18:18,080 can only have so many background tasks 552 00:18:18,080 --> 00:18:19,919 before it becomes overwhelming. The 553 00:18:19,919 --> 00:18:21,440 limit varies with the person and 554 00:18:21,440 --> 00:18:23,440 situation. It's not a perfect metaphor, 555 00:18:23,440 --> 00:18:25,760 but in general, map out where you want 556 00:18:25,760 --> 00:18:27,760 to invest your cognitive load and think 557 00:18:27,760 --> 00:18:29,280 about when it's too much and when you 558 00:18:29,280 --> 00:18:31,919 need to move a cognitive task to appear. 559 00:18:31,919 --> 00:18:33,280 These background cognitive loops, 560 00:18:33,280 --> 00:18:35,280 they're mental tasks that require 561 00:18:35,280 --> 00:18:36,960 constant or near constant awareness. 562 00:18:36,960 --> 00:18:39,440 Some of them are technical, like anytime 563 00:18:39,440 --> 00:18:41,440 I'm looking at code related TLSerts, I 564 00:18:41,440 --> 00:18:42,720 have to have a loop going on the back of 565 00:18:42,720 --> 00:18:44,240 my brain of searching for the following 566 00:18:44,240 --> 00:18:46,240 300 things that would result in using 567 00:18:46,240 --> 00:18:48,720 TLS unsafeely. Sometimes they're social 568 00:18:48,720 --> 00:18:50,880 tasks like, "Hey, have I forgotten to 569 00:18:50,880 --> 00:18:52,480 finish reviewing someone's PR? Are they 570 00:18:52,480 --> 00:18:54,559 waiting on me? Are they mad at me?" The 571 00:18:54,559 --> 00:18:56,160 more senior you get in your career, the 572 00:18:56,160 --> 00:18:57,360 more of these that you're expected to 573 00:18:57,360 --> 00:18:59,520 just juggle simultaneously. They don't 574 00:18:59,520 --> 00:19:01,919 get easier over time. 575 00:19:01,919 --> 00:19:03,600 Being overloaded leads to increased 576 00:19:03,600 --> 00:19:05,760 error rates and general misery. Looking 577 00:19:05,760 --> 00:19:07,200 back in the first half of this talk, 578 00:19:07,200 --> 00:19:08,880 many of these loops can be moved into 579 00:19:08,880 --> 00:19:11,280 support tools. Don't make people have to 580 00:19:11,280 --> 00:19:13,679 remember, did I forget a PR? Have a 581 00:19:13,679 --> 00:19:16,240 Slackbot post them once or twice a day. 582 00:19:16,240 --> 00:19:18,000 Now, that's an easy one. Many of these 583 00:19:18,000 --> 00:19:21,200 are way harder to automate. But, uh, as 584 00:19:21,200 --> 00:19:22,720 much as I've said unkind things about 585 00:19:22,720 --> 00:19:25,520 AI, if you think through and you decide 586 00:19:25,520 --> 00:19:27,440 given the balance of stressors, if this 587 00:19:27,440 --> 00:19:30,160 net gives you more bandwidth to focus on 588 00:19:30,160 --> 00:19:31,760 high priority things, if you can throw a 589 00:19:31,760 --> 00:19:34,080 clawed agent at some low priority thing 590 00:19:34,080 --> 00:19:35,919 that you were otherwise not going to get 591 00:19:35,919 --> 00:19:39,039 to, maybe that's great. If so, do it. 592 00:19:39,039 --> 00:19:41,440 And talk to your team. Uh, they are 593 00:19:41,440 --> 00:19:42,880 there to support you as much as you 594 00:19:42,880 --> 00:19:44,880 support them. and shifting tasks around 595 00:19:44,880 --> 00:19:46,720 to match the eb and flow of availability 596 00:19:46,720 --> 00:19:49,039 can help that a lot. 597 00:19:49,039 --> 00:19:51,520 Phew, that was a lot. What did we cover 598 00:19:51,520 --> 00:19:52,960 today? We went through some of the 599 00:19:52,960 --> 00:19:54,720 common failure modes of human brains and 600 00:19:54,720 --> 00:19:56,240 how to supplement squishy humans with 601 00:19:56,240 --> 00:19:58,480 checklists. We talked about objective 602 00:19:58,480 --> 00:20:00,080 support tools and how they fit into a 603 00:20:00,080 --> 00:20:01,760 development workflow and how to set up 604 00:20:01,760 --> 00:20:03,679 review workflows that are resilient to 605 00:20:03,679 --> 00:20:05,120 human errors. And finally, we talked 606 00:20:05,120 --> 00:20:07,039 about cognitive load and how to shift 607 00:20:07,039 --> 00:20:09,679 load before it shifts you. 608 00:20:09,679 --> 00:20:11,200 And as a note to leave on, treat 609 00:20:11,200 --> 00:20:12,640 yourself and your team with kindness and 610 00:20:12,640 --> 00:20:14,240 compassion because you can be bad at 611 00:20:14,240 --> 00:20:16,320 things and that should be okay. Thank 612 00:20:16,320 --> 00:20:20,334 you very much. Any questions? [applause] 613 00:20:22,559 --> 00:20:24,480 Thank you, Noah. Well, we have some time 614 00:20:24,480 --> 00:20:28,194 for some questions. [snorts] 615 00:20:33,679 --> 00:20:36,159 In the last six months or so, me and my 616 00:20:36,159 --> 00:20:39,280 team have been using AI a lot more for 617 00:20:39,280 --> 00:20:43,760 our coding, and we're finding 618 00:20:43,760 --> 00:20:45,679 it's it's changed the way we've worked 619 00:20:45,679 --> 00:20:49,039 and the way we think a lot. um for 620 00:20:49,039 --> 00:20:51,760 example, trying to switch between tasks 621 00:20:51,760 --> 00:20:53,919 while you're waiting for one prompt to 622 00:20:53,919 --> 00:20:57,039 finish or working through very very 623 00:20:57,039 --> 00:21:02,240 dense um responses from the AI. 624 00:21:02,240 --> 00:21:06,320 Have you given more thought to how that 625 00:21:06,320 --> 00:21:09,200 might change the way we think and work 626 00:21:09,200 --> 00:21:12,080 and work through these kind of issues as 627 00:21:12,080 --> 00:21:17,400 AI for coding becomes more common? 628 00:21:17,679 --> 00:21:20,240 The the unkind answer is it basically 629 00:21:20,240 --> 00:21:21,919 puts everyone into the same seat that 630 00:21:21,919 --> 00:21:24,880 staff engineers are in every day. Uh you 631 00:21:24,880 --> 00:21:27,039 know I I get interrupted pretty much 632 00:21:27,039 --> 00:21:28,799 constantly if I get four hours to work 633 00:21:28,799 --> 00:21:31,679 on a problem. That is amazing unless I'm 634 00:21:31,679 --> 00:21:33,200 working on the weekends which I never do 635 00:21:33,200 --> 00:21:35,600 and my team should never do either. Uh 636 00:21:35,600 --> 00:21:38,080 but it it is pushing that that 637 00:21:38,080 --> 00:21:40,240 interruption load into everyone because 638 00:21:40,240 --> 00:21:42,960 every task now gets interrupted every 15 639 00:21:42,960 --> 00:21:44,559 minutes when your tokens finish 640 00:21:44,559 --> 00:21:48,000 rendering. It's not good. Uh I mean the 641 00:21:48,000 --> 00:21:49,840 answer is we should be giving people 642 00:21:49,840 --> 00:21:51,520 more focus time. That has been the 643 00:21:51,520 --> 00:21:54,080 answer continuously since the 1970s and 644 00:21:54,080 --> 00:21:59,480 we haven't been doing it. So yikes. 645 00:22:08,080 --> 00:22:11,039 I um thank thank you. No, I really liked 646 00:22:11,039 --> 00:22:12,960 the there was a really good oneliner 647 00:22:12,960 --> 00:22:14,960 that I tried to write down like verbatim 648 00:22:14,960 --> 00:22:16,799 about uh the blameless culture of 649 00:22:16,799 --> 00:22:19,120 following the checklists as being about 650 00:22:19,120 --> 00:22:20,880 uh having incomplete information but 651 00:22:20,880 --> 00:22:22,400 having the best intentions for the team. 652 00:22:22,400 --> 00:22:24,880 Is that a common I haven't encountered 653 00:22:24,880 --> 00:22:26,080 that thought before and I really liked 654 00:22:26,080 --> 00:22:28,159 it. Is it a common thought that I should 655 00:22:28,159 --> 00:22:30,000 look into reading more about? I come 656 00:22:30,000 --> 00:22:32,400 from an S sur background. So the idea of 657 00:22:32,400 --> 00:22:34,960 a blameless retrospective is the 658 00:22:34,960 --> 00:22:37,600 standard. It's that but for every 659 00:22:37,600 --> 00:22:39,520 meeting and every human interaction you 660 00:22:39,520 --> 00:22:42,400 are in of I don't know I in my life I 661 00:22:42,400 --> 00:22:44,480 find blame is pointless and doesn't help 662 00:22:44,480 --> 00:22:46,799 me with anything. So like this is just 663 00:22:46,799 --> 00:22:49,600 how I am and so it's easy for me to say. 664 00:22:49,600 --> 00:22:51,280 Uh I I appreciate that this is not 665 00:22:51,280 --> 00:22:52,960 always easy for especially the people 666 00:22:52,960 --> 00:22:54,880 above me who are getting very pointed 667 00:22:54,880 --> 00:22:56,640 questions about why you know why prod 668 00:22:56,640 --> 00:22:58,320 was down why we shipped this bug and 669 00:22:58,320 --> 00:23:00,640 like I don't know guys cuz it happened 670 00:23:00,640 --> 00:23:03,440 cuz you asked me to ship four PRs in the 671 00:23:03,440 --> 00:23:05,679 same day. Uh 672 00:23:05,679 --> 00:23:08,080 but you know just the the same culture 673 00:23:08,080 --> 00:23:09,919 as a blameless retrospective but applied 674 00:23:09,919 --> 00:23:13,159 to everything. 675 00:23:18,720 --> 00:23:23,280 Um hi. Um with the talk um I think I 676 00:23:23,280 --> 00:23:25,679 resonate a lot um what you just covered 677 00:23:25,679 --> 00:23:28,320 here. Um there's probably one thing I'd 678 00:23:28,320 --> 00:23:31,200 like to ask is um because I'm a default 679 00:23:31,200 --> 00:23:34,640 code reviewers at my work and usually 90 680 00:23:34,640 --> 00:23:36,640 to 80% of the times I'm the one who's 681 00:23:36,640 --> 00:23:39,360 doing the heavy lifting of code review. 682 00:23:39,360 --> 00:23:43,200 So usually my day would be like 50% code 683 00:23:43,200 --> 00:23:47,120 review or 50% actually getting work 684 00:23:47,120 --> 00:23:51,039 done. Um we are trying to incentivize um 685 00:23:51,039 --> 00:23:52,640 more people to the code reviews like do 686 00:23:52,640 --> 00:23:55,280 you have any pointers to uh like promote 687 00:23:55,280 --> 00:23:57,679 this kind of culture like just because 688 00:23:57,679 --> 00:23:59,919 you're default reviewer doesn't mean 689 00:23:59,919 --> 00:24:03,679 it's your job to do it like 50 or 70% of 690 00:24:03,679 --> 00:24:06,799 the times. So speaking from a a GitHub 691 00:24:06,799 --> 00:24:08,559 ccentric point of view because that's 692 00:24:08,559 --> 00:24:10,960 where I live my life. Um they do have 693 00:24:10,960 --> 00:24:12,960 features to auto assign to different 694 00:24:12,960 --> 00:24:16,640 people. Um so that that will make you no 695 00:24:16,640 --> 00:24:18,240 longer the default reviewer. If you set 696 00:24:18,240 --> 00:24:20,159 the code owner to be a team and you set 697 00:24:20,159 --> 00:24:22,080 the team to be default assigned to 698 00:24:22,080 --> 00:24:25,279 people it'll roundroin them. Um that 699 00:24:25,279 --> 00:24:27,679 that can help. Uh I will point out that 700 00:24:27,679 --> 00:24:29,760 that doesn't necessarily always fix the 701 00:24:29,760 --> 00:24:32,640 problem. you know, if you are on the 702 00:24:32,640 --> 00:24:34,480 more senior side on your team, you will 703 00:24:34,480 --> 00:24:36,880 very often still be expected to take a 704 00:24:36,880 --> 00:24:39,760 look at things. Um, on the nonwork side, 705 00:24:39,760 --> 00:24:41,600 a thing that I like, I would probably 706 00:24:41,600 --> 00:24:43,200 never do it at work because it would 707 00:24:43,200 --> 00:24:45,679 create perverse incentives, but uh, the 708 00:24:45,679 --> 00:24:48,640 Twisted Project has a a leaderboard for 709 00:24:48,640 --> 00:24:50,799 code reviews and triage actions that 710 00:24:50,799 --> 00:24:52,880 every one of them gets you points. Um, 711 00:24:52,880 --> 00:24:54,720 so they've just kind of gified it. It's 712 00:24:54,720 --> 00:24:56,400 great. I love it. I I feel like it 713 00:24:56,400 --> 00:24:58,080 worked. This wouldn't go well. Maybe I'd 714 00:24:58,080 --> 00:25:00,159 try it if it was a team that really 715 00:25:00,159 --> 00:25:02,000 thought that it could be fun, but I 716 00:25:02,000 --> 00:25:03,120 would certainly be worried that it 717 00:25:03,120 --> 00:25:04,400 wouldn't go well. 718 00:25:04,400 --> 00:25:06,720 We have I'm going to talk at the 719 00:25:06,720 --> 00:25:08,400 leaderboard for months. 720 00:25:08,400 --> 00:25:11,840 Well, uh, tell them to get you a nice 721 00:25:11,840 --> 00:25:15,087 trophy or something for it. [laughter] 722 00:25:15,440 --> 00:25:18,080 Any other questions? 723 00:25:18,080 --> 00:25:20,159 Okay. 724 00:25:20,159 --> 00:25:21,840 So, think about that checklist, which I 725 00:25:21,840 --> 00:25:23,520 think is a really great point. Um, a lot 726 00:25:23,520 --> 00:25:25,360 of times we see suggestion checklist. 727 00:25:25,360 --> 00:25:28,400 you mentioned aviation. Uh surgery is a 728 00:25:28,400 --> 00:25:29,679 big one where it's made a huge 729 00:25:29,679 --> 00:25:32,240 difference. These are all very 730 00:25:32,240 --> 00:25:34,880 dangerous and regulated industries where 731 00:25:34,880 --> 00:25:36,720 it's seems like it's much easier to 732 00:25:36,720 --> 00:25:38,400 enumerate the things that you should 733 00:25:38,400 --> 00:25:40,480 check like count all the tools and count 734 00:25:40,480 --> 00:25:42,400 all the sponges before and after the 735 00:25:42,400 --> 00:25:45,679 operation. How do you balance 736 00:25:45,679 --> 00:25:48,320 um creating a checklist of that's not 737 00:25:48,320 --> 00:25:51,679 too generic but that is specific enough 738 00:25:51,679 --> 00:25:53,600 for each of the tasks that you have 739 00:25:53,600 --> 00:25:55,120 without sort of blowing it out with a 740 00:25:55,120 --> 00:25:56,720 lot of hypotheticals. 741 00:25:56,720 --> 00:25:58,720 Think about the harms that your team can 742 00:25:58,720 --> 00:26:02,799 can create. Um you know I I know Allan 743 00:26:02,799 --> 00:26:04,159 specifically is more on the academic 744 00:26:04,159 --> 00:26:05,840 side. So it's probably going to be like 745 00:26:05,840 --> 00:26:08,720 we for your team it's we can't let bad 746 00:26:08,720 --> 00:26:10,559 research out into the world. Like that 747 00:26:10,559 --> 00:26:12,240 that is what you can do that can create 748 00:26:12,240 --> 00:26:13,679 harm. So, you want to enumerate the 749 00:26:13,679 --> 00:26:15,520 things that will cause that to happen. 750 00:26:15,520 --> 00:26:17,600 For me, it would be, you know, people 751 00:26:17,600 --> 00:26:19,440 getting overcharged for furniture or 752 00:26:19,440 --> 00:26:20,880 people getting shipped the wrong 753 00:26:20,880 --> 00:26:23,520 furniture. Uh, you know, again, it's not 754 00:26:23,520 --> 00:26:24,960 nearly on the same level of surgery, and 755 00:26:24,960 --> 00:26:28,400 I'm extremely glad for that. Um, I this 756 00:26:28,400 --> 00:26:29,840 this keeps my blood pressure at a much 757 00:26:29,840 --> 00:26:31,440 lower level, but there are still real 758 00:26:31,440 --> 00:26:33,120 harms. You know, people getting charged 759 00:26:33,120 --> 00:26:35,039 an extra $2,000 if they don't have it. 760 00:26:35,039 --> 00:26:37,039 That can be a life-threatening problem 761 00:26:37,039 --> 00:26:40,320 for some people. Um, so try and think 762 00:26:40,320 --> 00:26:41,919 through where the harms would come from 763 00:26:41,919 --> 00:26:43,919 and build armor around those points 764 00:26:43,919 --> 00:26:46,919 specifically. 765 00:26:49,760 --> 00:26:51,919 Yeah, I have a question which I can't 766 00:26:51,919 --> 00:26:53,520 work out if it's on topic or not, so I'm 767 00:26:53,520 --> 00:26:55,440 just going to ask it anyway, which is 768 00:26:55,440 --> 00:26:57,679 really around like technical debt 769 00:26:57,679 --> 00:27:00,320 because I think like with with AI tools 770 00:27:00,320 --> 00:27:02,960 or less experienced developers or anyone 771 00:27:02,960 --> 00:27:04,960 in a rush like one of the things that 772 00:27:04,960 --> 00:27:06,960 can go out the window is just like 773 00:27:06,960 --> 00:27:09,200 elegant coherent coding across a 774 00:27:09,200 --> 00:27:12,400 codebase. Can you checklist do can can 775 00:27:12,400 --> 00:27:15,200 the these concepts apply to that? They 776 00:27:15,200 --> 00:27:16,960 can if you can figure out what defines 777 00:27:16,960 --> 00:27:18,480 elegance but that is a difficult 778 00:27:18,480 --> 00:27:20,799 question. Uh some of them you can you 779 00:27:20,799 --> 00:27:23,279 know you can have like please don't name 780 00:27:23,279 --> 00:27:25,360 four of your variables x in the same 781 00:27:25,360 --> 00:27:29,120 function. Um I find mandating type 782 00:27:29,120 --> 00:27:31,200 annotations tends to help a lot because 783 00:27:31,200 --> 00:27:33,840 it forces just enough friction to make 784 00:27:33,840 --> 00:27:36,159 people think about this kind of stuff. 785 00:27:36,159 --> 00:27:38,000 Maybe as the bots get better at writing 786 00:27:38,000 --> 00:27:39,600 type annotations that will no longer be 787 00:27:39,600 --> 00:27:41,919 true. But at least for the moment, um, 788 00:27:41,919 --> 00:27:43,760 if you if you have the workflow of the 789 00:27:43,760 --> 00:27:45,840 human writes the function signature and 790 00:27:45,840 --> 00:27:48,799 then possibly a bot fills it in, that 791 00:27:48,799 --> 00:27:50,799 that slows things down enough to 792 00:27:50,799 --> 00:27:54,279 generate elegance. 793 00:27:58,080 --> 00:28:01,760 All right then, Noah. Thank you so much 794 00:28:01,760 --> 00:28:04,559 once again 795 00:28:04,559 --> 00:28:07,954 and on behalf of Pyon AU 796 00:28:07,954 --> 00:28:08,559 [sighs and gasps] 797 00:28:08,559 --> 00:28:10,720 26, here's your mark. Thank [applause] 798 00:28:10,720 --> 00:28:13,720 you.