1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:16,000 --> 00:00:20,800 Hello and welcome back to Pyon AU. And 4 00:00:20,800 --> 00:00:23,519 now I am going to introduce you to Ben 5 00:00:23,519 --> 00:00:24,800 Taylor who's going to talk about 6 00:00:24,800 --> 00:00:26,240 experiments in engineering without 7 00:00:26,240 --> 00:00:28,320 reading code. Please give him a big 8 00:00:28,320 --> 00:00:30,130 applause. 9 00:00:30,130 --> 00:00:32,150 [applause] 10 00:00:34,715 --> 00:00:36,735 [applause] 11 00:00:36,880 --> 00:00:40,000 Hello everyone. Uh when I do talks, I 12 00:00:40,000 --> 00:00:42,239 like to do audience interaction. Um and 13 00:00:42,239 --> 00:00:44,000 when you're doing audience interaction, 14 00:00:44,000 --> 00:00:46,719 it's important to warm up the audience. 15 00:00:46,719 --> 00:00:48,640 Otherwise, none of you say anything when 16 00:00:48,640 --> 00:00:51,039 I ask you things. So, I'm going to start 17 00:00:51,039 --> 00:00:53,680 with a question and you have to shout 18 00:00:53,680 --> 00:00:56,320 out the answer, which is what is the 19 00:00:56,320 --> 00:00:59,199 best domain of science? You have to 20 00:00:59,199 --> 00:01:03,760 physics, maths, biology. Okay. Uh you're 21 00:01:03,760 --> 00:01:05,119 all wrong. It's obviously computer 22 00:01:05,119 --> 00:01:07,680 science. We're here [laughter] at um 23 00:01:07,680 --> 00:01:11,760 Pyon. Uh so experiments in engineering 24 00:01:11,760 --> 00:01:14,240 without reading code. When I proposed 25 00:01:14,240 --> 00:01:16,880 this talk, I added a disclaimer that I 26 00:01:16,880 --> 00:01:18,640 didn't know what was going to happen in 27 00:01:18,640 --> 00:01:20,560 the next 6 months and that I might 28 00:01:20,560 --> 00:01:22,479 totally change my opinion by the time I 29 00:01:22,479 --> 00:01:25,040 went to give this talk. I think mostly 30 00:01:25,040 --> 00:01:26,479 my thoughts are following the same 31 00:01:26,479 --> 00:01:29,040 themes, but the capabilities of AI 32 00:01:29,040 --> 00:01:30,960 agents have grown substantially in that 33 00:01:30,960 --> 00:01:32,960 time. So I'm going to talk about how my 34 00:01:32,960 --> 00:01:35,439 thinking has evolved as well. 35 00:01:35,439 --> 00:01:37,439 Over the last 3 months, I've given four 36 00:01:37,439 --> 00:01:39,040 different versions of this talk. First 37 00:01:39,040 --> 00:01:41,520 at AI engineer in Melbourne, then at Yao 38 00:01:41,520 --> 00:01:43,920 Tech leaders, then at Melbourjs, and now 39 00:01:43,920 --> 00:01:46,479 here at Pyon AU. I've updated it each 40 00:01:46,479 --> 00:01:48,880 time so you're getting the best and most 41 00:01:48,880 --> 00:01:52,640 up-to-date version possible. 42 00:01:52,640 --> 00:01:54,320 Today I'm going to talk through three 43 00:01:54,320 --> 00:01:55,840 different experiments I've done in 44 00:01:55,840 --> 00:01:58,079 engineering without reading code. Each 45 00:01:58,079 --> 00:02:00,159 of them explores a different part of 46 00:02:00,159 --> 00:02:02,320 this topic that I've become a little 47 00:02:02,320 --> 00:02:05,600 obsessed with. I hope my obsession can 48 00:02:05,600 --> 00:02:08,560 show you some cool things you can do. 49 00:02:08,560 --> 00:02:10,720 And I also want to say these are 50 00:02:10,720 --> 00:02:13,280 experiments. I'm not trying to pitch 51 00:02:13,280 --> 00:02:16,400 this as best practice. I don't think 52 00:02:16,400 --> 00:02:18,959 that you should be closing the editor 53 00:02:18,959 --> 00:02:21,599 and not reading any code anymore. But I 54 00:02:21,599 --> 00:02:23,599 think it is interesting to play around 55 00:02:23,599 --> 00:02:25,280 with this idea and to see what's 56 00:02:25,280 --> 00:02:28,080 possible. 57 00:02:28,080 --> 00:02:30,720 To anchor these experiments in time, I 58 00:02:30,720 --> 00:02:32,560 want to show you them on a timeline of 59 00:02:32,560 --> 00:02:35,440 AI capabilities. This will help frame 60 00:02:35,440 --> 00:02:37,599 what was possible at the time, how 61 00:02:37,599 --> 00:02:39,440 things are changing month by month, and 62 00:02:39,440 --> 00:02:42,640 then where we are now. One way that I 63 00:02:42,640 --> 00:02:44,560 think about this progression is the 64 00:02:44,560 --> 00:02:47,519 length of a useful session with AI. 65 00:02:47,519 --> 00:02:49,440 Pre2025, 66 00:02:49,440 --> 00:02:52,000 it was one short output. Um, you could 67 00:02:52,000 --> 00:02:54,959 have a session that lasted seconds. Uh, 68 00:02:54,959 --> 00:02:58,560 in 2025, a few back and forths were 69 00:02:58,560 --> 00:03:00,080 possible. Um, and you could have a 70 00:03:00,080 --> 00:03:02,560 useful session that lasted minutes. Um, 71 00:03:02,560 --> 00:03:05,200 and in 2026, especially right now, I'm 72 00:03:05,200 --> 00:03:07,040 finding that I can have useful sessions 73 00:03:07,040 --> 00:03:09,840 that last hours um, autonomously without 74 00:03:09,840 --> 00:03:13,360 me monitoring it. 75 00:03:13,360 --> 00:03:16,159 So, experiment one, uh, can science 76 00:03:16,159 --> 00:03:20,400 writers make interactive web apps. 77 00:03:20,400 --> 00:03:23,280 When this one started, uh, you could 78 00:03:23,280 --> 00:03:26,640 mostly get working HTML and CSS, uh, and 79 00:03:26,640 --> 00:03:29,280 JavaScript out of Claude, uh, sessions 80 00:03:29,280 --> 00:03:32,400 would become useless pretty quickly. Uh, 81 00:03:32,400 --> 00:03:34,560 so if you spent a lot of time iterating, 82 00:03:34,560 --> 00:03:37,680 it would become garbage. 83 00:03:37,680 --> 00:03:39,519 To help you understand this experiment, 84 00:03:39,519 --> 00:03:42,000 I'll need to set up some context. 85 00:03:42,000 --> 00:03:44,720 I work at a science education company uh 86 00:03:44,720 --> 00:03:46,799 called style education. We build a 87 00:03:46,799 --> 00:03:48,640 platform for teaching and learning along 88 00:03:48,640 --> 00:03:50,799 with a core science curriculum product. 89 00:03:50,799 --> 00:03:52,400 The way to think about that is that 90 00:03:52,400 --> 00:03:54,560 teachers use style to teach science 91 00:03:54,560 --> 00:03:57,439 every day in the classroom. We provide 92 00:03:57,439 --> 00:03:59,120 lesson plans, presentations, 93 00:03:59,120 --> 00:04:01,200 simulations, worksheets, videos, labs, 94 00:04:01,200 --> 00:04:02,959 revision material, and everything else 95 00:04:02,959 --> 00:04:05,439 you might need to teach science. Some 96 00:04:05,439 --> 00:04:07,519 lessons are live classroom experience 97 00:04:07,519 --> 00:04:10,400 with laptops open. Some are on paper 98 00:04:10,400 --> 00:04:12,959 with laptops closed and labs are run 99 00:04:12,959 --> 00:04:16,762 without laptops, especially wet labs. 100 00:04:16,762 --> 00:04:18,160 [snorts] When we're designing our 101 00:04:18,160 --> 00:04:20,000 lessons, we think about the best way to 102 00:04:20,000 --> 00:04:22,479 explain the concept. And sometimes we 103 00:04:22,479 --> 00:04:24,400 decide the best way will be with 104 00:04:24,400 --> 00:04:26,720 something we call an interactive. These 105 00:04:26,720 --> 00:04:29,199 are small interactive websites that help 106 00:04:29,199 --> 00:04:30,960 students play with concepts in a way 107 00:04:30,960 --> 00:04:33,919 that responds live. Here's an example 108 00:04:33,919 --> 00:04:36,000 interactive that demonstrates projectile 109 00:04:36,000 --> 00:04:37,759 motion. Uh, now I'm going to need 110 00:04:37,759 --> 00:04:40,000 audience participation. What release 111 00:04:40,000 --> 00:04:42,720 angle should we use? 112 00:04:42,720 --> 00:04:45,919 45. Everyone's going for optimal. Um, 113 00:04:45,919 --> 00:04:47,680 and what release speed should we go 114 00:04:47,680 --> 00:04:51,840 with? One. Excellent. [laughter] 115 00:04:51,840 --> 00:04:55,280 All right. Down to one. And let's throw 116 00:04:55,280 --> 00:04:58,738 it. Wow, [laughter] 117 00:04:58,960 --> 00:05:02,400 that was great. Fantastic. Um, so this 118 00:05:02,400 --> 00:05:05,759 is the kind of thing um that we make u 119 00:05:05,759 --> 00:05:08,960 when we're talking about interactives. 120 00:05:08,960 --> 00:05:10,240 We've been building building 121 00:05:10,240 --> 00:05:12,479 interactives and simulations like this 122 00:05:12,479 --> 00:05:15,039 for more than 10 years. And we've made 123 00:05:15,039 --> 00:05:17,360 some really great ones. However, they 124 00:05:17,360 --> 00:05:20,560 are expensive to build. They require a 125 00:05:20,560 --> 00:05:23,600 science writer, an illustrator, and at 126 00:05:23,600 --> 00:05:26,160 least one engineer. There's iteration 127 00:05:26,160 --> 00:05:27,919 backwards and forwards as they try out 128 00:05:27,919 --> 00:05:29,680 different ideas to figure out what is 129 00:05:29,680 --> 00:05:32,479 possible, what is pedagogically sound, 130 00:05:32,479 --> 00:05:35,919 what is intuitive, and what is fun. 131 00:05:35,919 --> 00:05:38,960 Here's an interactive we made pre2025 132 00:05:38,960 --> 00:05:40,880 that demonstrates natural selection with 133 00:05:40,880 --> 00:05:43,120 the peppered moth. It took a team of 134 00:05:43,120 --> 00:05:45,520 three people a few weeks to build. Um, 135 00:05:45,520 --> 00:05:47,360 now this has a loud chirping noise, so 136 00:05:47,360 --> 00:05:49,440 depending on the volume, um, it's going 137 00:05:49,440 --> 00:05:54,160 to Okay, we're good. Um, so I'm I'm 138 00:05:54,160 --> 00:05:57,280 clicking on these these black moths you 139 00:05:57,280 --> 00:06:01,199 can see on the screen. And uh I found 140 00:06:01,199 --> 00:06:03,520 six of them. Um, and apparently Oh, 141 00:06:03,520 --> 00:06:06,000 there were white moths as well. Um, 142 00:06:06,000 --> 00:06:07,440 let's go have a look at the polluted 143 00:06:07,440 --> 00:06:09,440 environment. Wow, the white moths are so 144 00:06:09,440 --> 00:06:11,919 much easier to spot. Almost like there's 145 00:06:11,919 --> 00:06:13,840 some sort of natural selection effect 146 00:06:13,840 --> 00:06:15,840 going on. 147 00:06:15,840 --> 00:06:17,280 Wait, are there black moths on the 148 00:06:17,280 --> 00:06:19,440 screen as well? 149 00:06:19,440 --> 00:06:22,400 We didn't find any. But uh this whole 150 00:06:22,400 --> 00:06:24,960 interactive is to demonstrate that in a 151 00:06:24,960 --> 00:06:27,440 dark environment, black moths have an 152 00:06:27,440 --> 00:06:29,280 advantage and in a light environment, 153 00:06:29,280 --> 00:06:32,720 white moths have an advantage. 154 00:06:32,720 --> 00:06:35,440 In 2024, we built two of these 155 00:06:35,440 --> 00:06:37,280 interactives. 156 00:06:37,280 --> 00:06:40,639 We had appetite for way more, but on the 157 00:06:40,639 --> 00:06:42,560 engineering side, we had to trade it off 158 00:06:42,560 --> 00:06:44,560 against platform features. Features 159 00:06:44,560 --> 00:06:46,639 usually win since they benefit every 160 00:06:46,639 --> 00:06:49,360 lesson, not just one. So instead, our 161 00:06:49,360 --> 00:06:51,360 writing team would make a graphic or at 162 00:06:51,360 --> 00:06:54,960 most an animation. But then in 2025, we 163 00:06:54,960 --> 00:06:57,440 tried out vibe coding. 164 00:06:57,440 --> 00:06:59,840 This interactive was vibe coded back in 165 00:06:59,840 --> 00:07:02,960 2025 before the models got really good. 166 00:07:02,960 --> 00:07:05,039 This took a writer working with an 167 00:07:05,039 --> 00:07:07,520 illustrator a few days to create. I can 168 00:07:07,520 --> 00:07:09,199 guarantee you this would have been a few 169 00:07:09,199 --> 00:07:11,680 weeks of engineering work in 2024. Um, 170 00:07:11,680 --> 00:07:13,680 I'm just going to click this is a little 171 00:07:13,680 --> 00:07:16,560 demo that I built into it. So it's a 172 00:07:16,560 --> 00:07:21,840 guitar. you can adjust um the um cappo 173 00:07:21,840 --> 00:07:25,199 and then you end up yes uh you have the 174 00:07:25,199 --> 00:07:27,370 capability to play songs on it as well. 175 00:07:27,370 --> 00:07:28,000 [laughter] 176 00:07:28,000 --> 00:07:30,800 Uh and you could use this to demonstrate 177 00:07:30,800 --> 00:07:34,400 uh how waves change uh pitch depending 178 00:07:34,400 --> 00:07:38,840 on how long the length of a string is. 179 00:07:39,919 --> 00:07:42,240 Where previously writers had to iterate 180 00:07:42,240 --> 00:07:43,919 through an engineer to understand what 181 00:07:43,919 --> 00:07:46,000 they could do, now they could try it out 182 00:07:46,000 --> 00:07:47,759 directly. They could come up with a 183 00:07:47,759 --> 00:07:50,400 concept and then iterate focusing on fun 184 00:07:50,400 --> 00:07:53,199 scientific accuracy and pedagogy. We'd 185 00:07:53,199 --> 00:07:55,680 removed the barrier of iterating through 186 00:07:55,680 --> 00:07:58,720 another person and massively sped up the 187 00:07:58,720 --> 00:08:02,560 process. In 2024, we built two of these. 188 00:08:02,560 --> 00:08:06,080 In 2025, we built almost 50, which is a 189 00:08:06,080 --> 00:08:09,120 massive increase. 190 00:08:09,120 --> 00:08:11,120 Something we've noticed over the last 2 191 00:08:11,120 --> 00:08:12,879 years building with AI is that 192 00:08:12,879 --> 00:08:15,280 bottlenecks move. AI allows you to move 193 00:08:15,280 --> 00:08:17,599 so fast in one place that somewhere 194 00:08:17,599 --> 00:08:20,319 surprising becomes a bottleneck. 195 00:08:20,319 --> 00:08:22,000 Once our science writers were unlocked 196 00:08:22,000 --> 00:08:23,759 and could create any interactives they 197 00:08:23,759 --> 00:08:25,680 could dream of, we suddenly had a 198 00:08:25,680 --> 00:08:28,000 bottleneck in review and deployment. 199 00:08:28,000 --> 00:08:29,440 Even though the writers could build 200 00:08:29,440 --> 00:08:31,599 something that worked for them, they 201 00:08:31,599 --> 00:08:33,760 couldn't deploy their work. They didn't 202 00:08:33,760 --> 00:08:35,760 stick to design systems. They didn't 203 00:08:35,760 --> 00:08:37,760 know how to do accessibility testing. 204 00:08:37,760 --> 00:08:39,039 They couldn't extract out 205 00:08:39,039 --> 00:08:41,200 internationalization strings. And 206 00:08:41,200 --> 00:08:43,039 generally they weren't reviewing for 207 00:08:43,039 --> 00:08:45,120 engineering quality. So we had to do the 208 00:08:45,120 --> 00:08:48,560 last mile in engineering. 209 00:08:48,560 --> 00:08:51,200 We didn't like this. Finishing off 210 00:08:51,200 --> 00:08:53,600 someone else's half finished work 211 00:08:53,600 --> 00:08:55,839 doesn't feel good, even when you give it 212 00:08:55,839 --> 00:08:59,279 a fancy name like productionizing. Plus, 213 00:08:59,279 --> 00:09:02,320 it was slowing us down. We decided that 214 00:09:02,320 --> 00:09:04,160 we needed to get engineering out of the 215 00:09:04,160 --> 00:09:06,640 loop. If writers could deploy their work 216 00:09:06,640 --> 00:09:08,560 straight to production, then we could 217 00:09:08,560 --> 00:09:10,640 remove a step in the process. speed 218 00:09:10,640 --> 00:09:12,720 everyone up and hopefully ship more 219 00:09:12,720 --> 00:09:15,360 great work. Of course, we can't just 220 00:09:15,360 --> 00:09:17,519 remove engineering. There was actual 221 00:09:17,519 --> 00:09:20,560 work being done there. So, instead, we 222 00:09:20,560 --> 00:09:22,959 needed to do engineering work so that we 223 00:09:22,959 --> 00:09:26,080 didn't have to do engineering work. 224 00:09:26,080 --> 00:09:28,160 So, now we get to experiment two, 225 00:09:28,160 --> 00:09:30,399 building a system that enforces 226 00:09:30,399 --> 00:09:32,800 engineering standards. 227 00:09:32,800 --> 00:09:34,240 Off the back of our experience with 228 00:09:34,240 --> 00:09:36,320 experiment one, we wanted to see if we 229 00:09:36,320 --> 00:09:38,560 could build a system that enforced the 230 00:09:38,560 --> 00:09:40,080 engineering standards we were practicing 231 00:09:40,080 --> 00:09:42,640 ourselves. This would allow us to move 232 00:09:42,640 --> 00:09:44,480 remove engineering from the loop and let 233 00:09:44,480 --> 00:09:46,240 riders deploy their work straight to 234 00:09:46,240 --> 00:09:48,399 production. 235 00:09:48,399 --> 00:09:51,760 So this we did very much after AI got 236 00:09:51,760 --> 00:09:54,080 good um and we could start offloading 237 00:09:54,080 --> 00:09:57,360 significant tasks to AI. So this started 238 00:09:57,360 --> 00:10:01,360 um at the beginning of this year. 239 00:10:01,360 --> 00:10:03,839 So, as a bit of background to this, I've 240 00:10:03,839 --> 00:10:05,519 started to make a distinction when I 241 00:10:05,519 --> 00:10:07,760 talk about AI. And this distinction is 242 00:10:07,760 --> 00:10:10,080 between engineering and coding. I'm 243 00:10:10,080 --> 00:10:12,080 really good at coding. Uh, I've been 244 00:10:12,080 --> 00:10:14,399 doing it for more than 20 years. I'm 245 00:10:14,399 --> 00:10:15,920 comfortable in all common programming 246 00:10:15,920 --> 00:10:18,000 languages, including Python. It was my 247 00:10:18,000 --> 00:10:20,000 first love of a programming language. 248 00:10:20,000 --> 00:10:23,120 I'm fast. I comprehend quickly. I've 249 00:10:23,120 --> 00:10:26,800 made a successful career out of coding. 250 00:10:26,800 --> 00:10:29,680 Claude is way better than me at coding. 251 00:10:29,680 --> 00:10:31,360 Not only does it have deeper knowledge 252 00:10:31,360 --> 00:10:34,240 of every programming language, it's also 253 00:10:34,240 --> 00:10:36,720 faster. It knows more libraries. It can 254 00:10:36,720 --> 00:10:39,680 spot bugs that I would miss. And it can 255 00:10:39,680 --> 00:10:42,480 comprehend large code bases in seconds 256 00:10:42,480 --> 00:10:45,440 without onboarding, 257 00:10:45,440 --> 00:10:49,767 but it's not better at me yet. 258 00:10:49,767 --> 00:10:51,600 [laughter] 259 00:10:51,600 --> 00:10:53,839 So, we have another audience activity. 260 00:10:53,839 --> 00:10:55,519 Um, and I want you to sort of think 261 00:10:55,519 --> 00:10:58,720 about is this concept engineering or 262 00:10:58,720 --> 00:11:01,360 coding? Yell it out. 263 00:11:01,360 --> 00:11:03,200 Coding definitely feels like coding. 264 00:11:03,200 --> 00:11:06,320 Async controversial 265 00:11:06,320 --> 00:11:08,399 coding. Okay, we're hearing more coding. 266 00:11:08,399 --> 00:11:10,880 Duct typing. 267 00:11:10,880 --> 00:11:14,160 Coding. Okay. Decorators. Wow, this 268 00:11:14,160 --> 00:11:16,399 random ordering happened to be very bad 269 00:11:16,399 --> 00:11:19,680 for this activity. Uh, coding. Coding. 270 00:11:19,680 --> 00:11:22,800 Okay. There are Oh, okay. Oh my gosh. 271 00:11:22,800 --> 00:11:25,120 Peppe, engineering. Okay. Um, I'm going 272 00:11:25,120 --> 00:11:27,757 to say that's coding. Um, accessibility. 273 00:11:27,757 --> 00:11:28,640 [laughter] 274 00:11:28,640 --> 00:11:32,160 Engineering. Making it work. 275 00:11:32,160 --> 00:11:34,480 Security. [laughter] 276 00:11:34,480 --> 00:11:36,480 Context managers. 277 00:11:36,480 --> 00:11:39,200 Coding. I This one actually I think is 278 00:11:39,200 --> 00:11:41,440 like controversial. Anyway, um, error 279 00:11:41,440 --> 00:11:43,279 handling. 280 00:11:43,279 --> 00:11:46,399 Coding. Or is it engineering? [laughter] 281 00:11:46,399 --> 00:11:47,600 I'm going to put it in coding because I 282 00:11:47,600 --> 00:11:50,720 heard coding first. Time complexity. 283 00:11:50,720 --> 00:11:52,160 Definitely engineering. Definitely 284 00:11:52,160 --> 00:11:55,279 engineering type hints 285 00:11:55,279 --> 00:11:59,040 coding maybe uh tradeoffs 286 00:11:59,040 --> 00:12:02,560 absolutely engineering okay so I I hope 287 00:12:02,560 --> 00:12:04,079 this has helped you think about this and 288 00:12:04,079 --> 00:12:05,360 there's definitely some in there that I 289 00:12:05,360 --> 00:12:08,240 would say bridge the gap right there's 290 00:12:08,240 --> 00:12:11,040 some details like types where if you 291 00:12:11,040 --> 00:12:13,440 have a fully type- safe code that result 292 00:12:13,440 --> 00:12:15,440 code base that results in an engineering 293 00:12:15,440 --> 00:12:19,040 practice that is higher quality uh 294 00:12:19,040 --> 00:12:21,200 so for my entire career coding and 295 00:12:21,200 --> 00:12:23,839 engineer ing have really been conflated. 296 00:12:23,839 --> 00:12:25,839 You do the engineering by doing the 297 00:12:25,839 --> 00:12:28,480 coding. You think through the problem by 298 00:12:28,480 --> 00:12:30,639 writing out the implementation. Sure, 299 00:12:30,639 --> 00:12:33,360 you did planning beforehand, but so many 300 00:12:33,360 --> 00:12:35,360 issues would only come up once you'd 301 00:12:35,360 --> 00:12:37,360 started implementing it. Loading the 302 00:12:37,360 --> 00:12:39,440 model of the code into your head would 303 00:12:39,440 --> 00:12:42,079 allow you to deeply understand the gaps. 304 00:12:42,079 --> 00:12:44,800 You'd spot the potential security issue. 305 00:12:44,800 --> 00:12:47,200 You the user experience issue. you'd 306 00:12:47,200 --> 00:12:49,279 understand whether an async process 307 00:12:49,279 --> 00:12:51,120 would be consistent because you'd 308 00:12:51,120 --> 00:12:53,040 imagine the different orders that things 309 00:12:53,040 --> 00:12:56,000 would happen. 310 00:12:56,000 --> 00:12:58,800 When we're using AI agents to code, 311 00:12:58,800 --> 00:13:02,480 we're no longer doing this work, but we 312 00:13:02,480 --> 00:13:06,000 still need those same quality standards. 313 00:13:06,000 --> 00:13:09,200 So, how do we enforce them? 314 00:13:09,200 --> 00:13:10,639 To try and reduce the amount of 315 00:13:10,639 --> 00:13:12,639 engineering work we've be we're doing, 316 00:13:12,639 --> 00:13:14,560 we've been building tools that do that 317 00:13:14,560 --> 00:13:17,120 work for us. 318 00:13:17,120 --> 00:13:19,680 So an example is we built a coding agent 319 00:13:19,680 --> 00:13:22,399 for our writers. In that coding agent, 320 00:13:22,399 --> 00:13:25,120 we were able to build in accessibility 321 00:13:25,120 --> 00:13:28,079 and design standards. 322 00:13:28,079 --> 00:13:29,680 This is the first prototype of our 323 00:13:29,680 --> 00:13:32,240 interactive builder. We built into it an 324 00:13:32,240 --> 00:13:35,519 axe checker, an i8N checker, a zoom 325 00:13:35,519 --> 00:13:37,920 tool, a resize tool. These things are 326 00:13:37,920 --> 00:13:41,600 really important for um 400% zoom. Um, 327 00:13:41,600 --> 00:13:43,760 and 328 00:13:43,760 --> 00:13:45,440 the result was that we got better 329 00:13:45,440 --> 00:13:47,440 quality interactives. 330 00:13:47,440 --> 00:13:49,040 And it turns out that also building 331 00:13:49,040 --> 00:13:51,120 custom coding agents is actually super 332 00:13:51,120 --> 00:13:54,560 easy. Um, and this itself was vibe coded 333 00:13:54,560 --> 00:13:56,800 as an electron app um to make it really 334 00:13:56,800 --> 00:13:59,440 simple to distribute. 335 00:13:59,440 --> 00:14:01,279 We also introduced a pipeline that 336 00:14:01,279 --> 00:14:03,680 ingests interactives. 337 00:14:03,680 --> 00:14:05,760 We take the interactive that someone has 338 00:14:05,760 --> 00:14:09,760 vibe coded um and then we uh rewrite it 339 00:14:09,760 --> 00:14:12,240 to meet our specifications. This helps 340 00:14:12,240 --> 00:14:14,240 to make sure that everything is written 341 00:14:14,240 --> 00:14:16,560 in the exact same structure so that we 342 00:14:16,560 --> 00:14:18,639 can then have standardized uh operations 343 00:14:18,639 --> 00:14:21,279 on top of that structure. And it also 344 00:14:21,279 --> 00:14:23,199 allowed us to migrate all our old 345 00:14:23,199 --> 00:14:25,199 interactives, the backlog that we've had 346 00:14:25,199 --> 00:14:27,360 for the last 10 years over to the same 347 00:14:27,360 --> 00:14:30,320 shape and structure. 348 00:14:30,320 --> 00:14:33,199 We sandbox the interactives via headers 349 00:14:33,199 --> 00:14:36,560 and in an iframe. This is obvious. Um we 350 00:14:36,560 --> 00:14:38,240 were already doing this um but it does 351 00:14:38,240 --> 00:14:39,839 help us feel safe from a security point 352 00:14:39,839 --> 00:14:41,680 of view. 353 00:14:41,680 --> 00:14:43,440 And we built these things that we call 354 00:14:43,440 --> 00:14:46,000 critics which are systems that provide 355 00:14:46,000 --> 00:14:49,360 criticisms of PRs. And we like to think 356 00:14:49,360 --> 00:14:51,199 about that as the whole change not just 357 00:14:51,199 --> 00:14:53,279 the code. They deal with different kinds 358 00:14:53,279 --> 00:14:55,519 of failures. We've seen most of these 359 00:14:55,519 --> 00:14:58,160 are prompts, but they don't have to be. 360 00:14:58,160 --> 00:15:00,320 Our accessibility critic runs a manual 361 00:15:00,320 --> 00:15:02,639 audit based on Microsoft's accessibility 362 00:15:02,639 --> 00:15:05,519 insights tool. 363 00:15:05,519 --> 00:15:07,279 There's much more that we want to do, 364 00:15:07,279 --> 00:15:09,120 and we're not happy with where it is at 365 00:15:09,120 --> 00:15:11,600 yet, but the early results are are good. 366 00:15:11,600 --> 00:15:14,000 In 2025, we shipped 50 interactives 367 00:15:14,000 --> 00:15:16,959 total. In 2026, we've shipped more than 368 00:15:16,959 --> 00:15:19,279 50 interactives so far. Uh, and we 369 00:15:19,279 --> 00:15:22,160 expect to do 100. Our engineering work 370 00:15:22,160 --> 00:15:24,639 on individual interactives has shrunk 371 00:15:24,639 --> 00:15:28,560 from days to hours to minutes. 372 00:15:28,560 --> 00:15:30,800 Instead of making sure each interactive 373 00:15:30,800 --> 00:15:32,880 was highquality, we're working on 374 00:15:32,880 --> 00:15:35,120 building a system that makes sure all of 375 00:15:35,120 --> 00:15:37,360 our interactives are high quality. I 376 00:15:37,360 --> 00:15:39,360 used to say that if I implemented the 377 00:15:39,360 --> 00:15:42,320 same thing three times, I'd failed as an 378 00:15:42,320 --> 00:15:45,839 engineer. Now, I feel like if I give the 379 00:15:45,839 --> 00:15:48,880 same feedback three times, I failed as 380 00:15:48,880 --> 00:15:51,839 an engineer. Whatever rule I'm applying 381 00:15:51,839 --> 00:15:56,199 should be encoded into the system. 382 00:15:56,560 --> 00:15:59,759 So getting on to experiment three. Can I 383 00:15:59,759 --> 00:16:01,600 build highquality software without 384 00:16:01,600 --> 00:16:03,600 reading code? And this is really the 385 00:16:03,600 --> 00:16:06,240 title of this talk. So at the start of 386 00:16:06,240 --> 00:16:08,560 this year, we did an agent building 387 00:16:08,560 --> 00:16:11,279 hackathon with Jeff Huntley at work. The 388 00:16:11,279 --> 00:16:13,120 message he was pushing was that we 389 00:16:13,120 --> 00:16:14,720 should be building our own tools and 390 00:16:14,720 --> 00:16:18,079 workflows. So I decided that I wanted to 391 00:16:18,079 --> 00:16:20,320 see if I could build high-quality 392 00:16:20,320 --> 00:16:22,399 software without reading code by 393 00:16:22,399 --> 00:16:24,160 building an agent that had the standards 394 00:16:24,160 --> 00:16:27,440 that I expected. At the time this idea 395 00:16:27,440 --> 00:16:32,000 was controversial uh to me as well. 396 00:16:32,000 --> 00:16:33,519 So I've been working on and improving 397 00:16:33,519 --> 00:16:35,920 this system for the last 8 months and 398 00:16:35,920 --> 00:16:38,399 over the last three it has been really 399 00:16:38,399 --> 00:16:40,480 flying. Um, and particularly that's 400 00:16:40,480 --> 00:16:42,399 because of the release of uh new models 401 00:16:42,399 --> 00:16:44,720 with new capabilities. 402 00:16:44,720 --> 00:16:47,519 But before we jump into that, I want to 403 00:16:47,519 --> 00:16:50,880 take a step back and talk about why I 404 00:16:50,880 --> 00:16:53,920 think this idea is so interesting. Um, 405 00:16:53,920 --> 00:16:56,160 and if you're Lee and Beno's talk, I 406 00:16:56,160 --> 00:16:59,600 think there's a lot of overlap 407 00:16:59,600 --> 00:17:01,839 and it's because of this thing called 408 00:17:01,839 --> 00:17:04,319 vigilance decrement. 409 00:17:04,319 --> 00:17:06,640 I think that when we're using tools like 410 00:17:06,640 --> 00:17:09,600 clawed code, we're no longer engaging 411 00:17:09,600 --> 00:17:12,240 directly in coding. We've moved from 412 00:17:12,240 --> 00:17:15,280 active participation coding it yourself 413 00:17:15,280 --> 00:17:18,480 to monitoring the system. Research shows 414 00:17:18,480 --> 00:17:20,720 that humans monitoring a system 415 00:17:20,720 --> 00:17:23,600 experience more vigilance decrement than 416 00:17:23,600 --> 00:17:26,079 active participants. That means we'll 417 00:17:26,079 --> 00:17:28,720 start missing things. The details 418 00:17:28,720 --> 00:17:30,880 matter. Does it make an incorrect 419 00:17:30,880 --> 00:17:32,799 assumption about the way an external 420 00:17:32,799 --> 00:17:35,200 system operates? Does it propagate 421 00:17:35,200 --> 00:17:37,760 errors in the right way? Is it using a 422 00:17:37,760 --> 00:17:40,720 pattern copied from some outdated code? 423 00:17:40,720 --> 00:17:43,760 Does the combination of new behavior and 424 00:17:43,760 --> 00:17:46,400 an existing system create a security 425 00:17:46,400 --> 00:17:49,440 threat without manually writing code? 426 00:17:49,440 --> 00:17:52,080 You miss these details. The industry 427 00:17:52,080 --> 00:17:55,360 frames this as human in the loop, but 428 00:17:55,360 --> 00:17:57,440 you're not in the loop. You're 429 00:17:57,440 --> 00:18:00,400 monitoring as the agent does the work 430 00:18:00,400 --> 00:18:02,320 and that means that you're losing 431 00:18:02,320 --> 00:18:04,960 situational awareness, system state 432 00:18:04,960 --> 00:18:07,520 understanding, and the ability to make 433 00:18:07,520 --> 00:18:10,799 an effective intervention. 434 00:18:10,799 --> 00:18:13,120 You might think I'm advocating for less 435 00:18:13,120 --> 00:18:16,240 automation. Um, but I'm not. We've 436 00:18:16,240 --> 00:18:18,960 opened Pandora's box. Uh, and I don't 437 00:18:18,960 --> 00:18:21,360 think we can close it. The solution is 438 00:18:21,360 --> 00:18:24,000 more AI. 439 00:18:24,000 --> 00:18:27,760 Compare Tesla self-driving to Whimo. In 440 00:18:27,760 --> 00:18:30,880 a Tesla, you have to keep your hands on 441 00:18:30,880 --> 00:18:32,559 the wheel while it's in full 442 00:18:32,559 --> 00:18:35,039 self-driving. It will yell at you if you 443 00:18:35,039 --> 00:18:38,080 take your hands off the wheel. Way cars 444 00:18:38,080 --> 00:18:41,520 have no driver. Teslas are partially 445 00:18:41,520 --> 00:18:43,120 automated. 446 00:18:43,120 --> 00:18:45,440 When push comes to shove, you are the 447 00:18:45,440 --> 00:18:48,640 backup for the automated system. In most 448 00:18:48,640 --> 00:18:51,039 cases, your workload decreases. You're 449 00:18:51,039 --> 00:18:52,640 sitting, you're chilling on the highway, 450 00:18:52,640 --> 00:18:55,039 everything's going fast, it's great. You 451 00:18:55,039 --> 00:18:57,039 don't have to do anything. But in 452 00:18:57,039 --> 00:18:59,840 extremely rare and critical moments, 453 00:18:59,840 --> 00:19:02,720 your workload increases sharply and you 454 00:19:02,720 --> 00:19:04,720 are unprepared. 455 00:19:04,720 --> 00:19:06,160 Imagine holding on to the steering 456 00:19:06,160 --> 00:19:07,520 wheel, not paying attention, and 457 00:19:07,520 --> 00:19:09,200 suddenly you're in the middle of a car 458 00:19:09,200 --> 00:19:10,640 crash. How are you going to make a 459 00:19:10,640 --> 00:19:13,120 decision in that moment about what is 460 00:19:13,120 --> 00:19:15,679 the right thing to do? 461 00:19:15,679 --> 00:19:17,840 In a Whimo, the engineering work has 462 00:19:17,840 --> 00:19:20,559 been done to take the human entirely out 463 00:19:20,559 --> 00:19:23,679 of the system. The human isn't a backup 464 00:19:23,679 --> 00:19:25,760 and so the automation has to be able to 465 00:19:25,760 --> 00:19:29,200 operate safely. In uh Benu and Lee's 466 00:19:29,200 --> 00:19:30,640 talk, they talked about the human 467 00:19:30,640 --> 00:19:33,200 crumple zone. In a partially automated 468 00:19:33,200 --> 00:19:35,440 system, the human is the crumple zone. 469 00:19:35,440 --> 00:19:37,520 That's where the failures happen on on 470 00:19:37,520 --> 00:19:40,640 the human part. 471 00:19:40,640 --> 00:19:42,480 Applying this to software engineering 472 00:19:42,480 --> 00:19:45,440 reveals a clear parallel. The solution 473 00:19:45,440 --> 00:19:48,720 isn't more partial automation. We need 474 00:19:48,720 --> 00:19:52,000 full Whimo style automation where robots 475 00:19:52,000 --> 00:19:54,480 write, review, and merge code 476 00:19:54,480 --> 00:19:58,720 automatically with no human involvement. 477 00:19:58,720 --> 00:20:00,799 So to play around with this idea, I 478 00:20:00,799 --> 00:20:03,360 built my own software factory. 479 00:20:03,360 --> 00:20:05,120 You may have heard this called a dark 480 00:20:05,120 --> 00:20:07,600 factory or heard of Steve Yaggi's Gas 481 00:20:07,600 --> 00:20:10,559 Town. The basic concept is that you put 482 00:20:10,559 --> 00:20:13,120 your ideas in and highquality working 483 00:20:13,120 --> 00:20:16,000 software comes out. In the middle, stuff 484 00:20:16,000 --> 00:20:18,320 happens, but that stuff doesn't involve 485 00:20:18,320 --> 00:20:20,799 you reading code or looking at diffs. 486 00:20:20,799 --> 00:20:23,280 The stuff is agents doing work to break 487 00:20:23,280 --> 00:20:25,280 down the problem, implement it, and 488 00:20:25,280 --> 00:20:28,559 enforce the quality of the output. 489 00:20:28,559 --> 00:20:31,600 My software at its core is just a pile 490 00:20:31,600 --> 00:20:34,034 of markdown documents. 491 00:20:34,034 --> 00:20:34,559 [snorts] 492 00:20:34,559 --> 00:20:36,400 The simplest way to think about what it 493 00:20:36,400 --> 00:20:39,200 does is in this order. It opens an 494 00:20:39,200 --> 00:20:42,400 issue, submits a PR, reviews the PR, 495 00:20:42,400 --> 00:20:46,000 merges the PR, and closes the issue. 496 00:20:46,000 --> 00:20:47,840 But the actual way it works is in 497 00:20:47,840 --> 00:20:50,559 reverse. If there is work that fixes an 498 00:20:50,559 --> 00:20:52,880 issue, close it. If there is an approved 499 00:20:52,880 --> 00:20:55,600 PR, merge it. If there is an unapproved 500 00:20:55,600 --> 00:20:58,240 PR, review it. If there are review 501 00:20:58,240 --> 00:21:00,720 comments, fix them. If there is an open 502 00:21:00,720 --> 00:21:02,960 issue, implement it. If there are no 503 00:21:02,960 --> 00:21:07,400 open issues, open an issue. 504 00:21:07,840 --> 00:21:10,559 When I started, it was a full Python 505 00:21:10,559 --> 00:21:13,039 server, JavaScript client with a custom 506 00:21:13,039 --> 00:21:15,600 agent and a harness that managed jobs. 507 00:21:15,600 --> 00:21:18,400 Over time, I've found that written 508 00:21:18,400 --> 00:21:21,440 guidance on top of a general capability 509 00:21:21,440 --> 00:21:24,080 works better than a structured harness. 510 00:21:24,080 --> 00:21:26,320 It still has some scripts to help with 511 00:21:26,320 --> 00:21:28,960 determinism, but it's mostly cron jobs 512 00:21:28,960 --> 00:21:32,159 and markdown documents. 513 00:21:32,159 --> 00:21:33,600 What actually makes this work 514 00:21:33,600 --> 00:21:35,360 successfully is the same sort of work 515 00:21:35,360 --> 00:21:36,640 we've been doing in engineering 516 00:21:36,640 --> 00:21:39,039 leadership for years. It's creating 517 00:21:39,039 --> 00:21:41,039 working norms and standards for what 518 00:21:41,039 --> 00:21:43,120 good looks like and then enforcing those 519 00:21:43,120 --> 00:21:46,240 standards automatically. 520 00:21:46,240 --> 00:21:50,679 I have a side project called Hyper. 521 00:21:51,440 --> 00:21:54,159 Ironically, it's a code review app. Yes, 522 00:21:54,159 --> 00:21:56,320 I still do code review, though I'd like 523 00:21:56,320 --> 00:21:58,400 to think about it more as change review. 524 00:21:58,400 --> 00:22:01,120 Now in that repository, I've created 525 00:22:01,120 --> 00:22:02,880 documentation that describes the 526 00:22:02,880 --> 00:22:06,000 standards I have. Here are some examples 527 00:22:06,000 --> 00:22:08,159 of my standards. All code must be 528 00:22:08,159 --> 00:22:10,480 strictly typed. All errors must be 529 00:22:10,480 --> 00:22:12,559 handled. Display a useful error screen 530 00:22:12,559 --> 00:22:15,600 or hard crash. No silent errors. Every 531 00:22:15,600 --> 00:22:17,120 new feature must have an end-to-end 532 00:22:17,120 --> 00:22:19,440 test. All screens must be WIKAG 533 00:22:19,440 --> 00:22:21,200 compliant. Every meaningful 534 00:22:21,200 --> 00:22:23,440 architectural decision is recorded. When 535 00:22:23,440 --> 00:22:25,120 a decision is changed, the old decision 536 00:22:25,120 --> 00:22:27,679 is marked as superseded. No dangling 537 00:22:27,679 --> 00:22:30,480 follow-ups. Every issue with this PR is 538 00:22:30,480 --> 00:22:33,280 resolved in this PR or is documented. 539 00:22:33,280 --> 00:22:36,159 Small loose ends are completed. Now, 540 00:22:36,159 --> 00:22:38,159 manually verify the work before claiming 541 00:22:38,159 --> 00:22:40,320 it's done. and provide evidence of your 542 00:22:40,320 --> 00:22:42,799 verification to a reviewer. 543 00:22:42,799 --> 00:22:44,880 That last one is great. It means that 544 00:22:44,880 --> 00:22:47,360 every PR comes with screenshots, videos, 545 00:22:47,360 --> 00:22:49,760 or a trace. That makes it really easy to 546 00:22:49,760 --> 00:22:52,640 check that it was done correctly. 547 00:22:52,640 --> 00:22:55,440 This is paired with review guidance like 548 00:22:55,440 --> 00:22:57,679 must meet our existing standards. This 549 00:22:57,679 --> 00:22:59,600 encompasses explicitly checking for all 550 00:22:59,600 --> 00:23:02,000 of the things that I said before. No 551 00:23:02,000 --> 00:23:04,000 excuses. A documented reason for 552 00:23:04,000 --> 00:23:05,520 breaching our standards is not a 553 00:23:05,520 --> 00:23:07,760 substitution for the standard. This one 554 00:23:07,760 --> 00:23:09,440 is kind of important when working with 555 00:23:09,440 --> 00:23:11,520 AI. They love to prompt inject each 556 00:23:11,520 --> 00:23:13,120 other. 557 00:23:13,120 --> 00:23:14,960 CI checks must be green. This one seems 558 00:23:14,960 --> 00:23:17,960 obvious. 559 00:23:26,799 --> 00:23:28,799 better than the work it was doing ago on 560 00:23:28,799 --> 00:23:31,440 Opus 4. 561 00:23:31,440 --> 00:23:33,919 I released Hyper Review to test and I 562 00:23:33,919 --> 00:23:35,840 got a report that the GitHub signin flow 563 00:23:35,840 --> 00:23:38,559 had crashed and that in general the 564 00:23:38,559 --> 00:23:41,440 sign-in was difficult to use. I filed 565 00:23:41,440 --> 00:23:43,520 this issue including the report from the 566 00:23:43,520 --> 00:23:45,840 tester. GitHub signin is mildly 567 00:23:45,840 --> 00:23:49,200 difficult um and then uh a crash. The 568 00:23:49,200 --> 00:23:50,960 tester included this diagnostic log 569 00:23:50,960 --> 00:23:53,679 which demonstrated the failure. And then 570 00:23:53,679 --> 00:23:55,440 after I posted the issue, I added two 571 00:23:55,440 --> 00:23:57,360 comments. I wanted a notification when 572 00:23:57,360 --> 00:23:58,960 it was resolved so I could tell that 573 00:23:58,960 --> 00:24:01,440 person. And I suggested an updated user 574 00:24:01,440 --> 00:24:04,559 experience in plain text. 575 00:24:04,559 --> 00:24:06,640 The agent triaged the report, reproduced 576 00:24:06,640 --> 00:24:09,520 the failure. Then it submitted a PR to 577 00:24:09,520 --> 00:24:11,760 fix the ticket. It updated the sign-in 578 00:24:11,760 --> 00:24:13,679 flow like I'd suggested and added a 579 00:24:13,679 --> 00:24:16,080 polling retry for transient failures. 580 00:24:16,080 --> 00:24:18,240 When it merged, I got a ping so I could 581 00:24:18,240 --> 00:24:20,640 tell the tester it was resolved. And 582 00:24:20,640 --> 00:24:22,240 because of the verification standard, 583 00:24:22,240 --> 00:24:24,480 the PR comes with evidence. On the left 584 00:24:24,480 --> 00:24:26,080 hand side, you can see the before 585 00:24:26,080 --> 00:24:28,480 screen. On the right, you can see after. 586 00:24:28,480 --> 00:24:30,320 I skimmed the implementation of this, 587 00:24:30,320 --> 00:24:32,320 but what I was most interested in was 588 00:24:32,320 --> 00:24:34,640 the evidence that it works. That's the 589 00:24:34,640 --> 00:24:38,760 part I'm scrutinizing. Now, 590 00:24:38,799 --> 00:24:40,080 there's other things we've been doing at 591 00:24:40,080 --> 00:24:43,039 style and uh that I've been doing with 592 00:24:43,039 --> 00:24:44,640 my personal projects that I think are 593 00:24:44,640 --> 00:24:46,960 really interesting in the space, but I'm 594 00:24:46,960 --> 00:24:49,600 going to call it there for this talk. 595 00:24:49,600 --> 00:24:51,520 Across all of this, the biggest insight 596 00:24:51,520 --> 00:24:53,120 that I want to drive home for everyone 597 00:24:53,120 --> 00:24:54,799 is that you can get higher quality 598 00:24:54,799 --> 00:24:58,159 results with AI by building systems that 599 00:24:58,159 --> 00:25:01,440 let you take your hands off the wheel. 600 00:25:01,440 --> 00:25:03,279 When you're working in a tight loop with 601 00:25:03,279 --> 00:25:05,840 AI, when it's asking you questions about 602 00:25:05,840 --> 00:25:08,640 what it can do and you have to decide, 603 00:25:08,640 --> 00:25:10,559 when it asks for permission to run that 604 00:25:10,559 --> 00:25:12,960 shell command or whether you want option 605 00:25:12,960 --> 00:25:15,919 A, Floop retry, or option B, Floop 606 00:25:15,919 --> 00:25:18,720 event, you've disempowered yourself. 607 00:25:18,720 --> 00:25:21,120 You've become a human tool call that the 608 00:25:21,120 --> 00:25:24,799 AI uses to let it do stuff. 609 00:25:24,799 --> 00:25:27,039 The empowering way to use these agents 610 00:25:27,039 --> 00:25:28,960 is to focus on building systems that 611 00:25:28,960 --> 00:25:31,360 help you get highquality outputs that 612 00:25:31,360 --> 00:25:33,840 creates the guarantees you believe are 613 00:25:33,840 --> 00:25:35,840 necessary from an engineering point of 614 00:25:35,840 --> 00:25:37,760 view. That requires getting that 615 00:25:37,760 --> 00:25:40,000 knowledge out of your head and into your 616 00:25:40,000 --> 00:25:42,960 codebase so it can be executed like code 617 00:25:42,960 --> 00:25:45,039 by the agents. 618 00:25:45,039 --> 00:25:46,799 If you want to chat later, some threads 619 00:25:46,799 --> 00:25:49,120 that I'm interested in talking about are 620 00:25:49,120 --> 00:25:50,720 when should we be doing requirements and 621 00:25:50,720 --> 00:25:52,720 planning and what strategies can we use 622 00:25:52,720 --> 00:25:55,840 to verify our solutions. Thanks. You can 623 00:25:55,840 --> 00:25:57,679 find me on LinkedIn. I post about this 624 00:25:57,679 --> 00:26:00,000 stuff if you find it interesting. Also, 625 00:26:00,000 --> 00:26:02,640 if you're in Melbourne and studying, for 626 00:26:02,640 --> 00:26:05,120 example, rocketry, um, come talk to me 627 00:26:05,120 --> 00:26:07,679 or add me on LinkedIn. Um, we have an 628 00:26:07,679 --> 00:26:09,520 emerging internship program and I'd love 629 00:26:09,520 --> 00:26:12,673 to talk to you about it. Thank you. 630 00:26:12,673 --> 00:26:14,693 [applause] 631 00:26:17,258 --> 00:26:19,278 [applause] 632 00:26:19,279 --> 00:26:21,440 Thank you so much, Ben. Um, and looking 633 00:26:21,440 --> 00:26:23,440 at your presentation, um, I think 634 00:26:23,440 --> 00:26:25,200 working in adtech really gives you an 635 00:26:25,200 --> 00:26:27,279 unfair advantage in in in making 636 00:26:27,279 --> 00:26:30,080 engaging slideshows. Um, we do have time 637 00:26:30,080 --> 00:26:31,840 for questions. We started a little bit 638 00:26:31,840 --> 00:26:33,919 late, so with the question, we might run 639 00:26:33,919 --> 00:26:35,919 a few minutes into the lunch break. If 640 00:26:35,919 --> 00:26:36,240 you're 641 00:26:36,240 --> 00:26:37,039 Don't do that. 642 00:26:37,039 --> 00:26:39,840 If you're starving, uh, feel free to go. 643 00:26:39,840 --> 00:26:43,880 But do we have any questions? 644 00:26:47,760 --> 00:26:51,039 I'm uh so I'm curious about how when you 645 00:26:51,039 --> 00:26:54,640 started um handing over to um the um 646 00:26:54,640 --> 00:26:57,360 nontechnical teams in 2025, what kind of 647 00:26:57,360 --> 00:26:59,039 enablement did you do to allow people 648 00:26:59,039 --> 00:27:00,960 with a background in writing to 649 00:27:00,960 --> 00:27:03,200 immediately jump into VIP coding? How 650 00:27:03,200 --> 00:27:05,520 did you kind of um get them into that 651 00:27:05,520 --> 00:27:08,000 and uh help them overcome fears and so 652 00:27:08,000 --> 00:27:08,640 on? 653 00:27:08,640 --> 00:27:10,400 Uh honestly, it was being driven by 654 00:27:10,400 --> 00:27:13,039 them. um they were becoming frustrated 655 00:27:13,039 --> 00:27:14,320 that we couldn't build the things that 656 00:27:14,320 --> 00:27:16,159 they wanted and they could see that they 657 00:27:16,159 --> 00:27:18,640 had this new tool claude that could um 658 00:27:18,640 --> 00:27:20,400 and so they really came to us and they 659 00:27:20,400 --> 00:27:21,520 were like we want to put this in the 660 00:27:21,520 --> 00:27:24,240 platform um we have been doing a lot of 661 00:27:24,240 --> 00:27:27,039 training um so we have uh someone from 662 00:27:27,039 --> 00:27:30,480 my team who's a junior engineer sits in 663 00:27:30,480 --> 00:27:32,799 uh the writing room every Friday uh and 664 00:27:32,799 --> 00:27:35,120 they can come to them with questions um 665 00:27:35,120 --> 00:27:37,600 and so that part we sort of do both 666 00:27:37,600 --> 00:27:39,039 sides but yeah definitely they were the 667 00:27:39,039 --> 00:27:41,520 ones that wanted this more than 668 00:27:41,520 --> 00:27:43,360 Do you ever find yourself still needing 669 00:27:43,360 --> 00:27:45,200 to manually intervene? 670 00:27:45,200 --> 00:27:47,039 Uh, do I need to manually intervene? 671 00:27:47,039 --> 00:27:50,640 Absolutely. Um, I would I would say it's 672 00:27:50,640 --> 00:27:52,320 very similar. Um, we've been talking 673 00:27:52,320 --> 00:27:54,880 about this metaphor of it's a lot like 674 00:27:54,880 --> 00:27:58,480 management. Um, and you sort of guide 675 00:27:58,480 --> 00:27:59,840 the people on your team, but every now 676 00:27:59,840 --> 00:28:01,120 and then you spot something and you're 677 00:28:01,120 --> 00:28:03,039 like, "That looks totally wrong. 678 00:28:03,039 --> 00:28:05,279 Everything about what's going on here is 679 00:28:05,279 --> 00:28:07,120 freaking me out." So then you have to 680 00:28:07,120 --> 00:28:09,120 dive in and really like have a look and 681 00:28:09,120 --> 00:28:11,520 guide things in the direction you want. 682 00:28:11,520 --> 00:28:13,039 Um and that's how I've been approaching 683 00:28:13,039 --> 00:28:17,120 it. Uh this morning um I noticed that a 684 00:28:17,120 --> 00:28:19,039 issue in my software factory had been 685 00:28:19,039 --> 00:28:21,919 closed but there was no uh 686 00:28:21,919 --> 00:28:24,000 implementation. Um and it had been 687 00:28:24,000 --> 00:28:26,159 closed because the agent had written a 688 00:28:26,159 --> 00:28:28,399 plan for how to implement it and had 689 00:28:28,399 --> 00:28:30,480 then decided that that was sufficient to 690 00:28:30,480 --> 00:28:33,120 mark it as done. Um and so I needed to 691 00:28:33,120 --> 00:28:35,279 add explicit guidance that a plan is not 692 00:28:35,279 --> 00:28:38,480 sufficient to mark a issue as done. 693 00:28:38,480 --> 00:28:40,960 Yeah. 694 00:28:40,960 --> 00:28:44,600 Any other questions? 695 00:28:49,919 --> 00:28:52,000 Um presumably you got into programming 696 00:28:52,000 --> 00:28:54,480 and engineering because you like coding. 697 00:28:54,480 --> 00:28:55,200 Uhhuh. 698 00:28:55,200 --> 00:28:57,919 Do you feel like you're losing that part 699 00:28:57,919 --> 00:29:00,240 of yourself now because you are coding 700 00:29:00,240 --> 00:29:02,480 less? Do you feel like because you've 701 00:29:02,480 --> 00:29:04,320 taken your hands off the wheel, 702 00:29:04,320 --> 00:29:06,640 you aren't enjoying the journey as much? 703 00:29:06,640 --> 00:29:09,039 This is the most fun I've had like in 704 00:29:09,039 --> 00:29:12,159 the last 20 years. Uh I think there's 705 00:29:12,159 --> 00:29:15,360 it's it's different, right? The thing 706 00:29:15,360 --> 00:29:17,360 that was really exciting early in my 707 00:29:17,360 --> 00:29:20,720 career was building building stuff. Um 708 00:29:20,720 --> 00:29:23,039 and I was the one building the stuff, 709 00:29:23,039 --> 00:29:25,120 writing code, and now the exciting thing 710 00:29:25,120 --> 00:29:27,520 is still building stuff. Um it's just 711 00:29:27,520 --> 00:29:30,000 yeah, just a little bit different. Um 712 00:29:30,000 --> 00:29:32,080 the 713 00:29:32,080 --> 00:29:33,760 the other side of this is it gives me 714 00:29:33,760 --> 00:29:36,159 huge ambition like the capability to 715 00:29:36,159 --> 00:29:38,720 have huge ambition. Um there's projects 716 00:29:38,720 --> 00:29:41,279 that I would have loved to do that are 717 00:29:41,279 --> 00:29:43,520 just really impossible outside of 718 00:29:43,520 --> 00:29:45,520 working hours and now I can totally 719 00:29:45,520 --> 00:29:47,760 tackle those projects and the parts that 720 00:29:47,760 --> 00:29:49,919 previously were a bit wrote or 721 00:29:49,919 --> 00:29:52,080 repetitive now are very easy to paper 722 00:29:52,080 --> 00:29:55,880 over. Yeah.