1 00:00:00,000 --> 00:00:08,469 foreign 2 00:00:00,500 --> 00:00:08,469 [Music] 3 00:00:11,840 --> 00:00:18,900 next up we have advanced testing and 4 00:00:15,599 --> 00:00:21,300 rust uh and we have turban Harding uh 5 00:00:18,900 --> 00:00:22,800 come to speak to us today about as you 6 00:00:21,300 --> 00:00:24,840 would expect from the title of the 7 00:00:22,800 --> 00:00:27,060 speech uh some Advanced Techniques for 8 00:00:24,840 --> 00:00:30,199 testing your rust code please give 9 00:00:27,060 --> 00:00:30,199 turbine a warm welcome 10 00:00:33,120 --> 00:00:36,059 how's everyone doing 11 00:00:34,920 --> 00:00:38,760 good 12 00:00:36,059 --> 00:00:41,579 cool I'm Tobin I'm going to talk about 13 00:00:38,760 --> 00:00:44,100 program um testing and rust Advanced is 14 00:00:41,579 --> 00:00:45,660 a bit of a subjective term so by 15 00:00:44,100 --> 00:00:47,520 Advanced I mean I feel like I've 16 00:00:45,660 --> 00:00:48,780 advanced in the last six months and I 17 00:00:47,520 --> 00:00:51,180 thought something was pretty cool so 18 00:00:48,780 --> 00:00:52,980 hopefully you'll think so too can I get 19 00:00:51,180 --> 00:00:55,879 a show of hands how many people write 20 00:00:52,980 --> 00:00:55,879 rust for a living 21 00:00:56,160 --> 00:01:01,820 written a few projects in Rust 22 00:00:59,640 --> 00:01:04,559 Court never heard of it 23 00:01:01,820 --> 00:01:07,020 so I always wanted to do always wanted 24 00:01:04,559 --> 00:01:09,600 to see a talk on testing 25 00:01:07,020 --> 00:01:11,159 cool no worries um if you want to look 26 00:01:09,600 --> 00:01:12,420 um play along there's a repo there I'm 27 00:01:11,159 --> 00:01:13,680 going to be showing a bunch of code I'm 28 00:01:12,420 --> 00:01:16,860 just going to be in the editor in the 29 00:01:13,680 --> 00:01:19,320 terminal most the time so yep 30 00:01:16,860 --> 00:01:21,299 um also it's not a formal talk it's just 31 00:01:19,320 --> 00:01:22,680 like a show and tell so if you want to 32 00:01:21,299 --> 00:01:24,540 know something yell out I can take 33 00:01:22,680 --> 00:01:27,000 questions anytime if I'm wrong feel free 34 00:01:24,540 --> 00:01:28,979 to stop me I'm not a QA expert I'm not a 35 00:01:27,000 --> 00:01:30,720 rust expert I'm just got six or seven 36 00:01:28,979 --> 00:01:33,119 years programming professionally and I'm 37 00:01:30,720 --> 00:01:34,380 just a guy likes rust 38 00:01:33,119 --> 00:01:36,299 cool 39 00:01:34,380 --> 00:01:37,560 so I'll run quickly over unit tests and 40 00:01:36,299 --> 00:01:40,380 integration tests and directory 41 00:01:37,560 --> 00:01:42,299 structure then I'll talk about rustock 42 00:01:40,380 --> 00:01:43,920 examples a couple of different ways to 43 00:01:42,299 --> 00:01:45,900 do mutation testing on what that is and 44 00:01:43,920 --> 00:01:47,520 why I think it's so exciting 45 00:01:45,900 --> 00:01:49,439 we'll touch on property-based testing 46 00:01:47,520 --> 00:01:51,600 and why I haven't done it much and then 47 00:01:49,439 --> 00:01:53,880 we'll do fuzzing and code verification 48 00:01:51,600 --> 00:01:56,280 I'm a bit light on knowledge on the last 49 00:01:53,880 --> 00:01:57,780 two topics but uh you know when I wrote 50 00:01:56,280 --> 00:01:59,520 the submission I was hoping to learn a 51 00:01:57,780 --> 00:02:00,930 bit more about it before I got here but 52 00:01:59,520 --> 00:02:02,399 yeah 53 00:02:00,930 --> 00:02:04,680 [Music] 54 00:02:02,399 --> 00:02:07,740 cool basic stuff unit testing 55 00:02:04,680 --> 00:02:09,959 integration test is language agnostic 56 00:02:07,740 --> 00:02:11,520 yeah it's not all important about the 57 00:02:09,959 --> 00:02:14,099 layout and whatnot there's rust specific 58 00:02:11,520 --> 00:02:15,540 ways to do it but integration tests the 59 00:02:14,099 --> 00:02:17,520 main difference is they test your API 60 00:02:15,540 --> 00:02:18,840 surface unit tests have access to your 61 00:02:17,520 --> 00:02:21,660 internals 62 00:02:18,840 --> 00:02:24,660 so the question that often faces me is 63 00:02:21,660 --> 00:02:26,520 you come to some new code looks alright 64 00:02:24,660 --> 00:02:28,680 it's got tests they look all right bit 65 00:02:26,520 --> 00:02:30,120 messy about their tests right 66 00:02:28,680 --> 00:02:32,700 um I read through the code 67 00:02:30,120 --> 00:02:34,140 Linus doesn't test so we're right to go 68 00:02:32,700 --> 00:02:36,420 right 69 00:02:34,140 --> 00:02:38,099 um but no 70 00:02:36,420 --> 00:02:39,660 how can we check how can we assure 71 00:02:38,099 --> 00:02:42,720 ourselves that we're not full of bugs 72 00:02:39,660 --> 00:02:45,000 and that things work 73 00:02:42,720 --> 00:02:46,739 one metric is test coverage people like 74 00:02:45,000 --> 00:02:48,239 to say a line of code was executed by 75 00:02:46,739 --> 00:02:52,319 their tests 76 00:02:48,239 --> 00:02:55,379 another way is mutation testing 77 00:02:52,319 --> 00:02:57,840 so mutation testing is as the name 78 00:02:55,379 --> 00:03:00,180 suggests mutating the code so you have a 79 00:02:57,840 --> 00:03:02,340 test Suite you have software that 80 00:03:00,180 --> 00:03:04,800 changes a line of code and then you run 81 00:03:02,340 --> 00:03:06,599 the tests if the tests pass then 82 00:03:04,800 --> 00:03:09,000 obviously you didn't test it because you 83 00:03:06,599 --> 00:03:11,040 just broke it and the test will pass 84 00:03:09,000 --> 00:03:12,480 so it's kind of a cool concept I never 85 00:03:11,040 --> 00:03:14,099 thought of it before until a few months 86 00:03:12,480 --> 00:03:16,800 ago and I came across it and I was like 87 00:03:14,099 --> 00:03:19,800 sweet so we have this one is Cargo 88 00:03:16,800 --> 00:03:21,599 mutants what cargo mutants does this 89 00:03:19,800 --> 00:03:23,459 default default is it just drops the 90 00:03:21,599 --> 00:03:25,080 body of any functional method and 91 00:03:23,459 --> 00:03:26,700 Returns the default type for the return 92 00:03:25,080 --> 00:03:28,860 value so if you're returning an INT 93 00:03:26,700 --> 00:03:30,720 it'll just return zero 94 00:03:28,860 --> 00:03:33,379 it's cool 95 00:03:30,720 --> 00:03:33,379 all right 96 00:03:33,599 --> 00:03:37,920 now I'm going to jump around I might 97 00:03:36,120 --> 00:03:39,300 talk a bit fast I get a bit excited so 98 00:03:37,920 --> 00:03:41,580 if I'm going too faster you want to know 99 00:03:39,300 --> 00:03:43,680 something really stop me 100 00:03:41,580 --> 00:03:45,180 this is just a wrapper type something 101 00:03:43,680 --> 00:03:46,440 you do in Rust often you're wrapping 102 00:03:45,180 --> 00:03:48,900 into something that you want to add 103 00:03:46,440 --> 00:03:50,760 abstraction to so I hacked this up it's 104 00:03:48,900 --> 00:03:52,319 a bit contrived but it's the aim is to 105 00:03:50,760 --> 00:03:55,260 show different tests right so we have a 106 00:03:52,319 --> 00:03:57,840 struck num represent a signed integer we 107 00:03:55,260 --> 00:04:00,239 have a Constructor from a sign type we 108 00:03:57,840 --> 00:04:02,459 have a Constructor from an unsigned type 109 00:04:00,239 --> 00:04:04,680 we have a getter two signed and two 110 00:04:02,459 --> 00:04:06,239 unsigned and then we have a more 111 00:04:04,680 --> 00:04:06,800 interesting method ABS to get the 112 00:04:06,239 --> 00:04:08,519 absolute value 113 00:04:06,800 --> 00:04:11,040 [Music] 114 00:04:08,519 --> 00:04:13,739 then we have some standard rust things 115 00:04:11,040 --> 00:04:15,540 triform is conversion functions and some 116 00:04:13,739 --> 00:04:16,979 error logic because I work on a library 117 00:04:15,540 --> 00:04:19,139 all day I do all the error handling 118 00:04:16,979 --> 00:04:21,959 manually and 119 00:04:19,139 --> 00:04:23,699 we have two tests for abs 120 00:04:21,959 --> 00:04:24,660 a positive test case and a negative test 121 00:04:23,699 --> 00:04:26,460 case 122 00:04:24,660 --> 00:04:28,620 sorry testing a positive value and a 123 00:04:26,460 --> 00:04:29,699 negative test value 124 00:04:28,620 --> 00:04:32,460 sorry 125 00:04:29,699 --> 00:04:35,460 looks good right 126 00:04:32,460 --> 00:04:37,440 all the non-trivial methods I tested the 127 00:04:35,460 --> 00:04:41,520 one non-trivial method is tested 128 00:04:37,440 --> 00:04:45,500 it's got two tests so is it all right 129 00:04:41,520 --> 00:04:45,500 let's see what cargo mutants thinks 130 00:04:52,020 --> 00:04:55,440 cool so if you remember we replaced the 131 00:04:54,120 --> 00:04:58,259 return value with the default return 132 00:04:55,440 --> 00:05:00,180 value it's as it shows there we're going 133 00:04:58,259 --> 00:05:02,340 to ignore display 134 00:05:00,180 --> 00:05:05,040 for now and we'll just look at so we can 135 00:05:02,340 --> 00:05:07,979 see num2 signed a num to unsigned are 136 00:05:05,040 --> 00:05:12,020 both not tested 137 00:05:07,979 --> 00:05:12,020 and if we go to the tests 138 00:05:12,419 --> 00:05:15,740 we can see that's clear right 139 00:05:25,560 --> 00:05:31,020 cool 140 00:05:27,240 --> 00:05:33,120 so we've added 141 00:05:31,020 --> 00:05:34,860 some stuff 142 00:05:33,120 --> 00:05:37,500 let's talk about that 143 00:05:34,860 --> 00:05:40,139 rust docks 144 00:05:37,500 --> 00:05:41,520 rust docs are specific to rust but I 145 00:05:40,139 --> 00:05:43,560 believe they're based on the python 146 00:05:41,520 --> 00:05:45,840 system they're similar to what the 147 00:05:43,560 --> 00:05:48,300 kernel uses now as well they pull the 148 00:05:45,840 --> 00:05:50,280 documentation out of the source code and 149 00:05:48,300 --> 00:05:51,840 one neat thing I don't know too much 150 00:05:50,280 --> 00:05:53,880 about the others but one neat thing is 151 00:05:51,840 --> 00:05:57,660 you can have code that's actually run in 152 00:05:53,880 --> 00:06:01,500 the examples so you can have 153 00:05:57,660 --> 00:06:02,759 oh and they only test the public API 154 00:06:01,500 --> 00:06:05,820 they're good because they demonstrate 155 00:06:02,759 --> 00:06:08,759 usage and 156 00:06:05,820 --> 00:06:11,240 their document better yep 157 00:06:08,759 --> 00:06:11,240 so 158 00:06:15,960 --> 00:06:19,800 see now 159 00:06:17,220 --> 00:06:22,919 from sign now has a bunch of stuff the 160 00:06:19,800 --> 00:06:24,600 standard examples header and then with 161 00:06:22,919 --> 00:06:26,220 the three back texts we have code there 162 00:06:24,600 --> 00:06:29,639 that runs 163 00:06:26,220 --> 00:06:32,160 now this is nice because 164 00:06:29,639 --> 00:06:33,180 um it renders I didn't open that before 165 00:06:32,160 --> 00:06:34,620 I'll just leave that if you want to see 166 00:06:33,180 --> 00:06:36,300 the rendered version and I'm going too 167 00:06:34,620 --> 00:06:37,979 slow we can look at that 168 00:06:36,300 --> 00:06:39,600 yeah so you've got when people read it 169 00:06:37,979 --> 00:06:41,160 in code they can see the example it gets 170 00:06:39,600 --> 00:06:43,680 rendered to HTML and all looks lovely 171 00:06:41,160 --> 00:06:47,900 syntax highlighting and whatnot and the 172 00:06:43,680 --> 00:06:47,900 cool thing is we can now 173 00:06:52,199 --> 00:06:55,400 spell nightly correctly 174 00:06:55,680 --> 00:06:58,680 so you see the top two are our unit 175 00:06:57,180 --> 00:07:02,280 tests and the next two other tests we 176 00:06:58,680 --> 00:07:06,080 just added the examples 177 00:07:02,280 --> 00:07:06,080 and let's run cargo mutants 178 00:07:10,139 --> 00:07:13,380 awesome 179 00:07:11,160 --> 00:07:14,460 so we killed two of the mutants that one 180 00:07:13,380 --> 00:07:16,020 like I said we're just going to ignore 181 00:07:14,460 --> 00:07:18,720 for now 182 00:07:16,020 --> 00:07:20,400 um but we've only done one test 183 00:07:18,720 --> 00:07:22,500 and it's telling us the whole function 184 00:07:20,400 --> 00:07:24,479 is tested so that's probably not true 185 00:07:22,500 --> 00:07:26,280 right 186 00:07:24,479 --> 00:07:27,900 we had a bug just the other day where it 187 00:07:26,280 --> 00:07:29,699 was a wrapper type and we had a sub 188 00:07:27,900 --> 00:07:31,500 function and an ad function and all 189 00:07:29,699 --> 00:07:33,599 these things and there was cut and paste 190 00:07:31,500 --> 00:07:35,400 error where the sub function called add 191 00:07:33,599 --> 00:07:38,099 so like it's pretty embarrassing thing 192 00:07:35,400 --> 00:07:40,199 to merge but this would have caught it 193 00:07:38,099 --> 00:07:42,720 so it is useful 194 00:07:40,199 --> 00:07:44,160 cool so now we have awesome docs we've 195 00:07:42,720 --> 00:07:46,020 got a good test coverage of the API 196 00:07:44,160 --> 00:07:47,819 because if anyone changes the API the 197 00:07:46,020 --> 00:07:50,580 test will fail straight away so if you 198 00:07:47,819 --> 00:07:51,900 see a patch come in with changes to the 199 00:07:50,580 --> 00:07:53,940 examples 200 00:07:51,900 --> 00:07:55,319 then you know that change it to the 201 00:07:53,940 --> 00:07:57,720 example code then it's a good hint that 202 00:07:55,319 --> 00:08:00,479 you've changed your API right 203 00:07:57,720 --> 00:08:02,940 cool but is it tested sufficiently 204 00:08:00,479 --> 00:08:05,039 let's give mutagen a go 205 00:08:02,940 --> 00:08:07,319 so mutagen is a mutation testing as well 206 00:08:05,039 --> 00:08:10,259 but mutagen is a little bit more 207 00:08:07,319 --> 00:08:11,400 aggressive than cargo moons cargo 208 00:08:10,259 --> 00:08:12,720 mutants you didn't have to change the 209 00:08:11,400 --> 00:08:14,580 source tree did you notice we could just 210 00:08:12,720 --> 00:08:16,919 run it we can run it on any tree and it 211 00:08:14,580 --> 00:08:19,620 does every method mutagen needs some 212 00:08:16,919 --> 00:08:21,960 Source tree changes but it can change a 213 00:08:19,620 --> 00:08:24,139 lot more things so let's have a look at 214 00:08:21,960 --> 00:08:24,139 that 215 00:08:30,900 --> 00:08:35,659 we've got a dependency on mutagen coming 216 00:08:33,000 --> 00:08:35,659 straight from git 217 00:08:36,120 --> 00:08:41,459 we've got this line 218 00:08:38,219 --> 00:08:43,320 10 got numbers yep line 10 um 10 and 11 219 00:08:41,459 --> 00:08:44,820 we add a use statement and I use an 220 00:08:43,320 --> 00:08:46,260 attribute don't worry about that it's 221 00:08:44,820 --> 00:08:47,640 not super important it just means that 222 00:08:46,260 --> 00:08:48,839 only include that line under certain 223 00:08:47,640 --> 00:08:51,480 conditions 224 00:08:48,839 --> 00:08:53,339 and then we're mutating as first start 225 00:08:51,480 --> 00:08:54,720 this one method 226 00:08:53,339 --> 00:08:56,160 so again the attribute is a little bit 227 00:08:54,720 --> 00:08:58,320 complicated it's not all that important 228 00:08:56,160 --> 00:09:00,660 it just says to the compiler too I'm to 229 00:08:58,320 --> 00:09:02,459 mutagen to mutate this 230 00:09:00,660 --> 00:09:06,260 so mutagen is going to mutate from 231 00:09:02,459 --> 00:09:06,260 unsigned let's give it a run 232 00:09:12,480 --> 00:09:17,279 cool so it's created three mutants and 233 00:09:15,360 --> 00:09:19,140 they've all survived 234 00:09:17,279 --> 00:09:20,580 um mutagens written by Geeks so it's all 235 00:09:19,140 --> 00:09:21,959 about survivors and mutants and zombies 236 00:09:20,580 --> 00:09:22,980 and stuff 237 00:09:21,959 --> 00:09:24,779 um 238 00:09:22,980 --> 00:09:26,880 we need to kill them 239 00:09:24,779 --> 00:09:30,000 so what to do you can see here all these 240 00:09:26,880 --> 00:09:32,610 numbers the first number 51 is the line 241 00:09:30,000 --> 00:09:35,730 so we can go over to 51. 242 00:09:32,610 --> 00:09:35,730 [Music] 243 00:09:36,860 --> 00:09:40,920 and as this line effects is less than or 244 00:09:39,300 --> 00:09:43,019 equal Max 245 00:09:40,920 --> 00:09:44,940 you can see it says what the mutant is 246 00:09:43,019 --> 00:09:45,899 doing replace less than or equal with 247 00:09:44,940 --> 00:09:48,540 less than 248 00:09:45,899 --> 00:09:49,980 so it becomes there's only a few it does 249 00:09:48,540 --> 00:09:51,839 a bunch of things I don't know all the 250 00:09:49,980 --> 00:09:53,880 things it does but a few of them um 251 00:09:51,839 --> 00:09:56,940 often survive and they're usually 252 00:09:53,880 --> 00:09:58,800 boundary cases it changes most num and 253 00:09:56,940 --> 00:10:00,000 if Clauses it changes most numbers but 254 00:09:58,800 --> 00:10:02,459 and it's usually at the boundary that 255 00:10:00,000 --> 00:10:03,779 that's interesting right so now we see 256 00:10:02,459 --> 00:10:06,060 already that this is pretty useful 257 00:10:03,779 --> 00:10:08,820 testing boundary cases is useful 258 00:10:06,060 --> 00:10:10,500 so we have three on one line we can see 259 00:10:08,820 --> 00:10:12,959 it's a boundary case so of course let's 260 00:10:10,500 --> 00:10:15,920 check if the value is Max so we go down 261 00:10:12,959 --> 00:10:15,920 let's 262 00:10:20,339 --> 00:10:24,560 I added another we added another test 263 00:10:26,940 --> 00:10:32,519 and here we go so we set X to Max we 264 00:10:29,940 --> 00:10:34,860 create a number and then we send it to 265 00:10:32,519 --> 00:10:35,940 sum we use it's a round trip so we 266 00:10:34,860 --> 00:10:37,200 create it with the front with the 267 00:10:35,940 --> 00:10:38,940 Constructor and then use the getter and 268 00:10:37,200 --> 00:10:40,740 assert that they're the same 269 00:10:38,940 --> 00:10:43,040 so yep that should look good that looks 270 00:10:40,740 --> 00:10:43,040 good right 271 00:10:44,880 --> 00:10:49,620 profit both killed except one 272 00:10:48,480 --> 00:10:52,079 all right 273 00:10:49,620 --> 00:10:53,640 everyone good so far 274 00:10:52,079 --> 00:10:55,260 yep you can read the fonts and 275 00:10:53,640 --> 00:10:57,740 everything's fine 276 00:10:55,260 --> 00:10:57,740 awesome 277 00:11:01,260 --> 00:11:06,000 all right so this 278 00:11:04,560 --> 00:11:09,959 um the next one is a little bit more 279 00:11:06,000 --> 00:11:11,459 curly we've got the mutant here is less 280 00:11:09,959 --> 00:11:13,680 than a replace less than or equals with 281 00:11:11,459 --> 00:11:16,380 greater than or equals so that seems 282 00:11:13,680 --> 00:11:17,519 like it's obvious right but when I saw 283 00:11:16,380 --> 00:11:19,680 this I was like well why is that even 284 00:11:17,519 --> 00:11:21,240 working of course like it seems strange 285 00:11:19,680 --> 00:11:22,200 it's not a boundary case it seems quite 286 00:11:21,240 --> 00:11:23,820 wrong 287 00:11:22,200 --> 00:11:26,100 and then 288 00:11:23,820 --> 00:11:28,260 I looked a little harder and I worked 289 00:11:26,100 --> 00:11:30,360 out that there was an invariant that 290 00:11:28,260 --> 00:11:36,300 wasn't documented 291 00:11:30,360 --> 00:11:39,240 and that is that from unsigned 292 00:11:36,300 --> 00:11:41,640 takes an unsigned 32-bit integer but 293 00:11:39,240 --> 00:11:43,560 ours is assigned 32-bit integer so 294 00:11:41,640 --> 00:11:45,180 obviously there's some bits that are out 295 00:11:43,560 --> 00:11:47,579 of range right 296 00:11:45,180 --> 00:11:49,560 so 297 00:11:47,579 --> 00:11:51,839 the test that I wrote 298 00:11:49,560 --> 00:11:55,500 is in the documentation so it's an 299 00:11:51,839 --> 00:11:58,440 examples so we added this error section 300 00:11:55,500 --> 00:11:59,820 that's like in Java where you it will 301 00:11:58,440 --> 00:12:01,980 put a heading errors and everyone knows 302 00:11:59,820 --> 00:12:05,160 that's describing the error tone so I 303 00:12:01,980 --> 00:12:07,920 have um described the range 304 00:12:05,160 --> 00:12:09,480 and then added a test that round trips 305 00:12:07,920 --> 00:12:10,560 to and from the Constructor in the 306 00:12:09,480 --> 00:12:12,240 gutter 307 00:12:10,560 --> 00:12:13,500 this is nice because if someone comes 308 00:12:12,240 --> 00:12:14,760 and changes one and doesn't think to 309 00:12:13,500 --> 00:12:16,140 change the other then you know and it's 310 00:12:14,760 --> 00:12:18,300 like really obvious then that they get 311 00:12:16,140 --> 00:12:19,980 us and um set us for people new to the 312 00:12:18,300 --> 00:12:22,440 language 313 00:12:19,980 --> 00:12:24,540 the library I work on Rust Bitcoin we 314 00:12:22,440 --> 00:12:26,040 try to do things that help people that 315 00:12:24,540 --> 00:12:27,420 are new to us because there might be a 316 00:12:26,040 --> 00:12:29,339 hardcore Engineers come in that haven't 317 00:12:27,420 --> 00:12:31,200 done rust so we try to explain things if 318 00:12:29,339 --> 00:12:33,000 they're a bit funny and also we try to 319 00:12:31,200 --> 00:12:34,380 explain Bitcoin stuff so that people who 320 00:12:33,000 --> 00:12:35,940 are new to bitcoin can learn from the 321 00:12:34,380 --> 00:12:38,360 code base too so it's kind of a nice 322 00:12:35,940 --> 00:12:38,360 principle 323 00:12:40,800 --> 00:12:44,899 all right let's see if we killed them 324 00:12:42,720 --> 00:12:44,899 all 325 00:12:47,459 --> 00:12:50,600 win all dead 326 00:12:51,600 --> 00:12:55,320 so 327 00:12:52,920 --> 00:12:57,660 now we have pretty good confidence in 328 00:12:55,320 --> 00:12:59,040 our tests right and here I'll segue to 329 00:12:57,660 --> 00:13:00,959 why I put this talk in in the first 330 00:12:59,040 --> 00:13:03,779 place testing something that I always 331 00:13:00,959 --> 00:13:05,760 want to have done good then it's often 332 00:13:03,779 --> 00:13:08,639 messy you feel bad doing code chain 333 00:13:05,760 --> 00:13:10,920 fixing people's things but and it's also 334 00:13:08,639 --> 00:13:12,360 sometimes boring but 335 00:13:10,920 --> 00:13:13,620 there's the idea that you can read the 336 00:13:12,360 --> 00:13:14,940 code and know what's going on but it's 337 00:13:13,620 --> 00:13:17,459 never the same as if you wrote it right 338 00:13:14,940 --> 00:13:19,980 whereas when you when I was doing using 339 00:13:17,459 --> 00:13:22,380 mutagen I was breaking the code and then 340 00:13:19,980 --> 00:13:24,300 really having to look why did it break 341 00:13:22,380 --> 00:13:26,279 and I so I really had to understand the 342 00:13:24,300 --> 00:13:27,660 algorithm so having to write those tests 343 00:13:26,279 --> 00:13:29,639 was really good and I really thought 344 00:13:27,660 --> 00:13:31,440 that after writing those tests I had 345 00:13:29,639 --> 00:13:33,180 covered the whole thing 346 00:13:31,440 --> 00:13:35,220 there's definitely could be ways you 347 00:13:33,180 --> 00:13:36,899 could break it that are not like that 348 00:13:35,220 --> 00:13:38,220 mutagen doesn't cover but I felt much 349 00:13:36,899 --> 00:13:40,079 more common that I'd really looked at 350 00:13:38,220 --> 00:13:42,120 each line so it's yeah it's pretty cool 351 00:13:40,079 --> 00:13:44,160 and I was pretty confident after doing 352 00:13:42,120 --> 00:13:46,320 mutagen after doing mutation testing 353 00:13:44,160 --> 00:13:48,060 that the API service was covered that 354 00:13:46,320 --> 00:13:50,519 the internals were like pretty good and 355 00:13:48,060 --> 00:13:51,720 that all the boundaries were tested 356 00:13:50,519 --> 00:13:54,600 but 357 00:13:51,720 --> 00:13:57,260 is it always correct 358 00:13:54,600 --> 00:13:57,260 what's always 359 00:13:58,680 --> 00:14:02,540 all right do you have any questions on 360 00:14:00,120 --> 00:14:02,540 fuzzing 361 00:14:03,000 --> 00:14:07,160 oh sorry not on fuzzing on mutation 362 00:14:04,740 --> 00:14:07,160 testing 363 00:14:14,160 --> 00:14:17,940 um I don't mutagen needs nightly I don't 364 00:14:16,019 --> 00:14:19,620 know exactly what it's um what's using I 365 00:14:17,940 --> 00:14:23,300 didn't look in there's yeah one of the 366 00:14:19,620 --> 00:14:26,459 dependencies requires nightly yeah 367 00:14:23,300 --> 00:14:28,620 awesome so um the question was why did I 368 00:14:26,459 --> 00:14:30,120 use nightly so that plus nightly that 369 00:14:28,620 --> 00:14:32,399 there's a nightly tool chain of stable 370 00:14:30,120 --> 00:14:34,620 tool chain and a bunch of old ones and 371 00:14:32,399 --> 00:14:36,420 so nightly has some features so most 372 00:14:34,620 --> 00:14:38,220 likely one of the dependencies uses a 373 00:14:36,420 --> 00:14:40,920 feature that's not in stable yet so 374 00:14:38,220 --> 00:14:42,480 that's why I was using nightly 375 00:14:40,920 --> 00:14:45,320 that was the question 376 00:14:42,480 --> 00:14:45,320 oh sorry 377 00:14:46,279 --> 00:14:50,220 yeah more 378 00:14:48,300 --> 00:14:53,399 cool um does anyone want to know about 379 00:14:50,220 --> 00:14:55,320 the attributes did they look funny 380 00:14:53,399 --> 00:14:57,480 if that's what you wanted to know about 381 00:14:55,320 --> 00:15:00,320 okay also I've got plenty of time so 382 00:14:57,480 --> 00:15:00,320 I'll show you guys that 383 00:15:03,240 --> 00:15:05,519 so 384 00:15:04,260 --> 00:15:06,899 um CFG 385 00:15:05,519 --> 00:15:08,459 and then the attribute is everything 386 00:15:06,899 --> 00:15:11,160 inside the brackets 387 00:15:08,459 --> 00:15:15,600 and it's an all is just um like a 388 00:15:11,160 --> 00:15:17,639 logical and so CFG says if um test and 389 00:15:15,600 --> 00:15:19,320 mutator both enabled they exist and 390 00:15:17,639 --> 00:15:20,519 they're enabled and then include the 391 00:15:19,320 --> 00:15:24,180 next line 392 00:15:20,519 --> 00:15:25,380 test is enabled by um cargo test these 393 00:15:24,180 --> 00:15:28,260 are called conditional compiler 394 00:15:25,380 --> 00:15:29,940 configuration options cargo test enables 395 00:15:28,260 --> 00:15:32,699 one called test they can be any string 396 00:15:29,940 --> 00:15:35,339 and I created one called mutate and 397 00:15:32,699 --> 00:15:39,060 you'll see when I run it I pass um 398 00:15:35,339 --> 00:15:40,500 Pastor rather compiler CFG mutate so 399 00:15:39,060 --> 00:15:42,240 that mutate can be anything and that 400 00:15:40,500 --> 00:15:45,360 just sets the variable to true 401 00:15:42,240 --> 00:15:49,079 so then that'll be included 402 00:15:45,360 --> 00:15:51,839 and then CFG attribute what it does is 403 00:15:49,079 --> 00:15:55,560 if the first thing is true it changes 404 00:15:51,839 --> 00:15:58,440 that line to be CFG the next thing so if 405 00:15:55,560 --> 00:16:00,240 test and mutate are enabled this will 406 00:15:58,440 --> 00:16:02,279 change to B this line will be changed by 407 00:16:00,240 --> 00:16:05,339 the compiler before code gets compiled 408 00:16:02,279 --> 00:16:07,740 to be CFG mutate which tells our mutagen 409 00:16:05,339 --> 00:16:09,420 to mutate the method 410 00:16:07,740 --> 00:16:11,579 okay 411 00:16:09,420 --> 00:16:13,079 so the if you look up online how to do 412 00:16:11,579 --> 00:16:14,760 this they don't do the rust Flags 413 00:16:13,079 --> 00:16:16,740 business that's something I added 414 00:16:14,760 --> 00:16:18,360 because I wanted I didn't want mutagen 415 00:16:16,740 --> 00:16:19,920 to run anytime anyone ran the tests I 416 00:16:18,360 --> 00:16:21,959 only want to just run when I read NCO 417 00:16:19,920 --> 00:16:23,519 and I really wanted it to run so that's 418 00:16:21,959 --> 00:16:26,040 why I did that we do the same with 419 00:16:23,519 --> 00:16:27,660 benchmarks um let's yeah it's kind of 420 00:16:26,040 --> 00:16:29,360 neat 421 00:16:27,660 --> 00:16:31,380 yeah so if you look up how to run 422 00:16:29,360 --> 00:16:34,199 benchmarks is another nightly thing 423 00:16:31,380 --> 00:16:38,519 where often benchmarks use this crate 424 00:16:34,199 --> 00:16:41,339 test and it needs nightly and 425 00:16:38,519 --> 00:16:43,920 if you do want the suggested online is 426 00:16:41,339 --> 00:16:46,199 put a line in the code that says if 427 00:16:43,920 --> 00:16:48,839 you're running tests enable this test 428 00:16:46,199 --> 00:16:50,880 grade and that breaks compiling with all 429 00:16:48,839 --> 00:16:53,759 features so you now cannot compile with 430 00:16:50,880 --> 00:16:55,500 cargo check or build dash dash all 431 00:16:53,759 --> 00:16:57,180 features and I like to be able to type 432 00:16:55,500 --> 00:16:59,160 that because it's heaps easier when 433 00:16:57,180 --> 00:17:01,920 you're I'm developing so I came up with 434 00:16:59,160 --> 00:17:04,940 this trick yeah to add a CFG flag and 435 00:17:01,920 --> 00:17:04,940 use that yeah 436 00:17:08,160 --> 00:17:14,819 cool sorry fuzzing um I don't know a lot 437 00:17:12,000 --> 00:17:16,919 about fuzzing but what I do know is that 438 00:17:14,819 --> 00:17:19,439 it's taking arbitrary data and feeding 439 00:17:16,919 --> 00:17:21,720 it into your functions and checking that 440 00:17:19,439 --> 00:17:24,540 they don't explode 441 00:17:21,720 --> 00:17:29,120 I don't have anything particularly good 442 00:17:24,540 --> 00:17:29,120 to fuzz in this code so I added 443 00:17:30,960 --> 00:17:35,520 oh sorry I do want to see the last few 444 00:17:33,900 --> 00:17:38,419 mutations 445 00:17:35,520 --> 00:17:38,419 thought that went quick 446 00:17:42,059 --> 00:17:45,900 so I added mutations to all add a 447 00:17:44,220 --> 00:17:48,660 mutagen to all the rest of the 448 00:17:45,900 --> 00:17:51,240 non-trivial methods and so now we've got 449 00:17:48,660 --> 00:17:54,559 13 mutants and 450 00:17:51,240 --> 00:17:54,559 they're all killed by two 451 00:17:55,740 --> 00:18:01,679 so you can see now from signs done 452 00:17:58,799 --> 00:18:06,559 two unsigns done and abs is done so 453 00:18:01,679 --> 00:18:06,559 we've got two unkilled mutants at 86 454 00:18:07,860 --> 00:18:10,460 all right 455 00:18:10,860 --> 00:18:15,240 all on this 456 00:18:13,080 --> 00:18:16,919 self-zero just gets the inner field so 457 00:18:15,240 --> 00:18:18,660 the inner field which is the assigned 458 00:18:16,919 --> 00:18:20,340 integer if you remember the sign integer 459 00:18:18,660 --> 00:18:22,440 is less than zero 460 00:18:20,340 --> 00:18:24,240 the mutations are replacing the zero 461 00:18:22,440 --> 00:18:28,340 with a one and the less than with less 462 00:18:24,240 --> 00:18:28,340 than or equal so standard boundary cases 463 00:18:28,620 --> 00:18:31,880 can anyone see what's the problem 464 00:18:36,600 --> 00:18:40,340 this if Clause determines which branch 465 00:18:38,880 --> 00:18:43,980 to take right 466 00:18:40,340 --> 00:18:47,400 changing less than or equal to from to 467 00:18:43,980 --> 00:18:49,559 eat less than to less than an equal and 468 00:18:47,400 --> 00:18:51,720 or changing the zero to one functionally 469 00:18:49,559 --> 00:18:53,460 just makes the zero value take a 470 00:18:51,720 --> 00:18:55,320 different branch 471 00:18:53,460 --> 00:18:56,700 but the zero for the value is zero it 472 00:18:55,320 --> 00:18:59,340 doesn't matter which branch you take so 473 00:18:56,700 --> 00:19:01,200 zero minus zero is the same as zero 474 00:18:59,340 --> 00:19:03,840 so I talked to the guy who wrote mutagen 475 00:19:01,200 --> 00:19:05,460 and he called this I don't care Claus I 476 00:19:03,840 --> 00:19:06,720 don't care statement so the value is 477 00:19:05,460 --> 00:19:08,820 zero I don't care you can Branch either 478 00:19:06,720 --> 00:19:10,559 way but mutagen can't actually handle 479 00:19:08,820 --> 00:19:12,059 that because it it can't tell what's the 480 00:19:10,559 --> 00:19:13,140 difference like why you can branch on 481 00:19:12,059 --> 00:19:15,600 either 482 00:19:13,140 --> 00:19:17,160 so an open question that um you guys 483 00:19:15,600 --> 00:19:19,320 could help me with is I don't know how 484 00:19:17,160 --> 00:19:21,000 to put this information into the code 485 00:19:19,320 --> 00:19:22,500 because I don't want I don't want to go 486 00:19:21,000 --> 00:19:24,419 and spend two weeks writing mutation 487 00:19:22,500 --> 00:19:26,280 testing do all this stuff and then a 488 00:19:24,419 --> 00:19:28,679 month later someone else comes back and 489 00:19:26,280 --> 00:19:30,299 they try and run it and there's 1500 490 00:19:28,679 --> 00:19:31,559 false positives 491 00:19:30,299 --> 00:19:33,059 and then they've got to go through 492 00:19:31,559 --> 00:19:34,980 everyone and check them that seems like 493 00:19:33,059 --> 00:19:37,020 a waste of time so the best I came up 494 00:19:34,980 --> 00:19:39,059 with is putting a code comment putting 495 00:19:37,020 --> 00:19:42,059 those lines and saying like lying such 496 00:19:39,059 --> 00:19:43,799 and such is false positive but that's 497 00:19:42,059 --> 00:19:46,380 like pretty hard to maintain it's going 498 00:19:43,799 --> 00:19:47,760 to go style pretty quickly but I thought 499 00:19:46,380 --> 00:19:49,500 at least if someone's just trying to 500 00:19:47,760 --> 00:19:50,760 skim over it they can go oh there's two 501 00:19:49,500 --> 00:19:51,960 thing false positives there that's 502 00:19:50,760 --> 00:19:55,620 probably all right 503 00:19:51,960 --> 00:19:58,679 but yeah I'm open to tips on that one 504 00:19:55,620 --> 00:20:01,200 so tests 505 00:19:58,679 --> 00:20:03,179 ABS handle zero 506 00:20:01,200 --> 00:20:05,280 that's an obvious um is that a boundary 507 00:20:03,179 --> 00:20:06,960 case when it's right in the middle 508 00:20:05,280 --> 00:20:08,280 um it's kind of obvious that those that 509 00:20:06,960 --> 00:20:10,880 should have been tested 510 00:20:08,280 --> 00:20:10,880 and 511 00:20:16,500 --> 00:20:21,360 so the suggestion is a conditional 512 00:20:18,120 --> 00:20:23,760 compilation and explicitly test for zero 513 00:20:21,360 --> 00:20:26,000 I think I did try that at some stage 514 00:20:23,760 --> 00:20:26,000 because 515 00:20:27,179 --> 00:20:32,880 so when you mutate your test for zero 516 00:20:29,760 --> 00:20:35,700 and then I'm I believe I did try that 517 00:20:32,880 --> 00:20:37,140 it's a good idea because I had it I 518 00:20:35,700 --> 00:20:38,640 believe I did try it and you get another 519 00:20:37,140 --> 00:20:40,380 false positive 520 00:20:38,640 --> 00:20:42,059 yeah there's I I think that's what 521 00:20:40,380 --> 00:20:44,100 happened at the end if we have this time 522 00:20:42,059 --> 00:20:46,340 I can type it up we can see but thanks 523 00:20:44,100 --> 00:20:46,340 thanks 524 00:20:46,380 --> 00:20:51,539 any questions about those tests not 525 00:20:48,960 --> 00:20:54,500 nothing crazy there 526 00:20:51,539 --> 00:20:54,500 all right that's fuzz 527 00:21:02,400 --> 00:21:06,380 all right to fuzz I added 528 00:21:14,039 --> 00:21:18,960 a from string implementation so that's a 529 00:21:17,100 --> 00:21:20,100 trait from straw that converts from a 530 00:21:18,960 --> 00:21:22,260 string to whatever it is you're doing 531 00:21:20,100 --> 00:21:24,419 you're whatever type 532 00:21:22,260 --> 00:21:26,160 and that's a candidate for fuzzing it's 533 00:21:24,419 --> 00:21:27,299 a bit contrived in this example but like 534 00:21:26,160 --> 00:21:29,340 I said I couldn't come up with any 535 00:21:27,299 --> 00:21:31,140 better so that's the best we've got so 536 00:21:29,340 --> 00:21:33,960 we take a string 537 00:21:31,140 --> 00:21:35,820 and I'd like to know that you can pass 538 00:21:33,960 --> 00:21:38,100 any string in the whole world and my um 539 00:21:35,820 --> 00:21:39,360 code's not going to blow up 540 00:21:38,100 --> 00:21:43,580 so 541 00:21:39,360 --> 00:21:43,580 to do that we create a crate called fuzz 542 00:21:48,080 --> 00:21:52,380 the um when I was going over this talk 543 00:21:50,520 --> 00:21:56,460 yesterday I realized does everyone know 544 00:21:52,380 --> 00:21:58,679 about Edition 2018 2021 2015. 545 00:21:56,460 --> 00:22:01,440 so every so many years Russ puts out a 546 00:21:58,679 --> 00:22:03,120 new a new addition and that's when they 547 00:22:01,440 --> 00:22:04,440 bundle up a whole like a whole heap of 548 00:22:03,120 --> 00:22:06,780 change or like right now here's a new 549 00:22:04,440 --> 00:22:09,059 edition and then people can jump from 550 00:22:06,780 --> 00:22:11,000 one to the next and this is Cargo cop 551 00:22:09,059 --> 00:22:13,320 program from code that used Edition 552 00:22:11,000 --> 00:22:15,179 2015. so some of it's going to seem 553 00:22:13,320 --> 00:22:16,260 strange like the fields down the bottom 554 00:22:15,179 --> 00:22:17,340 but if you don't know about that I 555 00:22:16,260 --> 00:22:20,000 wouldn't worry about it if it's strange 556 00:22:17,340 --> 00:22:20,000 just don't worry 557 00:22:28,140 --> 00:22:31,640 so I just write a little script that was 558 00:22:30,179 --> 00:22:33,539 just going to run it for 10 seconds 559 00:22:31,640 --> 00:22:36,240 fuzzling needs to run for a lot more 560 00:22:33,539 --> 00:22:37,860 than 10 seconds obviously 561 00:22:36,240 --> 00:22:40,200 so it's just going to keep feeding data 562 00:22:37,860 --> 00:22:41,100 into this hung fuzz provides this macro 563 00:22:40,200 --> 00:22:44,159 fuzz 564 00:22:41,100 --> 00:22:45,720 and that takes a closure D is the data 565 00:22:44,159 --> 00:22:47,280 it's going to pass in 566 00:22:45,720 --> 00:22:50,100 and it's going to call the function that 567 00:22:47,280 --> 00:22:51,480 I defined do test now do test is a 568 00:22:50,100 --> 00:22:53,780 little bit contrived 569 00:22:51,480 --> 00:22:56,640 here the standard library is pretty much 570 00:22:53,780 --> 00:22:58,380 taking care that I don't blow up so it's 571 00:22:56,640 --> 00:23:00,240 not really this is not really useful but 572 00:22:58,380 --> 00:23:02,460 you could imagine a time when this would 573 00:23:00,240 --> 00:23:05,100 be more useful so I convert the data to 574 00:23:02,460 --> 00:23:08,340 a string using from utf-8 575 00:23:05,100 --> 00:23:10,320 and then if it converts so if that 576 00:23:08,340 --> 00:23:13,320 random data happened to be valid utf-8 577 00:23:10,320 --> 00:23:15,539 then we can drop in and we can try and 578 00:23:13,320 --> 00:23:18,000 create a string from it this is because 579 00:23:15,539 --> 00:23:20,580 the from string extra accepts it takes 580 00:23:18,000 --> 00:23:23,700 us through as argument so 581 00:23:20,580 --> 00:23:24,840 I create a string and then imagine that 582 00:23:23,700 --> 00:23:26,280 we've written a library and we've got 583 00:23:24,840 --> 00:23:27,299 another Library somewhere else and we 584 00:23:26,280 --> 00:23:29,400 want to test our inflammation 585 00:23:27,299 --> 00:23:30,960 implementations against them in this 586 00:23:29,400 --> 00:23:32,340 case I'm using the stood lib which is a 587 00:23:30,960 --> 00:23:34,860 bit pointless but you get what I'm doing 588 00:23:32,340 --> 00:23:37,080 so I create a string from using from 589 00:23:34,860 --> 00:23:39,179 string Radix if that error is I assert 590 00:23:37,080 --> 00:23:40,980 that our library are it as well and if 591 00:23:39,179 --> 00:23:43,080 that doesn't error I assert that we got 592 00:23:40,980 --> 00:23:44,760 the same thing 593 00:23:43,080 --> 00:23:46,860 now I said I was going to mention prop 594 00:23:44,760 --> 00:23:49,200 testing prop testing property-based 595 00:23:46,860 --> 00:23:51,120 testing and the idea is that you pass 596 00:23:49,200 --> 00:23:52,860 random values in and then you write a 597 00:23:51,120 --> 00:23:54,539 test that tests the prop just the 598 00:23:52,860 --> 00:23:56,580 property not the exact value 599 00:23:54,539 --> 00:23:58,799 property testing is a version of fuzzing 600 00:23:56,580 --> 00:24:01,200 and this is like prop testing would 601 00:23:58,799 --> 00:24:02,940 probably be better in this case if like 602 00:24:01,200 --> 00:24:04,380 for all the other Constructors the 603 00:24:02,940 --> 00:24:06,480 weather line is between fuzzing and prop 604 00:24:04,380 --> 00:24:08,039 testing I don't know exactly 605 00:24:06,480 --> 00:24:11,059 and I'll explain to you why I don't use 606 00:24:08,039 --> 00:24:11,059 prop testing in a minute 607 00:24:16,500 --> 00:24:20,159 cool so it's kind of fast for 10 seconds 608 00:24:18,600 --> 00:24:22,320 and miraculously is going to find no 609 00:24:20,159 --> 00:24:24,179 bugs 610 00:24:22,320 --> 00:24:27,059 but that really hasn't given us that 611 00:24:24,179 --> 00:24:29,039 much more confidence right 612 00:24:27,059 --> 00:24:30,179 um yeah we have fuzzing and we run it 613 00:24:29,039 --> 00:24:32,100 during the night for a set amount of 614 00:24:30,179 --> 00:24:33,960 time and it gives us some confidence in 615 00:24:32,100 --> 00:24:36,360 some cases but we'd like something more 616 00:24:33,960 --> 00:24:38,520 we want to know if it's right all the 617 00:24:36,360 --> 00:24:41,419 time 618 00:24:38,520 --> 00:24:41,419 code verification 619 00:24:41,940 --> 00:24:45,900 so 620 00:24:43,919 --> 00:24:47,520 prop testing passes some number of 621 00:24:45,900 --> 00:24:50,159 random variables to your method or 622 00:24:47,520 --> 00:24:53,220 function but imagine if you have a 623 00:24:50,159 --> 00:24:54,480 function that takes two u64s the space 624 00:24:53,220 --> 00:24:55,919 that you're going to test is pretty 625 00:24:54,480 --> 00:24:58,200 small compared to the whole thing right 626 00:24:55,919 --> 00:25:01,260 so it'll be nice if we could have some 627 00:24:58,200 --> 00:25:04,260 way of knowing if it's for everything 628 00:25:01,260 --> 00:25:07,200 and what we can use is something called 629 00:25:04,260 --> 00:25:11,760 canny Kenny's a library that does code 630 00:25:07,200 --> 00:25:13,620 verification it uses math magic to 631 00:25:11,760 --> 00:25:15,900 verify that all of any value 632 00:25:13,620 --> 00:25:19,080 mathematically every line can't be 633 00:25:15,900 --> 00:25:21,240 reached or and can't crash it can't 634 00:25:19,080 --> 00:25:23,039 handle all logic it chokes on Loops a 635 00:25:21,240 --> 00:25:24,840 bit as you can imagine the mask quite 636 00:25:23,039 --> 00:25:27,539 complex and if there's like nested Loops 637 00:25:24,840 --> 00:25:30,539 it just takes forever to verify so how I 638 00:25:27,539 --> 00:25:32,340 did it is I would add cany tests run it 639 00:25:30,539 --> 00:25:33,480 and if they ended within like a 640 00:25:32,340 --> 00:25:35,159 reasonable amount of time a minute or 641 00:25:33,480 --> 00:25:36,480 something then I was like sweet if not 642 00:25:35,159 --> 00:25:37,620 there's some things you can do otherwise 643 00:25:36,480 --> 00:25:39,900 you have to use a different type of 644 00:25:37,620 --> 00:25:43,279 testing so it's good for a subset of 645 00:25:39,900 --> 00:25:43,279 things that you can test 646 00:25:53,220 --> 00:25:56,220 foreign 647 00:25:58,820 --> 00:26:02,940 's pretty easy to use you don't even 648 00:26:01,020 --> 00:26:04,260 have to add a dependency because you're 649 00:26:02,940 --> 00:26:06,360 going to we run it with a different 650 00:26:04,260 --> 00:26:08,520 command you just add this with a 651 00:26:06,360 --> 00:26:12,360 compiler attribute again conditional 652 00:26:08,520 --> 00:26:14,279 configuration attribute and then 653 00:26:12,360 --> 00:26:16,559 where you have the attribute proof which 654 00:26:14,279 --> 00:26:18,480 says this function's a proof 655 00:26:16,559 --> 00:26:20,580 now similar to if you've done prop 656 00:26:18,480 --> 00:26:22,100 testing so County any will give just 657 00:26:20,580 --> 00:26:25,020 gives me a random 658 00:26:22,100 --> 00:26:26,100 sorry it um signals that this can be any 659 00:26:25,020 --> 00:26:28,380 value 660 00:26:26,100 --> 00:26:30,539 and then I write the test you can also 661 00:26:28,380 --> 00:26:32,640 limit that so I could say any i32 and 662 00:26:30,539 --> 00:26:34,380 then I there's a method I can function I 663 00:26:32,640 --> 00:26:36,299 can call that says assume that it's less 664 00:26:34,380 --> 00:26:39,059 than 10 if I want to test everything 665 00:26:36,299 --> 00:26:40,860 less than 10. and then I just write a 666 00:26:39,059 --> 00:26:43,020 test very similar to what a prop based 667 00:26:40,860 --> 00:26:45,240 test would be 668 00:26:43,020 --> 00:26:47,640 sorry very similar to what I testify 669 00:26:45,240 --> 00:26:49,140 pasta value because I know 670 00:26:47,640 --> 00:26:51,000 it's a property isn't it because we 671 00:26:49,140 --> 00:26:52,440 don't know the value so we converted it 672 00:26:51,000 --> 00:26:54,360 and then we just checked that it was the 673 00:26:52,440 --> 00:26:55,620 same so unsynabs is coming from the 674 00:26:54,360 --> 00:26:57,840 standard Library 675 00:26:55,620 --> 00:26:59,820 ABS is the function we wrote and we're 676 00:26:57,840 --> 00:27:01,740 just comparing they're the same so again 677 00:26:59,820 --> 00:27:03,179 a little bit contrived but we were doing 678 00:27:01,740 --> 00:27:04,620 this with like a hashing library and 679 00:27:03,179 --> 00:27:06,419 then we use another hashing library and 680 00:27:04,620 --> 00:27:09,080 we check if we get the same thing so it 681 00:27:06,419 --> 00:27:09,080 can be useful 682 00:27:13,620 --> 00:27:19,020 cool so I've got one failure 683 00:27:16,260 --> 00:27:20,940 that's nice we've got three successes I 684 00:27:19,020 --> 00:27:22,020 won't dig too much into it because I I'm 685 00:27:20,940 --> 00:27:23,159 like I said I don't know I don't 686 00:27:22,020 --> 00:27:25,740 understand it all that much it's got 687 00:27:23,159 --> 00:27:27,000 good documentation but basically I just 688 00:27:25,740 --> 00:27:29,220 looked for red and then tried to work 689 00:27:27,000 --> 00:27:32,640 out why it was red 690 00:27:29,220 --> 00:27:35,659 so if we look here line 89 attempt to 691 00:27:32,640 --> 00:27:35,659 subtract with overflow 692 00:27:46,520 --> 00:27:50,520 all right so it's the same line we're 693 00:27:48,480 --> 00:27:52,559 looking at before now it took me a fair 694 00:27:50,520 --> 00:27:53,700 while to work this out and actually I 695 00:27:52,559 --> 00:27:54,840 had to work it out again yesterday when 696 00:27:53,700 --> 00:27:56,820 I was going over it because I wrote this 697 00:27:54,840 --> 00:27:58,740 a little few weeks ago 698 00:27:56,820 --> 00:28:02,600 um Can anyone say what's wrong 699 00:27:58,740 --> 00:28:02,600 attempt to subtract with overflow 700 00:28:04,440 --> 00:28:09,480 the clue is our inner type is assigned 701 00:28:07,500 --> 00:28:11,840 integer and we're returning an unsigned 702 00:28:09,480 --> 00:28:11,840 integer 703 00:28:13,980 --> 00:28:19,980 how's your two's complement knowledge 704 00:28:17,400 --> 00:28:22,799 so the minimum value the minimum 705 00:28:19,980 --> 00:28:24,840 negative value of a type of an unsigned 706 00:28:22,799 --> 00:28:26,279 to other sign type can't fit in the 707 00:28:24,840 --> 00:28:28,500 unsigned type 708 00:28:26,279 --> 00:28:30,179 so if it was is it two bytes would be 709 00:28:28,500 --> 00:28:32,760 minus four three bytes minus four 710 00:28:30,179 --> 00:28:34,080 doesn't fit into three bytes three is 711 00:28:32,760 --> 00:28:35,880 the maximum value 712 00:28:34,080 --> 00:28:38,419 does that make sense 713 00:28:35,880 --> 00:28:38,419 yes 714 00:28:41,100 --> 00:28:45,179 not correct 715 00:28:42,900 --> 00:28:47,720 well 716 00:28:45,179 --> 00:28:47,720 the gate 717 00:28:48,779 --> 00:28:53,030 and let me 718 00:28:51,179 --> 00:28:56,109 let me see 719 00:28:53,030 --> 00:28:56,109 [Music] 720 00:28:57,179 --> 00:29:01,200 so three bytes there if it was a three 721 00:28:59,760 --> 00:29:03,059 bit it's a 32-bit number if it was a 722 00:29:01,200 --> 00:29:07,320 three bit number the minimum value would 723 00:29:03,059 --> 00:29:10,580 be -4 and for an unsigned 724 00:29:07,320 --> 00:29:10,580 you can't have positive four 725 00:29:12,000 --> 00:29:15,179 so that you think that's not the reason 726 00:29:13,500 --> 00:29:17,600 that that's that that arm test is 727 00:29:15,179 --> 00:29:17,600 triggering 728 00:29:18,899 --> 00:29:22,460 yeah so that's why the Overflow 729 00:29:28,200 --> 00:29:30,690 the 730 00:29:29,460 --> 00:29:33,739 fixes 731 00:29:30,690 --> 00:29:33,739 [Music] 732 00:29:35,520 --> 00:29:40,140 outside the the what was said was a 733 00:29:37,980 --> 00:29:42,059 suggestion is to change 734 00:29:40,140 --> 00:29:43,919 to convert to signed 735 00:29:42,059 --> 00:29:45,840 if you cast a sign then you get another 736 00:29:43,919 --> 00:29:46,140 warning for that you're losing value I 737 00:29:45,840 --> 00:29:49,219 believe 738 00:29:46,140 --> 00:29:49,219 [Music] 739 00:29:51,000 --> 00:29:54,600 it's already signed 740 00:29:52,919 --> 00:29:55,799 so yeah we can't cast one sign because 741 00:29:54,600 --> 00:29:58,320 we don't have the we don't have the 742 00:29:55,799 --> 00:30:00,240 value right so I didn't add the fix the 743 00:29:58,320 --> 00:30:01,440 fixes to to return an error right and 744 00:30:00,240 --> 00:30:04,500 then check the value and if it's out of 745 00:30:01,440 --> 00:30:07,220 range you have to return an error 746 00:30:04,500 --> 00:30:07,220 would that be right 747 00:30:07,410 --> 00:30:12,600 [Music] 748 00:30:09,539 --> 00:30:15,559 you can simply check for the 749 00:30:12,600 --> 00:30:15,559 value and return 750 00:30:15,960 --> 00:30:19,460 value 751 00:30:17,880 --> 00:30:23,220 in this case for the other approach 752 00:30:19,460 --> 00:30:24,899 involves magic involving ignoring errors 753 00:30:23,220 --> 00:30:28,520 on arithmetic 754 00:30:24,899 --> 00:30:28,520 in Gold that has a bit stream 755 00:30:29,100 --> 00:30:33,419 so if I got you right you're saying um 756 00:30:31,020 --> 00:30:35,580 ignore errors 757 00:30:33,419 --> 00:30:37,500 if you negate 758 00:30:35,580 --> 00:30:39,679 so who's talking so I can look at you oh 759 00:30:37,500 --> 00:30:39,679 yeah yeah 760 00:30:40,620 --> 00:30:45,360 wise representation 761 00:30:42,419 --> 00:30:47,700 of your answer is the same regardless of 762 00:30:45,360 --> 00:30:51,299 whether it's signed or unsigned 763 00:30:47,700 --> 00:30:53,760 so in effect if you can convince 764 00:30:51,299 --> 00:30:55,860 rust to take the bits 765 00:30:53,760 --> 00:30:58,320 and just manage it you get the right 766 00:30:55,860 --> 00:30:59,820 answer yeah yeah awesome awesome yeah 767 00:30:58,320 --> 00:31:02,480 good idea 768 00:30:59,820 --> 00:31:02,480 thank you 769 00:31:04,740 --> 00:31:09,779 so we saw that um verification is useful 770 00:31:08,399 --> 00:31:11,399 we saw that it found a bug that I 771 00:31:09,779 --> 00:31:13,620 wouldn't have seen like that that's 772 00:31:11,399 --> 00:31:14,700 pretty that's reasonably I'm reasonably 773 00:31:13,620 --> 00:31:16,020 confident to say that I wouldn't have 774 00:31:14,700 --> 00:31:17,340 seen that I wrote it and I didn't see it 775 00:31:16,020 --> 00:31:20,039 I think that could have gone through 776 00:31:17,340 --> 00:31:22,620 review and not been seen so is that um 777 00:31:20,039 --> 00:31:25,200 Kenny for the win 778 00:31:22,620 --> 00:31:28,559 yeah pretty sweet right it is um can be 779 00:31:25,200 --> 00:31:30,779 slow to run and mutant ninja testing has 780 00:31:28,559 --> 00:31:32,399 a problem that I wrote like a bunch of 781 00:31:30,779 --> 00:31:34,200 tests and then I looked back and I'm 782 00:31:32,399 --> 00:31:35,880 like that's all super trivial that's two 783 00:31:34,200 --> 00:31:37,440 hours of time that I don't think that 784 00:31:35,880 --> 00:31:39,000 really adds any value 785 00:31:37,440 --> 00:31:41,100 so I don't know exactly where the line 786 00:31:39,000 --> 00:31:42,960 is like how much time to spend on this 787 00:31:41,100 --> 00:31:44,520 I'm still learning that 788 00:31:42,960 --> 00:31:46,320 and I don't know how much time to wait 789 00:31:44,520 --> 00:31:48,299 for the verification was one minute too 790 00:31:46,320 --> 00:31:50,779 long too long or too short I'm pretty 791 00:31:48,299 --> 00:31:50,779 impatient 792 00:31:50,880 --> 00:31:54,179 and yeah if you've got 20 000 lines of 793 00:31:52,860 --> 00:31:57,240 code and you start putting you know you 794 00:31:54,179 --> 00:32:01,279 put in 100 200 mutations 795 00:31:57,240 --> 00:32:01,279 so yeah a few open questions 796 00:32:03,360 --> 00:32:08,760 so summing up we looked at rust dock we 797 00:32:06,600 --> 00:32:11,220 looked at examples we saw that the tests 798 00:32:08,760 --> 00:32:14,039 run those we looked at mutation testing 799 00:32:11,220 --> 00:32:16,020 basic with mutants cargo mutants and you 800 00:32:14,039 --> 00:32:18,000 didn't have to change the source tree we 801 00:32:16,020 --> 00:32:20,460 looked at mutagen which needed Source 802 00:32:18,000 --> 00:32:22,440 tree fixes and possibly maintenance 803 00:32:20,460 --> 00:32:24,000 burden it's definitely a pain in the ass 804 00:32:22,440 --> 00:32:26,100 to run because you get a lot of false 805 00:32:24,000 --> 00:32:27,240 positives and it takes ages once you 806 00:32:26,100 --> 00:32:29,100 start putting a lot of these it takes 807 00:32:27,240 --> 00:32:30,120 ages 808 00:32:29,100 --> 00:32:31,919 um I was trying to mess around with 809 00:32:30,120 --> 00:32:33,600 attribute so you could run specific bits 810 00:32:31,919 --> 00:32:36,779 but I couldn't find a really nice way in 811 00:32:33,600 --> 00:32:39,059 the end I was just like using said to 812 00:32:36,779 --> 00:32:40,740 change everything change all the lines 813 00:32:39,059 --> 00:32:42,419 the attributes and then making a change 814 00:32:40,740 --> 00:32:44,039 in one place and then resetting the um 815 00:32:42,419 --> 00:32:46,200 the index 816 00:32:44,039 --> 00:32:46,930 we've got Hong fuzz for fuzzing uncanny 817 00:32:46,200 --> 00:32:49,620 for verification 818 00:32:46,930 --> 00:32:51,419 [Music] 819 00:32:49,620 --> 00:32:53,410 my talk 820 00:32:51,419 --> 00:32:56,549 so listen 821 00:32:53,410 --> 00:32:56,549 [Music] 822 00:32:56,880 --> 00:33:02,179 you guys want to say anything or ask any 823 00:32:58,860 --> 00:33:02,179 questions or give me any ideas 824 00:33:13,620 --> 00:33:18,659 thanks very much for the talk thank you 825 00:33:15,860 --> 00:33:21,840 coming to us so like my background's 826 00:33:18,659 --> 00:33:23,460 Java kotlin Dart and they're like the 827 00:33:21,840 --> 00:33:27,360 most common like 828 00:33:23,460 --> 00:33:29,159 rough measure of unitesque is coverage 829 00:33:27,360 --> 00:33:30,480 I'm just wondering like you didn't 830 00:33:29,159 --> 00:33:34,159 mention that or would you be able to 831 00:33:30,480 --> 00:33:37,080 contrast like using like like code 832 00:33:34,159 --> 00:33:38,700 versus for instance mutation testing and 833 00:33:37,080 --> 00:33:41,519 like do you think one's more useful in 834 00:33:38,700 --> 00:33:43,080 the other in terms of rust or like one 835 00:33:41,519 --> 00:33:45,360 doesn't like coverage doesn't coverage 836 00:33:43,080 --> 00:33:47,100 doesn't make so much sense in Rust sure 837 00:33:45,360 --> 00:33:49,080 I can only give an opinion it sounds 838 00:33:47,100 --> 00:33:51,179 like Tully's my personal opinion 839 00:33:49,080 --> 00:33:54,059 um I've never been a cure like employed 840 00:33:51,179 --> 00:33:56,279 as a QA guy I always was suspicious of 841 00:33:54,059 --> 00:33:58,200 coverage because I just have to execute 842 00:33:56,279 --> 00:34:00,059 that line of code in a test 843 00:33:58,200 --> 00:34:03,019 but like 844 00:34:00,059 --> 00:34:05,940 there's different kinds of Courage c0c10 845 00:34:03,019 --> 00:34:08,280 ah cool okay there are different models 846 00:34:05,940 --> 00:34:10,440 of coverage um C zero coverages did this 847 00:34:08,280 --> 00:34:12,000 line execute yeah one coverages did you 848 00:34:10,440 --> 00:34:13,679 test all the branches 849 00:34:12,000 --> 00:34:16,679 etc etc so you get a different depth of 850 00:34:13,679 --> 00:34:18,060 coverage cool levels of things and a lot 851 00:34:16,679 --> 00:34:19,440 of people just stop and did I execute 852 00:34:18,060 --> 00:34:21,419 this line but that is not the only model 853 00:34:19,440 --> 00:34:22,679 of coverage that exists excellent okay 854 00:34:21,419 --> 00:34:24,240 thank you that's awesome all right 855 00:34:22,679 --> 00:34:25,980 that's something I've learned 856 00:34:24,240 --> 00:34:27,300 so yeah I can't answer is better and I'm 857 00:34:25,980 --> 00:34:28,859 definitely going to go home and check in 858 00:34:27,300 --> 00:34:30,300 coverage I always just thought the badge 859 00:34:28,859 --> 00:34:33,320 and was like what does that tell me 860 00:34:30,300 --> 00:34:33,320 thank you that's excellent 861 00:34:34,919 --> 00:34:39,560 I fully answer your question 862 00:34:37,020 --> 00:34:39,560 cheers 863 00:34:40,260 --> 00:34:43,379 any other questions or anything else 864 00:34:41,700 --> 00:34:45,000 you'd like to see further I'm about all 865 00:34:43,379 --> 00:34:47,280 day so you can um come up to me if you 866 00:34:45,000 --> 00:34:48,720 want to chat yeah I literally got really 867 00:34:47,280 --> 00:34:50,220 excited when I was doing that mutagen 868 00:34:48,720 --> 00:34:52,560 stuff look in this algorithm when I 869 00:34:50,220 --> 00:34:54,659 found a code path uh false positive and 870 00:34:52,560 --> 00:34:57,060 I worked out that the the algorithm was 871 00:34:54,659 --> 00:34:58,320 correct but it was so complex it took me 872 00:34:57,060 --> 00:34:59,880 like 10 minutes to work out that was 873 00:34:58,320 --> 00:35:01,380 correct and I was like that's awesome I 874 00:34:59,880 --> 00:35:03,780 wouldn't have looked at it that hard I'd 875 00:35:01,380 --> 00:35:06,380 just gone yep sweet looks right so yeah 876 00:35:03,780 --> 00:35:06,380 it was really cool 877 00:35:07,380 --> 00:35:12,740 off the back 878 00:35:09,119 --> 00:35:12,740 I've been asked before yeah 879 00:35:14,579 --> 00:35:18,599 I suggested before about conditional 880 00:35:16,619 --> 00:35:20,220 compilation of an explicit zero test and 881 00:35:18,599 --> 00:35:24,440 you said there was an issue with another 882 00:35:20,220 --> 00:35:24,440 false positive do you want to try it yep 883 00:35:41,940 --> 00:35:45,119 so what was the idea we wanted to check 884 00:35:43,560 --> 00:35:48,680 if the value is zero and then do 885 00:35:45,119 --> 00:35:48,680 something different on zero to one right 886 00:35:55,859 --> 00:36:03,260 so is your idea to when mutation is on 887 00:35:59,400 --> 00:36:03,260 when mutation testing is on to do that 888 00:36:37,940 --> 00:36:41,760 I don't actually use this form factor of 889 00:36:40,380 --> 00:36:44,119 keyboard at home so there's a lot of 890 00:36:41,760 --> 00:36:44,119 stabbing 891 00:36:49,339 --> 00:36:54,140 so that didn't work at 90 892 00:36:58,800 --> 00:37:02,099 . cool I'm happy to say this is what I 893 00:37:00,599 --> 00:37:03,780 did last time and that's cool that it 894 00:37:02,099 --> 00:37:06,140 did do so now we've got the zero one 895 00:37:03,780 --> 00:37:06,140 problem 896 00:37:08,400 --> 00:37:12,119 oh yeah so here we can't add another 897 00:37:10,079 --> 00:37:13,680 line the zero one problem is a false 898 00:37:12,119 --> 00:37:15,300 positive you can't fix so we've replaced 899 00:37:13,680 --> 00:37:16,800 one false positive with another at least 900 00:37:15,300 --> 00:37:18,720 there 901 00:37:16,800 --> 00:37:21,380 and then we've still got a problem in 902 00:37:18,720 --> 00:37:21,380 94. 903 00:37:21,780 --> 00:37:25,200 the same problem we could we could 904 00:37:24,060 --> 00:37:27,180 assume that we could change the next 905 00:37:25,200 --> 00:37:28,619 line to remove that and we'd get another 906 00:37:27,180 --> 00:37:29,880 so we've got another false positive 907 00:37:28,619 --> 00:37:32,060 we'll replace one false positive with 908 00:37:29,880 --> 00:37:32,060 another 909 00:37:32,160 --> 00:37:35,040 do you think you could get rid of those 910 00:37:33,420 --> 00:37:37,820 are you convinced that I didn't face it 911 00:37:35,040 --> 00:37:37,820 any further than that 912 00:37:39,480 --> 00:37:42,260 the quest 913 00:37:42,960 --> 00:37:45,720 it's not very clear why we've got the 914 00:37:44,520 --> 00:37:47,760 new false positive 915 00:37:45,720 --> 00:37:49,260 oh that's because it's the this boundary 916 00:37:47,760 --> 00:37:51,680 case where you've got a zero so you 917 00:37:49,260 --> 00:37:51,680 could change 918 00:37:52,619 --> 00:37:56,520 oh no if that's a one that should be 919 00:37:54,300 --> 00:37:58,500 fine that's 920 00:37:56,520 --> 00:38:03,599 oh yeah because one is two zero minus 921 00:37:58,500 --> 00:38:05,960 one is one and one is one 922 00:38:03,599 --> 00:38:05,960 no 923 00:38:07,140 --> 00:38:10,920 yeah so that's the same right I know 924 00:38:09,060 --> 00:38:14,359 that's not because it is trans zero that 925 00:38:10,920 --> 00:38:14,359 is not clear why that's false positive 926 00:38:16,380 --> 00:38:19,640 ah because we haven't written a test 927 00:38:20,940 --> 00:38:24,119 so we have to add it so we'd have to add 928 00:38:22,500 --> 00:38:27,420 a test you want to see me do that have a 929 00:38:24,119 --> 00:38:30,200 good time we'd have to add the test 930 00:38:27,420 --> 00:38:30,200 for one 931 00:38:46,140 --> 00:38:49,700 it's a bit lazy do it like this 932 00:38:55,140 --> 00:38:58,140 foreign 933 00:39:09,780 --> 00:39:12,740 system works 934 00:39:13,680 --> 00:39:17,280 so now we're left with the ones at 94 so 935 00:39:15,720 --> 00:39:20,820 we could probably get rid of those so I 936 00:39:17,280 --> 00:39:22,740 guess then the question is how much 937 00:39:20,820 --> 00:39:25,560 um 938 00:39:22,740 --> 00:39:28,440 the function's a lot more messy now 939 00:39:25,560 --> 00:39:30,660 it's got that is that is that a net win 940 00:39:28,440 --> 00:39:32,520 to remove a false positive 941 00:39:30,660 --> 00:39:34,619 I don't know I tend to think that's not 942 00:39:32,520 --> 00:39:36,660 a net win 943 00:39:34,619 --> 00:39:37,859 yeah I kind of get I kind of feel like I 944 00:39:36,660 --> 00:39:39,660 don't want to come into the code base 945 00:39:37,859 --> 00:39:41,280 and make a bunch of changes and then the 946 00:39:39,660 --> 00:39:42,960 next guy comes along is like wow what's 947 00:39:41,280 --> 00:39:45,000 this 15 lines of [ __ ] that I have to 948 00:39:42,960 --> 00:39:46,500 look past now just so that you could fix 949 00:39:45,000 --> 00:39:48,240 one small thing that probably wasn't a 950 00:39:46,500 --> 00:39:49,859 problem in the first place 951 00:39:48,240 --> 00:39:51,960 so I'm always a bit cognizant of that 952 00:39:49,859 --> 00:39:53,880 what are you out of blocks I reckon that 953 00:39:51,960 --> 00:39:57,079 more experience blocks is that true or 954 00:39:53,880 --> 00:39:57,079 is it worth having all this 955 00:39:57,900 --> 00:40:02,640 probably not worth having it yeah 956 00:40:01,440 --> 00:40:04,680 yeah 957 00:40:02,640 --> 00:40:06,300 do you have an opinion on the idea of 958 00:40:04,680 --> 00:40:08,220 putting the false positives in the code 959 00:40:06,300 --> 00:40:10,320 comment do you think that's worth the 960 00:40:08,220 --> 00:40:11,740 noise 961 00:40:10,320 --> 00:40:14,950 probably not was the comment before 962 00:40:11,740 --> 00:40:14,950 [Music] 963 00:40:16,619 --> 00:40:23,099 yeah 964 00:40:18,200 --> 00:40:26,040 yeah so I used I put it above the 965 00:40:23,099 --> 00:40:27,660 like just here 966 00:40:26,040 --> 00:40:28,800 and I just wrote 967 00:40:27,660 --> 00:40:30,480 um 968 00:40:28,800 --> 00:40:31,680 muted in false positives and put a 969 00:40:30,480 --> 00:40:33,300 couple lines then I use the same form 970 00:40:31,680 --> 00:40:34,680 everywhere so it was easy once you've 971 00:40:33,300 --> 00:40:36,800 seen one you didn't need to keep reading 972 00:40:34,680 --> 00:40:36,800 them 973 00:40:39,900 --> 00:40:45,540 suppress it yeah yes yeah so in in the I 974 00:40:43,920 --> 00:40:47,579 the the comment was there needs to be a 975 00:40:45,540 --> 00:40:49,200 way of suppressing that in the test and 976 00:40:47,579 --> 00:40:51,420 I went into the refund talked to the guy 977 00:40:49,200 --> 00:40:52,800 and it shouldn't be that difficult but I 978 00:40:51,420 --> 00:40:54,240 only put two hours work into it I 979 00:40:52,800 --> 00:40:56,099 couldn't get it done in two hours and he 980 00:40:54,240 --> 00:40:57,599 doesn't want to do it yet so yeah it 981 00:40:56,099 --> 00:41:00,000 shouldn't be that hard 982 00:40:57,599 --> 00:41:01,380 you have to um it's like reasonably 983 00:41:00,000 --> 00:41:04,500 Advanced for us you have to mess around 984 00:41:01,380 --> 00:41:08,400 with sin and proc macros and things 985 00:41:04,500 --> 00:41:09,960 but yeah once you're um passed rust 101 986 00:41:08,400 --> 00:41:12,720 then you can mess around with Brock 987 00:41:09,960 --> 00:41:14,280 macros and lose a lot of days 988 00:41:12,720 --> 00:41:16,579 but yeah it would be really cool if we 989 00:41:14,280 --> 00:41:16,579 had that 990 00:41:17,099 --> 00:41:21,140 awesome thanks for your time thanks for 991 00:41:19,020 --> 00:41:21,140 coming 992 00:41:22,000 --> 00:41:25,449 [Applause]