1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:16,000 --> 00:00:17,840 Um, 4 00:00:17,840 --> 00:00:19,600 it's been a great conference so far for 5 00:00:19,600 --> 00:00:21,439 the last day. Um, I'd like now to 6 00:00:21,439 --> 00:00:26,760 introduce Sam to come on stage. 7 00:00:26,855 --> 00:00:28,875 [applause] 8 00:00:32,079 --> 00:00:34,320 Okay. So, yeah, I'm Sam Bishop. I'm a 9 00:00:34,320 --> 00:00:35,760 freelance consultant, which means you 10 00:00:35,760 --> 00:00:37,280 can hire me, and you can do that by 11 00:00:37,280 --> 00:00:38,719 checking out my contact info on the 12 00:00:38,719 --> 00:00:40,399 website. But, uh, other than the crazy 13 00:00:40,399 --> 00:00:42,960 side projects I have, I tend to end up 14 00:00:42,960 --> 00:00:45,200 having to do a lot of testing. 15 00:00:45,200 --> 00:00:49,320 So, I'm I'm going to start real 16 00:00:52,719 --> 00:00:53,920 shenanigans that have been happening 17 00:00:53,920 --> 00:00:55,440 over the last week. you will get to see 18 00:00:55,440 --> 00:00:56,985 some real good highlights of those. 19 00:00:56,985 --> 00:00:57,760 [snorts] 20 00:00:57,760 --> 00:01:00,399 So, what a test does is it's there to 21 00:01:00,399 --> 00:01:02,320 check that our code is working. Like in 22 00:01:02,320 --> 00:01:04,159 this case, they're really simple tests. 23 00:01:04,159 --> 00:01:05,680 We're just testing that the encoding and 24 00:01:05,680 --> 00:01:07,119 the decoding works. I'm sure that 25 00:01:07,119 --> 00:01:08,159 everyone's written a couple of basic 26 00:01:08,159 --> 00:01:10,159 tests like this at some point, and these 27 00:01:10,159 --> 00:01:13,680 all pass. But eventually, you're going 28 00:01:13,680 --> 00:01:16,560 to find that the tests don't pass. And 29 00:01:16,560 --> 00:01:17,759 then there's the worse and more 30 00:01:17,759 --> 00:01:19,920 insidious problem of what happens when 31 00:01:19,920 --> 00:01:22,000 the tests pass, but there's still a bug. 32 00:01:22,000 --> 00:01:23,840 it's crept through the gaps somewhere. 33 00:01:23,840 --> 00:01:25,520 [snorts] Because at the end of the day, 34 00:01:25,520 --> 00:01:27,600 a thorough test suite is most of the 35 00:01:27,600 --> 00:01:28,720 time going to be everything you've 36 00:01:28,720 --> 00:01:31,119 managed to think of, not everything that 37 00:01:31,119 --> 00:01:33,520 could possibly go wrong. That's the 38 00:01:33,520 --> 00:01:35,520 assumption that lets things creep in. 39 00:01:35,520 --> 00:01:37,280 Edge cases will come around the tests 40 00:01:37,280 --> 00:01:39,360 you've handwritten. And the bugs are 41 00:01:39,360 --> 00:01:41,360 always going to be in the test cases you 42 00:01:41,360 --> 00:01:43,280 haven't written [snorts] because it is 43 00:01:43,280 --> 00:01:44,720 fundamentally in the last place you 44 00:01:44,720 --> 00:01:46,240 look. 45 00:01:46,240 --> 00:01:47,920 Because program testing can be used to 46 00:01:47,920 --> 00:01:49,600 show the presence of bugs, but never to 47 00:01:49,600 --> 00:01:51,600 show their absence. you can't prove the 48 00:01:51,600 --> 00:01:54,960 negative. So, one of the ways to try and 49 00:01:54,960 --> 00:01:56,960 avoid this being a problem is property 50 00:01:56,960 --> 00:01:59,200 based testing. So, what is property 51 00:01:59,200 --> 00:02:01,040 based testing? Well, the basic idea is 52 00:02:01,040 --> 00:02:03,119 that instead of choosing a specific set 53 00:02:03,119 --> 00:02:05,200 of inputs and asserting that you're 54 00:02:05,200 --> 00:02:07,280 going to have a specific output, what 55 00:02:07,280 --> 00:02:08,800 you do is you describe the connections 56 00:02:08,800 --> 00:02:10,479 between them. You you assert that 57 00:02:10,479 --> 00:02:11,920 something is going to be always true for 58 00:02:11,920 --> 00:02:14,879 a set of inputs. You tell a library like 59 00:02:14,879 --> 00:02:17,520 hypothesis, I want to put these kinds of 60 00:02:17,520 --> 00:02:19,760 inputs into my function. If it's 61 00:02:19,760 --> 00:02:21,280 something that does like a math library, 62 00:02:21,280 --> 00:02:23,440 maybe you say, "Give me integers." And 63 00:02:23,440 --> 00:02:24,879 then it will run that for those 64 00:02:24,879 --> 00:02:27,200 integers. And the framework ideally is 65 00:02:27,200 --> 00:02:28,720 going to work through intelligently 66 00:02:28,720 --> 00:02:30,319 trying to find cases that it thinks 67 00:02:30,319 --> 00:02:32,160 might break that. In the case of 68 00:02:32,160 --> 00:02:34,160 hypothesis, when you don't tell it to do 69 00:02:34,160 --> 00:02:35,840 anything differently, it's going to try 70 00:02:35,840 --> 00:02:37,440 100 things every time you give any 71 00:02:37,440 --> 00:02:39,680 function a decorator to try and run as a 72 00:02:39,680 --> 00:02:41,120 test. So, it's giving you a 100 test 73 00:02:41,120 --> 00:02:42,800 cases for every one you write by 74 00:02:42,800 --> 00:02:44,080 default. 75 00:02:44,080 --> 00:02:45,680 And then when it finds something that 76 00:02:45,680 --> 00:02:47,519 breaks that test, it's going to try and 77 00:02:47,519 --> 00:02:49,200 tell you the simplest thing that broke 78 00:02:49,200 --> 00:02:50,879 the test. And there's going to be more 79 00:02:50,879 --> 00:02:52,319 about that later on, so don't worry too 80 00:02:52,319 --> 00:02:53,440 much about how it knows what the 81 00:02:53,440 --> 00:02:56,000 simplest thing is. And on top of that, a 82 00:02:56,000 --> 00:02:58,400 good framework like Hypothesis is knows 83 00:02:58,400 --> 00:03:00,160 how to find the nasty edge cases that 84 00:03:00,160 --> 00:03:02,319 are likely to break stuff. It's taken 85 00:03:02,319 --> 00:03:04,560 advantage of years of work trying to 86 00:03:04,560 --> 00:03:06,319 find what is most likely to cause an 87 00:03:06,319 --> 00:03:09,840 issue. And how it's doing that is by an 88 00:03:09,840 --> 00:03:12,159 adversarial generator. It knows certain 89 00:03:12,159 --> 00:03:14,000 classes of bugs come from certain kinds 90 00:03:14,000 --> 00:03:15,840 of inputs. It knows that there are 91 00:03:15,840 --> 00:03:17,599 usually things like unicode decoding 92 00:03:17,599 --> 00:03:19,360 errors in the higher code spaces. It 93 00:03:19,360 --> 00:03:20,640 knows that maybe your code doesn't 94 00:03:20,640 --> 00:03:22,159 handle that extra bite at the end of 95 00:03:22,159 --> 00:03:24,080 that particular kind of character input. 96 00:03:24,080 --> 00:03:26,319 It knows that maybe if you use positive 97 00:03:26,319 --> 00:03:27,920 or negative infinity on a float that 98 00:03:27,920 --> 00:03:29,360 that might give a different kind of an 99 00:03:29,360 --> 00:03:31,200 error. It knows to check that negative 100 00:03:31,200 --> 00:03:33,040 zero you often forget about as opposed 101 00:03:33,040 --> 00:03:34,799 to just the zero that most people 102 00:03:34,799 --> 00:03:37,519 remember. And as well as things like 103 00:03:37,519 --> 00:03:39,120 gigantic lists that might overflow 104 00:03:39,120 --> 00:03:42,080 memory or stuff like imaginary numbers 105 00:03:42,080 --> 00:03:43,840 that tend to not be somewhere in the 106 00:03:43,840 --> 00:03:45,840 middle of your code until well someone 107 00:03:45,840 --> 00:03:47,440 accidentally uses a function the wrong 108 00:03:47,440 --> 00:03:49,440 way and now there's an imaginary number 109 00:03:49,440 --> 00:03:51,760 crept in somewhere. It's taking your 110 00:03:51,760 --> 00:03:53,599 imagination for all of those obscure 111 00:03:53,599 --> 00:03:56,560 ways your code can break out of the 112 00:03:56,560 --> 00:03:58,159 problem. You are no longer responsible 113 00:03:58,159 --> 00:03:59,760 for thinking of all the ways your code 114 00:03:59,760 --> 00:04:02,720 can break. That's its job now. It's kind 115 00:04:02,720 --> 00:04:04,400 of like fuzz testing if you've ever seen 116 00:04:04,400 --> 00:04:06,560 that before. But it's not exactly the 117 00:04:06,560 --> 00:04:08,799 same. They both sort of get a general 118 00:04:08,799 --> 00:04:10,640 improvement in the same area, kind of 119 00:04:10,640 --> 00:04:12,319 parallel directions, but very different 120 00:04:12,319 --> 00:04:14,080 focuses on how they achieve those 121 00:04:14,080 --> 00:04:17,359 results. And and where it shines is in 122 00:04:17,359 --> 00:04:18,880 things like parsing and serializing 123 00:04:18,880 --> 00:04:21,040 where you've got a free round trip. It's 124 00:04:21,040 --> 00:04:22,720 fantastic for validation and business 125 00:04:22,720 --> 00:04:24,000 rules where you want to make sure that 126 00:04:24,000 --> 00:04:25,360 all the various conditions and 127 00:04:25,360 --> 00:04:27,440 invariance and special things that 128 00:04:27,440 --> 00:04:29,199 you've got to check on various report 129 00:04:29,199 --> 00:04:31,759 formats and data classes is able to be 130 00:04:31,759 --> 00:04:33,840 ticked off. you know, if it's been sent 131 00:04:33,840 --> 00:04:35,840 on a Tuesday, do they always have the 132 00:04:35,840 --> 00:04:37,040 specific thing and there's no way for 133 00:04:37,040 --> 00:04:39,199 them to break it? And anytime you've got 134 00:04:39,199 --> 00:04:40,720 a pure transformer, if you've just got a 135 00:04:40,720 --> 00:04:42,320 function or a method that's just in and 136 00:04:42,320 --> 00:04:44,720 out, pure and simple, it's fantastic for 137 00:04:44,720 --> 00:04:47,199 that. And one of the big ones now that 138 00:04:47,199 --> 00:04:48,800 most people might not start thinking of 139 00:04:48,800 --> 00:04:50,320 until it's sort of pointed out is that 140 00:04:50,320 --> 00:04:51,919 if you've got a big block of code that 141 00:04:51,919 --> 00:04:53,440 you didn't write and don't necessarily 142 00:04:53,440 --> 00:04:55,759 fully understand how it works, but you 143 00:04:55,759 --> 00:04:58,320 know what it supposed to take in and 144 00:04:58,320 --> 00:05:00,560 what's supposed to come out, like say 145 00:05:00,560 --> 00:05:02,320 something that Claude or Codeex might 146 00:05:02,320 --> 00:05:04,560 generate. It's good for that because you 147 00:05:04,560 --> 00:05:06,000 know what that function is supposed to 148 00:05:06,000 --> 00:05:07,759 do and you don't necessarily have to 149 00:05:07,759 --> 00:05:10,080 know as much about how it does it as 150 00:05:10,080 --> 00:05:12,560 long as you can very thoroughly test 151 00:05:12,560 --> 00:05:15,680 that it does it correctly. 152 00:05:15,680 --> 00:05:17,680 So before we get into how it actually 153 00:05:17,680 --> 00:05:21,360 works, the quick rundown here 154 00:05:21,360 --> 00:05:24,160 regular example test 155 00:05:24,160 --> 00:05:26,560 inputs and outputs. With a property 156 00:05:26,560 --> 00:05:29,199 based test, you're defining a rule that 157 00:05:29,199 --> 00:05:31,280 for these kinds of inputs, it will 158 00:05:31,280 --> 00:05:34,000 always do this. For an example based 159 00:05:34,000 --> 00:05:36,479 test, all you're going to get covered is 160 00:05:36,479 --> 00:05:38,479 everything you think of writing. For a 161 00:05:38,479 --> 00:05:40,400 property based test, it's going to be 162 00:05:40,400 --> 00:05:42,160 everything it can invent based on what 163 00:05:42,160 --> 00:05:44,639 you give it. For an example based test, 164 00:05:44,639 --> 00:05:46,320 anytime you get a failure, it's just 165 00:05:46,320 --> 00:05:48,320 telling you this one thing broke. 166 00:05:48,320 --> 00:05:49,840 Whereas for a property based test, it's 167 00:05:49,840 --> 00:05:50,880 going to be telling you something like 168 00:05:50,880 --> 00:05:52,560 these kinds of numbers broke or this 169 00:05:52,560 --> 00:05:54,320 size of list or this order of things. 170 00:05:54,320 --> 00:05:59,080 It's finding values that broke the test. 171 00:06:04,880 --> 00:06:07,039 doing a negative test, but it's just 172 00:06:07,039 --> 00:06:09,360 saying these inputs do this thing. It's 173 00:06:09,360 --> 00:06:11,440 almost documentation. That's why there's 174 00:06:11,440 --> 00:06:13,600 dock tests, but for a property based 175 00:06:13,600 --> 00:06:15,974 test, it's good at finding the unknown. 176 00:06:15,974 --> 00:06:17,120 [snorts] 177 00:06:17,120 --> 00:06:21,280 So, what is it actually giving us? 178 00:06:21,280 --> 00:06:23,520 Well, it's giving us reproducibility 179 00:06:23,520 --> 00:06:25,039 because it's going to remember those 180 00:06:25,039 --> 00:06:27,360 failures. It's generating a bunch of 181 00:06:27,360 --> 00:06:28,960 stuff. And you might think, well, how do 182 00:06:28,960 --> 00:06:30,560 I know it's going to generate the same 183 00:06:30,560 --> 00:06:33,039 thing next time? Is it going to forget 184 00:06:33,039 --> 00:06:36,000 to check that bug? No, it remembers. So, 185 00:06:36,000 --> 00:06:37,600 it actually keeps a database of past 186 00:06:37,600 --> 00:06:38,880 failures and those are the first things 187 00:06:38,880 --> 00:06:40,720 it checks the next time. [snorts] So, if 188 00:06:40,720 --> 00:06:43,120 you've got these done once, it will find 189 00:06:43,120 --> 00:06:45,199 them very fast again until you fix that 190 00:06:45,199 --> 00:06:47,440 bug. And on top of that, you can do 191 00:06:47,440 --> 00:06:49,280 things like use a fixed seed. So, if 192 00:06:49,280 --> 00:06:50,720 you've got a weird bug that only happens 193 00:06:50,720 --> 00:06:52,240 on your machine and you can't quite work 194 00:06:52,240 --> 00:06:54,560 out why it's not happening in CI, you 195 00:06:54,560 --> 00:06:57,120 can take that generation seed, put that 196 00:06:57,120 --> 00:06:59,039 seed into the CI or hand it to another 197 00:06:59,039 --> 00:07:00,960 developer to help you, and you can 198 00:07:00,960 --> 00:07:02,639 actually try to reproduce much more 199 00:07:02,639 --> 00:07:04,960 accurately than you would in other kinds 200 00:07:04,960 --> 00:07:07,759 of circumstances like that. And on top 201 00:07:07,759 --> 00:07:08,880 of that, you can do things like giving 202 00:07:08,880 --> 00:07:11,199 the CI a longer time span. One of the 203 00:07:11,199 --> 00:07:13,360 things I've seen done is before a 204 00:07:13,360 --> 00:07:15,919 release, they'll do a deep run. And so 205 00:07:15,919 --> 00:07:17,280 instead of having like the regular 206 00:07:17,280 --> 00:07:19,520 default of 100, they might say, "Yeah, 207 00:07:19,520 --> 00:07:21,840 go do 10,000 and let the CI run 208 00:07:21,840 --> 00:07:24,240 overnight and really thoroughly execute 209 00:07:24,240 --> 00:07:25,919 the code." And it [snorts] gives you the 210 00:07:25,919 --> 00:07:28,479 power to do that. And then what it's 211 00:07:28,479 --> 00:07:29,759 giving you, we saw a little of this 212 00:07:29,759 --> 00:07:31,520 earlier, it gives you strategies, 213 00:07:31,520 --> 00:07:33,919 strategies to decide what is my function 214 00:07:33,919 --> 00:07:36,319 or method supposed to take. It's got a 215 00:07:36,319 --> 00:07:38,000 whole bunch of these built in for things 216 00:07:38,000 --> 00:07:39,840 like integers and text and floats and 217 00:07:39,840 --> 00:07:42,000 datetimes and more complicated stuff 218 00:07:42,000 --> 00:07:44,720 like generating an email address. So, do 219 00:07:44,720 --> 00:07:46,000 do you think that your email address 220 00:07:46,000 --> 00:07:47,919 function correctly handles the fact that 221 00:07:47,919 --> 00:07:49,840 you can have an emoji in certain places 222 00:07:49,840 --> 00:07:52,479 but not other places or an emoji for the 223 00:07:52,479 --> 00:07:54,879 domain name part? There are a lot of 224 00:07:54,879 --> 00:07:56,639 ways that you might not think and that's 225 00:07:56,639 --> 00:07:58,479 what it's good at, including things like 226 00:07:58,479 --> 00:08:01,039 dictionaries with strange keys. Like 227 00:08:01,039 --> 00:08:02,879 there's no reason you can't have a 228 00:08:02,879 --> 00:08:05,440 non-printable space character as the key 229 00:08:05,440 --> 00:08:07,325 in a dictionary 230 00:08:07,325 --> 00:08:07,919 [snorts] 231 00:08:07,919 --> 00:08:09,759 and things like picking from various 232 00:08:09,759 --> 00:08:11,680 options and generating tupils and 233 00:08:11,680 --> 00:08:14,000 one-offs. But you can also use it to 234 00:08:14,000 --> 00:08:15,440 build strategies for your own classes. 235 00:08:15,440 --> 00:08:16,879 So if you've got like a data class like 236 00:08:16,879 --> 00:08:19,199 we have here, you can ask hypothesis, 237 00:08:19,199 --> 00:08:20,879 hey, I might want to generate this a lot 238 00:08:20,879 --> 00:08:23,520 of times. I would like a shortcut and 239 00:08:23,520 --> 00:08:24,960 you can build that shortcut and make it 240 00:08:24,960 --> 00:08:26,720 a strategy and then use it the same as 241 00:08:26,720 --> 00:08:28,800 everything else. You can use it and fold 242 00:08:28,800 --> 00:08:30,479 them together. You can have an optional 243 00:08:30,479 --> 00:08:32,560 input. It can either be blank or an item 244 00:08:32,560 --> 00:08:34,399 or it could be pick an item from these 245 00:08:34,399 --> 00:08:36,560 three or it could be pick an item or a 246 00:08:36,560 --> 00:08:38,399 different thing. And it lets you combine 247 00:08:38,399 --> 00:08:41,200 these things to get your inputs. And so 248 00:08:41,200 --> 00:08:43,039 by saying, "Hey, build me an item," by 249 00:08:43,039 --> 00:08:45,200 taking the name and the price as those 250 00:08:45,200 --> 00:08:46,800 two other strategies, we just get one 251 00:08:46,800 --> 00:08:49,246 out the other side as a new strategy. 252 00:08:49,246 --> 00:08:50,959 [snorts] And then when we go to actually 253 00:08:50,959 --> 00:08:52,720 put them into our functions, it gives us 254 00:08:52,720 --> 00:08:54,080 a bunch of decorators. It's a very 255 00:08:54,080 --> 00:08:56,560 decorator heavy framework, but that 256 00:08:56,560 --> 00:08:58,000 makes it quite easy to read when you're 257 00:08:58,000 --> 00:08:59,839 going through code. So you get a given 258 00:08:59,839 --> 00:09:02,720 set of inputs, hand those to your tests, 259 00:09:02,720 --> 00:09:04,880 and then you can specifically say, "Hey, 260 00:09:04,880 --> 00:09:06,399 I want to make sure that even though 261 00:09:06,399 --> 00:09:08,640 you're going to randomly generate stuff, 262 00:09:08,640 --> 00:09:10,720 do make sure you check that one. that 263 00:09:10,720 --> 00:09:12,320 specific one and that's when you can 264 00:09:12,320 --> 00:09:14,399 give it an example and then sometimes 265 00:09:14,399 --> 00:09:15,519 you may actually want to change the 266 00:09:15,519 --> 00:09:17,519 settings like that default of 100 that's 267 00:09:17,519 --> 00:09:19,440 fine most of the time but [snorts] 268 00:09:19,440 --> 00:09:21,839 sometimes the code might be a bit slow 269 00:09:21,839 --> 00:09:24,399 and that 100 can be like 6 minutes and 270 00:09:24,399 --> 00:09:25,920 you're like h I don't really feel like 271 00:09:25,920 --> 00:09:27,360 waiting 6 minutes every time my test 272 00:09:27,360 --> 00:09:29,920 cases run maybe we only want to do 10 273 00:09:29,920 --> 00:09:31,360 and there's a little settings decorator 274 00:09:31,360 --> 00:09:33,279 for you to specifically tell certain 275 00:09:33,279 --> 00:09:34,720 functions to do things a little 276 00:09:34,720 --> 00:09:36,240 differently and then it's got a couple 277 00:09:36,240 --> 00:09:38,480 of helpers for inside your test function 278 00:09:38,480 --> 00:09:40,880 leaving notes for yourself or assuming 279 00:09:40,880 --> 00:09:42,959 that certain values need to be skipped. 280 00:09:42,959 --> 00:09:44,959 We'll talk more about those later. And 281 00:09:44,959 --> 00:09:46,480 and putting them together looks a bit 282 00:09:46,480 --> 00:09:48,160 like that. You know, we've got a a 283 00:09:48,160 --> 00:09:49,600 function here, two of them, in fact. 284 00:09:49,600 --> 00:09:52,000 We've got both of them taking one's a 285 00:09:52,000 --> 00:09:53,360 list of integers, and we've got a 286 00:09:53,360 --> 00:09:55,200 specific example there. And then the 287 00:09:55,200 --> 00:09:56,720 other one is just like, yeah, just take 288 00:09:56,720 --> 00:09:58,480 just take integers. [snorts] And and 289 00:09:58,480 --> 00:10:00,480 there's an obvious bug in this one. It's 290 00:10:00,480 --> 00:10:03,200 going to divide by zero. And uh it 291 00:10:03,200 --> 00:10:05,440 helpfully tells us that there note there 292 00:10:05,440 --> 00:10:07,040 pointing out that yes, dividing by zero 293 00:10:07,040 --> 00:10:09,680 was doomed to fail. But you can see that 294 00:10:09,680 --> 00:10:11,040 it's quite easy to just put it around 295 00:10:11,040 --> 00:10:13,120 your regular code. It's not particularly 296 00:10:13,120 --> 00:10:15,120 obscure. It's just a it looks a lot like 297 00:10:15,120 --> 00:10:16,560 Piest when you start to get familiar 298 00:10:16,560 --> 00:10:18,959 with it. And then there's the bit that 299 00:10:18,959 --> 00:10:20,079 is a little trickier to get your head 300 00:10:20,079 --> 00:10:21,760 around which is the rules and invariants 301 00:10:21,760 --> 00:10:24,560 where you can do state machine testing. 302 00:10:24,560 --> 00:10:26,720 So you can define a rule for a function 303 00:10:26,720 --> 00:10:29,040 or a method and then you can define an 304 00:10:29,040 --> 00:10:31,040 invariant and then hypothesis will 305 00:10:31,040 --> 00:10:33,839 randomly try your rules until it breaks 306 00:10:33,839 --> 00:10:35,680 your invariant. That will make more 307 00:10:35,680 --> 00:10:37,120 sense when we sort of walk through it 308 00:10:37,120 --> 00:10:39,120 later, but I'll just sort of show the 309 00:10:39,120 --> 00:10:41,200 simple example here of how it's doing a 310 00:10:41,200 --> 00:10:43,360 bad job sorting a list because it goes 311 00:10:43,360 --> 00:10:45,760 to try and sort the list and well, yeah, 312 00:10:45,760 --> 00:10:47,440 it doesn't do it. It it actually doesn't 313 00:10:47,440 --> 00:10:49,440 sort the list, 314 00:10:49,440 --> 00:10:51,760 but what you saw there is an example of 315 00:10:51,760 --> 00:10:53,920 shrinking because Hypothesis would have 316 00:10:53,920 --> 00:10:55,839 tried a few examples longer than the 317 00:10:55,839 --> 00:10:57,920 list it just showed us. It would have 318 00:10:57,920 --> 00:11:00,480 tested maybe six or seven by the time it 319 00:11:00,480 --> 00:11:02,560 actually got through that thing. But by 320 00:11:02,560 --> 00:11:03,600 the time it got through that thing, it 321 00:11:03,600 --> 00:11:05,360 walked backwards and came out with a 322 00:11:05,360 --> 00:11:07,600 shorter list of options to get us there. 323 00:11:07,600 --> 00:11:09,360 It came out with two steps instead of 324 00:11:09,360 --> 00:11:11,760 however many it started with. It shrank 325 00:11:11,760 --> 00:11:13,760 the list down. So it actually has a 326 00:11:13,760 --> 00:11:15,360 bunch of heruristics and algorithms in 327 00:11:15,360 --> 00:11:17,680 it to try and walk down the difficulty 328 00:11:17,680 --> 00:11:20,560 and complexity of what it produces until 329 00:11:20,560 --> 00:11:22,399 it gives you what it thinks is the 330 00:11:22,399 --> 00:11:25,120 simplest example of the value that broke 331 00:11:25,120 --> 00:11:27,440 your code or the simplest set of 332 00:11:27,440 --> 00:11:30,000 operations that produced an invalid 333 00:11:30,000 --> 00:11:33,040 state. So for something like a like a 334 00:11:33,040 --> 00:11:35,040 large class doing business logic that 335 00:11:35,040 --> 00:11:37,200 might have maybe eight or nine steps or 336 00:11:37,200 --> 00:11:39,120 operations talking to systems or 337 00:11:39,120 --> 00:11:40,720 something like that, it's going to 338 00:11:40,720 --> 00:11:43,120 produce a smaller sequence of operations 339 00:11:43,120 --> 00:11:45,279 that you can much more easily test by 340 00:11:45,279 --> 00:11:47,200 hand and pull up in a debugger and 341 00:11:47,200 --> 00:11:49,760 actually try to reason with just no, you 342 00:11:49,760 --> 00:11:51,279 have to do exactly these steps to break 343 00:11:51,279 --> 00:11:53,279 it. It has huristics to make that more 344 00:11:53,279 --> 00:11:56,720 friendly for you. 345 00:11:56,720 --> 00:11:58,560 So let's walk through an example of how 346 00:11:58,560 --> 00:12:01,760 you actually use it. 347 00:12:03,128 --> 00:12:05,148 [laughter and snorts] 348 00:12:05,279 --> 00:12:07,040 Let's take a basic function. We've got 349 00:12:07,040 --> 00:12:08,720 our lovely little split bill here. It's 350 00:12:08,720 --> 00:12:10,320 going to take some sense and it's going 351 00:12:10,320 --> 00:12:12,399 to take some people and then we'll just 352 00:12:12,399 --> 00:12:13,839 write some tests for it, right? Two 353 00:12:13,839 --> 00:12:16,480 tests. Nice and easy. Super simple. 354 00:12:16,480 --> 00:12:18,000 We've even gone and tested a couple of 355 00:12:18,000 --> 00:12:20,560 combinations there. And look, great. 356 00:12:20,560 --> 00:12:21,839 That should be enough, right? Well, 357 00:12:21,839 --> 00:12:25,680 okay, two tests, two tests pass. Great. 358 00:12:25,680 --> 00:12:27,519 Well, let's run the same thing through 359 00:12:27,519 --> 00:12:29,040 with hypothesis. We've got our 360 00:12:29,040 --> 00:12:31,200 decorator. We've pulled in the imports. 361 00:12:31,200 --> 00:12:32,959 We've put the decorator on the function. 362 00:12:32,959 --> 00:12:36,000 We've said, "Okay, uh, I'm going to take 363 00:12:36,000 --> 00:12:37,920 any integer over zero because I don't 364 00:12:37,920 --> 00:12:39,200 really care about splitting a bill that 365 00:12:39,200 --> 00:12:41,519 is 0 cents. Yeah, I'll split my 0 cent 366 00:12:41,519 --> 00:12:44,000 bill with you. Thanks any day." But then 367 00:12:44,000 --> 00:12:45,440 you're going to take between say 1 and 368 00:12:45,440 --> 00:12:47,360 20 people. Maybe you just don't care 369 00:12:47,360 --> 00:12:49,120 past 20 people. But for whatever reason, 370 00:12:49,120 --> 00:12:50,480 those are the choices we've got for 371 00:12:50,480 --> 00:12:52,000 this. And then we're going to go through 372 00:12:52,000 --> 00:12:52,880 and we're going to run these 373 00:12:52,880 --> 00:12:54,720 combinations. And it's going to see if 374 00:12:54,720 --> 00:12:56,480 it can find anything here that breaks 375 00:12:56,480 --> 00:12:58,880 this. and it immediately does because it 376 00:12:58,880 --> 00:13:00,160 turns out that if you try to split one 377 00:13:00,160 --> 00:13:02,079 cent between two people, you've got a 378 00:13:02,079 --> 00:13:03,920 problem. You can't really split one cent 379 00:13:03,920 --> 00:13:06,639 conveniently. So there it's actually 380 00:13:06,639 --> 00:13:08,720 saying, hey, this is the specific one 381 00:13:08,720 --> 00:13:12,079 that it found. 1 cent, two people, no. 382 00:13:12,079 --> 00:13:14,720 Now we can pin that as an example here, 383 00:13:14,720 --> 00:13:16,560 saying every time you run this test, 384 00:13:16,560 --> 00:13:17,839 make sure that you actually check that 385 00:13:17,839 --> 00:13:20,639 one for future. And we can also easily 386 00:13:20,639 --> 00:13:21,920 fix the bug there by actually just 387 00:13:21,920 --> 00:13:23,600 putting in the div mod instead of doing 388 00:13:23,600 --> 00:13:26,240 the regular division. And if we do both 389 00:13:26,240 --> 00:13:28,320 of those things, what we end up doing is 390 00:13:28,320 --> 00:13:30,320 we get a third test that actually does a 391 00:13:30,320 --> 00:13:32,000 better job. And now we're much more 392 00:13:32,000 --> 00:13:33,440 confident that this will work all the 393 00:13:33,440 --> 00:13:35,120 time. We didn't have to take away the 394 00:13:35,120 --> 00:13:36,959 tests we had already written. We simply 395 00:13:36,959 --> 00:13:39,120 added more rich testing to what we had 396 00:13:39,120 --> 00:13:41,839 already done. Now that was a bit of an 397 00:13:41,839 --> 00:13:44,639 overly contrived example, but I think if 398 00:13:44,639 --> 00:13:45,920 we walk through a slightly better one, 399 00:13:45,920 --> 00:13:48,560 that'll make more sense. So we've got a 400 00:13:48,560 --> 00:13:49,760 class here and a function that's going 401 00:13:49,760 --> 00:13:51,279 to use it. We've got an item, and this 402 00:13:51,279 --> 00:13:52,959 is a function that's going to total the 403 00:13:52,959 --> 00:13:54,959 value of a bunch of items. So, it takes 404 00:13:54,959 --> 00:13:57,680 a list of items. So, we're going to make 405 00:13:57,680 --> 00:13:59,040 a strategy to build ourselves some 406 00:13:59,040 --> 00:14:01,040 items. We saw this one earlier. It's 407 00:14:01,040 --> 00:14:02,399 going to take a name, which has to have 408 00:14:02,399 --> 00:14:04,240 at least a single character in it. And 409 00:14:04,240 --> 00:14:06,480 it's going to have a price that is at 410 00:14:06,480 --> 00:14:09,120 least zero and not negative. And then 411 00:14:09,120 --> 00:14:11,279 we're going to have a function that goes 412 00:14:11,279 --> 00:14:13,199 make sure that the basket of items, that 413 00:14:13,199 --> 00:14:15,199 list of items we're going to work with 414 00:14:15,199 --> 00:14:16,959 is the same forwards and backwards. That 415 00:14:16,959 --> 00:14:18,399 somehow we've not done anything strange 416 00:14:18,399 --> 00:14:20,560 in our class that might make it add up 417 00:14:20,560 --> 00:14:22,560 differently forward or backward. 418 00:14:22,560 --> 00:14:24,880 wouldn't think it's going to fail and no 419 00:14:24,880 --> 00:14:26,959 it doesn't. But it's a good example of 420 00:14:26,959 --> 00:14:28,320 something that you might need to do if 421 00:14:28,320 --> 00:14:30,000 it's a more complicated thing where if 422 00:14:30,000 --> 00:14:32,480 there is a way things stack in terms of 423 00:14:32,480 --> 00:14:35,519 discount codes or vouchers that subtract 424 00:14:35,519 --> 00:14:36,959 from certain products and things like 425 00:14:36,959 --> 00:14:39,199 that. If there's more complexity hiding 426 00:14:39,199 --> 00:14:42,240 in these things or you add them in later 427 00:14:42,240 --> 00:14:44,000 and didn't think about adding extra 428 00:14:44,000 --> 00:14:46,000 tests for them, this is where that can 429 00:14:46,000 --> 00:14:49,120 catch that kind of an increasing little 430 00:14:49,120 --> 00:14:50,720 edge case that creeps in around the 431 00:14:50,720 --> 00:14:53,360 side. Now, let's add that cart we were 432 00:14:53,360 --> 00:14:54,880 just kind of talking about here. So, 433 00:14:54,880 --> 00:14:56,399 we'll put the items in a cart. We have a 434 00:14:56,399 --> 00:14:58,800 new class. The class is going to take a 435 00:14:58,800 --> 00:15:00,399 list of items. I'm going to point out 436 00:15:00,399 --> 00:15:01,920 the obvious bug there because uh 437 00:15:01,920 --> 00:15:03,360 otherwise it might not be easy to see 438 00:15:03,360 --> 00:15:04,959 straight away. But, uh we're not 439 00:15:04,959 --> 00:15:06,320 actually checking if the item is in the 440 00:15:06,320 --> 00:15:07,839 cart. 441 00:15:07,839 --> 00:15:10,079 But what we're going to do is we're 442 00:15:10,079 --> 00:15:11,040 going to write one of those more 443 00:15:11,040 --> 00:15:13,440 complicated tests I mentioned earlier. 444 00:15:13,440 --> 00:15:15,360 And we're going to make a rule. We're 445 00:15:15,360 --> 00:15:16,880 going to say we can add things to the 446 00:15:16,880 --> 00:15:19,199 cart and those things can be items. 447 00:15:19,199 --> 00:15:21,360 We've already wrote the item creator 448 00:15:21,360 --> 00:15:23,040 earlier. So, we're literally just asking 449 00:15:23,040 --> 00:15:24,800 hypothesis, hey, that items thing we 450 00:15:24,800 --> 00:15:26,639 made earlier, make that your input. 451 00:15:26,639 --> 00:15:28,560 That's the strategy. And then we're 452 00:15:28,560 --> 00:15:29,760 going to say the same thing. You can 453 00:15:29,760 --> 00:15:31,760 remove an item from the cart. And then 454 00:15:31,760 --> 00:15:33,360 we're going to write our invariant that 455 00:15:33,360 --> 00:15:35,440 the cart should never be any less than 456 00:15:35,440 --> 00:15:37,440 zero. Sensible enough. You don't want to 457 00:15:37,440 --> 00:15:39,839 give the customer negative money. And 458 00:15:39,839 --> 00:15:42,880 then we'll run our test. And so, yep, it 459 00:15:42,880 --> 00:15:44,720 found one that broke it. Zooming in a 460 00:15:44,720 --> 00:15:46,000 bit so you can see it a bit easier. 461 00:15:46,000 --> 00:15:47,760 Yeah, it it very quickly found that it 462 00:15:47,760 --> 00:15:49,279 could take an item out of the cart that 463 00:15:49,279 --> 00:15:51,600 didn't exist in the cart and yep, there 464 00:15:51,600 --> 00:15:54,800 it is. It's less than zero. So, what 465 00:15:54,800 --> 00:15:58,160 we've managed to do there is give it a 466 00:15:58,160 --> 00:16:00,240 list of steps. It's [snorts] executed 467 00:16:00,240 --> 00:16:02,560 those steps and it's found a sequence of 468 00:16:02,560 --> 00:16:04,720 them. In this case, it found that the 469 00:16:04,720 --> 00:16:07,199 sequence is the very first operation, 470 00:16:07,199 --> 00:16:09,279 but it will have reduced that down from 471 00:16:09,279 --> 00:16:11,519 however many it started out trying to 472 00:16:11,519 --> 00:16:13,519 just the smallest number that produce 473 00:16:13,519 --> 00:16:15,680 our failure. It could have added one by 474 00:16:15,680 --> 00:16:17,600 default. The first method it was adding 475 00:16:17,600 --> 00:16:19,440 in that class was add. It probably did 476 00:16:19,440 --> 00:16:22,079 add first, but then it realized, oh, add 477 00:16:22,079 --> 00:16:24,880 then subtract. That that doesn't need to 478 00:16:24,880 --> 00:16:26,160 happen. We can actually go back a couple 479 00:16:26,160 --> 00:16:28,320 of steps. Let's try subtract directly. 480 00:16:28,320 --> 00:16:30,000 And it produces the shortest example 481 00:16:30,000 --> 00:16:32,320 through those. 482 00:16:32,320 --> 00:16:36,560 So, we fix it easily enough. 483 00:16:36,560 --> 00:16:38,320 And yeah, now it's actually working 484 00:16:38,320 --> 00:16:41,040 properly. So, we're talking about these 485 00:16:41,040 --> 00:16:43,040 properties a lot. Well, let's actually 486 00:16:43,040 --> 00:16:44,399 try to sort of talk about what makes a 487 00:16:44,399 --> 00:16:47,440 good property in this. 488 00:16:47,440 --> 00:16:49,279 So, so what makes a good property is 489 00:16:49,279 --> 00:16:51,839 that it should be true for every input, 490 00:16:51,839 --> 00:16:54,240 not not just most of your inputs. Uh, 491 00:16:54,240 --> 00:16:56,480 you can narrow down those inputs with 492 00:16:56,480 --> 00:16:58,639 those things that the library is giving 493 00:16:58,639 --> 00:17:00,320 you. You've got things like assume to 494 00:17:00,320 --> 00:17:01,440 skip [clears throat] something inside of 495 00:17:01,440 --> 00:17:03,040 the function. If you've got like a a 496 00:17:03,040 --> 00:17:04,480 corner case that only happens very 497 00:17:04,480 --> 00:17:07,439 rarely, you can say assume no, and it'll 498 00:17:07,439 --> 00:17:09,919 skip that. That won't break your test. 499 00:17:09,919 --> 00:17:12,079 It lets you filter them down. We saw a 500 00:17:12,079 --> 00:17:13,600 lot of times through there we never had 501 00:17:13,600 --> 00:17:15,280 a price that was negative. We we always 502 00:17:15,280 --> 00:17:16,559 said, "Hey, don't generate negative 503 00:17:16,559 --> 00:17:18,240 numbers." We're reducing the size of the 504 00:17:18,240 --> 00:17:20,240 property. So it's the sensible test 505 00:17:20,240 --> 00:17:24,240 input. And a good function input is 506 00:17:24,240 --> 00:17:25,919 going to be checkable without 507 00:17:25,919 --> 00:17:27,120 reimplementing. You don't want to have 508 00:17:27,120 --> 00:17:29,440 to change anything if suddenly that 509 00:17:29,440 --> 00:17:31,520 breaks things because why should the 510 00:17:31,520 --> 00:17:33,360 test have to change to retest after you 511 00:17:33,360 --> 00:17:35,039 fix the code? That's not necessarily the 512 00:17:35,039 --> 00:17:36,880 right property to be testing about that 513 00:17:36,880 --> 00:17:38,640 function. you you want to be able to 514 00:17:38,640 --> 00:17:41,280 test things that don't change and then 515 00:17:41,280 --> 00:17:42,880 it's going to fail nice and loudly. It's 516 00:17:42,880 --> 00:17:45,600 not going to be hard to tell what has 517 00:17:45,600 --> 00:17:47,360 busted about the code. Like we were 518 00:17:47,360 --> 00:17:49,039 looking at simple ones here where it's 519 00:17:49,039 --> 00:17:52,080 hey does the cart add up correctly as 520 00:17:52,080 --> 00:17:54,480 opposed to does something go wrong when 521 00:17:54,480 --> 00:17:56,559 we're putting things into or out of the 522 00:17:56,559 --> 00:17:59,120 cart. You're testing the property of 523 00:17:59,120 --> 00:18:01,360 cart is not negative 524 00:18:01,360 --> 00:18:02,799 and and it says something that humans 525 00:18:02,799 --> 00:18:04,480 care about like hey the cart is not 526 00:18:04,480 --> 00:18:06,160 negative. If it was negative, I would be 527 00:18:06,160 --> 00:18:09,440 giving the customer money. 528 00:18:09,440 --> 00:18:11,039 You don't want them to be vague. You 529 00:18:11,039 --> 00:18:12,240 know, you don't want something that is 530 00:18:12,240 --> 00:18:13,679 just like, hey, assert that the function 531 00:18:13,679 --> 00:18:15,760 didn't fail. You know, you you could 532 00:18:15,760 --> 00:18:16,960 make that better by saying something 533 00:18:16,960 --> 00:18:18,880 like, hey, maybe we definitely got valid 534 00:18:18,880 --> 00:18:20,400 JSON out the other side of this by, you 535 00:18:20,400 --> 00:18:22,720 know, asserting that the JSON was 536 00:18:22,720 --> 00:18:25,120 parsible. But what would be a better 537 00:18:25,120 --> 00:18:26,880 example of a property to test is that 538 00:18:26,880 --> 00:18:28,880 the parsing of the output recovers the 539 00:18:28,880 --> 00:18:30,320 input. [snorts] That you'd take an 540 00:18:30,320 --> 00:18:31,919 input, generate it, run it through, and 541 00:18:31,919 --> 00:18:33,600 then recover it out the other side. that 542 00:18:33,600 --> 00:18:35,280 would be the best property to test about 543 00:18:35,280 --> 00:18:37,120 a function like that because it's not 544 00:18:37,120 --> 00:18:38,640 really about like how cleverly you can 545 00:18:38,640 --> 00:18:40,480 test every little individual step. It 546 00:18:40,480 --> 00:18:42,000 it's about the leverage that this 547 00:18:42,000 --> 00:18:44,160 library gives you to try and find what 548 00:18:44,160 --> 00:18:45,840 is correct and provable about your 549 00:18:45,840 --> 00:18:48,080 codebase so that you know it's doing its 550 00:18:48,080 --> 00:18:50,000 job. 551 00:18:50,000 --> 00:18:52,000 So where does this all sort of fit in in 552 00:18:52,000 --> 00:18:55,039 terms of getting tests written? 553 00:18:55,039 --> 00:18:56,559 Well, I'll start with some stuff that's 554 00:18:56,559 --> 00:18:58,160 obvious, you know, some patterns worth 555 00:18:58,160 --> 00:19:00,400 borrowing and and it's things like 556 00:19:00,400 --> 00:19:02,160 roundtrip decodes like I just mentioned 557 00:19:02,160 --> 00:19:03,600 where you can say, "Hey, I've got an 558 00:19:03,600 --> 00:19:05,200 input and I'm going to decode it and 559 00:19:05,200 --> 00:19:06,240 encode it. I'm going to run it through 560 00:19:06,240 --> 00:19:07,520 two of my functions and make sure that 561 00:19:07,520 --> 00:19:09,360 they both do their jobs." You know, 562 00:19:09,360 --> 00:19:10,559 things like making sure that if you 563 00:19:10,559 --> 00:19:12,400 query from a database one way and you 564 00:19:12,400 --> 00:19:14,240 query from database another way, you get 565 00:19:14,240 --> 00:19:16,080 the same thing. You know, normalizing 566 00:19:16,080 --> 00:19:17,440 stuff is another great example. If you 567 00:19:17,440 --> 00:19:18,799 normalize a thing twice, it really 568 00:19:18,799 --> 00:19:20,880 shouldn't have changed. Uh, another 569 00:19:20,880 --> 00:19:22,640 great one that most people don't sort of 570 00:19:22,640 --> 00:19:24,559 think of at first is any kind of fast 571 00:19:24,559 --> 00:19:27,120 path like Oracle testing where you do a 572 00:19:27,120 --> 00:19:28,960 thing like, hey, I've got a slow version 573 00:19:28,960 --> 00:19:31,760 that works for every input that I have, 574 00:19:31,760 --> 00:19:33,600 but I've also got this fast one if it's 575 00:19:33,600 --> 00:19:36,240 less than like a thousand items. I want 576 00:19:36,240 --> 00:19:37,840 to make sure that that short fast 577 00:19:37,840 --> 00:19:40,559 version does the exact same thing for 578 00:19:40,559 --> 00:19:43,600 every one of its inputs as the slow one. 579 00:19:43,600 --> 00:19:45,600 And you can use hypothesis to very 580 00:19:45,600 --> 00:19:48,000 efficiently test that kind of code. and 581 00:19:48,000 --> 00:19:49,840 and the other one there like you know 582 00:19:49,840 --> 00:19:51,679 sums unchanged after ordering is sort of 583 00:19:51,679 --> 00:19:53,120 the regular run-of-the-mill kind of a 584 00:19:53,120 --> 00:19:56,160 test but it it's important to not forget 585 00:19:56,160 --> 00:19:57,840 and like I said earlier it works with 586 00:19:57,840 --> 00:19:59,679 piest so if you've already got a bunch 587 00:19:59,679 --> 00:20:01,280 of tests you don't have to rewrite the 588 00:20:01,280 --> 00:20:03,039 test suite to take advantage of this 589 00:20:03,039 --> 00:20:05,120 it's the same test runner the same CI 590 00:20:05,120 --> 00:20:06,559 that you're already using you get the 591 00:20:06,559 --> 00:20:08,640 same piest report out the other side you 592 00:20:08,640 --> 00:20:10,960 can just decide hey this this new thing 593 00:20:10,960 --> 00:20:12,640 we're writing let's use hypothesis on 594 00:20:12,640 --> 00:20:15,120 that or hey this old thing that we 595 00:20:15,120 --> 00:20:16,559 missed a bug because we didn't write 596 00:20:16,559 --> 00:20:17,919 enough tests for 597 00:20:17,919 --> 00:20:19,360 Maybe we should write a new test that 598 00:20:19,360 --> 00:20:21,200 catches that, you can just add 599 00:20:21,200 --> 00:20:22,960 hypothesis to that one spot without 600 00:20:22,960 --> 00:20:25,679 breaking the rest of the test suite. 601 00:20:25,679 --> 00:20:27,280 So, a couple of the common mistakes if 602 00:20:27,280 --> 00:20:29,200 you do want to try and pick it up is 603 00:20:29,200 --> 00:20:30,720 testing a property that can't actually 604 00:20:30,720 --> 00:20:33,200 fail. Now, that might seem obvious, but 605 00:20:33,200 --> 00:20:34,640 when you actually start trying to think 606 00:20:34,640 --> 00:20:37,440 of these inputs and outputs, you do 607 00:20:37,440 --> 00:20:39,440 start to realize, oh yeah, no, that did 608 00:20:39,440 --> 00:20:41,280 actually not exercise the function of 609 00:20:41,280 --> 00:20:42,640 the code. That would have always been 610 00:20:42,640 --> 00:20:44,880 true. uh you don't want to reimplement 611 00:20:44,880 --> 00:20:46,320 the whole test function. If you've 612 00:20:46,320 --> 00:20:48,000 written complicated tests before, you've 613 00:20:48,000 --> 00:20:50,159 sort of had that tension where this test 614 00:20:50,159 --> 00:20:51,600 is getting bigger than the thing it's 615 00:20:51,600 --> 00:20:54,320 testing. That's not ideal. You don't 616 00:20:54,320 --> 00:20:56,480 really want to have to generate that 617 00:20:56,480 --> 00:20:58,720 much complexity in your test input. You 618 00:20:58,720 --> 00:21:00,400 ideally want to try and find a simpler 619 00:21:00,400 --> 00:21:02,799 thing to test. Uh that's the other part 620 00:21:02,799 --> 00:21:04,640 of over complicating the strategy 621 00:21:04,640 --> 00:21:06,799 because those strategies are chained. If 622 00:21:06,799 --> 00:21:08,720 you've got a very complex object, you 623 00:21:08,720 --> 00:21:10,240 might find yourself building a strategy 624 00:21:10,240 --> 00:21:12,480 that is generate this object from those 625 00:21:12,480 --> 00:21:14,240 objects which are generated by these 626 00:21:14,240 --> 00:21:16,000 inputs which are filtered with those 627 00:21:16,000 --> 00:21:17,840 queries. And it gets quite messy 628 00:21:17,840 --> 00:21:20,159 sometimes. And if you need that, you 629 00:21:20,159 --> 00:21:21,919 want to maybe build that in a reusable 630 00:21:21,919 --> 00:21:24,559 way up front and not write a specific 631 00:21:24,559 --> 00:21:26,960 test that does that on the fly in one 632 00:21:26,960 --> 00:21:29,919 place only. And using assume is the 633 00:21:29,919 --> 00:21:31,360 other sort of mistake. You really want 634 00:21:31,360 --> 00:21:33,440 to avoid that. use it sparingly because 635 00:21:33,440 --> 00:21:35,360 it's basically saying, "Hey, I spent all 636 00:21:35,360 --> 00:21:38,320 this time generating an input and you 637 00:21:38,320 --> 00:21:40,320 don't care. Why why would you want to 638 00:21:40,320 --> 00:21:42,080 generate an input you don't care about? 639 00:21:42,080 --> 00:21:44,320 It's an escape hatch, not something to 640 00:21:44,320 --> 00:21:47,200 use on every function you try, but use 641 00:21:47,200 --> 00:21:48,400 it while you're learning if you have 642 00:21:48,400 --> 00:21:50,880 to." And the other thing is try to make 643 00:21:50,880 --> 00:21:53,280 everything a hypothesis test. Sometimes 644 00:21:53,280 --> 00:21:55,280 you just need a simple test. Sometimes 645 00:21:55,280 --> 00:21:56,799 you want to write some dock tests so 646 00:21:56,799 --> 00:21:58,240 that it's easier for people to see how 647 00:21:58,240 --> 00:21:59,919 to use the code. You know, behavior 648 00:21:59,919 --> 00:22:02,640 that's easier to show than to state is a 649 00:22:02,640 --> 00:22:05,360 fantastic place to not use hypothesis. 650 00:22:05,360 --> 00:22:06,640 You know, tests where defining the 651 00:22:06,640 --> 00:22:08,799 properties is basically rewriting the 652 00:22:08,799 --> 00:22:10,640 function. You know, specific 653 00:22:10,640 --> 00:22:13,039 regressions, we have an example, but do 654 00:22:13,039 --> 00:22:16,799 we really want to make 8 9 10 50 655 00:22:16,799 --> 00:22:19,039 decorators on a function? Maybe you just 656 00:22:19,039 --> 00:22:20,799 want to have a list of single functions 657 00:22:20,799 --> 00:22:22,559 that test those instead. [snorts] Same 658 00:22:22,559 --> 00:22:24,640 thing goes for known edge cases. You can 659 00:22:24,640 --> 00:22:26,720 leave the boring tests using just 660 00:22:26,720 --> 00:22:28,640 regular testing. You don't need to make 661 00:22:28,640 --> 00:22:31,919 everything hypothesis. 662 00:22:31,919 --> 00:22:33,360 So, you know, where do you actually 663 00:22:33,360 --> 00:22:36,480 start using a library like this? 664 00:22:36,480 --> 00:22:38,159 Well, you can start with one round trip 665 00:22:38,159 --> 00:22:40,000 on some simple code. You you can find 666 00:22:40,000 --> 00:22:41,039 something in the code that you don't 667 00:22:41,039 --> 00:22:42,880 already trust. Like, as I said earlier, 668 00:22:42,880 --> 00:22:44,480 if you've got some AI generated code 669 00:22:44,480 --> 00:22:46,000 that you know is supposed to do 670 00:22:46,000 --> 00:22:47,280 something and it's supposed to take 671 00:22:47,280 --> 00:22:49,039 certain inputs, you can wrap a 672 00:22:49,039 --> 00:22:50,640 hypothesis test around that really 673 00:22:50,640 --> 00:22:53,120 easily and then you can let the computer 674 00:22:53,120 --> 00:22:55,360 do the work validating what the other 675 00:22:55,360 --> 00:22:57,360 computer wrote. you know, keep the 676 00:22:57,360 --> 00:22:59,440 strategy simple. Don't necessarily try 677 00:22:59,440 --> 00:23:01,120 to make the most complicated part of 678 00:23:01,120 --> 00:23:02,960 your codebase tested with hypothesis. 679 00:23:02,960 --> 00:23:04,640 Find some easy stuff. Get the hang of it 680 00:23:04,640 --> 00:23:07,280 first. The documentation is very good 681 00:23:07,280 --> 00:23:09,120 and includes such nice helpers as things 682 00:23:09,120 --> 00:23:11,120 for generating Django classes. You know, 683 00:23:11,120 --> 00:23:12,720 it's got a bunch of stuff in the helpers 684 00:23:12,720 --> 00:23:15,120 and extras area in the documentation 685 00:23:15,120 --> 00:23:17,200 that really show you places where you 686 00:23:17,200 --> 00:23:18,400 can just, oh yeah, I could add it for 687 00:23:18,400 --> 00:23:20,159 that. You know, you got to install it 688 00:23:20,159 --> 00:23:22,000 obviously, but at the end of the day, 689 00:23:22,000 --> 00:23:23,600 it's it's not magic. you know, it 690 00:23:23,600 --> 00:23:24,799 doesn't read your code and divine 691 00:23:24,799 --> 00:23:26,960 magically what you meant to do. It's 692 00:23:26,960 --> 00:23:28,799 just a better form of leverage. You 693 00:23:28,799 --> 00:23:30,480 describe what needs to be true all the 694 00:23:30,480 --> 00:23:32,400 time, and it spends its time hunting for 695 00:23:32,400 --> 00:23:34,720 the actual way to break that. [snorts] 696 00:23:34,720 --> 00:23:37,919 So, with that, I know this is a bit of a 697 00:23:37,919 --> 00:23:41,366 hands-on, so there's time for questions. 698 00:23:41,366 --> 00:23:43,386 [applause] 699 00:23:51,200 --> 00:23:54,960 in your use of hypothesis. What's the 700 00:23:54,960 --> 00:23:58,000 sort of what's the largest impact that 701 00:23:58,000 --> 00:24:00,640 you've managed to get out of this? 702 00:24:00,640 --> 00:24:03,039 Well, one of the places I first came 703 00:24:03,039 --> 00:24:04,240 across the library was when I was trying 704 00:24:04,240 --> 00:24:06,480 to test some domain name stuff [snorts] 705 00:24:06,480 --> 00:24:08,960 and I ended up actually contributing to 706 00:24:08,960 --> 00:24:10,880 the library the first time I used it 707 00:24:10,880 --> 00:24:12,480 because it didn't test thoroughly 708 00:24:12,480 --> 00:24:14,240 enough. But it also happened to be that 709 00:24:14,240 --> 00:24:16,720 I was at a Pyon and one of the 710 00:24:16,720 --> 00:24:18,400 developers was here doing a sprint on 711 00:24:18,400 --> 00:24:20,000 it. So I was able to very quickly go 712 00:24:20,000 --> 00:24:22,480 from it doesn't do exactly what I want 713 00:24:22,480 --> 00:24:24,720 to no it does now very quickly and 714 00:24:24,720 --> 00:24:26,159 suddenly had a much better email 715 00:24:26,159 --> 00:24:28,559 validator in my in my code than I did 716 00:24:28,559 --> 00:24:31,559 originally. 717 00:24:32,000 --> 00:24:34,960 Hey um when I've previously used uh fuzz 718 00:24:34,960 --> 00:24:36,559 testing libraries some features that 719 00:24:36,559 --> 00:24:38,080 I've really appreciated a coverage 720 00:24:38,080 --> 00:24:41,039 guidance. So um recording the seeds that 721 00:24:41,039 --> 00:24:43,039 resulted in additional like code paths 722 00:24:43,039 --> 00:24:44,799 being followed by tracking the code 723 00:24:44,799 --> 00:24:47,039 execution with with a coverage tool. Uh 724 00:24:47,039 --> 00:24:48,880 what features does hypothesis provide 725 00:24:48,880 --> 00:24:50,159 for like that sort of coverage 726 00:24:50,159 --> 00:24:52,159 integration and then storing your seed 727 00:24:52,159 --> 00:24:54,720 mutation corpus? Uh if it's in piest 728 00:24:54,720 --> 00:24:55,520 it's probably going to work with 729 00:24:55,520 --> 00:24:57,440 hypothesis. There's a couple of bits 730 00:24:57,440 --> 00:25:00,320 that don't um like notably this may have 731 00:25:00,320 --> 00:25:01,279 changed. It's been a while since I 732 00:25:01,279 --> 00:25:02,480 looked, but uh there was a bit of an 733 00:25:02,480 --> 00:25:04,559 issue with subtests not working nicely 734 00:25:04,559 --> 00:25:06,640 with hypothesis, but they're a bit of an 735 00:25:06,640 --> 00:25:08,799 not as commonly used feature of Piest, 736 00:25:08,799 --> 00:25:10,159 but all the standard stuff that you 737 00:25:10,159 --> 00:25:11,279 would use for things like coverage 738 00:25:11,279 --> 00:25:12,880 testing and tracking coverage tests 739 00:25:12,880 --> 00:25:17,240 between test runs all works. 740 00:25:17,919 --> 00:25:20,640 Um more and more code is in Python is 741 00:25:20,640 --> 00:25:22,480 now gaining type annotations. Lots of 742 00:25:22,480 --> 00:25:24,000 libraries will have type annotations in 743 00:25:24,000 --> 00:25:27,919 them now. um is the like are the type 744 00:25:27,919 --> 00:25:30,559 annotations usable by hypothesis so that 745 00:25:30,559 --> 00:25:32,240 you don't have to like write your type 746 00:25:32,240 --> 00:25:33,840 annotations and then also write on the 747 00:25:33,840 --> 00:25:36,640 line above the type generators for those 748 00:25:36,640 --> 00:25:39,039 those types. I think there has been some 749 00:25:39,039 --> 00:25:40,880 work to try and do that, but off the top 750 00:25:40,880 --> 00:25:42,640 of my head, I'm not sure if it got 751 00:25:42,640 --> 00:25:44,720 finished. But you will find that if 752 00:25:44,720 --> 00:25:46,240 you're using any kind of sort of type 753 00:25:46,240 --> 00:25:48,400 ahead assistance like any of the 754 00:25:48,400 --> 00:25:51,279 reasonably smart autocompletes from an 755 00:25:51,279 --> 00:25:53,120 IDE like Jet Brains or VS Code with any 756 00:25:53,120 --> 00:25:55,039 of the better plugins, you you'll 757 00:25:55,039 --> 00:25:56,880 probably find that it'll suggest the 758 00:25:56,880 --> 00:25:58,400 right strategy because it can sort of 759 00:25:58,400 --> 00:26:00,000 guess that oh yeah, the last function 760 00:26:00,000 --> 00:26:02,240 you had with an integer input, you you 761 00:26:02,240 --> 00:26:04,159 used an integer. And so it'll suggest 762 00:26:04,159 --> 00:26:05,760 the strategy for an integer straight 763 00:26:05,760 --> 00:26:06,880 away when you start writing the 764 00:26:06,880 --> 00:26:09,760 decorator. So it it's not necessarily as 765 00:26:09,760 --> 00:26:12,559 much of an autogeneration issue as it 766 00:26:12,559 --> 00:26:14,720 might seem at first, but I believe there 767 00:26:14,720 --> 00:26:16,559 is some work to try to make a robust way 768 00:26:16,559 --> 00:26:19,760 to automatically use the types. 769 00:26:19,760 --> 00:26:21,440 I think there's still a little time. 770 00:26:21,440 --> 00:26:24,400 Any more questions? 771 00:26:24,400 --> 00:26:26,480 Silence. 772 00:26:26,480 --> 00:26:28,080 Tests usually get more questions. 773 00:26:28,080 --> 00:26:30,159 Yeah. All right. Um [clears throat] I'd 774 00:26:30,159 --> 00:26:31,440 like to [snorts] thank Sam again. If we 775 00:26:31,440 --> 00:26:35,653 all get a round of applause. [applause] 776 00:26:39,520 --> 00:26:42,559 And here is your fresh mug. [laughter] 777 00:26:42,559 --> 00:26:44,768 Thank you. 778 00:26:44,768 --> 00:26:46,788 [applause]