1 00:00:00,480 --> 00:00:03,480 foreign 2 00:00:09,139 --> 00:00:16,379 Au education 3 00:00:11,840 --> 00:00:18,420 session and uh Stephen here has the 4 00:00:16,379 --> 00:00:21,859 enviable position of being the last 5 00:00:18,420 --> 00:00:21,859 session before the afternoon break 6 00:00:22,260 --> 00:00:26,100 we're going to give him all of our Focus 7 00:00:23,939 --> 00:00:27,779 Stephen is a senior lecturer at the 8 00:00:26,100 --> 00:00:29,279 University of Sydney in the field of 9 00:00:27,779 --> 00:00:31,320 Statistics data science and machine 10 00:00:29,279 --> 00:00:34,260 learning and teaches python in all of 11 00:00:31,320 --> 00:00:37,100 those and what I found exciting is that 12 00:00:34,260 --> 00:00:38,940 he also has a bit of a pawn shot for 13 00:00:37,100 --> 00:00:40,860 microcontrollers and has a fully 14 00:00:38,940 --> 00:00:44,160 automated Greenhouse in his backyard so 15 00:00:40,860 --> 00:00:47,460 we're going to be talking later okay 16 00:00:44,160 --> 00:00:49,140 thank you uh thank you for coming to 17 00:00:47,460 --> 00:00:52,140 probably the most boring titled 18 00:00:49,140 --> 00:00:53,700 presentation of pycon assurance of 19 00:00:52,140 --> 00:00:56,219 learning through testing I'm presenting 20 00:00:53,700 --> 00:01:00,020 this on behalf of work from both myself 21 00:00:56,219 --> 00:01:03,000 and Allison who you saw earlier 22 00:01:00,020 --> 00:01:04,500 and this is work that we have sort of 23 00:01:03,000 --> 00:01:06,060 developed over a number of years it's 24 00:01:04,500 --> 00:01:07,799 not something that we have put together 25 00:01:06,060 --> 00:01:09,720 just for pycon this is something that 26 00:01:07,799 --> 00:01:12,119 has been an ongoing project for us and 27 00:01:09,720 --> 00:01:14,400 in this talk I'm going to go over the 28 00:01:12,119 --> 00:01:16,439 problems we're facing and some 29 00:01:14,400 --> 00:01:18,420 suggestions about how we've solved them 30 00:01:16,439 --> 00:01:20,659 and hopefully they're helpful for you in 31 00:01:18,420 --> 00:01:20,659 the future 32 00:01:20,759 --> 00:01:25,680 so first of all I want to talk about 33 00:01:23,040 --> 00:01:26,820 um cover off what is Assurance of 34 00:01:25,680 --> 00:01:29,040 learning 35 00:01:26,820 --> 00:01:30,360 um the key thing that we're trying to 36 00:01:29,040 --> 00:01:31,439 get across with Assurance of learning is 37 00:01:30,360 --> 00:01:33,479 that you need to make sure that students 38 00:01:31,439 --> 00:01:34,920 are actually learning what we want them 39 00:01:33,479 --> 00:01:36,420 to learn 40 00:01:34,920 --> 00:01:37,560 um or in other cases we want them to 41 00:01:36,420 --> 00:01:39,360 achieve the learning outcomes that we 42 00:01:37,560 --> 00:01:40,680 set so for our units and our in our 43 00:01:39,360 --> 00:01:42,540 context the University of Sydney we set 44 00:01:40,680 --> 00:01:43,500 learning outcomes for our units so these 45 00:01:42,540 --> 00:01:45,540 are the things we want our students to 46 00:01:43,500 --> 00:01:47,040 achieve by the end and we want to make 47 00:01:45,540 --> 00:01:48,299 sure they do that and you could take 48 00:01:47,040 --> 00:01:49,439 that further and say that there are 49 00:01:48,299 --> 00:01:51,180 learning outcomes that each week 50 00:01:49,439 --> 00:01:52,439 subtopics that you want them to learn as 51 00:01:51,180 --> 00:01:53,820 well 52 00:01:52,439 --> 00:01:57,299 so we want to demonstrate this through 53 00:01:53,820 --> 00:01:58,320 assessment and it's not just for the 54 00:01:57,299 --> 00:01:59,700 sort of promise that we give to the 55 00:01:58,320 --> 00:02:01,619 students through those learning outcomes 56 00:01:59,700 --> 00:02:03,540 and what we want to give as Educators 57 00:02:01,619 --> 00:02:06,420 but we actually at the University have 58 00:02:03,540 --> 00:02:08,160 some requirements that we assure that we 59 00:02:06,420 --> 00:02:09,840 do this 60 00:02:08,160 --> 00:02:11,400 and what I'm going to talk about today 61 00:02:09,840 --> 00:02:14,660 is mostly in the context of using Ed 62 00:02:11,400 --> 00:02:16,680 stem but tools like reflect and GitHub 63 00:02:14,660 --> 00:02:18,599 through Renee stalk you would have seen 64 00:02:16,680 --> 00:02:21,120 earlier actually have the facility to do 65 00:02:18,599 --> 00:02:23,099 this sort of testing of your students 66 00:02:21,120 --> 00:02:24,239 code so keep in mind that even though 67 00:02:23,099 --> 00:02:27,379 I'm going to give examples to show 68 00:02:24,239 --> 00:02:29,760 screenshots of ads this is quite General 69 00:02:27,379 --> 00:02:31,500 so don't think that you have to use add 70 00:02:29,760 --> 00:02:33,239 to uh to do this 71 00:02:31,500 --> 00:02:34,920 and here's an example of one of the 72 00:02:33,239 --> 00:02:36,900 exercises that we might actually give 73 00:02:34,920 --> 00:02:38,819 our students to do the description of 74 00:02:36,900 --> 00:02:41,220 one of them anyway so we might get them 75 00:02:38,819 --> 00:02:42,260 to write a simple code a Python program 76 00:02:41,220 --> 00:02:44,640 to 77 00:02:42,260 --> 00:02:45,959 read in a number from the user so 78 00:02:44,640 --> 00:02:47,099 collect some user input and then tell 79 00:02:45,959 --> 00:02:48,360 the user whether that was a big number 80 00:02:47,099 --> 00:02:50,519 or not so this is a very simple exercise 81 00:02:48,360 --> 00:02:51,900 we might give our students in week one 82 00:02:50,519 --> 00:02:53,220 or week two of an intro programming 83 00:02:51,900 --> 00:02:55,739 class 84 00:02:53,220 --> 00:02:57,599 and the students will get this exercise 85 00:02:55,739 --> 00:03:00,000 description and then they'll write some 86 00:02:57,599 --> 00:03:01,440 code the code might look something like 87 00:03:00,000 --> 00:03:03,720 this I've intentionally put a bug in 88 00:03:01,440 --> 00:03:04,640 there hopefully you can spot it for that 89 00:03:03,720 --> 00:03:07,459 exercise 90 00:03:04,640 --> 00:03:10,980 and then they'll go ahead and submit it 91 00:03:07,459 --> 00:03:12,780 so after they've submitted it they'll 92 00:03:10,980 --> 00:03:15,360 get some feedback from a number of test 93 00:03:12,780 --> 00:03:16,680 cases that we've written ourselves so in 94 00:03:15,360 --> 00:03:18,420 this example there are four test cases 95 00:03:16,680 --> 00:03:19,860 shown we have the they're just based on 96 00:03:18,420 --> 00:03:22,019 the number that we input into the 97 00:03:19,860 --> 00:03:23,640 program to test where the student passed 98 00:03:22,019 --> 00:03:25,680 or not so in this case the student 99 00:03:23,640 --> 00:03:28,080 didn't pass the 107 test case because 100 00:03:25,680 --> 00:03:29,640 they actually put out or printed out on 101 00:03:28,080 --> 00:03:32,060 the console that's a big number when 102 00:03:29,640 --> 00:03:35,400 they shouldn't have 103 00:03:32,060 --> 00:03:36,480 and in this example through this talk 104 00:03:35,400 --> 00:03:39,659 we're going to go through the problems 105 00:03:36,480 --> 00:03:40,980 of this sort of environment of uh making 106 00:03:39,659 --> 00:03:42,060 sure that students are doing what we 107 00:03:40,980 --> 00:03:43,379 actually want them to be doing and 108 00:03:42,060 --> 00:03:46,140 learning what we want them to learn 109 00:03:43,379 --> 00:03:47,760 through this automatic automatic testing 110 00:03:46,140 --> 00:03:49,560 environment 111 00:03:47,760 --> 00:03:51,060 and just as a little bit of context for 112 00:03:49,560 --> 00:03:53,340 behind the scenes this is what one of 113 00:03:51,060 --> 00:03:57,000 those diff based tests would look like 114 00:03:53,340 --> 00:03:58,620 on Ed's back end we tell Ed this is the 115 00:03:57,000 --> 00:04:00,239 name of the test case on the left hand 116 00:03:58,620 --> 00:04:02,519 side this is the sort of the parameters 117 00:04:00,239 --> 00:04:03,840 or configuration and then we tell it 118 00:04:02,519 --> 00:04:05,220 what's the expected input for the 119 00:04:03,840 --> 00:04:06,840 program what's the expected output for 120 00:04:05,220 --> 00:04:08,580 the program and then above I have some 121 00:04:06,840 --> 00:04:11,159 examples of what those input and output 122 00:04:08,580 --> 00:04:12,140 files look like creatively named in and 123 00:04:11,159 --> 00:04:15,180 out 124 00:04:12,140 --> 00:04:17,459 and from here Ed will do all the heavy 125 00:04:15,180 --> 00:04:19,500 lifting for us normally of actually 126 00:04:17,459 --> 00:04:21,540 producing all that nice pretty feedback 127 00:04:19,500 --> 00:04:24,300 for the student 128 00:04:21,540 --> 00:04:25,740 and this works okay for the great 129 00:04:24,300 --> 00:04:28,880 majority of cases but there are cases 130 00:04:25,740 --> 00:04:33,000 where it's it falls down 131 00:04:28,880 --> 00:04:34,440 and the cases are really where we want 132 00:04:33,000 --> 00:04:35,580 to do more complicated testing like 133 00:04:34,440 --> 00:04:37,979 longer programs that are more 134 00:04:35,580 --> 00:04:39,960 interactive and dynamic or where we have 135 00:04:37,979 --> 00:04:42,300 students that aren't trying to hack the 136 00:04:39,960 --> 00:04:45,080 test cases so only when it only works 137 00:04:42,300 --> 00:04:48,540 and we have compliance students 138 00:04:45,080 --> 00:04:51,060 and so we by creating this sort of test 139 00:04:48,540 --> 00:04:52,740 case setup and evaluating our students 140 00:04:51,060 --> 00:04:54,600 in this way we've created a sort of 141 00:04:52,740 --> 00:04:56,220 student versus teacher Paradigm where 142 00:04:54,600 --> 00:04:57,479 we're kind of chasing each other's Tails 143 00:04:56,220 --> 00:05:00,440 they're trying to beat us all the time 144 00:04:57,479 --> 00:05:00,440 and we're trying to beat them 145 00:05:01,080 --> 00:05:06,240 um so here's an example of a devious 146 00:05:02,699 --> 00:05:09,780 student's solution for that example they 147 00:05:06,240 --> 00:05:11,580 might just uh hard code in the answers 148 00:05:09,780 --> 00:05:13,560 for the test cases so for the other test 149 00:05:11,580 --> 00:05:15,060 cases like two 107 you actually don't 150 00:05:13,560 --> 00:05:17,580 need to print any output to pass this 151 00:05:15,060 --> 00:05:18,960 test case so this is not what we want 152 00:05:17,580 --> 00:05:20,759 the student to do we want them to use an 153 00:05:18,960 --> 00:05:22,259 if statement and produce a more general 154 00:05:20,759 --> 00:05:24,300 program we don't want them to be hard 155 00:05:22,259 --> 00:05:25,800 coding against our test cases 156 00:05:24,300 --> 00:05:27,720 we want them to we want them to 157 00:05:25,800 --> 00:05:29,400 demonstrate their General understanding 158 00:05:27,720 --> 00:05:31,440 and have the ability to generalize to 159 00:05:29,400 --> 00:05:33,240 new problems 160 00:05:31,440 --> 00:05:35,340 here's another example of a more 161 00:05:33,240 --> 00:05:37,740 complicated case where we might want 162 00:05:35,340 --> 00:05:39,419 them to do some sort of simulation and 163 00:05:37,740 --> 00:05:41,820 then that simulation results in a random 164 00:05:39,419 --> 00:05:44,479 output in this case this is a Monte 165 00:05:41,820 --> 00:05:46,500 Carlo simulation so they're sort of 166 00:05:44,479 --> 00:05:48,600 evaluating an integral the area under 167 00:05:46,500 --> 00:05:50,940 the curve by actually just generating 168 00:05:48,600 --> 00:05:52,979 around a bunch of numbers or coordinates 169 00:05:50,940 --> 00:05:54,600 on the on the grid and seeing which 170 00:05:52,979 --> 00:05:55,500 proportion of those fall underneath the 171 00:05:54,600 --> 00:05:57,240 Curve 172 00:05:55,500 --> 00:05:58,320 so each time you run it as shown on the 173 00:05:57,240 --> 00:06:00,300 right hand side you might get a slightly 174 00:05:58,320 --> 00:06:01,680 different result and our diff based 175 00:06:00,300 --> 00:06:03,120 testing that Ed has provided in the past 176 00:06:01,680 --> 00:06:06,180 is not actually going to be suitable for 177 00:06:03,120 --> 00:06:07,500 that case because we have a fixed input 178 00:06:06,180 --> 00:06:09,539 and output 179 00:06:07,500 --> 00:06:11,280 because we can't tolerate different 180 00:06:09,539 --> 00:06:15,020 outputs from our student even though 181 00:06:11,280 --> 00:06:15,020 they've got a perfectly valid answer 182 00:06:15,900 --> 00:06:19,560 um and here's another example of a 183 00:06:17,699 --> 00:06:21,600 longer form program that a student we 184 00:06:19,560 --> 00:06:23,940 might ask a student to build 185 00:06:21,600 --> 00:06:25,860 um so they have this console program 186 00:06:23,940 --> 00:06:29,100 that is just uh collecting key value 187 00:06:25,860 --> 00:06:31,259 Pairs and sort of a data entry program 188 00:06:29,100 --> 00:06:32,880 and in this case the student has to has 189 00:06:31,259 --> 00:06:34,740 to implement a whole bunch of uh 190 00:06:32,880 --> 00:06:36,479 commands like the add get and print 191 00:06:34,740 --> 00:06:37,919 commands and these will also have 192 00:06:36,479 --> 00:06:39,600 Dynamic outputs based on the previous 193 00:06:37,919 --> 00:06:43,620 state on what has been entered into the 194 00:06:39,600 --> 00:06:46,860 data storage system beforehand so being 195 00:06:43,620 --> 00:06:47,940 able to test the the state of the 196 00:06:46,860 --> 00:06:50,160 program is correct and the students 197 00:06:47,940 --> 00:06:53,280 doing what we want using that diff style 198 00:06:50,160 --> 00:06:55,080 test is not really uh very user friendly 199 00:06:53,280 --> 00:06:56,759 to us as teachers because it means we 200 00:06:55,080 --> 00:06:58,860 have to write thousands of tests to make 201 00:06:56,759 --> 00:07:00,180 sure that everything actually works 202 00:06:58,860 --> 00:07:02,340 and we want something a little bit more 203 00:07:00,180 --> 00:07:04,259 General and flexible that can cut down 204 00:07:02,340 --> 00:07:07,199 on the work for us when testing long 205 00:07:04,259 --> 00:07:09,180 complicated programs like this 206 00:07:07,199 --> 00:07:11,819 there's also other issues that are a 207 00:07:09,180 --> 00:07:13,919 little bit more python technical like if 208 00:07:11,819 --> 00:07:15,900 we want students to demonstrate their 209 00:07:13,919 --> 00:07:17,580 understanding of some object-oriented 210 00:07:15,900 --> 00:07:21,020 programming Concepts we might want them 211 00:07:17,580 --> 00:07:23,340 to use like pythons inbuilt inheritance 212 00:07:21,020 --> 00:07:25,500 features like on the top we might expect 213 00:07:23,340 --> 00:07:27,500 them for their checking class of a bank 214 00:07:25,500 --> 00:07:30,120 account to use the super 215 00:07:27,500 --> 00:07:33,000 method rather than just sort of 216 00:07:30,120 --> 00:07:36,300 re-implementing that same code in the 217 00:07:33,000 --> 00:07:38,460 child class and doing a diff based test 218 00:07:36,300 --> 00:07:40,080 on this is kind of weird like I don't 219 00:07:38,460 --> 00:07:43,139 think it's possible 220 00:07:40,080 --> 00:07:44,460 uh and again there are some cases where 221 00:07:43,139 --> 00:07:46,199 we want students to do things by hand 222 00:07:44,460 --> 00:07:48,900 rather than using an existing library 223 00:07:46,199 --> 00:07:51,419 and doing a diff on that is not going to 224 00:07:48,900 --> 00:07:53,039 reveal that to us so these are some uh a 225 00:07:51,419 --> 00:07:55,500 devious student might go oh this 226 00:07:53,039 --> 00:07:58,080 exercise asks us to calculate the 227 00:07:55,500 --> 00:08:00,060 average or the mean so I know numpy 228 00:07:58,080 --> 00:08:01,500 implements that for me so I will just 229 00:08:00,060 --> 00:08:03,000 use numpy's implementation rather than 230 00:08:01,500 --> 00:08:04,440 write it from scratch squares we want 231 00:08:03,000 --> 00:08:06,080 them to write it from scratch to 232 00:08:04,440 --> 00:08:11,780 demonstrate their understanding of 233 00:08:06,080 --> 00:08:11,780 variables data types loops and so on 234 00:08:12,840 --> 00:08:16,560 so in summary we want students to solve 235 00:08:14,880 --> 00:08:18,780 Dynamic interactive and interesting 236 00:08:16,560 --> 00:08:19,800 programs and we want them to be able to 237 00:08:18,780 --> 00:08:21,840 demonstrate achievement of learning 238 00:08:19,800 --> 00:08:24,440 outcomes not to take shortcuts and 239 00:08:21,840 --> 00:08:24,440 things like that 240 00:08:25,020 --> 00:08:29,580 so the start of our solution here is to 241 00:08:27,720 --> 00:08:33,360 use a python Library called P expect 242 00:08:29,580 --> 00:08:34,800 based on the expect console program if 243 00:08:33,360 --> 00:08:36,539 you've used it before 244 00:08:34,800 --> 00:08:38,599 the important thing about p-expect is it 245 00:08:36,539 --> 00:08:41,219 allows you to 246 00:08:38,599 --> 00:08:43,919 programmatically control another script 247 00:08:41,219 --> 00:08:46,260 and you can feed it input and read the 248 00:08:43,919 --> 00:08:47,940 output as if you were a human typing 249 00:08:46,260 --> 00:08:52,800 those commands yourself 250 00:08:47,940 --> 00:08:54,600 so p-expect is very powerful and you 251 00:08:52,800 --> 00:08:56,040 don't actually to be useful you don't 252 00:08:54,600 --> 00:08:59,040 actually have to use the full gamut of 253 00:08:56,040 --> 00:09:01,800 what it gives us but we only use a very 254 00:08:59,040 --> 00:09:04,200 small number of he expects features to 255 00:09:01,800 --> 00:09:05,880 get our good results 256 00:09:04,200 --> 00:09:06,920 here's an example of how you might use 257 00:09:05,880 --> 00:09:09,839 pay expect 258 00:09:06,920 --> 00:09:11,700 so you just need to import it and then 259 00:09:09,839 --> 00:09:15,180 the next sort of block of code there is 260 00:09:11,700 --> 00:09:17,760 to spawn your students codes like that 261 00:09:15,180 --> 00:09:20,940 big number python file in a sub process 262 00:09:17,760 --> 00:09:22,440 a child process set up the logging so 263 00:09:20,940 --> 00:09:24,779 that you can collect the output of that 264 00:09:22,440 --> 00:09:27,140 program as it's running and then send 265 00:09:24,779 --> 00:09:30,360 line you can send input to that program 266 00:09:27,140 --> 00:09:32,820 and then you can expect so child.expects 267 00:09:30,360 --> 00:09:33,540 the peer expect eof end of file just 268 00:09:32,820 --> 00:09:35,160 means that you're going to wait until 269 00:09:33,540 --> 00:09:37,019 that program ends so in this case it's 270 00:09:35,160 --> 00:09:40,140 very simple that big number exercise 271 00:09:37,019 --> 00:09:41,459 only requires us to send one line of 272 00:09:40,140 --> 00:09:43,140 text in and then just wait for the 273 00:09:41,459 --> 00:09:44,940 program to end 274 00:09:43,140 --> 00:09:46,440 afterwards you can collect the output 275 00:09:44,940 --> 00:09:50,120 it's in byte format and then you can 276 00:09:46,440 --> 00:09:50,120 decode it into something sensible 277 00:09:52,440 --> 00:09:56,940 so here's a little cheat sheet of things 278 00:09:54,300 --> 00:09:58,560 you might want to use or do with peer 279 00:09:56,940 --> 00:10:01,019 expects so spawning a child process 280 00:09:58,560 --> 00:10:02,160 number one thing sending lines of text 281 00:10:01,019 --> 00:10:05,940 as well 282 00:10:02,160 --> 00:10:07,320 you can expect things that are quite 283 00:10:05,940 --> 00:10:10,019 interesting so you can use regular 284 00:10:07,320 --> 00:10:12,240 Expressions to wait for output so you 285 00:10:10,019 --> 00:10:14,220 can wait until the program from your 286 00:10:12,240 --> 00:10:16,620 student actually matches a regular 287 00:10:14,220 --> 00:10:19,560 expression if you want or if you know 288 00:10:16,620 --> 00:10:21,839 that it's not going to be changing you 289 00:10:19,560 --> 00:10:23,100 can expect an exact string as shown at 290 00:10:21,839 --> 00:10:24,660 the bottom so if you wanted to the 291 00:10:23,100 --> 00:10:28,700 student to Output hello world you could 292 00:10:24,660 --> 00:10:28,700 expect exactly that string 293 00:10:29,880 --> 00:10:34,740 so this gets us onto the pathway of 294 00:10:32,940 --> 00:10:38,880 those of defeating those earlier 295 00:10:34,740 --> 00:10:41,220 problems so if we want to input a random 296 00:10:38,880 --> 00:10:43,019 number into our students program like we 297 00:10:41,220 --> 00:10:45,120 didn't win with those earlier test cases 298 00:10:43,019 --> 00:10:47,880 we only put in four numbers for them to 299 00:10:45,120 --> 00:10:50,880 test and they could then counter that 300 00:10:47,880 --> 00:10:53,339 we can instead feed in a random number 301 00:10:50,880 --> 00:10:54,480 from Python's random library or maybe if 302 00:10:53,339 --> 00:10:56,579 you want to use numpy or something like 303 00:10:54,480 --> 00:10:58,560 that you can do it too so the first 304 00:10:56,579 --> 00:11:01,220 example at the top is showing how to 305 00:10:58,560 --> 00:11:03,839 send in a random number 306 00:11:01,220 --> 00:11:04,860 with the the heart to defeat hardcoding 307 00:11:03,839 --> 00:11:06,360 you probably also want to repeat these 308 00:11:04,860 --> 00:11:08,100 tests a few times not just send in one 309 00:11:06,360 --> 00:11:09,540 random number they might get lucky so 310 00:11:08,100 --> 00:11:10,800 you might want to repeat your tests as 311 00:11:09,540 --> 00:11:12,540 well 312 00:11:10,800 --> 00:11:15,060 here's an example at the bottom of how 313 00:11:12,540 --> 00:11:17,820 you could send in some random text so on 314 00:11:15,060 --> 00:11:20,100 most sort of uh Linux or Unix 315 00:11:17,820 --> 00:11:22,860 environments you there's a file called 316 00:11:20,100 --> 00:11:24,420 words under user shared dict and then 317 00:11:22,860 --> 00:11:25,980 you can just randomly sample from those 318 00:11:24,420 --> 00:11:27,240 words and send them in as well if you if 319 00:11:25,980 --> 00:11:30,200 you wanted to send it some random text 320 00:11:27,240 --> 00:11:30,200 to your students program 321 00:11:34,440 --> 00:11:38,459 I'm going to duplicate side there it 322 00:11:36,660 --> 00:11:41,040 also allows you to tolerate Randomness 323 00:11:38,459 --> 00:11:43,740 so in the example of that Monte Carlo 324 00:11:41,040 --> 00:11:45,899 simulation doing the integration 325 00:11:43,740 --> 00:11:47,100 this is a regular expression that you 326 00:11:45,899 --> 00:11:50,399 might use this is going to give you 327 00:11:47,100 --> 00:11:54,060 allow a student to report a result by 328 00:11:50,399 --> 00:11:57,660 printing it out that's a a digit a 329 00:11:54,060 --> 00:11:59,339 number up to four decimal places this is 330 00:11:57,660 --> 00:12:00,959 just a simple example you might have 331 00:11:59,339 --> 00:12:03,839 something that's tailored more to your 332 00:12:00,959 --> 00:12:04,920 needs and you can help your students a 333 00:12:03,839 --> 00:12:06,300 little bit here or restrict the 334 00:12:04,920 --> 00:12:07,800 complexity of your regular Expressions 335 00:12:06,300 --> 00:12:10,680 by saying that okay you need to print 336 00:12:07,800 --> 00:12:14,420 out the final answer to a specific 337 00:12:10,680 --> 00:12:14,420 number of decimal places and so on 338 00:12:16,440 --> 00:12:20,399 so how do you actually do more 339 00:12:17,820 --> 00:12:23,100 complicated interaction this requires a 340 00:12:20,399 --> 00:12:25,680 little bit more code 341 00:12:23,100 --> 00:12:29,220 the strategy that we've adopted is to 342 00:12:25,680 --> 00:12:32,399 have a file maybe it's called inputs.txt 343 00:12:29,220 --> 00:12:33,779 that lists on each line all the inputs 344 00:12:32,399 --> 00:12:36,720 that you're going to send into your 345 00:12:33,779 --> 00:12:38,940 students code so we can just read that 346 00:12:36,720 --> 00:12:42,240 in and split it into the separate lines 347 00:12:38,940 --> 00:12:44,940 and then use just a quick little Loop to 348 00:12:42,240 --> 00:12:46,019 send in those new lines so the first 349 00:12:44,940 --> 00:12:49,320 thing we're going to do is just wait 350 00:12:46,019 --> 00:12:51,720 expect a new line maybe you the way we 351 00:12:49,320 --> 00:12:53,519 tend to structure our um or the ways I 352 00:12:51,720 --> 00:12:54,480 have been structuring the programs is to 353 00:12:53,519 --> 00:12:56,820 get the student to print something out 354 00:12:54,480 --> 00:13:00,060 the first time and then wait for the uh 355 00:12:56,820 --> 00:13:02,940 Slash r n which is the new line or End 356 00:13:00,060 --> 00:13:04,680 of Line character set and then 357 00:13:02,940 --> 00:13:06,540 afterwards at the end buried at the 358 00:13:04,680 --> 00:13:08,279 bottom is to send in the next line of 359 00:13:06,540 --> 00:13:09,720 text so this way you can just feed in a 360 00:13:08,279 --> 00:13:11,399 whole bunch of text and just listen to 361 00:13:09,720 --> 00:13:13,220 see what your student has produced at 362 00:13:11,399 --> 00:13:15,959 the end 363 00:13:13,220 --> 00:13:17,100 afterwards you could then using the the 364 00:13:15,959 --> 00:13:19,920 um the example that I gave before 365 00:13:17,100 --> 00:13:20,880 collect all the output and compare it to 366 00:13:19,920 --> 00:13:23,519 um to see whether it was what you 367 00:13:20,880 --> 00:13:26,220 expected if you wanted to this example 368 00:13:23,519 --> 00:13:27,480 sorry this example will just spew in a 369 00:13:26,220 --> 00:13:29,040 bunch of text into the students program 370 00:13:27,480 --> 00:13:31,019 and it doesn't care what the student 371 00:13:29,040 --> 00:13:32,160 puts back every time if you cared what 372 00:13:31,019 --> 00:13:33,300 the student gave you back every time 373 00:13:32,160 --> 00:13:35,519 then you might have a corresponding 374 00:13:33,300 --> 00:13:37,200 outputs file and then sort of go Tit for 375 00:13:35,519 --> 00:13:38,240 Tat with the inputs outputs file in your 376 00:13:37,200 --> 00:13:41,040 Loop 377 00:13:38,240 --> 00:13:42,540 and maybe if they failed at a certain 378 00:13:41,040 --> 00:13:45,680 point you could stop them early if you 379 00:13:42,540 --> 00:13:45,680 want to stop their program early 380 00:13:46,139 --> 00:13:51,060 okay so that's the the relatively 381 00:13:48,000 --> 00:13:52,860 straightforward stuff we do encounter 382 00:13:51,060 --> 00:13:54,779 more difficult problems which I want to 383 00:13:52,860 --> 00:13:57,240 talk about now and this is the stuff 384 00:13:54,779 --> 00:14:00,959 that is a little bit more Niche and 385 00:13:57,240 --> 00:14:03,420 maybe more applicable to higher levels 386 00:14:00,959 --> 00:14:07,139 of Education in in high school like is 387 00:14:03,420 --> 00:14:11,519 11 and 12 or university onwards 388 00:14:07,139 --> 00:14:14,880 so cases where you have going back to 389 00:14:11,519 --> 00:14:16,740 the checking bank account example doing 390 00:14:14,880 --> 00:14:17,820 a diff based test is not going to reveal 391 00:14:16,740 --> 00:14:19,920 whether a student has actually 392 00:14:17,820 --> 00:14:24,660 implemented the child class Constructor 393 00:14:19,920 --> 00:14:28,200 properly but you can use Python's unit 394 00:14:24,660 --> 00:14:30,779 test library to help with that so you 395 00:14:28,200 --> 00:14:33,779 can patch the parent class Constructor 396 00:14:30,779 --> 00:14:37,200 in this case bank account init 397 00:14:33,779 --> 00:14:40,380 and listen to see whether the the uh 398 00:14:37,200 --> 00:14:43,139 that was called by the child class 399 00:14:40,380 --> 00:14:44,699 so here's an example of doing that so 400 00:14:43,139 --> 00:14:46,980 um to do the monkey patch you just use 401 00:14:44,699 --> 00:14:48,839 this nice decorator from the unit test 402 00:14:46,980 --> 00:14:52,680 mock Library 403 00:14:48,839 --> 00:14:54,540 you supply the name of the function that 404 00:14:52,680 --> 00:14:57,480 you want to actually patch 405 00:14:54,540 --> 00:14:59,940 and then associate that with a function 406 00:14:57,480 --> 00:15:02,579 in this case it's called test super and 407 00:14:59,940 --> 00:15:04,860 inside that function I'm just creating a 408 00:15:02,579 --> 00:15:06,720 new instance of the child class and then 409 00:15:04,860 --> 00:15:09,240 after it checking whether the child 410 00:15:06,720 --> 00:15:12,060 class was actually called so mock super 411 00:15:09,240 --> 00:15:14,220 assert called and that refers back to 412 00:15:12,060 --> 00:15:16,079 the the name of the function in Grain 413 00:15:14,220 --> 00:15:18,300 and then to run this you just run it as 414 00:15:16,079 --> 00:15:20,399 a normal function like test super 415 00:15:18,300 --> 00:15:22,139 so that way you can get an error if the 416 00:15:20,399 --> 00:15:24,240 chart if the student hasn't actually 417 00:15:22,139 --> 00:15:26,399 called the function that you expected 418 00:15:24,240 --> 00:15:28,639 them to call 419 00:15:26,399 --> 00:15:30,540 this doesn't just isn't just limited to 420 00:15:28,639 --> 00:15:32,940 object-oriented programming you could 421 00:15:30,540 --> 00:15:34,740 apply this to almost anything you can 422 00:15:32,940 --> 00:15:36,980 think of that is in the form of a 423 00:15:34,740 --> 00:15:36,980 function 424 00:15:40,320 --> 00:15:45,540 the next issue is literal problems 425 00:15:43,620 --> 00:15:47,279 um 426 00:15:45,540 --> 00:15:49,500 sometimes you might want to prevent 427 00:15:47,279 --> 00:15:50,760 students from using things like you 428 00:15:49,500 --> 00:15:53,820 might want to prevent a student from 429 00:15:50,760 --> 00:15:55,620 using a or the built-in python 430 00:15:53,820 --> 00:15:57,660 dictionary for example so going back to 431 00:15:55,620 --> 00:15:59,579 the earlier data storage program that 432 00:15:57,660 --> 00:16:01,980 long interactive program part of that 433 00:15:59,579 --> 00:16:03,959 exercise was to actually get students to 434 00:16:01,980 --> 00:16:06,300 implement a subset of the Python 435 00:16:03,959 --> 00:16:08,399 dictionary class itself but I didn't 436 00:16:06,300 --> 00:16:10,620 want students to just wrap the 437 00:16:08,399 --> 00:16:12,600 dictionary class and hide it in their 438 00:16:10,620 --> 00:16:15,180 own little implementation I wanted them 439 00:16:12,600 --> 00:16:16,500 to try and apply Concepts that they'd 440 00:16:15,180 --> 00:16:19,079 been learned they've taught and been 441 00:16:16,500 --> 00:16:20,940 learned sorry they'd learned in class to 442 00:16:19,079 --> 00:16:23,040 actually Implement some of a dictionary 443 00:16:20,940 --> 00:16:24,839 themselves but what I discovered was 444 00:16:23,040 --> 00:16:27,600 that students were actually just 445 00:16:24,839 --> 00:16:29,820 creating an empty dictionary as a 446 00:16:27,600 --> 00:16:32,339 literal and then using that as their 447 00:16:29,820 --> 00:16:34,680 sort of backhand and hiding their sort 448 00:16:32,339 --> 00:16:37,440 of deviousness away from me 449 00:16:34,680 --> 00:16:39,779 so once I discovered that I did some 450 00:16:37,440 --> 00:16:41,579 research into this problem and the 451 00:16:39,779 --> 00:16:45,300 solution that I've come up for this is 452 00:16:41,579 --> 00:16:48,000 to use the abstract syntax tree and this 453 00:16:45,300 --> 00:16:50,699 is a little bit more scary so just a 454 00:16:48,000 --> 00:16:52,560 quick mention to explain what the 455 00:16:50,699 --> 00:16:53,940 abstract syntax tree is the python 456 00:16:52,560 --> 00:16:55,920 interpreter will convert all your python 457 00:16:53,940 --> 00:16:58,560 code into the abstracts into an abstract 458 00:16:55,920 --> 00:17:02,040 syntax tree format before it gets turned 459 00:16:58,560 --> 00:17:07,079 into bytecode so you can leverage that 460 00:17:02,040 --> 00:17:09,780 with Python's built-in AST module 461 00:17:07,079 --> 00:17:13,740 to actually inspect what the student's 462 00:17:09,780 --> 00:17:17,419 code is doing and detect things like the 463 00:17:13,740 --> 00:17:19,919 creation of a literal dictionary 464 00:17:17,419 --> 00:17:21,480 so this is the solution that I've come 465 00:17:19,919 --> 00:17:23,459 up with because monkey patching is not 466 00:17:21,480 --> 00:17:25,140 going to work in this case because it's 467 00:17:23,459 --> 00:17:27,419 treated somewhat differently to a 468 00:17:25,140 --> 00:17:29,340 function in Python 469 00:17:27,419 --> 00:17:32,220 so here's an example of how you might 470 00:17:29,340 --> 00:17:35,460 iterate over a student's class 471 00:17:32,220 --> 00:17:37,740 definition to 472 00:17:35,460 --> 00:17:40,559 deep drilled drill in and dig down and 473 00:17:37,740 --> 00:17:43,620 see if there is a creation of a 474 00:17:40,559 --> 00:17:46,020 dictionary in there somewhere and you 475 00:17:43,620 --> 00:17:49,320 could put this in as a supplemental test 476 00:17:46,020 --> 00:17:51,299 case to say uh I detected that you are 477 00:17:49,320 --> 00:17:55,280 actually cheating you need to go back 478 00:17:51,299 --> 00:17:55,280 and fix that or change your code 479 00:18:04,500 --> 00:18:09,600 and I think this brings us to the last 480 00:18:06,860 --> 00:18:10,500 technique I recommend less last 481 00:18:09,600 --> 00:18:11,700 technique before I get to the 482 00:18:10,500 --> 00:18:14,220 recommendations 483 00:18:11,700 --> 00:18:15,539 is checking for illegal Imports so if 484 00:18:14,220 --> 00:18:19,140 you want to stop your student from using 485 00:18:15,539 --> 00:18:21,480 numpy or pandas or something like that 486 00:18:19,140 --> 00:18:24,600 to take a shortcut then you can use the 487 00:18:21,480 --> 00:18:25,400 built-in module finder Library so this 488 00:18:24,600 --> 00:18:27,960 is 489 00:18:25,400 --> 00:18:29,880 relatively straightforward to use but I 490 00:18:27,960 --> 00:18:31,799 think it's a little bit unknown 491 00:18:29,880 --> 00:18:33,919 I think a lot of python programmers out 492 00:18:31,799 --> 00:18:36,840 there don't know that this exists 493 00:18:33,919 --> 00:18:38,100 so it's quite straightforward to use you 494 00:18:36,840 --> 00:18:40,740 just need to create a module finder 495 00:18:38,100 --> 00:18:42,840 object evaluate that so run the script 496 00:18:40,740 --> 00:18:45,900 whatever that script is you just refer 497 00:18:42,840 --> 00:18:48,419 to it by its name and then you can go 498 00:18:45,900 --> 00:18:51,059 through the list of modules that were 499 00:18:48,419 --> 00:18:52,620 imported and if that one of the matches 500 00:18:51,059 --> 00:18:54,960 one of the ones that you want to ban you 501 00:18:52,620 --> 00:18:57,260 can then fail the student on that test 502 00:18:54,960 --> 00:18:57,260 case 503 00:18:59,039 --> 00:19:02,940 okay 504 00:19:00,660 --> 00:19:04,679 um onto recommendations before before I 505 00:19:02,940 --> 00:19:07,919 get there I want to point out that these 506 00:19:04,679 --> 00:19:11,700 are examples of starting points for 507 00:19:07,919 --> 00:19:13,440 these sort of detection strategies I 508 00:19:11,700 --> 00:19:16,140 don't recommend that they are always 509 00:19:13,440 --> 00:19:18,000 used and this should only be seen as 510 00:19:16,140 --> 00:19:19,980 sort of a starting point for your 511 00:19:18,000 --> 00:19:20,960 journey in the sort of automatic testing 512 00:19:19,980 --> 00:19:23,580 space 513 00:19:20,960 --> 00:19:25,860 don't feel like you shouldn't do it 514 00:19:23,580 --> 00:19:27,600 because you don't see any value in doing 515 00:19:25,860 --> 00:19:29,640 this hard stuff even some of the easy 516 00:19:27,600 --> 00:19:32,220 stuff is quite important so I don't want 517 00:19:29,640 --> 00:19:35,240 to I want you want you to remember that 518 00:19:32,220 --> 00:19:39,240 earlier easy examples that I showed you 519 00:19:35,240 --> 00:19:40,700 so the first recommendation is think 520 00:19:39,240 --> 00:19:44,400 first so 521 00:19:40,700 --> 00:19:45,539 this automatic testing procedure and 522 00:19:44,400 --> 00:19:47,220 detecting 523 00:19:45,539 --> 00:19:49,980 um sort of irregularities in students 524 00:19:47,220 --> 00:19:52,200 code or devious students is time 525 00:19:49,980 --> 00:19:53,640 consuming so it may not be necessarily 526 00:19:52,200 --> 00:19:56,160 worth your time doing you might think oh 527 00:19:53,640 --> 00:19:58,799 I saw Steven's talk I'm going to go try 528 00:19:56,160 --> 00:20:00,780 it out but you might spend far too long 529 00:19:58,799 --> 00:20:01,919 doing it and not get any results and it 530 00:20:00,780 --> 00:20:03,179 might not actually be applicable to your 531 00:20:01,919 --> 00:20:05,700 students so you need to think about 532 00:20:03,179 --> 00:20:08,940 whether it's actually worthwhile 533 00:20:05,700 --> 00:20:12,299 there are also going to be imperfections 534 00:20:08,940 --> 00:20:14,220 in your test cases and these test cases 535 00:20:12,299 --> 00:20:16,620 tend to be a little bit of a hurdle for 536 00:20:14,220 --> 00:20:18,059 students students will come to you and 537 00:20:16,620 --> 00:20:20,700 complain and say hey I couldn't pass 538 00:20:18,059 --> 00:20:22,140 this test case and 539 00:20:20,700 --> 00:20:24,600 often the case with his automatic 540 00:20:22,140 --> 00:20:26,460 marking system is that students will see 541 00:20:24,600 --> 00:20:28,740 the feedback from the market and say 542 00:20:26,460 --> 00:20:31,380 that they see that they failed and they 543 00:20:28,740 --> 00:20:33,480 won't think to look at their own code to 544 00:20:31,380 --> 00:20:35,700 sort out the problem they'll come to the 545 00:20:33,480 --> 00:20:37,799 teaching staff first to try and fix 546 00:20:35,700 --> 00:20:39,660 their issues so you may have to deal 547 00:20:37,799 --> 00:20:41,240 with students coming back to you a 548 00:20:39,660 --> 00:20:43,320 little bit too soon 549 00:20:41,240 --> 00:20:45,539 and you might have to send them away and 550 00:20:43,320 --> 00:20:46,919 say hey you need to go and try again 551 00:20:45,539 --> 00:20:48,600 um you've been given some information 552 00:20:46,919 --> 00:20:50,280 about how to fix that 553 00:20:48,600 --> 00:20:52,020 so just be aware that there's a 554 00:20:50,280 --> 00:20:54,299 additional time complexity of students 555 00:20:52,020 --> 00:20:55,919 complaining at you all the time you 556 00:20:54,299 --> 00:20:57,720 might want to start with simple tests as 557 00:20:55,919 --> 00:20:59,340 well I don't think there's any real need 558 00:20:57,720 --> 00:21:01,559 to start with really complicated tests 559 00:20:59,340 --> 00:21:04,679 this is again ties into the time issue 560 00:21:01,559 --> 00:21:06,000 you don't have to write crazy tests you 561 00:21:04,679 --> 00:21:07,440 should think about what is going to 562 00:21:06,000 --> 00:21:09,360 bring you the most value in terms of 563 00:21:07,440 --> 00:21:11,160 your time and also the most benefit to 564 00:21:09,360 --> 00:21:13,320 your students 565 00:21:11,160 --> 00:21:15,299 um and you can also limit yourselves you 566 00:21:13,320 --> 00:21:17,220 don't have to deploy this across all of 567 00:21:15,299 --> 00:21:18,960 your assessment items or all of your 568 00:21:17,220 --> 00:21:21,360 exercises or all your homework at once 569 00:21:18,960 --> 00:21:23,760 maybe one week out of a term or a 570 00:21:21,360 --> 00:21:26,220 semester you might try this out and then 571 00:21:23,760 --> 00:21:28,140 slowly over time you can come back to 572 00:21:26,220 --> 00:21:31,080 your test cases and improve them or 573 00:21:28,140 --> 00:21:33,720 extend them to broader parts of your 574 00:21:31,080 --> 00:21:35,700 assessment strategy 575 00:21:33,720 --> 00:21:37,640 the other thing is that I wouldn't 576 00:21:35,700 --> 00:21:41,280 recommend this for 577 00:21:37,640 --> 00:21:43,679 everyone everywhere or all students and 578 00:21:41,280 --> 00:21:45,659 it's not a complete replacement for sort 579 00:21:43,679 --> 00:21:46,700 of a human connection and feedback from 580 00:21:45,659 --> 00:21:50,520 a person 581 00:21:46,700 --> 00:21:51,780 so don't think that if you're well 582 00:21:50,520 --> 00:21:53,340 particularly for us it's attractive 583 00:21:51,780 --> 00:21:55,980 because we have large number of students 584 00:21:53,340 --> 00:21:57,480 but even still we think that it's 585 00:21:55,980 --> 00:21:59,400 actually quite helpful to get feedback 586 00:21:57,480 --> 00:22:02,340 directly from your the instructor the 587 00:21:59,400 --> 00:22:03,720 teacher person to person rather than 588 00:22:02,340 --> 00:22:05,600 just having a computer yell at them that 589 00:22:03,720 --> 00:22:08,460 they got something wrong 590 00:22:05,600 --> 00:22:10,380 the other thing is that most students 591 00:22:08,460 --> 00:22:12,179 will do the right thing so the devious 592 00:22:10,380 --> 00:22:14,700 students are usually a small proportion 593 00:22:12,179 --> 00:22:16,559 of your students and actually throwing 594 00:22:14,700 --> 00:22:19,140 these test cases at all of the students 595 00:22:16,559 --> 00:22:20,280 can be a negative thing so you need to 596 00:22:19,140 --> 00:22:22,500 again evaluate whether it's going to be 597 00:22:20,280 --> 00:22:25,919 worthwhile to do 598 00:22:22,500 --> 00:22:27,600 uh and lastly you may accidentally 599 00:22:25,919 --> 00:22:30,120 narrow the path 600 00:22:27,600 --> 00:22:32,460 so often we want students to be creative 601 00:22:30,120 --> 00:22:35,580 and to be thinking for themselves but 602 00:22:32,460 --> 00:22:38,220 sometimes these test cases make students 603 00:22:35,580 --> 00:22:39,780 sort of funnel down a path towards a 604 00:22:38,220 --> 00:22:41,100 standardized solution which you might 605 00:22:39,780 --> 00:22:42,600 not want 606 00:22:41,100 --> 00:22:44,580 um and it's not necessarily true that 607 00:22:42,600 --> 00:22:46,200 students have to follow a narrower path 608 00:22:44,580 --> 00:22:48,539 it's just more that they feel that way 609 00:22:46,200 --> 00:22:50,580 or perhaps at prior examples that 610 00:22:48,539 --> 00:22:53,640 they've learned from lead them down that 611 00:22:50,580 --> 00:22:55,860 narrow path with your test cases 612 00:22:53,640 --> 00:22:57,179 so just be aware that sometimes your 613 00:22:55,860 --> 00:22:58,620 test cases can be a little bit too 614 00:22:57,179 --> 00:23:01,500 strict and you need to make a trade-off 615 00:22:58,620 --> 00:23:04,280 between those strict test cases and your 616 00:23:01,500 --> 00:23:04,280 students needs 617 00:23:04,799 --> 00:23:13,199 okay that's all for today thank you 618 00:23:08,780 --> 00:23:13,199 [Applause] 619 00:23:14,640 --> 00:23:18,539 thank you very much Stephen do we have 620 00:23:16,679 --> 00:23:21,980 questions in the room gosh we have many 621 00:23:18,539 --> 00:23:21,980 questions in the room let's go 622 00:23:22,380 --> 00:23:30,299 okay a great talk my question was around 623 00:23:27,299 --> 00:23:33,059 um when these are used for assessments 624 00:23:30,299 --> 00:23:34,620 do you give the students all of the test 625 00:23:33,059 --> 00:23:36,179 cases beforehand like do you let them 626 00:23:34,620 --> 00:23:38,039 run all of the tests and kind of 627 00:23:36,179 --> 00:23:39,600 essentially know exactly what Mark 628 00:23:38,039 --> 00:23:41,880 they're getting before they submit do 629 00:23:39,600 --> 00:23:44,159 you give them a subset or once you go 630 00:23:41,880 --> 00:23:47,100 there uh it varies 631 00:23:44,159 --> 00:23:50,159 um it's normally well the test cases are 632 00:23:47,100 --> 00:23:52,679 actually for us in our tutorials as well 633 00:23:50,159 --> 00:23:54,120 so we have them in tutorials 634 00:23:52,679 --> 00:23:55,919 um almost all the exercises are covered 635 00:23:54,120 --> 00:23:57,720 by test cases 636 00:23:55,919 --> 00:23:59,880 um and in the assessments for my 637 00:23:57,720 --> 00:24:02,400 assessments personally I kind of run 638 00:23:59,880 --> 00:24:04,679 them as a bit of a test fail or you 639 00:24:02,400 --> 00:24:06,720 succeed or you um don't there's no sort 640 00:24:04,679 --> 00:24:08,340 of marking in between them and then but 641 00:24:06,720 --> 00:24:09,840 the trade-off with that is that I give 642 00:24:08,340 --> 00:24:12,059 the students quite a lot of help to get 643 00:24:09,840 --> 00:24:13,620 over the hurdle I can say this is going 644 00:24:12,059 --> 00:24:15,120 to be difficult for you I know it's 645 00:24:13,620 --> 00:24:17,340 going to be difficult I'm going to give 646 00:24:15,120 --> 00:24:19,140 you lots of help and you you there you 647 00:24:17,340 --> 00:24:21,059 can see the test cases straight away as 648 00:24:19,140 --> 00:24:23,280 soon as I unlock the 649 00:24:21,059 --> 00:24:24,840 um the homework or the assessment they 650 00:24:23,280 --> 00:24:28,280 can see it they can try hacking against 651 00:24:24,840 --> 00:24:28,280 the test cases immediately 652 00:24:30,600 --> 00:24:34,100 still a question over here yes 653 00:24:37,200 --> 00:24:40,320 I've got like three hopefully they're 654 00:24:39,000 --> 00:24:42,419 quick 655 00:24:40,320 --> 00:24:44,400 um first of all how is this deployed uh 656 00:24:42,419 --> 00:24:47,700 is it like all on replay do you have to 657 00:24:44,400 --> 00:24:49,320 create an API how does this it's on Ed 658 00:24:47,700 --> 00:24:51,720 um I could give you a demo afterwards 659 00:24:49,320 --> 00:24:54,900 but um yeah if they have a system where 660 00:24:51,720 --> 00:24:56,159 you can uh input information about uh an 661 00:24:54,900 --> 00:24:58,799 exercise so you give the description 662 00:24:56,159 --> 00:25:01,200 information which I showed uh was an 663 00:24:58,799 --> 00:25:02,280 example of a description here's an 664 00:25:01,200 --> 00:25:04,260 example of a description on the left 665 00:25:02,280 --> 00:25:06,600 that the student sees and then on the 666 00:25:04,260 --> 00:25:07,919 back end as well we can put in uh 667 00:25:06,600 --> 00:25:10,200 information about our marketing 668 00:25:07,919 --> 00:25:13,140 procedure and with ads they allow us to 669 00:25:10,200 --> 00:25:14,520 just run an arbitrary bit of code to do 670 00:25:13,140 --> 00:25:17,159 it to do the marking and we just have to 671 00:25:14,520 --> 00:25:20,400 return the results to add in a specific 672 00:25:17,159 --> 00:25:21,900 format they give you uh it's in Json and 673 00:25:20,400 --> 00:25:23,940 they say you need to return your marking 674 00:25:21,900 --> 00:25:26,220 to the ad system from your marking 675 00:25:23,940 --> 00:25:29,760 script in that format and they do all 676 00:25:26,220 --> 00:25:31,640 the rest Json Ed there's some that is an 677 00:25:29,760 --> 00:25:34,980 online education platform okay 678 00:25:31,640 --> 00:25:36,960 edstem.org Ed stem and the feedback that 679 00:25:34,980 --> 00:25:38,400 students get it's in entirely after the 680 00:25:36,960 --> 00:25:39,779 problem is submitted or is there 681 00:25:38,400 --> 00:25:42,960 anything ongoing while they're building 682 00:25:39,779 --> 00:25:46,080 it uh no so for the students interface 683 00:25:42,960 --> 00:25:47,100 basically they have two options run and 684 00:25:46,080 --> 00:25:49,200 then they can sort of interact with 685 00:25:47,100 --> 00:25:50,760 their own program or they can Mark and 686 00:25:49,200 --> 00:25:53,279 then the Mark will then marking button 687 00:25:50,760 --> 00:25:55,740 will run our marking code and then they 688 00:25:53,279 --> 00:25:58,380 will be presented with the example of a 689 00:25:55,740 --> 00:25:59,760 feedback like these test cases here so 690 00:25:58,380 --> 00:26:01,200 you've passed all the test cases right 691 00:25:59,760 --> 00:26:02,760 so they get the feedback after they run 692 00:26:01,200 --> 00:26:05,460 off yeah after they click the mark 693 00:26:02,760 --> 00:26:07,799 button yep okay so you can't really run 694 00:26:05,460 --> 00:26:09,179 the your regex thing in a loop and tell 695 00:26:07,799 --> 00:26:11,039 them hey you're ticking off you've done 696 00:26:09,179 --> 00:26:13,200 this and then this and then this well we 697 00:26:11,039 --> 00:26:14,340 can do uh in our marketing code we can 698 00:26:13,200 --> 00:26:15,840 do anything we want we can run the test 699 00:26:14,340 --> 00:26:17,279 cases multiple times if we wanted to 700 00:26:15,840 --> 00:26:19,140 like we can reevaluate the student's 701 00:26:17,279 --> 00:26:21,480 code each of those test cases you should 702 00:26:19,140 --> 00:26:26,220 think of as like uh so the test case two 703 00:26:21,480 --> 00:26:29,100 107 199 that would be us running our own 704 00:26:26,220 --> 00:26:30,900 marking script for that test case uh and 705 00:26:29,100 --> 00:26:32,520 we within that we can do whatever we 706 00:26:30,900 --> 00:26:34,200 like basically cool and the last 707 00:26:32,520 --> 00:26:37,620 question is one word answer there was a 708 00:26:34,200 --> 00:26:40,860 special library for check King o o 709 00:26:37,620 --> 00:26:43,799 program structure I forgot the name of 710 00:26:40,860 --> 00:26:45,720 program structure like object oriented 711 00:26:43,799 --> 00:26:46,740 uh well you could you can monkey patch 712 00:26:45,720 --> 00:26:49,799 things 713 00:26:46,740 --> 00:26:52,080 um to check that uh certain functions 714 00:26:49,799 --> 00:26:55,940 have been called as you expect as the 715 00:26:52,080 --> 00:27:00,260 instructor but then there's also the uh 716 00:26:55,940 --> 00:27:00,260 ASD yeah thank you 717 00:27:03,120 --> 00:27:07,440 thanks very much for the presentation I 718 00:27:05,279 --> 00:27:09,600 was curious to if you could provide more 719 00:27:07,440 --> 00:27:11,520 details on that narrowing of the path 720 00:27:09,600 --> 00:27:13,200 like coding is a very creative thing 721 00:27:11,520 --> 00:27:15,299 that really brings a lot of creativity 722 00:27:13,200 --> 00:27:18,120 from students 723 00:27:15,299 --> 00:27:20,460 um and when you try to put all these 724 00:27:18,120 --> 00:27:22,380 blockages like could have like some 725 00:27:20,460 --> 00:27:24,059 negative trade-off like sometimes brute 726 00:27:22,380 --> 00:27:26,159 force or hard coding things is the easy 727 00:27:24,059 --> 00:27:28,919 way but sometimes hard coding or using 728 00:27:26,159 --> 00:27:31,620 brute force it may be more complex and 729 00:27:28,919 --> 00:27:33,779 maybe may show that the person he or she 730 00:27:31,620 --> 00:27:35,460 may definitely know the stuff and he's 731 00:27:33,779 --> 00:27:37,200 just trying he or she may be trying to 732 00:27:35,460 --> 00:27:38,580 trick the system but at the end that's 733 00:27:37,200 --> 00:27:39,900 great for me if the person understands 734 00:27:38,580 --> 00:27:43,140 what they are doing well that's that's 735 00:27:39,900 --> 00:27:45,360 fantastic so you mentioned that you have 736 00:27:43,140 --> 00:27:47,279 to manage that trade-off but what can 737 00:27:45,360 --> 00:27:49,620 you do to avoid the risk of penalizing 738 00:27:47,279 --> 00:27:51,360 someone that really knows the stuff and 739 00:27:49,620 --> 00:27:54,840 it's just just try to trick the system 740 00:27:51,360 --> 00:27:57,480 yeah so for me my strategy is actually 741 00:27:54,840 --> 00:27:59,039 outside of this automatic marking it's 742 00:27:57,480 --> 00:28:02,340 to offer other assessment opportunities 743 00:27:59,039 --> 00:28:04,380 that are more free form so this I I 744 00:28:02,340 --> 00:28:07,020 haven't run a unit where this is the 745 00:28:04,380 --> 00:28:08,760 only way we assess I have always had 746 00:28:07,020 --> 00:28:10,799 other assessment items that give the 747 00:28:08,760 --> 00:28:12,960 students way more opportunity to express 748 00:28:10,799 --> 00:28:15,480 themselves and do what they like instead 749 00:28:12,960 --> 00:28:18,779 and meet a more I guess 750 00:28:15,480 --> 00:28:20,039 less well-defined goals and if as long 751 00:28:18,779 --> 00:28:21,659 as they meet those 752 00:28:20,039 --> 00:28:23,760 um according to our marketing rubric or 753 00:28:21,659 --> 00:28:25,919 whatever that's fine so that that's the 754 00:28:23,760 --> 00:28:28,980 opportunity for them to to take that 755 00:28:25,919 --> 00:28:31,320 creative path yeah 756 00:28:28,980 --> 00:28:33,779 okay thank you very much here is your 757 00:28:31,320 --> 00:28:37,039 mug and your thank you card can we give 758 00:28:33,779 --> 00:28:37,039 Steven another round of applause