1 00:00:05,239 --> 00:00:07,259 [music] 2 00:00:09,825 --> 00:00:11,845 [music] 3 00:00:15,679 --> 00:00:19,039 Okay. So, I'm here to talk about uh feed 4 00:00:19,039 --> 00:00:21,439 forward, not feedback. Uh rethinking 5 00:00:21,439 --> 00:00:25,439 projectbased assessment. So I'll start 6 00:00:25,439 --> 00:00:28,000 off with some context and then go over 7 00:00:28,000 --> 00:00:29,599 um a little bit about how the unit 8 00:00:29,599 --> 00:00:32,160 started uh and what I did to it, how I 9 00:00:32,160 --> 00:00:35,040 tried to change it to uh improve things 10 00:00:35,040 --> 00:00:38,960 and also how that plays with AI. So I 11 00:00:38,960 --> 00:00:40,879 teach this unit intro to dashboarding 12 00:00:40,879 --> 00:00:43,280 and data visualization. It's an elective 13 00:00:43,280 --> 00:00:45,200 unit at the University of Sydney for 14 00:00:45,200 --> 00:00:47,920 mostly masters of commerce students and 15 00:00:47,920 --> 00:00:49,600 it's a project-based unit requiring that 16 00:00:49,600 --> 00:00:51,680 students design and build a dashboard 17 00:00:51,680 --> 00:00:55,199 over the semester. And uh like I said, 18 00:00:55,199 --> 00:00:56,399 I'm going to go over how this has 19 00:00:56,399 --> 00:00:58,879 changed over time and how being o open 20 00:00:58,879 --> 00:01:01,199 to AI isn't sometimes the end of the 21 00:01:01,199 --> 00:01:03,280 world. 22 00:01:03,280 --> 00:01:05,360 So for more information about the unit, 23 00:01:05,360 --> 00:01:07,200 uh the first delivery or the first time 24 00:01:07,200 --> 00:01:10,960 it ran was in semester 2 of 2022. Uh we 25 00:01:10,960 --> 00:01:14,080 get about 400 to 500 students each year 26 00:01:14,080 --> 00:01:16,159 enrolling in this unit. Uh and 27 00:01:16,159 --> 00:01:17,439 interestingly being in the business 28 00:01:17,439 --> 00:01:20,000 school we get more female students or 29 00:01:20,000 --> 00:01:22,320 female identifying students than male. 30 00:01:22,320 --> 00:01:25,200 Uh and we have a the cohort is vastly 31 00:01:25,200 --> 00:01:27,759 international. Uh in descending order we 32 00:01:27,759 --> 00:01:29,680 have Chinese, Indian, uh Taiwanese and 33 00:01:29,680 --> 00:01:32,079 Vietnamese students. Most of them are 34 00:01:32,079 --> 00:01:34,640 from mainland China. Uh and we have 35 00:01:34,640 --> 00:01:36,720 about 50/50 commencing to continuing 36 00:01:36,720 --> 00:01:38,479 ratio. So some students are coming in, 37 00:01:38,479 --> 00:01:39,840 it's their first semester, they've 38 00:01:39,840 --> 00:01:41,520 picked this elective. Some of them are 39 00:01:41,520 --> 00:01:42,799 at the end of their degree and they 40 00:01:42,799 --> 00:01:45,600 they've picked this elective and as a 41 00:01:45,600 --> 00:01:48,640 result about half of them are coming in 42 00:01:48,640 --> 00:01:50,000 uh to the master of commerce or this 43 00:01:50,000 --> 00:01:52,079 this unit having done a specialization 44 00:01:52,079 --> 00:01:53,600 in business analytics and got some 45 00:01:53,600 --> 00:01:56,880 programming experience with Python and 46 00:01:56,880 --> 00:02:00,560 all assessments are open to AI use. 47 00:02:00,560 --> 00:02:02,479 So this is not something that I control 48 00:02:02,479 --> 00:02:03,840 myself. This is part of the University 49 00:02:03,840 --> 00:02:05,520 of Sydney's policy. So, there's a 50 00:02:05,520 --> 00:02:08,399 two-lane policy with open and closed 51 00:02:08,399 --> 00:02:10,080 assessments. Maybe we can talk more 52 00:02:10,080 --> 00:02:11,840 about that at the end or answer that 53 00:02:11,840 --> 00:02:14,000 during question time, but for now, we're 54 00:02:14,000 --> 00:02:15,120 going to assume that everything is open 55 00:02:15,120 --> 00:02:17,520 to AI use. And to the best of my 56 00:02:17,520 --> 00:02:18,959 knowledge, it's the largest university 57 00:02:18,959 --> 00:02:22,000 course that uses Python and Plotly Dash 58 00:02:22,000 --> 00:02:25,200 to teach dashboarding. 59 00:02:25,200 --> 00:02:27,040 Now the unit is quite challenging since 60 00:02:27,040 --> 00:02:28,879 building a good dashboard requires 61 00:02:28,879 --> 00:02:30,800 wide-ranging knowledge and skills in 62 00:02:30,800 --> 00:02:33,519 user experience, data visualization, web 63 00:02:33,519 --> 00:02:35,920 technologies, and programming. 64 00:02:35,920 --> 00:02:38,239 Overall, students are quite scared at 65 00:02:38,239 --> 00:02:39,440 the start of the unit even though 66 00:02:39,440 --> 00:02:41,200 they've enrolled in it. In the first 67 00:02:41,200 --> 00:02:43,360 lecture, I tell them you guys will be 68 00:02:43,360 --> 00:02:45,519 doing this in Python and the reaction is 69 00:02:45,519 --> 00:02:48,319 usually shock. Um, most of them don't 70 00:02:48,319 --> 00:02:51,720 read the syllabus. 71 00:02:52,800 --> 00:02:55,599 So, how do we do this? Well, we use 72 00:02:55,599 --> 00:02:58,319 Python of course and plotly dash. Who 73 00:02:58,319 --> 00:03:00,480 here knows about plotly dash or who's 74 00:03:00,480 --> 00:03:02,480 heard of it? Got a few hands. Who here 75 00:03:02,480 --> 00:03:04,959 has actually used it? We've got a couple 76 00:03:04,959 --> 00:03:09,840 of people. Okay. So, plotly dash uh is 77 00:03:09,840 --> 00:03:11,920 uh a library where you can build and 78 00:03:11,920 --> 00:03:14,080 define your dashboard entirely in 79 00:03:14,080 --> 00:03:16,319 Python. You can also do it in R and 80 00:03:16,319 --> 00:03:18,000 JavaScript, but I'll just stick to 81 00:03:18,000 --> 00:03:20,720 talking about Python today. So your web 82 00:03:20,720 --> 00:03:22,800 layout, your HTML and things like that 83 00:03:22,800 --> 00:03:25,200 is defined in Python and functionality 84 00:03:25,200 --> 00:03:27,200 like user interaction, live updates to 85 00:03:27,200 --> 00:03:28,640 graphs and everything else also happens 86 00:03:28,640 --> 00:03:30,799 in Python and Dash translates your 87 00:03:30,799 --> 00:03:33,040 Python layout to a React front end um 88 00:03:33,040 --> 00:03:34,640 and hooks up all your callbacks to make 89 00:03:34,640 --> 00:03:38,280 all the magic happen. 90 00:03:38,799 --> 00:03:40,720 The other scary thing is I I'm this is 91 00:03:40,720 --> 00:03:42,400 like the first week lecture I'm that I'm 92 00:03:42,400 --> 00:03:44,159 giving to students. I tell them there's 93 00:03:44,159 --> 00:03:46,959 no PowerBI, no Tableau. And this is also 94 00:03:46,959 --> 00:03:49,040 even scarier for students. They some of 95 00:03:49,040 --> 00:03:50,319 them think that they come into this unit 96 00:03:50,319 --> 00:03:52,000 going, "Oh, I know PowerBI. I know 97 00:03:52,000 --> 00:03:54,319 Tableau. This unit's going to be easy 98 00:03:54,319 --> 00:03:58,080 for me." So, my philosophy is that once 99 00:03:58,080 --> 00:03:59,519 you've learned the hard way, you already 100 00:03:59,519 --> 00:04:02,959 know the easy way. But I'm not entirely 101 00:04:02,959 --> 00:04:05,840 dead inside. And we try as hard as 102 00:04:05,840 --> 00:04:07,599 possible to support our students through 103 00:04:07,599 --> 00:04:11,640 this very challenging process. 104 00:04:12,239 --> 00:04:13,760 There are four motivating goals that I 105 00:04:13,760 --> 00:04:15,200 always have in the back of of my mind 106 00:04:15,200 --> 00:04:17,519 for this unit. I want them to work on 107 00:04:17,519 --> 00:04:19,040 something that makes the the world or 108 00:04:19,040 --> 00:04:21,280 their lives better. This is quite 109 00:04:21,280 --> 00:04:23,440 different from the abstract mathematical 110 00:04:23,440 --> 00:04:24,960 or fixed topic assignments a lot of the 111 00:04:24,960 --> 00:04:26,880 students have done in the past in a 112 00:04:26,880 --> 00:04:29,520 business analytics degree. 113 00:04:29,520 --> 00:04:31,199 I want them to gain real experience 114 00:04:31,199 --> 00:04:32,560 which involves wrestling with 115 00:04:32,560 --> 00:04:34,560 challenging or illdefined problems, 116 00:04:34,560 --> 00:04:36,160 building a product from start to finish 117 00:04:36,160 --> 00:04:38,960 and working in a team. And I want them 118 00:04:38,960 --> 00:04:41,680 to gain and uh confidence and succeed. 119 00:04:41,680 --> 00:04:43,600 And lastly, I want them to walk away 120 00:04:43,600 --> 00:04:45,440 with uh from the unit with a tangible 121 00:04:45,440 --> 00:04:48,479 dashboard that they can use uh as part 122 00:04:48,479 --> 00:04:50,960 of a portfolio or maybe as something of 123 00:04:50,960 --> 00:04:53,840 a talking point in a job interview. And 124 00:04:53,840 --> 00:04:55,360 I try to tell these exact things to my 125 00:04:55,360 --> 00:04:56,720 students at the start that this is what 126 00:04:56,720 --> 00:04:58,639 I want from you by the end and hopefully 127 00:04:58,639 --> 00:05:01,600 you can get on board. 128 00:05:01,600 --> 00:05:04,639 So back in 2022 when the uh unit first 129 00:05:04,639 --> 00:05:07,039 was delivered uh this is the basic 130 00:05:07,039 --> 00:05:10,080 structure that I had. 131 00:05:10,080 --> 00:05:12,000 I was thrust into teaching this unit 132 00:05:12,000 --> 00:05:14,160 back then as basically the only person 133 00:05:14,160 --> 00:05:16,160 in the department that was capable of 134 00:05:16,160 --> 00:05:18,000 delivering it. I'm sure that's a 135 00:05:18,000 --> 00:05:21,039 familiar experience for many teachers. 136 00:05:21,039 --> 00:05:23,120 So there was a single design brief or 137 00:05:23,120 --> 00:05:26,639 design proposal and lastly at the end of 138 00:05:26,639 --> 00:05:30,080 the semester a final big dashboard that 139 00:05:30,080 --> 00:05:32,320 they had to submit. 140 00:05:32,320 --> 00:05:34,320 At the bottom of my diagram hopefully 141 00:05:34,320 --> 00:05:36,880 you can read it. Um there's a homework 142 00:05:36,880 --> 00:05:38,720 sort of path that's along the side. So 143 00:05:38,720 --> 00:05:40,320 this is what I did to get students who 144 00:05:40,320 --> 00:05:41,919 weren't familiar with Python to pick up 145 00:05:41,919 --> 00:05:44,639 Python over the course of the semester. 146 00:05:44,639 --> 00:05:45,840 But I'm going to ignore this for the 147 00:05:45,840 --> 00:05:47,120 rest of the talk. This is just there for 148 00:05:47,120 --> 00:05:49,600 completeness at the moment to say that 149 00:05:49,600 --> 00:05:52,000 um there are some other assessment items 150 00:05:52,000 --> 00:05:55,039 along the way. 151 00:05:55,039 --> 00:05:57,120 So what happened in this first delivery? 152 00:05:57,120 --> 00:06:00,400 Um well there's a large deliverable and 153 00:06:00,400 --> 00:06:01,919 procrastination. 154 00:06:01,919 --> 00:06:04,319 So there were some disastrous outcomes 155 00:06:04,319 --> 00:06:06,560 for some students. 156 00:06:06,560 --> 00:06:08,160 It was also very difficult to give 157 00:06:08,160 --> 00:06:10,160 feedback to individual students because 158 00:06:10,160 --> 00:06:13,759 we had 400 to 500 students and those 159 00:06:13,759 --> 00:06:15,280 that the feedback that we did provide 160 00:06:15,280 --> 00:06:16,800 was at the end of the semester when 161 00:06:16,800 --> 00:06:18,240 students were already on holiday and not 162 00:06:18,240 --> 00:06:19,759 thinking about it and what's the point 163 00:06:19,759 --> 00:06:22,720 of the feedback if they can't act on it. 164 00:06:22,720 --> 00:06:24,080 Students were also pretty disconnected 165 00:06:24,080 --> 00:06:25,520 from one another and really didn't 166 00:06:25,520 --> 00:06:26,880 achieve graduate outcomes on 167 00:06:26,880 --> 00:06:30,600 communication and teamwork. 168 00:06:31,120 --> 00:06:33,680 And if you can squint, there's something 169 00:06:33,680 --> 00:06:36,479 off in the distance in 2022. 170 00:06:36,479 --> 00:06:39,520 It's the looming spectre of Sam Alman 171 00:06:39,520 --> 00:06:41,919 uh and GPT 3.5 172 00:06:41,919 --> 00:06:45,440 and behind that Anthropic and Claude. So 173 00:06:45,440 --> 00:06:46,720 it was around this time at the end of 174 00:06:46,720 --> 00:06:49,199 2022 that I created my first OpenAI 175 00:06:49,199 --> 00:06:51,840 account and I started to see in the 176 00:06:51,840 --> 00:06:53,680 classroom some students toying around 177 00:06:53,680 --> 00:06:55,919 with chat GBT, but they had pretty 178 00:06:55,919 --> 00:06:59,840 limited success at that point. 179 00:06:59,840 --> 00:07:02,319 So 2023 comes around and I have a new 180 00:07:02,319 --> 00:07:04,639 plan. 181 00:07:04,639 --> 00:07:06,080 The first step was to break the project 182 00:07:06,080 --> 00:07:08,240 down to further scaffold the development 183 00:07:08,240 --> 00:07:10,560 of their skills and emphasize the 184 00:07:10,560 --> 00:07:13,360 process of learning rather than the 185 00:07:13,360 --> 00:07:16,400 final product of the dashboard. So I 186 00:07:16,400 --> 00:07:18,240 deweeded the final dashboard that they 187 00:07:18,240 --> 00:07:21,599 were to deliver uh and upweed the other 188 00:07:21,599 --> 00:07:23,599 assessment items. 189 00:07:23,599 --> 00:07:25,280 The idea was to lower the final 190 00:07:25,280 --> 00:07:26,800 workload, get students to complete their 191 00:07:26,800 --> 00:07:29,120 work on time, and ultimately try to get 192 00:07:29,120 --> 00:07:33,280 a better quality dashboard in the end. 193 00:07:33,280 --> 00:07:35,120 I've got some information about the 194 00:07:35,120 --> 00:07:36,880 assignments. [snorts] 195 00:07:36,880 --> 00:07:38,720 So, the first assignment, which is a 196 00:07:38,720 --> 00:07:40,880 design brief or a proposal on the left, 197 00:07:40,880 --> 00:07:42,560 you can sort of see the the motivating 198 00:07:42,560 --> 00:07:43,759 statement, which is getting the students 199 00:07:43,759 --> 00:07:45,680 to come up with a proposal that's going 200 00:07:45,680 --> 00:07:47,039 to solve a problem for one or more 201 00:07:47,039 --> 00:07:49,280 users. And they have to find a problem 202 00:07:49,280 --> 00:07:51,360 themselves from areas like their 203 00:07:51,360 --> 00:07:52,880 personal life, hobbies and interests, 204 00:07:52,880 --> 00:07:55,120 their work, public service or needs like 205 00:07:55,120 --> 00:07:58,000 charity, society, government, etc. And 206 00:07:58,000 --> 00:08:00,080 then they have to go through the process 207 00:08:00,080 --> 00:08:01,680 of specifying the problem that they're 208 00:08:01,680 --> 00:08:03,280 going to work on, determine the 209 00:08:03,280 --> 00:08:05,759 requirements and draft the solution. 210 00:08:05,759 --> 00:08:07,919 This is a pretty non-committal stage and 211 00:08:07,919 --> 00:08:10,000 pie in the sky. Students can propose a 212 00:08:10,000 --> 00:08:12,240 dashboard that they have uh later on 213 00:08:12,240 --> 00:08:14,560 abandon or revise at later stages. So 214 00:08:14,560 --> 00:08:18,440 they're not locked into this. 215 00:08:20,080 --> 00:08:22,479 For the second assignment, students form 216 00:08:22,479 --> 00:08:24,319 groups and they're tasked with 217 00:08:24,319 --> 00:08:26,639 critically evaluating each other's first 218 00:08:26,639 --> 00:08:29,120 assignment. Marks are based on the 219 00:08:29,120 --> 00:08:31,360 quality of the critiques as a group, not 220 00:08:31,360 --> 00:08:33,839 what they originally submitted. 221 00:08:33,839 --> 00:08:35,440 So 222 00:08:35,440 --> 00:08:36,959 they're not going to affect each other's 223 00:08:36,959 --> 00:08:39,599 marks uh that way. 224 00:08:39,599 --> 00:08:40,959 Then they select which dashboard they 225 00:08:40,959 --> 00:08:42,399 want to work on and describe any changes 226 00:08:42,399 --> 00:08:43,599 that they're going to make because maybe 227 00:08:43,599 --> 00:08:45,040 their initial idea needed some 228 00:08:45,040 --> 00:08:46,560 improvement. Maybe they got some 229 00:08:46,560 --> 00:08:49,040 feedback from that first assignment. And 230 00:08:49,040 --> 00:08:50,240 finally, they write a plan on how 231 00:08:50,240 --> 00:08:51,680 they're going to build this dashboard in 232 00:08:51,680 --> 00:08:53,760 the remainder of the semester. And this 233 00:08:53,760 --> 00:08:56,000 is a nice hack to minimize freeloading 234 00:08:56,000 --> 00:08:57,519 because they have to specify and agree 235 00:08:57,519 --> 00:08:59,839 in advance on everyone's workload ahead 236 00:08:59,839 --> 00:09:02,720 of time. 237 00:09:02,720 --> 00:09:04,320 And for the final assignment, they have 238 00:09:04,320 --> 00:09:05,760 to build and submit their dashboard 239 00:09:05,760 --> 00:09:08,399 code. This is supported by a short video 240 00:09:08,399 --> 00:09:10,640 walk through. Um and again this is a bit 241 00:09:10,640 --> 00:09:13,440 of a marking hack because um we get to 242 00:09:13,440 --> 00:09:15,519 see the students explain uh their 243 00:09:15,519 --> 00:09:17,200 dashboard informally before we actually 244 00:09:17,200 --> 00:09:18,959 start marking the dashboard. So we get a 245 00:09:18,959 --> 00:09:20,080 better understanding of what their 246 00:09:20,080 --> 00:09:22,640 dashboard is uh or a refresher on what 247 00:09:22,640 --> 00:09:23,920 that that is because we will have 248 00:09:23,920 --> 00:09:25,839 actually seen their dashboard a few 249 00:09:25,839 --> 00:09:28,880 times before this point. 250 00:09:28,880 --> 00:09:30,560 Um it also helps build their 251 00:09:30,560 --> 00:09:32,240 communication skills. So they have to 252 00:09:32,240 --> 00:09:35,360 learn uh how to succinctly present a 253 00:09:35,360 --> 00:09:37,760 complicated idea um to people that are 254 00:09:37,760 --> 00:09:39,279 unfamiliar with it or maybe they can 255 00:09:39,279 --> 00:09:40,320 pretend that we're some kind of 256 00:09:40,320 --> 00:09:42,240 stakeholder. 257 00:09:42,240 --> 00:09:44,399 And intentionally the code of the 258 00:09:44,399 --> 00:09:47,200 dashboard is not marked and instead we 259 00:09:47,200 --> 00:09:48,480 focus on the design and the 260 00:09:48,480 --> 00:09:50,720 functionality things that are a little 261 00:09:50,720 --> 00:09:57,399 bit trickier to apply AI to. 262 00:10:00,720 --> 00:10:03,120 This is supported by rubrics that we've 263 00:10:03,120 --> 00:10:05,279 developed that provide continuity over 264 00:10:05,279 --> 00:10:07,680 the entire semester. So students get 265 00:10:07,680 --> 00:10:09,120 multiple passes at the same topic from 266 00:10:09,120 --> 00:10:10,320 slightly different angles. So in the 267 00:10:10,320 --> 00:10:14,079 three columns I've got um the an element 268 00:10:14,079 --> 00:10:16,160 of the design part of the rubrics for 269 00:10:16,160 --> 00:10:17,760 each of the three assignments. So on the 270 00:10:17,760 --> 00:10:19,200 left is assignment one, the middle is 271 00:10:19,200 --> 00:10:20,640 assignment two, and then on the right we 272 00:10:20,640 --> 00:10:23,920 have assignment three. So the rubrics 273 00:10:23,920 --> 00:10:25,360 are progressively more challenging over 274 00:10:25,360 --> 00:10:28,880 the semester. Um, and [snorts] you can 275 00:10:28,880 --> 00:10:30,399 also see that as we progress from the 276 00:10:30,399 --> 00:10:31,839 individual assignment, the depth 277 00:10:31,839 --> 00:10:33,680 required is is going to increase too. So 278 00:10:33,680 --> 00:10:36,000 the students get multiple runs at this 279 00:10:36,000 --> 00:10:39,279 uh at this process. They're not left um 280 00:10:39,279 --> 00:10:41,200 sort of only having one shot at it and 281 00:10:41,200 --> 00:10:44,839 hoping for the best. 282 00:10:45,360 --> 00:10:48,160 So now we get to the title of the talk. 283 00:10:48,160 --> 00:10:49,920 I added in what I call feed forward 284 00:10:49,920 --> 00:10:52,320 workshops. Uh and these happen in 285 00:10:52,320 --> 00:10:53,839 lecture time the week before an 286 00:10:53,839 --> 00:10:56,079 assignment is due. So the point of the 287 00:10:56,079 --> 00:10:57,920 workshop is to guide students and give 288 00:10:57,920 --> 00:11:00,560 advice before they submit so they don't 289 00:11:00,560 --> 00:11:01,680 have to wait until the marks are 290 00:11:01,680 --> 00:11:04,079 returned for feedback and the feedback 291 00:11:04,079 --> 00:11:07,040 they receive is actually actionable. 292 00:11:07,040 --> 00:11:08,640 We also get a much better insight into 293 00:11:08,640 --> 00:11:10,640 the students progress than if we only 294 00:11:10,640 --> 00:11:13,440 saw their final submission. 295 00:11:13,440 --> 00:11:15,200 So you can see on each of the three 296 00:11:15,200 --> 00:11:16,640 assignments we have a feed forward 297 00:11:16,640 --> 00:11:20,320 workshop assigned in that week prior. So 298 00:11:20,320 --> 00:11:22,000 what is a feed forward workshop? Well, 299 00:11:22,000 --> 00:11:24,000 it's anything you want it to be. I'm not 300 00:11:24,000 --> 00:11:25,839 your boss, but I provide a self- 301 00:11:25,839 --> 00:11:27,440 assessment worksheet to students to get 302 00:11:27,440 --> 00:11:29,440 them started. Uh, and that gets them to 303 00:11:29,440 --> 00:11:31,839 reflect on their progress. The worksheet 304 00:11:31,839 --> 00:11:33,440 also helps narrow down questions that 305 00:11:33,440 --> 00:11:36,560 students might have. U, so that saves us 306 00:11:36,560 --> 00:11:37,920 time when we're going around and 307 00:11:37,920 --> 00:11:41,120 individually consulting those students. 308 00:11:41,120 --> 00:11:42,640 And as we as students are doing the 309 00:11:42,640 --> 00:11:44,399 worksheet, the teaching team works their 310 00:11:44,399 --> 00:11:46,880 way through the students, sometimes 400 311 00:11:46,880 --> 00:11:49,360 students, sometimes 500 students, uh, 312 00:11:49,360 --> 00:11:53,519 providing individual or group feedback. 313 00:11:53,519 --> 00:11:55,040 Recurring problems are noted by the 314 00:11:55,040 --> 00:11:56,640 teaching team and reported back to the 315 00:11:56,640 --> 00:11:58,240 whole class at the end so they can learn 316 00:11:58,240 --> 00:12:00,320 from each other and no one misses out on 317 00:12:00,320 --> 00:12:03,959 anything that's important. 318 00:12:06,079 --> 00:12:07,839 And finally, I added an element of 319 00:12:07,839 --> 00:12:10,079 informal peer evaluation and reflection 320 00:12:10,079 --> 00:12:11,839 by adding a showcase at the end of the 321 00:12:11,839 --> 00:12:13,440 semester to highlight exceptional 322 00:12:13,440 --> 00:12:15,279 dashboards that's noted in green at the 323 00:12:15,279 --> 00:12:19,360 end. And we also at the same time review 324 00:12:19,360 --> 00:12:21,680 the previous year's showcases at the 325 00:12:21,680 --> 00:12:24,480 start. So in lecture one, I'll show the 326 00:12:24,480 --> 00:12:27,120 students the prior year showcases uh and 327 00:12:27,120 --> 00:12:28,720 they will be able to refer back to that 328 00:12:28,720 --> 00:12:31,839 at any point over the semester. And I 329 00:12:31,839 --> 00:12:33,360 think it's most useful at the start of 330 00:12:33,360 --> 00:12:34,880 the semester when they're generating 331 00:12:34,880 --> 00:12:36,880 ideas for what they want to work on uh 332 00:12:36,880 --> 00:12:39,600 and learn about what is the end state of 333 00:12:39,600 --> 00:12:41,040 the the dashboard that they need to 334 00:12:41,040 --> 00:12:43,360 achieve. 335 00:12:43,360 --> 00:12:44,880 And overall, this seems to have 336 00:12:44,880 --> 00:12:47,600 addressed to a large extent the three 337 00:12:47,600 --> 00:12:49,839 problems that I outlined. The work is 338 00:12:49,839 --> 00:12:51,839 now spread out over the semester to 339 00:12:51,839 --> 00:12:54,720 avoid a lastminute disaster. Pressure is 340 00:12:54,720 --> 00:12:56,720 taken off me to provide detailed post 341 00:12:56,720 --> 00:12:58,480 assignment feedback and students 342 00:12:58,480 --> 00:13:00,399 actually prefer the feedback ahead of 343 00:13:00,399 --> 00:13:02,720 time. So since implementing this, I 344 00:13:02,720 --> 00:13:04,000 haven't really had any complaints from 345 00:13:04,000 --> 00:13:05,920 students saying you didn't provide me 346 00:13:05,920 --> 00:13:07,920 enough with enough feedback after they 347 00:13:07,920 --> 00:13:10,160 they received their marks. 348 00:13:10,160 --> 00:13:11,360 And students now feel much more 349 00:13:11,360 --> 00:13:13,040 connected to each other as they see 350 00:13:13,040 --> 00:13:14,880 students in the workshops and have the 351 00:13:14,880 --> 00:13:19,160 showcases as a reference point. 352 00:13:21,279 --> 00:13:24,880 What about AI? So, it turns out I think 353 00:13:24,880 --> 00:13:27,279 that the strategy that I've presented 354 00:13:27,279 --> 00:13:28,720 here lends itself naturally to an 355 00:13:28,720 --> 00:13:30,639 environment where AI will be used by 356 00:13:30,639 --> 00:13:32,399 students and we have limited control 357 00:13:32,399 --> 00:13:35,519 over that. The now cheap and easy parts 358 00:13:35,519 --> 00:13:38,000 of code generation have been strongly 359 00:13:38,000 --> 00:13:39,920 deemphasized. 360 00:13:39,920 --> 00:13:41,519 And in its place, I've emphasized and 361 00:13:41,519 --> 00:13:43,040 rewarded the students judgment and 362 00:13:43,040 --> 00:13:45,120 evaluation skills, which I think are 363 00:13:45,120 --> 00:13:47,440 skills that matter most for applied 364 00:13:47,440 --> 00:13:50,440 projects. 365 00:13:54,240 --> 00:13:55,760 The feed forward sessions create a 366 00:13:55,760 --> 00:13:57,120 built-in point of reflection for 367 00:13:57,120 --> 00:13:59,040 students. And even if students have 368 00:13:59,040 --> 00:14:02,000 retired relied on AI, they will get 369 00:14:02,000 --> 00:14:04,000 feedback from the teaching team on what 370 00:14:04,000 --> 00:14:06,880 needs improving. Very few of them get a 371 00:14:06,880 --> 00:14:09,839 good result from AI the first time. 372 00:14:09,839 --> 00:14:12,639 And again, if they take our feedback to 373 00:14:12,639 --> 00:14:14,560 try and make improvements, they 374 00:14:14,560 --> 00:14:16,160 themselves need to judge whether those 375 00:14:16,160 --> 00:14:18,639 improvements have been successfully made 376 00:14:18,639 --> 00:14:20,560 or else they'll lose out on potential 377 00:14:20,560 --> 00:14:22,240 marks. 378 00:14:22,240 --> 00:14:24,079 But this does require carefully 379 00:14:24,079 --> 00:14:26,639 delivered feedback. We intentionally 380 00:14:26,639 --> 00:14:29,279 provide verbal feedback only, which 381 00:14:29,279 --> 00:14:30,800 makes it somewhat harder for students to 382 00:14:30,800 --> 00:14:32,959 dump the feedback into an LLM. They need 383 00:14:32,959 --> 00:14:35,199 to reynthesize what we've said to them 384 00:14:35,199 --> 00:14:38,000 in person. And we minimize direct 385 00:14:38,000 --> 00:14:39,839 instruction where we can during these 386 00:14:39,839 --> 00:14:42,880 sessions. Instead, we try to ask probing 387 00:14:42,880 --> 00:14:44,639 questions to try and lead students 388 00:14:44,639 --> 00:14:46,079 towards improving their assignment 389 00:14:46,079 --> 00:14:49,079 themselves. 390 00:14:51,920 --> 00:14:53,360 Generally, feed forward sessions 391 00:14:53,360 --> 00:14:54,880 encourages encourage the students to 392 00:14:54,880 --> 00:14:56,880 start earlier on their assignments. And 393 00:14:56,880 --> 00:14:58,399 I think this is really helpful for 394 00:14:58,399 --> 00:15:02,079 reducing assignment stress which in turn 395 00:15:02,079 --> 00:15:03,839 will prevent them or make them less 396 00:15:03,839 --> 00:15:06,079 likely to reach for AI at the last 397 00:15:06,079 --> 00:15:09,440 minute to try and save them. 398 00:15:09,440 --> 00:15:10,959 We also use this time to reassure 399 00:15:10,959 --> 00:15:13,199 students where they are performing well 400 00:15:13,199 --> 00:15:16,160 and set realistic expectations on what 401 00:15:16,160 --> 00:15:19,199 is required to do better. 402 00:15:19,199 --> 00:15:20,720 The end result generally is that many 403 00:15:20,720 --> 00:15:21,760 students feel like they don't need to 404 00:15:21,760 --> 00:15:24,800 rely on AI to do well and to maximize 405 00:15:24,800 --> 00:15:27,839 their marks. 406 00:15:29,199 --> 00:15:30,959 And the scaffolding structure also seems 407 00:15:30,959 --> 00:15:32,880 to have a moderating effect on the use 408 00:15:32,880 --> 00:15:35,120 of AI. The main effect seems to come 409 00:15:35,120 --> 00:15:36,399 from breaking down the project into 410 00:15:36,399 --> 00:15:38,880 these more manageable stages which 411 00:15:38,880 --> 00:15:40,880 students then can feel like they are 412 00:15:40,880 --> 00:15:43,279 more comfortable approaching. 413 00:15:43,279 --> 00:15:44,880 I also tell students from the start that 414 00:15:44,880 --> 00:15:46,560 the assignments build on each other and 415 00:15:46,560 --> 00:15:48,000 they get to access to the assignment 416 00:15:48,000 --> 00:15:50,399 descriptions early on and so that they 417 00:15:50,399 --> 00:15:51,440 can see that there are possible 418 00:15:51,440 --> 00:15:53,519 consequences such as a greater volume of 419 00:15:53,519 --> 00:15:55,440 work for not carefully considering each 420 00:15:55,440 --> 00:15:56,880 stage. So if they leave their assignment 421 00:15:56,880 --> 00:15:58,160 to the last minute and come up with a 422 00:15:58,160 --> 00:16:00,079 design proposal that's not very good. 423 00:16:00,079 --> 00:16:02,160 It's going to that is going to put 424 00:16:02,160 --> 00:16:04,000 pressure on them at later stages in the 425 00:16:04,000 --> 00:16:06,320 semester. 426 00:16:06,320 --> 00:16:07,759 So this gets them engaged because they 427 00:16:07,759 --> 00:16:10,639 can't just blindly trust AI output for 428 00:16:10,639 --> 00:16:12,480 their assignment. They have to verify 429 00:16:12,480 --> 00:16:17,000 what's written if they're using it. 430 00:16:19,120 --> 00:16:20,320 There's also some potential 431 00:16:20,320 --> 00:16:22,480 embarrassment here of presenting work 432 00:16:22,480 --> 00:16:24,160 that they can't explain or don't 433 00:16:24,160 --> 00:16:27,519 understand and that forces them to check 434 00:16:27,519 --> 00:16:30,959 and revise AI generated output. So this 435 00:16:30,959 --> 00:16:32,320 happens at multiple points over the 436 00:16:32,320 --> 00:16:34,399 semester. There's feedback seeking that 437 00:16:34,399 --> 00:16:36,399 they do during the feed forward sessions 438 00:16:36,399 --> 00:16:38,320 and the semi-public discussion required 439 00:16:38,320 --> 00:16:40,959 for peer evaluation um which happens 440 00:16:40,959 --> 00:16:42,320 during the the middle assignment and 441 00:16:42,320 --> 00:16:43,680 there's also the risk that their project 442 00:16:43,680 --> 00:16:46,000 gets selected for the showcase. So a lot 443 00:16:46,000 --> 00:16:47,759 of them are a little bit shy and nervous 444 00:16:47,759 --> 00:16:49,199 about being selected for the showcase 445 00:16:49,199 --> 00:16:50,800 and if they don't feel comfortable 446 00:16:50,800 --> 00:16:52,160 presenting their work it's it's going to 447 00:16:52,160 --> 00:16:54,639 discourage them from relying on AI 448 00:16:54,639 --> 00:16:57,639 entirely. 449 00:17:00,480 --> 00:17:02,160 There are also a couple of nice side 450 00:17:02,160 --> 00:17:04,959 effects too. The feed forward session 451 00:17:04,959 --> 00:17:06,319 simulates potential stakeholder 452 00:17:06,319 --> 00:17:08,160 discussions that they might have 453 00:17:08,160 --> 00:17:10,079 building a dashboard in the real world 454 00:17:10,079 --> 00:17:11,600 because most students don't have a real 455 00:17:11,600 --> 00:17:13,520 stakeholder to talk to over the 456 00:17:13,520 --> 00:17:15,679 semester. So we do our best to act as 457 00:17:15,679 --> 00:17:16,880 one for the students during the 458 00:17:16,880 --> 00:17:18,720 workshops. 459 00:17:18,720 --> 00:17:20,559 And lastly, the showcase is showcase 460 00:17:20,559 --> 00:17:22,240 encourages students to put in effort 461 00:17:22,240 --> 00:17:24,079 because they want to put it on their CV 462 00:17:24,079 --> 00:17:26,079 or use it as a talking point in a job 463 00:17:26,079 --> 00:17:29,440 interview. And in that situation again, 464 00:17:29,440 --> 00:17:30,960 if they don't understand it or can't 465 00:17:30,960 --> 00:17:35,320 explain it, they're not getting the job. 466 00:17:35,760 --> 00:17:38,400 So, how's it gone? 467 00:17:38,400 --> 00:17:40,000 We've run surveys to collect student 468 00:17:40,000 --> 00:17:42,080 perspectives on the assessment design 469 00:17:42,080 --> 00:17:45,760 early on. So, over 2022 and 2023, 470 00:17:45,760 --> 00:17:47,440 when asked about the most memorable and 471 00:17:47,440 --> 00:17:48,640 meaningful learning experience in this 472 00:17:48,640 --> 00:17:51,200 unit, students expressed pride that they 473 00:17:51,200 --> 00:17:53,200 built a working dashboard, which really 474 00:17:53,200 --> 00:17:54,720 underlines the value of authentic 475 00:17:54,720 --> 00:17:57,120 assessment in this unit. And students 476 00:17:57,120 --> 00:17:58,960 are generally very satisfied with the 477 00:17:58,960 --> 00:18:01,679 assessment design. The unit consistently 478 00:18:01,679 --> 00:18:03,919 receives excellent scores in the final 479 00:18:03,919 --> 00:18:06,480 unit evaluation survey on intellectual 480 00:18:06,480 --> 00:18:09,520 challenge, rewarding experience, 481 00:18:09,520 --> 00:18:11,520 amount, and quality of feedback, 482 00:18:11,520 --> 00:18:14,960 engagement, and sense or feeling of 483 00:18:14,960 --> 00:18:19,080 being part of a learning community. 484 00:18:19,360 --> 00:18:21,760 I've collected some quotes uh that I've 485 00:18:21,760 --> 00:18:23,440 received from students and also from the 486 00:18:23,440 --> 00:18:24,960 survey. 487 00:18:24,960 --> 00:18:28,559 So um the first one says that they feel 488 00:18:28,559 --> 00:18:29,760 like they've played lots of roles in 489 00:18:29,760 --> 00:18:31,440 this project such as a project manager, 490 00:18:31,440 --> 00:18:34,320 coder and artist. So again we've 491 00:18:34,320 --> 00:18:36,640 deemphasized the the programming aspect 492 00:18:36,640 --> 00:18:38,480 and they feel like they've taken on 493 00:18:38,480 --> 00:18:41,039 various roles that form an integration 494 00:18:41,039 --> 00:18:42,960 of skills. 495 00:18:42,960 --> 00:18:44,960 And the middle student said that they 496 00:18:44,960 --> 00:18:46,080 that the units made them think 497 00:18:46,080 --> 00:18:47,520 critically about how to build dashboard 498 00:18:47,520 --> 00:18:49,360 that serves purposes not merely 499 00:18:49,360 --> 00:18:51,120 visualizing data without some clear 500 00:18:51,120 --> 00:18:54,640 objective. Uh and lastly, this student 501 00:18:54,640 --> 00:18:56,240 wrote that they thought the unit was 502 00:18:56,240 --> 00:18:58,240 amazing uh because they learned how to 503 00:18:58,240 --> 00:18:59,840 make a dashboard when they thought it 504 00:18:59,840 --> 00:19:04,039 was impossible for them in the past. 505 00:19:05,760 --> 00:19:07,760 There's specific comments uh about the 506 00:19:07,760 --> 00:19:09,440 feedback too and the feed forward 507 00:19:09,440 --> 00:19:12,000 session. So the first uh student wrote 508 00:19:12,000 --> 00:19:13,280 that they were able to receive feedback 509 00:19:13,280 --> 00:19:14,320 for the assignments before the 510 00:19:14,320 --> 00:19:15,760 submission date which helped them 511 00:19:15,760 --> 00:19:18,240 continuously improve it. 512 00:19:18,240 --> 00:19:20,880 And the last uh comment here says, 513 00:19:20,880 --> 00:19:22,720 "Really good assignment structure, how 514 00:19:22,720 --> 00:19:24,240 it iterates and builds upon previous 515 00:19:24,240 --> 00:19:26,160 ones, and they were fortunate enough to 516 00:19:26,160 --> 00:19:28,240 be in a great group, and I like the 517 00:19:28,240 --> 00:19:31,120 embrace of AI, too." Uh I think they're 518 00:19:31,120 --> 00:19:33,360 probably extending the embrace of AI. I 519 00:19:33,360 --> 00:19:35,039 don't particularly encourage or dissuade 520 00:19:35,039 --> 00:19:37,039 students from using it. It's up to them. 521 00:19:37,039 --> 00:19:39,760 Um but they're this student feels like 522 00:19:39,760 --> 00:19:44,600 I've I've pushed them towards it. So, 523 00:19:46,080 --> 00:19:48,799 okay. So the strategies that I've 524 00:19:48,799 --> 00:19:50,960 discussed so far can be generalized to 525 00:19:50,960 --> 00:19:53,120 other contexts. I think these feed 526 00:19:53,120 --> 00:19:55,760 forward sessions can be run uh ahead of 527 00:19:55,760 --> 00:19:57,353 all kind of submission deadlines 528 00:19:57,353 --> 00:20:00,400 [snorts] at all levels of education. 529 00:20:00,400 --> 00:20:02,000 Large project-based units can be broken 530 00:20:02,000 --> 00:20:05,360 down into smaller authentic stages and 531 00:20:05,360 --> 00:20:06,799 peer feedback can be integrated 532 00:20:06,799 --> 00:20:08,160 authentically and in such a way that 533 00:20:08,160 --> 00:20:09,919 doesn't directly influence other 534 00:20:09,919 --> 00:20:12,080 students marks. 535 00:20:12,080 --> 00:20:14,080 Finally, the showcases can help reduce 536 00:20:14,080 --> 00:20:16,640 the feeling of isolation and increase 537 00:20:16,640 --> 00:20:18,240 connectedness. 538 00:20:18,240 --> 00:20:20,960 So, this is still a work in progress. 539 00:20:20,960 --> 00:20:23,360 Uh, and we're constantly changing things 540 00:20:23,360 --> 00:20:26,240 in the unit to adjust for changes in 541 00:20:26,240 --> 00:20:29,280 particularly quality of AI output. 542 00:20:29,280 --> 00:20:31,520 The rubrics that we've put together uh 543 00:20:31,520 --> 00:20:33,600 in particular uh have reflected those 544 00:20:33,600 --> 00:20:35,520 changes the most. 545 00:20:35,520 --> 00:20:38,400 Um, so hopefully I've encouraged you to, 546 00:20:38,400 --> 00:20:41,440 if you're a teacher or an educator, to 547 00:20:41,440 --> 00:20:44,720 rethink how you might compensate for a 548 00:20:44,720 --> 00:20:47,120 world in which AI is potentially 549 00:20:47,120 --> 00:20:49,360 unavoidable, such as in an open-based 550 00:20:49,360 --> 00:20:52,320 assessment and project-based assessment. 551 00:20:52,320 --> 00:20:56,359 And thank you for your time. [applause] 552 00:21:00,000 --> 00:21:01,760 Thank you very much, Stephen. And here 553 00:21:01,760 --> 00:21:03,200 is your speaker gift. 554 00:21:03,200 --> 00:21:06,184 Thank you. [applause] 555 00:21:08,080 --> 00:21:10,320 We have time for some questions. So, if 556 00:21:10,320 --> 00:21:12,480 you've got any, raise your hand and 557 00:21:12,480 --> 00:21:13,760 Amanda will run around with the 558 00:21:13,760 --> 00:21:16,760 microphone. 559 00:21:20,080 --> 00:21:22,320 Amanda, you have to run. [laughter] 560 00:21:22,320 --> 00:21:26,280 I was looking the wrong way. 561 00:21:30,000 --> 00:21:33,520 Thank Thanks for that. And uh I feel 562 00:21:33,520 --> 00:21:35,919 like it's actually very getting very 563 00:21:35,919 --> 00:21:38,960 close to what the agile methodology 564 00:21:38,960 --> 00:21:42,960 should be. Um so I really want to see 565 00:21:42,960 --> 00:21:45,679 workplaces uh implement this idea that 566 00:21:45,679 --> 00:21:48,720 they actually have constructive feedback 567 00:21:48,720 --> 00:21:51,760 with their design teams on an ongoing 568 00:21:51,760 --> 00:21:56,559 basis. Um, the thing I'm curious about 569 00:21:56,559 --> 00:21:59,360 here is that it feels like in your 570 00:21:59,360 --> 00:22:02,640 description how a lot more of your work 571 00:22:02,640 --> 00:22:05,600 is now assessing how students work 572 00:22:05,600 --> 00:22:08,559 together and I was interested to see 573 00:22:08,559 --> 00:22:10,799 that the feedback that you're getting is 574 00:22:10,799 --> 00:22:12,720 still saying we learned how to make a 575 00:22:12,720 --> 00:22:14,240 dashboard. So, how do do you feel like 576 00:22:14,240 --> 00:22:17,360 there's any tension between the two? Um, 577 00:22:17,360 --> 00:22:20,480 so I've selected the feedback. Um, 578 00:22:20,480 --> 00:22:22,159 cherrypicked it a bit, but yeah, there 579 00:22:22,159 --> 00:22:24,240 there is still there's comments in 580 00:22:24,240 --> 00:22:27,200 there. Um, I'm not sure if there like 581 00:22:27,200 --> 00:22:28,559 this last person wrote that they were 582 00:22:28,559 --> 00:22:29,919 fortunate enough to be in a great group 583 00:22:29,919 --> 00:22:32,480 and there are some comments scattered in 584 00:22:32,480 --> 00:22:34,799 these ones about it, but um, and there 585 00:22:34,799 --> 00:22:36,400 are comments that I' I've left out that 586 00:22:36,400 --> 00:22:37,760 say, "Yeah, I enjoyed the teamwork 587 00:22:37,760 --> 00:22:41,520 aspect." Um, yeah, part of that is the 588 00:22:41,520 --> 00:22:44,640 design of the the 589 00:22:44,640 --> 00:22:47,360 project plan. um getting the students to 590 00:22:47,360 --> 00:22:50,159 agree ahead of time on what they're 591 00:22:50,159 --> 00:22:51,520 going to do and who's going to do what 592 00:22:51,520 --> 00:22:52,880 and when it's going to be delivered by 593 00:22:52,880 --> 00:22:54,320 that's greatly reduced the amount of 594 00:22:54,320 --> 00:22:57,440 sort of um complications and troubles 595 00:22:57,440 --> 00:23:01,840 that that occur with group work. Um 596 00:23:01,840 --> 00:23:04,480 I think uh and prior to having that in 597 00:23:04,480 --> 00:23:06,640 the 2022 version of the unit that there 598 00:23:06,640 --> 00:23:08,720 was a lot of complaints about group work 599 00:23:08,720 --> 00:23:11,679 difficulties. Um so yeah, I don't think 600 00:23:11,679 --> 00:23:14,080 that you can't really avoid any of those 601 00:23:14,080 --> 00:23:17,760 sorts of things. Um uh I and but at the 602 00:23:17,760 --> 00:23:19,200 same time I don't really particularly 603 00:23:19,200 --> 00:23:21,440 emphasize it beyond that one part of the 604 00:23:21,440 --> 00:23:25,640 assignment either. So yeah 605 00:23:33,520 --> 00:23:35,520 um I know you've de you mentioned that 606 00:23:35,520 --> 00:23:37,600 you deemphasize the actual final 607 00:23:37,600 --> 00:23:38,159 product. 608 00:23:38,159 --> 00:23:41,039 Yep. Um, however, I'm curious to know, 609 00:23:41,039 --> 00:23:43,760 did the actual final product improve 610 00:23:43,760 --> 00:23:46,799 over the 3 or 4 years um, since the 611 00:23:46,799 --> 00:23:50,880 inception of it in 2022? Uh, yeah, I I 612 00:23:50,880 --> 00:23:52,480 have noticed an improvement in the 613 00:23:52,480 --> 00:23:54,640 quality. I don't know if I can, uh, 614 00:23:54,640 --> 00:23:58,400 quickly bring this up now. Um, I did 615 00:23:58,400 --> 00:24:02,080 have some bonus slides prepared. Oh no, 616 00:24:02,080 --> 00:24:05,601 it's going to make me two factor. 617 00:24:05,601 --> 00:24:06,960 [laughter] 618 00:24:06,960 --> 00:24:09,200 Um maybe I can show you later, but yeah, 619 00:24:09,200 --> 00:24:10,720 I have showcases going back four years 620 00:24:10,720 --> 00:24:12,320 now and and the quality definitely has 621 00:24:12,320 --> 00:24:15,120 seen an improvement. Some of that is use 622 00:24:15,120 --> 00:24:18,720 of AI, the tools get better. Um and some 623 00:24:18,720 --> 00:24:21,120 of it is also the way the rubric has 624 00:24:21,120 --> 00:24:22,720 changed over time has really encouraged 625 00:24:22,720 --> 00:24:24,880 students to focus more on the design and 626 00:24:24,880 --> 00:24:28,159 like does this dashboard solve that 627 00:24:28,159 --> 00:24:30,559 original problem that they defi defined 628 00:24:30,559 --> 00:24:32,720 back in the design proposal. So there's 629 00:24:32,720 --> 00:24:35,200 like a a very strong link and strong 630 00:24:35,200 --> 00:24:38,159 mark associated with that in the final 631 00:24:38,159 --> 00:24:40,000 report that they have to deliver. They 632 00:24:40,000 --> 00:24:42,080 they have to explain to me and the 633 00:24:42,080 --> 00:24:44,799 marking team how does how have they met 634 00:24:44,799 --> 00:24:46,880 or solved that problem to whatever 635 00:24:46,880 --> 00:24:49,039 degree and also provide a reflection to 636 00:24:49,039 --> 00:24:51,919 say what parts didn't they quite get or 637 00:24:51,919 --> 00:24:53,360 what would they do differently if they 638 00:24:53,360 --> 00:24:57,880 had more time. Um yeah. 639 00:24:58,000 --> 00:25:01,360 Uh when you interview students, do you 640 00:25:01,360 --> 00:25:03,279 have a scaffold or a process that you do 641 00:25:03,279 --> 00:25:04,720 there or do you just do it as a choose 642 00:25:04,720 --> 00:25:06,240 your own adventure when you're sort of 643 00:25:06,240 --> 00:25:07,600 teasing out what they can actually 644 00:25:07,600 --> 00:25:10,880 explain out of their own projects? So 645 00:25:10,880 --> 00:25:14,320 yeah, it's not really um a fixed process 646 00:25:14,320 --> 00:25:16,880 or anything. Uh usually we start off by 647 00:25:16,880 --> 00:25:18,320 saying how are you feeling about the 648 00:25:18,320 --> 00:25:20,720 project? Where where are where are you? 649 00:25:20,720 --> 00:25:22,480 What do you what do you need help with? 650 00:25:22,480 --> 00:25:23,919 Do you have any questions? and sort of 651 00:25:23,919 --> 00:25:26,480 just open up a bit of a dialogue. Um, 652 00:25:26,480 --> 00:25:28,000 yeah, so there's no real fixed process 653 00:25:28,000 --> 00:25:30,000 and I'm only I'm the coordinator for the 654 00:25:30,000 --> 00:25:32,159 unit, but I'm in the room with usually 655 00:25:32,159 --> 00:25:34,720 at least two or three other people and 656 00:25:34,720 --> 00:25:36,640 they all have their own approaches to 657 00:25:36,640 --> 00:25:37,760 getting feedback from students and 658 00:25:37,760 --> 00:25:39,120 guiding them as well. So, it's all a bit 659 00:25:39,120 --> 00:25:42,080 it's a bit different. Yeah. 660 00:25:42,080 --> 00:25:44,400 I have a I have a question as a teacher. 661 00:25:44,400 --> 00:25:46,640 Um, what percentage of students have 662 00:25:46,640 --> 00:25:48,320 finished the assignment a week before 663 00:25:48,320 --> 00:25:49,760 it's due? 664 00:25:49,760 --> 00:25:50,640 Most of them. 665 00:25:50,640 --> 00:25:52,320 Oh, wow. [laughter] 666 00:25:52,320 --> 00:25:53,679 They have they have something 667 00:25:53,679 --> 00:25:55,679 submittable that they could submit at 668 00:25:55,679 --> 00:25:58,720 that point which is great. Um yeah, it's 669 00:25:58,720 --> 00:26:00,320 not perfect. That's why we have the feed 670 00:26:00,320 --> 00:26:01,520 forward workshop so we can tell them 671 00:26:01,520 --> 00:26:03,840 what to fix. Um but yeah, it's 672 00:26:03,840 --> 00:26:05,919 submittable. 673 00:26:05,919 --> 00:26:07,200 Okay. 674 00:26:07,200 --> 00:26:09,039 All right. I just want to remind people 675 00:26:09,039 --> 00:26:11,039 that we have to move. 676 00:26:11,039 --> 00:26:12,480 Uh are you on there? 677 00:26:12,480 --> 00:26:13,440 Ask a question. 678 00:26:13,440 --> 00:26:13,840 Yeah. 679 00:26:13,840 --> 00:26:15,600 I wanted to know of all the dashboards 680 00:26:15,600 --> 00:26:17,279 that have been produced, which was the 681 00:26:17,279 --> 00:26:19,520 most interesting topics or themes that 682 00:26:19,520 --> 00:26:21,200 you saw? 683 00:26:21,200 --> 00:26:23,360 Uh so 684 00:26:23,360 --> 00:26:26,799 there's quite a variety. Um some 685 00:26:26,799 --> 00:26:29,520 students uh a lot of students reach for 686 00:26:29,520 --> 00:26:30,799 things where there is data readily 687 00:26:30,799 --> 00:26:32,640 available like uh from government 688 00:26:32,640 --> 00:26:35,360 websites and government data databases. 689 00:26:35,360 --> 00:26:37,039 Um the more interesting ones are the 690 00:26:37,039 --> 00:26:39,840 ones that I think are personal. So, a 691 00:26:39,840 --> 00:26:43,039 student uh had a particular medical 692 00:26:43,039 --> 00:26:44,799 condition and she built herself a 693 00:26:44,799 --> 00:26:47,840 dashboard so that she could basically 694 00:26:47,840 --> 00:26:50,000 review all the medications that she was 695 00:26:50,000 --> 00:26:52,720 taking and try to figure out which ones 696 00:26:52,720 --> 00:26:54,640 potentially were causing various 697 00:26:54,640 --> 00:26:56,240 complications and side effects with each 698 00:26:56,240 --> 00:26:58,320 other. Um, and so that that could be 699 00:26:58,320 --> 00:27:00,159 used by other people with the same 700 00:27:00,159 --> 00:27:03,039 condition. Uh, and another student built 701 00:27:03,039 --> 00:27:04,799 something that was a Stardew Valley 702 00:27:04,799 --> 00:27:06,240 planner so that they could plan their 703 00:27:06,240 --> 00:27:08,799 farm in Stardew Valley. Um, so yeah, 704 00:27:08,799 --> 00:27:10,799 there's a big big variety of of topics 705 00:27:10,799 --> 00:27:12,080 and things that the students can work 706 00:27:12,080 --> 00:27:15,080 on.