[Music]
welcome back everybody and for our next
speaker we have Dr Steven tiny who works
for the University of Sydney with
Allison Wong who got a place on the
slide even though she's decided to stay
in her seat today um yeah so uh he works
in the School of Business uh teaches
python staff he's going to talk bit
about I don't actually know how to say
it is it Carell or Carol it Carol I call
it Carol too okay I'm right I'm right
everybody it's fine um okay so he's
going to talk about playful learning
with Carol thank you thank you
Stephen okay so our talk is enhancing
programming ability with playful
learning and Carol um as you can see
there's already a little bit of a demo
going on with Carol um and I'll show
some more demos of Carol in action later
uh this is a now turned into a larger
scale research project and it's been
something we've been working on uh since
the start of the
year um and to give you some context uh
so I work in the business school uh in
the department or discipline of business
analytics uh so we focus on teaching
statistics sta uh data science
optimizing ation data visualization uh
within a business
context uh and our students um both
Allison and mine uh are at quite
different levels we teach undergraduate
and postgraduate students however in
both contexts most students don't come
into these degree programs with any
python or programming experience uh and
we also have a bit of a time crunch
where teaching we usually only have the
students for 13 weeks and we only have
them a couple of hours each week so we
have very limited time to teach these
students programming alongside all the
other content that we have to try and
teach them about the statistics and the
data science too the theory behind it so
we have a limited time to teach and
learn skills which are slowly acquired
so we know that for most people
acquiring programming as a skill is
something that takes time and practice
so things that might be slowly acquired
are understanding syntax or learning a
language is is hard understanding how
program execution and how to trace
through a program that's a skill that
can also take time to develop and then
there's the third stage of understanding
and developing a model of program state
in your mind so those things I really
want to enforce that they they take a
long time to learn and we have a very
limited time to teach
those uh and the result of this is that
we end up with students with relatively
weak computational thinking skills so we
want them to improve or do better than
what they're currently doing um often
they are unable to write code
independently uh even though they are
exposed to and see a lot of code
throughout a semester some of them
struggle to do this by themselves and
often they're unable to debug their own
code that they've
written so what we are uh trying to do
is encourage greater participation with
programming outside of the classroom as
an extracurricular activity and make
things more fun and engaging for the
students um so that they can see that
okay programming is not just something
that they do to uh help them complete
their homework and their assignments but
is actually an enjoyable activity that
has broader applications beyond the
classroom
uh and we've decided to um sort of
promote this engagement with programming
by introducing an extracurricular
activity with Carol and Carol is a
simplified programming environment it we
use a it uses a subset of python uh and
the environment of Carol is a 2d grid
world and you move Carol around this
grid world uh and interact with the grid
World um through
Carol and it's all code based so
students control Carol only with code uh
and the point of this is that students
can develop their computational thinking
skills by understanding the concepts of
a program rather than trying to
understand really complicated
syntax there's also immediate
visualization of the program state so
all parts of the program state are
actually in the the Carol world there's
no way to create things like variables
so everything actually is visible in the
2D space uh and as a result there's
reduced need to maintain a mental model
so students can sort of defer or
Outsource their mental model to the
Carol world and they don't need to sort
of as they're writing code think about
or as they're actually testing their
code think about what's happening inside
their program because they can just see
it um okay so I've told you what Carol
is in theory so let's take a look at
some examples of Carol hopefully this is
working let's see if I can zoom this in
so we teach primarily on Ed let's see if
I can make that font a bit bigger for
you
all uh and we have a set of I'll provide
the students with a set of information
uh on Ed about what Carol is and how to
use it through these slides you can see
on the left hand side I'll start off
with some basics of Carol uh we give
them instructions about how they can
move Carol around the world and a nice
thing about Ed is that you can have
these uh integrated uh and runnable code
Snippets so for example I can run this
code here um you define your Carol
program by defining a main function and
then just put all your instructions in
there and later on you can Define your
own other functions which you can call
so that will just move Carol uh in two
steps to the right uh and then Carol has
a very limited subset of instru set of
instructions that you can do so here we
can turn left and then move and move uh
and so we step the students through
these basic instructions to hopefully
get to a point where we can do something
more
interesting Carol can also interact with
these things called beepers which are
little objects that can be placed on
each of the tiles so we can place one
beeper down put another beeper down and
we can see that there are two beepers
now placed in the world um and so
basically we can represent rather
complex program state in these 2D grid
environments with only beepers and also
sometimes by coloring some of these
environments um okay so now I'll move on
to uh a more complicated example which
we'll do try and do live I
think have we hidden the Carol exercises
wrong wrong Oh wrong I'm on
challenges yep all right so let's try
the Beeper drop exercise so this is a a
an exercise that we give the students
then as part of these this
extracurricular
activity uh we give them a program sort
of goal on the left hand side and then
give them a gift to to show them uh what
this might look like when their program
actually runs and so for the challenge
for the students is then to write their
code over on the right hand side in the
code editor and then they can try and
run their code if they like so if we
want to do that we can then say
move uh and then we can probably turn
left and then move
again and so on and I click the Run
button and it will very quickly execute
I have set the speed way too fast for
Carol um that's a useful feature on Ed
you can actually speed up car running so
when you have a really long running
program you can make it move really fast
uh and then students don't get to see
this but I'm just going to grab the
solution over on the uh from the
solution
page and I will try running my code make
sure that it works we place the Beeper
in the center as required by the
program uh I'm just going to edit out
the uh a comment out placing the Beeper
and I will mark my
program um so we have an automated
marking setup so that we can compare the
the Carol world at the final State um to
what is expected they get um some
information description
about what actually went wrong uh and
they also get a little asky art diagram
of hey you didn't place a beeper in the
right spot so um the one there indicates
that one beeper should be placed in the
middle and that Carol should be at the
bottom left corner of the screen so
hopefully if I uncomment down my code I
can mark this and I should pass the test
case Okay so that's the the basics of
the Carol environment that we have and
the sorts of things that we can do with
Carol or students can do with Carol
okay so now that we know about the
basics what did we actually do over
these past two semesters so we developed
a playful inquiry and inquiry-driven set
of extracurricular programming
activities uh and challenges using Carol
we did this in two subjects an
undergraduate unit and a postgraduate uh
unit both are uh at the early stages of
the students uh course or course work
and we tried to encourage participation
in these challenges by creating a
competition so we created um a
leaderboard system students were awarded
points for completing uh the challenges
that we set for
them uh and uh at the end we rewarded
those students that did well and
actually rewarded a lot of students who
even for just participating the
competition runs over four or five weeks
depending on the unit uh because of some
scheduling issues and each week we
released a new set of instructional
slides on Ed and uh we release then five
challenges on that week's topic so each
week five new challenges for them to try
questions were worth some maximum number
of points and incorrect submissions were
given a small penalty so what we wanted
to try to encourage by imposing a
penalty uh is getting students to test
and evaluate their own code before they
hit that Mark button so one of the
issues that we find uh with our students
is that on systems like Ed when they're
writing uh code for something that's in
a tutorial or maybe it's their homework
they will just Spam The Mark button
until they get it right rather than
trying to actually test and evaluate
their own code so we wanted to encourage
this behavior of thinking about their
own code uh and being able to test and
evaluate it
themselves okay and in terms of
difficulty I can show you what the range
of difficulty is uh that we we pose to
our students so we saw the first
exercise which was the Beeper drop you
just go into the middle of the room and
place a a beeper and then return to the
corner uh we had some more difficult
ones like filling in the potholes where
students were given this random uh sort
of almost one-dimensional world where
they had to go in and find the locations
where a beeper was present and then
replace that beeper oh sorry where a
beeper wasn't present in these holes and
place a beeper in so that's a relatively
sort of intermediate difficult
difficulty U exercise and then we really
ramped up the difficulty um later on so
we did exercises like uh maze solving
within
Carol um so students implemented sort of
like a a wall holding or or War
following algorithm there um we made
them do uh QR codes which we called KR
codes um and we embedded a nice Rick
Roll in there for the students um and
then other we created some really quite
intricate and very challenging exercises
like coloring in these squares in a very
um you know predetermined pattern or
order which is actually really
challenging because you are unable to in
this carol World Define variables and
keep track of where you are other than
sort of by manually stepping through the
world placing beepers and changing
colors um so the students that were able
to do this we think were really quite
smart um and we tried as well to add
some things which are sort of related to
uh the work that we do in business
analytics like optimization so this was
sort of finding the minimum number of
beepers in in a random World um and this
was also surprisingly deviously
difficult to do given the constraints of
of
Carol
okay so this is an example of the
leaderboard I apologize it's a bit small
but uh we were able to keep track of and
students were able to see in real time
their position on the leaderboard um
they got their name on the leaderboard
the number of points uh and also which
tutorial they belong
to uh this is met with uh some students
like this some students don't like this
we'll get to that later um so we had
some quite good success in our first
semester um semester one these are some
of the the winners of awards um so I
think on the left these are the sort of
the top five or 10% of students in the
unit that I coordinated and then on the
right Allison is presenting the award to
the winner from uh her unit cubus 1040
and then these are the similar uh
cohorts in semester
2 so we had quite a large number of
students participating these are people
who we are sort of at the top of the
cohort and were winning the larger
Awards we also a rewarded participation
at different levels so um I've got some
trophies here as an example and if I can
get Nikki or someone to pass these
around as a little bit of a prop These
are 3D printed trophies um that we put
together um for in recognition of the
students
achievement um so so we also had
personalized certificates uh and also
smaller 3D printed keychains maybe if
you've got a good question we can hand
out some of those um and we did was a
little bit laborious making these but I
think the students appreciated
it uh and we did see quite a bit of
evidence of this uh so students uh
posted on in particular Chinese social
media because a lot of our students are
are Chinese International students U and
in the translations um they said they
were really happy and proud of their
achievement which is really great to see
and one of these students on the right
hand side was quite upset because she
actually missed out on the presentation
ceremony but we were able to get a
reward to a later um so in terms of
participation and impact uh we've had we
had sort of a thousand students in the
first semester and uh about 900 students
in the second semester and 200 200 and
350 students participating so we had a a
k degre of
participation uh generally the feedback
has been positive um so students uh have
reported that they felt that Carol a FL
over me um have helps them understand
things Concepts like if El statements
and Loops in particular those are two
things that our students really have
struggled with in the past and also uh
students have reported that they felt uh
that it made them think that programming
was more fun and developed things like
critical thinking and problem solving
skills and they reported that part of
the reason for that is that they can
visualize exactly what's happening in
their program at any
time um in terms of uh we have done some
informal surveys from semester one and
now we've we've done a a formal survey
in in semester 2 but I can't discuss the
results of those just yet so I'm going
to report the results from semester one
um generally students thought that Carol
uh had a positive impact on the way they
thought about programming and improved
their understanding of programing
Concepts and they also found that it was
fun which is part of our a broader goal
to make programming fun for the
students uh students felt the sort of
comfortable with this process which is
really great to see uh and in general
there was an uplift in responses to
things like being able to uh being able
to use uh python to solve uh problems
and that they they indicated to us that
they would continue to undertake uh
courses which required programming which
is the majority of courses in our
field we also asked them why they didn't
participate most of the time it was
because they didn't have time uh or some
of them reported they didn't have uh
didn't know about it or weren't
interested in it um so that was from
semester one and we had a greater uplift
in semester 2 so hopefully we can reduce
this number or have a nicer looking
graph for semester
2 now um one of the things I wanted to
say is that what we're doing here is not
really about Carol itself or using Ed um
but what we're trying to do is create a
playful learning environment for our
students where they can learn and
experiment and build confidence with
programming um so our checklist for
Success um does not really involve Carol
at all um if you wanted to implement
something like this in your own teaching
context this is what we would recommend
so we would recommend aligning with your
learning outcome so for us we chose
Carol specifically because it's a subset
of python so if we were using a
different programming language we might
have gone with some other
tool um it also specifically addressed
student weaknesses because it
specifically targeted use use of
conditionals and
Loops uh we wanted to keep it
interesting and fun and we recommend
that this is well we think this is
really part of the success of this
program so we design challenges with a
range of difficulty levels and
seriousness so try to put some
playfulness in your exercises um don't
make them sort of super boring move
Carol to this position and return try to
get them to do something fun like make a
QR code or solve a
maze try to create a safe space for
mistakes so this had no impact on the
student marks students were were
participating completely voluntarily so
if students did didn't perform very well
it was there's no harm no fou and
students were sort of free to make as
many mistakes as they
wanted we also think you should give
Fast feedback uh so we gave automatic
feedback and immediate feedback through
an automatic marking process so if
students aren't getting that um fast
feedback from something like this
program I we feel like they're not
likely to continue doing it um and that
feedback could be sort of a yes no youve
passed a test case or not and it can
also be part of the actual Carol
visualization process too so seeing that
2D grid world is also part of the
feedback uh motivate so we we motivated
our students through a live leaderboard
with mixed success some students liked
it some students didn't and we awarded
students prizes and we awarded prizes um
for a medium level of participation all
the way up to doing really well like
scoring the the highest points so we
think you should actually not just award
those um who did really well but go back
down and try to encourage participation
ahead of time and what we did as well
with our students is we said hey if you
meet this minimum Benchmark which was
sort of doing half the challenges you
would still get some sort of um
recognition of your achievement in this
case a
certificate we also encouraged student
communication through use of a
discussion board and we tried to limit
staff intervention on the uh discussion
board for these specific questions
because we wanted students to build a
bit of a community around Carol and
again I think I've already touched on
this point of recognizing achievement at
all levels um don't try to don't just
limit it to the top performing
students so what's going to happen in
the future so I already mentioned this
but in semester 2 we actually launched
this as a formal research project within
the university um this involved larger
scale surveys so we have much uh higher
uh response rates to these surveys and
we've also already run focus groups
we're waiting waiting for final exams to
be completed so that when we can match
Carol activity to assessment scores like
exam questions that Target programming
and we're also going to be able to
control somewhat for predis
predisposition and self- selection
because we have access to past student
performance data through their entry
scores um scores in other units and
things like that so we'll hopefully be
able to um accurately model whether the
effect is due to Carol itself um or just
the students's predisposition to
participation and we expect to publish
these results in 2025 sometime and our
plan is actually to kind of keep
repeating this and improving on uh our
initiative into 2026 uh as a part of a
longer term
study uh and lastly I'd like to wrap up
by saying thank you to the university
for funding us to do this we did this as
part of a playful Innovations Grant
within the uh within the business school
uh and we spent most of our money on a
3D printer which is really nice
um but yeah we'd like to thank the
university for supporting us uh in and
letting us do this um and that's all I
have to
[Applause]
say okay thank you Stephen uh we have
already got hands up all over the
place that was really awesome thanks um
do you have a sense of whether there's a
point at which the Simplicity of the
environment starts limiting what people
can learn or is the Simplicity good all
the way through yeah there that's a good
point there is a limit to it so we
introduced this in the first half of the
semester where students are learning
those fundamental concepts like
condition we tried to match it to the
times when they're Lo uh learning
conditionals and loops and things like
that and yeah I think beyond that um it
has less of an impact on those learning
outcomes that you're probably trying to
Target towards the end of a course where
concepts are much more difficult like
objectoriented programming things like
that there's no real way to sort of uh
mirror that with Carol so yes there is a
limit a point which we probably suggest
that you maybe move on to something a
bit more real or different than than
Carol make people angry or frustrated
just oh the student seem to understand
they and in their focus groups they've
reported that they can see the clear
connections between the the unit content
and what is happening in Carol um but
yeah they they appreciate that also it's
kind of just a bit of fun that has has a
limited lifespan yeah for
them I was wondering do you combine the
like you said give rapid feedback but
you also said you know give prizes
rewards that kind of thing do you
combine those in any way and give prizes
sort of throughout the use of this like
you know each week or you know at the
midpoint or whatever or is it just the
prizes at the end as the like main goal
so we've limited it to prizes at the end
mostly I think for uh simp licity of
administration um we're trying to do
this alongside teaching the normal stuff
so yeah I think if if we could we could
we have well in the past actually the
first semester I ran this I in my
version of the unit I had quite a
complex Awards uh sort of structure much
more complicated than Allison's and I
gave out uh prizes for things like first
to complete all the exercises in that
week and then uh or first or person who
scored the most within a week but it was
still only at the end because I didn't
have time to keep up with it throughout
throughout the semester but yeah I think
it that would definitely help uh improve
participation and cons consistent
participation because one of the things
that students have reported in focus
groups is that they really want like the
the prize and the award at the end but
they will leave things to the last
minute as classic student Behavior
they'll try to do all the challenges a
day or two before the due date closing
date
yeah yeah thank you um is your course
workk open sourced it feels like it
could be useful to schools as well uh
our coursework is not open sourced um
where Alice and I working on a different
project that's not affiliated with the
university to do something like that um
but uh yeah not not this specific unit
course Stanford thing right yeah so the
Carol uh thing came from came from
Stanford um you don't need to use Ed or
anything like that there are open source
Carol packages so that you can um just
pip install a Carol package and run
Carol locally on your machine or some
other environment
yeah hi thank you so much for your
presentation I really enjoy enjoyed that
I was just wondering did you have any
data from prior to the Carol being used
in the courses to Baseline student
engagement and participation uh we do
have data on that um so we have data
through Ed of um all of our tutorial
material is hosted on Ed and has a quite
a com comprehensive analytics system so
we can see which exercise students have
uh participated in and completed and
things like that number of attempts and
so on was there much of a difference we
haven't got to got to that point in the
analysis the other thing is that our
ethics approval for studying student
data is limited to from semester 2
onward so we're not actually allowed
within the university to include student
data from prior to that
unfortunately um from sorry that was
really cool but from like after students
were finished with Carol did you notice
a difference in sort of like onboarding
them into uh more widely used uh
programming languages and that sort of
thing for us we stay with python
throughout the entire course and most of
their degree is done in Python some
units will touch on are so I can't
really comment on that yet uh hopefully
we can get some insight into uh Pro
improvements in programming performance
um once we get to analyzing their sort
of exam scores and things like that at
the end of the
semester um I noticed in the graph of
like people participating or not
participating uh that a lot of them said
that it looked too hard um what exposure
do they get to like what it's going to
be like before they get started on it
and then like what are you doing to help
them like get over that hurdle yeah so I
mean the they get to see exactly uh some
of what I've shown you before um so we
have the set of introductory slides um
and in some of our tutorials as well we
also um demonstrate this and get them to
work through the first exercise um so
that is was imp the in putting it into
tutorials was implement in semester 2 so
those results were from semester one
yeah so so this is a question but I have
to give you a premise I have to give
people a premise I um I used some of
this stuff with my year n students and I
found that I was I was
blindsided about halfway through I went
to check how people were
progressing and a quarter of my class
had not used a single Loop and just
copied and pasted move and turn left a
billion times y
um uh this is not a judgment of me of
course um so I'm wondering if you've
come across that and if you have thought
about how to guide people to like
getting their code a little you know
more yeah we definitely have encountered
that uh one of the problems that
students report in their focus group is
that sometimes their Carol programs took
too long to run so even though you can
run it at 100 times speed some students
report to take some of those programs
running for half an hour with 100 time
speed um so they are definitely not
writing things well a lot of the time
um in terms of actually addressing that
problem we haven't really come up with
any solutions for that the best thing I
can come off the top of my head is
probably having some sort of dedicated
out of class creating sort of like a
peer learning group or sort of like a
study session for Carol to help and
guide those those students
um change a scrip link a
we we could do those sorts of things uh
we we wanted to make it fun and we
didn't really we wanted to kind of
encourage independent problem solving we
kind of didn't really care how they they
did it too much because part of the goal
is encourage participation don't force
them down a narrow path
yeah great thank you very much Stephen
can we give him another round of
applause please