hello and welcome back to pike online
2021 the education track we hope you all
had a fabulous lunch break um and i am
joined here by stephen tierney and ali
wong
on what promises to be a bit of an epic
tale i'm really excited i was going to
give a nice intro to both of our
speakers but apparently they've got that
covered so i don't have much of a job
here at all other than to welcome you um
but but i'm really bad at not doing that
i need a job too so steven and ali are
from the university of sydney both of
them and they're going to take us on an
epic journey of what i can only assume
will be
hope uh despair and i very much hope
success
um i i really want to see this heroes
heroes tale the the arc of learning
um
i'm i'm excited to go on this journey
with you both um they will be asked they
will be answering questions perhaps they
will also be asking questions uh at the
end so as always throw those questions
uh in the chat uh and
i'm i'm keen tell me a story
all right hi everyone uh thanks for
tuning in to our talk today
um this talk is about our recent
experiences with teaching uh remotely
due to covid over the sort of last year
and a half and we're going to be telling
our tale through the lens of lord of the
rings there's going to be spoilers so
just be warned but i make no apologies
because it has been 20 years since it
came out
hi again everyone so i'm stephen i'm a
lecturer at the university of sydney
i kind of teach a mix of statistics data
science and machine learning and i work
in the discipline of business analytics
within the business school
and i'm ali i'm an astrophysics phd
student at the university of sydney and
i tutor programming in data science and
computational modelling units and i've
also recently joined the super lovely
team at ed
so we'll talk about ed a little bit
later
but
first i want to go back to our proposal
and we originally wrote that we can
reach any student no matter their
location time zone or computing device
with a real-time collaborative coding
environment
that sounds like a pretty big brag
and that's not the main point of today's
talk we don't want to just show off we
also want to share our journey uh how we
got to that point uh so that you can all
learn from our mistakes
and what we consider must-haves to be
able to do that
and whether that's
for small classes or teaching
one and a half thousand students as in
our case and also how we actually went
about implementing those ideas and
techniques
and we're coming from a tertiary level
position but we think our talk is going
to be useful for educators for all
education levels whether that's primary
or secondary as well
so just to give you some context uh we
have been teaching this unit bus 6002
data science and business for a few
years
it's a postgraduate unit at the
university of sydney business school
it's a sort of as an introduction to
data science and also an introduction to
programming
and we teach with python
and we teach with a view to actually
doing applied data science
unfortunately for us though uh our
students often have zero programming
experience uh they're coming into this
uh program this degree uh usually
pivoting out of some other undergraduate
program
that's typically unrelated entirely
this is really a challenging environment
to work in
although i think a lot of you are
working in similar environments where
you're teaching novices for the first
time
and just to give you an idea of what
we're teaching over our sort of 12-week
semester period
we start with python so the basics of
variables conditionals iteration those
sorts of things so programming
fundamentals and then we move through
some data wrangling exploratory analysis
and even numerical computing using
standard packages like pandas matplotlib
and numpy and then finally in the last
few weeks we get the students to
actually do some statistical inference
and modeling using psychic learn
for the pythonistas amongst you that's
probably a piece of cake but for these
students that's really fast
and just to give you an idea in sort of
an undergraduate computer science course
usually
sort of the programming fundamentals are
covered over an entire semester not just
the first few weeks like we teach it
and in the past we used to teach using
the one ring to rule them all jupiter
so every tutorial was uh the material
was distributed as a jupiter notebook um
and we chose notebooks because they are
the de facto tool for data science uh so
we thought okay well we need to give
this equip the students with the ability
and knowledge of actually using the real
tools
but
over time we started to notice some
problems with that
so often our notebooks are really large
they were really long and tedious
just trying to cram all that information
in
and because of that they
students kind of fell a little bit into
a passive learning mode they would just
sort of sit back and run the pre-written
examples rather than writing much code
themselves
and there are other some technical
problems with notebooks like when we
when we start students on notebooks they
kind of don't understand the notebook or
the jupiter model
they sort of don't understand out of
order execution and then how there's a
sort of a jupiter server running in the
background that they access through a
browser that's quite confusing for
students and you have to get students to
install jupiter at some point as well
and we used to teach in
physical classrooms that would normally
be some sort of computer lab or a
learning studio here's a picture from
one of the newest buildings of the
business school
in a learning studio
and in this sort of setting we would
have two staff members always for us
a class size of about 30 students we
would have a tutor
who sort of runs and manages the class
and then we have a python helper who is
there to provide assistance to that
tutor and also individual assistants to
the students
after all frodo wouldn't have got far
without sam
and
this made it really easy to monitor
student progress and build rapport with
students over the semester and students
also formed bonds and friendships with
each other which was really nice
so we had a system the system was
working it was great but then our
semester one last year around about week
four something that no one saw coming uh
was covid so very quickly the university
was locked down and we kind of just had
to like migrate everything online
overnight so as some of you guys know
teaching programming is already
challenging enough but then teaching
online without all the proper resources
in place felt like a near impossible
fate and so there we were coverts
happening and we've just been thrust
into this remote teaching with zero
experience and so we're kind of
struggling a little uh we felt lost we
felt like we were blind our usual
methods of teaching where we'd be
hands-on in the classroom talking to
students helping them with their coach
heading with them to see how they were
doing uh that was all just like gone
and we needed a way of replicating the
success of our in-person tutorials to
work in an online environment through
video conferencing tools like zoom
and i'll be honest uh we didn't do well
initially so attendance was just
dropping questions weren't being asked
and students just weren't uh completing
the tutorials
but the worst was yet to come so little
did we know there was an army of
students approaching that we would have
to teach we were facing 1200 students in
the next semester and then 1600 in the
semester after that and so we thought
well we need our own little army of
tutors equipped with the best
educational tools in order to survive
so we didn't have that much time to
prepare but we needed to kind of stop
and think about what we needed in the
classroom for our mission
uh the first thing that was really
important was we needed to get the
students engaged so we know from the
literature that students learn best in
an active environment but unfortunately
our students they're all hiding behind
their screens
silently watching their tutors walk
through each jupiter notebook and a lot
of students were unfortunately just not
participating in class or doing any of
the exercises and attendance was pretty
dismal
then another really important thing that
we needed online was we needed
visibility and we just had none we were
totally blind so all of the students are
sitting there with their cameras off and
we had no idea if they were
understanding what was going on whether
they were just confused or just bought
out of their minds or to be honest even
if they were there
at all so we knew some of the students
were working but we just couldn't see
what they were doing we tried all sorts
of things like breakout rooms or screen
sharing but it just wasn't effective
because logistically you can't have 30
or so students trying to share their
code simultaneously and also online
students just weren't as proactive
in seeking help and we couldn't really
just jump around the classroom and talk
to the students as we had done
in the past now related to
visibility is feedback so
for all of the students still continuing
on this journey with us we needed to
support them with feedback and that's a
really hard thing to do when you can't
see how they're going
not to mention it's like really super
awkward if
you're trying to give some feedback to
an individual student and there's a
whole class watching because no one
wants to be wrong or kind of feel silly
in front of a class appears except
sometimes you do get the one student who
is okay with that
but most students they needed space to
make mistakes
and to learn from them
now students also expected and they
deserved consistency so we needed to
teach and be consistent along our own
modes of teaching whether they were on
campus or uh learning remotely uh
they're paying the same fees they're
earning the same degree and in later
semesters we had parallel streams online
and face-to-face and we needed the
classes to be the same so we needed a
way to get the learning materials and
also methods of feedback in a way in
some sort of way that worked in both um
settings
then finally probably the biggest
challenge we had was scalability so with
enrollments climbing uh quickly we
needed to be able to find a way to
deliver high quality education to
literally hundreds and hundreds of
students not only for this semester but
for every semester afterwards
so previously these things had seemed
impossible but i come to you at the turn
of the tide so we came up with this idea
to use the newly developed lesson
feature on the ed platform and the
lessons allow you to generate
interactive content which students can
work through alongside
their tutors so what we'll do is we'll
quickly and briefly kind of show you
what this all
means
so on ed we have these things called
lessons and each lesson consists of a
number of slides so these can be
anything from like static information
think powerpoint and like lecture
presentations and like here we are about
hobbits and if it so interests you you
can come and learn about hobbits uh but
what's really cool is you can have
interactive code examples on each page
so
you can have these little experiments
that students can come and play with um
so here's a code snippet where you enter
your name here let's put in our name
aragon
and
we can go ahead and run that
so it's just really nice to have this
kind of interactive slide for the
students
now another really cool thing on ed is
that you can host quizzes and surveys
so it's really nice because students can
test their understanding and as cheers
we can kind of see what concepts our
students are stuck on so let's have a
look here
does anyone remember what gimli asks
galagio for i think it's a locket of
hair
and also it means that students when
they're testing their understanding they
can ask uh tutors questions um in real
time and then tutors are also able to
look at the analytics and check to see
what concepts our students are getting
stuck on
so they don't have to be proactive when
asking for help
now probably the most exciting thing
about the lessons is that we have these
um code challenges and here students can
try and solve these code problems by
typing their code they can go ahead and
run it and they can also go ahead and
mark it which means they get feedback
straight away um
and so they can they don't have to wait
for a tutor to tell them if they're
right or not and the most exciting part
about this is that tutors can dive into
a student's workspace so we can kind of
come in and spy on the students um the
way you would kind of do so by leaning
over a student's shoulder in class and
they can also come in and code alongside
a student in the same manner as like you
would work with someone through a google
doc so stephen is now going to
demonstrate this
okay so i'm going to switch over to
the sort of
tutor perspective
of the analytics page
so hopefully you can see this
so i'll just quickly explain the typical
setup of how we would run a class
so normally if we're teaching online
then the tutor would create a zoom
session which the students all join
and the tutor would share their
their screen with the students and walk
through those concept slides and quizzes
and allow time for students to do those
quizzes and so on uh that asks students
questions and and then when we get to
the
challenge questions the programming
challenge questions the tutors would
then uh direct the students actually
complete those and give them uh
depending on the complexity of the
question uh anywhere from a couple of
minutes to 10 minutes to complete those
questions
and during that time we as tutors or
instructors would be monitoring those
the student progress through the
analytics page
and so on this analytics page this is a
class and we've got sort of a class of
characters from lord of the rings
and left on the column left column you
can see the the students and then along
the top row here you can see the actual
name of the particular slide challenge
quiz and so on
the green ticks
indicate that the student has passed or
read that section just fine yellow means
they've opened it but they haven't
completed it or they haven't clicked
mark or so on and the red means that
they have tried to mark and they've
they haven't passed that exercise
now this will update in relative free
time real time all we need to do is
click the refresh button so i can see
now that
alison who is playing the role of aragon
here
has gone through and looked at some of
these slides and attempted some of the
quizzes and so on
and if we keep going we can monitor that
student's progress
and what would normally happen is
when we get to
a point where we can see that a student
has failed and
particularly a programming challenge we
would then click on that workspace so it
looks like
aragon's up to this how would you get to
mordor exercise so let's go take a look
at what aragon's up to so we can jump
directly into their workspace and type
alongside them let me just hide the
slides on the side
and hopefully you can see that i can see
aragon's cursor here and that's allison
typing in real time
and they
haven't they've had some difficulty with
this challenge and i'm playing the role
of the tutor here and typically what we
see is that students will be working
alongside and they may leave a note as a
comment and say hey it's not working and
then as an instructor we come in and say
okay here's the problem or give the
students a little hint so in this case
it looks like there's a
string quotation mark issue or string
closure issue um because of this
single quote
inside the other single quote so i would
say
hey aragon uh just check your
string
uh looks like
uh you've
got a single quote inside it
and hopefully that's enough of a hint
uh
for the student to go oh yeah i remember
i i need to use a different style of
quote when i've got a single quote uh
inside a string
and then uh
from this point onwards uh
the student would then click mark and
check whether they've got the right
answer again and we could go back to the
analytics page and then hopefully
if
alison can click mark
sorry i need to switch tabs to see what
alison's up to
yep there we go we can see that uh
aragon's now past that exercise and
aragon can move on and we can now we can
move on to
another student
so that's the basic idea of how we teach
we found it to be very effective
going back to the sorts of things that
we can do in challenges
they can be quite simple we can do sort
of like text based or diff based marking
so we can get students to write a
program and then compare what their
program outputs for a fixed set of
inputs and we can even do some more
interesting things like
uh
actually introspect a
matplotlib plot for example so in this
exercise example i've got here
we've got the middle earth sensor so the
the students are asked to
generate a
uh a bar chart that shows the
distribution of races within middle
earth and so if i go ahead i've got a
sort of a partially complete bit of code
here and click mark
we can see that the specific elements of
the plot haven't been achieved so we can
we actually introspect those plots and
go okay does the
y-axis label match does the title match
so for example if i wanted to add a
title i'm going to say plot dot title
and then we'd say races of middle earth
or something like that i think that's
the desired title races middle earth so
if i click mark again
we should see that i passed that
specific title test
okay
that's our demo
um
so what actually happened after the
first time we did this so we rolled this
out in semester 2 of 2020
during the semester we ran our own
survey just to get some feedback on what
was happening and how the students felt
particularly about using ed
and we asked them a question
do you think ed is an effective way to
learn
94.6 of them agreed on a leicard scale
so they either picked strongly agree or
agree
and follow-up questions on the
difficulty of the exercises the breadth
the pacing presentation revealed very
similar levels of agreement
and here's some quotes from that uh
survey so
students often felt that uh ed was their
favorite part of the unit
so students thought it was magical they
loved it
and then they they felt like uh it was
making uh the work more exciting and
therefore easier to learn
and we also
got the highest ever unit of study
survey outcome as a result so this is a
centrally run survey by the university
um and we believe that a large uh
one of the reasons for the uplift in our
unit of study score was because of how
we used ed
and
we've continued to have uh higher uss
scores since then
and typically students can be pretty
harsh on this survey uh because they're
responding on anonymously so they're
they're pretty harsh and honest so we're
really happy with this result
um and so we think this sort of
experimental tutorial design paid off
um and
uh we were really happy to see at the
end from that unit of study survey
students saying that they felt like
there was really no difference to
learning off campus as to learning on
campus and they
felt like even though everything was
online
they were able to really learn
successfully from it which was a big
change from our first semester during
covert
so just to close the loop on our
on our sort of our must-haves that we
defined earlier
on engagement we saw increased
attendance so attendance was uh much
more consistent over the semester
and in within the classrooms students
were much more active
with visibility we have great visibility
we can see what students are doing in
real time
and with feedback we can give them
actionable and immediate feedback
there's very little barrier to giving
feedback it's kind of limited to how
quickly you can type back to the
students
and then on consistency students as you
saw in the previous quotes tended to
think that okay learning this way is
basically the same as learning in person
to a certain degree so students were
happy with the consistency aspect
and for us the scalability side of thing
uh we definitely didn't have we didn't
have any issues that prevented us from
teaching uh during the semester so
in semester two last year we used this
system with over 1200 students and then
in semester one this year was sort of
over 1500 students and we have about 44
tutorials every week with many of those
concurrent
so definitely
scalability wasn't a problem
there are also pos some positive side
effects from using ed this way
it turns out that ed is quite student
paced
students can work at to whatever pace
they like the um
the automatic marking allows the slower
students to to safely fall behind
because they can catch up later and use
the automatic marking
for self-directed learning the faster
students we equip those students with
extension exercises so they could move
ahead of the class and get to those more
difficult challenging exercises if they
wanted to
and the other side effect of using ed
was that students no longer had to
install anaconda or jupiter
all they needed was a browser because ed
is a browser-based environment
so we sort of saved a lot of time
debugging and
providing technical support to students
with jupiter
what we did wasn't perfect by any means
uh there are still areas of improvement
uh for example it's really hard to make
a personal connection with the students
even though we do have the ability to
give students immediate feedback a lot
of the students are resistant to turning
on their cameras in zoom meetings and
and so on so it's hard to make that
personal connection with them with the
students and then between students it's
also hard to make personal connections
so there's
very little opportunity for peer
learning but there are some features
being added to ed that we've seen in the
pipeline which might make that a bit
easier
other than the sort of the social aspect
it is actually still quite a lot of work
to do this
moving to ed required a huge amount of
time and effort from allison and a
handful of other people
and
not only effort but actual expertise in
designing and actually
using designing exercises and actually
writing tests
so for example for each exercise you
need to come up with a description
hints examples so students don't get
stuck you need a scaffold or a starting
point for those students
you need to collect any data or whatever
the students need and make it available
to them you need a solution for that
normally
and you also need like i said a suite of
tests to actually do the automatic
marking
so over the past few semesters we've
built up a huge library of programming
challenges so we kind of have over 200
it's probably closer uh
it's i think well over 200 programming
challenges built as a result of this
but it's not for free you have to do the
work yourself
here's an example of what such a test
would look like
so it might be a little bit small to sit
and hard to read on your screen but this
is just the to give you an idea of the
scope of a test uh that's required for
one
exercise so this is the marking code or
smart code that's similar to the um the
middle earth census graph that i showed
earlier
so there's a lot of work to be done you
have to do this for every exercise
and
i think you also need to do this as a
team it's very difficult to do this on
your own it would be very difficult to
do this on your own so to give you an
idea we have one of the largest teams of
instructors in the business school we
had 20 instructors in semester 2 last
year and over 25 in semester 1 this year
and it really wouldn't have been as
successful without such a big team
we had people writing
material like alison and priyanka and
daniel and david also wrote a small
amount of material
people did qa on our exercises and
material each week to make sure
everything was working as expected ahead
of time and then everyone or the rest of
the instructors would always give
thoughtful and feedback and deliver the
material every week so
on the scalability side of things you do
need a large number of people when
dealing with a problem of our scale but
um
i think you can uh
get away with a much smaller team if
you're delivering this to it to a maybe
like a in a primary audience or a
secondary audience so a special thank
you to all the people that i've listed
here who went the extra mile and
contributed heavily to bus 6002
so that brings us to the end of our talk
we'd really like to extend a sincere
thanks to the pycon organizers and
volunteers who make pycon possible every
year so thank you everyone
and
that's it from us
thank you so much when i return remember
to turn my mic back on um that was
fantastic we are almost out of time but
we can see if we can sneak in
um one or two questions uh let me just
get it
um there was the one question on can the
students collaborate
yes they can collaborate
do you want to take this allison ah sure
do you want to pull up the workspaces
uh yeah let me just grab that link so
we've been demonstrating the lessons but
there's also a feature on ed called the
workspaces
and uh it'll be in the same kind of
course so here you can see
we have a workspace here stephen can
type
and if you click the kind of little
button that looks like a speech bubble
next to the share that will also bring
up a chat so students can also come in
and talk here
or come in and say hi and photo can come
in and say hi so students can kind of
work on larger kind of code
problems together
there's also
another example where they can code on
like a jupiter sort of environment
collaboratively as well
fantastic and that actually also
partially answers this other question is
there a chat function in a workspace so
fantastic excellent
um let's see if we've got another second
but yeah we've got another we can feed
um
so this question is in my class the
difference between the completion rate
of the front and the back of the class
is huge how did you manage that
we have the same problem i think um
yeah same sort of distribution of
completion rates um i don't know whether
i don't think we have the answer to that
or at least i don't can you think of
anything alison
uh no but it's kind of nice at least
that even if you're not completing
things in the tutorial um like in the
live one-hour time slot you have our
students can kind of keep completing
things at their own pace whether they're
fast or slow so that's kind of a nice
thing about the way we've set up the
lessons
that that certainly is a
a pressing challenge for anyone who is
teaching coding and computer science in
general um
yeah it's
uh that that was a fantastic talk thank
you so much thank you for answering
questions there are a few more questions
um that i will invite you to answer in
the uh education hallway chat and i'll
copy them in in just a minute but we are
out of time here for now so we all have
a
15 minute break um
before our next talk um which is katie
bell talking about real python made of
blocks so thank you ali steven thank you
so much for joining us um thank you
fantastic talk what an epic journey it
was
less walking than i remember there being
um uh and
thank you thank you very much uh we hope
you uh we'll see you all in 15 minutes
cheers
thanks guys
bye
you