hello and welcome back
our last speaker for the science data
and analytics track today
is now based in geneva but he used to
attend the melbourne python users group
back when he lived here
it feels really good to be able to
introduce him to the pycon australia
audience
last yenken is a swedish australian
engineering leader and data scientist
currently leading the tech team at our
world in data where he tries to wrangle
the world data sets into shape and share
insights into how we can make the world
better using that data
however most of the world data sets are
scattered to the wind or worse
stuck in excel files of massive
organizations in tiny addendums to
research papers
in pdf format and even in government
tweets
lars will take us for a dive into
initiatives by our world in data
to unlock this information he'll share
the joys and pains of data harmonization
and ask how meta should data really be
so please give a warm virtual applause
to las janken
and to his talk systems of the world
cataloging the world's data for great
good
thanks so much fabio
um and i should say yeah it's really fun
to be able to uh join remotely uh
having you know being an old melbourner
so as having mentioned uh i'm from our
world in data i'm going to be talking to
you about our efforts to catalog the
world's data um
a little bit about me you know i
actually i'm born in melbourne but i'm
also part swedish
i've spent the last five or six years
living in stockholm in sweden i had a
little boy there we moved our little
family to geneva recently and recently
i've been working for our world in data
which is what i want to talk to you
about today
um
most of my kind of working career has
been in data science or in
leading engineering teams in the health
space but at our world i should say even
the short experience so far at our world
in data is quite different from these
other data environments and i thought
this would be something people might
enjoy
today you're going to learn a bit about
our world and data and our mission
you're going to learn about what's
involved in cataloging the world's data
and you'll learn how to find and query
some of this data for yourself
so let's dive in
a world in data is a small non-profit
it's spun out of oxford university
by our founder max rosa and you know
it's basically like a handful of
researchers a handful of data managers
and a handful of developers who are
working together to try to
shed some light on the world's big
problems
and our mission is really to make the
data on these really big problems
understandable and accessible
we do that by
writing articles that try to explain a
lot of detail about these problems and
make it accessible
and we write about
poverty
and inequality
we write about hunger
about disease
and about climate change
amongst a range of other topics
and now
what one thing that really distinguishes
the writing here other than the kind of
depth of research behind it is that
everything is backed up with these
really rich interactive charts
so for example
here's a chart of
code vaccine
copper 19 vaccinations over time
and
this chart you know you can
change the time frame you're interested
in
you can
add countries so we can add australia
and i know people are suffering in
lockdown and trying to vaccinate as fast
as possible in australia
so i um but you can see that this curve
is definitely on the rise
it also does interesting things like if
you
click the time frame
you know you get an appropriate
visualization like a bar chart
you can
visualize things in maps you get a lot
of source information and you can easily
get to the underlying data too
in order to
you know in order to write about these
topics and have a lot of these rich
charts then actually we need a really
solid data catalog behind them
so let's take a look at
what's involved in building this data
catalog
so
we want to kind of help promote this
evidence-based world view on a really
wide range of global issues and if we
want to do that then we need really good
data from a wide range of sources
if you've tried to pull data from
different places you will notice that
all sources use their own special format
generally speaking i would say things
are getting better year by year
but you know the world doesn't share a
single uh
a single format for exchanging data
and
so that what that means is that you have
to deal with a huge amount of variety
from different institutions and
different researches
you'll also find that uh every
institution has its own view on the
world even if we just talk about
countries so
institutions often report countries by
name rather than by iso code and they
also pick they use their own names
sometimes
it's uh
uh it's kind of semi-political which
name a country is used for a country
countries have disputed borders and
different organizations decide what's in
or out of a country
um
they also
based on the the slice of global
problems that they're looking at then
the organizations also will pick like
their own
regions for the world
so we might think of the world as being
needly split up into different regions
like europe and
asia and north america and things like
that but actually the organizations cut
the world up into special categories
like
low-income countries middle income
countries
east asia excluding china south asia
excluding india things like that
if you want to work with all this
variety you need to bring everything
into this sort of common format
and then
think about it from like a database
perspective if you want to if you want
to be able to merge or compare things
from different data sets anything that
you're going to join on you need to
harmonize
so uh that's especially things like
countries but it could also be things
like diseases and the words they use for
genders and things like that
now
the covet 19 data set that we collect
and republish i mean this is actually um
an extreme case of
what our world and data goes through to
collect data
roll back a year the world was just
entering this pandemic
governments are scrambling even to
accurately they're scrambling just to
test
let alone report their testing
so
the data reporting is only just
beginning to come online for different
countries
and
the really big institutions like the
world health organization that we might
expect
to collect and publish this data they it
took them actually quite a long time to
come to the table
so that our world and data team had a
tricky pivot to decide on
most of the big problems in the world
like child mortality you know they don't
change day by day
these big problems you know the data
sets on them they're updated a few times
a year or maybe even once a year
but covert 19 data was coming in every
single day
but the team basically leaned in to try
and fill this gap
and that involved an immense amount of
work but that work enabled
a lot of
countries to be able to respond to or to
have better evidence to base their
policy responses on
the
one area or just generally speaking all
this country reporting is getting better
and better big organizations have come
to the table
so a lot of the things that used to be
hard are a lot easier like
testing and cases and deaths
but vaccinations is still a really
interesting and difficult one
so
how well the data has a nature paper
where we discuss uh the assembly of a
vaccination data set but when it comes
down to actually pulling it together
there's like all the usual suspects that
you can imagine when you're trying to
pull data in from wherever you can get
it
and
you know there's apis that there's
csv and excel neither of which are that
terrible pdf was actually kind of
prominent early on in the pandemic like
the world health organization was
publishing data and pdf
and
you know also
numbers embedded in images and
infographics
and actually we still have them
but you might be surprised to realize
that
you know in this data set there's also
daily figures that are only given by
video
or daily figures that are posted to
facebook only or daily figures that are
only posted to twitter
and uh there's still 12 countries that
our world and data fetches the daily
figures from directly from
twitter
interestingly as well as like this
difficulty getting access to data there
was one interesting case of like too
much data
the government of paraguay actually
published
vaccination status for every single
citizen by name
so if you want to know how they're doing
with vaccinations you used to have to
actually
fetch this entire data set and then of
all like you know people's names stuff
you don't want to know but actually then
roll that up into summary statistics
thankfully now they do calculate those
summary statistics so we don't need to
be fetching all of this very personal
data on their citizens
so
we've got data from these crazy sources
and we're trying to bring it in and what
does it look like what are we trying to
bring it into
our core data model is a tuple
of four elements
an entity which is often a country but
it could be something else it could be a
disease
a variable
a time point which is used at either a
year or a date and a value
so for example here we could we could
read this tuple as
australia had a life expectancy in 1921
of 61 years
and if you're curious the life
expectancy today is more like 80 to 83
years so australians are living 20 years
longer than they used to 100 years ago
so
when it came to our world and data i had
you know i've been working in these
commercial environments doing data
science
and
there was a couple of things though that
were really really different about the
data we're working with
in a commercial environment let's say
you're a successful company you might
have millions of customers
um
and
for these customers
you probably
have a lot of data terabytes of data
uh it will be split in a few different
categories some of it is like the direct
stuff that
is about your product
some of it will be like marketing
acquisition data some of it will be
automatic machine logs
especially for debugging
but so you have this massive data set
but a ton of it is machine generated
automatically captured
and probably it's only a few hundred
variables actually but a huge number of
time ports a huge number of recordings
at our world in data you know there's
not that many entities we care about
like we're especially looking at
countries there's only a few hundred
countries
you add in some debian diseases and
types of fish and different things you
know still the number of entities are
small the number of time points will get
us small
but the data set has like over 100 000
different variables and this is by far
the most diverse data set that i've ever
worked with
every one of those variables is not
machine
captured but is instead something that
is
like a research team has painstakingly
put together and people have had to go
out and collect and estimate so there's
a huge value in this data set as well
it's completely different shape to a
commercial data set
so if we've got 100 000 variables then
uh you know we need more structure on
them than that to even navigate our own
data set and in fact we do have more
structure we organize these variables
into data sets and data sets in name
spaces
so for example
the one namespace might be the world
health organization
and they release for example uh data
that one day set their release as their
global health observatory and so that
might have that might be you know a
gigabyte or two and that will have
thousands of variables and all of that
will kind of come together
something else that's very different
compared to commercial data sets is the
amount of metadata that we keep
and we keep that metadata for two
reasons one is for provenance and the
other one is for visualization
first start with provenance you know you
might have been reading about these
scandals in behavioral economics or
social science recently
and this kind of
dog ate my homework excuses that the
data's missing and
you know i'm sorry
but actually we know that in science
you should never have to take anyone's
word for anything you should be able to
go back to that raw data at any time
it's exactly the same thing with our
articles and with our charts you should
never have to take our word for anything
at all you should always be able to get
straight back to you should be about to
get the raw data from our chart in two
clicks and you should be able to get to
the original source of that data as well
in two clicks
and in that way
you know in that way you can come to
know that you can trust our charts
because
the thing the data that we show you is
really transparent about where it came
from
so we keep this at we keep a lot of
metadata around provenance at this level
the other side of metadata is around
visualization
so
think about it this way when was the
last time you just took a data frame and
you plotted that data frame and you were
like yes this is a chart i can show
people this is beautiful
it almost never happens
so you end up like changing access
labels and titles and adding all of this
stuff all of that stuff is actually
information about the data that you're
kind of adding later
so we have a lot of rich metadata about
entities and variables that let us
visualize them much more smoothly
so that we have to do
we do more work when we get the data in
but we do much less work when we have to
visualize that data
just as an example there if i take a
look at life expectancy
here we can actually see that for some
countries life expectancy
goes all the way to all the way back to
1543.
um
things like the title of the chart is
metadata around a variable things like
the units in years
it might be really hard to see but
there's a little source description here
that says
where this data came from and a note
that explains
what life expectancy at birth or what it
sort of means
and one way you can think about it is
that this is a chart that's designed to
be shared like in a tweet out of context
and it should be pretty hard to it
should be pretty hard to misinterpret it
if even if it's shared out of context
other metadata that would keep is of
course
this provenance metadata and in this
case for life expectancy there's quite a
detailed explanation of how this data
comes together and that's because it's
stitched from several data sets
i'll just talk to you a little bit about
our architecture that we use
under the hood
so
we keep an internal data catalog
in my sequel and this is our kind of
source of truth for the world
at the moment we publish over a thousand
datasets from that catalog we republish
them onto github but uh
our goal actually in the coming year is
to end up republishing
more like 90 of that nearly everything
the only things that we wouldn't
republish would be things that are like
embar data sets that are embargoed or
data sets that are
uh that we're unable to share publicly
so
uh our welding data is actually a jam
stack site it's statically rendered the
entire site
so that means that all the interactive
charts you see are actually
rendered in a kind of baking or compile
process
into the live into the site that you see
live but this data set this
data catalog is never used
when you use our live site
there's three channels for data to get
into the data catalog one of them is
around covert because covet is just such
a special situation for the world
um and
we combined the automatic channels of
reliable data that we know about
including johns hopkins university
perhaps the world health organization
along with some of the manual things
that i showed you
pulling official data from tweets
facebook posts video things like that
and we combine all of this in to make a
reference coveted 90 data set
and we published that on github
we have published a data explorer for
that on our site
and um
yeah
you will if you see global data being
compared if you you'll see it in your in
a newspaper perhaps if you read the age
of city morning herald or something
similar you often see references to this
particular data set
i also mentioned that there are these
big institutions like the who
different departments of the united
nations they go to immense effort to try
to collect
painstakingly
information about global progress
so they usually release that a few times
a year
and we do a process of bulk cleaning and
uploading that anytime they do these
releases
and this work is actually the work of
four full-time people in a data team
finally if you are interested in uh
well finally i should say if you're
looking at these big global problems
there are a lot of tiny facets to these
problems
for example
there are um
you know there was new data that came
out on
diet affordability which offered a kind
of different perspective on
global hunger
and this sort of research data set
actually our researchers themselves have
a process an admin that they can
harmonize and bring this data in
so
let me
talk to you a little bit about getting
your hands on this kind of global data
so that you can actually um you know
so that you can dig into these issues
yourself if there's a major global issue
and you want to get your hands on that
data how can you do it
so
like everything else these days search
is still your friend
if you do a search i would say you know
often you're hoping to get data from a
big institution because you know that
their data set is going to be pretty
reliable that they stake their
credibility on it
but there
if you see our world in data in the
search results even if you don't care at
all about our articles there you can
it's still worth coming to us to try and
get that data out and i can explain to
you a little bit why
let's take a look at indoor air
pollution
we can do a google search for indoor air
pollution
uh and it says that oh this is uh dust
dirt and gases in the air and that could
be harmful to you
you might think wait indoor air
pollution is this one of the world's big
global problems
and it could be surprising to you to
discover that you know this kills 3.6
million people every year which is six
times more than all
deaths from war terror attacks and
murders combined
so it's definitely a major cause of
death in the world
and
well let's keep let's look down and see
what we find in these search results we
do see some stuff from the world health
organization and this is probably one
channel that you can go and get data on
this problem
but we also see an article here in our
world in data
now
even if you are
not interested in the article you can
actually come down
and look at the charts each chart is
kind of representing a little small data
set
and
you can download this data
directly if you want to
and here we see that uh outdoor and
indoor pollution air pollution together
is
the fourth highest risk factor for death
in 2017.
and maybe just a reminder for indoor air
pollution
the reason in the air pollution is a
problem it's for people who do not have
good clean energy sources to
feed or to
to cook food and to heat their homes and
that means they have to burn solid fuels
and then you have this like
bad smoky environment in the home
actually it affects 3 billion people
worldwide we can see that especially it
affects sub-saharan africa east asia
and you might not be aware that
nearly 11 percent of our neighbors in
papua new guinea die because of uh
indoor air pollution
so it's actually a reasonably
significant problem
again
uh to get the data on this
it's very simple just to click and get
ssd and this is easy to work with and to
use
suppose you do this search and you don't
either
well for whatever reason you come dry
you
don't find an article from us that lets
you quickly get into a chart and get to
a data set or you get to an
international institution but you end up
chasing your tail in circles because
they don't make the data super easy to
get
so then there's two really good open
data catalogs published on github one of
them is from us at our world and data
our owed datasets catalog the other one
is from gapminder
and gapminder you might especially know
from the late hans rosling that used to
do really excellent ted talks
their mission is to combat devastating
ignorance in the world and they are also
trying to collect and republish global
progress data in a really nice way
both of these data sets both of both us
and gap minder we actually publish in a
format called frictionless data now i
had never come across frictionless data
until recently so this was kind of news
to me and it's interesting when you've
worked in data science for ages but you
haven't had a chance to actually
where you discover a format that people
are using that you've never used before
um i mentioned all sorts of like weird
ways of getting data in like well i
imagine we have to check videos and
tweets and facebook
csvs are definitely not the worst format
on the other hand
when you think about the last time you
used a csv file to do some serious work
first you load it in super easy but then
the first thing you have to do is fix
all the types
and
this is a problem with csv that it's
kind of lossy with respect to types or
the schema if you save your csv file you
load it back into a data frame it's not
going to be the same data frame
so this is one problem csvs don't have a
schema
the other problem with csvs is that they
don't have any room for the type of
metadata i described to you before
like where did this column come from
what does it mean what units are in
things like that
i think of the frictionless format as
being csvs without shame
so you've got csv files they're actually
there's some nice things about them like
you can really easily look at them as a
human
but they lack this kind of machine
really readability and they like this
metadata
so if you basically add a json sidecar
file
datapackage.json that includes these
things you've more or less got this
frictionless format
um
yeah so maybe we can take a look at that
actually so if i go to our
github i'm in our world in data and our
world and data data sets
you can
uh we can click straight into data sets
and we see that uh
well there's over a thousand data sets
here covering all sorts of things work
and
work and leisure access to electricity
electricity literacy tuberculosis
let's take a look at the affordability
of diets
again a nice thing about frictionless
format
is that the csv is still there you can
still just go and look at it
um
and here we are
we're able to see various uh costs of
eating enough calories versus the sort
of diet that your body needs
but you can also
look in the datapackage.json file and
here then you would see a lot of this
metadata where did the data come from
you see a schema
and
including descriptions for different uh
for
different columns and also some of this
display data like the cost of a calorie
sufficient data
diet the short unit here is we if we
don't have space we call it a dollar but
if we've got more space we say this is
actually technically a 2017 us dollar
per day
um
there's a frictionless package in python
that makes it super easy to load in this
data and to
get to both the metadata and the data
frame out and so i recommend that if
you're interested in that format
just some like thoughts to close here um
so
at our world in data we talk about
massive global problems and
the world has really big problems
and on the other hand when people get
lonely you look at the news the news is
also kind of giving you
giving you all of these problems showing
you all these things that are wrong with
the world every single day
and you combine these things and it's
really easy for people to get cynical
and to feel like
there's not much
to feel like there's not much uh worth
in putting their efforts into trying to
make the world better
and surveys of young people show that a
lot of them feel like the world is in
decline
at our world in data we think that the
really big picture that we should be
considering is both the massive global
challenges and the massive global
achievements that we've had
if we talk about some of these
achievements
there they're things like
moving from a world where nearly
everyone lived in extreme poverty 200
years ago
to a world where we've been able to
close and close and close this gap and
we can really foresee that we'll end
extreme poverty
you know
likely in the next 50 years
we live in the world where
more than 40 of children used to die
before the age of five
and we've been able to really really
drastically cut child mortality
especially in the last hundred years
uh so that more and like neil yeah
even this gap that's still left it's
still unacceptable but yet we're doing
so much better now than we were before
more and more people can read get access
to ideas
uh navigate their
local communities better
and this corresponds with a massive
increase in the global level of
education something that's projected to
continue well through the end of the
century
and people are also living in greater
and greater
political freedom than they have in the
past and especially the last hundred
years we see the end to the colonial era
so
you know you can understand someone
writing an article like this this has
been the best year ever except
you know the moment they wrote it the
global pandemic was beginning and i know
you're all in lockdown so you'll be like
lars this is not the best year ever
but
i really believe that if we put our
energies into these massive global
problems you know this pandemic we're
going to see our way through it
on the other side
soon i think we'll be able to say again
that each year will have been the best
year ever
and
yeah that's me thank you
thank you lars
this has been um sovereign talk and uh
people on the back channels were asking
lots of questions we have time for a
couple
so
the first one is the most voted by the
audience how does a world of data relate
to fair data principles
and what would make your work easier in
that regard
um
so i if there is a kind of technical
concept of fair data principles then
unfortunately it's still something for
me to learn but i would say
there's two things that would make our
work a lot easier
one would be a really nice standardized
format for
uh
[Music]
that big institutions and researchers
both publishing that would be super nice
um
maybe on the other hand if i if if i
instead of talking about what my gal
work is if i just talk a little bit
about a fairness angle
um our world and data now gets quite a
lot of traffic about a lot of issues and
when i was a researcher you know when i
collected data i was always terrified
about sharing that data because i was
sure that someone else would scoop me
they would publish something they would
find something amazing in my data that i
didn't
uh our world and data we try and be
super super good about
any time we grab data from researchers
uh we'll firstly talk to the researchers
but we also make sure to try and
direct all credit and we direct people
back to the researchers because that's
definitely what's needed
to get more people to be willing to
share
their data and i think uh
yeah it's just like it's it's uh
the more people who are able and willing
to share what they have i think the
better it is for the community
um people are asking also the
visualizations look great what tools do
the team use
we developed an in-house tool uh which
we call grapher and that's i've actually
been doing some work to try and port
that to a nice
python interface that you could use in a
jupyter notebook
in some ways it felt
you know you could say well there's so
many good data visualization tools it's
crazy to write your own why would the
team do that
i think a big piece of this is the
need to support all of this metadata
like with the right metadata
the charting
of these issues should become really
easy and you should be able to get a
really rich interactive chart in just
like
uh
you know just a few lines of code but um
the existing charting tools they really
require you
to specify everything about a chart so
that's a massive difference in the kind
of open source tools that are out there
today and the tool we're using
i should mention that grapher is open
source but i would say it's not
split out
into a code base in in a nice enough way
for external contributions at the moment
thank you
uh we have many more good questions and
i invite you to go and venue less and
interact directly with the audience if
you have a moment and i also want to
thank you thank you lars jenkin for
being the closing speaker at the pycon
2021 data science and analytics track
thanks so much everyone thanks have you
bye
and for everyone who's still following
in the we thank you also for your
attention and we would like to give you
a small goodbye in 10 minutes from now
at 5 45 melbourne time see you then
you