[Music]
uh I'd like to introduce Ed Shield who
today will be talking about better data
frames big hand for Ed thanks hi
thanks for coming so the pictures here
uh oh there's nothing showing yet uh can
we have there we are beautiful all right
so we have a a panda we have a polar
bear and we have an Ibis these are three
of the data frame libraries I'll be
talking about in this in this uh session
thanks for coming so uh now pandas is
sort of the heavyweight data frame
library in Python it's been around um
for about 10 15 years and uh at the
moment there's a kind of proliferation
of upstart packages which have different
ideas that they've been exploring um and
um I'd like to help you know what these
are and make sense of these so you know
which package to choose for your next
project or if you're a library
maintainer that you know how you can
support different kinds of data frame in
the uh in inputs to your functions for
example so here's the outline of this
talk so i' like
to just go over the history of data
frames briefly to help you understand
some design limitations that pandas
has and then describe some of the
Alternatives and efforts at
interoperability and then a few minutes
if there's time um
with some more philosophical thoughts on
the nature of the kind of creative
process this creative destruction
process in open source and some
recommendations about which packages to
use and which techniques to use for
different tasks and if we have time at
the end um I'd welcome questions as
well so my name is Ed scoffield I have a
background mathematics and computer
science and I did a PhD in language
models before
most people knew who what they were in
fact um
uh I while procrastinating writing up my
PhD I I got involved in uh development
with scipi and at the time actually the
tools were quite kind of um primitive
and broken so numpy and scipi uh needed
to be developed to be useful for um for
you know machine learning research at
the time and uh
um yeah anyway Matt plot lib didn't
exist and so that eventually came around
around 2005 or so um so I founded python
Charmers after that it's a training
business and we're based in Melbourne
and Singapore and we've given about 700
training courses over the time about 500
in Australia um most of them are kind of
live with you know Hands-On uh
coding um so um I was involved in the
python future project when I was
concerned that the Python 3 migration
was happening too slowly so it looked
for a time like Python 3 might be dead
on arrival and it might become a a pearl
sick situation and thankfully the
community rallied to to do a big push to
make this migration possible but it it
looked it looked quite kind of close for
a while so I got involved in the python
future project to S of assist with that
effort uh and um
uh for fun at the moment I'm doing a set
of courses through Stanford in in AI
particularly in reinforcement learning
which I'm finding really really fun um
something I haven't really explored but
it's a it's a brilliant field so anyway
that's um what I'm doing at the moment
for fun so so first of all what is a
data frame most of you probably use them
uh can I a show of hands please actually
who uses data frames most of you okay um
so it's not a vector it's not a matrix
it's not a multi-dimensional array it's
it's a bit like a matrix in a sense but
you've got heterogeneously typed
columns uh
[Music]
so so how is that different from just a
a table I'll come back to that in a
moment but um this is another view of a
data frame you've got a column index or
of column names and you've got multiple
series of data and this is the
terminology used by both pandas and
polers notice there's no index in
general that's a a pandas concept but
most other data frame libraries don't
have the concept of a row
index and so
how are these different
from from tables in SQL databases or
tables in
spreadsheets um well suppose terminology
wise um tables in relational databases
have know row relationships
um and um and
traditionally um relational databases
store data rowwise so so operations
analytic operations are not optimized um
as they are in the new breed of columna
uh data databases olap databases like
Duck DB and data frame libraries okay so
data frame libraries store data
columnwise and this allows F vectorized
operations traditionally also there's
been this s of terminology that a data
frame is something that lives in memory
it's not always true now but and the
lines are blurring but um traditionally
you would do a a kind of import of data
suck in lots of data until until you've
got it all in Ram okay which which of
course naturally puts a a limit on the
size of data sets you can work with all
right but that's changing these
days so data frames
really
support um kind of janitorial kinds of
uh um task which are necessary before
you put data into a machine learning
model for example or do a regression or
something like that uh so um data frames
are kind of have yeah features for for
cleaning up data uh for aggregating data
with time series and things like that
okay um now uh um here's a post from Big
Data Borat uh 11 years ago in data
science 80% of time spent prepare data
20% of time spent complain about need
for prepare data uh so he was there was
he yeah understood understood that very
early uh all right so
now here's um brief sort of set of words
about the design of pandas and where
it's come from so in the beginning we
had this but then a few billion years
passed and we had uh the 1970s and um so
in the' 70s the S language came out of
bell labs and SQL was sort of developed
and started to be standardized and the
first spreadsheet package vial um became
widespread um in the 80s we had data.
frame as an object in in the S language
python came out in 1983 R came out um
later that decade
um and uh and in the 90s there were
these packages numeric and nay which
made use of Python's great C API to
extend python to make it useful for
scientific Computing in particular and
Engineering Computing um some of the
early applications of it the mySQL
database also came out in the 1990s just
for historical context as part of the
lamp stack postgrad was in 2000
it was until 2005 that numpy came out in
its first release in its current form
which is a kind of amalgam of those two
earlier array packages numeric and num
and scipi came out at around the same
time Matt plot lib a year or so later um
when I started working on scipi it
contained three separate plotting
libraries and all of them were broken uh
so I was very happy to have Matt plot
lib to create some plots in my my thesis
um uh Panda's I think um development
started around 2008 which is pretty
early so this is sort of interesting
this historical note so it
predates uh the kind of Innovations in r
on other kinds of data frames um it
predates spark um and uh Big Data tools
like like big query and so on so the
first of the kind of
um now tidy ver packages and R came out
in 2011 uh and uh and deer and tidy R
came came later and the Tidy verse was
sort of formalized more in 2016 and
that's also when when development
started on the Apache Arrow project now
this was really an initiative of Wes
mckin who's the is the uh original
author of pandas and Hadley Wicken from
the r community
so so Wes mckin wrote a blog post in
about
201 13 or so I think um perhaps a little
bit later
2015 uh called 10 Things I Hate About
pandas and he gave a talk at the New
York City python Meetup or something
like that um about that and so he was
aware of these design limitations from
the beginning but but by then Panda's
adoption was really growing and I think
we've got pandas to thank for the fact
that python is where it is today as as a
tool for manipulating data so R was was
very much loved by users um and still is
uh some some there are users out there
who say know you'll take R out of my
cold dead hands you got to claw it out
um uh and the Tidy roast is a is a
wonderful kind of ecosystem of packages
um but uh but it's quite possible that
without pandas python wouldn't have had
a big role to play in in um in many
different kinds of sub fields of of data
analytics and data science or at least
not as big a role as it does today uh so
pandas builds upon numpy as a foundation
and that also gives it a couple of
limitations uh
so um
numpy I want to
um sorry an old version of this slide
but numpy um provides
a um
a multi-dimensional array object but no
support for masked well it supports a
masked array it doesn't support sort of
missing data natively for for a range of
data types the master array feature
never really took off and and it
provides record arrays and structured
arrays those are also aren't really a
substitute for for data frames in in
Panda so pandas built upon
numpy uh and provided that kind of
tabular analytics function ity okay so
there's a data frame in pandas it's like
the diagram you saw before but there's
this row index down down the side um now
there are people who use the row index
to its advantage and you can get better
performance by making use of it um but a
lot of people just don't know what to do
with and just find it an annoyance and
try to drop it or reset it or something
like that but um it's an interesting
design decision because
it um
it it does complicate learning pandas
um um uh but yeah I think a lot of the
the sort of new breed of data frame
libraries don't don't use a row
index many SQL databases also
traditionally haven't returned data in a
consistent order and uh unless you
explicitly call order by as a
clause all right so I'd like to say
again sort of three cheers for pandas
it's it's done us well uh it's it's used
by millions and millions of people and
what it does it does well so it does in
memory uh analytics it's expressive it's
got an enormous API which is uh it means
you can do a lot with it but it also you
know makes it harder to maintain and
things like that it its focus isn't
there and it's it's got a lot of open
bugs partly because there are few
maintainers doing putting in a Hulan
effort to kind of uh support you know
millions of users who are all
complaining that my sort of such and
such Doesn't work um so it's it's um
played a very important role in the
python data ecosystem uh now Panda 2 was
released last year uh and there are a
couple
of architectural
improvements uh it supports optional
backing of all of columns in a data
frame with the pi Arrow sort of a
package with the arrow standard that
this supports and this
provides the ability to um to have
missing data on all data types uh it
also provides better string handling
than numpy is traditionally provided but
numpy um uh
2.0 does change that um but numpy 2.0
doesn't have a good answer about missing
data still uh so there's this limitation
there there are some um other
limitations as well about s of memory
usage in particular that limit the scale
you can of data you can work with with
pandas that there's assumption that you
pull in all your data into memory for
example but there are some new copy on
write features which can assist a little
bit but the pandas 3.0 release is still
s of in an early design phase and the
design is not decided yet uh there's
been some kind of back and forth about
whether Pi arot types should be
supported by default and initially that
was going to be the case and then
someone said hey wait this brings in
another 500 megabytes of of dependency
is what's going on here uh it's a very
heavyweight um dependency at turns out
so so
uh it's unclear what the future of the
project is and and I'm thinking it um it
will probably not go away anytime soon
but it's going to become continually
less relevant in relation to the other
data frame libraries uh so um so I think
Marie condo says a sort of you know say
bless something and then they say if you
don't no longer need to bless it and
then throw it in the bin so so uh so
this is perhaps what we should be doing
here and I I don't think um this would
offend W mckin who's very aware of its
uh design limitations and has been
actively kind of contributing the last
sort of well yeah the last 10 years of
his his career towards improving
improving this current
situation uh so so here are some
Alternatives now this is a set of GitHub
Stars uh look at the Blue Line That's
polar the green line is Duck DB the red
line is pandas okay so King of the
castle right now um but it's um it's
acceleration it sort of yeah it seems to
be um smaller it's a second moment and
we've got um yeah various other kind of
little tools there um but in particular
the blue and green lines are
interesting so one thing that's
not really listed which is super
important is this so this was as I
mentioned the initiative between um W
mckin and Hadley Wickham to to define a
cross language standard for how how data
can be accessed in memory which supports
you know nullability of of all data
types um and and efficient zeroc copy
you know reads and
writes so this is critical and this is
better than numpy in a way numpy is
python specific arrow is cross language
so it's it's really very powerful and
duct
DB is based on Arrow polers is based on
Arrow uh the a lot of the current breed
of new data frame libraries are based on
this standard okay so this this is sort
of the invisible stitching behind the
scenes that makes this all work so if it
looks like there's a complicated
ecosystem of new ways to doing things
with data frames in a way that most of
them are linked together by this okay so
it's it's not as incoherent as it as it
seems okay so Arrow solves these kinds
of problems that pan is as tightly
coupled with numpy with its limitations
pandas also oh yeah um Arrow also
supports um other languages there's this
Nano Arrow package for example and a
variety of other C libraries you can use
and linking with various languages so
Arrow does say something about what you
can do to process larger than memory
data sets Pand has never had a good
answer to this story for example um uh
you can't uh use yeah sort of memory
yeah like like dis backed sort of um
arrays and memory and um what am I
talking about okay mental blank there
but um yeah it won't support um
sort of this transparent mapping of
memory maap files is what I'm trying to
say yeah um okay so pandas provide uh
creates lots of intermediate objects in
memory which is quite wasteful of memory
so this creates an intermediate series
and then filters on that which creates
an intermediate data frame and um and so
Arrow can help with that um and I
mentioned memory mapping okay so polers
is a very interesting Library uh as I
mentioned it's based on
Arrow it does have a limitation
fundamental kind of
limitation um that it is designed at the
moment anyway for local inprocess data
processing so it it's not designed to
run across a
cluster it's not an alternative to to
Apache spark or pie spark um but duck DB
is also targeting this Niche they're
very very closely related projects um so
it's fast neck and neck with duck DB in
terms of uh benchmarks um it's written
in Rust and has kind of documentation
and interfaces in Python and rust and
the API is is quite nice uh it's not as
Wy or clunky as the Panda's API it's a
bit more of a Bose um but people like it
just like people like the the D plier
API uh from r
so duct DB you notice some similarities
it's also based on Arrow it's also for
local in in process data
processing it does however have good
support for spilling data to to dis to
some degree okay you probably don't want
to go about more than 10 times Ram um
certainly not more than 100 times Ram
but it it duct DB and polers will let
you handle um data sets potentially up
to 100 gabes or a terabyte if you've got
lots and lots of ram um but you can't
really push them beyond that very well
um duct DB has interfaces in most
languages which is nice
so some of you would have gone to the
duck DB talk um it's uh you can think of
it as like SQL light but for analytical
queries but the python API is is for
Bose at the moment and it hasn't really
had the same attention as the underlying
du DB Library so I wouldn't recommend
and the python API for duct DB at this
stage um there was this original plan to
build out a data frame API like pandas
or poas or something with the duck DB
backing which is not a bad plan but now
others are doing it better so they're
focusing on where they contribute which
is the basic Library so here's Ibis it
was started by W mckin around the same
time as the Pache Arab
project um and what Ibis lets you do is
to scale
from local to cloud with the same API
which is a really
interesting kind of um s of value
proposition I'd say so originally um
where implemented apparently as an
interface for Impala um but now it
supports about 20 different backends uh
including including duck DB and polers
it is what pandas but that's being
removed now because pandas is so
different from all of the other
engines so it was like using 40 or 50%
of the entire code base and it was not
worth it since it was slower anyway it's
still but but this will quite happily
read data from Panda data frames in
memory that's not a problem but it just
no longer pushes data out to pandas as
an execution
engine
um so I think of this
uh as being like SQL Alchemy for
analytical queries I haven't seen that
kind of epithet used but that that would
be a good tagline for the project SQL
Alchemy is brilliant of course we don't
need to write you know custom code for
each database and inte D portability and
so on and and that's what this aims to
do to this project okay and it's
actively developed and growing doesn't
have the same traction or or kind of uh
hoo-ha as as polers but it probably
deserves more than it's getting so um
here's a quote from the the site uh the
blog blog post from the IIs project if
you're considering polers for new code
give Ibis a try with a duck DB backend
you'll get better performance than
polers on some workloads with a broader
cross backend API that helps you scale
from development to
production okay so you can use the same
API for processing data on your local
machine as for processing data through
big query or or um click house or
something like that so there's a uh the
current list of backends from the Ibis
website that Ibis
supports so it's really interesting
you'll notice that there's one data
frame library polers and the others are
all SQL
engines so it does a good job of
converting SQL uh these days it's making
use of a package called SQL glot to do
that um which is a something you yeah
it's probably worth watching um does
yeah so you can just translate between
SQL dialects there now there's this
deferred execution mod model that IIs
offers you can kind of connect to some
remote database whatever the store is
you perform some SQL query you get your
results back but it feels like a local a
local database and that's especially
important when the connection between
your local machine and the cloud is is
weak uh it's like some some slow
connection um okay so you got all this
data up there and You' got a fast
computer but this this tiny little wire
connecting them so I saw this was an
important kind of news announcement
recently um so until we have a new
carrier pigeon based Network that's
competitive it's good idea to do the
processing on the remote instance
instead of uh instead of pulling all the
data over the wire just
a couple of others this is a GPU Library
it's limited by vram it's Nvidia cards
only so it won't run on your Mac but it
now has nice Integrations with polers
and
pandas
um modin and dar data frames are both
eort efforts to scale out Panda's data
frames to a cluster so they try to mimic
the Panda's API which is good for
compatibility but it has its warts and
so on as well um narwals is worth
using if you want
cross uh data frame compatibility if
you're writing a library that needs to
accept different data frames okay you
can use it for example like this okay so
this would be a a function that accepts
a variety of different data frames um
and does what's necessary preferably
without memory copies to support those
um now there just note on benchmarks um
polers and du are often neck and neck in
performance but they're both a kind of
big step up from these other kinds of
libraries and and pandas and deer are
ort of in the dust I'm not sure what
arrows doing there uh I'm not sure
because these are based on Arrow so
that's a bit surprising I'm not quite
sure what that's about um so arrow is a
very very well design
specification um here's a quote from
will IED from an interesting blog post
about the The Arrow C data interface so
um so one you'll want to look at this
for napari and anyone else who wants to
support um
uh um zeroc copy memory access from data
frames the the mechanism is this Arrow C
stream the C data interface it's
becoming the standard mechanism to
support Arab based data um with a lot of
kind of benefits there so if widespread
this would support zero copy
operations um without requiring Pi Arrow
which as I mentioned is a big
dependency so
um so this is the the goal of data frame
agnosticism currently there's a an
effort underway toward removing pandas
from a whole bunch of different projects
as a dependency which is quite
interesting um but still accepting the
data frames of course just no longer
requiring it there so I will just skip
to the end as I'm out of time um so
um uh if you're interested please
approach me sorry sorry there and please
um got to go that to that there and that
there so be happy to kind of give you a
live demo I have prepared a live demo to
um yeah of alternative data frame
libraries how you can use them in
practice and and what what the um
experience is there so please please
email me and be happy to uh chat to you
more um yeah and Le come and find me
afterwards I'll be hang out probably
outside yeah all right so uh thank you
very
[Applause]
much thank you very much Ed for that
talk I would like to present you with
the pon thank you uh I think we've got
time for one question okay uh anyone
have a
question oh yeah we've got one at the
back
thanks that was brilliant um I come from
Earth Science community and there we use
xray n CDF hdf5 how do you see the
relationship between that and what you
presented
yeah
these say data frame libraries are are
not really focusing on on that use case
um there's a little bit of overlap but I
don't yet see any efforts underway to
for example generalize xarray to to use
polar data frames under the hood or
something like that
um yeah so I I
don't I don't see any overlap at the
moment yeah uh there will naturally be
improvements available in performance
for example with working with large data
sets by virtue of the Apache Arrow
format if if these libraries start to
use that so that might be the first kind
of point of entry um for for those
libraries to to support uh for example
missing data on at a deeper level than
numpy does
yeah well thank you if could you please
join me in thanking Ed once more
[Applause]