foreign
welcome everyone to the all things data
track I'm super stoked for the lineup we
have today my name is Ned I'm one of the
hosts of The Specialist at the
specialist track we also have Ray and
Pedro who you'll see later on uh
throughout the day they're down the
front
um the only minor housekeeping thing I'm
gonna mention before we jump straight in
is that we have a Discord Channel look
for the data track uh Discord Channel
pop questions in there for when folks
are going to be taking questions if we
have time and just say hello uh chat
there's a big shout out to everyone in
the online audience um it's great that
you're tuning in as well so with that
said let's jump into our proceedings
and uh first up we have Christine seliga
and long dang uh Dan they're going to be
talking about uh unpacking the
geospatial engineering toolbox an
overview of data science techniques for
spatial data just some quick intros
Christine is the technical lead of wsp
digital's data science and analytics
team a scientist by training she has
worked across a wide variety of
different fields from computer science
and software engineering to biomedical
research fintech and now civil
engineering she is continuously looking
for new and interesting challenges to
expand her Knowledge and Skills long is
a junior data scientist WSB who likes
all things math deep learning and
computer science long is it most at home
crunching through some algorithm coding
problems with too much scratch paper and
a pen when long is not at work you can
find Long watching anime playing gacha
games going on random walks if the
weather is nice meeting friends and new
people or trying to get a neural network
loss to go down so please Round of
Applause for our first talk
yep
um hi everyone
um thank you for spending some of your
time at the first half for this track
um so I think when you think about it
science you probably think about
statistical techniques like the things
that you can work on for tables like the
things you find in Excel or you may be
thinking about unstructured data like
images and texts the things that you
work with when using deep learning
models like strategypt or Sable
diffusion they're all cool and stuff but
at my workplace wsv we have a lot of
contact with civil engineering
disciplines so one of the data type that
we work a lot with is actually
geospatial data these are everywhere
they're very useful and in this
presentation we would like to give you a
brief overview of what geospatial data
is what you can use it for we'll give
you some overviews of patent libraries
and techniques that we use and we'll
give you some examples of the actual use
cases or projects that we deliver to
clients and give them insights from
their geospatial data
we'd like to begin by acknowledging that
we're meeting on the traditional country
of the Ghana people of the Adelaide
pains and pay respects to the elders
past presented emerging my name is long
and second Sunday with me is Christine
and we are data scientists at the team
so the first question is probably the
most important one is what is your
spatial data and why is it important to
us
so the long and short story is that
geospatial data represents all the
physical things that surround us so it
can represent the buildings this
building this room so the chairs the
roads around us and can represent the
electrical lines that bring power into
our homes the water pipes that bring
water and waste away from our homes
and Google Maps as you can see to the
right it's probably one of the best
example of how we use geospatial data in
our everyday life so we use all this
geospatial information about the layout
of the buildings of the cities and the
public transport networks to help me get
to this convention center today
we also use it to find places to go to
on weekends for example
and this is what geospatial data is
important we want to use all of this
important information to make better
decisions not only how to move from
place to place but also how to build
better cities better environments and
better communities so for example we may
want to use your spatial data to find
the best place to build new social
housing new parks for just
underprivileged communities we may want
to understand how our infrastructure
work how they are all connected where
faults may happen and if they do happen
where can we best solve them
before we can delve into like solving
all these important problems we first
need to understand how we actually
represent geospatial data for computers
to understand
there are many ways that you can
represent your spatial data depending on
whether you want to include verticality
or not but for this presentation we'll
mostly focus on the 2D case so you know
like a map where things live in two
dimensions in two Dimensions everything
left on the coordinate system so they
can be represented as Vector data and
these are the intuitive things that you
find your everyday life like shapes
lines points you can abstract over this
representation using grads where the
shape of things no longer really
mattered only the connectivity of the
objects you care about matter
2D data also include things like images
and raster where things are recorded in
pixels so for each pixel it represents
some actual real-life geographic area
and for that area you measure some
measurements maybe the amount of
rainfall the amount of solar potential
and so on
so this is probably the simplest Vector
data that you can think of it's just a
triangle and a swear but you may be able
to do more interesting to them when you
start moving them around rotating them
finding the intersection between them
which is highlighted in purple here
you're going to merge them you'll use
one shape to match the other shapes
solve these are common operations that
you may think of when we're dealing with
Vector data
and you can think of this representation
as not unique to geospatial data alone
because they are shapes they are
practically everywhere they reduce a lot
in computer Graphics as well so there
are a lot of literature and algorithms
that have been developed over the years
that we can leverage file geospatial
data analysis
speaking of tools and algorithms at my
team we use
um very common Suite tools to extract
insights from your spatial data the
favorites are qgis jio pandas Chef Lee
and network X which we will go over
shortly we also use raster IO and
lastify which are libraries for raster
and lighter data but we will not talk
about in this presentation because we
don't have enough time
so the first stop is qgis
this where when we receive data from a
client or we download data from the
internet we will usually just plug them
into this tool and it will plot
everything onto the map and then we can
scroll around just learn Google Maps we
can also click on the actual shapes
itself
to fight attributes associated with that
[ __ ] so here we are having a lot of
polygons and orange shapes that are the
footprints of houses and we also have
dots that represent
um
points of interest
and qgis also have a lot of tools
associated with it it's a full power
Analytics tool so it can also do all
that geometric operations that I talk
about it can merge things find
geometries of Interest as an open source
tool qgis also has a lot of third-party
plugins that you can use to improve your
analytics workflow and you can actually
write Python scripts that can run in
qgis as well
if we have determined that qgis is not
sufficient for our work and we really
want to dig deep into things and start
automating things with code the second
stop is usually geopanders it's kind of
like pandas which is a library for
manipulating tabular data except we have
an additional geometry column
and it can read and work with the most
common geospatial data formats like SHP
LG or Json or GDP so in this code
snippet on the right I'm importing your
pandas and using the read5 API to read
all the geometries for suburbs in South
Australia and this read file API is very
similar to the pandas read CSV API and
you can pass arguments to it to specify
which format you want to read it
and the next line of code to CIS is
where the geometry column really shines
here I'm using the 2cis API to perform
an operation that transforms one
coordinate system to another so when you
think of coordinate system you probably
think of like latitude and longitude
that can locate anywhere on the globe
but it's not the only coordinate system
and in fact it's not even useful for
some use cases where we want to measure
precise distances in meters you can't
really have a distance in degrees it
doesn't really make a lot of sense so
what we do is we'll convert that corner
system into one that is locally precise
for South Australia where we can measure
distances in meters and so that's what
we're doing with this API call
and then we can perform common pandas
operations so things that can work on
tables look similar here here I'm just
indexing some columns and showing it in
my tributor notebook
if we want to go a bit further and
directly manipulate the geometry itself
we will turn to Shipley Shipley actually
underlies of the geopanders geometries
objects so when I print the typed of the
first geometry in the geometry column it
actually comes out as shapley the
geometry the polygon
and I can also use a lot of the common
geometry operations that Chevrolet
provides and here I'm trying to use the
unary union operation and what this does
is it just merges all the geometries
together
and because submerging the geometries of
suburbs in South Australia I'd expect to
see what the state of South Australia to
emerge and it kind of does except
there's this big hole to the left which
I don't really know why maybe there's
just no one living there
but um as an old disciplines of data
science data quality is probably one of
the biggest problem that we have and
geospatial data is now stranger to that
problem as well
to really show how powerful these
libraries are though we need to really
dig into the use cases and the projects
that we work with our clients to bring
value to them
so the first project I would like to
share with you is search pipe
maintenance
so the problem that we're trying to
solve for our clients here which is a
water utility company in New South Wales
is to optimize the sewage means
maintenance program to achieve the best
performance for their budget
so the idea is sewage pipe
sdh they get more prone to breakage or
leakages and it releases a lot of
harmful things environment and also
cause a ton to fix up so ideally we
would like to replace or maintain the
pipes before it gets too old
um obviously you cannot really replace
everything everywhere all at once so you
have to make decisions on which pipe to
replace now within your budget
so for the scope of this work we want to
automate this automation optimization
program and we also want to add some new
additional features we want to tell the
client how accessible a pipe is
to maintenance equipment and this kind
of breaks down into two sub-measures one
is how much of a pipe is built over Say
by property so you can see the pipe
highlighted in red is kind of built over
in certain places
the second dimension is how accessible a
pipe is from the road behind like all
these properties so let's say I want to
access the red pipe
from the road then I may need to go over
like the building in Gray and the
building May does not have a large
enough right way for my equipment to
drive through
so for this problem we have data that
are the pipes and lines we also have the
outlines of all the buildings and these
orange polygons
and these are actually generated from
satellite images using the Deep learning
model so that's an interesting
application there
and finally we have some outlines of the
lots for each property and this data is
publicly available from the new software
government
so to stop the first measure which is
how much of a pipe is spills over the
algorithm is relatively simple
um intuitively you can think of well if
I want to find how much of a pipe is
built over I just find the exact section
of pipe that is underneath any other
geometry
and this is a geometry intersection
problem
and then Shipley it can do that with a
single API call the intersection API
so energy geometry have an intersection
method that can be called on knowledge
geometry and then Shipley will handle
the specific algorithms to intersect a
line with the line or a live with the
polygon
Etc
and it's all well and good
however the problem of determining how
much of the pipe is acceptable
accessible for an equipment behind
properties a bit harder there's no
really single unique operation that you
can just do and get the answer
so our first naive approach is to try to
measure the gap between the building and
the large fence line so the idea is well
if the pipe is in front of the property
then surely is accessible so that's the
trivial case if it's behind the property
then the issue is is there like a big
enough gap between the building and a
lot of fence line for the equipment to
drive straight in like a driveway that's
not blocked by your garage
so visually we're trying to measure the
distance of that red line between the
building and the lot
and in shortly distance is very easily
performed consuming you are in the
projected Corner system that I mentioned
earlier you can use a distant API
between any two geometries again and
Shipley will handle all the specifics to
measure the shortest distance between
any two point on those two geometries
this approach however kind of just
breaks down when you add more complexity
to the problem so here I'm trying to add
some extra like
sheds and garage and maybe a poles so
there's extra polygons now in the lot
and so this minimum distance between the
building and a lot of fence light
doesn't really tell the whole story
because maybe there's sufficient
distance but a shared filter data lines
blocking as you can see or maybe
so now you have to include the distance
between the building and the shed as
well but if that distance is too small
maybe you also need to consider the
first distance that we measured because
there may still be a path straight
through
so I guess you can come up with some
algorithms to iterate over all these
combinations of distances but it's not
very naive algorithm anymore and it's
starting to be very complex and error
prone fortunately there is a better
approach which takes which uses
buffering and buffering is just
expanding or shrinking a geometry by a
certain set distance
so in this illustration we are buffering
expanding the original building
geometries by one meter so you can see
the original geometries in Orange with
the X cross on it
and a buffered outlines in a more
transparent orange color
I'm also shrinking the original large
geometry by one meter and the remaining
space is color in green
so the inside of this approach is that
if we make some simplifying assumption
on the shape of the equipment
that we care about and we're just gonna
see there is a sphere of some radius and
we can also tweak this radius to
actually account for the actual shape of
the equipment
than the center of the sphere must lie
in the green area and you can kind of
see this for yourself if you try to like
imagine placing the center of the sphere
anywhere that is not in the green area
then when you draw the sphere it must
intersect the large fence line on the
original building geometry somewhere
because the radius would exceed the
buffer distance
so if we now
check if the green area contains both
the pipe and access to the road then
surely some equipment that is a sphere
would be able to go from the road to the
pipe we cannot do this in shiftly using
the difference and buffer API which does
exactly what they say it does
we compute the difference between the
negatively buffer a lot so that's the
strength lot and the positively buffer
buildings envelope to give the green
area
we then iterate through each polygons
within that green area just in case that
the buildings went above for drops the
area in half and for each of that
polygon you can check if it intersects
the road and the pipe geometries if it
does then the pipe is accessible
the outcomes of all of this algorithm is
some condensed summarization
measurements for H Pi let's say how much
of it's built over and how much of it is
accessible
and all of these measurements can then
be fed to Downstream applications for
example like in the dashboard or in some
reporting program so that the client can
make decisions or it can even be fed
into another machine learning program to
actually perform that optimization
program in the first place
sometimes though we don't really want to
have this summarization measurements
from a geospatial data we instead want
some geospatial data as outputs from our
initial geospatial data because we may
want to for example visualize them on
the map
and the second project I want to share
with you is one where this is the case
in this project heavy vehicle load
access
sorry
the the problem is we want to evaluate
Road networks in New South Wales against
electric trucks Dimensions so the idea
is the state of New South Wales wants to
adopt electric trucks but these are
really large compared to original trucks
they are larger and longer to account
for their batteries and you have ever
driven behind trucks you know that you
don't want to be
in size turning curve because as it
turns it create this really large web
path which is this total area that it
Trace out as it turns and you don't want
to be inside it because then it's going
to hit you and that's not going to be a
fun experience
so the client wants to understand if the
road networks in New South Wales is
sufficient for all these new vehicles
so for that purpose we
yep sorry
I think my laptop is not liking me today
yep
yep so we have developed a simple
algorithm that can simulate the swept
path given the path that the Mover of
the vehicle takes
but this is actually the start of a new
problem because this simulation
algorithm
needs the path that the Mover takes but
the data that we were given actually do
not have any information on what actual
pass it can take to turn
so the information we're given is a
single line for each Road geometry that
starts and end where the road starts and
end so in this intersection we have many
separate roads but we have no
information that tells the algorithm
that it can turn
that you can take Row one two and three
to turn right from the top so none of
that information is available to the
algorithm itself all it says is this
separate collections of Roads
so we need a way to given this
collection of Roads it numerates the
possible hats that the truck can take
and then output some actual Road
geometry that the truck can the
simulation algorithm that can then take
yeah sorry for that all right
um so the way that we solved It Is by
using
uh Network X Library which constructs
graph
okay so let's see if I can get this to
work now
so what we did is we take each
individual Road
and turn it
just go to the last yeah and don't click
we take each individual Road and compute
and treat it as an abstract Edge so the
start and end of the road will be the
starting n vertex in the graph and we
ignore all the actual coordinates that
link those two vertex together
then if we Define some start and
entrance vertices to the graph to the
intersection so maybe you can use if
there's only one Edge to the node then
that is an entrance on exit vertex then
we can use a graph algorithm to kind of
find the shortest path between those two
vertex and that will give us a list of
edges in this graph
and then we can extract the coordinates
of the road that is associated with the
edge and then concatenate them and that
will form the final Road the truck can
take
so
yep
when in networks X this is done using
the code in the bottom right so we first
initialize this graph component this
graph class then we iterate through each
geometries
and then we can index into the core in
this array and find the first and final
coordinate entry let us start an N
vertex Network X actually allows you to
input any hashable immutable python data
structure to be the vertex so you can
just insert the coordinates pair right
into it
and we can also attach the coordinates
of the original live stream geometry as
well and just make it easy makes it
easier to just extract the coordinates
and then concatenate all the
the the paths together
we can then smooth out this path using
some additional numpy geometries numpy
vectors and the math to create this red
geometry that informs the simulation
algorithm that the truck can now take
this path to turn right and we can do
the same for every other pairs or
starting inverses vertex
and that helps create all this paths
that navigate this intersection
now if I click next it will jumps all
the way again so
I will just present without any visual
aid before passing it onto Christine so
the output of this algorithm would be
this swap pad geometry that we have
simulated for the truck and then we can
kind of look into the widths along the
square path
and then we look at the width of the
road and then we say oh at this location
the width of this web hat is too large
and so there will be a problem because
this drug will now go into the opposite
lane and cause problems for other
vehicles
oh it may intersect some roadside
geometric assets like maybe fence or
traffic lights I don't know
the problem is we don't really have very
good labels on how Wire Road is at any
point it would be good if we have the
actual geometries for the road like a
polygon that covers the area of the road
but that is actually quite hard to get
so as in all other
data science problems QT label is also a
problem in our case where we don't have
a truthy geometry labels that we can
compare this webpad simulation with
so with that said I will now pass it on
to Christine who will take it over from
here so yeah and it's to move to the
next slide
yes so with this last use case
um we are pushing the libraries and
tools that long was talking about a
little bit to the site they're not the
center stage of it but they are still
very important to produce inputs and
outputs for our analysis our team
frequently supports traffic modeling
simulation tasks with data analytics and
Automation and occasionally the tools
and Frameworks that are in use by those
teams are necessarily suitable for for
the problem at hand and in this case one
of the issues they encountered was
modeling reversal of vehicles
so what was the problem and we were
asked to look at a Precinct in the
mountainous region that gets frequented
particularly on weekends by a lot of
tourists and these Precinct is to access
roads
um that are very narrow and in a lot of
places
um they're just single files so you can
imagine if you're driving down the road
with a lot of traffic you're
encountering other vehicles going the
other direction quite frequently which
causes you to either try to be brave and
scramble past them or what is probably a
little bit safer is to reverse and find
a spot where you can safely let them
pass
The Operators of this road were
particularly worried about like the
latter people reversing up and down the
road and creating too much traffic on it
so they asked us to provide some
foundations to intuitive traffic
operations that they could use to
alleviate that problem for example using
shuttle buses to get the vast majority
of people down the road with different
schedules and different sizes or to
implement boom gate Solutions on either
end of the road
um so how did we approach this problem
we needed something that could iterate
over a lot of different situations
fairly quickly and
um we decided to break the road up into
cells and each of these cells would have
properties such as whether they are
single or double file and for example if
the speed associated with traveling on
this particular location
and it will have a state which
represents a vehicle being on the cell
whether this vehicle is traveling east
or west or whether it is currently
reversing or in a holding State waiting
for a vehicle to pass
um for each time step in the simulation
the cells would update their state in a
random order based on the surrounding
cells and we introduced some more
variability for example the arrival
times of cars at either end of that road
based on distributions we got from
traffic counts that were actually
measured up front
what allows us what this allows us to do
is very quickly test different operating
scenarios introduce the boom Gates and
buses and other things and simulate
thousands of typical days how traffic
could evolve on this road
how did we implement this at the core of
this is a custom simulation that we
wrote in Python and one of the key
inputs to this is a shapefile that
contains the geometry of the road and
you can imagine that you can easily load
it and then use shapely that Don was
talking about before to break this road
down into evenly sized segments
in addition we put the vehicle counts in
and the config file that then relates
the change along that road and therefore
it's relation to the cells with regards
to what the speed limits are on or the
average speeds on those sections of Road
and which ones are the single file
locations
now once you've done that your
simulation can essentially forget about
the spatial context of this however it's
really valuable to keep this information
in the back of your head because what we
did is we implemented the visualization
in pi game for this which is basically
translating the coordinates of the cells
into screen coordinates and we could use
this to take the client on our journey
of developing developing the simulation
and show them along the way our vehicles
are actually moving along the Bendy
Mountain Road
then the simulation has a batch mode
where you can run thousands of
simulation and it's just chucking out
files with the parameters that we are
interested in
so the last bit to this analysis this is
missing is what do we do with our
startup once simulation is produced it
and in good old data science style we
developed a python notebook that reads
all of this in for single scenario and
then creates aggregated outputs and
again the spatial representation of
these cells becomes really quite
important because it allows us to
produce a spatial representation of our
results that then can be loaded for
example in qgis and put on maps to show
people the actual content of what's
Happening
and these are some example outputs I
need to say that the y-axis went missing
from these plots but it's a percentage
of vehicles and it for example shows the
travel time overall on the road the
intergate travel time the number of
interactions that vehicles have and on
the left and the top right in particular
on the left you can see the hot spots of
these reversal interactions that we were
mostly interested in and on the right
you can see General vehicle interactions
which also includes Vehicles holding or
queuing up behind others
now this approach really worked quite
well and after this we conducted a
second study
um with a very different topic but the
same methodology which basically looked
at a mine site in wa and the road that
is required to access it and this is a
dirt road and the biggest problem there
is dust development so we basically use
this methodology to simulate different
operating scenarios water trucks Convoy
scenarios to get workers in and out and
heavy Vehicles traveling up and down
this road in machinery
um yeah so a very useful little
methodology and as you can see the
special context is really important and
useful to show your results
um that's it from us if there's still
time I think happy to take questions
otherwise thanks for listening
thank you sir
thank you so much for that talk um we
are running a bit late and so um you did
a wonderful job with a few minutes extra
but we might give that back to folks too
because we're going to try and stick to
the 11 o'clock schedule so uh we'll jump
into saying thank you so much Christine
and and long
um and we have some some gifts for you
foreign
thank you
um