foreign
welcome back everybody wonderful to see
you all here for the last session of our
connected Universe I'm really excited to
hear in this session uh from Ram a
engineer at spiral data all about
improving water networks using AIML take
it away rum
sorry I haven't been presenting for a
while so give me a second apologies for
the
okay
just sorry
much better
hi everyone uh thank you very much for
staying on until the last session I will
try to keep it short and but I'll try to
make it worthwhile there were amazing
talks all day so it's a little bit
honestly intimidating to talk the last
person but
just uh has a different perspective this
talk will not have a lot of code
Snippets but it's more like a zoomed out
talk trying to talk about the problem I
personally care a lot about and the
problem but my organization spiral data
cares about a lot and all the aspects of
the problem and how Python and its
ecosystem is helping us to try to solve
that problem so and I'm happy to take
questions at the end our connect
so just a bit of an outline we are an
Adelaide company based out of the
tonsley precinct the main thing that we
do is Big Data and machine learning
mainly for water and we do all the way
from Consulting trying to understand the
problem to build to support and we are
also I mean we are also proud of the AWS
award that we got recently
and these are some of our customers
is why do we do what we do and why do I
personally connect with the problem for
those of you who don't know about the UN
sets up sustainable development goals
for all the world all the countries as a
Target and goal six talks about ensuring
availability of water and sanitation for
all
and this is a mission that we as a I
care deeply personally because I come
from a place where I had to suffer for
clean water when I was in childhood and
had personal experience but this is not
just a developing country's problem this
is a problem that currently exists even
in Australia more than half a million
Australians especially in remote
Aboriginal communities do not have
access to clean drinking water and
sanitation
and just
why is that
the problem is extremely complex water
networks are complex everything around
water is complex just to give you a
perspective in Adelaide South Australia
most of the top people live within the
Adelaide Metro but the state is
extremely large
the density is around one to two percent
per kilometer and so what happens is you
have Mains that are running for tens of
thousands of kilometers and same as with
the sewer main so this is data just from
the sa water the biggest utility here
but there are also other utilities who
manage especially the Superman and the
sub IT Supplies more than 220 billion
liters of water
so just the magnitude is so complex and
so and so is the problems around it
what are the problems there is a lot of
them I just had to pick few just to keep
it short with aging infrastructure so
most of the pipes that are across sea
Adelaide and most of the cities are over
100 years old constantly needing
monitoring and upgrading and also the
issue with climate change because uh
there is the encroachment that is coming
in that is affecting all the water
networks hurricanes and also the
environment impact like any
leakage from the sewer systems can have
like catastrophic effects especially in
very sensitive areas
so we strongly believe most of these
problems can be addressed and at least
mitigated technologically but then even
technological challenges exist in water
networks the main being technological
maturity because water being an old
industry they're also very resistant to
change and for good reason because they
are transferring
water which is being ingested by us so
any mistakes can be catastrophic and
also the other problem is siled solution
so the most of the solutions that they
build are specific for those use cases
and cannot be leveraged across multiple
problems so the and also and with that
comes the issue with data because every
vendor has their own data standards data
provenance data management
but then it's not all dull and gloomy
there are also a lot of opportunities
the biggest that in my opinion is the
collaboration water industry is very
highly collaborative utilities talk to
each other constantly sharing
technological advantages and there is
also a growing sense of Fleet of iot4
devices that is exponentially increasing
but then the last part of the puzzle is
how do you use this data how do you use
this data to solve problems that are
important for them that to make it a lot
more easier for the water abilities to
maintain the infrastructure and also to
meet up with existing demand
so the question is do we have a silver
bullet no we don't and the idea is can
we at least design a framework but the
framework is not just technological the
framework includes people who are going
to build this people who are going to
use this the data that needs to be part
of this the tools that is going to be
part of it and what we I mean we are
just going to tell about the the
framework that we come about the mainly
on how do we leverage all these pillars
of data people platform and process and
especially using AI machine learning uh
to create what we call as decision
advantages decisions around operational
improvements decision around
strategic Investments like for example
the talk in the morning in Hall C talked
about the pipes monitoring using GP
geospatial data that things like we
don't use your special but it is a big
deal because it involves hundreds of
millions of dollars in to do any upgrade
and
so
what does and you need to do that by
maximizing value by reducing cost effort
and time but and what we see is python
has a role to play in every part of that
python helps us to handle data better
display data better it helps us to
streamline the process of development it
helps us to integrate platforms and it
also helps us to communicate better with
people using dashboards uis everything
so I'll just give a brief outline of how
we go through the process
so the main challenge here is
as I said the technological maturity is
very low in water utilities and as a
result people don't know what they want
until they see it so the idea is how do
we rapidly build solutions that the
customers can have a look and feel and
use it before they say yes this is what
we want this is what something we don't
want and the idea is there are multiple
stakeholders here the customers who are
going to use it people who are going to
pay for it people who are building it
and testing it and the idea and what we
see is we python provides a framework
especially the Ripple framework that
helps us to prototype it rapidly share
with the customers using streamlit and
if and once it is iterated we can we
also use like UI ux using bokeh
colorways and also apis using fast API
so there are these are a lot of them are
big ones but especially I wanted to call
out the small
people like the packages that are being
maintained by one person two person
especially like jupy text jupit text is
a godsend because data scientists love
the Ripple framework they want the
notebooks but
but software Engineers don't want to
touch it with a 10 feet Pole
so the idea is up text helps us to use
light person formats where
the data scientists are happy and go on
to do the job but the software Engineers
are also happy because now it's plain
text files you can lint it you can test
it you can do code reviews on it and you
can do Source control on it so yeah I
mean the environment is so rich that it
helps to solve a lot of the problems
so this is the platform that we are I
mean at a very schematic level again
there is data mainly the sensor data
that comes in and then there is a whole
heap of data engineering that happens
and then there is the platform itself
with the core is being building machine
learning models Rapids devops and
security and then the output gets
through the alerts analytics and API
gateways
so just to give a context regarding the
challenges in data engineering data can
come in any form some utilities are very
Advanced and they have like extremely
good data provenance and data management
some utilities they just have flat files
that they copy it into a hard drive and
ship it to you
and similarly the data velocity is also
quite different there are sensors that
can stream up to like hundreds of Hertz
that needs to be captured and maintained
and then the extraction patterns are
also complex sometimes real sometimes
patched and the volume is also sometimes
massive again there is a lot of the
tools that we have to use in order to
integrate them across different use
cases different
customers
again like you see the richness of the
like the python ecosystem that is
helping us to leverage this like we use
all SQL Alchemy is a godsend because we
can use that across all kinds of
databases and it and also similarly we
are fast API and especially I wanted to
call out the python wrapper that is
being maintained by Sergey Z standard
the core implementation is by a meta for
Facebook but then there are python
wrappers that are being maintained by
single developers who are just doing it
for passion and that helps us to
compress data at a massive scale and
being able to solve these problems
so the core of our work is AML workbench
the plenary talk talked a lot about AI
bias the the other aspect of the AI is
the accuracy and the reliability
especially in critical infrastructures
like water because if you build an AI
model that gives unphysical results like
negative pressure the customer is never
going to talk to us again so the idea is
how do you build reliable machine
learning models with but the speed to
Value needs to be high but then the
other important challenge is
most of the utilities are extremely busy
they have very little time and then you
need to make like whatever little time
you have you need to utilize it again
the way we are trying to see is a lot of
our work is built on anomaly detection
and what we try to do is we try to use
unsupervised model again start start
model SK learn to build a labeling
pipeline again it's based on python
label studio and smes gives us feedback
on how to I mean what
kind of data is an actual anomaly what
is not an anomaly and then we use that
to again build supervised models and
then again we have model Registries
using ml flow and then once the mod and
this gets iterated it's not a single
process right we constantly iterate this
a lot of times until the customers are
satisfied with the reliability before it
can get deployed again like one of this
amazing tools that is just maintained by
a single person is piode which is an
anomaly detecting package and anybody
who is working on anomaly detection I
strongly encourage to take a look at
that
so I talked a lot about the problems I
talked a lot about in terms of abstract
of how python is using we just wanted to
give a couple of real world examples so
this is a real world example this was a
massive fight break a few years back in
King William Street just down across the
road
so this is caused by what the engineers
call as pressure transience so there is
constant flow of water happening through
the network that is at a certain
pressure but then if there is a sudden
increase in demand like say during half
time of the metal does game if everybody
went to the restroom
that is going to be a sudden spike in
demand that is gonna what happens is it
creates like a hammer effect you are
hitting the network with the hammer at a
very high velocity and very high force
uh the technical term is pressure
transience and these are very I mean
these happen across the network but the
understanding is there is very little
analytics around this where does it
happen when does it happen how strong is
the impact and what is the likely impact
on that so just to give a schematics of
how this works so the transients are
being generated and this Hammer effect
that keeps ongoing it's not one time it
keeps happening continuously and that
creates what again in mechanical terms
is fatigue and then fatigue is like
similar to human fatigue which leads to
burnout but here it leads to breaks and
cracks so if it cracks it's not just a
challenge of fixing it there is not just
an operational cost that is also a huge
Capital cost invest in replacing and
more importantly there is a large
Downstream effect of this effects water
quality for a long time few days because
there is uh I mean there is I would say
the unnecessary sediments that coming
into the network that cannot be cleaned
easily and that affects a lot of things
especially vulnerable population if they
need access to clean water it this is a
major issue
and so the first thing is
detecting it requires sensors at really
high frequency slow so the sensors that
we worked on this with they say what are
that what they deployed is over 100
Hertz and that roughly translates to a
few billion records per sensor per year
it's not manually possible to analyze
any of them so the first kind of models
that we built is trying to extract
patterns from these data that are
equivalent to the pressure transient
there is also a lot of noise in this
and then but what they want is a
specific pattern where there is a sudden
increase or a decrease and then there is
a flattening out we were able to design
these algorithms the main criteria for
them was the reliability because without
that nothing moves forward but then the
other thing is also in the the approved
the solution needs to be fast because
they need real-time monitoring because
if there is an impending break they need
to be proactive rather than being
reactive and the other thing is all so
scalable you can do it for One sensor
but then the idea is can you do it for
100 sensors
what happened is we were able to extract
it but it's still a hundred more than
100 events per week per sensor and
that's still not possible for any humans
uh to do it because just because they
are maxed out they do it they have
minimal time and then they still can't
uh analyze each and every event
independently
and so the next step was we wanted to
group them automatically again we used a
lot of the information within the
characteristics of Time series using
various metrics Dynamic time warping and
all the other things but also we worked
with smes to understand the Tran the
type of transience like for example
what's the transient that is going to be
created by a fire testing versus a large
custom like a large plan for example or
like a large high-rise apartments and
because that was important for them to
figure out what's the source and then
again needs to be reliable yes it
significantly improved now they only had
to analyze a few tens of groups
but the challenge is this is One sensor
what happens if there are a hundred
sensor or a thousand sensors
uh what we did was we started to build
analytics around it and then like across
all the groups we tried to look for
patterns on daytime and then what we we
were able to surface those anomalies and
one of them is shown here which is like
the same kind of pattern happens every
Friday in the morning and and these are
the information that is useful for them
because now they can go and do field
testing and then they came back and said
yes this was a fire testing that was
happening especially in Peak demand
times and that's and I think they are
working with them to try to move the
testing time so that it doesn't affect
the infrastructure
and similarly there were other work that
we did in trying to understand is that
the characteristics change over time is
the new transient behavior that is
coming in that is stopping and and so
the idea is this is that what we call as
decision Advantage how do you take the
raw data and then try to create as much
insights as possible automatically for
them that are reliable and accurate that
they can act upon and create some kind
of a value
and so the second one I wanted to talk
about is alexandrina Council Alexander
is just 50 kilometers down South for
those of you who are new in Adelaide
have strongly recommend going there
because it's Where the River Murray
meets the sea and it's a beautiful place
the challenge is it's a large geography
the council is over a thousand square
kilometer it's very ecologically
sensitive there are a few people but
then there are even fewer employees and
maintaining the Wastewater network is
really hard for such a large geography
with such few people excuse me
and the challenge is so the drains can
get blocked there are a lot of reasons
that it can get blocked the predominant
one being that trees Roots trying to
force its way into the sewer Network
and what this happens is if there is a
Blog there is a back pressure happening
and then this can cause spills and then
spills especially around the
Delta of the Murray river is very is in
one it has a strong environmental impact
the challenges there are no alerts that
are present at this time because most of
the alerts and in the modbus system to
control these pumps is based on just
thresholds as it reached the start limit
or is it has reach the stop limit and so
this is there are no alerts they have
these issues happening from time to time
and so we are working with them to
create a solution that is of operational
value to them
so the data then again we worked with is
for over five areas and then they had 54
pump stations and over 800 million
samples
so the idea here is so this is what they
call as there are multiple pump stations
and then there are sums that are
associated with them so there is sewer
water coming in from every residential
or other commercial buildings and then
it gets into the main Network which then
goes to the pump station it gets pumped
all the way to the wastewater treatment
plant and so what happens is as you can
see the red there it's slowly racing but
then it's not below the lower limit to
raise an alarm and it is not above the
higher limit to raise an alarm so unless
somebody sees this data manually
constantly inspecting the dashboards
there's no way for them to know and I
and likewise in this specific scenario
they had no idea about this and it
caused a spill where where state
government had to get involved and
federal government got to get involved
and there are a lot of other cascading
issues as well it affects tourism the
main source of
economy and a lot of things so what we
were trying to do is can we find
patterns that can prevent these large
blockages to get spill
so that and as you can see from the data
the red circle shows that was a small
block that happened that got released
for a short period of time and then the
larger block ensured so we were able to
pick up this from the historical study
so ideally if a solution was in place an
alert would have gone out when this
small block happened and released and
then the field investigators could have
taken a look solved the block before the
environmental impact happened
so
there are a lot of problems in water
networks and what I'm trying to do is
just giving you a flavor of some of the
a very very small tiny piece of the
problems that we are trying to solve and
python is helping us to solve from all
sides like it is helping us to better
communicate with the customers build
these problems solutions faster deploy
them better and also maintain them
better so
summary if there are just three things
that I would like people to remember
it's just that
machine learning has a huge potential to
improving water Network utilities
especially automatically analyzing data
identifying patterns and creating alerts
and it helps them to do better decisions
and python is the base for all of the
technology to be made possible but more
importantly that a lot more people with
the right skill sets needs to be
involved in solving these problems a lot
of people are involved in energy
transition towards net zero but then we
also need a lot more people to be
involved in solving water problems
as Gibson said the future is here it's
just not evenly distributed especially
not for the water Networks
so
yeah if you want to learn more about us
I'm here in bike control someday and or
you can go online we also have a
newsletter where we just don't talk
about our work but we talk about the
tech in general across the water
networks and yeah I mean feel free to
email us and we are looking for good
people people who are motivated to make
a difference to solving socially
important problems people who are happy
to work with ambiguity but even in
general if you want to know more about
the area so
thank you
[Applause]
thank you ram that was a fascinating
session we may have time for one or two
questions if anyone in the audience
would like to ask a question I think I
can see a hand
yes excellent we have a question
uh that was a really interesting talk
um personally I work in a similar space
where I'm delivering Ai and ml based
solutions to sort of Legacy or very
traditional Industries and have a lot of
trouble building confidence with those
those customers especially uh when
you're sort of the first pass at
bringing Information Systems in and
you're bringing in AI stuff I'd be
interested to know how your experience
has been building confidence in in
systems like this especially hands-off
automated ones
yes it is a challenge it is an ongoing
challenge but what we realized is
they are a lot more
they are I mean they are open to hearing
ideas that are not operational so if you
go and tell them that we can help you to
make strategic decisions of analyzing
historical data to say like for this uh
part of your asset is bad this is needs
more maintenance they're more uh they're
all years but if you want to do any
modifications to the operational
day-to-day operations in terms of
maintaining then they are a lot more
resistant so the idea is you try to get
in and start communicating with them at
a more strategic level and trying to
solve those problems first and then
slowly get into the operational I mean
that's been what our experience has been
I can't see all your wonderful faces
through the lighting so is there a
second question in the audience
I feel like an Auctioneer going twice
no there isn't Ram thank you very much
for a fantastic session here is a small
gift to say thank you for your talk
thank you very much
[Applause]
Chris would you like to come down and
join me uh we won't do an extended uh
track close but uh from myself and from
Chris uh we wanted to say thank you very
much for coming and joining us at the
our connected Universe track it's the
first time we've run this track together
uh and we're delighted that you could
all be here thank you
next time
[Music]
[Applause]