welcome back everyone hope you had a
nice break got some food bit of exercise
hopefully a little bit of sunshine from
wherever you are in the world
and uh welcome back to the second half
of devops
and for our first talk uh here back here
in the devops track um we have a talk by
peter chu
called things might go wrong in a data
intensive application
i'm intrigued to find out what possible
things could go wrong as we all know
there's an endless possibility so with
that let's take it away peter
hello everyone i'm here to talk about my
dev oops experience about building a
data intensive application
so what is data intensive application
well i think there is no formal
definition
and i will adopt an idea from deciding
data intensive application written by
martin equipment in my talk
in this book we call it application data
intensive if data is its primary
challenge
the quantity of data the complexity of
data are speed at which lead is changing
the next view circles are bottlenecked
for this idea we can say almost all
success for our business
a based on data intensive system
to decide the data intensive application
we need to consider not only functional
requirements but non-functional
requirements
such as reliability and scalability
and we already have many success stories
from google amazon and facebook
it tends to let the thing
we can just follow their architecture to
build reliable and scalable system
we can just introduce things like the
release or kubernetes and then
whoa now we have system can keep
variable and scalable
well i won't say totally wrong
and i thought we should notice the bios
here
those stories are just in effect as the
sharp layer current situation that of
course
uh
they maybe have different requirements
compared to your place
and they don't tell you how to get there
how many mistakes they made in the past
and that's why we are here today
i am a software engineer from taiwan
as well as a python start like all you
guys
i am a psf contributing member and
organizing pycon in taiwan
by the way taikan taiwan will be home
virtually this year
so welcome everyone to join us online
from october 2nd to 3rd
you can get more informations on our
website if you're interested
back on track the main point is
i've been working on a data intensive
systems for many years
and i'm glad to share my experience with
you today
let me introduce the case we study today
it's a data managed
we host user data for various usage
patterns and workloads
such as online streaming ld data
abbreviation
file distribution things like that
currently it hosts several petabytes and
transfers several several terabytes a
day
in case you don't have a feeling about
these numbers
i google some vectors with you
for example size of
github
arctic code world is 21 terabytes and
java's data lake has 34 petabytes of
data currently
and here is a fun fact
if we put all our basic disks on the
ground we can cover whole football field
and
of course this case cannot be compared
to those strands but i think is no
practical case for most people since not
everyone has the opportunity to view
facebook or google from scratch
this is what they look like
i think it's a game architecture of a
large-scale distributed system
on the top of the diagram we can have
reverse proxy and stellis application
servers
which are responsible for receiving and
serving your service
at the bottom of the diagram you can see
we have various kinds of data
technologies to store structure and
unstructured data
we use starting and partitioning to
distribute loading to different nodes
also we adopt this review file systems
to store unstructured data
there are thunderstorms there are some
other subsystems for drop processing and
data analysis
this is just a very rough diagram i know
to do just like you have an idea
i won't go through deeply since today we
focus on mistakes we made
now how we succeed
in the following time i will tell you
four of many incidents i made in the
past years
the first two are about scalability
the others are about reliability
finally we will review what we can learn
from these incidents
so here we go instead of one
one day we had a new customer
they upload their data generated by
thousands of devices to our platform
24 7 f365
we never saw this usage pattern before
and as you might expect it
we cannot handle it
i trust the issue and figure out a
problem using database
for if for efficiency we use
optimistic locking in our system
optimistic locking means
the system doesn't lock the data
explicitly before we manipulate it
we use it because we ensure
our workloads will now cause contention
very often
but we're wrong
a straightforward solution is to switch
to
pessimistic locking
which ensure only one tray can
manipulate data at each time
but it actually causes many problems
performance of other usage patterns
don't worry since their workload are
suffering from unnecessary locking
operations
in the end i decided hybrid and adaptive
approach to address the issue over here
means we introduce both
optimistic and pessimistic logging in
our system
operations that may encounter contention
such as writing
will use pessimistic login
the others give use optimistic login as
before
on the other hand
adaptive means by
by default
operations just need to obtain a lock
from
application servers before processing
this is not real pessimistic lock since
operations may still complete while
doing updates in database
recall is local lock
well carefully this detected
will complete is actually occur
the system will ensure the rear
pessimistic locking automatically
we call is global log
and
[Music]
it will be more clear in this diagram
you can see local logs can be obtained
from each application server
and global logs are obtained from
databases
in this approach we can satisfy all
users and their usage patterns
this is how because how this case was
solved but what can we learn from it
well in my opinion
the root cause is we didn't predict a
usage pattern like that
in all our test scenarios
optimistic lucky just works fire until
we encounter this scenario
this is a classical uh skeleton
scalability challenge we must be while
building a data intensive application
we will go back to reveal our release
and learn later
let's go to the next one first
so what happened this time
we have an abdul we have an optional
data manager feature for users
basically it scans and removes expire
files for user
it's just a prototype at latin since
almost no one used it for many
years
a lot time it was actually implemented
as a simple quest job
we didn't care about it until one day
the user found it
and made a million times more requests
to this poor contract
now i knew this feature is necessary for
our customers
so i will implement it in a robust and
production ready way
why need is a job processing system
we found one from github called rescue
which use radius as a java q
and can process tasks in a
distributed way
so what can we learn from this incident
again of course this is a scalability
challenge
the difference is
the pattern is exactly what we expected
this time
what we are expected is the law
next one is about reliability
we know security is also an important
non-version of requirement for the data
intensive application
so we
most specifically might both
decide to outsource our data protection
module to a
professional security service provider
the project runs mostly until we deploy
it
we start to receive complaints
that their data is corrupted
that's very weird
because not all but just part of data is
corrupted
what's going on here
i found the problem is the encryption
process
the encryption algorithm is black based
so we have it
we have to add padding before we encrypt
it
let me explain how padding works in case
if you don't have related background
the assure block size is 16 bytes in the
following examples
first example if the input size is 12
bytes
we need to append four fourths
to the data
after that it becomes 16 bytes
and satisfy the rule or not the size of
data must be multiples of block size
in other case
the data size is 16 bytes this time
what should we do we still have to add
in padding
this time we add 16 16
at the end of the data
it's already satisfied rule why should
we do that you may not ask
because in the decryption process we
need to remove padding to recover
original data
lag reason use the last bite to identify
how many bytes of padding need to be
removed
that's why we need to padding in this
example
even though its original data size is
already
multiple multiples of block size
so now we came back to the original
parliament
data corruption occurs because
the implementation of our integration
algorithm only consider the case running
our example but not case too
that's why some users complement their
data is corrupted the other is not
fixed fixed list bug is trivial but we
have to fix corrupted data in the system
as well
this is where reliability issue differs
from others
we are not only fixed local but the data
corrected correct data is also trivial
we just filter corrupted data by
metadata starting database
then we can correct corresponding data
this is stable as an example
data corresponding to first and second
rule is not affected by the five since
it's not processed
by an incorrect algorithm
the cellular is also not effect affected
since its size is
multiples of black size
even though it's indeed processed by an
defective algorithm
the fourth row is affected but is
already fixed
with this thing issued by the version
number
we need this column since we will
correctly split curves in a distributed
way
we will discuss it later
the last row is the lonely one need to
be fixed here
since it was processed by an
incorrect algorithm
and the size is multiples of black sides
it's time to fix us f9 as a fixed
recall
indeed in general we will say this is
just a silly bag
and most times we can fix it with our
pen
but for a large scale system like us
it's not funny
the fact affects affects millions of
files in our system
you can imagine it's a very very
straightforward emergency situation
we if we
evaluate the situation and
we found out if we use
a
basic
python script to fix it
we need hundreds of days to correct all
effective data
nowhere we can do that no user will wait
with us to do that
so we decide to use a
distributed way to get things done
we use a job processing system called
gearman this time
we don't use radius as a job queue since
we want tasks can be kept in a
persistent storage
not immediate release time
so the new cost is obvious
we had an unreliable provider
but why can we figure out a problem
before we deploy it
right
remember the data is affected by this
bug only if its size is multiples of
black size and the block size is 128
bytes
then we only have a 0.7 percent chance
to file back by testing
so the
uh the listener here is
besides increasing taste coverage we
also need to consider how to
tolerate software phones and human
errors
in the production
because there is no way to prove a code
is created or not
and
okay last one
this is the last incident i will show
today
we didn't just screw out the user data
guess what
we lost data too
we lost it not because we don't care
about reliability we d we d we store
multiple replicas in multiple data
centers while receiving user data
that's why that's what we're doing our
system design
so but but what is happen
the background is the system not only
consider reliability but also not
balancing
we will consider both distant usage and
failure domains while choosing a
location to store data
this implies that
newly added node will have a higher
chance to receive data since they have
more resources
available compared to all nodes in
general
here is another fact is
new machines have a higher chance of
damage this is called
bathtub curve
compare these two things cause the
incidence happen
data right into newly added node and you
not
broken that's why
this is another classical reliability
trends caused by hardware fault
on small scale these things are really
happening
many people need more backline their
computer because they don't think this
will happen on their
on their
yeah on their computer
but in a large scale system there are
lots happen regularly
and
the value is is it's almost impossible
to solve it completely
i can tell you how many works we
definitely use
with the week long run test
we
keep writing and reading data in the
system in that while we're doing lab
tests
we even go to the data center and plug
on the block of the
laser cable directly to see what's
happening
we also analyze this system reliability
by some theoretical methods such as
markov trim
if you're interested you can go to and
see
soa
that means service level agreement of
your club provider
i think there is no one guarantee 100
reliability
in the end what can we learn from these
stories
three takeaways
before we start i have to remind you
like it's um a lot of the subjective
you can agree
or not and also welcome to discuss it
later
first no sleep bullet classical
we've seen successful stories from
google or facebook
where i was a junior junior engineer i
may think
their story are so amazing we should
keep following their tech stacks to
build reliable and scalable system
however after the journey through
my journey
i
know that there are many things behind
the thing
but why why can we just follow some
blueprints to build a reliable and
scalable system
like building a bridge
or a tower
in my opinion first is how to enumerate
all possible reliability reliability
causes
just like what i showed you previously
even though we do many preparations
that are still
almost
second
pattern and law
keep changing while your business
expands
you cannot get a final version of the
specs that tell you how many kinds of
workload
we will have and how many requests per
second are needed
as a software engineer i think that's a
very interesting and very
unique challenge we need to deal with
second
you may think
exposure those incidents will not happen
if you have some
fancy techniques
yeah yeah that's fair
i believe a talent engineer can prevent
prevent some of many incidents we have
but
she can use
and tree in most rice scenarios she can
i don't know use eventhousing to scale
system out or she can introduce puzzles
to get strong consistency or adapt
calcite consistency hash ring and
irrational coating in their system
but there's one more thing
i have mentioned yet
the platform reliability and scalability
maintainability is also a key maker
in his book
mapping complements they lack
reliability scalability and
maintainability are the three most
important factors
while designing a data intensive
application
although the insert incidents i
mentioned are primarily about
scalability and reliability
maintainability also play an important
role behind the scene
generally speaking
a data intensive
application it's not also a large scale
system right
it's essential to have a talent team to
support and involve it
no one no one can do it by herself
uh here is this
example i heard
kafka we know it's a messenger kill or
string in processing system right
usually it's just a component in a data
intensive allocation
but you know what in leaking they had
hundreds of engineers to maintain it
why do i mention this here because i
know you guys are those who are willing
to challenge yourself that's a good
thing that's a good thing
but i i recommend you to think twice
before you introduce some advanced
techniques in your stack
consider your thing have ability to
maintain it or not
now you may think
you talk a lot but
the slot meaning it doesn't have
anything to do to build a reliable and
scalable system
i have no answers here but
i think the most important thing within
policing studies
people
people behind the systems and machines
are matter
people
are the most important part of a service
actually we engineers provide service to
our customers
nano's machines
machines are just our interface and
tools
we need talent engineer to not escape
problems and fix them behold before bad
things happen
we need engineers to tolerate and
troubleshooting reliability problems
also we need sre engineers to build an
infrared that has this opportunity to
consumer errors
the infrared can also help developers to
figure out problems in a more productive
way
finally we need supervisors and project
manager to understand it's important to
have a good development culture
this is my zen of data intensive system
design
if you have experience i share today
you may have
similar feelings about these things if
not i learned these things the hard way
and
i hope this talk will save you from
repeating the same mistakes
thank you
well thanks peter that is that was
excellent um lots that i agree with
there which is just tremendous i
particularly like the emphasis on
maintainability uh that you mentioned
there towards the end of the talk i
think that's something that we often
overlook and the fact that it's it's all
being looked after and managed by humans
and largely as all of us here in the
devops track are well aware the humans
are where a lot of the challenge comes
from so thank you so much for that talk
um now peter is actually here in the
chat so if you have questions about
anything that was in in the talk there
do drop it into the chat and hopefully
peter will be able to answer those
questions
um
thank you so much that was tremendous
now between uh now and the next talk we
have another break
so
drop in some questions uh have a chat
amongst yourselves um otherwise yes grab
a quick drink
stretch your legs and we'll be back at
about 3 p.m for the next talk see you
soon