[Music]
hello everyone and welcome back to the
kiara theater at linux conference
australia 2022 uh next up in this slot
we've got matthew oh i have matthew
oliver uh who is a senior systems and
software engineer working at nvidia
where he primarily works in upstream
openstack as a swift core
today we're talking through the benefits
and challenges of integrating open
tracing into openstack swift now we're
going to try and have time for questions
but if we run out of time there will be
follow on question time in the post chat
post talk chat kia ora theater
accessible by the
left sidebar in venulis it's all yours
matthew take it away
thanks thanks for the introduction and
yes i say we might need to there might
be
uh i tend to waffle on a bit uh golf and
tangent for those who know me and i'm
going to try and keep it crisp and and
quick so
this this talk all started uh because i
was trying to
um
benchmark swift on some new i've some
new hardware at uh at nvidia
and the way swift is made it's all
modular and i wanted to be able to start
tracing and so i want to start timing
between these middlewares we'll get all
this in a sec but i then showed our sre
guys i loved it and it's kind of
snowballed into this whole
uh tracing thing and it's upstream so we
started working on upstream
now i'm trying to figure out why am i
not going to the next slide and there we
are so like i said my name is matthew
oliver i work at nvidia um on all the
socials you'll see me i'm usually up as
matt oliver you um i've been involved in
swift uh since 2014 and i've been to
swift court since 2015.
so i've been doing it for a while
uh
i am going to tempt the demogors today
and most might well say most like half
my talk is going to be
a live demo um or locally here but
it's going to be interesting
but to get before you guys can start
looking
at the fun traces
you kind of i want you guys to
understand what you're looking at
otherwise it's not as exciting uh well
it won't be as exciting as this for me
so
i'm going to try and bootstrap the
audience into understanding so we're
going to go through this really quickly
because i want to get to the fun stuff
so
let's start with
what is swift it's a fully open source
end to end object storage solution
it can be standalone
and integrated into your environment
this is what we do at nvidia or a part
of the rest of the openstack ecosystem
it's highly extendable it's mostly
written in python and that's not
it is written in python so it's very
extendable but that's not the only
reason why it's extended we'll get to
more on that a little bit later as we
look how traces work
um
basically it's done with whiskey so you
can insert me to wherever you want where
it's not written in python stuff like
erasure coding it's done in c
um it supports a swift api you'd hope it
does because it's swift
and also the s3 api
you can do replicated or ratio coded
policies
it's great at scale um
the bigger actually turns out bigger the
better we're we're massively scaling it
at nvidia uh in fact we've hired a bunch
of the swift cores to make like these
bottleneck huge scaling issues kind of
better go away get better and better so
they're actually a really fun time and a
really fun community and fun problems
we're solving
the community upstream community is
actually really awesome like there's a
reason i started in 2014 and
i've gone through
different roles in different different
places and stayed in the community
because they're awesome so if you're a
dev an operator you ride docks you're
kind of just interested in the space you
just want to hang it's a really it's a
really awesome community anyway let's
keep moving because otherwise i'll run
out of time we don't get much time for
questions at the end so
let's have a brief look at the swift
topology we need to know this because
when you start looking at traces and
we're jumping around through the cluster
you kind of want to know what you're
looking at so
we'll talk on about the very basic paco
installs these are what you tend to see
in really small deployments
where every node
is stored has all the swift services the
swift proxy server account service
tennis server and object server
but then as you stand to scale up people
tend to start doing this they start
splitting out the proxy from the storage
system and this makes perfect sense the
swift proxy is stateless
um and so it's perfect for
containerization
and
uh if you put more of them in it if you
want more throughput and more and handle
more connections you just scale them and
if you want more storage you scale the
other side that's it's pretty
straightforward right the swift proxy uh
is also if you're doing things like at
rest uh encryption the swift proxy is
the one that's going to be talking to
uh
is the one that's talking into your
getting the encryption keys and stuff so
it's also nice to have that separated
and then you just get a encrypted stream
hidden to the storage notes anyway
i just told you tangence is what i do so
i should have got someone to like ping
me or hit me when i start going off on a
tangent too much
okay so at greater scale this is what we
tend to see right so when we get even
higher
it's an object uh swift is an object
storage system so
99 of its storage is objects and so
having
uh
just being able to separate that those
object nodes away from the rest of the
cluster i mean
you know what i mean
uh isolating those makes sense
the account a container side of it is
our metadata layer in swift um it's
keeping track of you know acls and other
metadata and other stats like an account
will have a list of
a list of containers and the containers
will have a list of objects
and we'll have all the acls and
metadatas and anything else and other
aggregating stats up the stack so you
can think of the
ac layer we see in this diagram as
the metadata layer so makes sense to
this nice approach this is what we do at
nvidia
there's also when we scale swift also
knows about other availability zone
issues too but we're not going to go
into any of that stuff we don't have
time so
here's a quick look
at what happens uh when swift
uh we got a basic request so i think my
mouse works i'm assuming people can see
my mouse do stuff
but maybe you
can't i don't know anyway uh i'll assume
you do so
the request will come in and hit the
swift proxy and the proxy this is
obviously going to be uh with an object
put on probably a three times replicated
system
um so what will happen is a request
comes in
the
swift proxy will look up uh on a
modified consistent hash ring
for that policy of where which storage
nodes are responsible for storing that
object and we'll make three connections
oh no we're going to see this in this
trace this is the only worries when i
bring it up right and here is a snapshot
of trace this is not live obviously
this is a snapshot of our trace just to
give you an idea of what you've learned
so far so the blue here is the proxy
server
and it's making an object put
and so it's making three connections to
one object server two object servers and
three object servers
now
we can you can also follow along that
over here you can set over here is a
container server
and so the object is being put into the
object server and then it's up going to
go update the metadata layer and so we
can see this in our in a glance which is
kind of cool and we're getting our
knowledge
now we'll quickly talk about open
tracing
this i know this slide goes against all
um
what you should and what you shouldn't
do with with slide presentation because
there's way too much information on this
slide
but i put it here in case you're
interested and you get these um you get
these slides and you want to go back and
have a look at stuff
so we'll go over very quickly so the
project information for open tracing is
opentracing.io it's available in nine
languages
i obviously use python because that's
what swift is
and
it basically is an api specification
frameworks and libraries that have that
have implemented that specification and
allows us as developers especially open
source developers uh to be able to
instrument our code
using apis that do not fender lock us in
no one likes vendor locking but what's
even better is if other
if other projects are using open tracing
now we can start you know integrating
with each other and start tracing
let's not think of swift let's see the
greater community the greater
open open source open source environment
anyway
the concepts in terminology i'm only
saying these because i will accidentally
spout them out um and you guys won't may
not understand what i'm talking about so
concepts and terminology so there's a
tracer so when you're instrumenting your
code
you're dealing with a tracer all the
time and the tracer creates spans more
on that in a sec
um and keeps track of the active span
it's kind of important um it also does a
few things it injects which is really
serialized and has a term extracts which
is really deserializing the tracing
context and so we can move it across
through
process
boundaries
um
a span let's move back for a bit so
here's this is back a slide
going to come up you know here's a slide
so we can we can say that this is a span
each of these lines represents one span
and
this whole inject extract thing is
when i get to the proxy swift proxy and
i want to send a request to a different
server or a different process boundary
then i'll be able to inject this context
of the trace and then send it off and
then the the
in this case the object server can pick
it up and continue the trace and then we
can knit these things all together at
the end
and so a span is the basic building
block um it references other spans it
kind of has this parent of child of if
we go back again i'm sorry for jumping
around so much but you can see that this
band has a child and this child has a
parent you know it keeps track of um
the tree i guess
and it's starting to finish time stamps
and has things called tags logs and
baggage
all of which are key value pairs the
only real difference is a tag is kind of
can be filtered by or queried and so
it's kind of it's useful for when you've
got it when you've pulled out the trace
they expect to use standard tags which
makes sense stuff like http method and
status code and stuff like that so we
can all be
use the same tags across
a trace or custom so in swift we
tag every request that goes through our
cluster
if every every request gets a unique
transaction id so for in our case having
that as a tag makes perfect sense
logs are just key value pairs again that
are just stored to span so you can pull
information out
i guess another example that i like in
our in the swift deployment uh swift
version of this is a proxy server
is smart enough to when it has trouble
talking to a node
it can
update it doesn't it you can error limit
that node for for a certain time
now when you're coming back and you're
finding you're doing a trace because
you're trying to find a problem that's
the kind of information you want to know
about your cluster so it's perfect thing
you can pull in and get a snapshot of
time in your trace too
and the other thing is baggage now
baggage
it's funny name it it allows you to
predict
these key value pairs
um
when injected or when serialized are
stored with that serialization so it
means you can do some communication
between
traces which is between tracing entities
between processes between whatever you
want to call it
okay i've probably rambled on way too
long then
quickly we talk about whiskey because
it's going to make sense
when you look at us one of these traces
i'm going to show you um
there's a lot of there's a lot of spans
and most of them are because of the way
whiskey works whiskey is a python
rest framework um well we're going to
delve into too much details
pictures are worth a thousand words so
in essence here's a very basic example
it's built up a bunch of middlewares and
an application at the end and as a
request comes in it'll walk it'll work
its way down the chain
and then back out and send that back the
response
now a middleware you can interact with
the request as it comes in
when a response comes out or
you can kind of i call short-circuiting
it so you can
in this for this this example we say
that third middle wears an
authentication middleware and it may
have gone talk to your ldap server
whatever you want and said no not
authorized to send 401 unauthorized back
and we don't actually have to make it
all the way to the application
here's a cut down version of what this
looks like in the swift proxy server
just so you can see that
it's it's not nothing's rocket science
here in fact you also notice that a lot
of the swift features are middlewares in
their own right so if you want to
um not have any rate limiting i don't
know why you don't want that but say you
don't care about living because the
system's all for yourself
you could just remove that middleware
from the pipeline definition which is a
list of um middlewares and we wouldn't
talk about it it would just the request
would skip it completely so that's
really cool um
and earth is also where that
extensibility comes in so you can write
your own middleware that puts that
inserts your own business logic
and inserts that uh and you can send it
anywhere you want so you can do
something on a request as it gets coming
into the proxy
or when it comes back out or better yet
it turns out the swift uh
account server container servers and
object servers are all whiskey and have
their own pipelines too so you can even
insert a middlewares to do something
before it hits disk or when it comes out
so swift is highly extensible it's
pretty cool
here's a here's another trace example
where it shows this this is an
authentication
to our tempo system which is only for
development
but you can see
now we look on this side we can see that
it's walking down through the list of
middleware and we can see actually watch
it
chain along going working its way down
and then
coming walking back up the chain never
made it to the application never made it
to the proxy server and walked its way
back up and received a response
so we're almost we're almost at demos
this is the exciting we're going to also
design a bit thanks for sticking with me
we've been going for a while
um
so let's see so
high level integration so like the rest
of swift it's done
similarly well i'm a swift developer so
it kind of makes sense
uh it's a basically
uh
middleware that we insert um and there's
a trace mix in for uh that all the
middle was inherent because
it helps
and i'll explain in a sec so the trace
middleware as it comes in will be
inserted in the pipeline of all the
storage nodes too
and when a request its sole purpose is
when a request comes in and is to decide
if tracing should start or not
it's um
it's also i guess it's also to note that
because it starts and stops tracing that
anything to the left of it so the
gatekeeper here
it won't get it won't show up in any in
any traces so it's good to have it as
far left as you can go uh gatekeeper
it's always on the end and swift because
it stops yuckiness getting out
data you don't want to getting out so
the request comes in and hits a trace
now there's three ways that the trace
that that
that the tracer will start a trace
um so the trace middle will start a
trace and the first one is have you
given it a specific hmac signature so
the whole purpose of this
and for our use case is
we have a customer who's having issues
and
at the moment
we can usually come tell us they're
having issues we look at the logs and
swift does great logging but sometimes
there's still that level of read the
logs come with a hypothesis and then
they go test to see if that would make
sure that's correct or whatever
but with this tracing middleware
inserted we can go give that customer
use or whatever uh hmac signature to
attach to the end of their query
and
um
and as soon as the trace sees that it'll
start tracing
and it'll work its way my mask on it'll
work its way back up this chain and then
work your way back down and then stop
tracing
so that's one way of starting tracing
the second way is a continuation from
that so once we get to the proxy and
we're making those additional requests
out to storage nodes
we enjoy like i mentioned before we
inject the tracing context if it's
tracing
into the request and so that's the
second type so if a trace middleware
sees that there's an injected
trace context it knows to continue
tracing
the third is the what i'm using
in this demo because
i'm lazy and that is log every
x request in essence and i'll have it
set as log every request because i want
to test
um but obviously this could be like you
want to do a health check so you might
want to run it every thousand or ten
thousand or whatever you want you pick
and so the three ways the trace mix in
just means a whole bunch of helper
methods are available for
instrumentation to each middleware it um
it automatically starts
traces and enters and and ends the trace
that's sorry not the trace starts the
span
when it enters and ends the span context
when it finishes and just so that's why
we just for free get that nice uh
graphing the tracing
and another than that you know it just
helps oh the other thing it does is also
it calculates the
the response um
the status code as it's um of the
response
we do this through the line because
things like the s3 api
they
they might return a different response
code for
uh status code for different
apis than what swift does so we it's
nice to be able to track what it was at
a certain point in time that's just an
example
so the high level integration this is
the hmx signature i'm going to talk
about this is the same slide as before
but just annotated and so the trace in
this case is going to come in hit the
swift proxy go through that pipeline
it's going to see this hmac signature
which is time limited and then we'll
start tracing the request as it's going
to send inserts the trace context is
pretty straightforward
so without further ado we can do some
demoing
so let's see if this works well um it is
lca so you know take it off with a grain
of salt
i'm using uh jaeger tracing because it
was just a docker install away they
implement the open tracing api so
it just works right when i say just
works i had to make a tracer this is a
document that i created for
the openstack ptg so if you want to have
the exact same environment that i have
i'll give you a link in fact the link to
this google doc is in my um
is in these slides
so this is the whopping amount of jaeger
tracer code
i think it's after imports what three
lines of code so
yeah it was very it's very easy and
that's because this is the whole
advantage of open tracing
okay so this is
jaeger tracer and this is what we're
going to use to show some examples but
first we need to actually make some
things so let me go to a terminal and
step one this is a fresh
swift all-in-one uh virtual swift
all-in-one a demo environment a dev
environment that we use so i'm just
going to do that that's just going to
use the swift command line tool to
to run an authentication
and then
uh where i'm sorry i'm lost
we should see
a swift
and if we do find traces
where you have our first trace
this is the transaction id
we know what to get and if i click on it
this should look very familiar because
there's an authentication so you'll see
it worked its way down to tempo so this
screen is a little bit
smaller so
whatever
uh so we see it's what we saw before but
if we
if we like look at the first line we can
start looking at all these tags so
you've got hey it was a get it got a 200
its url was
the temple the url
and we have logs and stuff like that
very exciting stuff
but let's look at more interesting
things
um
sorry be there a sec
so let's see so next
we'll quickly
put in
let's put a container because we need
somewhere to put objects
so i'm going to put in
just a container called c because i'm
lazy
um
and i have no imagination on naming
and at the same time i'll put in an
erasure come on apparently i can't
it'd be fast if i just type this i'm
having trouble
copying and pasting
and so where you can see these
transactions happen we've got some
transaction ids here
four three five nine and four three six
b
uh if i go back here and do a fine trace
here they both are
first one
has 175 spans there's a reason for that
it's a lot one it's more in that path
that code path is switched up a lot more
instrumented
than the ec put the container but um
also because this is like i said this is
a clean brand new um class um swift
environment so the temp auth is going to
create
the account doesn't exist so it's going
to have to create it so you see this
really long line
the first view are probably going to be
like hey does this account does this
account exist
um
say it's an account server and it's
getting 404 so the account doesn't exist
so there's a whole bunch of extra effort
going on here because it has to create
these things
um if we go look at the
towards the end we should see
it actually putting
so many
we should see it actually starting to
put this container server in and then
it's going to go update the account
server metadata so we can watch that
happen and it should happen three times
because
my container ring is three times
replication
now i'll show you the same for this one
it should be less because it's um
still very long it looks like maybe my
cache my memcache um
is not is either ejecting things very
quickly uh because you would have
thought that would have been cached but
hey see here's an advantage straight
away of kind of getting to know what
these traces look like um you can't
start seeing a long line problem
i guess another thing to note is what
i'm
the stuff that i'm putting in
are kind of all kind of junk objects
right like why don't put an object in
let's let's do that
um
so right so
that first container was so many so many
spans because it said hey does it can
exist no go get
it doesn't so i'll go create it it
doesn't contain exist no i'll go create
it
and then finally go create so there's
lots of extra requests that usually
shouldn't happen but
because it's a nice
new greenfield system that's what
happens so let's put
uh
what i'm just trying to copy i'm really
struggling with my copy and pasting
today
i blame
doing this live and not in a video
sure
and so now i'm just putting an object uh
into the c container which is replicated
and while we're at it let's put one into
the eurasian coded one which i think are
called ec
now
these
are silly objects they're four bytes
aren't they they've just got like one
two three four in them um
so you know
especially for erasure coding you
wouldn't want to rate your code 4 bytes
it's a little bit overkill but anyway
let's see what happened
so if we find these traces now we should
see
two new puts uh it doesn't matter really
matter one's a bit more again
hold on these are objects so this is
this is correct okay so we've put us on
object in
it's let's see what this first one is
it is
um yeah okay so this should this should
have been cached because in our memcache
server we should already know that uh
auth test kind of exists so
there's a bug um although there's a mem
cache setting issue on my dev
environment but i'm glad i can see that
so the important stuff here is this
stuff then
where we're actually putting the object
in
testo and again because a standard
object uh by default i've got my default
policy as a three times replicated we
should have three so that's not as
exciting what is more interesting is
some of these other spans that i've
created
so i can't read it so
this is a span
so i've instrumented the path going
through so there's a store object method
a bit like flame graphs right so it's
getting put connections it's gathering
all the the connections to the storage
nodes it needs to be able to put this
um
and there's connect to the one node this
one that's connects to another node and
we see the nodes interleaved in which is
nice
um
down here we've got check for failure
put connections transfer data well
there's four bytes right so that's not
going to do anything and then we've got
get put responses
um the idea is
was that durably written if it wasn't it
would go out make another connection
that kind of stuff
um
looking at the ec one should be
identical except it's an ec policies and
i think the policy was a 4-2 policy
that's four times what's our data and
two times um
uh parity
so we should have six connections we
have one two three four five six and
we've got the same kind of stuff we're
transferring data takes longer well
actually is it taking longer or is it
there's less data to send
but there's actually more because it's
four bytes and so there's a bit of
overhead this is why you wouldn't use a
razer coating on a four byte object
um but again i don't want to go find a
tangent and we can also see doing
updates
and so there it is it's doing a put to
the ec container it got a 201
so
it's kind of fun to watch what it should
look like but
they're not all that super exciting
because it's just putting some objects
in i mean that is the main i guess
reason for an object store system let's
try and get that ec container back out
object back out
so for pudding we obviously had to write
to all the
all six nodes
when we're getting on the other hand
it only has to talk to four so we should
see four
two four four b e
i don't know why i'm trying to make
confirmed transaction ids they're always
going to be at the top
and again it seems to be doing these
stem
is it
yeah it's doing it's doing auth checks
my memcache is not working and that's
weird
but
hey i guess so if you're gonna have a
demo failure it's good to see an error
that i could see is wrong straight away
um i guess that's the whole point of
tracing
and if that's the only problem we're
going to get then hey that's great in a
demo
uh so we got we're connecting to one two
three four so only if it's connected to
four to be able to rebuild this object
also it doesn't have to update anything
better data lakes we're just getting the
object we're not like adding or deleting
or doing like a
create
remove
operation
okay so that's all great that's all
exciting um
but let's do some more interesting ones
i should probably do a time check uh
yeah it's good
so
let's go back to the slides i don't know
why my that slide didn't come back where
are you
have i lost my slides
not there they are they're just not
listening to me
let's look at some more interesting
features of swift now so
first one let's do a bulk insert so bulk
insert is a interesting
middleware it allows you to or feature i
guess to say what we know implement as a
middleware
it allows you to upload something like
a tar file
excuse me and
it'll extract it
and store all the contents objects
inside your cluster now this is kind of
cool um it's very serial
it's cool because it's kind of cool the
con the the con is it's
it's serial and so if speed's an issue
you wouldn't use this feature but if you
were wanting something to run in the
background and you're just doing it for
like you know archival purposes then
it's kind of cool so let's see what this
looks like in um
while we trace it
so in this directory
i'm just going to type it
i've got a to
extract.gz and it has
a subfolder called extract and it's got
a bunch for four
files in it some text to monorail cat
dyno and whatever what some rather
random random text files because i'm
lazy again
so what we expect to happen is it will
go and create a container called extract
and then inside that container it'll
have all these objects which is kind of
neat if it works
and i have
bulk insert i've got it here if i can
copy it
and i can paste it so here's the command
we'll do it we're saying extract archive
it's a taji zed file so you're saying
here's a taji zed and go do your thing
and i'm using curl instead of any other
things because you can see you can
easily talk to swift
so we can see here that it was
successful
we got a transaction id
um
it created four files
the response code was 201 and there were
no errors so that's great
so if i come back here
there's 352 spans so we know it did a
lot we can see all the things that talk
different servers that talk to they're
all local on this laptop on this virtual
environment
but we can we can see right we can see
that it's one of these should be go and
create that container that extracted
container if i scroll down to it
it's uh put to extract yeah so we're
we're making a uh we're putting the
extract container
that means these guys which one would be
our monorail cut so it would have been
the second one i think right so let's go
look at this one this one in theory
where or if it did it in serially like i
think it does
this should be monorailcat
and it is it's monorailcat and so
you can see from here that it's a pretty
cool feature it uh it will go
it starts extracting it and then starts
making newer kind of sub requests out to
all the clusters and and each of them uh
because
because i just put it into
by default my default policy is
replication is the replicated ring um
it is
only three times replication it's not
erasure coding it you could obviously
point it to a ratio uh you make it
create a razor coded container uh to
store objects in
okay
another time check for me okay we're
getting we're getting toward the end um
there's lots more i could talk about
this for hours there's lots more to talk
i could talk about
but
and you asked the guys at the ptg i just
kept on going
but if we go to the next one what did i
go too far i don't know if i think it's
listening to me
no no i don't have a conclusion go back
there so another interesting feature
from the other side of things is
static large objects so the idea here is
like mtu um multi-part i'd multi-part
things in in aws
so you go you have your object you can
divide it up into what we call segments
and you upload them separately so again
hit different swift proxies you can do
concurrently you can do it quick if it's
a really large item you can cut into
thousands
uh if it's or if you just want if it's a
smaller item but you just want to upload
it quicker then this is the way you
could do it um and then it used to slide
last it'll send a json manifest uh to
kind of enter to stitch it all together
so when you go get that json manifest it
will
it will then be able to pull these and
stitches back together for you um so we
can see that in action now
i'm going to use this this time i'm not
going to use curl because it's a little
bit more um
involved so i'll just let the swift
client do its thing
so this command is saying
swift upload use slo
uh what's that 5ma about five mega
chunks
to my 10 megabyte image which is i think
i think i did it by you random so
nothing useful
um
and it uploaded three segments i'm
assuming it's five five-ish five-ish in
a little bit um and that last so so it's
made one two three requests and the last
request should have been inserting
manifest which has the name
10 mb dot image because that's what i
told it i wanted to be called
so if we go look at this in our trace
it won't be super interesting because
it's going to be a bunch of extra new
requests because it's inserting
one each time and we've seen inserting
an object the last one we'll confirm
that the last one is in fact um actually
i'll just look at the top here
is it is in fact uh
sending in the manifest file
well it's more this is my favorite bit
so what's more interesting uh is if we
try and get that object back out now
um
i'll just use the good old cut and paste
because i don't think you want 10
megabytes of random crap on my screen
or on the dev
so we downloaded it i don't know what
the
um
transaction id was but turns out it
doesn't matter
here it is
it doesn't seem like much but look at it
it's a bit of a crazy um
up here is the full
kind of time separated
uh
request
and it's interesting right like so what
it's done is it's gone to do a get of
the manifest this should be a get of the
manifest file
yep good i'm good i'm i do know my stuff
that's good um and then it's starting to
build so what so what whiskey does let
me see the time check again cool so what
whisky does
is it
will send back
um the response headers and stuff and it
will wrap up the body in a file like
iterator and
so it will send back
uh all that to the client in this case
curl um and then curl will start you
know reading the body uh to the body of
the request
so we can see it is
getting that manifest file at this point
it's actually probably going to go get
that first segment
just getting that iterator ready the
file like iterator ready
yep it's getting segment zero i'm
pointing at a screen that you can't see
me point at use the mouse mat use the
mouse um
zero first one zero based because it's
programming
because it's i t
and then there's this big gaps this gap
is caused by
curl so as curl reads
um it's it as it's dealt with the first
segment it reads and then gets a second
segment and we can actually see this
happening and it's kind of really cool
well i think it's really cool i get
excited about this stuff i don't know if
you guys do
um
i definitely do
let me i can do other things let me see
how time goes i think we've probably
only got what five minutes
before i want to try and stop for
questions
so
let me
finish the last few slides of the
presentation and if we want and if
anyone's interested i can go because
i've shown you all the happy path things
right we can also go in and like go kill
a server and then do requests and see
what the traces look like this is all
the fun stuff because when
just be honest when you're like doing a
trace request you're not tracing the
happy path you tend to be tracing
because an issue and we can try and
demonstrate that stuff
only more than 45 minutes
so
conclusion
i wrote this up about an hour ago so
it's a bit more verbose um
being able to visualize a request
at least to me has been amazing like in
my i've been involved in swift for a
long time i know how the code works and
you know build up my mental model i know
what's happening but to see it's like
seeing your baby or maybe not that far
but it's like
it's being able to see it it's actually
been really exciting for me uh if you
can't tell um
and i i also think that the human
ability to patent match means we tend to
spot potentially quickly
i i guess when i saw it before right i
saw that the things are not being stored
in memcache like they're supposed to be
so
we saw an issue
um but that only comes down to you need
to know what normal looks like so
there is going to be a level of having
to train and this is where our sres are
at in nvidia now that we've got some dev
environments we haven't rolled out to
full production yet um it's more in some
of our staging environments so we can
get an idea of what these things look
like and and get it integrated into our
um into our
manner what's the word
i can't think of it our pro stack our
monitoring there we go that's what i'm
looking for our monitoring stack
um
but so it first is learning what normal
looks like because if you don't know
what normal looks like then how you know
the pattern match doesn't work
can we see we can see what happened in
request rather than hypothesize maybe
maybe you guys are good at an sres and
you
sis ads out there are amazing at being
able to figure out what happened from
logs
and i struggle sometimes but being able
to actually look at what happened at the
time and being able to pull in
information and yes we're not pulling
all the information yet but there's the
code there and in that mix in there we
can continue to instrument the parts
that we need to to pull out the
information we need um
and just a nice have another way of
pulling out information uh get back to
those error limited nodes in the swift
proxy server
we do dump those to logging if you if
you turn your debugging up to debug but
in production we don't want to do that
especially on really really really large
clusters um
so being able to have a way of pulling
that data out it's kind of exciting and
interesting
there are a few interesting problems
that we had
along the way
um this is maybe more for people who are
path and developers who have had to deal
with eventlet and green threads and
stuff but weirdly i told you that the
the transat that
come on brain the um tracer
was in charge of keeping track of the
active span so anyway what i'd find was
once we get into an event like um thread
pool and now at the other side so it's
in an instant threading environment
it just had lost its um
active span from inside the tracer so we
had to like figure out ways of pinning
that in
um because
you know
suddenly tracing doesn't work and that's
not really useful um response iterators
that last one you saw with the slo
static clutch object getting what also
tends to happen is because whiskey kind
of
sends back the response with that with
that file like iterator it decides that
it kind of cleans up all the context all
the span contexts and stuff and so they
don't try and exist when we're um
when we're still
making new calls to the next segment for
example and so the same same kind of
solution to the green thread turns
eventlet seemed to here too it's just
kind of keeping those things alive
making sure we've got them pinned so
they don't just disappear when their
context
uh ends
um well by default um
and yeah because the other thing is
request tracing has been interesting
it's um
whiskey tends to have a lot of these
middlewares in memory and our request
coming through is really like is really
an event dict dictionary that kind of
works its way around environment
dictionary and because we're trying to
keep the scope to of tracing to a single
request
it means the kind of tracing mechanics
had to to some extent while the state of
them had to live inside these dicks and
that there's like
it's just interesting it going to
interesting problems because we didn't
want issues to bleed into other issues
um anyway i think i
am running out of time so i'll end this
here well actually i'll uh give you
these things if you want to contact me
uh email me at nvidia at my thing there
or at
uh oliver.edu again i'm on matt oliver
you on all the socials if you want to
come check out the swift project we all
hang out on ifc
not not quite the cool kids anymore i
think
but anyway we're on openstack swift on
oftc
there's our project website
swift.openstack.org
there is a link to the contributing
guide you should come and have fun with
us
um and there's some interesting links
there's the link to this code in garrett
as it stands and that google doc that
will
get you bootstrapped into doing exactly
what my system's doing right here if you
don't have a play
i'm really looking forward to the google
doc link in
oh yeah well i'll share this yeah yeah
it's not great maybe i should have like
shortened it but you know i don't think
about that until you can put the slides
into the text chat afterwards and okay
yes i'll do that i'll um i'll do
definitely do that
uh so that's it any questions
actually what time do we have we do have
a few questions we've got
two and a half minutes to talk right
um
yeah
so the first question that we've got is
open tracing is only application
specific does it use any tracing
mechanism from linux kernel
no it's just application specific it's
just
the idea is it's just an api that would
be cool though actually like if we could
do some tracing down the kernel level
i would like that too because there's
sometimes the way you know some of our
swift issues are not always
application specific right when we in
nvidia are looking at
how can we speed up transfers how can we
talk directly to disks can we do
drmaying and all this kind of fun stuff
so being able to kind of get under the
hood a bit more would be great but no
this is at the moment at least as far as
i'm aware it's um it's very application
specific
cool uh another question we have is what
overhead have you seen having tracing
going would you leave it on
uh this is great um because this is the
question i would have asked myself so
here we go basic benchmarks now you take
this with a grain of salt because this
is just on this death machine and i did
it during lca right so because that's
the way we do things in lca um so it's
very basic this is pretty much just a
thousand get puts and deletes it's
um just one k
object so take of the grain of salt but
the whole point is a similar environment
one there's no tracing no tracing at all
no code this it's like master without
this patch applied then we've got
tracing off that is the red one it is
got no tr it's got all the tracing code
enabled but it does not have not tracing
anything and tracing on is not how you
would ever run this thing it's tracing
every single request but you know we
want to test this stuff right and so
tracing on is the code with tracing
turned on every request as expected when
you got tracing on it gives us a
performance here
but
it seems at least in this small one and
we will be testing this um
as we before we roll it in production
too as we like test and i have bigger
bigger staging systems just to confirm
but in all my tests so far it it has
been negligible in a lot of the coding
if you have a look at the code
i've gone to great f efforts to try and
make any of the tracing up stuff if it's
not tracing as no op as possible
um so i didn't want to affect our spin
so
our latency distribution so we've got a
thousand puts we got a thousand uh
support yeah a thousand puts um and most
of them
were identical to no tracing um there's
slight differences as we can see so it
does have a does have it will have a
marginal effect as you'd expect um the
ops per second
we can see that
tracing on well yeah it's never going to
be great right um but tracing off for no
tracing of fairly similar i'd like to do
a bigger scale of this but thanks to
have asked that question because it's
exactly the question that i was thinking
which is why i preempted
does that answer that question well the
best i can answer that question right
now
is there anything else
uh you're muted
i'm muted i'm so sorry
uh i've got a couple more questions that
i'll jump into the text chat uh in the
press talk chat for kia ora
and then i will uh
yeah we've got afternoon tea now thank
you so much for your talk matthew
um
green great to have you
and we'll have see everyone back at 3 45
pm aedt for sam bishop with boldly going
running linux in space
awesome
thanks everyone i'll be in the chat