[Music]
welcome back everybody from morning tea
i hope you've had a great snack time to
catch up with people and have some
refreshment
apologies this is my first time emceeing
and it all happened within the last 10
minutes so this is really agile software
development on the fly so apologies if i
put my foot in my mouth so this morning
i'd like to introduce brian who's coming
to us from sony ontario in canada
currently -30 outside for all of those
of you in sydney that are suffering from
30 plus or from the west coast of
australia and it's even hotter just
think of them in canada minus 30. i
think our sunny australian coasts are a
little warmer than canada at this time
so brian uh is joining us to so we can
learn about how hootsuite used load
testing to evaluate the performance of
prometheus and grafana based metric
platforms
developing a framework based on open
source tooling locust that can be used
to quickly evaluate the performance
impact of configuration and component
changes prior to deployment now brian's
talk is pre-recorded in case we have any
technical glitches across the pacific
but brian will be joining us again for a
q a at the end so brian will be live
that is him there live at the moment
nodding along it's not just a naughty
and those plants in the background are
real they're not even a virtual
background so we really really have him
here
uh thanks enjoy the session there'll be
a few seconds pause as we switch over to
the recording
see you soon
welcome to the inaugural voyage of
benchmarking prometheus metrics
platforms my name is brian gru and i'm a
senior software developer at hootsuite
and i mainly focus on observability
so some background we implemented a
prometheus thanos grafana stack to
support multi-window multi-burn rate
alerting for most of our product
software but also for some of our
infrastructure as well
after the initial implementation
we received reports from developers that
their dashboards were loading slowly and
while we could load these dashboards and
see that some were slow others were
performing fine so we really didn't have
an idea as to the extent of the problem
we really had no empirical data
to prove that things were actually
running slow or how many things were
running slow
so
we set out and we thought to ourselves
how can we verify and also tune the
performance of the current metrics
platform we have but how could we also
possibly benchmark other
prometheus-based solutions
and
how so our answer was basically we want
to performance to start infrastructure
so you may be asking yourself why
performance test your infrastructure
it's our belief that infrastructure
critical to assisting product
development should be just about as
critical as the product itself
it also helps you understand baseline
performance it helps you validate
changes against existing baselines
before changes actually get integrated
it helps you watch for regressions in
the system itself
and also to allow you to be proactive so
if you're running this in an automated
fashion or manually on a cadence you can
catch problems before your users
actually start complaining
first we set up to figure out how to
model these tests correctly
so generally speaking we want to
understand end user performance and we
want to understand end user performance
from the perspective of the client that
our users are accessing to
query the metrics platform so more
specifically we were looking to assess
the performance of
our prometheus or thanos platform which
was accessed primarily through grafana
dashboards so to this end we thought we
would actually model the performance
test themselves after
actual graffana dashboard loads
so the first question became how is
grafana loading dashboards and chrome
developer tools to the rescue here uh we
basically loaded a dashboard and watched
and saw what was happening so
unsurprisingly grafana is making http
requests to its own back end and then it
would proxy these requests over to the
prometheus or thanos backend and
their request path was basically the
same as what you would throw on a
prometheus api request so it's easy to
copy pasta
the next question became what are these
requests actually doing and again
wearing chrome developer tools so first
we could see that template variable
queries uh were being run and that
there's actually dependency resolution
between the variables so if you had
multiple template variables uh grafana
would actually figure out the order in
which to make those requests and then
kind of request them top down uh and
next grafana was loading uh panel
queries and it would start by requesting
all visible panels concurrently and then
as you scrolled it would request the
other panel queries as those panels
actually became visible
so the question became do we have to
replicate this complex behavior
precisely and my thought was no you just
need to determine the best way to
approximate the client's behavior in a
way that your
performance tests
still make sense and for this use case
we decided to approximate the behavior i
just talked about by first serially
loading all the variable queries and
then loading all the panel queries in
parallel and this would kind of provide
us a uh worst case scenario for loading
a dashboard and that all panels were
actually loading at the same time are
visible at the same time
so next the question was what
specific request should we actually be
making and as i said prior we wanted to
simulate the user behavior so we came to
the conclusion that no one knows how to
break our stuff better than our own
users or our developers
so we should actually model these tests
based on existing dashboards and the
alternative would have been just to
generate some synthetic load perhaps
requests that we know are problematic
but maybe not as true to form as
behaving like a user
so
the last question was what is grafana
actually sending in these requests and
surprise surprise it's it's prom ql with
the variables evaluated and it's like
wait what more variables yes uh so there
are template variables in here that we
had already kind of talked about uh and
these are either static or or generated
dynamically from like a prom ql query
and there's also built-in variables
things like range and underscore
so how did we decide to handle variable
evaluation well we decided for the
built-in variables that we just set some
same defaults to things like interval
for five minutes and you know range was
obviously going to be calculated based
off the start and end dates uh we
decided for the query based template
variables uh that we would actually
still make those requests to prometheus
or thanos
and that uh and that this would help
better mimic uh the dashboard load
however we just we also decided that uh
you know uh we would use the template
variable default values in the actual
panel queries themselves as these were
the default queries with those default
values for those variables that were
actually run when the dashboard came up
and decide we didn't have a great way to
say
which value we should actually be
substituting anyway other than the
default
so to kind of recap our modeling section
we decided to make multiple http
requests to our metrics back end to
replicate the load
we are going to serially request all
template variables first then move on to
concurrently request all the panel
queries
we we're going to model the request
order and content based on existing
dashboards that we already had and we're
going to set sane built-in and template
variable
default values but we're still going to
make those variable requests the
variable query request to the back end
uh so before we actually set out to do
this we wanted to come up with a testing
methodology and also some boundaries for
our tests so in the center here you see
a standard software development life
cycle that everyone should be familiar
with and we thought we could apply this
to our performance testing as well and
that we could come up with a hypothesis
you know
this thing is slow because of this or
that and then we could come up with a
design of how to prove or disprove that
hypothesis maybe we can throw more
resources at it or maybe we can
tweak configuration in some manner
then we'd actually make those changes
once we figured out what to do
then we would run a standardized test
suite to verify and benchmark those
changes and then the results from the
test suite could feed back into a
subsequent hypothesis
so just to kind of recap some of the
tenets of this methodology is that it's
systematic in that it provides us a
framework to develop and prove out these
hypotheses and it's consistent in that
the same tests are on on every iteration
which actually allows every
run to be compared to previous runs and
it's data driven test feedback the test
data actually feeds back into the
decision making process
and in addition to the test data itself
external data can also be integrated
into the analysis phase so if you have
metrics from somewhere else or logs or
traces or whatever else you may have
you can use that to influence your
analysis phase as well and lastly the
methodology is iterative it results in
this kind of tight loop
where changes can be evaluated quickly
and hypothesis can kind of evolve with
each iteration
which hopefully leads to an increase in
velocity
so just a quick note about data validity
here uh a data-driven process like this
one is really only as good as its input
data
so we started thinking about what
happens if during one of these tests if
a service goes offline or if it
temporarily slows down
let's say because someone else is using
the service and the testing environment
so we actually had decided to smooth out
the data for the tests by running the
tests multiple times and then using an
aggregate of the result instead of just
running the test once and using that in
case uh the test run happened to be uh
botched or influenced by external
factors
so lastly we needed to decide on some
boundaries for our tests our we found
our longest common uh use use case for
dashboard was basically about 30 days
and we noticed grafana didn't have many
concurrent users so we decided to cap
the maximum number of virtual users we
simulate to about 10.
uh then we had decided uh that we would
run each kind of test suite for user
loads of one five and ten and each one
of those for one day three days seven
days and 30 day windows so
each one of these combinations would
actually be ran
10 times to smooth out the data and just
kind of
ignore any external factors and maybe
influencing the test to kind of smooth
that out
so now we had a methodology we wanted to
come up with a testing platform and a
design for our tests
so a little note on technology selection
uh we needed a tool uh for load testing
that could make http requests and
there's lots out there uh we also needed
to simulate multiple users so many load
testing tools kind of fall into this
into those requirements and locus had
actually been used in the past at
hootsuite uh for load testing different
things and we'd actually used it for low
testing individual prometheus uh servers
and time had also been invested in the
tool to make sure it ran on kubernetes
for us and some other tweaks here and
there so given all this we decided to
move forward
with locust
so to bring forward some of our
requirements from the modeling and
methodology sections and how these
implements our tool
we needed to seriously make http
requests for these template variables
which is met by the default locust http
client and we needed to concur
concurrently request
all the panel queries which is something
locus does not handle we also needed to
execute requests for multiple dashboards
multiple times and provide an aggregate
of the results
which isn't really met by locus but it's
something we'll address in our test
design in a minute and we also needed to
expose data to feedback into the
analysis phase of our testing loop which
was met by locust as it has this web ui
that allows you to export csvs and while
it's not terribly automated it worked
for us where we could just take that
data and dump it into another uh sync
like
google sheets for instance is what we
used for doing most of our comparisons
so the first step was adding concurrency
into locust and
locus itself is python based both the
tool and the authoring and as uh people
who are familiar with python no
concurrency isn't necessarily known to
be one of its strong suits but we did
find this aio async io http client to be
the kind of recommended solution uh for
doing this thing so we proceeded to
implement a locus client based on this
package
it did take some time to kind of figure
out uh the exception and error handling
uh with async io
so we weren't doing things like
exploding the event loop or swallowing
exceptions in the event loop and not
passing those back to locusts so they
could actually be
reported
to the
error statistics that are displayed on
these dashboards as well
also locus that can run n virtual users
per node essentially and we never really
figured out how to get this working with
async io
just by trying things like creating an
event loop per python process or per
async io client
so the client overview i mean the idea
here was to expose them expose a method
to make a list of requests concurrently
but we thought it would also make sense
to encapsulate a base http client so the
consumers can request both serially and
concurrently with the same client
it also as i said prior like it creates
this new
async i o event loop per client instance
and we're still not sure this is the
right approaches we never quite got it
working
the async request mechanism itself it
passes back a list of these custom
resolved objects so if
an exception does actually occur during
the request they are caught and they're
passed back inside this result object
which allows our handling code on the
other side of the event loop to tie that
back into locusts error reporting
a little overview of the test layout uh
so as you can see here at the top we
have a uh
a encapsulating class locus user tasks
and inside that there is uh two of these
dashboard one and dashboard two tasks
both uh are a sequential task set which
is a locust construct from the api and
you'll notice here that the
the embedded classes dashboard one
dashboard two
they would declare this kind of tasks
variable
which is something that's part of locust
api and this will define
the
sequence of the tasks that are executed
and in the case of the dashboards
themselves
these task variables are set to methods
right so they load the variables they
load the panels and then they stop the
dashboard load and dashboard 2 would do
the same thing it would load the
variables like the panels and stop the
dashboard load and then you can see for
the locus user
tasks outer class that there's also a
task variable declared and that's
basically deferring to the inner classes
so it wants to load dashboard one and
dashboard two and then to repeat this
for the send times you just kind of
repeat that for as many times as you
want to run dashboard one dashboard two
dashboard one dashboard 1.42 etc and
then at the end you just say stop test
again and stop doesn't stop dash
they're not shown here but they're
really just methods they call an
interrupt on the locus test itself to
stop things from executing over and over
so now we knew what we wanted to do we
actually had to implement these things
so to bring forward some of our
requirements from modeling methodology
and the platform itself the tests had to
be written in python for locust the
tests must be modeled based on existing
dashboards
the test should be consistent between
runs so that they're fully repeatable
and some other considerations we had was
that it would be great to increase the
velocity for operators if the test could
easily be updated
so what we came up with was to actually
generate
the tests themselves from
grafana dashboard json and we already
had some experience in the team with the
graphone dashboard json model so that
lent itself well
this also allowed us to
model the tests after the dashboards a
little more precisely and it enables a
quick adjustment to the tests so if you
needed to tweak something you could just
tweak it and rerun the generator and we
have a bunch of new tasks coming on the
other side and the nice thing is that it
generates a consistent test suite every
time you don't really have to worry
about user error as long as the input's
good the output should be good as well
so the overview of our implementation of
the generator itself we identified some
key dashboards in grafana
things that were slow things were fast
things that had different kinds of
panels things that had
lots of template variables things that
had no no template variables etc i think
overall we probably identify close to a
dozen or so and we pulled these via
the grafana
rest api and uh we actually ended up
saving a copy of these in the generator
repo and that was mainly due to the fact
that if somebody edited the frontal
dashboard upstream we didn't want to
have our
tests change on us we needed them to
remain consistent
so we actually saved this copy like the
copy of all these dashboards uh the json
into our
tester repo to kind of codify them
so we wrote this python script to parse
that json and it extracts the template
variables and the panel queries and it
performs that variable substitution
and it actually uses this data to
template the tests uh with jinja which
are just python test templates written
in jinja 2
they in this template would really
define the structure for each test right
for each dashboard uh
handle the variable loads handle the
panel loads and uh these templates uh
this python that that was templated it
mainly used uh the new async locus
client to make all those requests
and
from the output of this whole system was
a bunch of locus tests that are all
valid python and they're ready to run
and we would actually
we ended up generating uh one test per
query window so one day three days seven
day and thirty days and this was really
just for us i mean it the multiple tests
it allowed us to test each query window
distinctly
but we could load all the tests at once
so we could rerun a specific query
window or something if you want to
without having to rerun everything and
again for us just logistically this made
more sense
so
uh now we were just about ready to go we
need to figure out what kind of data we
wanted to get out of these for tests uh
these tests and kind of what we wanted
to compare on
uh so it came down to choosing some
meaningful indicators so error errors to
us are always meaningful they help track
your failures under load more data is
always better than less so we actually
chose to record basically everything
that locus provided in our google sheet
but we'd really only compare on
uh on error rates uh your median your 90
and 99th percentile so just to dig in
there a little more error rates they're
not only helpful to understand what's
currently failing
but they help you understand the point
at which your system starts to fail so
you can actually increase load uh using
your load testing tool until you until
uh you're either content with the
performance under load or things start
to fail
the next thing is quantiles these are
extremely useful in the observability
world just that most latency based slos
are already expressed as a percentage so
for example you might have something
like 99 of your requests complete within
x seconds
and like the quantiles
chosen they should be meaningful to that
slo
but they should also be
wide enough and varied to kind of help
understand
the spread or how frequent those slow
requests are
so for example your p50 or your median i
mean half year requests are basically
completing in this time
so if this is really good or acceptable
performance then you're kind of already
doing okay
uh at your 90th percentile only 10 of
the requests are actually slower than
this so is this is this performance
acceptable for
the majority of users who are accessing
your system and you know uh the 99th
percentile just one percent of your
requests will actually be slower than
this so is this a tolerable edge case
and kind of how does this align to your
slo if your slo was 99
uh for that dnc
um so now we built this thing and we
needed to turn it loose so our first
step was to identify
the current performance of the system we
already had and this is what it looked
like
so 30 day windows
you could see were pretty horrendous and
added load really only exacerbated the
issue
and uh 10 user 30 day windows as you can
see here there's actually no data for
those
they are timing out
except for the p50
so clearly things
weren't great
and just note here the y-axis and all
these graphs
this is the time in milliseconds for the
entire test suite to run
so iteration 2
we thought hey maybe other metrics
platforms are faster and thanos is the
problem it couldn't possibly be us
so this is what happened we implemented
a proof of concept for amazon managed
prometheus which you see here is amp and
we implemented a proof of concept for
prom scale and we ran those against our
current uh thanos system
so as you can see here from the error
rates uh slide that as a user range
increase so did the error rates in
general uh most of the solutions saw
nowhere little error rates at the
shorter smaller ranges and uh virtual
users and prom scale had error rates
kind of thrift
and uh
amazon managed prometheus and thanos
tracked very similarly although
uh the management solution was slightly
ahead
uh in terms of performance we found
extremely similar trends uh between the
platforms for the most part as user
range increased uh performance basically
became unacceptable again
so the question really became why do
none of these platforms actually work
well for us
um so the conclusion we came to is
basically that given that amp which i
believe is cortex-based
and prom scale and thanos all degraded
pretty severely approaching 30-day query
windows
uh
we came to that conclusion that our data
was likely suspect and that
the way we were
recording metrics uh was not lending
itself well to the way prometheus tsdb
or thanos or something else worked and a
future investigation would actually
confirm this
in that our certain series had major
cardinality issues and it was something
to the tune of less than one percent of
our metric series by name accounted for
almost 50 percent of the total samples
uh per scrape basically um but in the
meantime until we got to that
investigation uh we decided to actually
use this testing platform to tune thanos
itself
um so this was really iterations three
and on uh we were benchmarking changes
to the existing platform
um so uh the tuning we focused on 10
user and 30-day query ranges as this was
our worst case and you know if we could
make our worst case better than our best
cases only got even better right
so we decided on a potential bottleneck
we investigated how to solve that via
configure resource options and
implemented and this is a tight loop
again we talked about and we went
through about 17 iterations of these in
totals
and every time we went through an
iteration if any increase to the
performance or
or or decrease in error rate was found
we actually implemented that as the
iterations continued
and if you look at the graph here uh
what you can see is that against our
baseline our tuned
thanos actually performed
much better
than what we had seen previously
and not only was performance up uh we
were able to get error rates down from
about 21
to one percent
so uh summary of the changes uh for
those that are interested things that
had a positive impact were increasing
resources surprise surprise like memory
and cpu specifically for uh storing
query pods uh thanos store we actually
changed that from being a time based uh
partitioning scheme to hash base which
is just a configuration option and this
led to some better query distribution so
the thanos store or store instances that
were responsible for the most commonly
queried ranges didn't always get
obliterated uh we also increased the
dano's storage erpc concurrency just so
more things could get through the door
at once we moved from an in cluster
memcached to an elastic managed
memcached instance and you know this
gave us kind of benefits of a managed
service but also it gave us access to
much larger much larger nodes without
having to worry about resource
constraints inside kubernetes and
and working on nodes where everybody
else is also running workloads
so we also tuned thanos store index
cache to increase the timeout increase
async concurrency and increase the
buffer size but we actually decreased
the max number of auto connections just
so things would get booted a little more
aggressively we also enabled thanos
store caching bucket which actually just
speeds up chunk loading
and things that we had tried that really
didn't work out for us
we tried to overly shard thanos store
and we basically found it had
diminishing returns uh after a certain
point uh we tried thanos star
store shard replicas
but we really found no benefit running
that with kubernetes in in just that
like if a
store shard goes down kubernetes
replaces that pod pretty quickly and the
way this worked was that when you
queried it would actually query both it
actually query across both replicas at
the same time so there wasn't any real
performance gained from this
and also we had tried thanos store in
memory index caching but it just put too
much memory pressure on our kubernetes
nodes
so wrapping it all up the tldr slide is
that we identified a lack of visibility
into the performance of our systems we
ended up designing and implementing this
performance testing framework
and we established baseline performance
and we also evaluated other solutions
with that framework
this allowed us to draw then later prove
out some conclusions from our test
results and using that same framework in
a tight loop fashion we were able to
successfully tune our thanos and
prometheus instance
so for more information i'm going to
make the slides the locust aio http
client and the grafana test generator
available at the github repo you can see
there
and if you want to contact me or if you
have questions or you want to share some
opinions
my email is on the screen there and also
you can hit me up at the cncf for
grafana slack and my handle is just
brian grew
so uh thank you everybody for
sitting through the presentation and now
we will move on to
q a
thanks so much
hey welcome back everybody brian thanks
for an amazing talk and the wonderful
memes um i didn't know what hootsuit was
before your recording so thanks so much
for your uh doing that for us very
enjoyable
so brian we've got a couple of questions
that have come through in the chat um
the first one's a double question the
user behavior was codified as python and
locus
go ahead sorry yeah no
if you want to finish reading the second
half that's fine sure and the second
half is do the number of developers
affect the number of load test users
were implemented right uh so yeah uh the
first half we did choose uh several
dashboards and uh the user behavior of
loading those dashboards or what it
would look like coming from grafana
that was definitely codified into uh the
tests we wrote and it did impact uh the
number of users right i mean that we
found we had relatively low uh
simultaneous users on griffon at any one
given time
so we decided to uh max out at about uh
10 virtual users for locust
um
and then uh
and then i think yeah like we were
trying to determine i mean why the
performance was slow and not try to find
the breaking point of the system so i
mean that's really the reason we didn't
uh ratchet up the number of virtual
users too high
excellent thanks brian and team please
do continue to put chats questions
through into the the chat the great av
guys etc in the background and the
moderators will post them through to us
so please keep more questions coming
we've got about 15 minutes we'll finish
at
uh
12 25 australian eastern standard time
on the east coast for sydney apologies i
can't work out the rest of the six or
seven times zone in australia on the fly
but hopefully you can so we've got about
15 minutes left before we go to lunch
the second question is
what's
sorry that's the most common type sorry
there should be what's the most common
type of metrics you have seen people
overlook
what that would add value to their
polling and graphing
uh yeah for sure so i've been racking my
brain over this one a bit uh
but i mean in terms of general metrics
for services or applications i mean it's
kind of hard to say because everyone's
stack is so different right
but in general i do find a lot of people
they tend to focus on uh resources right
where's my memory at where's my cpu and
i mean that is very important uh but i
always like to think of things in terms
of user experience and uh whether it's a
real user out there in the world that's
external
or
whether it's like another service that's
using it or like for our use case where
we were uh providing grafana to our
eu for our users and i mean in a lot of
these cases
like latency does come in handy and as i
said in the talk as well and
and not just running with an average or
maxis but actually understanding
uh the quantiles and kind of how that
aligns that's low and how many users are
being impacted uh through those
quantiles
but i think something that's also
generally overlooked is that like the
impact of downstream or managed uh
services your service may be dependent
on and you know and remembering to take
their performance and their status into
account
both when you're performing you're
monitoring your writing dashboards but
also when you're coming up with
dslos for your service as well
i'm not getting any audio from you
michael
thank you i fell for the oldest trap in
the book
sorry the next question is and apologize
for my pronunciations here greek was not
my strength what is the benefit of
thanos over prometheus
uh yeah for sure um so the reason why
hootsuite went with uh thanos is that
we had a requirement that we wanted to
provide our users with uh up to a year
or so of of metrics and
we necessarily didn't want to keep all
those metrics live and prometheus and
have a massive
tsdb
so the benefit for us uh was really
implementing thanos as that uh long-term
store uh but thanos uh and and the other
side of it is also uh the uh high
availability side right uh we run uh
multiple uh prometheus or promethei
pairs uh and thanos uh provides the
query across that pair uh the aj pair
prometheus instances and it will also uh
detube the results on the other side
through the query mechanism
so
it it kind of bulks out prometheus where
maybe vanilla prometheus falls a bit
short in terms of h.a or uh long-term
storage and uh long-term storage on
thanos is uh can can be sampled down as
well
excellent thank you
well that's the last formal question
please do ask more questions they did
foolishly say that i'm allowed to add
live which is always very dangerous
so
my ad-lib question is what was the
biggest surprise you found when running
your first test
uh yeah so the biggest surprise was just
uh
how absolutely horrific the performance
on querying some of these series were
and uh and it became pretty obvious uh
when we started running these tests
manually to begin with uh just like
just testing if stuff worked uh
uh it became pretty obvious who the
worst defenders were and uh
like in terms of the series right like
not the people but yeah but it became
pretty obvious who the worst uh metric
series were so when we did perform
subsequent investigations it kind of
gave us the okay i i kind of know where
the bodies are buried i know where to
look
excellent and what was the biggest waste
of time like you spent all of this time
coding something up and it didn't
actually show any in performance and
what could you learn from that
right uh for sure so i mean
uh like are you asking the biggest waste
of time that we've hit or
yeah it doesn't you thought you would
find value in a space and you didn't
actually find any value uh yeah that's a
good question i mean in terms of the
overall uh system and the approach which
using locust uh for performance and load
testing
we've found value there i don't think
we've fallen down anywhere
um
i think the only thing uh
that's maybe fallen slightly short is
just uh
now that this has worked so well for us
we've been trying to figure out how to
scale it uh to the rest of the team and
you know we're trying to figure out if
locust is that tool to do that going
forward and we're actually not sure um
so if you want to talk about perhaps
burning time then yeah we put a lot of
time into this and into locust and you
know uh we may be starting over with
something that's uh slightly more
scalable or that we can relate to the
whole org
excellent thank you as an ex-software
tester i never found testing or
performance testing was a waste of time
even though the other people on the
other side of the quad sometimes
wondered what we were doing on ourselves
excellent and what would be your one key
takeaway if you if someone walked in and
say what should i learn take away from
this talk right
yeah yeah like i think the
best thing we've got out of the system
uh
was definitely uh
tuning thanos and just how easy it was
to uh make a change run the unit or
sorry make a change and run the
performance test and make a change and
run the test again and then actually see
those uh benefits or detriments that
sometimes happened as as as we went
along so i mean the one takeaway is just
say yes uh performance testing is useful
even in infrastructure even if it's not
an end user facing product uh your users
the developers at the end of the day
well thank you
good good excellent
that long that dying between the
developer and the end user is sometimes
a little blurred with management and
cost of performance testing and
continuous integration so yeah i'm a
great believer in it and as a manager
i'm encouraging it so yes wonderful
excellent
good good
um i think that's all the questions
we've had come through
we might give people an early mark for
lunch if they like you'll get a few
minutes early but brian will be heading
across to the uh the chat in the venulis
if you haven't found the channels yet so
this is my second lca took me a while to
find them scroll down on the left go to
browse channels and then you'll find a
whole pile of channels to join and if
that's why you missed out on some other
things sorry that um you didn't hear
about that earlier but please do join
brian and brian a great thank you thank
you for taking your time your evening
thanks your your family and plants as
well for the after hours work you're
having to do in canada and please do
stay warm over there
great thanks so much
thanks bye everyone enjoy lunch
you