welcome back everyone we're ready for
the first talk of the day here in the
devops track
and the first one we have today is an
amazing talk by jonathan hall about how
a single bit
changed his career i've been looking
forward to this talk for a while and i
can hardly wait to hear what he has to
tell us
so without further ado let's kick off
the devops track we talked by jonathan
hall
hi everyone i'm jonathan hall thank you
for attending my presentation
uh i hope you enjoy it let me jump over
to the slides and we'll get started
so as you all saw from the program the
title of my talk of course is the
butterfly effect uh how a single bit
changed my career
my agenda for this talk of course is uh
to tell you the story of how i
uh sort of started my career and in
software delivery
i want to encourage you on your own path
in that regard to whatever extent you
were involved in software delivery which
i expect
most of you are as you're probably
developers of some sort
and then maybe give you some pointers on
how you can improve in some of these
areas if that's interesting for you
of course my name is jonathan hall and
uh i am uh the tiny devops guy if you
google for that you'll find me
i have a podcast by that name
and uh i do devops consulting primarily
for small companies
i'm i'm a lifelong developer uh
operations person uh i now do coaching
i'm a trekkie a podcast host i also
dance salsa i'm a father many many other
things i can say about myself but you're
not here to learn about that i i think
most of you are more interested in the
technical part of my career so
i started i got into computers actually
at the age of eight uh when i started
programming on my dad's commodore 64.
uh and for many years i just played
around made games in various uh silly
little programs on the computer
and then in 2006 is when i got my first
uh
professional programming job
and the the interview was interesting
the the owner of this small company
drew three boxes on a whiteboard and
said these boxes represent the three new
servers we just bought
we would like to hire somebody maybe you
to come in and build these servers out
to run our spam filtering software
and up until that point their spam
filtering software that they that they
had written in-house
they would install it on an appliance
meaning like a 1u server along with the
operating system and they would ship
that to the customer's
office and the customer would install
this physical server next to their
probably physical exchange server
and they had the idea of switching to a
cloud-based model what we now call sas
or software as a service it was
relatively new at that point
so that's what i was hired to do i was
hired to uh take this existing spam
filtering software and turn it into a
sas that we could sell uh as a single
installation rather than all of these
countless installations we had
physically installed around the world
the author of this spam filtering
software had just left for for another
job uh up until that point he had been
the only software developers technical
support person network engineer all of
that
in one person
and so when he left they decided to
replace him with with two they hired a
developer to take over the day to day
and then hired me uh to to help turn it
into a sass
jump ahead just a month and after i had
started and it was thanksgiving day and
uh for those of you who have not
experienced thanksgiving day in the
united states let me just give a little
bit of context here
of course thanksgiving day is a national
holiday in the united states so
everybody uh practically everybody was
not working that day even restaurants
closed down
it's almost impossible to buy a stick of
gum in the united states on thanksgiving
day uh because everything is closed
but what you may also know is that the
day after thanksgiving day is a big deal
because that's known as black friday and
that's when everybody rushes out to get
the best deals they can on electronics
and on vacuum cleaners and on whatever
else stores might be trying to sell
for the holiday shopping season
this particular black friday uh while
the entire country was outside shopping
malls fighting over tv sets
uh we had a half day of work uh at the
office and our
our newest developer not me the other
guy who joined at the same time
was pushing his first change out to
customers
what could possibly go wrong here right
nothing
this is what our phone lines maybe
normally look like uh
soon they look more like a christmas
tree
every single customer we had 150 plus
appliances installed in the wild every
one of them
stopped working
our phone lines were jammed and not just
that day this lasted into the next week
everyone in the company not just the
people working on the spam filtering
software the sales representatives the
president of the company the janitor
were taking phone calls from from
customers whose mail was not working
what happened
well
our developer had made a small a single
bit mistake
he had failed to set the execute bit on
a startup script
so this prevented of course the mail
from flowing but it also prevented
future updates from working so we
couldn't even
create a patch and put it on the on the
central server for the servers to
download because the all these
appliances wouldn't wouldn't even start
enough to download a new update
so we had to actually log in by ssh to
every customer machine
and do it mod plus x on this particular
script
for every customer now you might think
couldn't you script some of this well we
could and we did to an extent
but many of these machines were behind
firewalls uh or they the the owners had
changed the password or deleted ssh keys
for security reasons so in some cases we
actually had to call customers over the
phone and walk them through the process
you know if they weren't able to give us
access to their through their network
then we would
walk somebody through going into their
server room
plugging a monitor into the physical
machine logging in with root on the
console and typing these commands to get
things working again it was quite an
ordeal it took the better part of the
following week to get everything fixed
over the following months uh in the
aftermath of this uh this incident
uh we had some other problems as you
might imagine you know problems come up
all the time on software uh one that
jumps to mind is uh we had some reports
that were taking 20 minutes or so to
report to to generate from our database
we were using my sql
and uh
the developer uh the the lead developer
and remember my my job was mainly uh
getting the sas ready his job was the
core development so the core developer
uh and i had a
disagreement on how to do this i
suggested partitioning by date so that
we could only query the dates we cared
about and he said that would be too
complicated and take too long
well our mutual boss came to me
uh
in private and said jonathan you think
you get that partitioning working
and uh i threw together a prototype in a
few hours and it proved that it would be
worth investing in
um
so that upset the developer um that
wasn't the only thing of course um but
he ended up leaving the company so i
became the lead developer of the project
so i essentially set out on my own of
course we did over time hire other
people to join the team and i remained
the team lead for for several years and
during this time i was reading
a lot
of course every hero's journey needs a
guide and i had no shortage of guides
um my guides were people like uh like
joel spolsky who you may know as the
co-founder of stack overflow uh y'all he
also had a blog uh i guess he still does
it's called joel on software he used to
blog very frequently now it's
about once a year maybe
but i read his blog
front to back uh maybe more than once
during this time
michael feathers
he's the author of the famous book
working effectively with legacy code i
would say more than any other book this
book has changed my career it is an
excellent book especially if you're
working with legacy code and who isn't
kent back he's the author of extreme
programming he's also the so-called
inventor of test driven development and
uh he's a very influential person one of
the co-authors of agile manifesto
along with bob martin another co-author
of the agile manifesto bob martin wrote
a number of books in the clean coding
series clean code clean architecture
he has some great youtube talks all of
these people and many many others were
my guides during the the years that i
spent on this project
uh to
to help me
take this project to the next level
one of the lenses i have used to sort of
evaluate the project i was working on at
the time
became the joel test which is a a blog
post that joel spolsky wrote on his joel
on software blog
i think the post was written in 2000 so
i read it six seven years later
but he goes through 12 uh
steps and and 12 questions you can ask
about your project
to decide the maturity or the the
usefulness of your project
um many of the of the items on this list
are timeless they're still valid um do
you fix bugs before writing new code for
example or or does your team
get distracted with new things before
they finish the old things
and leave bugs that sit for months or
years or forever
can you make a build on a single step
this was kind of profound back then at
least
on the other hand some of the items on
his list are a little bit outdated
um
do you use source control i mean that's
still valid uh but i think practically
everybody does if you don't use source
control i would love to hear why
but it's at least assumed that you
should be using source control
do you make daily builds maybe not
everybody does this but it's another one
of those things that
the bar has been raised
nowadays we tend to talk about are you
doing
daily deployments not daily build
but i felt like it's time for something
new so this is why i have created the
jonathan test which are my 12 steps for
better software delivery it's a little
bit different focus than joel's
um but it also is a little bit more
updated you know 20 years newer
as i already mentioned i feel like
everybody should be using source control
that's kind of assumed uh if you're not
get with the game
now maybe you're not doing one uh build
per day but uh it's it's not exciting if
you are so let's raise the bar on that
one
there's a lot of new technology it's
easier to build things faster more
frequently than it was in 2000
um also we have some new science not a
lot there's not a whole lot of science
that's gone into the software
development process but we do have some
so we we do have some
qualitative and quantitative data that
we didn't in 2000 about what works and
what doesn't for example we have we have
evidence now that that code review is
one of the
um sure things to reduce defect count uh
reviewing to this code that didn't we
didn't know that in 2000 maybe people
guessed but now we have some hard
science to back some of that up
so here's my test number one
do devs merge multiple times per day
also known as continuous integration
this is another thing that science has
proven is uh
is beneficial uh in the sense that
companies that do continuous integration
outperform those in terms of delivery
and revenue
versus those that do not
is all code tested thoroughly before
merging into mainline
when i when i took on the the developer
project this was not assumed uh our main
branch was broken frequently and a big
part of the software release process
was testing the main branch to make sure
that it wasn't broken and fixing any
bugs that had crept in in the
intervening months since the last
release
nowadays with things like trunk based
development we can assume that maine is
always ready to be
released continuous deployment
this is your project built and deployed
automatically after every merge to
mainline this was
almost inconceivable in 2000 when the
dual test was written
in fact it was in 2009
when the the famous talk was done called
10 plus deploys per day
uh and
that's what kind of birthed the devops
movement
so nowadays
you know back then we were talking about
is it even possible to deploy that
frequently now we're pretty well
confident that it is possible it's more
a business question now should we
i believe you should
is all code reviewed by another
developer before merge
like i said this is one of the things
that that the little science we have has
demonstrated pretty conclusively that
code review is effective
do you fix bugs before writing new code
this is when i borrowed straight from
the drill test i believe that you should
fix all bugs uh all significant bugs
before writing code maybe if your
alignment is off by two pixels you don't
care enough to ever fix it but if it's
something that affects users you should
fix that before doing new
work do you have a prioritized backlog
of features and bugs now
bugs should be the top of that list in
my view
and then features should be ranked
according to whatever criteria your
project or product manager
has has in mind
do you have a reasonably complete
automated test suite now i say
reasonably complete i don't say 100 or
95 because i think those percentages are
are almost worthless they're so easy to
game
that you might as well not use those
what's more important is do you have the
confidence when your test suite runs
that you haven't introduced a regression
uh if you don't have that confidence you
should improve it you should prove your
test suite don't focus on percentages
focus on confidence
does everybody have quite working
conditions another one i borrowed from
the joel test i think this is really
important
especially for knowledge work like we're
doing in software development you should
be able to concentrate
you use the best tools money can buy
another one i borrowed
if you need a better ide you should be
able to get one if your boss is not
willing to pay for it you might consider
getting a new boss
get the best ide for your job uh get the
best monitor the best keyboard
nobody wants to fight with a mouse that
doesn't work correctly when you're
trying to code get the best tools you
can buy whether that's
hardware software
the best trick ticket tracking system
the best
meeting software whatever you need to
get your job done effectively
the money spent is it pales in
comparison to the productivity lost
developers are expensive people uh
get the best tools you can buy and save
your time
do you do hallway usability testing this
is one that joel talked about talks
about a lot in his blog it's one i don't
do well
well enough
but there's a lot you can gain just by
having some random person walking down
the hallway test your code and i don't
mean like reading your your code but
like
show them the prototype show them the
software
look at this screen what does it say to
you just simple stuff like that just
getting a second pair of eyes somebody
who hasn't been looking at that all day
to look at your code or to look at your
your product uh can go a long ways
no handoffs this one gets at the heart
of the devops movement do developers
control their workflow from start to
finish without handoffs
it could be to qa with operations or to
any other team if you have handoffs
you're not doing devops devops the whole
idea of devops is to kill those silos
can your developers work effectively
from an airplane or anywhere else where
there's no wi-fi connection
this doesn't necessarily mean that you
should be able to build an entire
working application and run it from your
laptop but you should be able to do a
meaningful amount of work without an
internet connection
and this is just in my view it's just
common sense um especially if you have
developers who are going to conferences
or uh they have a long commute through
it on on a train or something like that
or they like to work from a cafe
somewhere it should be possible to work
when you're offline and that's not just
because wi-fi doesn't exist everywhere
else sometimes github goes down
or sometimes your production server goes
down or sometimes anything can happen
that prevents your entire network from
being online and available
you should not [ __ ] your developers
in those instances let them keep working
so with my 12 step test in mind let's
let's jump back to when i started uh
with double check
uh what did things look like back then
do develop do contributors
merge their changes into mainline
multiple times per day no we were not
doing this we did have subversion we
were using version control
but there was no rule about continuous
integration whatsoever
code was not thoroughly tested in fact
as i explained a minute ago we just
assumed that mainland was broken at all
times until we were ready to do a
release then we would test mainline
our project was not built or deployed at
all uh manually or otherwise until we
were ready to do a release so we failed
there
is code reviewed by other developers no
we had no code review policy in place
we did not fix bugs before writing new
code we fixed bugs when they became
quote priority
do you have a prioritized backlog of
features as i recall when i started here
we did have a backlog but it was a text
file in subversion that the developer
had been keeping so he would just as
request came in he would add a an item
to the to his text file and when he was
done he would remove it and that's fine
if it works um
the problem of course is that there's no
visibility to the rest of the company uh
you know customer support can't see that
it didn't matter for him because he was
a solo operation he was technical
support
once you have more than
two people uh you probably want
something a little more centralized than
a text file of course if it works it
works
do you have a reasonably complete
automated test suite uh no we i think
there were some automated tests but most
of them didn't work correctly um so it
certainly wasn't being used and it was
not a reasonable complete test suite
whatsoever
does everyone have quiet condition
working conditions yes uh we did uh we
were a small office uh we uh but it was
relatively quiet so that one uh is a win
to use the best tools money can buy
honestly i don't remember the answer on
this one i'm assuming so i don't
remember any complaints about the tools
we had at our disposal
so i'm going to give them a pass
hallway usability testing no we didn't
really do that
the developers control their work from
start to finish without handoffs
i mean in a sense i suppose so because
uh
there was just one developer or two
once i started um but it certainly
wasn't like a mindset that developers
should be able to do all this it was it
was by accident if anything
can your developers work productively
from an airplane uh you know i don't
remember the details uh this time in
2006 uh probably not i'm sure it was not
intentional that if it was possible to
do meaningful work from an airplane it
wasn't was by accident
final score 2.5 out of 12.
not so great
i stayed a double check for nine years i
left in 2015 in september so almost
exactly nine years uh to the day later
uh is when i left uh let's see how we
scored uh by the time i left
all of these highlighted in green are
things that had improved uh almost
everything improved the only things it
did not
were of course the ability to uh deploy
automatically after every merge we were
not doing continuous deployment
um
and we did not we still did not do
usability testing that's one i fail on
frequently i mean i've done it at times
but it's not something that's ever
become a habit for me
and could we work productively from an
airplane uh to some extent we we had
gotten better about that but there were
still parts of our software that just
didn't work without a a network in fact
i remember writing a pearl module to
allow us to do dns tests do dns based
tests without a network so we could
cache dns services so there was effort
being done there but we hadn't reached
uh our goal
final score 9.5
so we gained seven points over a nine
year period um but i guess that's
something right uh keep in mind i was i
was uh
learning as i went here
um
when i this was my first programming job
it lasted nine years
uh we had two or three other developers
join come and go over that time period
so not bad
let's jump ahead to 2020 last year uh in
january i started with a small startup
called lana they're based in madrid
spain uh they do uh their fintech
startup
when i started in january um
only the cto was able to do deployments
uh so developers were at his mercy to uh
when they wanted something deployed they
had to ask the cto and if he had time
and agreed that it was uh something that
should be deployed or and didn't have
any external dependencies and so on he
would deploy it within a day or two
probably
um
the the only of these 12 items that that
lana had was quite working conditions
and that was mainly because it was
mostly a remote company so most people
work from their homes
final score won when i started in
january
less than 12 months later december when
my contract ended
these are the the changes i had been
able to implement um of course i didn't
do it alone but i was a driving force
behind most of this
you see it's a it's a big change um
almost everything the only things that
weren't uh weren't done
were
completely automated test suites we're
still doing a fair amount of manual
testing
um
and you know you never completely get
rid of manual testing but we were doing
more than we would have wanted
we did not uh have the best tools money
could buy um i had some developers ask
for
an ide and they were they were denied i
disagreed with that but i didn't control
the budget so i didn't get to make that
call
we still didn't do much hallway
usability testing
and we didn't have a no handoffs culture
yet so we were still only part way
through the devops uh sort of mindset
transition
in the sense that we did have a manual
qa and some other developers were
handing things off to our manual qa to
do work uh to do testing before
deployment
but there's still pretty good progress
uh i would say um final score of nine
after 12 months a plus eight over over
12 months that's pretty um
pretty encouraging in my view
i would like to invite you to join me on
this uh goal to improving your jonathan
test score uh you don't have to use my
test use whatever test you want but the
the idea is
can you improve on these important areas
if you want to see the jonathan test you
can go to this url jhal.i o
test
score for yourself
i'd also like to invite you to my free
10 day email
uh course uh called the lean cd boot
camp it's it's really designed for
teams that aren't doing continuous
deployment yet so if you're not doing
continuous deployment this breaks it
down into 10 simple steps that you can
apply and in spoiler alert you don't
have to do automated testing before
you're doing continuous deployment and
that's kind of the point of the lien
here
get yours get your continuous deployment
in place first and then you can start
add the steps for automation later if
this is interesting to you i encourage
you to go to leanciditybootcamp.com
it's completely free sign up for the 10
day course
if you need any help
with that with any of the things i've
talked about i'm always happy to talk to
you you can go to my website you can
contact me
by email follow me on linkedin or
twitter at tinydevops
and if there's time i'd love to take
some questions uh so let me show you one
last slide though if you follow this qr
code or the url at the bottom there that
will take you to this slide deck and
this list of resources so you can find
everything that i've talked about today
with that what questions can i answer
for the audience
so uh we have a couple of questions that
have been posted in the venulis
now i'm not sure whether or not jonathan
hall is actually here in the chat with
us
that we can have a look at
uh hopefully we can
oh someone's got a 404 on codes oh boo
never mind um
so if you've got some questions uh throw
them into the chat oh and i will approve
them as fast as i can when i can see
them and um we'll just have a chat
once we've uh so feel free to chat um in
the chat that'll be great
um we can also do that in the uh
in the hallway as well after this
thing end ends
and once that's finished um we when we
come back we'll have a break uh the next
one that we have the next talk we have a
break until 10 15 so you've got about 15
20 minutes grab some water um
take a break get up walk around we don't
want to just be standing here looking at
screens the whole time
um
but do come back at 10 15 we have a talk
by yakov called making swiss cheese of
ticket sales and yakov grab it gave a
tremendous talk last year so i'm really
looking forward to see what he's come up
with for this year i'm sure it'd be
entertaining
we'll see you again in about 15-20
minutes