foreign
[Music]
good morning again everyone I hope you
enjoyed your morning tea
now we're getting a talk from Paul
McKinney he's a software engineer who's
best known for his work in RCU on the
Linux kernel
he's been coding for more than four
decades so he has a bit to teach us and
today he's going to to some of the
lessons he's learned in that time please
welcome Paul
[Applause]
thank you Russell
so as Russell said I've been doing this
for a while uh for almost 50 years in
terms of just coding uh 45 supporting
myself
and uh I can sum up some cautionary
quotes which are mostly saying that if
you want to actually get what you want
you have to first know what you want
and this is all very nice to say but uh
you know here we live in the real world
so I'll go back to 1990 something called
stochastic fairness cueing
and that involved these guys this is a
Cisco AGS Gateway from late 80s early
90s and I'm one of the people that's
actually programmed the innards of this
without actually ever working for Cisco
this was part of a research project and
the problem we were looking at was this
kind of thing a queuing problem so you
have a router like the one we saw on the
previous page we got a bulk Source it's
just maybe FTP or something just
blasting bits out there and a couple of
interactive sources that are just trying
to Echo characters
problem is back then a one megabit
network was really fast and these poor
little one characters get stuck in this
mass of red packets that are filling up
the router's queue and your echoing
takes forever
now the uh theoretical solution back
then was something called Fair queuing
so what you do is you identify each flow
of traffic with its TCP IP quadruple the
two addresses and the two ports and uh
you've used a data structure to give
each flow its own cue
well the problem with that uh and this I
guess hasn't changed much is the CPUs
have a hard time keeping up networking
and back then the CPUs were really
really slow
and so this wasn't really practical
so the trick we used was instead to use
hashing so we wanted to give each flow
its own cue but only with high
probability all right so sometimes you'd
Collide and that'd be bad most the time
you wouldn't
and so we hashed the IP port quadruple
wonderful end in fairness with the end
end principle it was great
and my vision at that time was we were
going to have stochastic Fair queuing
routers throughout the internet and that
would be wonderful we'd have fairness
and be great of course you haven't heard
of this
and there's a good reason why you
haven't heard of this
and that's because this was not a vision
this was a delusion this never happened
and what happened instead was that the
backbone tended to be over provisioned
and there was not end-to-end fairness
Supply but rather hop to hop hop by hop
and they had the fairness applied at the
endpoints into the houses shown at the
bottom there
in fact I would get calls from Engineers
who'd read my paper and say hey this
this sounds great we'd really like this
but is it okay if we use Ethernet Mac
addresses instead of using the IP Tuple
quadruple I go well yeah that'll work
but why okay thanks bye
anyway uh the problem was I solved the
wrong problem
I was gonna solve end in fairness
because that's what the people around me
thought was important the correct
problem was instead hop by hop fairness
and endpoint fairness but by sheer dumb
luck my algorithm handled both
the other problem I had was that I was
writing research quality code I might
have been insulted by that at the time
but I've learned differently since
I just need to get a paper out uh
fortunately nobody asked for the code
they just implemented good code based on
the paper
this was used heavily until about 2015
and for this sort of thing 25 years is
not a bad run
there's something called fq codal and
cake that also addressed buffer bloat
and these three guys did a lot to make
that happen I suppose they're gonna be
obsoleted you could do a lot worse than
we obsoleted by these three guys
so the thing was I had a bad idea that
was badly implemented I was trying to do
the wrong thing and didn't do a very
good job of it but fortunately my idea
was resuscitated by sheer dumb luck
uh and uh here the thing is here that
premature abstraction is the root of all
evil used to be performance now I think
it's abstraction
uh the problem was that the end-to-end
abstraction was a very powerful one and
happened to work in a lot of places this
just was not one of them
and the problem was that I failed to
live among my users I was living among a
bunch of researchers and standards
people
but Engineers who were living among the
users were took made things right
so let's go back even further to the
1980s this is customer relationships
management I did an application based on
one of these things
this is an ultrona attache which was
very popular for a couple of years in
the early 1980s the reason was so
popular was that it used a z80 and ran
CPM instead of using the more expensive
16-bit processors so it was quite a bit
cheaper which was fine until the 16-bit
processors got cheap enough that it
didn't matter
but at that point there was a customer
that asked me to build it to spec under
contract so I did that work and they
loved it it was great they just they
were really happy I'd done all the stuff
and made it work and that was great but
their prospective customers weren't
quite so happy
in fact their prospective customers were
so unhappy that they went out of
business
uh by sure dumb luck they paid me before
they went bankrupt so uh you know uh
this is a bad idea I claim it's well
implemented the code's lost in the midst
of time so who knows
but hey I got paid
the problem here was just the computer's
field was quite immature I was the only
one on the team that had any experience
and I didn't have any contact whatsoever
with the customer and that was just
built to fail which it did
so this quote from about that same time
every nose debugging is twice as hard as
writing a program in the first place
well that's what we got hit by
the thing is
while you're programming you're living
in Blissful ignorance of a lot of the
requirements yeah you got the ones you
know about and you're quoting those and
it's great uh but those other Pro
requirements are really still there okay
and they hit you really hard during
debugging and this lack of requirements
lack of knowledge requirements is only
one cause of karnahan's observation
my problem was that I failed to
understand that I was competing with a
file cabinet
and you know what
the file cabinet one
so again uh it's a live among your users
thing let's go back even further to the
1970s uh this was uh the first paying
assignment I had now I worked my way
through University the first time uh
working on the program that figured out
which student went into which dorm room
and also how much they should be charged
uh this is not the computer used but it
was the same type A CDC 3300 uh it was
slightly configured slightly differently
but it filled a room just like that one
did
and we were using Punch Cards in Fortran
I kid you not Fortran for a business
application program but such was the
time
but we moved to the times later on we
moved to a cyber 73 which is a successor
of the famous CDC 6600 supercomputer
from the mid 60s
and uh we went to Punch Cards on Cobalt
now there were a lot of issues with this
being as things as they were one of them
was that the code I received had some
temporal confusion
normally the students were charged for
an entire term so this didn't matter but
uh one feature they'd asked before I
showed up was to make it so that a
student could arrive part way through
the term and they could be have their
charges prorated over the amount of the
term they actually resided in the dorm
room
okay so that's a problem was the the
computer had the computer program had
some uh
interesting ideas about different
days of having different value
and a student started on Friday and was
not amused with the bill
my manager had some interesting
suggestions for what that student might
have wished you the money instead but my
job was to fix the problem
and the problem was the months vary in
length I mean this month for example has
31 days next month has 30 days last
month had 28 days except that last month
the next year has 29 days
and there's a lot of corner cases and
easy to get that wrong and uh the code
had gotten it wrong
the solution was something called the
jaded algorithm and no this has nothing
to do with the website this is before
the web showed up this is just a nice
numerical algorithm that turns month day
year into a sequential number for each
day going up and also switches them back
which makes this sort of thing trivial I
applied that and life got easy really
quickly
uh this was a good idea implemented
poorly
now I've kind of broken the rules here
because I'm telling on my predecessor
not myself so I should fix that and to
do that we'll go ahead to the mid-1990s
this is a sequent symmetry system and
these were used to make cluster database
servers
and the idea was you had a shared disk
system so each server here could access
all the disks
and the cool thing about that there was
a locking algorithm distributed lock
algorithm according to all coordinated
all of that but the cool thing was that
one of those guys go down you could
still access all the data
of course if you're doing something like
that failover is kind of tricky so you
really want to make sure that you test
this thing all right you need to test it
frequently more recently we have a
concept called a chaos monkey to do that
but that was a long time in the future
back in the early 90s
but for some reason one of our customers
objected
to having this thing tested every
evening
which we were inadvertently doing
fortunately for us it uh system crashed
immediately after backups completed
but still this was not a good look and
the customer complained vociferously
problem was I mean the standard thing
back then you collect your crash dump
from the customer analyze it and fix the
fix except that the crash dump was just
a mess there was no useful information
in it and we were for a long time unable
to reproduce in the lab we tried what
they did it worked fine
until finally we did find a test case
that uh between five hours and 24 hours
77 hours it would fall over
problem with this it takes a long time
to test an alleged fix and uh there's
this U.S holiday at the end of May
called Memorial Day and it was the start
of that Memorial Day three day weekend
but you know we had a problem we need to
fix it the customer was getting really
upset
uh we finally got a hint from a stack
trace and that pointed the figure at
this piece of the code
and it translated an address to a data
structure that represented a two Meg
section of memory unfortunately memory
was very scarce expensive and so we
couldn't afford to line the memory
region so that meant that we'd have
something like this the address was up
there that yellow arrow
where the beginning of the region was
down in one two Meg region and the
actual address while still in that
region was in another was in the next
two Meg chunk of memory
and that was okay we had some code to
fix that we would go it would translate
it to the thing it would say wait a
minute this entry they rape is that blue
guy up there not the yellow guy so let's
go to the previous one and yay life is
happy and let's go
except that compilers sometimes mess you
up and then and this was an 80 86
processor which has and 32 bits and
doesn't have very many registers so the
compiler decided that uh no I don't have
any of our registers we're going to
refetch the stupid thing every time I
don't care about your temp variable I'm
not going to use it
and in that case of course things can go
bad uh we go and we translate to the
thing we check it's a null pointer we
say nope it's not null but then the guy
up there goes away now the pointer is no
we compare a no point is less than any
other pointer so this is great we got
the right guy we won't decrement and
bang things go bad
that's what was happening
okay this is an easy fix I've shown the
fix in Linux kerneles uh you just do
read once basically tell the compiler
you can't trust this pick it up and use
what you picked up don't pick it up
again unless until I tell it to you tell
you to and in that case it works great
the memory can be free we still have the
old value which is not null we do the
comparison everything works out and it's
great
this is a good idea implemented poorly
and for any compiler writers that might
happen with the audience despite what
you may have heard and what you may feel
volatile is your friend okay
it's needed by device drivers so if
volatile doesn't work your credit card
doesn't work either all right
and uh that's what rewinds does it
doesn't volatile read and that is the
story of how I provide deprive myself
and my colleagues of Memorial Day
Weekend okay didn't work out very well
but here I am
well now I'd like to go back to the
1970s this is my very first professional
project I said earlier about a first
project but that was the first paid
project this uh involved a computer that
was off-site inside our high school this
is what you saw a Model 33 teletide
paper tape the whole show author Cat
uppercase only
the actual computer was connected by a
lease line 110 bod it was a couple hours
drive away
and I was really impressed with this
when I first saw it it could type faster
than I could all right wow
um and it didn't involve a bunch of
cards
now this was a pro bono computer dating
program they had a dance the National
Honor Society it was a fundraiser
and uh
I was in charge of writing the code that
uh did the matching now I had a pretty
good idea with the boys were after but
um at that time and place doing similar
research on what the girls would after
would have been considered highly
inappropriate so instead I interrogated
my home economics teacher I didn't
actually take any courses from her but I
interrogated her anyway
and extracted some questions from her
and that allowed me to make a simple
Hemi distance matching with the expected
1970s constraints on matches
so we sold the students questionnaires
they filled them out and gave them back
to us we constraint transcribe the
result of paper tape and then read that
paper tape into the program
this symbol was effective it worked
great
except that we did have one dissatisfied
customer
it seems that a senior girl got matched
only with freshman boys which I've said
her greatly and we looked at the Forum
and she really did check the seniors
only box
uh you know check the program and it
looked to be fine and so we checked the
paper tape and sure enough this is a
data entry error
let this be a lesson to you having a
correct program is not enough
the environment in which that program
runs and the processes surrounding that
that program are also very very
important
now I could claim hey the code was
correct I'm good except uh and this was
a good idea implemented properly as far
as the code is concerned
but I was also the overall Project Lead
I was in charge of this whole thing as
well as the guy writing the code
that meant that if there's a problem
it's my responsibility no matter where
the problem is
and you know uh looking back on it it
might have been a mistake to entrust the
data entry to the Freshman boys
but also
I missed a chance to learn from that
young lady
and there's a quote more recently a lot
of success in life and business comes
from knowing what you want to avoid
and I don't really know if you knew what
she wanted but she sure knew what she
didn't want
and if I had learned that that was
important rather than just being uh you
know I've got this angry customer
I could have saved myself a lot of
trouble later in life I eventually
learned it so here we are
let's get into this Century I've been
doing a lot of stuff from the past
Century let's at least get into this
century
with the real-time Linux I'm not showing
a picture of a machine here just because
there's so much variety there's all just
any range of things that might use
real-time Linux these days
but I'll go back to 2004 and uh my
employer at the time IBM got a lot of
requests for something called real-time
Linux but it was actually Enterprise
grade real-time Linux by which the
people making the requesting proposals
meant something that IBM would stand
behind and fix everything for
problem was that no such thing existed
and IBM had rather strict rules for
contracts calling for mythical creatures
and these rules can be stated in two
words and those two words are no bid
so there was a lot of no bid on
real-time Linux contracts
except
at that time there were some chips
coming out
there were armed chips and they had
multiple CPUs
I saw a a relief press release on these
things it's like wait a minute
we can take one of the CPUs and Mark it
as non-real time so CPU zero will be our
sacrificial lamb
and then all the other CPUs are
restricted to only running real-time
code
and to make it so I can get rid of the
arrows I'm going to change the legend so
we have nrt for non-real time and RT for
real time we can drop the arrows at that
point
and then we have the green real-time
code running on CPUs one through three
and if anything happens non-real time
it's on CPU zero
of course at that point even doing a
system called Linux kernel was non-real
time pretty much any system call was
going to get the big kernel lock and it
was game over but that we could deal
with that what happens is if that
happens we migrate
so
that guy that just turned red over there
in the in the lower left CPU two he just
did a system call well we're not gonna
let him do a system call we're gonna
migrate him
to CPU zero
then we'll let him do a system call
and then when he gets stolen system call
Will migrate him back and they continue
being real time
this is simple it's pretty
straightforward easy concept
of course an easy concept uh so I
implemented a patch
and uh it worked tests out worked out
great
there was a real-time effort spending
that's a really big thing they were
reading whole kernels crashing
everywhere uh but you know would they be
successful if they're successful when
would be successful
uh what I was doing is very pragmatic
and and I knew I could commit to it so
what I did I informed my Executives that
real-time Linux is real and we didn't
need to know a bit anymore
and you know what
those guys actually listen to me
and so we run won a large contract
you know had a nice celebration and
everything except that
uh the customer rejected my idea
that wasn't going to work for them
now
I Can't Tell You Why my brilliant idea
was rejected because I didn't have a
security clearance and so they couldn't
tell me
but that's all of the good because I can
share my speculation with you
and my guess is if you were on a high
value Target and there was Munitions
coming at you at a high rate of speed
you just might want all the CPUs working
out what to do about that
but in any case uh my idea was rejected
we still had these contractual
commitments to me this cracked contract
have been signed before I found out
about this
you remember a couple slides ago there
was that effort I was throwing shade on
well now it was the only game in town
and one of the reasons I was didn't
trust it is because I was pretty sure
they're gonna have problems with RCU and
and I was right and so it was clearly my
job to help them with RCU
and it was something like three from
scratch implementations to get an RCU
that worked well and scaled did all that
stuff but fortunately the first one out
of the gate was good enough for that
contract
and that might sound like kind of a
problem but you know that effort was on
the highlights of my career now it's
working with some really bright guys we
were doing stuff that where I wasn't
clear we could get it done and we got it
done and and changed the world it was it
was a lot of fun that's uh it was it was
a real great experience
sort of a nerve-wracking experience at
times but a great experience
so there it is and uh I had a nice idea
that colliding with reality and you know
when that happens reality wins okay
reality wins
um I'm going to uh
go through this kind of quickly
uh first off somebody may like formal
verification some may have been burned
by it so it's worth asking why bother in
practice with formal verification and
I'm going to view my experience we're
going to go back to that National Honor
Society computer dating program I wrote
in the 1970s and there was only one of
them right we only had one user so if we
had a million year bug in that code I'm
not sure how you you'd have a million
year bug and a code that just checks and
does Hamming Edition but if we did it
would happen once in a million years and
at that point you know you know Murphy
at that point he's kind of a nice guy
sure everything that can happen will
eventually
maybe in geologic time
but over time I've written code this had
more and more and more users
and in 2017 a guy from arm told me that
there were at least 20 billion that's
dealing with a B is in 10 to the ninth
power as in 20 times 10 to the ninth
power Linux systems out there
at that time across the stall base a
million year bug is happening several
times per hour
worse yet if the Linux kernel were to
gain any reasonable fraction even a
small fraction of the iot market and
that goes as much as people think it
will well
uh my fear is that Mr Murphy might
transition from a nice guy to a
homicidal maniac
and I don't know about you guys but I
feel really really bad if I could end up
killing somebody
so
um what do we do about that
in some cases it's surprisingly enough
in some cases you can actually test your
way out of it for example if you had a
bunch of smartphones and people actually
use them as phones like us old guys do
but for servers and for smartphones
using the new age way forget it
plus Linux is starting to use for safety
critical applications not really heavy
duty ones but still safety critical
applications and when you're faced with
something like that
formal verification is full State space
search gets kind of attractive all right
and uh I've done some stuff with it uh
but most of the stuff was verifying
design and to make this work for real
you need regression testing as well
and the thing is you do a design
verification then you do a bug fix and
is your verification still valid well
maybe it is maybe it isn't
uh the problem is this stuff's expensive
at best it's exponential often excitable
and if you do the full up verification
you have full specification that's
software it can contain a lot of bugs
itself which means that if you do your
final verification it's your full
specification might be driving bugs into
your software which is the opposite of
what we want
as a result formal verification really
is used heavily in practice but its use
is highly restricted it's powerful when
properly used but you need to verify the
verification
this is some examples of places it's
used in it's uh the lineage kernel is
subject to formal verification fairly
often
now the way we deal with the verif
verifying the verifiers we do a one-way
bet
so we look for a problem if we find a
problem we treat we do a bug report and
treat it as real if we don't find a
problem we say yeah well whatever we're
not gonna We're Not Gonna trust that
okay so the formal verification can find
bugs for us but it can't lie to us well
it can lie to us all as well we just
won't believe it if it says that the
software is correct we'll just ignore it
so let's get back to this cautionary
quote a lot of success in life and
business comes from knowing what you
want to avoid
uh and uh here are some quotes that are
also pertinent I'll focus on the last
one from the initial Will Rogers
we don't know what we want but we're
ready to bite somebody to get it
and of course that notion leads us to
something called natural selection
and if we're to talk about natural
selection we should have the great man
here Charles Darwin looking on the
proceedings and there he is
and the thing is Charles Darwin was
studying National selection in life
forms
sometime later about a century later
we learned that this natural selection
is based on DNA which has codons which
controls proteins and much else besides
and it's a weird form of software
so Charles Darwin didn't realize it but
he was studying natural selection in
software in living software
and a if natural selection Works in
listening software why not apply it to
real software
so we've got our randomly generated
software instead of mutations
um some of you may object to me calling
your carefully crafted soft randomly
generated
as Russell said at the beginning of this
I've been doing this for Far Over more
than four decades
if you want me to change my assessment
I'm happy to do that after you change
your behavior after you change your
behavior
okay so we got this randomly generated
software you make a validation function
to our selection that kicks out bugs we
fix the bugs and hopefully we inject
fewer bugs than we fix if we manage to
do that we've got this nice virtual
Circle here we go around
and uh we have robust software dropping
out the bottom and you can use agile
methods and well and they attempt to
push this methodology back to the
specification to deal with the what do
users want problem uh with the varying
degrees of success
okay uh there's one problem with this
uh this applies to
software and unfortunately there's a
particular kind of software that it
applies to and that kind of software is
something called a bug
so we don't get robust software dropping
out of the bottom there much as we might
like to instead we get software plus
bugs that have adapted to the
verification
okay so what do we do about that well
one thing we can do is we can get bug
reports in the field now of course we
get the bug report we fix the bug in the
software but it's important to realize
that bug report also indicates a bug in
the validation
the validation has a bug because it
failed as fought a bug in the software
so don't just fix the bug in the soft or
fix the bug in the validation as well if
you do that then you've got a better
chance you can at least get rid of the
bugs that people run into
and make it so that such bugs will be
caught in the future
but there's another issue
okay you're going to validate your
intended use cases naturally right you
write test cases and validation whatever
you can do verification whatever you use
for your use cases you envision
you do some major development that
generates bugs if everything's working
perfectly it never is but let's let's
give it the benefit of the doubt you'll
get rid of all the bugs that the covered
by the current validation
then you do some more development get
rid of some more
and you know what somebody says hey we
can use this or something else
but your software is protected from
those new use cases by walls of bugs
one way out of this is open source
software
the thing is is that the people trying
to use for the news use cases can fix
the bugs or pay somebody else to fix the
bugs or do something anyway they have
some options with proprietary they they
don't release a software and it may just
be stuck
so that can help to some extent and of
course you know if
you have a choice
yes you're going to fix the software
uh that the fix of bugs that are
affecting your customers if you have two
bugs you'll you'll prioritize the one
affecting your customers before the one
that doesn't but if you only ever do
that if you only ever fix the bugs that
affect your customers this is where
you're headed
one way to get out of this is to add
paranoia to the bug reports and right
now a pretty good form of paranoia is
the fuzzer uh this been doing a pretty
good job in the latest kernel finding
all sorts of stuff
and hopefully that's helping and making
things better
all right all right
so
the other piece you got to keep in mind
is natural selection is is a euphemism
for something that's not very pretty
if you don't believe me on that just
think hard and carefully about what
happens to the organism that is not
selected okay
and the same thing applies to Software
if your tests are not failing they are
not improving your software
if your tests are not failing they are
not improving your software if they're
not failing there's a bug in your test
there's bugs in that software somewhere
and that test is failing to find them
in addition if your news users are not
complaining they are not improving your
software
this is a little bit harder to take
sometime it's really painful when users
complain but that's how they improve
your software
okay so why would they fail to complain
well the usual cause is the first one
there they are aren't using our software
at all
they might not know who to complain to
it might be the last end times they
complained either nothing useful
happened or worse yet they were yelled
at or otherwise belittled by the way if
you've been doing this last bit here
yelling at them and belittling them but
please stop that if you sell user
l-u-s-e-r please stop that too that's
that you're not you're not helping
yourself
or it could be your software successful
I mean it just works and nobody has to
do anything with it and it's faded into
the work and and it's wonderful
well that's called a technical success
of course if that happened you probably
have to find something else to do
but that's success
so another cautionary quote is from the
inevitable Steve Jobs customers don't
know what they want until we've shown
them and that's part of the problem here
now Henry Ford has a possibly apocryphal
quote involving horses but uh this one's
verified if there's only one secret of
success it lies in the ability to get
the other person's point of view and see
things from that person's angle as well
as from your own
in other words if your customers don't
know what they want you need to
understand them well enough to see what
they want or need
and that's not something I saw for guys
are really good at
um
we work with things rather than people
that's just that's what we're trained to
do that's what we're supposed to do and
that's why I keep saying You must live
among your users given that we're not
very good at it we have to have the
users around us full time in order so
that we can see what's going on we can
understand where they're coming from but
I repeated exposure
and here's the tricky part you have to
complain on their behalf they may be
perfectly satisfied they're doing
something as repetitive as taking a lot
of time it's causing problems for other
people that's consuming a lot of money
uh but they're getting their job done
it's they're turning the crank it's
working for them they're happy in that
case you have to complain on their
behalf which can be uncomfortable or
even sometimes dangerous but it's a very
important skill As you move up to more
senior levels and yeah I'm in a senior
level I don't bleach my hair this is in
fact my natural hair color
so to sum up you know users people don't
know what they want
but that's no excuse not just for
software Developers for engineers in
general uh we are nonetheless required
to get people things they want and need
perhaps it's some consolation to know
that you've only failed if you've given
up until then it's called learning maybe
learning among them and learning living
among them and learning what they want
and maybe more important you're not a
failure until you start blaming others
for your mistakes so you know the whole
point of this you're going to make
mistakes you need to own them and
improve based on them and then you can
give your users what they don't know
that they want
okay I had it easy actually uh the first
part of my career is just doing assigned
work uh it was easy to figure out what
to do to keep people out of court or
even out of jail that was most of us
doing in the early 80s
keeping users happy and filling research
contract terms wasn't too hard
prior to Unix it was pretty
straightforward I had to do now things
more challenging more recently the last
20 years there's a bunch of stuff that
uh I've had to work on and most recently
in my current employer he's the
administration large data centers which
has been a lot of fun a lot of really
cool challenges over that time and uh
you know I'm proud of my accomplishments
but a lot of you guys are working on
stuff that is much more complex and way
more user-centric okay
uh and uh hopefully what I've said here
has helped but in the meantime my jobs
provide reliable infrastructure for what
you guys are doing so I'm I'm here for
you down underneath
so uh at this point we've got some time
for questions and I'm going to show One
technical success I mean sure my code
has its problems on the planet Earth but
so far on planet Mars has been a
technical assess no complaints
along with everybody else in Linux
kernel back in the day
so in any case thank you for time and
attention
over to you
Israel we have about four minutes of
questions
on here
I think disappeared um do you have any
advice you mentioned earlier about uh
customers sometimes don't know what they
want do you have any particular advice
on how to coax them into a state where
they can know what they want
Beyond becoming a subject matter expert
in whatever it is they're doing
unfortunately becoming a subject and a
men are expert in what they're doing is
the safest and probably most productive
route that's what I'm getting at with
the live among them but
um
other than that uh one thing is to is to
what your communication skills learn how
to how to understand what people are
saying and what listen to them and talk
to people who are customer matter
experts the the thing is there's a bit
of an impedance match so somebody who
understands really well what the
customer is wanting may or may not
understand what your capabilities
limitations are so they may ask for
something that's that's impossible or
they may fail to ask for something as
simple that would really help them
because they think it's complicated
uh so uh if you're not going to become a
subject matter expert that's safest okay
live among your users and become a
subject matter expert but if you can't
do that you need to really really
seriously work on your communication
skills and uh how to understand what
people are getting at some of the agile
methods can help they're not perfect but
if you can produce prototypes for them
and show them that can help sometimes
oh so I'm sorry I don't know of a silver
bullet
but that's what I know right now
what else all right I've got one for you
uh what cool what bug caused you them
I think well yeah what bug caused you
the most heartache
what bug caused me the most heartache
well the the one I talked about that
cost myself and my colleagues in
Memorial Day certainly took me a while
to live that one down I'll give you that
right now
oh uh
that's uh uh more recently I suppose so
what ones caused me trouble recently A
lot of them are bugs that didn't seem
like bugs to start with so uh one of the
things that happened uh upon uh joining
Facebook and meta lace at least that
they needed a no new variant of RCU for
tracing it for BPF the the Berkeley
packet folder software so we talked and
we put something together and unknown to
all of us that had a bug
uh they had worked for an art test it
seemed to work fine in production and
everything right
but uh uh what happened was it was uh it
allowed things to it did better than we
thought and they ran into some update
side performance issues which weren't
bugs until they hit them and suddenly it
was a bug and that came over it came
along twice there wasn't didn't really
cause me heartache I don't think but it
I'm just giving you uh another way these
things could happen
um I did it I did things as best I could
and and thought I never really thought I
did fine for six or eight months and
then these new things came up
okay uh and and I'm going to give
another example uh it's a silicone
manufacturers they produce the same as
the old skills before and suddenly got
tons of bug reports
because suddenly people were trying to
do that instead of sending dials to zoom
in on the traces
so does it just
we've got time for one more question
thank you
um
with you said open source is sort of
part of the solution uh and I agree with
you on the principle that open source
gives customers the ability to sort of
directly contribute but I've also seen
the sort of patches welcome attitude
from open source how do we
sort of make open source more friendly
to customers and users so they don't
feel like they the only way to
contribute is by fixing their own
problem
that's an excellent question you make an
excellent point at the same time the
fact that they can fix the problem by
contributing as an option they have with
open source that they don't have in many
cases proprietary software so let's not
lose sight of that but you're right we
need to do better okay
uh the CII uh initiative is one way we
help doing that by funding some people
to make it work better
and perhaps there's more things done
along those lines
uh the
uh it may be that we can do a better job
of reaching out depending on what the
bug is okay I mean I'm not suggesting
doing this for an RCU bug all right
although I've come across some really
brilliant students in my time and some
of them a few of them a very few have
managed to actually jump in and do this
kind of thing
but perhaps we need to uh keep track of
the requests and see if we can get
people to come in people who are
interested in learning to use those
learning processes for some of the bugs
now that's not a full solution
uh but uh it's a problem bigger than I
am
but the problem we do need to take
seriously I agree with you
all right thank you Paul we'll have to
bring it to an end now we've got the
next session starting in about nine
minutes thank you again
thank you all for your time and
attention
[Applause]