foreign
it's my pleasure to introduce Peter from
the Australian energy Market operator
with 6qa tall QA tools that'll make your
python code better take it away Peter
thank you and welcome
who's this talk for
the Amo team in this case study right
critical Enterprise code that executes
on a lock lockdown serves in the cool
room we are very very conservative
I'll be advocating you spend about 45
percent of your Dev Budget on code QA
this may not be for you but we'll
proceed anyway
before I start I have a question and the
question is a question
what is the hardest question for a coder
to answer
give some thought to that although
scroll through the next few slides and
we'll talk about that more in a moment
first of all can you spot the bug in
this code
probably not a fair question because you
haven't seen the context but you can see
that it opens a text file it
iterates over it it pulls out gosh
column number six it adds the value to a
list it sorts it and it prints it out
what's the whole point of that well
here's the underlying data this is a
typical Market data that we deal with at
work it's available on the public
internet and it shows the generation in
megawatts against each generator in the
nem at a point in time the point in time
we do this every five minutes
The Columns of Interest are f
a value we call do it or dispatch unit
ID we call a generator A dispatch unit
and column G scada value which is the
megawatts of that unit is producing at
that point in time
you can often reverse engineer where the
unit is in the country by looking at the
name
c-a-p-t-l underscore WF for example on
row whatever it is six is Capitol Hill
Wind Farm outside Canberra
eyeballing the data you can see the
lowest value is around about negative
0.02 and the highest value is around
about 620
back to our code
it opens the file
it iterates that it pulls out column six
it adds it to It List It sorts it and it
prints out what we think going to be the
first value and the last value the
minimum megawatts and the maximum
megawatts and we get this
minus
0.0015 and 97.
the answer's wrong
so where's the bug
this is perhaps the worst sort of bug
you can get because it executes it
produces a number but the number's
absolute nonsense
so what we need is a crutch to lean on
to help us not make this error
let's get a bit smart now we'll
introduce a class structure to help us
class structure to lean on or a
different coding style to help us
avoid that error down the track and of
course the error was that we're adding
our numeric values as a string to a list
and sorting is a string and strings are
numbers sort differently so we're
getting the maximum and minimum value
based on the fact that it's a string not
a number
let's inject some code that forces it to
be a number
so I've created a class line eight in
the code called genval it has two
attributes do it and scada value do the
string scalar value is a number it has a
Dunder LT method or a magic python
method that will help us sort this class
where it compares the two scada values
it does a generation line 21 creates an
instance of the class line 22
sets the do it line 23 sets the value it
sets it as a decimal so we're getting
smarter now
it does its sort and it prints the
maximum and minimum value
but it's still wrong
like can you spot the error in this code
again probably not a fair question but
where we work scada value is a value
that's used all over the organization in
many systems and it's it's a word that
we hold in our heads and we probably
translate it without even thinking about
it from one form to the other so line 11
is scada value line
23 where assigning it is scada
underscore value and of course beautiful
infinitely
elastic permissive python will just
swallow that and will create scada
underscore value as an attribute on our
class without complaining
so we need a crutch to prevent us from
that error
more code
Can you spot the area in this code
again these are three areas that I have
made a hundred times and there areas my
team has made a hundred times
the error in this one is a result of
copying and pasting the headers from a
CSV into my code entering some Carriage
returns and then reformatting it so I'm
mapping CSV columns into object
attributes
and invertently left a trailing comma on
line 23. now beautiful permissive python
will just swallow that except it will
set our scada value to be a tuple with
one value in it rather than the actual
value itself
I've changed the context of the code
slightly it's now not looking for the
maximum minimum values but it's trying
to sum the generation available at that
um
uh time period and of course this code
will actually blow up on line 28 because
it's trying to add a tuple to a number
this is
this is a less worse error than those
previous ones because our code won't
execute that's a good thing
anyway we've had three hard to spot bugs
in Trivial code
these bugs would never have occurred in
a compiled language like Pascal which is
my background or C or C sharp as the
compiler would have caught them
what happens when the body of code grows
what happens when the number of code is
on your team grows from one to n and
what happens if you're simply having a
bad day
back to the question at the start of the
talk
what is the hardest question for a coder
to answer
in my experience
it's this
are you done
the answer always comes with a caveat
yes I'm done but I haven't finished the
unit test yes I'm done but I have some
documentation to write yes I'm done but
oh my goodness I have to refactor this
module over to the side
what we want is an objective unambiguous
definition of done
something that we all agree to see
something is all your team members agree
to
to allow this question to be answered
about these six and they're going to
help us answer that question are you
done
the tools that I saw black Pile in my
pipe Pi test and coverage we're going to
look at each one and look at the changes
those tools will make to the code as you
execute them
I thought as the first one and it's the
simplest this is the blurb of the
eyesort website I saw it as a python
utility or library to sort Imports
alphabetically and automatically
separate them into sections by type
that's all a bit ho-hum you can
implement the same behavior as I sort
will give you with Control Alt something
other in your IDE
um
it will sort your imported or get rid of
the ones that aren't used it will
eliminate git noise that's the main
reason for use this you're not going to
get any git conflicts in your import
section when you've used this install
use it and move on
black takes over your life a little bit
more
black is the uncompromising
uncompromising python code formator by
using it you agree to see control over
the minutia of hand formatting in return
black gives you speed certainty and
freedom from get noise that git thing
again black will save you time and
mental energy freeing you for more
important matters
let's have a look at some code before
and after we execute black
so our little fragment of code that
parses that text file with the generator
class has been run through black and
it's done two things to it it's injected
a blank line ahead of the class
definition that's pep eight thing it's a
bit ho-hum but it will solve get noise
and it has reformatted the dunder string
method that I've added to my class I
started off with a long uh
long line of code that spanned I don't
know 80 90 characters and black has
wrapped it for me and it's wrapped it
for me in a consistent way
a little bit ho-hum perhaps but again it
will get rid of git noise and it will
stop arguments over formatting Styles
within your team it also allows you as
your coding to just type any old
nonsense run it get the thing working
and then hit black and it's formatted
for you so it actually it does save a
lot of time because you don't have to
stress over do I enter a carriage return
here or should I be continuing on the
line
again black a little bit ho-hum install
it use it move on
piling is starting to get a bit more
interesting
piling has a is a python static code
analysis tool which looks for form looks
for programming errors it helps enforce
coding standards it sniffs for code
smells and it offers simple refactoring
suggestions
now a good ide will give you isort black
and pilint support
this is part charm and you can see line
six there's a squiggly yellow line under
my class definition if I hover the mouse
it tells me pep 8.
e302 expected two blank lines one found
well that's easy to fix and it tells you
another one class names should use camel
case convention okay that's a little bit
harder to fix you've got to control or
whatever the rename is in pycharm and
change that class name all through your
code
pilint will run in the IDE as I just
showed you but also run as a batch
script outside the IDE and we do both on
our project pilot will give an error
message like this it's the file name
followed by the line number and the
column number you can copy that to the
clipboard and control alt something or
other in pycharm and it'll take you to
the exact location it gives you a
numeric warning a message ID it gives
you a human readable message description
and then it gives you a text-based
message ID also you can use that
text-based message ID to suppress this
form of Warning by inserting a comment
into your code
messages from our sample code
the first one trailing new lines to bit
ho-hum the second one disallowed
trailing comma Tuple that's interesting
that's a real bug that pilant has
spotted for us that that absolutely
needs to be fixed
uh the next one redefining a built-in
sum okay so the word sum is a python
Library thing and I have a variable that
clashes with that so that should be tied
it up constant name input underscore
file doesn't conform to uppercase naming
style ho-hum but let's fix it missing
doc string we have a rule on our project
that all classes must have documentation
pilot wants a higher level of
documentation than that but we've wound
that back to only requiring
documentation for classes and there's a
couple of other ones that are getting
into the
um
uh semantic thing rather than hardcore
bugs but we have a policy of tidying
them up anyway because that noise could
be hiding something like our trailing
comma that actually matters
progress between runs with this message
here your code has been rated 10 out of
10. previous run
99.98 out of 10. it kind of gamifies the
quality process so our team members
collectively get this wonderful surge of
happy hormones as our as our progress
tracks put towards 10 out of 10 or 100
percent
here's a fragment of code before and
after
pilot
before and after the human intervention
to correct pylent errors again bit
ho-hum I've changed the name of the
input file variable to conform with pep
8 all uppercase I've changed my class
name to Pascal case and I've inserted a
comma that comma a comment
pilint allows you to write custom
linters and we do this quite a bit on
our project too one of our staffing
patterns is that we rotate our power
system engineer graduates through the
team who all come with coding skills but
not necessarily Enterprise type coding
skills and during peer reviews or peer
reviews of the code we've found
ourselves over and over again explaining
look please don't build up a file name
like that use Python's pathlib or please
don't create a python class like that
use a data class or please don't
concatenate strings like that use F
string so we have a series of custom
linters that pylin executes that
supports the new coder towards our
coding standards and relieves time
during the peer review process for
matters of architectural significance
rather than just
how did you concatenate a string
after running the customer letters a
code looks like this
the input file now has become a python
path object
because it's a python path object we can
open it as a file and iterate it using a
slightly different tersa syntax as shown
on line 26.
and we've created an instance of our
class using the new format where we're
passing important data into the
Constructor on line 30.
my pie is where things start to get more
interesting still my pie is a static
type Checker your note your code with
typings and my Pi will check for bugs
caused by type
incompatibility here are some typical my
Pi messages function is missing a type
annotation
or incompatible type assignment
some code before and after we've run it
through mypine manually made the
necessary changes
our Dundas LT method has got some
typings now so the other attribute or
the other argument that's passed in must
be of type gen and it must return a
Boolean
while we're here we've created a generic
based list of our generators so we can
pass this we can use this list in our
code without having to um
type the list
differently all through our all through
our coder that grows and we've created
an instance of our list of the correct
type down the bottom
Pi test is a unit test framework for
python
High test framework makes it easy to
write small tests while scaling to
support unit integration regression
testing of a full program Suite
now to run Pi test against your code
you're probably going to have to change
the structure of your code this is where
the tools really start to take over your
life
code was initially a large script
execution would start at the top it
would work through to the bottom and the
lines at the bottom would be the entry
point and would call the code
procedurally one line after the other
that is very hard to unit test code in
that style because you don't have a
thing that you can grasp hold of and
pass around you can't create an instance
of it in your unit tests and poke it
with the testing patterns
so we break our code from one file into
four
we start with a an entry point file you
know called done dismayne in this case
we have a file that contains the unit
tests test underscore gen in this case
we have a file that contains our data
test underscore bomb a naming standard
we use on our project bomb means
business object model
and then we have a file that contains
the behavior
gen to total
so the pattern we use is that we
separate data from Behavior nouns from
verbs in our code
our gen bomb file contains the code that
we created earlier on in our big
procedural script
definition of our gen class and the
definition of our gen list
our Behavior has been wrapped up into a
class called gen to total
it has a it has a entry point called
execute again the standard we use across
our team if you have a class that
contains Behavior it has a entry point
called execute
execute takes a single argument passed
to the input file and it returns a
single value of death of type decimal
which is the sum of all the generations
for that trading period as execute grows
you can break it down into a number of
methods that are private to the Gen to
Total class you've got to do what you've
got to do for readability
anything remarkable here if name equals
main create a reference to the file that
we want to read in reality you might
want to parameterize that somehow create
an instance of our gender total class
execute our gender total class grab the
value print the value to the screen
nothing remarkable there
our test code looks like this using the
pi test
standard we have a container class
called test gen read Pi test requires
the first part of a class that's part of
a test to be Capital t-e-s-t it has a
method or a bunch of methods on it that
can execute our test code Pi test
requires test underscore whatever as
part of its naming style line 21 we're
creating an input file with known data
that variable content contains three
lines of text that sets up our input
file with values that we have control
over
line 20 contains a pi test
fixture called temp path
temppath is a path to a temporary folder
that Pi test has created for us and Pi
test will take responsibility for
cleaning up
we create an instance of our gen to
Total class and line 25 we execute and
capture the value and then on line 28 we
compare the value against an expected
answer
and our test passes
and of course we have many many tests
like this in a production body of code
again to execute to work in this way we
must refactor our code and just a
reminder this the pattern is we have an
entry point we have tests we have data
and we have Behavior
it is the last tool in the suite
coverage is a tool for measuring code
that is actually executed in Python
coverage monitors the program while it
executes noting which parts of the code
have been run
it produces a report listing code that
could have been run but wasn't
coverage report on our frag on our test
code looks like this
you can see we've got coverage of 93
percent and we're missing lines 8 to 11
of Dundas Maine
how are we going to hit those lines
things start getting a little trickier
the code at the top shows the entry
point as we coded it
early on
if name equals main creating reference
to the file an instance of the reader
executed print the answer
we need a way of grabbing hold of that
code in pi test and poking it
so to achieve that we've wrapped it all
in a method called Main
same code is just contained in a in a
function called Main and the entry point
calls that function on line 14.
this allows us to write a unit test like
this
we use Python's import lib
module to import our Dundas main file as
if it was a module
we then use Python's mock library to
change the name of the doneness name
attribute on that file to Main
we then hit the main
method the main function in that file
a pi test has another fixture called
capsis which will capture anything
that's printed to the screen
so line 35 we're reading the contents of
what was printed to the screen
it gives you two outputs what was
printed to stand it out and what was
printed to standard error
we checked the contents of standard
error against our expected value and our
test passes
which now reports 100 percent
coverage
now
there'll be a question within your team
what level of coverage is sufficient
when we started this
we had our unit test and we had almost
prod quality code we had a coverage of
around about 95 percent
we all pulled our efforts and spent a
day and we got it up to 98 percent
and then we did some soul searching how
far do we want to take this because
we're spending our employers money and
to reach that extra two percent we're
starting to spend serious money
but
well and so we decided to give it a go
so we time boxed it we said by the end
of the week if we don't have 100
coverage we're going to back off and
lower our quality standards
we achieved 100 coverage there were some
patterns we had to nut through
particularly that how to hit an entry
point
but this has become our quality metric
from that day forward code doesn't go to
production unless it has 100 coverage
do I have to do this really is often
asked by the new members of our team
this is seriously expensive
and the answer is absolutely yes you
have to do this for our code base
for many reasons but one is that your
code will last a long long time
December 17 1999 is the birthday of the
system we support
23 years old and the code that was
written prior to December 17 1999 Pascal
code is still in production and is still
producing value for the organization so
we write our python code as if it's
going to last 25 years or more
what's the lifetime spend on QA
aimo's amp code base the project I work
on now has 71 000 lines of production
source
it has 52 000 lines of unit tests that's
around about 40 of our code is unit
tests
back to this question the hardest
question for a coda to answer
are you done we now have
an unambiguous objective way of
answering that question
I thought it's clean it's black is clean
is clean my pie is clean you have unit
tests and coverage is at a hundred
percent
we can talk about
the cost of peer review user acceptance
testing load testing cyber testing
integration testing usability testing
and all those other things you do before
you're truly done later but this is a
tool for the coder
thank you
thanks Peter we have some time for
questions yes
did anyone have a question
um do you uh
maintain the same level of checking for
products and related issues how do we
how do you fix those in case
say that again if you could in case
there is a products and related issue
and you need to push a chance rather
quick
uh yes we do
um we have a git row repo of course we
use git flow so we have a series of
branches we have feature branches hot
fixes going on
release branches if we have a prod type
quality will create a hotfix
we'll implement the fix
this is absolutely a quality metric that
everything that goes to prod has to
has to go through
um you think it's an overhead and it's a
burden and it's going to cost more than
it saves but because we have this
we're able to implement that prod fix
very quickly
[Music]
looks like we've hit our definition of
done no questions I think so
so much Peter it was a lovely talk thank
you
thank you very much