welcome back
uh we are joined by liam now to talk
about working with penetration testers
as a developer
liam is a former software developer who
switched to the dark side and started
pen testing many moons ago
formerly a director at assurance liam
now runs the research and development
program for the pen testers at cyber cx
liam is enthusiastic about ducks so
please get all the duck emojis in now
take it away liam
thank you
all right welcome
um
yeah thanks for the intro and
uh
all right who am i uh talked about that
a little bit
before um
my name's liam
you can find me on twitter
i am presenting to you from where
injuryland
um i am the research and development
director
for the security testing and insurance
practice which is essentially the pen
testers at cyber cx
and i'm also
acting director of the appsec team at
the moment
and
what what are we talking about why are
we here uh why am i here well
i like to think of myself as a
professional and for me that doesn't
necessarily mean wearing a suit and
collar means
actually caring about the outcomes of my
work and
the customer experiences for you know
when you're
working as a pen tester consultant
i get frustrated when
i see engagements not go as well as they
could
i get frustrated when i see customers
not getting
value or
uh spending too much money really
because like pen testing is expensive uh
mario's talk earlier on appsec you know
talked about how important uh budget and
cost is in information security
um because like let's be honest like we
we love security but every dollar spent
on security is a dollar that's not spent
on adding a feature or making it out
more accessible or something actually
constructive so what i'm hoping to
achieve in this talk is to try and
make things more efficient and avoid
wasted effort
so just for a bit of context if you're
not familiar with it this is the owasp
software assurance maturity model um
shout out to owasp who were another
great community run organization uh they
just released their latest version of
the
os top 10 so if you're keen about web
security go check that out it's great
um
but the reason i'm mentioning this is
just to point out that like pen testing
is a really small
it's in
part it's an important part but it's
only a small part of the the much bigger
security puzzle
um
so
i know some pen testers seem to think
that like pen testing is security and
security is pen testing but that's
definitely not the case
and
so if we think about
developers and pen testers like we we
have different
expertise we have different experience
we have different backgrounds i mean
everyone does that can be said for
everyone and that's like i was gonna say
that's okay but that's that's more than
okay like that's as it should be you
know people um
specialize in different things
and what what i'm not trying to do with
this talk is to turn developers into pen
testers like what i'm trying to work on
is that that bit of overlap so like when
we're working together um
how can that part run more smoothly
and if it seems like the talk is a
little bit one-sided in it where it's
all about
his things developers can do to work
better with pen testers um remember that
like in the shadow world there is a
reverse version of this talk called
you know working with developers
as a pen tester and it includes a whole
bunch of advice for how pen testers can
do their job
better and work with developers like
things like
remember that security testing isn't
actually everyone's highest priority
even if it's yours and
remember that some features are actually
more important than perfect security but
anyway enough about the shadow talk back
onto the real talk
um
so
it seems silly to define what a pen test
is but i think in the tech industry in
general um terminology isn't often a
source of miscommunication because like
the same words mean different things to
different people um
samantha mentioned in her talk earlier
about the difficulty of defining privacy
i've got a slightly easier task
defining pen testing so for me
pen testing is
a type of test that follows a
methodology and the methodology looks
something like this so
if something doesn't follow this
complete methodology so like for example
if we're doing a threat and risk
assessment on an application
it's important work um you know straight
modeling is
a great critically important task
everyone should do more threat modeling
i love threat modeling
just doing threat modeling is not a pen
test and you know by the same token
if you specialize in vulnerability
research
super interesting i love it
it's not pen testing
and pen testing's not vulnerability
research
same you know like
despite some of the marketing materials
coming out of certain bug bounty
companies like bug bounties and not pen
testing and pen testing is not bug
bounties
and the same thing for you know
simple activities like vulnerability
scanning
this one in particular it can be a
little bit frustrating because there are
organizations out there that will
sell people of onscan and say here's
your pen test results
um
it's it's a bit of a challenge for our
industry because it's
not a pen test if you're not doing like
the full
the full activity
um and thinking about this methodology
an interesting thing for me at least
because i love nerding out about this
sort of stuff is that as you work from
from left to right throughout the
methodology um
you need humans more and more so
scripts
and
bots or you know ai if you in marketing
uh are really good at doing discovery
and enumeration
um but as you get
more and more towards the right you you
really need more and more humans because
humans understand
context humans understand risk
humans understand
other human factors they know how to
communicate better with other humans
so
pen testing does require
a human to do it like it you know
needs saying that you know you can't
just have fully automated pen testing
um
and so who who who is this human that
does it um can i just
hack my own stuff um the answer to can
you hack your own stuff is
yeah
yes
absolutely please hack your own stuff
like
pen testers did not have a monopoly on
security testing um they shouldn't have
a monopoly on security testing um
anyone can hack stuff like you don't
have to wear a black hoodie you know you
don't have to work for a pen testing
company uh everyone can hack
everything
that they have permission to hack
um so you know if it's if it's your own
system assuming you do have permission
to hack that so yeah absolutely do your
own security testing um it's not really
pen testing though what you're doing
though like it's it's security testing
but it's not a pen test in fact
the
person who builds the system is probably
actually the worst person to pen test it
because
you know about how you built it so you
have certain assumptions that you made
as you build it
um which means you're gonna have really
big blind spots
about it so
really this sort of testing should be
done by someone other than the person
who built it
right so
pen test follows a
methodology it needs to be done by
someone who specializes in it like you
can't just uh take any
person with a tech background and throw
a methodology in front of them and say
now pen test this like you do need
experience training um and at least for
the purposes of this talk um i'm gonna
say that a pen test is a sort of
engagement based activity and that it's
project you know it has a beginning it
has an end um
some people are interested in the
concept of continuous pen testing you
know
is that a pen test is that sort of a
different part of the abstract program
you're sort of arguing semantics
i'm just going to dodge that question
for today
and move into
why
why why why would we want to do a pen
test like i've seen a whole bunch of
different reasons
that customers have had for uh
for what i do it um
many many more than this
um
the most common one these days is that
it's really hard to sell a product
without having it pen tested um and
what's what's interesting uh there's a
lot of talk in some of the earlier
presentations about government
legislation and
this one hasn't so much been pushed by
government because business themselves
have got a lot more serious about it
like as part of
good governance you know if you're
buying a sas
if you're going to pay for a sas
services offer provider you know you can
say
can you show us your pen test report
please and if they say now what you're
going to say
um i'm going to go with another vendor
so it's sort of business to business
compliance is is driving a lot of
testing and security which
in my mind is a good thing
all of these have one thing in common
and that is they're all about risk
management right
so
what risk in particular it's the risk
that the system has vulnerabilities that
you don't know about
okay so
when
so when would you not do a pen test so
like if you already know
about vulnerabilities you don't need
someone else to tell you about it like
you need to be working on fixing those
um another question we get sometimes is
hey we've been hacked we don't really
know how it happens but can you maybe do
a pen test and figure out
the vulnerability so we can fix it and
it's like you don't you don't need pen
testers you need incident responders um
they're gonna do a much better job of
figuring out and you know if you think
there's a chance that there's still
people live in your environment you
don't you don't need more attackers you
need fewer attackers
so
you pen test when it makes sense to from
a risk managed point of view so
uh what high-risk scenario is like i've
never had a pen test done before you
don't have a security baseline it's like
you probably should do some testing um
i'm about to release a new product i'm
about to release a new feature
it makes sense to to have testing done
to lower your risk there
all right so
you have established that you probably
need to
get a pen test you need to know some
things about yourself so
what is it that you're trying to test is
like this is a pretty easy question to
answer and the obviously the more stuff
that's in scope the more work it's going
to take right that that should be pretty
obvious
slightly less obvious question is like
exactly what about it
do you want to know like what are you
trying to test so like in infosec we
always talk about the cia triad like
that's the get out of jail free card
it's the obvious one we want to test for
these sort of things um sometimes you
want to test for your ability to detect
and respond to attacks
um
unknown unknowns as old mate rumsfeld
called them like where are the gaps like
we think we've got everything secured
but like we don't know what we don't
know
um or like i'm not actually sure what
i'm trying to secure about this thing
and you know what that's that's actually
fine for an answer like if you don't
know like cool you've found a good
question that you you need to answer and
part of answering this is doing some
basic threat modeling
everyone seems a little bit scared of
threat modelling like i think it's sold
as this big complex
activity um it doesn't have to be like
so here's a mini talk how to do threat
modeling in 10 seconds
what are the bad things you want to
avoid congratulations you're doing
threat modelling done
all right if that was too fast
here's the 30 second version of it
what
might be affected
who might do it
how might they do it
cool you've done straight modeling
okay another thing to know about
yourself
is how mature is the target and
this requires like a little bit of
honesty like everyone likes to think
that their stuff is super cool and super
high level
um
it's not always the case like how
security focused are you like and and
you know honestly you know i are you
uh
handling super ridiculously important
data or is it just
you know uh
of medium importance
um how risk adverse or accepting are you
like
a lot of people seem to think oh we're
very risk
averse you know we're focused on
security it's like no if you're a
startup who you know has another startup
working on the same sort of idea
um and the most important thing for you
is to get product to market like your
risk accepting like
i'm not saying ignore security
but it's not your number one priority
um do you have a baseline for the work
um do you have a blue team do you have
people who are working in like detection
and response
it's fine if you don't some
organizations aren't big enough to
justify having a blue team but it's
these are things that you need to know
about yourself
when you're trying to decide
what's going to be the best um
the best way to engage and the best way
to run pen tests
and who's even gonna who's gonna do this
pen test so if you work for a big
multinational
if you work for google for microsoft
apple uh if you work for like one of
australia's big four banks um you
already have an answer to this question
like your pen test is going to be done
by your internal pen test team
if you work for any organization that's
smaller than those and doesn't have an
internal pen testing capability you're
going to get an external partner to do
it
and who are they so this this would if
this talk was a sales pitch this would
be the bit where i'd do the sales pitch
about my company um
i'm not going to do that that's fine
um i will say
in the
security industry in general so this
isn't just a pen testing
problem
assessing the quality of
security services
is a hard problem like
you know tech is a pretty hot industry
right now cyber security is a
particularly hot area of the tech
industry there's a lot of money um
and so there's a lot of
people attracted to that money
it's hard to tell
who's there selling snake oil
and i'm not going to be able to solve
that problem for you today
but i will emphasize that when you're
getting a pen test partner
um this is a really high trust
relationship like the people who are
performing pen tests for you
are going to have the sort of access
that you only give to the most trusted
members of your own staff
so you know someone who's going to
become a domain admin of your windows
enterprise someone who's might get root
access to your aws account
um
think about that and then think
how
how much do you want to know about the
people who are doing this like you
probably don't want some yahoo
to be your next domain admin
um
but as i said it's a it's a hard problem
to
to assess um our industry has tried
uh various different approaches to like
make it more transparent you know
there's different certifications for pen
testers
organizations have tried
certifying organizations
um like there's the crest program
i
without getting into it
none of them have been completely
successful
um there's problems with all of them um
but i will give you one bit of advice
that is pretty universal in that if you
want to know who's good talk to your
peers like
have you had pen testing done who did it
were they good what was your experience
like um
most of the good pen testing firms in
australia and internationally trade on
their reputation you know like we do
quality work
and our customers tell
their peers that they do quality work
and that's what sort of keeps the ball
rolling
but there are sharks out there
all right so
you can have a pen test done you've sort
of thought about who might do it
but the most important thing is working
out the
scope of the test
so if you google a thing called the
project management triple constraint
you'll find all these slightly different
variations of the definition of the
triple constraint but they all seem to
agree that the triple constraint comes
down to cost
time
scope
and quality
these
four things make up the triple
constraint
so
ignoring the fact that project
management makes no sense at all
time scope cost quality we can simplify
this a little bit in this uh in a pen
testing uh model because when you're
sort of working on a consultancy model
like whether they're
internal pen testers external pen
testers um you typically have a day rate
so time is cost that's simple um quality
is probably not that much of a variable
because if you say hey i just need a pen
test for compliance purposes can you
just do a really bad job of it please um
most pen testing companies will say um
no
because we put our name on the report
and then you show that report to other
people and if it's later shown to be
fraudulent we look really bad we get the
bad reputation
we don't get follow-on work from anyone
so
at least the reputable pentest companies
will not
do poor quality work and we're not
saying that you know this is one of the
areas where the perfect is the enemy of
the good like no
no test is there a hundred percent
complete it's all done from a risk
managed point of view
but quality doesn't really hopefully
vary that much so it pretty much comes
down to
scope is cost and cost is scope so the
more scope
there is the more it's going to cost so
getting that scope right is absolutely
critical
um now there's different types of
penetration tests you know talk about
white box testing black box testing red
team
red team is a relatively
misunderstood term there's this like
perception among some people that like a
red team is like a pen test but better
and that's not really the case
at all so to to better understand sort
of the difference between some of these
types of testing
thinking about
target's knowledge of attack and
attackers knowledge of the target
so a red team or at my work we call them
adversary simulations
to try and help alleviate some of the
misunderstandings um
and red team or adversary simulation
sits here
so it's very close to what would happen
in in the case of like real attackers so
target doesn't know they're being
attacked but also
that you don't get any insider
information
and uh like a white box pen test which
is where
the testers get given as much
information as possible about the
environment to allow them to do the test
it's sort of at the opposite end of the
spectrum up here
and a lot of people ask
like real world attackers don't get the
insider info like why would i give it to
pen testers like isn't that cheating
it's like yeah it's it's absolutely
cheating but if you come back to this
methodology of everything that's
included in a pen test we've only got a
certain amount of time to do all these
activities so if we can shrink the
amount of time spent on these initial
tasks and if you spend less time doing
discovery it means you can spend more
time doing volume discovery and that
that's one of the key differences
between a pen test and a red test uh red
team is that with a pen test you're
trying to find as many vulnerabilities
as possible in the time that you have
you're not trying to be like a real
world attacker whereas with a red team
you are trying to be more like a real
world attacker you're trying to achieve
a specific objective so that might be
you know
get access to the hr documents it's like
cool i might do that through phishing i
might do that through physically going
inside an office
might do it through you know hacking
your hr system there's a whole bunch of
different ways and red teaming is all
about chaining vulnerabilities together
to achieve that objective
but you're not trying to find as many
vulnerabilities as possible
on a red team engagement you're just
trying to achieve the goal
whereas with a pen test you are trying
to find as many vulnerabilities as
possible and so as a result you don't
work like a real world attacker like
real world attackers need to be a bit
stealthy
during a pen test you're not trying to
be stealthy
you're actually the opposite you're
being as noisy as possible like
you should be showing up all over logs
during a regular pen test
so this is this is the last slide about
red teaming
it's good for certain types of
engagements uh it helps flash out those
sort of unknown unknowns and if one of
your objectives is to test your internal
blue team capability
red teams
are what you want
but
it's more expensive it takes longer and
you're going to find less
vulnerabilities so unless you have the
maturity
to be running a red team it's not going
to be the best value and so
an obvious follow-up question is like
can we get a bit of both um can we do a
sort of hybrid approach
the answer is of course you know the
customer's always right we can do a
hybrid approach but it's probably not a
great idea to do it because you'll end
up doing things like
seeing if your blue team can detect
if there were pen testers in your
environment which
like who cares you want your blue team
to detect real type of attacks
there is another type of hybrid approach
called a purple team which is pretty
much a red team but the red team sits
down next to the blue team and they go
okay we did this
did you detect it let's look at your
logs together oh cool you found it in
logs all right now i'm going to do the
next thing
did it appear in your logs and it's sort
of the objective of a purple team is
leveling up
your blue team
different type of hybrid approach purple
team makes sense trying to do a hybrid
between white box and red teaming
doesn't make sense so much
other things that we can adjust in the
scope what type of environment are we
gonna
target for the test like
target all your environments
probably going to take too long
and there's no one-size-fits-all answer
to this like
um
it makes more sense to test in prod a
lot of the time but there's always that
risk that you might break something
during a pen test and cause availability
issues to prod but on the other hand
real world attackers test in prod and
they test and test and they test in dev
they'll try and target any of your
environments
this is something you need to work
through about what makes the most sense
for your specific environment
and
okay what sort of authentication are we
going to give the pen testers so a
really common misunderstanding is
hey so the thing that we're most
concerned about is
an unauthenticated attacker can do
bad things so like say you've got what
should be a mainly read-only website so
you know it's a brochure website it's a
blog
something that where most most of your
users are unauthenticated
um
does that mean that the type of testing
that you should be doing and the type of
access you should give to testers is
uh
just unauthenticated testing like in
intuitively you'd say yeah
but in reality
what you're really testing is
can unauthenticated
users
do things that authenticated users can
and the easiest way to test that is to
give the testers access
to an
authenticated account
and then they can see if they can
replicate that in an unauthenticated way
otherwise they're sort of like flailing
around a little bit in the dark trying
to figure out what functionality is
there that they might be able to access
so
similar sort of thing when it comes to
assessing what sort of roles you want to
include in a pen test like you say oh
look we trust our admins so we just want
to test unprivileged users it's like no
but what you want to test is if your
unprivileged users can perform
privileged
actions so again it makes sense to
include admin roles if possible now the
number of roles that you test
is going to have a huge impact on
how much testing is actually required
it's like related to the the handshake
problem you know like if you have two
roles two people as one handshake
three people is
three handshakes
it it goes up and up and up the more
roles that you include and
so if you've got something like this
this is from like
sap um
this is a matrix of like roles to users
and a security architect will say oh
this is this is really good this is nice
fine-grained principle of least
privilege applied and a tester will look
at this and go like oh my god like
you want me to test all of these like
this will take me a year so
at the end of the day you need to make a
risk-based decision on what makes sense
to test in terms of roles
pick the most
commonly used roles
and go with them is is usually the
approach that you go with um similar to
roles is like the interfaces of your app
so like you've got a user facing in
interface um some apps will have the
admins log into the same interface other
apps will have like a totally separate
out-of-band admin interface like do we
want to test that as well um
yeah you probably will like
fewer people use it but this is the
high-risk part of the app and uh this is
where a lot of people get popped
and people think cool this is our whole
application the user facing interface
and optionally the admin facing
interface it's like did you forget about
this little guy
because a lot of people don't consider
this in scope for when they're testing
systems and
when your infrastructure is in the cloud
that means potentially other people can
access that if you've misconfigured
anything in your cloud environment if
you've got open s3 buckets if you've got
roles not right
guess what you'll get hacked just the
same as if you've got hacked through the
front door of your web app the good news
for testing cloud environments is
because they're all the same
it's
relatively efficient to test cloud
environments these days
uh
your web app's got a waff in front of it
should you
test with the waff it's like the waf is
there to slow attackers down
in the real world you want to slow bad
guys down you don't want to slow pan
testers down the best way to approach
with waffs is
disable it for the testers ip addresses
um
if you want to know if the waff was
being
effective if they found issues get them
to retest those issues again um with the
wife present
all right
we've got the engagement uh we're
getting ready to start
best thing to do is talk it up like we
need to communicate there's some really
obvious things that we're going to talk
about so like pen testers are going to
tell you the source ip addresses
the people being tested are going to
provide credentials to the testers and
some documentation
but
some less obvious stuff is like
doing proper introductions so like don't
just say this is alice this is bob they
work on the team it's like this is bob
he's the project manager he's
responsible for the project schedule
like if something's going to affect the
project schedule
tell bob this is alice she wrote the api
she's your technical contact for
questions about the api
uh have a comms plan if we find a
vulnerability
how are you going to tell us about it
how do you want to know and then during
the engagement
follow that that story a great thing is
if you can get testers embedded into
your
your work slack uh or teams like
whatever you're using internally um
talk it up
keep change to a minimum during the test
like try not to keep changing things
while people are testing it
um
if someone reports a vulnerability
during the engagement that's actually a
pretty high risk scenario because
they've told you about it early so you
can fix it early but it also means that
you haven't had any risk analysis done
so keep talking to the pen test it's
like hey sue thanks for reporting that
this is the fix we're thinking about
will that work
if you've got any other advice like
keep the communication open uh post
engagement if people um like we often
offer people a
uh follow-up meeting
take take take the pen testers up on the
meeting if there's anything you're not
sure about um again we're thinking about
addressing this issue
what do you think about this approach
[Music]
so
in summary
the key themes i guess from all this is
it's it's uh
this is all about the relationship
between developers and pen testers and
the keys to any good relationship are
open clear communication pen testers
only pretend
to be bad guys we're not actually bad
guys we're actually on your side
so we like to work with you we're not
working against you
and that's me
thank you very much
thank you very much i'm amazed how well
you sped up at the end there without
losing too much that was pretty good
work
thank you
um there are a couple questions for you
so i'll get them across you for uh
answering in the chat
and uh we'll be back in about a little
under 15 minutes at 3 o'clock australian
eastern time with our next talk