foreign
[Music]
good afternoon and welcome to Security
in the open
incident detection and response for open
source with Aleister Chapman
Alistar is a senior Cloud security
engineer at Red Hat specializing in
incident response and security
architecture for public and hybrid Cloud
environments today Alistair will present
a broad rundown of all the new and
exciting ways that things can go wrong
in your security when working in the
open please welcome Alistar
[Applause]
[Music]
all right thank you and welcome everyone
to the Melbourne convention Exhibition
Center and the home stretch of
everything open we're getting to the end
there before we get into today I do want
to acknowledge the traditional owners of
this land the Warren Jerry were wrong
people of the cooling Nation pay my
respects to their Elders past and
present and dwellers of all First
Nations communities that join us this
week and recognize this land's
significance to those people for over 2
000 generations and recognize that while
sovereigns are for all of us to enjoy
sovereignty was never seeded
one more step before we dive in and
that's covering who I am my name is
Alistair Chapman I'm a senior Cloud
security engineer based in Brisbane
Australia I do work for red hat but to
be clear this is all my view so while a
lot of what I'll be covering today has
been Guided by my experience at red hat
if you don't like it complain at me not
my boss you can find me basically
everywhere is AGC 93 and so I get to
enjoy the wonders of open source
security from both sides I work in
security for an open source company and
as outside of work I tend to build a lot
of Open Source software because I guess
I hate myself
so
we'll get into a the trust me it's okay
this is the only bit of audience
participation in the entire thing
I understand this is all the Spectrum
but if I had to ask the audience to pick
one of these three that you would
identify with can I get a show of hands
who would be in the developer camp
big chunk operations non-code something
like that all right and any fellow
security
that's us that's just a smaller number
as I expected
okay so with that out of the way let's
cover what I want to talk about today as
a few of you may have noticed there's
been quite a few excellent talks in open
source and security already this week so
I'm going to be focusing a little less
on areas like data privacy or
vulnerability management and a bit more
on my area of expertise which is
internet protection instrument response
and the cloud
if you're looking for a lot more details
there's been a bunch of other
presentations this week that you might
find interesting
um particularly I don't want to call out
William Brown and Fraser tweedale both
did excellent talks on specific
Technologies and Alex Murray from
canonical did a really good talk
yesterday on vulnerability management
that for people building the code side
of their projects
might be particularly valuable so so we
can go too far so that's what we're
going through today the risks what makes
it different working in open source what
to watch for which is detection how to
actually handle when something happens
response and how you can control the
risks in your project and learn from any
incidents that you do have which is
protecting
so security for open source is not
completely different to security for
proprietary projects or corporate
software any of the traditional software
markets so there's plenty of useful
information to be gained from security
communities that being said the most
common things that are really specific
to open source in our in my experience
boils down to probably three principles
there's a really really blurry line
between what's private and public both
from an information from infrastructure
from code all those sorts of things
you're having to constantly deal with
contributions from untrusted third
parties Because the Internet is a wild
and dangerous place and your tooling and
infrastructure is much more likely to be
available to everyone than you might be
used to in for example a proprietary or
more corporate focused development
environment that doesn't work with as
much open source
but if we're going to talk about open
source security we need to look at the
number one source of problems the
Boogeyman in the night the Specter
around the corner the demon that haunts
the dreams of every infosec person who
has to work with open source software
and that is the horror that is get push
because I cannot explain how many of the
problems that you'll find in open source
projects comes down to someone get
pushing something they shouldn't have or
not get pushing something they should
have now it is a bit reductive so we'll
it'll come up a few times but the
explanation I'd like to give is that you
need to treat get push like you do so
you know to borrow the excellent lecture
message respect the privacy of others
think before you type and with great
power comes great responsibility because
there is a lot of things you can do to
your project with a simple git push that
probably don't seem so important at the
time but very much will be often too
late
so let's have a bit of a bird's eye view
on instant response process now I know
flow charts are kind of not really
everyone's idea of a good time and
probably everyone's getting flashbacks
to every bad corporate training session
they've been sat through but here is the
the overall bird's eye view of how
incident response gets treated these
steps are roughly paraphrased from a
guide from the US nist document 861 the
computer security incident handling
guide
if you really want to learn more about
incident handling and how security
operations teams do incident response
it's an excellent resource there's a lot
of really good information in there it's
also an 80-page document published by a
US government agency so it's not exactly
the most accessible of resources but if
you are interested in this 861 is very
good
you say probably don't need all the
details so we'll just look at this basic
of view which is that you start with
preparation that's where you're doing
risk analysis threat modeling looking at
your project working out what you need
to be worried about you move into
detection and Analysis that's how do you
spot when something happens what do you
what are you watching in your projects
to make sure that things are working the
way they should
when something happens how do you know
it happened when something does happen
you move into contain and recover that
is the oh God something's gone terribly
terribly wrong how do I
fit X whether that's getting an attacker
out of your project fixing a
vulnerability that's been introduced
trying to recover infrastructure that
may have been impacted by an outage
there's lots of possible things and
those steps feedback into each other so
as you recover whatever incident has
happened you'll find more things recover
those find more things
after everything's cleared up you can
move to post internet activity which is
the simplified version is that that's
where you learn from all the absolute
Horrors that you've just had to walk
through the lessons from the post intern
activity will then feed back to your
preparation so that you can ideally be
better prepared and have better
detection analysis in place to stop it
from happening again
so that's the overall view uh it's gonna
today is going to jump around a bit but
we'll try and stick roughly to that sort
of view of things
so for that first part preparation and
planning how do you plan for the worst
case scenario because it's very easy to
think Ah that's not going to apply to me
but sometimes it will and the wonders of
the internet and both the most positive
and least positive ways means that
you're there's not really any reliable
predictor of whether your project or the
team you work with is actually going to
be affected by this there's plenty of
projects out there that have one
maintainer and six users that are just
as vulnerable to someone trying to
inject malware into their package or a
massive organization with huge security
operations teams that still get done
because someone pushed their AWS key to
GitHub there's not really a clear
division of what makes a project more or
less at risk
which means it's not just about I should
stop and point out too it's not just
about your code that while everyone's
immediate instinct is vulnerabilities in
my code there's a lot more to your
average project as they grow gain
complexity users there's a lot of
expectations from users around how
projects operate that means you've got
things like documentation demo sites uh
projects running hosted versions of
their projects whether that's for
demonstration purposes or as actual
Services there's a lot of infrastructure
and tools surrounding a project that are
just as important as the actual code
there's plenty of scenarios where you
might be the greatest c-sharp or Haskell
or Java developer but you'll still
introduce problems because while you
might be great at those you're probably
not quite so crash out at terraform
and there's a lot of Open Source
maintainers uh
let's go with uh hesitant to look at the
full process of proper risk management
and that's for a good reason risk
management is kind of hard and not
something that a lot of developers are
going to enjoy you're going to sit down
and go I've got two hours to work on
some stuff I can bang out this really
cool re-implementation of a feature I
don't like from two weeks ago or I could
spend two hours reading PDFs about how
to do accurate threat modeling of my
project and this has made no no one's
doing that so there is a a tendency to
lean towards winging it
I'm going to say from my point of view
bad idea because it's much better to be
over prepared than underprepared when it
comes to security
so what should you be worried about and
honestly that's the most important
question you need to answer if there's
anything you take from today you need to
have an answer to that question because
the answer to that question will vary
based on your project your team what
your users do how your software is
implemented what other services you have
all those sorts of things it's it's very
nice to be able to sit around go how am
I going to be able to improve my project
security posture but the only way to
answer that is to know what you need to
improve so as an example if you're an
upstream supplier to a lot of other
packages you probably need to be more
worried about supply chain attacks where
someone inserts bad code into your
package to try and get to someone else
conversely if you're working in some
ecosystems malicious package
substitution might be more important in
our experience things like the python
pit repository or npm are magnets for
not necessarily poorly maintained but
small scale packages that people don't
necessarily think about not necessarily
conducting audits of being used to
introduce all sorts of Badness
the infrastructure takeover is much more
important if you do have a lot more
infrastructure if you're running like a
something that can be self-hosted but
also running a hosted version someone
taking that over is going to impact your
project just as much as the code that
you're shipping likewise if you're doing
open source work for a company if you're
open source component is just one small
part of a much larger potentially not
open
project or organization
people might be trying to get into your
project as a way to get to the rest of
whatever they're actually after
there's a lot of scenarios you need to
be prepared for so you need to sit down
look at your project look at your users
and think about what do I what am I most
worried about
once you have that answer it becomes a
lot easier to do other things including
the fact you need to think like the bad
guy here and the first thing the bad guy
is going to want to do is reconnaissance
reconnaissance is a very easy term for
infrastic people to throw around because
basically it just means looking for
stuff what you're looking for varies
wildly but you're looking for as an
attacker you're looking for ways to get
in ways to get to what you want what
might be in there that you might want
all sorts of things what defenses are in
place do as the target I'm attacking
going to have a proficient security team
who are looking for me or is it just two
guys on GitHub there's a lot of things
there the problem is that basically all
of those questions are answered by your
excellent documentation and all your
very detailed bug trackers and
automations and everything are telling
the attackers quit answers to all the
questions they need to know they're
going to sit down and say oh I wonder
how this thing's been built and if
they're needing to answer that question
for a proprietary clothes Source product
they're probably gonna have to dig
around the server headers and work out
what where the oh sorry let's have
headers compilation artifacts and
workout what server might be building
this yours is a nice big yaml file your
repo that tells you exactly what GitHub
running you're using there's a lot of
things you're giving away for free
because that's how open source works
that attackers are going to be able to
use against you that's not to say you
shouldn't be providing those things it's
that you need to be aware that
everything you know they also know
and it's about a lot more than just your
code all those things there developer
tooling Parts if you have contacts of
you if you have a problem with X
component talk to so and so even small
details like you'll see automations for
issues of you know this if you open an
issue and tag it with this component
it'll automatically say oh the last
person who committed to this was so and
so now an attacker knows exactly who
wrote that code and can look into that
in more detail
there's a lot of issues that can come up
because you're having to provide a lot
of information to contributors
which comes to the actual working in the
open side of it there's these are these
are a reasonably surface level one so I
suspect a lot of you would intuitively
understand that there's downsides to
work in an open source from a security
point of view you can't fix something
without telling everyone exactly what
you fixed there's a reason that for
example cves are frequently embargoed
and teams like ours will have to deal
with patches and fixes and roll out new
packages without ever revealing what it
was that we fixed because it's a nice
big shopping list to attack us of ah yes
this specific function in these specific
versions has a buffer overflow that you
trigger using this code we don't really
want to give that away while that's
still out in the wild so every commit
you make is a nice convenient summary
for anyone who wants to introduce
Badness or find problems in your project
of any mistakes vulnerabilities anything
like that
one of the other things that's easy to
forget is that there is no quick enough
that the way the internet works is that
if you have pushed a commit to GitHub
it's gone it doesn't matter if you fixed
it 30 seconds later because github's API
picked it up published it to anyone who
uses their API and it's out there so to
use the example that we see very
frequently API keys there are plenty of
people who will push their AWS API key
then like 30 seconds later oh [ __ ] fix
that and they'll even do a proper job
with Git remove all the old references
to the commit and it's great but it
doesn't matter because github's already
published the original commit to their
API feed and it's probably on Twitter
because there are so many Bots running
doing exactly that of just scraping
GitHub apis looking for keys and secrets
the other big one being the obvious one
of there's no boundary you can't rely on
your firewall because you're on GitHub
there's no convenient oh you know they
won't be able to operate in this
environment because this environment's
only for this it's the internet there is
no perimeter it's just [ __ ] Wild West
out here so the traditional wisdom you
might find around security projects
won't always apply to open source
projects that are particularly if you're
running your infrastructure in the open
as much as you are your code
one also is that I would specifically
call that did briefly mention earlier is
the outer loop things that it's easy to
forget your public bug trackers are a
One-Stop shop of vulnerabilities not
even just in the sense of finding
vulnerability and putting a bug in there
but it's also an easy way for an
attacker to see how often bugs are
popping up in a certain chunk of code if
there's a really crucial part of your
code that no one's looked at in three
years that's probably where they're
going to look at and likewise if there's
you know to borrow a talk from earlier
if you've got this great kotlin app and
then there's this weird nugget of go
laying code in there that you're not
really sure why it's there they're
probably going to look there because the
odds are high that it might not have
been reviewed as well just based on the
numbers
likewise while in corporate environments
it's very good to sit down and say okay
this is restricted information this is
confidential information this is you
know public but not entirely and this is
everything it's the Internet it's all
public there's no boundaries there so
even if your project has guidelines on
what information can and can't be shared
that will only take you so far
and things like build pipelines we'll
cover by build pipelines in a bit more
detail but it's very easy to forget that
every push of code you run isn't just
code being put in a git repo it's code
that's being executed on someone else's
server and doing things and reporting
that output potentially to the entire
internet so the issues you can run into
there are not always super obvious
there are a lot of ways to help that
yeah we'll depend about help absolutely
depend about is great it'll tell you
exactly when you've got vulnerable
packages tell you when you need to
update your builds for out of date
Runners all these sorts of things is it
going to stop a determined attacker from
finding a hole absolutely not so make
use of the things you can make sure
you're prepared for the stuff that you
can't necessarily fix
ing out the white noise is very
important if you're running a large
scale project you are going to get a lot
of reports particularly I don't
specifically if you're running a bug
Bounty program if you're on a bug Bounty
program this is 100 times worse you will
get reports for every [ __ ] reason
you've ever heard sorry excuse me and
it's going to happen that everyone will
tell you oh do you know your github's
public like we get those reports at Red
Hat we're aware we noticed
we you'll get reports of oh your bug
track is free I'm sure Mozilla is very
sick of finding out their bugs ill is
open to the public it's kind of the idea
all of those things that we take for
granted is just going to clog up your
reporting parts so it's not to say that
these reports aren't valuable it's that
you need to have a way of tuning out the
white noise so that you're finding out
the stuff that is actually important
because you really don't want to have to
deal with every time someone sends you a
thing saying do you know your Pub your
files are public we know that what we
need to know is did you know your jira
configs are public because you probably
didn't want those to be public those
sorts of things so it's particularly
important if you're working with bug
bounties or where your project is part
of a larger one where there's going to
be things that you don't want to be
public and things you do want to be you
need to have a reliable way in your
project of tuning out the terrible white
noise from people whose reports you
probably don't care about from the ones
that are very important
that's particularly important there's
been a couple references this week to
disclosure policies have a disclosure
policy so the people who are doing the
proper way doing a dual and have found a
real security issue can follow a proper
process added bonus you can immediately
close in issues that don't follow the
proper process and a lot of people are
not going to be following that and if
they're not the odds of their report
being good are quite low so that's an
easy way to get a lot of the white noise
out of the way at once
so now let's look at specifically
infrastructure because a lot of Open
Source work will be from people who are
very very very very very good at code a
lot of developers that do are not great
sys admins as a lot of hard learned
lessons in the devops movement have
found the skill sets are not the same
but if you are running a project that
has a lot of infrastructure around it
and the vast majority of Open Source
projects have at least some you need to
be aware of that infrastructure I was
saying earlier there's everything from
docs to demo sites hosted versions
there's lots of scenarios where you
might have more infrastructure than
you're expecting to have
even one of the smallest details will
come back to is domain names
so what should you be looking for and
there's a lot of text but it's not too
bad we'll cover keys and secrets a bit
more in a second but the important one
is that there is so many keys and
secrets and you need to have a way of
managing them
runaway bills can be a problem if you
have an AWS account attached to your
project or you're using a paid GitHub
plan that has unlimited build minutes
it's very easy for you to knock off on
Friday go down the pub come back and you
owe them 150 000 because someone found a
bug in your build pipeline you don't
want to have to deal with that because
they're not going to follow your work
schedule the odds of the attacker being
someone in your time zone working to
business hours is pretty low so you need
to be aware that that's going to happen
and be prepared with both
well-documented processes for people who
are in that time zone and automation for
when no one's around
service accounts can be an issue there's
a lot of issues a lot of scenarios where
developers like oh just spin up the
service account it'll be fine and that
convenience is very good it does come at
a cost and frequently that cost is
measured with the number of zeros that
comes after it because again it's very
easy if someone gets that service
account that unless you are looking for
it you might not notice for ages what
bills have been run up by that account
pull requests is an important one that I
think most people will understand but
want to call out specifically you can't
trust the internet just generally that's
good life advice and you really can't
trust pull requests on GitHub so things
like GitHub actions that will fire off
in every pull request that's good but
you need to be you while GitHub will try
and protect you it can't protect you
from everything and you need to be aware
that someone's probably going to try and
put in a pull request that tries to
print out all your environment variables
which probably includes all of your API
Keys things like that can be a problem
so you can Implement things like not
running not running build pipelines on
first contributions or only running them
after someone okay is a code review
things like that there's a lot of
scenarios you can do because you want to
avoid as much as possible running
someone else's code that you've never
looked at because they have looked at it
and it's probably not good
secrets Secrets management is hard
apparently
because
it's
again I cannot under it I cannot under
convey how many issue security incidents
and open source projects come from
people pushing credentials to GitHub uh
while I was doing this slide I yeah they
previously I'd had those names on the
bottom that was just grabbed directly
from the GitHub events API wasn't even
looking for it found someone's AWS API
key in the GitHub events API because
this happens now sometimes a lot of the
time sorry I should say it's genuine
honest mistake someone's trying to put
in a placeholder and accidentally puts
in the real one someone's putting in
test data and the test data actually as
it turns out isn't test data and it's
real data
putting in configuration examples in
your documentation say no here's where
you set your AWS access key ID and you
set it just like this with my access key
ID maybe don't
things like that it can happen very
frequently but just like there's so many
things in security if you plan ahead and
use proper Secrets management the odds
of someone pushing their credentials
direct to GitHub are a lot lower same
vein of the sort of infrastructure one
if you're running your docs on your own
server or you have a hosted work one
don't open SSH to the internet and
certainly don't open SSH to the internet
with root SSH enabled with a password of
one two three four five six because it's
going to get popped in like seconds yeah
the days of Internet of obscurity and
the internet have long since passed and
if you leave vulnerable if you leave
weak infrastructure on the Internet it's
not yours anymore it's probably someone
doing crypto Mining and it's saying
something where someone dropping a
crypto monitor on your server is
realistically the best case scenario
because it could be a lot worse
service management is hard and I will
admit it actually is hard but it's
important to have a plan beforehand
there's no shortage of really good even
open source password managers out there
even tools like GitHub and gitlab will
provide you a lot of tools to try and
make Secrets management easier use them
even in small teams sign up for bit
Warden use gitlab Secrets whatever you
want to do ansible vault is a nice easy
one where as long as you can share the
password you can dump as much as you
want in there and that file can be
public because it's all encrypted to
that there's a lot of ways to do it use
them
so that's planning mostly
it's all about actually dealing with
incidents there's two halves of that
which is finding out what happened
finding out oh hang on that doesn't seem
good to oh no this is real bad
there's there's a lot of scenarios here
and this is one where I'm going to have
to be fairly high level and you need to
use the threat modeling we were talking
about earlier to adapt to your project
and the risks that you're trying to
prevent
so if we look first at detection you
need to know there's a fire before you
can put it out and there's only so that
only works yeah the classic scenario if
you'll be able to smell the smoke no
github's got the smoke you're at home
and it's a great time you need to be
able to detect when something's happened
to your project
now there's plenty of static analysis
tooling to find problems in your code
there's ways to improve the security of
your specific implementation
while it's very satisfying to have a
readme absolutely chock full of badges
covering all the infinite variations of
coverage tools test Suites you know
browser compatibility testing that sort
of thing you need to applying the same
level of rigor to everything else about
your project your GitHub actions
pipelines should be validated just as
much as your actual code is every one of
your configuration files you implement
even test ones and examples should be
running through linters so that you
don't for example push your AWS API keys
to GitHub are you spotting a recurring
theme yet
there's just as many potential problems
in your project infrastructure as they
might be in your code so while static
analysis tools are very easy to
implement this stuff is a lot harder
because you need to tailor it to your
project
sometimes it's also very easy to forget
the smallest unit of risk which is your
account your project can have the
greatest thing on the planet but if your
GitHub account has a terrible password a
no two Factor who cares they're just
going to log in as you and turn off all
that good stuff so your accounts are
just as important as the project
infrastructure so use two-factor use
Uber Keys use web orphan there's a lot
of ways you can do to actually improve
your security viewer account that
prevents also reduces the likelihood of
someone getting into the project through
you add a bonus you can gloat over
whichever of your fellow maintainers did
get popped that your account's fine they
were the problem get get it get their
[ __ ] together one of the simplest things
you can use for this is emails that
every platform will send you so many
emails so many emails and it's very
tempting to just have a rule that says
does it come from GitHub send it to
Archive because I don't care you don't
want to do that you want to actually sit
down and set up which of these alerts do
I care about turn on the ones you do
turn off the ones you don't and then
actually read the emails because they
will tell you important things you're
depend about summaries you're probably
just archiving you get a lot of them but
finding out that there's a critical
vulnerability in your project is a lot
more important than finding out that
leftpad got updated again as a lot of
very gray area in there and you need to
make sure that you can find the things
that matter
so again identify the key risks I was
talking about earlier that you need to
match that to your project we can't tell
you what the answer is there use what
the platform offers whether that's
emails or a lot of platforms will have
their own apps that can do push
notifications all those sorts of things
whatever it is you're watching you want
to find out about it as fast as possible
make it hard to undo this one's
important because it's not always of a
huge importance to you know small
projects with small number of
maintainers working on some code but if
your alerts can be bypassed by someone
who's really trying to they're not going
to be much help so again if your GitHub
account gets popped they can just go in
and turn off all those alerts you'll
spend a lot of effort putting in and you
need to be prepared for that scenario
and it's where we'll come back later to
logging but things like alerting on if
there's no alerts for a while that can
sometimes be a sign of problems of Their
Own
so you need to watch what matters and
across the entire team particularly if
you have other maintainers on the
project
remember two is very easy to Shadow it
yourself and immediate presumption is
probably oh I don't I'm you know three
maintainers I'm not worried about Shadow
I.T it's very easy to Shadow I.T
yourself because you'll have set
something up two years ago at night
because you're worried about something
and you just installed a few packages
and I'll look now it runs did you
document that because I'm guessing you
probably didn't I never do and you've
just Shadow at yourself and those will
come back to haunt you in very real
terms
it's obviously much more of a risk and
larger projects and organizations but
keep it in mind and on that same vein
you want the safe process to be simple
because otherwise whether it's you or
another maintainer you'll do it the
unsafe way and you'll probably pay for
that later
it's an example even a project with a
handful of maintainers who owns the
domains for your project
have a DOT IO or whatever else very cool
catchy short domain name whose account
is that on and is that account well
protected because if it's not someone's
going to redirect your Project's
excellent nice little Hugo built
Dockside to malware and now all of your
users are getting served malware every
time they open your Dockside that's
probably not what you want because it's
not being properly managed so created a
teams account with whatever registry I
use put the password in that Secrets
management you definitely have set up
and you're already a long way to being
able to avoid a lot of those scenarios
so and likewise for maintainers prepare
for the scenario where maintainers out
if you're dealing with a problem with
your domain name and the only maintainer
who has the red has the login to the
registry is on holiday for the next
three weeks
do you have a plan for that scenario
because a lot of teams won't and that
can make things very difficult to deal
with
particularly when you get to the
incident response stage I.E you've got
alerts that say that's not good what are
you going to do now because a lot of
teams it can be very easy you'll be like
oh I'm sure it won't happen it might
and a lot of teams including ours where
I thought yeah I work in a full-scale
full size team of incredibly capable
security engineers and we still can't
catch everything not in a million years
so we get very good at cleaning up
afterwards you guys want to also be as
good as you can at that process
we talked earlier about making sure
you're getting alerts for important
events here's where that's important
because that alert the time it takes for
you to get that alert find out what it
means action it and clean up after it is
crucial the shorter that time window the
better because speed is absolutely key
you need to know the moment something
happens and be able to very quickly
identify is that a problem and what's
the problem because if you can't your
attacker's just free to do whatever they
want to do in the first place
attackers will have automation to use
the cloud as an example if you don't
think attackers are using terraform to
deploy stuff to pop to AWS accounts have
I got bad news for you there's a lot of
terraform out there you need to be not
just automating things but automating
things better than your the attackers
will be
we all love over engineering things over
engineering things is great more things
to over engineer right here automate
stuff
log everything logging everything is
absolutely crucial for reasons we'll get
back to as well because you need to be
able to not just tell when something
happened but what happened and the
faster you can piece together exactly
what they did and in what order the
faster you can undo that so the more you
can piece together and the quicker you
get a holistic view of that the faster
you can undo damage from a potential
attacker
Don't Panic though it's very easy to go
this is the first time I've had to deal
with security incident and someone's
just changed our dock site to serve
crypto miners to unsuspecting browsers
oh God what do I do I'm just going to
turn off the server no when the
proverbials hit the fan you want to be
turning off the fan not tearing it out
of the ceiling and it's important to
respond as quickly as you can but
without panicking and that is part of
why again the preparation phase is so
important if you have documentation or
processes on what to do when X you can
just follow that thing followed quickly
but just follow the guide it reduces the
chance of a mistake you make making
things worse that one also actually
applies to vulnerabilities how many
times did it take to fix log4j too many
because it's very easy to go oh God we
need to fix something and introduce more
bugs likewise things like backboarding
backboarding is hard because you can
bring new bugs with you when you cherry
pick future commits there's a lot of
problems there same applies to incident
response that you want to be trying to
fix the problem without adding more
one of the big ones that is easy to
forget for non-security people is to
preserve what you can because you won't
be able to learn anything from an
incident if you don't have any evidence
of what happened so to use the example
of AWS if someone's in your AWS account
spinning up crypto miners in every
region they have available
stop those instances don't terminate
them because you terminate them they're
gone if you've rotated the keys and you
terminate the instances you have no idea
how they got in it's probably going to
happen again in two weeks because you
won't have known what happened if you
stop those instances you can get a
snapshot of the disk have a look
sometimes it's very simple look at the
disk look at bash history and oh look I
had a terrible root password on SSH sure
sometimes it's quite difficult and
that's when you go find someone who's
into security and has a lot of free time
for some reason to help you with
forensics to find out what actually
happened
because you do want to know what
happened we'll get to that the next one
the other one which is a very Niche case
but particularly important for really
big projects plan for a hostile attacker
year to use the AWS example if you've
pushed your AWS API keys to GitHub and
someone's in your AWS okay
they're going to be really
really and you might go in and go ah
I've got them now and rotate that
credential you've got them did you
rotate the three new keys they made in
three different regions one of which
with a different name possibly not
you're now going to have to go in and
undo each of those in order because if
you leave even one there too long they
use that key to generate more keys
likewise there's a lot of other weird
junk you can do create a new instance
with an attached IEM role now unless
you've also turned off that instance
they still have IAM access to your
account
the scenario where you have an actual
active hostile attacker is not something
most projects are going to be dealing
with but if you think your project might
plan for it and that's where that
process is writing down the process
becomes very important because things
like order of operations is crucial you
need to be able to kick someone out and
have them actually stay out
which comes back to prevention how do we
learn from having made some terrible
terrible mistakes
you need to know how it happened that's
looking at where things like forensics
that you're not always going to need
forensics sometimes it's going to be oh
that's my AWS key in our readme that's
probably why sometimes it'll be really
complicated like there's examples of for
example generating passwords that happen
to be the same as default passwords
that's very unlucky but could happen
there's lots of things that you need to
be able to know how it happened if
you're going to stop it from happening
again the second much more important
part how can you stop it from happening
again
to use the example of if you've push
your AWS kids to GitHub how do you stop
yourself doing that things like get
leaks is an excellent way too and we'll
cover that of
what do you need to know to stop it from
happening again and not necessarily the
exact same one you don't want to set up
a rule to a pre-commit hook to say don't
push this string you need to say how do
I stop pushing keys to get up maybe I
could start using ansible a bit Warden
or whatever else likewise for
infrastructure
if I've got automatic automatic
pipelines running how do I stop not just
this attacker don't Blacklist the user
but change how the build pipeline runs
to maybe not expose so much valuable
information
and the last one is limiting impact no
prevention is perfect again I work in a
massive team full of incredibly capable
security people and we can't stop it
happening Optus couldn't Medicare could
medibank couldn't but that's its own
problem and
you want to be able to limit how bad the
impact is just as much as you want to be
able to stop it happening if you can't
stop someone getting into your AWS
account it would be good if you have
really restrictive cost caps on there if
you're if you need to run GitHub actions
on every single commit from a third
party untrusted contributor that's great
maybe don't include all of your API keys
with every build pipeline because you
don't want them to have access to those
things no prevention is perfect but
limiting impact will make it a lot
easier on you and your team to stop the
worst happening
summary of that is prevention is still
better you want to catch things before
they happen as much as possible and even
better is stop the actual root class of
the problem so to prevent pushing API
keys to GitHub things like get leaks if
any of you've anyone heard of get leaks
one that's good get leaks is an
excellent project that you can use to
set up as pre-commered hooks or you can
run it on your own infrastructure it's
very flexible and it will do things like
scan your git and it will tell you when
you push API keys and you can set up to
find a lot of things AWS keys if your
application has its own secret Keys you
know C values anything like that you can
set those up to say tell me before this
happens and there's a lot of different
ways of using get leaks and it's not the
only one that's get Secrets as a few
others there's lots of ways of
preventing things happening before they
make it to the internet because remember
from way back at the start once it's on
the Internet it's gone you've lost it
it's it's all over it's in someone
else's hands so catching things first is
always much better
that also comes back to things like
guidance and documentation if you don't
have developer guides and contribution
guides a there's a bunch of docs people
in this room who are coming for you and
you should this is a good example to get
those that if you have contribution
guides and developer guides explaining
how to set up the tooling properly and
in a safe way a your developer is going
to be much much happier just being able
to follow three steps in a markdown
document than just yellowing it and
hoping they set things up right
added bonus you can say you're going to
need this pre-commit hook to make sure
you don't commit your own AWS keys
make security part of the overall
project process as I'm saying you've
probably already automated a lot of
things sometimes it's something as
simple as having a small amount of build
pipeline running sometimes it's really
complicated stuff being able to deploy
entire environments from GitHub or from
gitlab maybe running Jenkins for reasons
I'm not going to ask you there's a lot
of scenarios where you might have a lot
of automation running make those
automations a secure the automations and
make sure that the processes that you're
automating are secure processes because
the only thing worse than a automated
system being taken over is an automated
system introducing something if you have
automated systems that just deploy code
with zero user interaction the odds of
it deploying a vulnerability are quite
High
now there are middle grounds with that
where there can be too much automation
if you automate so that it just like the
entire release process is zero human
interaction then if someone manages to
sneak in a vulnerability you have no
chance to catch it and that can be bad
and that's a really a threat modeling
thing of is that something you're
worried about maybe you have a
completely automated process with one
step in the middle it requires an actual
human to say you know that looks good
there's plenty of those available for
GitHub or gitlab things like that or if
you're dealing with a larger team have
that as part of your process that one
person doesn't do the release one person
does most of the release and a second
person checks so you can over automate
but automation's still better than not
some solutions just aren't technical I
am a very technical person but I still
have to spend a lot of my days telling
people to stop making bad decisions
because at the end of the day we're all
fleshy humans at the end of the keyboard
and we're the ones making the mistakes
sometimes we'll make the same mistake
over and over again and that's where you
have to break out the Technical
Solutions sometimes it's someone using
the technical things exactly right but
differently to how you thought they
would and that's a case where you've got
to stop and think do I teach people how
to do it the way I thought or do I
prepare my project for someone doing
things differently to how I do summer
Solutions aren't going to be technical
but I suspect a lot of people who are
working on open source are looking for
Technical Solutions wherever they can
the hammer and nail problem strikes
again sometimes you're just going to
have to deal with the fact that humans
are the ones interacting with your
project
so
what have we learned theoretically
hopefully
something here it is a lot more than
just your code preventing
vulnerabilities from making it into your
code is very important but it's not the
only thing planning ahead will 100 pay
off no matter how much you think this is
a bunch of wasted effort it might be and
that is the best case scenario the ideal
world is you spent hours and hours
crafting an Immaculate security response
process and if you never have to use it
hell yes win because take it from me
it's much worse when you do have to use
it
document everything automate what you
can it's where we go back to there's
limits to automation but document
everything as much as possible not just
so that you reduce the chance of
mistakes but so that you have documented
process for fixing mistakes
use tooling and processes to reduce your
risks things like build pipelines are
very good for not introducing bugs use
them to also not introduce
vulnerabilities or introduce
misconfigurations you need to be able to
use things like it leaks or get secrets
to try and minimize how often your
leaking Secrets or leaking information
that turns out shouldn't be on the
internet there's a lot of situations
there that you need to adapt to your
project and if anything I should have
put one at the top there the number one
is make sure it matches your project
there'll be no shortage of think pieces
online about here's the steps you should
take to secure your project I'm doing
one of those live right now you need to
match all of these guidance to what your
project needs and the threats you're
likely to be facing don't push your API
keys to GitHub I don't know how many
times I have to say this but apparently
more
everyone's laughing going ah I'm not
going to do that yes you are one of you
is going to
don't don't open SSH to the internet
common mistake but worth pointing out
but particularly in the era of public
Cloud it's pretty easy to think of an
instance in Euros Euro account as oh
yeah that's just mine if it's got an
internet connection it's on the internet
if it's got SSH maybe don't enable root
like to start with and if you really
need route do not have a password and
really do not have a bad password yes
like when you're thinking of it's still
bad
but there's a lot of things you can do
to just cover off the low hanging fruit
very very quickly
that was very rapid
photo if there's a lot of it sounds like
a lot of effort for not a lot of work
but again take me it's better to do the
work and not have to use it than to have
not done the work and be flailing around
in the aftermath going oh the mistakes
we have made
[Music]
[Applause]
I think we have a couple of minutes
[Applause]
we've got a couple minutes for questions
yep we've got six minutes we've got any
questions I'll bring the mic around
[Music]
good day thank you for that excuse me
um feels like one of the ways that we
need to improve our Securities by
practicing
these kinds of responses how can
developers practice
yeah right it's hard the to use the
example that we would use an actual
response team it's a very common tactic
to actually have tabletop exercises
where we'll sit down and run through you
know it's like any other tabletop
scenario someone says X has happened
what do you do okay now why has happened
what do you do so and so forth and
sometimes that's really complicated
setting up lab environments and testing
out how it works and having to discover
things you go but sometimes it's just
sitting down and going oh I don't do X
all right you found y all right I go do
Zed that can be hard particularly if no
one involved in the project has a lot of
security experience to be able to
formulate those scenarios but there are
a lot of Guides Online for running those
they're particularly good if you're
working in a team if it's small scale
ones one of the easiest ones I can
probably recommend is just reading all
the postmortems you get from incidents
that when companies have an incident
they'll publish this is what happened
and they'll have big timelines of we
responded with X Y and Z or correction
they should be publishing those things
because it helps increase trust but just
as crucial you can read those to find
out what they did and get an idea you
know to use some even some of the
highest profile ones the first thing
that happens is someone leaked a key or
someone had a bad password and you can
follow their timeline of okay they did
this how would I do that they did that
okay I'm never going to have that
scenario cool who cares what if it was X
so I think if you've got a larger team
and you really want to get into it
tabletop's a really valuable exercise
for smaller teams probably the easiest
low commitment one I can recommend is
just really going to some detail on some
of those post-mortems
[Music]
dynamically allocated as a lot of Home
setups do
um so what would you do in terms of SSH
you said don't open it up to the
Internet
so yeah there's a couple of options
there
um there are some software solutions
that if you that I I won't say get into
if you don't already know them because
they're a fairly advisable learning
curve but things like cloudflare tunnels
or other L2 mesh technology can be quite
useful because you can install that
agent then close it down as far as that
that's concerned and it's you know
without getting into too much complexity
it's essentially being vpned into that
host so that's one effect the other one
that's a very low-tech solution but
honestly not the worst is just every
time your IP changes go change your AWS
Security Group rules you know I'm also
on a dynamic IP but my IP only changes
probably at most once a couple every
couple of weeks I just go update those
Security Group rules
um the other one being that there is a
little bit of obscurity don't put on in
Port 22 don't open Root because the vast
majority of probate amps are on 22 to
root if you have a different
administrative user you've anyone who's
looked at failed to ban logs will have
seen that A administrative user with a
name other than admin or root is
probably not getting tried very often so
those probably the easy ones is see if
there's another method activity just
update your firewall rules when it
changes
chat and a little bit of obscurity
probably got time for one more anyone
else in fraserhead one down here
how can we test the logging and
monitoring and alerting yeah that one's
tricky as well
um one of the things I'll call out with
this specifically like being able to
tell when something doesn't happen is
what we would refer to as canaries if
you have if you're monitoring the logs
for a system make sure through whatever
method there's a lot of different ways
of doing it that you get a consistent
log event sometimes it's a heartbeat or
a canary or whatever that just sends a
single log line that says alive once
every four hours whatever then you can
alert that it's been more than four
hours we haven't got one that's a
problem for being able to pick up actual
Badness
it's a lot of it is just having to trust
your work some of it is if you have
enough evidence you can basically replay
the attack if you're if you have enough
evidence from an attack to say okay
someone used an API key to issue these
S3 calls and then did X
get one of your developers to do that
just say use your API key do these steps
and hopefully you'll get a bunch of
alerts show up
um there's it's hard to just give the
universal answer for how do we test
logging into learning but the short
version is probably make sure you're
actually getting the logs and have a way
of finding out when you are or are not
getting them and as much as you can try
to do the thing that you're supposed to
pick up and make sure you actually pick
up on it
we do have another one back there if we
have time yeah there we go
time for one more
it wasn't circuitry
something I noticed a lot with cloud
services and other sasers is that auto
locks are usually behind some sort of
enterprising paywall as an open source
project that perhaps you know wants to
make sure that all of us don't have a
various sector or something in them
is there
um any movement or something in terms of
getting the auto locks for products or
like
how do you do it when you knock on a 10
000 a year the Enterprise plan yeah so
that one's I thought it was I guess I
didn't know that I cut for time on that
basis that there is a problem with
logging which is yet where do you put
the logs that a lot of services will ask
to ask you a lot of money to run their
own Auto logging situations but you also
can't really just say oh just run your
own elk stack and secure that and put
all the logs there it'll be fine if you
can help go for that run run out run
Splunk whatever you want to use and
alert on that stuff but you don't always
want to have to do that in terms of
getting audit logs out of the platform
there's not a lot we can do there was a
talk earlier in the week on ocsf which
is a event standard that like for
example AWS can output Cloud trails in a
nice predictable schema that you can
then process with any number of things
like yeah you can set up with a log
stack you can also just run GitHub
actions on the outputs lots of other
things you can do
they have their own stupid format with a
lot of extra detail you don't need a lot
of details you may want but aren't they
all those sorts of things the short
version which isn't very helpful answer
to your question is there isn't a
convenient answer
um and you will have to sort of adapt
how you get audit logs to what logs
you're looking for and what you have
available to you if you can run your own
stack and secure it do it there if you
can't you might have to Jank it up a bit
with CI jobs or schedule jobs that sort
of thing is that answer roughly
okay yeah sorry so GitHub GitHub to use
github's example it's audit logs are
only available Enterprise plans yet
there's no solution for that
unfortunately that's just them being bad
people and those logs those those logs
are only available if you pay the money
we can't really solve that the best
thing I can give is that if you're in a
position where you can like if you are
doing open source at a company make them
pay for Enterprise because that's the
value add is whatever amount of money
you have to pay GitHub to get those logs
it's worth it for the thing you can
avoid from being able to process those
logs
anyone but unfortunately there's not
really I haven't seen any particularly
effective solutions to be able to get
logs from platforms that would otherwise
charge you for them best I can say is
you're kind of gonna have to pay them
sometimes or change vendor if that's an
option
thank you for that we'll have to wrap it
up there and thank you very much
Aleister thank you
[Applause]