foreign
I would like to introduce your speaker
this morning uh Tom Eastman
Tom works for the Kraken sponsor that we
have and he'll be he has attended every
Pike on Australia we've had so far
I think that's 10 years worth which is
great
um Tom loves to talk about many many
things his great person to have
conversations with so if you do want to
talk more about this talk afterwards
find him at the Kraken booth at
lunchtime and he will happily talk your
ear off
I'll let you go if you need to
um please provide a warm welcoming
Applause for Tom
[Applause]
hey I'm Tom um who has heard me talk
about secateur before in this room
quite a few of you cool cool because
it's this is the third talk that I've
given on this what was just like a silly
little Django side project
um that kind of grew a little bit out of
control
and so there's a lot of detail that I
might skip over and I'm more than happy
to talk about that stuff or you can
check out my previous two talks on the
subject
um just a bit of a Content warning this
is a talk about a tool to protect people
from social media harassment so I'm
probably going to be mentioning
some of the things some of the sorts of
harassment that kind of happen on
Twitter
um I'm not going to dwell on any of it
but that means that there might be
mentions of transphobia and homophobia
and stuff like that just because that's
the kind of awful [ __ ] that happens on
Twitter
um
the Project's dead now and we're going
to go into detail we're going to go into
a little bit of detail on that so this
is kind of just a recap of why I built
it what I learned along the way it's
kind of a grab bag of of anecdotes and
things that happened maybe some of this
will be useful lessons
um maybe it won't be some of it will
hopefully be a little bit entertaining
some of it will be a little bit surreal
I'm going to try
to play a tiny bit of a question that I
received at 2019 when I first launched
this uh tool
you there we go have you done any threat
modeling on sort of the impact that this
could directly Place Onto You by being
the creator of this app with like
because your GitHub name is out there
how like your DNS creates that kind of
stuff have you done anything to sort of
cover yourself when it comes to building
a tool like this that you know [ __ ]
with too much time on their hands can
just start and play with
a little but only in so far as my GitHub
account has also got 2fa on it most of
most of what's important to me has
reasonable protections on it my real
name is of course everywhere
um I am Tom Eastman that is actually on
my birth certificate I'm findable
someone could hack my run keeper and
possibly find out where I lived there's
there's interesting things once you
become a Target yourself
um I again get to okay I'm going to stop
there but that's a little bit of
foreshadowing we'll come we'll come back
to that
a little bit
um don't don't worry about it it's cool
um so what what is secretary secature is
a little tiny web app it uses the
Twitter API to allow someone to block a
person who's harassed on Twitter and all
of their followers for a set period of
time so if someone is attacking you on
Twitter and they have like 20 000
followers and they're sending their
followers after you
um you can hit one button and you can
block them and you can block all of
their followers
uh the
it gave them it and it did it for a set
period of time so um it kind of gave you
two advantages over the usual sort of
Twitter protection mechanism that they
had it gave you the ability to
neutralize a large group of people who
are sort of trying to Dogpile you which
is one of Twitter's
great Innovations and harassment
um and it gave you the freedom to do it
fairly ruthlessly because you knew that
like because you could block people
temporarily
um
you knew that if you accidentally
blocked nice people while you were doing
it they would automatically be unblocked
eventually and I found that to be a
really useful tool
um in my original talk on the subject I
mentioned that Twitter allows you to
mute topics for a set period of time
um so they like have mechanisms inbuilt
they understand the value of being able
to protect yourself temporarily from
like Star Wars spoilers but not from
Nazis I don't know
um
it was a really unsophisticated UI this
is basically the only uh interactive
piece of the entire app uh this is what
it looks like because I'm a pretty good
python programmer I'm pretty good
back-end programmer
and I am just I'm just useless at HTML
and CSS and ux it's just not what I do
so for the python developers here
this app wasn't a lot to write home
about okay four Django models sort of
mirroring some of what um Twitter's data
model looks like a couple a half a dozen
or a dozen generic Django views you know
a list detail model detail uh literally
one form as which you just saw
um a dozen or so celery tasks and that's
where some of the clever stuff happened
um a crappy bootstrap UI
um not a lot to it uh but it worked
pretty hard and the salary tasks were
probably the more interesting part I'm
gonna run through uh an encapsulated
timeline of how this came about uh I
first had the idea
um in 2016 at LCA actually
um when I was talking to a
online activist who had been involved in
protecting people from gamergate
um and I had the idea of like maybe this
would be something that would be useful
for people
and then I didn't do anything with it
for two years because ideas are easy and
implementation is hard uh in 2018 I
finally started writing code and I wrote
it very much because I wanted it
um Twitter in 2018 was just post
Charlotte charlottesburg there was like
a lot of ugliness there
um and yet it was a place where because
of my toxic relationship with social
media I spent a lot of time there
um I wrote it in Django but I wrote it
for myself and that meant that it was
largely command line interface it was
really just local database and stuff but
I knew that it would be a benefit for
other people so I knew eventually I was
going to get it online but that would
involve
writing HTML and writing CSS and you
know the stuff that I don't like doing
that much
um
most of the development for secatura
happened in fits and starts and like do
nothing on it for months and then like a
Sprint of like a day or a weekend's
hyper focused work or or a week's worth
of work a lot of it was conference
driven development which I'm sure
several of you are familiar with so it's
not a coincidence that it only finally
got launched like a couple days before
Picone Au 2019 because I was giving a
talk on it and that's like oh crap I
better actually finish this thing
um it had its first couple users who
weren't me by then a couple people who
needed to protect themselves from uh
transphobic attacks
um
and
effectively people first started signing
into it during my talk in uh at pikon EU
in 2019. by the end of the year it had a
couple hundred users
in 2020 things went a little bit more
nuts uh lockdown happened uh everyone
was spending more time on Twitter I
in about let's see is my mouse on screen
in about May
I stopped it requiring an invitation to
join like you could sign up but then I
would have to push a button to enable
your account and I just let people sign
up freely
um and I also
changed one feature which was if I go
back to that form I added the forever
button there because I thought that the
key feature was the temporary blocking
but I added the ability to let someone
block forever
um
usage started to spread and started to
get popular and
the server that I was running it on
um started falling over all the time
because I was running it on a T2 tiny
then it teaches small then a T2 medium
I had I I go into a lot of detail on
this in a in a different talk but um I
spent a lot of time
working on keeping it running on one
instance of one Amazon device because I
wanted the cost to be predictable when
you think about a technical platform
like this you can easily design it in
your head such that it would be like I
don't know you'd use lambdas to do the
API calls and you'd use dynamodb and you
could make it infinitely horizontally
scalable and therefore infinitely
expensive which is a bad idea for a
hobby project
um it crashed a lot in 2020 and so the
talk that I gave uh in 2020 at paikon EU
is probably the most technical one on
the subject because I I learned a lot
about tuning postgres and AWS gp2 iops
exhaustion caused by disk rights from
the database and RAM exhaustion swapping
to the disk
um
but yeah like I thought I had solved the
scaling problems I thought I thought I
didn't think it was going to get too
much more popular than that I'm going to
just zoom the graph out a little bit for
the context of where we were
um
in by the end of 2021 it had 30 000
users so it it it kind of it's it's
spread entirely by like word of mouth I
think
um and it was basically just doing quite
a lot of work
um I didn't do pretty much any
development on it in 2021 I was too busy
um by the end of 2022 it had 60 000
users and it was making roughly six
million API calls uh to Twitter per day
uh by this stage it was running on the
T4 medium you know the budget was
starting to creep up a little bit
um in April of that year I did like one
of those hyper focusing Sprint things I
took a week off work ostensibly to have
a holiday and then I spent the whole
damn week programming
um you know I upgraded at the Django 4 I
upgraded to postgres 14 I reworked the
database code to make it more efficient
um I bootstrapped the whole thing into
open Telemetry and started sending data
to Honeycomb which I'll talk about more
in a moment
um I finally realized that I was an
idiot for running this thing in Amazon
Sydney because like you know I live near
Sydney well in Wellington
um I should run things in Sydney it's
spending all of its time talking to the
Twitter API where's the Twitter API
California so I finally realized that
actually if you're if you're gonna build
something you might as well put it
anyway you cut a lot of costs by moving
it to a graviton instance and putting it
in Oregon
um
and I set up Django Waffle and I set up
a patreon
um
and then finally in 2023 Twitter began
if those of those of you who use Twitter
probably know it seems to be in
basically a cultural relevance death
cycle uh it started letting all of the
extremists back on Twitter while at the
same time sort of ending its ability to
deal with abuse uh and they announced
that they'd be deprecating all of their
free apis in a attempt to sort of get
more money from developers
um the free API was cut off I was
suspended from Twitter on a sorry my
developer credentials were suspended uh
on Twitter
mid-april one week after I started my
new job
um
and
every single login attempt an API call
started returning 500s it was
effectively dead so that was basically
the life of the app from 2019 sorry 2019
to 2023 it died anonymously at night uh
woke him in the morning it was broken
um
end of talk not quite
um
that's the gist of it
um I'm gonna give you sort of a grab bag
of things that kind of happened along
the way first of all
just some numbers for a side project
like this that got kind of popular it
cost me about
not sure if that number is right it cost
me about six thousand dollars to run
for its entire lifespan
um and about a year into it like in 2020
I started accepting donations
um
and the donations
added up to just over six thousand
dollars so I just barely didn't lose any
money on it
um
it brought in yeah so over three years
it grew to 70 000 users as you saw uh
since early 2020 when I started logging
these it made about four billion Twitter
API calls total
um
according to their current pricing plan
that would be pretty expensive
um that'd be in the millions of dollars
so it's one of the reasons why it's not
really
um if you're wondering the most blocked
person on the planet using the tool is
someone you've never heard of
um it's someone who it's probably like a
bot account because they follow like 200
000 people so it stands to reason that
like the person who got blocked the most
by a tool that blocks the followers of
people is someone who just seems to
follow everybody on Twitter
um
one user on secretary used the site 8
000 times
so in their in their usage of the site
they asked to block the followers of
people 8 000 times total another user
triggered 10 million API calls on their
own so they they probably blocked a
total of 10 million people on Twitter
um
so I'm going to talk a little bit about
why this was good for me I guess
um I learned a lot of stuff doing this
um some of it I was already pretty good
at and some of it I absolutely wasn't um
I got a lot better at using redis for
celery and quiche I learned a lot more
about tuning postgres for performance
um some pretty cool Advanced celery
patterns because the hard parts of the
celery work was um
uh handling the rate limiting and the
back off algorithms and just sort of the
Fanning out of
this call triggers 100 calls to get
pages of followers which then trigger
5000 tasks each to do the block the
blocks of the followers and sort of
tuning all that stuff
um I got a lot better at Docker and
Docker compose uh because that was the
production platform was just a Docker
compose file
um running on an ec2 instance uh my pie
for Django stuff struct log a lot of
these things I then brought into my
workplace
um which was really handy it was like oh
how did you know about this cool tool
well I built it over here struct log
open Telemetry honeycomb
um psycho pg2 instrumentation and
traffic for sort of load loading um
sorry front-end load balancing stuff
um okay
I'm going to jump topics a little bit
for the next little bit because I'm just
going to talk about you know my favorite
parts of what kind of happened here
um the reason it stopped falling over
dead was because I finally put user
Centric rate limits onto secature um by
far the biggest early mistake I made was
not setting per user usage controls so
that like this person over here
who really needs the app because they're
being attacked can't use it because this
person over here
has triggered the blocking of 15 million
people and so it's going to take six
days for that backlog to drain before
coming over here
um
this is not interesting code but it's my
favorite code on the whole thing because
it was sort of simple and elegant and I
got to use high school math
who here has ever used high school math
it's amazing
okay some of you have used high school
math fine
[Applause]
I I don't get to use high school math I
was very excited it's like it's got a
gradient you know like the the gradient
thing and like anyway
um the original the original account
controls on the site where
Tom has to enable your account first
that's the thing that I got rid of
pretty early you're not allowed to block
all of your own followers because that
would be a recipe for some pain
um and I had to have a limit on like
you can't block someone you can't block
all the followers of someone with like
over 500 000 followers because once you
get into the millions it's just not
practical with the tools that Twitter
gives you
um for example to block all of Donald
Trump's followers you'd have to with
Twitter's rate limits it would take
about seven weeks just to download the
list so it's just not really practical
um
but once I built this mechanism I was
able to keep the site online by tuning
this and that's far better than like
trying to work out oh if I add more
threads or if I switch to G event or um
or or is it finally time to do the
dynamodb Lambda thing
um
eventually with a lot of experimentation
on these rate limits what I settled for
was um
if you first signed up to secretary you
had a bucket and you and your bucket was
full and you could block a lot of people
with the tokens in that bucket and then
the tokens refilled quite slowly so
maybe you could block 200 000 people
when you first join
um but then from then on it only refills
at the rate of like five thousand a day
and that seemed to be a very
sustainable
use model for this
um but all of that just comes from
experimentation and what was really
valuable was just having the ability to
if someone came to me and said hey I'm
being attacked by this person over here
and they actually have a ton of
followers can you please help
knowing that there was capacity in the
system for me to just say like
anyone who ever asked me that I was able
to go yep go for it I've just refilled
your your rate limit just go nuts um
and it meant that all the sort of
drive-by background usage uh didn't
bring the whole thing down
I told you that I didn't do basically
any work on the in 2021 and that was
pretty much because
it got really unwieldy and scary to do
because the database by then was really
big and it was on a really small
instance and it was very difficult to
test production conditions it was
impossible to test production conditions
and I thought that that meant that I was
failing as a developer right because
you're supposed to be able to do good
testing do good unit testing do good
integration testing do good load testing
in practice
that's really hard
um and when I was trying to build new
features or things for this a single
sequential scan would basically bring
down the whole server because the disk
IO would be way too much for the tiny
instance it was running on
um and if you're using postgres as your
backing store you don't necessarily know
if you've built something that's not
going to use a sequential scan because
postgres's query plan or behavior
changes depending on the size of your
tables so when you are working from home
when you're working on your local laptop
with a couple hundred rows or a couple
thousand rows postgres is going to
behave differently to when you have 150
million or 2 billion rows in your
database uh so it's just very hard to
deal with
at LCA last year at the online LCA uh
Liz Fong Jones gave a keynote address on
observability engineering and it really
opened my eyes to some cool new stuff
um the first one was open Telemetry
which I'm going to be giving a talk
about at kiwi pycon next month which you
guys should come to it'll be really cool
um
I'm going to give a talk on open
Telemetry there open Telemetry gave me
the ability to see exactly where the
slow and fast parts were in the running
production code
um
but that's not the only topic that Liz
discusses in in her keynote because the
other aspect of observability
engineering is the tacit admission that
there's no such environment action you
simply can't simulate production
anywhere else and so you just have to
learn how to safely develop on
production she's talking about things at
the scale of Google and stuff but it
really counted for something like what I
was dealing with where
the um
the constraints that I was under were a
bit weird they were smaller but they
were weird because the database was
really big on a really small box
um to be able to develop on production I
put Django waffle in and started doing
feature-based development where I could
deploy it and I could roll it out to
just myself
make it work for just myself make it
work for two people make it work for
four then one percent of the user base
then two then four then eight
um
it meant that I was able to get a crap
ton done that I simply was too scared to
do previously so I felt like that really
changed what I was able to do and I've
I'm a real sort of evangelist for it now
um
foreign
built this tool for myself
but I knew other people would want to
use it but I was only half right about
what people wanted being able to block
all the followers of someone attacking
you was incredibly valuable
um
but I thought the killer feature was the
temporary block I thought that was
absolutely what made the tool useful and
nobody gave a crap about that everybody
the overwhelming majority of the use was
blocking people forever they just didn't
care about the one thing that I cared
about so I'm sure that a product manager
type person will
have a good lesson for you there about
measuring what people want I don't know
it I was I I built what I wanted and it
was mostly what other people wanted
it took me a little while to come around
to asking for money but I'm glad I did
because it didn't stay as cheap to run
as I would have liked
um I never intended to monetize the site
I never wanted to like charge people who
needed protection who needed the tool
but what I really hoped was that the
people who um
who who could afford to donate
would do so to support the people who
couldn't it worked out but not really
how I wanted I'm still a little bit sad
about this but um well sad and grateful
the overwhelming financial support that
came to the tool was people like
yourselves it was friends of mine who
knew me it wasn't necessarily the people
who were using the site so much
um that was an interesting lesson I
don't really know what to take from that
uh apart from thank you
um
thank you this couldn't really have it
couldn't have helped as many people as
it did without the support of of the
people who did financially support it
but it was yeah it was
it would have been like a lot of people
who gave me good financial support
didn't need the tool so I'm glad they
supported it and yet I kind of yeah
um
I've I discovered that some of the
weirder some of the use cases that
happened just weren't um
what I think even Twitter expected
one thing that I learned that caused me
a lot of problems with the site is that
um there's a gigantic subculture on
Twitter that are constantly changing
their usernames
like they put on and take off usernames
like hats in the morning they're just
constantly changing them and I get the
impression that Twitter didn't really
realize that either because uh
the oauth credentials invalidate
themselves every time someone changes
their screen names so that would break a
lot of the mechanisms on the site the
unblock on Secretary of the unblocking
thing
um and it just was not a use case that I
uh ever really expected
so
discussion that I had in 2019 about
how almost all
abuse protection mechanisms can be
weaponized for abuse and I felt that
this couldn't because I felt that
blocking people was one of the things
that couldn't really be weaponized
against someone because it really is
curating your own online experience
and I'm pretty sure I was right about
that but last year there was an
interesting thing that
there's there's an argument that could
be made that it
some people who I wish didn't use
secretary used it for a benefit so I'm
going to tell you about how my life got
a little bit more surreal uh about a
year ago
um during
the early days of the conflict of the
Russia Ukraine war
um there's a gigantic
group of people on Twitter who are
sort of harassing people who are
spouting a lot of Russian propaganda
um and
I discovered this when I discovered that
my tool secretary got really really
popular among Russian propagandists
for blocking the attacks from this other
group of people I don't know if these
are very readable but you don't really
need to worry about it too much things
started to get a little bit weird
uh a lot of people were starting to
notice that
this group of people were using secateur
and going okay well what's the deal with
that tool who's who's made that what's
going on
um I felt a bit weird about this because
like on the one hand I do not at all
support
the Russian invasion of Ukraine and on
the other hand I built a tool that was
designed to stop people from harassing
people on Twitter and this group of
people were you know harassment on
Twitter so it was just
people using the tool for what it was
used for
um
at a certain point they decided to start
looking into me
um
so I woke up one morning about a year
ago and suddenly had a whole lot of
things in my inbox about how I was
helping the Russians and like who was I
and was I a Russian spy uh
was I was I an agent for the Russians
things like that it got a bit weird they
started digging into my websites and a
few other bits and pieces
um
at one point they messaged some people
from who claimed to be from Anonymous
and those people sort of said oh look
like this guy's clearly not a very good
developer the SSL cert on his Blog has
expired uh he clearly doesn't care much
about security
um I was staying way out of this because
there was nothing that I could say that
would like alleviate it either side
because I thought either side was kind
of problematic
um
but I really wanted to reply and say
dude
I get I get an email every 90 days from
my friend Lee
to tell me that my certificate has
expired on my blog it's the only way I
know anyone and anyone still reads it
and so like
things got weirder because
on the Russian side they started saying
hey how come we never heard about this
tool until a few weeks ago maybe it's a
false flag operation by the ukrainians
to get rip addresses so and then so they
started um
sort of playing mind games with each
other about whose side I was on at this
point though I went and talked to uh ZX
security who uh run kiwicon and koi Khan
um and our friends of mine from
Wellington
um
just sort of go okay well look if this
becomes a bit more targeted like how
much of my data really is online I was
talking to Simon and he said well Tom do
you still live on um Acura drive and I
was like no but [ __ ] how did you find
that
um I had gone through like my link I've
gone through my LinkedIn my run keeper
all those things like strive I turned
off all the maps I turned off all the
friend sharing I turned off everything
except you know made everything private
mode for a little while
um but it turns out that my phone number
and my home address
my old home address because I hadn't
updated in a while was still like
registered on a bunch of my New Zealand
domains because New Zealand only
recently added privacy protections to
their who is database but you have to
opt into them uh and I neglected to do
so for some of my some of my personal
domains
um finally things got wow no we're
almost to the end of the weirdness
um
some of the nepho people then said to
the Russian people that I had spoken to
the gcsb uh
to hand over my database now I do want
to just point out that the database just
had public data in it anyway right like
just but
they started saying oh yeah I just got
off the phone with Tom Eastman in New
Zealand the developer of the tool and he
has said that he is in touch with the
special intelligence service
um
again like I just didn't reply to any of
this stuff but um
I I I worked in Security in Wellington
and you have friends who work in
Security in Wellington every once in a
while they work for they start working
for the government and you sort of oh so
what department of the government do you
work for and they get really cagey about
it they're like just the government
just the government and and so one of my
friends messaged me one morning he just
pointed to that tweet
and he said Tom what the [ __ ] is this
and I think the I think I replied with
like a a face palm Emoji
um but the the biggest irony of all this
is yes because of this weird [ __ ] uh I
did actually have someone from a
government Department come and talk to
me about uh they're like like
he yeah he was like Tom what have you
got yourself into like dude I don't even
I don't even know
um I'm almost out of time and I am
almost out of slides
um
I wish I had like a point to all this it
was worth it it was really worth it I
helped a lot of people I learned a lot
of things
some [ __ ] got a little weird
um
I'm grateful to everyone who helped me
do it uh
I just thought I think one of the most
important things that I learned though
is that there's a there's a emotional
and physical upper limit to how much
effort that you can put into
um to fixing someone else's trash fire
when they don't want it fixed you know I
put this effort into Twitter because
people I cared about relied on the tool
and were using it to communicate if I
was in the same position today I would
say no just get off the platform like
just just it's like the the the the the
degradation of the platform uh in the
last year or so has been such that I
don't think
there's the an ability to protect on it
um I'm out of time pretty much the one
other thing that I wanted to discuss but
I'm happy to talk about outside is um
I put I put some effort into trying to
keep it going after the credentials were
busted there was a couple avenues that I
went down
um with credentials to like old apps
that had kind of leaked and things like
that
um and for reasons it just wasn't going
to work
um and finally this happened today which
kind of tells you this this this gives
you a good clear indicator of where the
future of this platform is going anyway
right so
thanks everyone who helped me out with
this thanks everyone for listening now I
can talk about this for hours like any
any slide that you saw here I I love
talking about this stuff there's a lot
more that I learned
um but yeah thank you for uh letting me
thank you for listening
thank you