1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:16,160 --> 00:00:19,600
welcome back

3
00:00:17,600 --> 00:00:21,840
now we have lindsey homewood who is here

4
00:00:19,600 --> 00:00:25,439
to tell us how to make users of your

5
00:00:21,840 --> 00:00:27,119
cryptographic library successful

6
00:00:25,439 --> 00:00:28,240
lindsay is a product and engineering

7
00:00:27,119 --> 00:00:32,079
manager

8
00:00:28,240 --> 00:00:34,399
uh working at cypher stash as the lead

9
00:00:32,079 --> 00:00:36,559
or the chief product officer

10
00:00:34,399 --> 00:00:38,320
previously served as the head of

11
00:00:36,559 --> 00:00:40,000
technology at the australian federal

12
00:00:38,320 --> 00:00:41,360
government's digital transformation

13
00:00:40,000 --> 00:00:44,800
agency

14
00:00:41,360 --> 00:00:47,280
he also won third prize at the 1996

15
00:00:44,800 --> 00:00:49,760
sydney royal easter show lego building

16
00:00:47,280 --> 00:00:51,600
competition

17
00:00:49,760 --> 00:00:52,960
lindsay will endeavour to allow time at

18
00:00:51,600 --> 00:00:56,800
the end of his

19
00:00:52,960 --> 00:00:59,120
presentation for questions so please

20
00:00:56,800 --> 00:01:00,960
leave your questions in the

21
00:00:59,120 --> 00:01:02,000
question tab

22
00:01:00,960 --> 00:01:03,199
over to

23
00:01:02,000 --> 00:01:04,879
lindsay

24
00:01:03,199 --> 00:01:07,280
great thank you so much greg for that

25
00:01:04,879 --> 00:01:09,520
lovely introduction and uh just a big

26
00:01:07,280 --> 00:01:11,040
shout out and thank you to all of the

27
00:01:09,520 --> 00:01:13,040
organizers as well who have been

28
00:01:11,040 --> 00:01:15,040
involved in in putting this production

29
00:01:13,040 --> 00:01:17,040
on it's pretty amazing to watch behind

30
00:01:15,040 --> 00:01:18,799
the scenes uh you know how much effort

31
00:01:17,040 --> 00:01:20,880
and and hard work they're putting in to

32
00:01:18,799 --> 00:01:23,280
to create an experience like this so

33
00:01:20,880 --> 00:01:26,640
big uh big hat tip to all of you

34
00:01:23,280 --> 00:01:28,400
so my name is lindsey homewood i am the

35
00:01:26,640 --> 00:01:30,320
chief product officer at cypher stash

36
00:01:28,400 --> 00:01:32,640
where we are trying to build

37
00:01:30,320 --> 00:01:35,280
uh a new searchable encryption

38
00:01:32,640 --> 00:01:37,280
technology uh commercializing that uh

39
00:01:35,280 --> 00:01:38,479
and so a big part of my job is focused

40
00:01:37,280 --> 00:01:39,920
on how do we actually make the

41
00:01:38,479 --> 00:01:43,119
cryptography technology that we're

42
00:01:39,920 --> 00:01:46,159
building uh usable by our users

43
00:01:43,119 --> 00:01:48,320
and uh in a product role uh typically uh

44
00:01:46,159 --> 00:01:50,399
you can go out and you can do you know a

45
00:01:48,320 --> 00:01:51,360
you know maybe a less technology focused

46
00:01:50,399 --> 00:01:52,880
company

47
00:01:51,360 --> 00:01:54,880
you do a lot of interviews with users

48
00:01:52,880 --> 00:01:56,479
but it's a bit of a different sort of

49
00:01:54,880 --> 00:01:59,040
set of challenges i guess when it comes

50
00:01:56,479 --> 00:02:00,399
to working with developers and looking

51
00:01:59,040 --> 00:02:01,840
at how they're writing code and how

52
00:02:00,399 --> 00:02:04,079
they're using the code that we're

53
00:02:01,840 --> 00:02:06,560
writing and shipping to our users so i

54
00:02:04,079 --> 00:02:08,720
spend a lot of time looking at academic

55
00:02:06,560 --> 00:02:10,879
research about

56
00:02:08,720 --> 00:02:13,599
cryptography cryptography usability

57
00:02:10,879 --> 00:02:16,080
overall and fortunately this intersects

58
00:02:13,599 --> 00:02:17,440
really really well with open source

59
00:02:16,080 --> 00:02:19,440
because pretty much all of the research

60
00:02:17,440 --> 00:02:21,440
that we've had into cryptography the

61
00:02:19,440 --> 00:02:23,920
security and the usability side of it is

62
00:02:21,440 --> 00:02:26,080
based around open source technologies so

63
00:02:23,920 --> 00:02:28,959
i've read probably close to 50 papers in

64
00:02:26,080 --> 00:02:30,800
the last couple of months about uh open

65
00:02:28,959 --> 00:02:32,560
source cryptography and the usability of

66
00:02:30,800 --> 00:02:34,480
that and what i'm going to be doing

67
00:02:32,560 --> 00:02:36,080
today is sort of taking you stepping you

68
00:02:34,480 --> 00:02:37,360
through sort of the highlights and the

69
00:02:36,080 --> 00:02:38,959
low lights of it

70
00:02:37,360 --> 00:02:40,800
in case you're wondering about what what

71
00:02:38,959 --> 00:02:43,120
the hell a foot gun is um it's sort of

72
00:02:40,800 --> 00:02:43,920
like an old unix sort of program a joke

73
00:02:43,120 --> 00:02:46,000
of

74
00:02:43,920 --> 00:02:47,680
it's a feature that is likely to shoot

75
00:02:46,000 --> 00:02:49,519
users in the foot

76
00:02:47,680 --> 00:02:51,680
which obviously we don't want to do when

77
00:02:49,519 --> 00:02:53,040
we're when we're doing cryptography and

78
00:02:51,680 --> 00:02:55,280
helping people be successful with the

79
00:02:53,040 --> 00:02:56,959
libraries that we're building

80
00:02:55,280 --> 00:02:58,239
but before i start i would like to

81
00:02:56,959 --> 00:02:59,680
acknowledge the traditional owners and

82
00:02:58,239 --> 00:03:01,680
custodians of country throughout

83
00:02:59,680 --> 00:03:03,519
australia which in my case is the dark

84
00:03:01,680 --> 00:03:05,440
and young people and recognize their

85
00:03:03,519 --> 00:03:07,040
continuing connection to land waters and

86
00:03:05,440 --> 00:03:09,360
culture we pay our respects to their

87
00:03:07,040 --> 00:03:10,959
elders past and present sovereignty has

88
00:03:09,360 --> 00:03:12,800
never been seeded and trees have never

89
00:03:10,959 --> 00:03:16,000
been signed it always was and always

90
00:03:12,800 --> 00:03:18,480
will be aboriginal land

91
00:03:16,000 --> 00:03:22,159
so cast your mind back to

92
00:03:18,480 --> 00:03:24,080
2020 um you might not want to because 20

93
00:03:22,159 --> 00:03:25,760
things have not been great since 2020

94
00:03:24,080 --> 00:03:27,440
but um let's talk about something

95
00:03:25,760 --> 00:03:29,120
particularly dark that happened in 2020

96
00:03:27,440 --> 00:03:31,280
you might not be aware of if you thought

97
00:03:29,120 --> 00:03:34,239
it was bad already just wait you to hear

98
00:03:31,280 --> 00:03:36,720
about this so there was a health tech

99
00:03:34,239 --> 00:03:38,640
startup in finland named vestamo that

100
00:03:36,720 --> 00:03:42,640
suffered a fairly significant data

101
00:03:38,640 --> 00:03:44,560
breach so they had over 300 000 patient

102
00:03:42,640 --> 00:03:47,280
records leaks and those were including

103
00:03:44,560 --> 00:03:49,440
detailed consult notes written by a

104
00:03:47,280 --> 00:03:52,560
psychologist or psychiatrist during

105
00:03:49,440 --> 00:03:55,040
during a session with a patient and this

106
00:03:52,560 --> 00:03:57,760
information was leaked on the internet

107
00:03:55,040 --> 00:04:00,000
and actually used to extort patients of

108
00:03:57,760 --> 00:04:01,599
this particular service

109
00:04:00,000 --> 00:04:04,640
so this had a really

110
00:04:01,599 --> 00:04:06,640
tangible human impact on the people that

111
00:04:04,640 --> 00:04:08,080
were involved in this data breach

112
00:04:06,640 --> 00:04:09,680
so this particular person from this

113
00:04:08,080 --> 00:04:12,720
article i highly recommend you give it a

114
00:04:09,680 --> 00:04:14,879
read from william ralston in wyatt

115
00:04:12,720 --> 00:04:17,199
this particular person got a message on

116
00:04:14,879 --> 00:04:19,199
on snapchat saying hey your data isn't

117
00:04:17,199 --> 00:04:22,400
is in this massive breach um and this is

118
00:04:19,199 --> 00:04:24,560
from uh when he used uh vestamos

119
00:04:22,400 --> 00:04:25,919
services uh a couple of years prior but

120
00:04:24,560 --> 00:04:27,600
hadn't hadn't been using it so you know

121
00:04:25,919 --> 00:04:29,199
these are these are people that have had

122
00:04:27,600 --> 00:04:31,120
significant trauma in their life and

123
00:04:29,199 --> 00:04:33,440
significant mental health issues and

124
00:04:31,120 --> 00:04:36,479
this information was just leaked right

125
00:04:33,440 --> 00:04:38,479
out there um and the uh the people that

126
00:04:36,479 --> 00:04:40,560
did this took one step further they

127
00:04:38,479 --> 00:04:42,240
actually were blackmailing people and

128
00:04:40,560 --> 00:04:45,120
saying hey you need to pay us a bit of

129
00:04:42,240 --> 00:04:46,880
bitcoin uh for us to permanently delete

130
00:04:45,120 --> 00:04:49,280
this information in the data breach that

131
00:04:46,880 --> 00:04:52,479
we've uh that we've that we've just done

132
00:04:49,280 --> 00:04:54,960
so a pretty bad situation overall

133
00:04:52,479 --> 00:04:56,479
um and when the researchers and the

134
00:04:54,960 --> 00:04:57,759
incident responders dug into it

135
00:04:56,479 --> 00:05:00,400
afterwards

136
00:04:57,759 --> 00:05:02,479
um there was a pretty glaring flaw in

137
00:05:00,400 --> 00:05:03,840
the end in their entire system which is

138
00:05:02,479 --> 00:05:05,919
it didn't encrypt

139
00:05:03,840 --> 00:05:08,000
any of the data at all in fact it was

140
00:05:05,919 --> 00:05:10,160
just a plain old i believe was a mysql

141
00:05:08,000 --> 00:05:11,680
database uh with a with a php based

142
00:05:10,160 --> 00:05:14,160
weber

143
00:05:11,680 --> 00:05:15,680
and yeah just some very basic network

144
00:05:14,160 --> 00:05:17,440
access controls

145
00:05:15,680 --> 00:05:19,280
but you know there was no encryption in

146
00:05:17,440 --> 00:05:21,840
the mix in any of this

147
00:05:19,280 --> 00:05:23,600
and you're probably thinking now wow

148
00:05:21,840 --> 00:05:25,919
that's pretty messed up but i would

149
00:05:23,600 --> 00:05:27,360
never do this because i am smart and

150
00:05:25,919 --> 00:05:30,000
intelligent and i am at a conference

151
00:05:27,360 --> 00:05:31,440
like this like lca and taking time out

152
00:05:30,000 --> 00:05:33,360
of my weekend

153
00:05:31,440 --> 00:05:35,120
and i would i would never if i was put

154
00:05:33,360 --> 00:05:36,639
in a position to to build and design a

155
00:05:35,120 --> 00:05:38,639
system such as this i would i would

156
00:05:36,639 --> 00:05:41,039
never uh build it in this way that i

157
00:05:38,639 --> 00:05:41,919
wouldn't use any encryption

158
00:05:41,039 --> 00:05:43,199
and

159
00:05:41,919 --> 00:05:46,160
the problem with this is it doesn't

160
00:05:43,199 --> 00:05:47,520
really matter what your intentions are

161
00:05:46,160 --> 00:05:49,840
in this particular study which we're

162
00:05:47,520 --> 00:05:51,360
going to talk about a little bit later

163
00:05:49,840 --> 00:05:53,440
that was comparing the usability of

164
00:05:51,360 --> 00:05:55,360
different cryptographic apis uh the

165
00:05:53,440 --> 00:05:56,960
researchers found that uh in these

166
00:05:55,360 --> 00:06:00,160
programming exercises where they had

167
00:05:56,960 --> 00:06:02,479
people uh writing python to encrypt and

168
00:06:00,160 --> 00:06:05,120
manage data they found that 20 of the

169
00:06:02,479 --> 00:06:06,560
submitted code was insecure but it was

170
00:06:05,120 --> 00:06:08,960
even worse than that because the

171
00:06:06,560 --> 00:06:10,960
developers who submitted that code

172
00:06:08,960 --> 00:06:13,440
strongly believe that that code itself

173
00:06:10,960 --> 00:06:15,120
was secure so it may not matter that you

174
00:06:13,440 --> 00:06:16,639
might say to yourself i would never do

175
00:06:15,120 --> 00:06:18,560
this the fact of the matter is that the

176
00:06:16,639 --> 00:06:20,400
way that we build

177
00:06:18,560 --> 00:06:23,120
cryptography tools and cryptography

178
00:06:20,400 --> 00:06:26,080
libraries directly impacts the security

179
00:06:23,120 --> 00:06:28,560
outcomes for our users or the developers

180
00:06:26,080 --> 00:06:30,400
and also their end users as well

181
00:06:28,560 --> 00:06:31,840
i think mark twain sort of said it best

182
00:06:30,400 --> 00:06:33,199
where he said it ain't what you don't

183
00:06:31,840 --> 00:06:34,800
know that gets you into trouble it's

184
00:06:33,199 --> 00:06:35,600
what you know for sure that just ain't

185
00:06:34,800 --> 00:06:37,360
so

186
00:06:35,600 --> 00:06:38,800
um so you might be super confident about

187
00:06:37,360 --> 00:06:39,840
this but it doesn't necessarily even

188
00:06:38,800 --> 00:06:42,319
matter

189
00:06:39,840 --> 00:06:45,039
um and the great news for us is that

190
00:06:42,319 --> 00:06:46,479
there is this burgeoning field um in the

191
00:06:45,039 --> 00:06:48,479
humanities it sort of lives at the

192
00:06:46,479 --> 00:06:50,319
intersection actually of comp sci and

193
00:06:48,479 --> 00:06:52,240
humanities called human factors and

194
00:06:50,319 --> 00:06:54,319
secure software development um and

195
00:06:52,240 --> 00:06:56,400
there's been this massive explosion of

196
00:06:54,319 --> 00:06:58,000
research over the last sort of 10 15

197
00:06:56,400 --> 00:07:00,160
years in particular but i've seen you

198
00:06:58,000 --> 00:07:02,720
know in looking at all these papers i've

199
00:07:00,160 --> 00:07:04,960
seen at least three uh doctoral

200
00:07:02,720 --> 00:07:07,039
dissertations in the last six months uh

201
00:07:04,960 --> 00:07:09,680
for people that are uh that are trying

202
00:07:07,039 --> 00:07:11,520
to acquire their phd um in human factors

203
00:07:09,680 --> 00:07:13,520
and secure software development so

204
00:07:11,520 --> 00:07:15,840
there's a rich vein original resource

205
00:07:13,520 --> 00:07:18,560
that we can tap into here to learn what

206
00:07:15,840 --> 00:07:20,479
the problems actually are and how we can

207
00:07:18,560 --> 00:07:22,000
not make those mistakes and only make

208
00:07:20,479 --> 00:07:23,440
new mistakes

209
00:07:22,000 --> 00:07:24,560
um so what we're really going to do is

210
00:07:23,440 --> 00:07:26,880
we're going to look at the usability

211
00:07:24,560 --> 00:07:28,400
traps in the cryptography libraries some

212
00:07:26,880 --> 00:07:30,560
of which you may be familiar with others

213
00:07:28,400 --> 00:07:32,639
you may not be uh and the sort of things

214
00:07:30,560 --> 00:07:33,520
that you can do to help your users avoid

215
00:07:32,639 --> 00:07:35,759
them

216
00:07:33,520 --> 00:07:38,160
uh and before we begin just a a quick

217
00:07:35,759 --> 00:07:40,720
note we are talking about cryptography

218
00:07:38,160 --> 00:07:42,880
not crypto and cryptocurrency uh we

219
00:07:40,720 --> 00:07:44,720
don't serve any of their kind here um

220
00:07:42,880 --> 00:07:46,639
and we're talking specifically about

221
00:07:44,720 --> 00:07:47,840
developers when we say the users of a

222
00:07:46,639 --> 00:07:50,000
particular piece of software we're

223
00:07:47,840 --> 00:07:52,639
talking about those being developers uh

224
00:07:50,000 --> 00:07:54,319
not the actual end users who may be you

225
00:07:52,639 --> 00:07:56,560
know using uh using a high level

226
00:07:54,319 --> 00:07:59,280
application to be able to store uh their

227
00:07:56,560 --> 00:08:00,639
data and perform a particular task

228
00:07:59,280 --> 00:08:02,800
the last thing as well to note is that

229
00:08:00,639 --> 00:08:04,639
we're looking at peer-reviewed research

230
00:08:02,800 --> 00:08:06,080
much of it has been replicated as well

231
00:08:04,639 --> 00:08:07,599
and i've got an extensive list of

232
00:08:06,080 --> 00:08:08,800
sources at the end that you can go and

233
00:08:07,599 --> 00:08:10,560
have a look at yourself if you want to

234
00:08:08,800 --> 00:08:13,440
dig into the details

235
00:08:10,560 --> 00:08:15,120
really thorough complicated interesting

236
00:08:13,440 --> 00:08:16,639
work and if you've got some time you're

237
00:08:15,120 --> 00:08:18,960
looking for some light reading on a

238
00:08:16,639 --> 00:08:20,400
sunday afternoon um i highly recommend a

239
00:08:18,960 --> 00:08:22,560
lot of the papers that we're going to be

240
00:08:20,400 --> 00:08:24,560
talking about today

241
00:08:22,560 --> 00:08:26,400
so i want to start with a base level

242
00:08:24,560 --> 00:08:27,680
fact which probably makes a few of you

243
00:08:26,400 --> 00:08:28,879
uncomfortable in fact you know i'm

244
00:08:27,680 --> 00:08:30,720
probably not talking to the people that

245
00:08:28,879 --> 00:08:32,399
are in the audience here today because

246
00:08:30,720 --> 00:08:34,800
if you care about security you're

247
00:08:32,399 --> 00:08:37,200
probably at this talk but on the whole

248
00:08:34,800 --> 00:08:39,200
developers pretty clearly don't really

249
00:08:37,200 --> 00:08:41,120
think about security at all

250
00:08:39,200 --> 00:08:43,440
so there's this fantastic paper uh

251
00:08:41,120 --> 00:08:45,519
published in 2014 called it's the

252
00:08:43,440 --> 00:08:47,120
psychology stupid how heuristics explain

253
00:08:45,519 --> 00:08:49,200
software vulnerabilities and how priming

254
00:08:47,120 --> 00:08:52,720
can illuminate developer blind spots

255
00:08:49,200 --> 00:08:53,519
this resources is led by daniela olivier

256
00:08:52,720 --> 00:08:55,519
and

257
00:08:53,519 --> 00:08:57,120
this is a really interesting experiment

258
00:08:55,519 --> 00:08:58,480
we're going to sort of rapid fire

259
00:08:57,120 --> 00:09:00,800
through a bunch of different papers here

260
00:08:58,480 --> 00:09:02,480
today so in this particular experiment

261
00:09:00,800 --> 00:09:04,480
what they did was that they showed

262
00:09:02,480 --> 00:09:06,399
developers six different code examples

263
00:09:04,480 --> 00:09:07,519
that contained vulnerabilities so the

264
00:09:06,399 --> 00:09:09,440
typical sort of things that you're

265
00:09:07,519 --> 00:09:13,519
probably familiar with like buffer

266
00:09:09,440 --> 00:09:15,839
overflows tls setup cross-site scripting

267
00:09:13,519 --> 00:09:18,000
ii time to check the time of use sql

268
00:09:15,839 --> 00:09:20,399
injection and brute force exhaustion so

269
00:09:18,000 --> 00:09:22,399
what they did was they showed these

270
00:09:20,399 --> 00:09:24,720
insecure code examples

271
00:09:22,399 --> 00:09:27,120
to to uh to participants in this

272
00:09:24,720 --> 00:09:28,800
particular study who are developers and

273
00:09:27,120 --> 00:09:31,040
so the developers in this particular

274
00:09:28,800 --> 00:09:33,360
study were grouped into three different

275
00:09:31,040 --> 00:09:35,680
cohorts so there was control there was a

276
00:09:33,360 --> 00:09:38,640
priming cohort and there was an explicit

277
00:09:35,680 --> 00:09:40,560
cohort and so in the control cohort they

278
00:09:38,640 --> 00:09:42,480
were asked what is the user input to

279
00:09:40,560 --> 00:09:44,240
this program and what happens when this

280
00:09:42,480 --> 00:09:46,399
code executes

281
00:09:44,240 --> 00:09:48,160
in the priming cohort they asked those

282
00:09:46,399 --> 00:09:49,839
same two questions but then they layered

283
00:09:48,160 --> 00:09:51,760
two additional questions on top of that

284
00:09:49,839 --> 00:09:54,240
they said could a developer experience

285
00:09:51,760 --> 00:09:55,440
unexpected results when they run such

286
00:09:54,240 --> 00:09:57,040
code

287
00:09:55,440 --> 00:09:58,959
and what could examples of these

288
00:09:57,040 --> 00:10:00,480
unexpected results be and where do they

289
00:09:58,959 --> 00:10:03,040
appear in the code

290
00:10:00,480 --> 00:10:04,880
and then in this third cohort

291
00:10:03,040 --> 00:10:06,240
where they were explicit they just said

292
00:10:04,880 --> 00:10:08,079
straight up this code has a

293
00:10:06,240 --> 00:10:10,079
vulnerability can you pinpoint the

294
00:10:08,079 --> 00:10:11,760
problem

295
00:10:10,079 --> 00:10:14,079
and the results of this were that the

296
00:10:11,760 --> 00:10:15,680
developers who were who are primed about

297
00:10:14,079 --> 00:10:17,360
their potentially being a security

298
00:10:15,680 --> 00:10:19,440
problem here identified security

299
00:10:17,360 --> 00:10:21,760
problems at almost twice the rate of

300
00:10:19,440 --> 00:10:23,040
unprimed developers

301
00:10:21,760 --> 00:10:24,880
so one of the things that they say in

302
00:10:23,040 --> 00:10:26,959
this paper is that security is not

303
00:10:24,880 --> 00:10:28,800
really part of the heuristic used by

304
00:10:26,959 --> 00:10:31,040
developers in their daily programming

305
00:10:28,800 --> 00:10:34,560
tasks and that's because developers on

306
00:10:31,040 --> 00:10:36,399
the whole focus mostly on functionality

307
00:10:34,560 --> 00:10:38,399
and performance can i get this code to

308
00:10:36,399 --> 00:10:41,040
run to meet a particular business or

309
00:10:38,399 --> 00:10:43,440
organizational objective and does it do

310
00:10:41,040 --> 00:10:44,959
it relatively quickly those are the two

311
00:10:43,440 --> 00:10:46,640
top two things that developers are

312
00:10:44,959 --> 00:10:48,959
actually focused on when they're writing

313
00:10:46,640 --> 00:10:51,440
code on a day-to-day basis

314
00:10:48,959 --> 00:10:52,959
um and the mental model that we have

315
00:10:51,440 --> 00:10:54,959
when it comes to

316
00:10:52,959 --> 00:10:57,120
security education that we think that

317
00:10:54,959 --> 00:10:58,880
developers should learn about security

318
00:10:57,120 --> 00:11:00,959
so often that's taking time out of their

319
00:10:58,880 --> 00:11:02,160
day to to go and learn that or maybe if

320
00:11:00,959 --> 00:11:04,000
they're going through an education

321
00:11:02,160 --> 00:11:05,680
institution that's part of the course

322
00:11:04,000 --> 00:11:07,120
work that they are doing and so they

323
00:11:05,680 --> 00:11:08,880
should just go and learn that and then

324
00:11:07,120 --> 00:11:10,959
they should apply

325
00:11:08,880 --> 00:11:13,200
what they learn when they need it

326
00:11:10,959 --> 00:11:15,040
and the fundamental reality here is that

327
00:11:13,200 --> 00:11:16,880
this is not actually how humans actually

328
00:11:15,040 --> 00:11:18,800
think you know we're great at acquiring

329
00:11:16,880 --> 00:11:21,040
a whole bunch of different knowledge

330
00:11:18,800 --> 00:11:22,959
but there are very specific triggers

331
00:11:21,040 --> 00:11:24,880
that are required for actually having us

332
00:11:22,959 --> 00:11:26,480
apply that knowledge at any particular

333
00:11:24,880 --> 00:11:27,519
point in time

334
00:11:26,480 --> 00:11:28,880
so

335
00:11:27,519 --> 00:11:31,760
they make this really strong

336
00:11:28,880 --> 00:11:33,360
recommendation that uh software that

337
00:11:31,760 --> 00:11:36,320
interfaces with the developers so that's

338
00:11:33,360 --> 00:11:38,000
like ides text editors compilers static

339
00:11:36,320 --> 00:11:40,640
analysis tools

340
00:11:38,000 --> 00:11:42,640
that those tools prime developers on the

341
00:11:40,640 --> 00:11:45,200
spot when they need it

342
00:11:42,640 --> 00:11:47,040
while they are actually coding so when

343
00:11:45,200 --> 00:11:49,279
they are cutting the code in their text

344
00:11:47,040 --> 00:11:51,600
editor and then running it that's when

345
00:11:49,279 --> 00:11:53,360
we should be providing feedback to users

346
00:11:51,600 --> 00:11:54,959
about their code potentially being

347
00:11:53,360 --> 00:11:56,720
insecure or actually telling them hey

348
00:11:54,959 --> 00:11:59,040
it's actually quite insecure the way

349
00:11:56,720 --> 00:12:01,600
that you're using this right now

350
00:11:59,040 --> 00:12:03,920
so the tldr for this particular paper is

351
00:12:01,600 --> 00:12:06,560
the devs don't really think about

352
00:12:03,920 --> 00:12:08,560
security but priming significantly

353
00:12:06,560 --> 00:12:11,680
increases the likelihood of discovering

354
00:12:08,560 --> 00:12:13,920
security bugs

355
00:12:11,680 --> 00:12:15,680
so another potentially uncomfortable

356
00:12:13,920 --> 00:12:17,360
truth something that you may know but

357
00:12:15,680 --> 00:12:19,760
you really wish you didn't know is that

358
00:12:17,360 --> 00:12:22,079
stack overflow provides functional yet

359
00:12:19,760 --> 00:12:23,519
insecure code examples

360
00:12:22,079 --> 00:12:25,600
to developers

361
00:12:23,519 --> 00:12:28,399
and so there's this fantastic piece of

362
00:12:25,600 --> 00:12:30,959
research that was uh published in 2016

363
00:12:28,399 --> 00:12:34,079
by uh jasmine a car michael

364
00:12:30,959 --> 00:12:36,000
bucks and uh sasha fall uh called you

365
00:12:34,079 --> 00:12:38,639
get where you're looking for the impact

366
00:12:36,000 --> 00:12:40,079
of information sources on code security

367
00:12:38,639 --> 00:12:42,639
and in fact this particular group of

368
00:12:40,079 --> 00:12:44,560
researchers um there's a they sort of

369
00:12:42,639 --> 00:12:45,600
have a nice sort of narrative thread

370
00:12:44,560 --> 00:12:47,040
through throughout everything that we're

371
00:12:45,600 --> 00:12:48,880
going to be talking today this this

372
00:12:47,040 --> 00:12:50,160
particular group uh based over in

373
00:12:48,880 --> 00:12:51,920
germany a bunch of educational

374
00:12:50,160 --> 00:12:53,600
institutions over there have done some

375
00:12:51,920 --> 00:12:55,519
pretty phenomenal work on the human

376
00:12:53,600 --> 00:12:57,760
factors uh in secure software

377
00:12:55,519 --> 00:12:59,760
development over the last sort of five

378
00:12:57,760 --> 00:13:01,680
to five to ten years

379
00:12:59,760 --> 00:13:05,040
and so in this particular experiment

380
00:13:01,680 --> 00:13:07,040
that they constructed back in 2016

381
00:13:05,040 --> 00:13:08,800
what they did was that they recruited

382
00:13:07,040 --> 00:13:10,480
android developers

383
00:13:08,800 --> 00:13:11,440
through github and a variety of other

384
00:13:10,480 --> 00:13:14,000
sources

385
00:13:11,440 --> 00:13:16,240
and they provided those uh those android

386
00:13:14,000 --> 00:13:18,560
developers with a skeleton app

387
00:13:16,240 --> 00:13:20,880
and they were also presented with four

388
00:13:18,560 --> 00:13:22,399
tasks in a random order here so they

389
00:13:20,880 --> 00:13:24,639
were it was a combination of secure

390
00:13:22,399 --> 00:13:26,880
networking secure storage

391
00:13:24,639 --> 00:13:28,000
inter-component communication and lease

392
00:13:26,880 --> 00:13:31,040
permissions

393
00:13:28,000 --> 00:13:33,600
and so the developers were time boxed to

394
00:13:31,040 --> 00:13:36,079
20 to 30 minutes per task and then at

395
00:13:33,600 --> 00:13:37,920
the end of that they were asked to fill

396
00:13:36,079 --> 00:13:39,440
out an exit survey about how they how

397
00:13:37,920 --> 00:13:41,360
they felt they performed and whether

398
00:13:39,440 --> 00:13:43,680
their code was secure

399
00:13:41,360 --> 00:13:45,279
and so the android devs that were that

400
00:13:43,680 --> 00:13:47,199
were run through this process were split

401
00:13:45,279 --> 00:13:49,839
into four separate groups

402
00:13:47,199 --> 00:13:50,639
so the first group they could only refer

403
00:13:49,839 --> 00:13:53,199
to

404
00:13:50,639 --> 00:13:54,959
information on stack overflow

405
00:13:53,199 --> 00:13:56,320
the second group could only refer to

406
00:13:54,959 --> 00:13:59,360
information in the official

407
00:13:56,320 --> 00:14:01,600
documentation the third group could only

408
00:13:59,360 --> 00:14:03,760
look at books and the fourth group could

409
00:14:01,600 --> 00:14:05,360
look at any of these information sources

410
00:14:03,760 --> 00:14:06,880
and the books weren't hard copy they

411
00:14:05,360 --> 00:14:08,399
were actually a digital copy of the book

412
00:14:06,880 --> 00:14:09,760
so they could use the table of contents

413
00:14:08,399 --> 00:14:12,240
they could use like the fine

414
00:14:09,760 --> 00:14:14,079
functionality and their pdf viewer

415
00:14:12,240 --> 00:14:15,600
but yeah so it wasn't wasn't entirely

416
00:14:14,079 --> 00:14:17,360
stone age but it's a relatively

417
00:14:15,600 --> 00:14:18,959
comparable sort of technological means

418
00:14:17,360 --> 00:14:21,279
to be able to access this information

419
00:14:18,959 --> 00:14:23,360
but that information itself what sources

420
00:14:21,279 --> 00:14:25,040
they could use was significantly

421
00:14:23,360 --> 00:14:27,120
restricted

422
00:14:25,040 --> 00:14:28,720
and the results well the results sort of

423
00:14:27,120 --> 00:14:30,320
fall into two separate buckets so

424
00:14:28,720 --> 00:14:32,639
there's the functional results where

425
00:14:30,320 --> 00:14:34,959
they're able to deliver code that was

426
00:14:32,639 --> 00:14:35,839
functional and did what it was intended

427
00:14:34,959 --> 00:14:37,760
to do

428
00:14:35,839 --> 00:14:39,920
and the second part was what is the

429
00:14:37,760 --> 00:14:41,920
overall security of of those particular

430
00:14:39,920 --> 00:14:44,000
results so let's look at the functional

431
00:14:41,920 --> 00:14:46,240
first and so in the functional they

432
00:14:44,000 --> 00:14:48,320
found that stack overflow people who use

433
00:14:46,240 --> 00:14:51,199
stack overflow

434
00:14:48,320 --> 00:14:52,959
had a significantly higher success rate

435
00:14:51,199 --> 00:14:54,800
in fact stack overflow and books funnily

436
00:14:52,959 --> 00:14:56,480
enough had a significantly higher

437
00:14:54,800 --> 00:14:58,240
success rate than people that were using

438
00:14:56,480 --> 00:15:00,480
only the official documentation or the

439
00:14:58,240 --> 00:15:03,199
people who are doing free choice

440
00:15:00,480 --> 00:15:04,480
uh in fact it's so bad that the the devs

441
00:15:03,199 --> 00:15:06,480
that were assigned to the official

442
00:15:04,480 --> 00:15:09,199
document official documentation actually

443
00:15:06,480 --> 00:15:10,880
performed the worst overall which is

444
00:15:09,199 --> 00:15:12,959
sort of a bit of a damning indictment i

445
00:15:10,880 --> 00:15:14,000
guess of a lot of official documentation

446
00:15:12,959 --> 00:15:15,760
particularly when it comes to

447
00:15:14,000 --> 00:15:17,360
cryptography

448
00:15:15,760 --> 00:15:20,079
so that's on the functional side but the

449
00:15:17,360 --> 00:15:21,440
security side of things

450
00:15:20,079 --> 00:15:22,880
was really interesting so they found

451
00:15:21,440 --> 00:15:25,199
that with the lease permissions with

452
00:15:22,880 --> 00:15:26,399
about 87 success rate that was the

453
00:15:25,199 --> 00:15:28,639
highest out of all of them everything

454
00:15:26,399 --> 00:15:30,639
else is about the same about sort of 38

455
00:15:28,639 --> 00:15:32,880
to 40 percent so for the secure

456
00:15:30,639 --> 00:15:35,040
networking the inter component coms and

457
00:15:32,880 --> 00:15:36,720
secure storage it was very very low

458
00:15:35,040 --> 00:15:38,720
success rate

459
00:15:36,720 --> 00:15:40,480
but the confidence was super high even

460
00:15:38,720 --> 00:15:42,720
though they were actually not producing

461
00:15:40,480 --> 00:15:44,800
something that was actually secure

462
00:15:42,720 --> 00:15:46,560
so on the whole developers self-assess

463
00:15:44,800 --> 00:15:48,800
their solutions to be secure when they

464
00:15:46,560 --> 00:15:51,839
mostly were not

465
00:15:48,800 --> 00:15:53,680
and what that means for us as developers

466
00:15:51,839 --> 00:15:55,519
is that in this particular study android

467
00:15:53,680 --> 00:15:58,079
developers who use stack overflow they

468
00:15:55,519 --> 00:16:02,720
do get functional code quicker but are

469
00:15:58,079 --> 00:16:02,720
more likely to produce insecure code

470
00:16:02,959 --> 00:16:06,320
see with that for a second like that's

471
00:16:04,720 --> 00:16:08,000
that's pretty wild when you think about

472
00:16:06,320 --> 00:16:09,759
it

473
00:16:08,000 --> 00:16:11,199
and when we think about you know the

474
00:16:09,759 --> 00:16:12,480
previous study we talked about a moment

475
00:16:11,199 --> 00:16:14,560
ago

476
00:16:12,480 --> 00:16:16,720
where you know we look at priming and

477
00:16:14,560 --> 00:16:18,639
heuristics and whatnot

478
00:16:16,720 --> 00:16:21,040
developers are mostly focusing on

479
00:16:18,639 --> 00:16:22,880
functionality and performance so of

480
00:16:21,040 --> 00:16:24,880
course they are going to continue to go

481
00:16:22,880 --> 00:16:26,639
to stack overflow because it produces

482
00:16:24,880 --> 00:16:28,320
functional code it helps them actually

483
00:16:26,639 --> 00:16:30,959
meet their objectives the things that

484
00:16:28,320 --> 00:16:33,440
they are often incentivized on within

485
00:16:30,959 --> 00:16:35,680
their organizations mostly functionality

486
00:16:33,440 --> 00:16:37,360
and performance

487
00:16:35,680 --> 00:16:39,839
so they have a pretty strong

488
00:16:37,360 --> 00:16:41,440
recommendation that it's really critical

489
00:16:39,839 --> 00:16:42,959
for people who build cryptographic

490
00:16:41,440 --> 00:16:45,519
libraries particularly in the open

491
00:16:42,959 --> 00:16:48,160
source space to develop documentation

492
00:16:45,519 --> 00:16:50,079
and resources that combine the useful or

493
00:16:48,160 --> 00:16:52,480
the usefulness of forums like stack

494
00:16:50,079 --> 00:16:54,800
overflow so a sort of a q a

495
00:16:52,480 --> 00:16:56,800
type situation or like a forum

496
00:16:54,800 --> 00:17:00,000
but with the security awareness of books

497
00:16:56,800 --> 00:17:01,120
or the official api documentation

498
00:17:00,000 --> 00:17:02,480
so

499
00:17:01,120 --> 00:17:04,079
one really interesting thing that you

500
00:17:02,480 --> 00:17:06,559
can do if you're a maintainer of an open

501
00:17:04,079 --> 00:17:09,039
source cryptographic tool

502
00:17:06,559 --> 00:17:11,280
is that you regularly spend time looking

503
00:17:09,039 --> 00:17:13,120
back at yourself on stack overflow

504
00:17:11,280 --> 00:17:15,199
looking at the questions that your users

505
00:17:13,120 --> 00:17:17,199
are asking on stack overflow about your

506
00:17:15,199 --> 00:17:19,919
library or about your tool

507
00:17:17,199 --> 00:17:22,079
use that as a way to identify gaps in

508
00:17:19,919 --> 00:17:24,000
your documentation that are being filled

509
00:17:22,079 --> 00:17:26,319
by stack overflow

510
00:17:24,000 --> 00:17:28,240
and then fill those gaps in your own

511
00:17:26,319 --> 00:17:29,520
official documentation

512
00:17:28,240 --> 00:17:30,799
because we're going to talk about this

513
00:17:29,520 --> 00:17:33,360
in a second actually with some of the

514
00:17:30,799 --> 00:17:35,440
other research but um typically the uh

515
00:17:33,360 --> 00:17:37,919
the official documentation

516
00:17:35,440 --> 00:17:39,679
tends to have the most secure uh results

517
00:17:37,919 --> 00:17:41,919
for for developers

518
00:17:39,679 --> 00:17:43,919
and so last thing to do here is to post

519
00:17:41,919 --> 00:17:45,919
links and follow our comments um on the

520
00:17:43,919 --> 00:17:47,520
accepted stack overflow answers back to

521
00:17:45,919 --> 00:17:49,840
your official documentation and then

522
00:17:47,520 --> 00:17:51,280
over time if you're looking at

523
00:17:49,840 --> 00:17:52,880
your website analytics for your

524
00:17:51,280 --> 00:17:54,320
documentation you should see an

525
00:17:52,880 --> 00:17:56,559
increasing number of people going to

526
00:17:54,320 --> 00:17:58,559
these pages and you should see your your

527
00:17:56,559 --> 00:18:00,080
official documentation ranking higher in

528
00:17:58,559 --> 00:18:02,160
google

529
00:18:00,080 --> 00:18:04,480
so the tldr for this particular study is

530
00:18:02,160 --> 00:18:06,960
that stack overflow answers they provide

531
00:18:04,480 --> 00:18:08,799
functional insecure but yet insecure

532
00:18:06,960 --> 00:18:10,400
code to developers and developers are

533
00:18:08,799 --> 00:18:12,960
also really bad at assessing whether

534
00:18:10,400 --> 00:18:14,799
that code is secure as well

535
00:18:12,960 --> 00:18:16,160
quick side quest for a second though

536
00:18:14,799 --> 00:18:18,240
there's this other paper that was

537
00:18:16,160 --> 00:18:20,480
published only a couple of months ago

538
00:18:18,240 --> 00:18:22,799
back in november of 2021

539
00:18:20,480 --> 00:18:24,640
um about the effect of google search on

540
00:18:22,799 --> 00:18:26,480
software security so

541
00:18:24,640 --> 00:18:28,640
security interventions and doing content

542
00:18:26,480 --> 00:18:31,600
re-ranking particularly for results

543
00:18:28,640 --> 00:18:33,120
showing stack overflow and so i'll skip

544
00:18:31,600 --> 00:18:35,120
over a lot of the detail of the study

545
00:18:33,120 --> 00:18:38,080
but they basically found that there is

546
00:18:35,120 --> 00:18:39,760
at least a 22.8 chance that one out of

547
00:18:38,080 --> 00:18:43,840
the top three google search results for

548
00:18:39,760 --> 00:18:46,160
stack overflow leads to insecure code

549
00:18:43,840 --> 00:18:47,520
again that's another damning statistic

550
00:18:46,160 --> 00:18:50,000
right there

551
00:18:47,520 --> 00:18:52,080
so dldr your insecure code ends up in

552
00:18:50,000 --> 00:18:53,520
the top results and is also clicked on

553
00:18:52,080 --> 00:18:55,280
more often there's a higher engagement

554
00:18:53,520 --> 00:18:56,720
with it so the study that they did for

555
00:18:55,280 --> 00:18:59,280
this was really interesting so they used

556
00:18:56,720 --> 00:19:02,720
like the google the google custom search

557
00:18:59,280 --> 00:19:05,360
engine functionality to uh to boost or

558
00:19:02,720 --> 00:19:07,679
unboost different bits of content and

559
00:19:05,360 --> 00:19:09,679
they asked the uh

560
00:19:07,679 --> 00:19:11,840
they asked the participants to perform

561
00:19:09,679 --> 00:19:13,440
google searches using this custom search

562
00:19:11,840 --> 00:19:15,520
engine that was built into the survey

563
00:19:13,440 --> 00:19:16,799
tool um that that the users were going

564
00:19:15,520 --> 00:19:18,720
or the developers were going through

565
00:19:16,799 --> 00:19:20,559
during during this particular

566
00:19:18,720 --> 00:19:22,320
this particular exercise

567
00:19:20,559 --> 00:19:24,720
and they found that the participants

568
00:19:22,320 --> 00:19:26,559
that used the modified search engine to

569
00:19:24,720 --> 00:19:28,960
look for help online submitted more

570
00:19:26,559 --> 00:19:32,240
secure and more functional results with

571
00:19:28,960 --> 00:19:35,120
a significant statistical significance

572
00:19:32,240 --> 00:19:36,960
so the gldr for that particular paper is

573
00:19:35,120 --> 00:19:38,640
this is a fairly novel solution to the

574
00:19:36,960 --> 00:19:40,880
problem that actually requires zero

575
00:19:38,640 --> 00:19:42,799
behavioral change from developers

576
00:19:40,880 --> 00:19:44,640
and my question is if there is anybody

577
00:19:42,799 --> 00:19:46,960
that is listening to this talk today

578
00:19:44,640 --> 00:19:48,880
that works at google can you please go

579
00:19:46,960 --> 00:19:50,240
and i'll provide a link to this

580
00:19:48,880 --> 00:19:51,520
particular paper at the end because i

581
00:19:50,240 --> 00:19:53,520
don't know how much traction it's really

582
00:19:51,520 --> 00:19:56,160
gotten outside of academic circles but

583
00:19:53,520 --> 00:19:59,120
there i guarantee there is some product

584
00:19:56,160 --> 00:20:01,440
manager uh or team within google that

585
00:19:59,120 --> 00:20:04,720
could actually do something about this

586
00:20:01,440 --> 00:20:07,039
and it would have an outsized effect

587
00:20:04,720 --> 00:20:08,640
on the overall security of code that the

588
00:20:07,039 --> 00:20:10,720
developers are shipping on a day-to-day

589
00:20:08,640 --> 00:20:12,080
basis really really novel solution to

590
00:20:10,720 --> 00:20:13,120
this problem and i would love to see it

591
00:20:12,080 --> 00:20:16,159
implemented

592
00:20:13,120 --> 00:20:17,760
particularly within google

593
00:20:16,159 --> 00:20:21,120
all right so

594
00:20:17,760 --> 00:20:23,440
the other um sort of hard to hear hard

595
00:20:21,120 --> 00:20:25,280
to listen to truth is that most what we

596
00:20:23,440 --> 00:20:27,440
call cryptography bugs actually come

597
00:20:25,280 --> 00:20:29,280
from the use of the libraries not the

598
00:20:27,440 --> 00:20:30,480
actual libraries

599
00:20:29,280 --> 00:20:32,320
themselves

600
00:20:30,480 --> 00:20:35,280
and so this particular study published

601
00:20:32,320 --> 00:20:37,039
in 2014 uh called why does cryptographic

602
00:20:35,280 --> 00:20:39,679
software fail a case study in open

603
00:20:37,039 --> 00:20:41,360
problems um they they go into a whole

604
00:20:39,679 --> 00:20:42,559
bunch of different areas in this paper

605
00:20:41,360 --> 00:20:43,679
they cover sort of three or four

606
00:20:42,559 --> 00:20:45,600
different topics and we're just going to

607
00:20:43,679 --> 00:20:46,400
talk about one of them today which is

608
00:20:45,600 --> 00:20:48,480
this

609
00:20:46,400 --> 00:20:50,159
uh this particular part of the study so

610
00:20:48,480 --> 00:20:55,120
what they did was they did an analysis

611
00:20:50,159 --> 00:20:57,039
of uh 269 uh cves that there were

612
00:20:55,120 --> 00:20:58,880
assessors having cryptographic ins uh

613
00:20:57,039 --> 00:21:01,200
issues which is from the common weakness

614
00:20:58,880 --> 00:21:03,440
enumeration from from mida and they

615
00:21:01,200 --> 00:21:05,760
looked at that from january uh 2011 to

616
00:21:03,440 --> 00:21:07,039
2014 and and that said this is the

617
00:21:05,760 --> 00:21:08,480
foundational paper there are other

618
00:21:07,039 --> 00:21:10,480
papers more recently that have

619
00:21:08,480 --> 00:21:12,000
replicated this but you just sort of

620
00:21:10,480 --> 00:21:14,080
articulated pretty clearly it was the

621
00:21:12,000 --> 00:21:15,840
sort of first movers in this field

622
00:21:14,080 --> 00:21:19,120
and what they did was that they

623
00:21:15,840 --> 00:21:22,000
categorized the vulnerabilities uh for

624
00:21:19,120 --> 00:21:24,640
for cryptographic issues um as along two

625
00:21:22,000 --> 00:21:26,000
different dimensions one was impact so

626
00:21:24,640 --> 00:21:28,480
uh whether it was a plain text

627
00:21:26,000 --> 00:21:31,760
disclosure a manipulator in the middle a

628
00:21:28,480 --> 00:21:34,000
brute force or a side channel attack and

629
00:21:31,760 --> 00:21:36,799
then also at the layer as well so was it

630
00:21:34,000 --> 00:21:40,400
at a low level cryptographic primitive

631
00:21:36,799 --> 00:21:42,240
like aes or diffie-hellman was it the

632
00:21:40,400 --> 00:21:44,080
protocol level so that could be some

633
00:21:42,240 --> 00:21:45,840
sort of library or abstraction that

634
00:21:44,080 --> 00:21:48,080
bundles up a bunch of cryptographic

635
00:21:45,840 --> 00:21:49,679
primitives to be able to provide an

636
00:21:48,080 --> 00:21:51,440
actual usable interface to the

637
00:21:49,679 --> 00:21:52,640
developers that were needed to access

638
00:21:51,440 --> 00:21:54,640
these primitives

639
00:21:52,640 --> 00:21:57,200
and the last layer was the application

640
00:21:54,640 --> 00:21:59,280
itself so you know issues particularly

641
00:21:57,200 --> 00:22:00,880
like with plaintext disclosure outside

642
00:21:59,280 --> 00:22:02,240
of the actual cryptographic libraries

643
00:22:00,880 --> 00:22:06,000
themselves

644
00:22:02,240 --> 00:22:10,159
um and the result of this was that 100

645
00:22:06,000 --> 00:22:11,280
of plain text disclosures of information

646
00:22:10,159 --> 00:22:13,360
86

647
00:22:11,280 --> 00:22:15,760
manipulated are in the middle attacks

648
00:22:13,360 --> 00:22:17,840
and 79 percent of brute force were in

649
00:22:15,760 --> 00:22:20,640
the app layer now that said zero percent

650
00:22:17,840 --> 00:22:23,600
of side channel happened at the the uh

651
00:22:20,640 --> 00:22:25,840
the uh the primitive and protocol layer

652
00:22:23,600 --> 00:22:27,600
um but that's the sort of problem that

653
00:22:25,840 --> 00:22:29,120
you would expect to to be manifesting at

654
00:22:27,600 --> 00:22:31,200
that layer right but a hundred percent

655
00:22:29,120 --> 00:22:32,720
of plain text discloses were actually

656
00:22:31,200 --> 00:22:35,039
happening in the

657
00:22:32,720 --> 00:22:36,559
application layer and that mostly came

658
00:22:35,039 --> 00:22:38,320
down to the way that the developers that

659
00:22:36,559 --> 00:22:40,159
were using these libraries were actually

660
00:22:38,320 --> 00:22:42,159
invoking them what the ergonomics of the

661
00:22:40,159 --> 00:22:43,520
apis were that these libraries were

662
00:22:42,159 --> 00:22:44,480
exposing

663
00:22:43,520 --> 00:22:47,440
and so

664
00:22:44,480 --> 00:22:50,080
what you can do as a cryptographic

665
00:22:47,440 --> 00:22:51,280
library author or maintainer

666
00:22:50,080 --> 00:22:53,120
is that you can try and expose

667
00:22:51,280 --> 00:22:55,840
interfaces in your libraries that are

668
00:22:53,120 --> 00:22:57,360
what the academic circles term as misuse

669
00:22:55,840 --> 00:22:59,200
resistant and that's really what the

670
00:22:57,360 --> 00:23:00,720
majority of the remainder of this talk

671
00:22:59,200 --> 00:23:03,360
is going to focus on how do we actually

672
00:23:00,720 --> 00:23:05,200
achieve misuse resistance and there's

673
00:23:03,360 --> 00:23:08,240
heaps of research into this but there

674
00:23:05,200 --> 00:23:11,280
are a fairly sparse actual directives as

675
00:23:08,240 --> 00:23:13,120
to how to actually achieve this

676
00:23:11,280 --> 00:23:15,600
so yeah what that fundamentally means is

677
00:23:13,120 --> 00:23:17,280
that about 83 of all cryptographic bugs

678
00:23:15,600 --> 00:23:19,919
come from the improper use of

679
00:23:17,280 --> 00:23:22,000
cryptographic libraries not actually due

680
00:23:19,919 --> 00:23:24,240
to cryptographic flaws inside those

681
00:23:22,000 --> 00:23:27,120
libraries themselves which again another

682
00:23:24,240 --> 00:23:28,080
pretty damning statistic

683
00:23:27,120 --> 00:23:29,760
so

684
00:23:28,080 --> 00:23:33,440
let's actually look at what you need to

685
00:23:29,760 --> 00:23:35,039
do to improve the misuse resistance of

686
00:23:33,440 --> 00:23:36,640
your libraries and the tools that you're

687
00:23:35,039 --> 00:23:39,200
shipping to developers

688
00:23:36,640 --> 00:23:40,880
and the tldr probably like the number

689
00:23:39,200 --> 00:23:43,679
one thing that you can take away from

690
00:23:40,880 --> 00:23:45,120
this talk today is that the features on

691
00:23:43,679 --> 00:23:46,880
the periphery of your library and i'll

692
00:23:45,120 --> 00:23:48,960
talk about what that means in a second

693
00:23:46,880 --> 00:23:52,559
and the documentation that you ship with

694
00:23:48,960 --> 00:23:54,799
your library are the absolute number one

695
00:23:52,559 --> 00:23:57,440
requirement for developers to use your

696
00:23:54,799 --> 00:24:00,320
library securely if you don't have an

697
00:23:57,440 --> 00:24:03,200
extensive range of features an extensive

698
00:24:00,320 --> 00:24:05,520
range of examples and recipes about how

699
00:24:03,200 --> 00:24:06,960
to use it and extensive documentation

700
00:24:05,520 --> 00:24:09,200
explaining all the different ways that

701
00:24:06,960 --> 00:24:12,320
you can that you can use that library

702
00:24:09,200 --> 00:24:14,480
you are doing a disservice to your users

703
00:24:12,320 --> 00:24:16,880
and so this particular paper published

704
00:24:14,480 --> 00:24:18,480
in 2017 like i i was originally just

705
00:24:16,880 --> 00:24:19,919
going to do a talk exclusively about

706
00:24:18,480 --> 00:24:22,640
this paper i can talk about this paper

707
00:24:19,919 --> 00:24:24,880
hours it's it's it's a brilliant read um

708
00:24:22,640 --> 00:24:26,960
it's super technical but it's also uh

709
00:24:24,880 --> 00:24:28,240
it's very readable at the same time and

710
00:24:26,960 --> 00:24:29,600
so this is by that same group of

711
00:24:28,240 --> 00:24:31,760
researchers that i referred to

712
00:24:29,600 --> 00:24:34,080
previously so yasmine car michael box

713
00:24:31,760 --> 00:24:35,279
and sasha phil and simon garfinkel

714
00:24:34,080 --> 00:24:37,440
and that's called comparing the

715
00:24:35,279 --> 00:24:39,200
usability of cryptographic apis and this

716
00:24:37,440 --> 00:24:41,039
has been cited heaps of times in the

717
00:24:39,200 --> 00:24:42,720
last five years and i highly recommend

718
00:24:41,039 --> 00:24:44,320
if you're going to read any paper out of

719
00:24:42,720 --> 00:24:45,600
the stuff that i'm talking about today

720
00:24:44,320 --> 00:24:47,919
go and have a read of this and i'll

721
00:24:45,600 --> 00:24:50,720
provide some links at the end as well

722
00:24:47,919 --> 00:24:52,320
so this particular experiment was really

723
00:24:50,720 --> 00:24:54,159
interestingly constructed so what they

724
00:24:52,320 --> 00:24:56,000
did was they recruited a bunch of python

725
00:24:54,159 --> 00:24:58,720
developers and they assigned those

726
00:24:56,000 --> 00:25:01,039
python developers five tasks with a

727
00:24:58,720 --> 00:25:03,200
bunch of stub code so those tasks were

728
00:25:01,039 --> 00:25:05,440
broken into two categories symmetric and

729
00:25:03,200 --> 00:25:07,840
asymmetric so for symmetric it was key

730
00:25:05,440 --> 00:25:10,000
generation and encryption and decryption

731
00:25:07,840 --> 00:25:12,000
for the asymmetric it was key generation

732
00:25:10,000 --> 00:25:14,400
encryption decryption and certificate

733
00:25:12,000 --> 00:25:16,320
validation and at the end they also ask

734
00:25:14,400 --> 00:25:17,840
the same sort of questions is the code

735
00:25:16,320 --> 00:25:19,679
that you're actually producing here

736
00:25:17,840 --> 00:25:22,559
secure

737
00:25:19,679 --> 00:25:24,640
and they uh used a bunch of very common

738
00:25:22,559 --> 00:25:25,760
well-known cryptography libraries that

739
00:25:24,640 --> 00:25:27,919
you if you

740
00:25:25,760 --> 00:25:29,520
a developer in the python ecosystem

741
00:25:27,919 --> 00:25:31,840
you've chances are you've used one of

742
00:25:29,520 --> 00:25:33,760
these libraries um so they were assigned

743
00:25:31,840 --> 00:25:36,320
libraries uh that one of these five so

744
00:25:33,760 --> 00:25:38,960
cryptography io kiza

745
00:25:36,320 --> 00:25:41,360
pineapple uh sounds like pineapple

746
00:25:38,960 --> 00:25:43,279
doesn't pineapple m2 crypto and pi

747
00:25:41,360 --> 00:25:45,039
crypto and they were instructed

748
00:25:43,279 --> 00:25:46,960
specifically to only use the included

749
00:25:45,039 --> 00:25:48,240
docs from these different libraries now

750
00:25:46,960 --> 00:25:50,080
you notice the the little sort of

751
00:25:48,240 --> 00:25:52,080
asterisks that i've got listed there the

752
00:25:50,080 --> 00:25:53,440
top three libraries that i that i

753
00:25:52,080 --> 00:25:56,640
mentioned here are ones that make

754
00:25:53,440 --> 00:25:59,279
explicit claims about the usability

755
00:25:56,640 --> 00:26:01,039
focus for actually building those apis

756
00:25:59,279 --> 00:26:03,679
and so trying to try to build a more

757
00:26:01,039 --> 00:26:05,360
usable misuse resistance style

758
00:26:03,679 --> 00:26:07,120
encryption

759
00:26:05,360 --> 00:26:08,640
it's also probably if you're familiar

760
00:26:07,120 --> 00:26:10,799
with the space as well it's worth noting

761
00:26:08,640 --> 00:26:14,320
that key kizar was deprecated in favor

762
00:26:10,799 --> 00:26:15,440
of tink by google in 2019 as well so

763
00:26:14,320 --> 00:26:17,919
there's a whole bunch of things that

764
00:26:15,440 --> 00:26:19,679
were fed into keys are but i actually

765
00:26:17,919 --> 00:26:22,159
haven't seen this research replicated

766
00:26:19,679 --> 00:26:23,840
with tink so if anybody is looking for a

767
00:26:22,159 --> 00:26:26,000
little research project and you work in

768
00:26:23,840 --> 00:26:27,919
an academic institution this would

769
00:26:26,000 --> 00:26:29,600
definitely be one to take a look at with

770
00:26:27,919 --> 00:26:31,760
tink

771
00:26:29,600 --> 00:26:33,120
and the experiment that they constructed

772
00:26:31,760 --> 00:26:36,000
was really interesting so what they did

773
00:26:33,120 --> 00:26:38,000
was they modified a jupiter notebook

774
00:26:36,000 --> 00:26:39,679
and include the stub code in here for

775
00:26:38,000 --> 00:26:41,360
each of these uh for each of these tasks

776
00:26:39,679 --> 00:26:42,320
that i mentioned in the previous slide

777
00:26:41,360 --> 00:26:43,919
and one of the other things that they

778
00:26:42,320 --> 00:26:46,000
did inside the jupyter notebook was that

779
00:26:43,919 --> 00:26:48,720
they would uh regularly take snapshots

780
00:26:46,000 --> 00:26:50,640
of the code and they also detected copy

781
00:26:48,720 --> 00:26:52,960
and paste events as well to see where

782
00:26:50,640 --> 00:26:54,240
people were actually copying code from

783
00:26:52,960 --> 00:26:56,080
elsewhere on the internet so they could

784
00:26:54,240 --> 00:26:58,000
trace it back to the original source

785
00:26:56,080 --> 00:26:59,600
whether they were actually only getting

786
00:26:58,000 --> 00:27:00,880
it from the official documentation like

787
00:26:59,600 --> 00:27:02,240
they were asked to or whether they were

788
00:27:00,880 --> 00:27:04,320
really actually going on to stack

789
00:27:02,240 --> 00:27:06,559
overflow

790
00:27:04,320 --> 00:27:07,919
and again the results are sort of in two

791
00:27:06,559 --> 00:27:10,320
separate buckets here the functional

792
00:27:07,919 --> 00:27:12,080
results and the secure results and not

793
00:27:10,320 --> 00:27:14,320
to give too much of a spoiler but um

794
00:27:12,080 --> 00:27:16,240
there's a similar discrepancy between

795
00:27:14,320 --> 00:27:18,720
the previous research that we're looking

796
00:27:16,240 --> 00:27:20,720
at and and this particular baby here

797
00:27:18,720 --> 00:27:23,279
so the functional results found that

798
00:27:20,720 --> 00:27:26,000
there was a wide variation of task

799
00:27:23,279 --> 00:27:28,000
success across different libraries on

800
00:27:26,000 --> 00:27:30,159
the whole asymmetric tasks were less

801
00:27:28,000 --> 00:27:32,159
successful and they they offer some

802
00:27:30,159 --> 00:27:34,640
discussion in the paper about why that

803
00:27:32,159 --> 00:27:36,640
might be i think that the the general

804
00:27:34,640 --> 00:27:38,640
sense of it is that uh asymmetric uh

805
00:27:36,640 --> 00:27:41,200
encryption is just

806
00:27:38,640 --> 00:27:42,880
generally less well understood

807
00:27:41,200 --> 00:27:44,880
and thus people are more likely to make

808
00:27:42,880 --> 00:27:46,640
mistakes with it but you know functional

809
00:27:44,880 --> 00:27:49,120
solutions uh if you look at the

810
00:27:46,640 --> 00:27:50,080
symmetric task for pineapple down there

811
00:27:49,120 --> 00:27:52,080
um

812
00:27:50,080 --> 00:27:53,520
very very high you know well i mean not

813
00:27:52,080 --> 00:27:56,080
not perfect but greater than eighty

814
00:27:53,520 --> 00:27:59,120
percent um so three out of the five had

815
00:27:56,080 --> 00:28:00,240
uh had a functional solutions uh uh

816
00:27:59,120 --> 00:28:01,360
fifty uh

817
00:28:00,240 --> 00:28:03,679
greater than eighty percent of the tasks

818
00:28:01,360 --> 00:28:04,960
had a functional solution lotus keys are

819
00:28:03,679 --> 00:28:06,720
there though as we mentioned before it

820
00:28:04,960 --> 00:28:09,919
was one of those uh libraries that made

821
00:28:06,720 --> 00:28:11,039
explicit claims about the usability um

822
00:28:09,919 --> 00:28:14,240
not good

823
00:28:11,039 --> 00:28:16,640
like 40 of tasks less than 40 of tasks

824
00:28:14,240 --> 00:28:18,480
had an actual functional solution

825
00:28:16,640 --> 00:28:19,679
um and don't worry it gets even worse

826
00:28:18,480 --> 00:28:20,640
when you actually look at the secure

827
00:28:19,679 --> 00:28:23,760
results

828
00:28:20,640 --> 00:28:25,520
but fun little anecdote from uh from the

829
00:28:23,760 --> 00:28:27,200
jupiter notebook little thing that they

830
00:28:25,520 --> 00:28:29,919
that they slipped in before they

831
00:28:27,200 --> 00:28:31,200
actually found that copy paste code that

832
00:28:29,919 --> 00:28:33,520
they were able to trace back to stack

833
00:28:31,200 --> 00:28:36,320
overflow was three times more likely to

834
00:28:33,520 --> 00:28:38,799
be functional on symmetric tasks so

835
00:28:36,320 --> 00:28:40,559
developers love copy pasting

836
00:28:38,799 --> 00:28:42,960
so if they're going to be copy pasting

837
00:28:40,559 --> 00:28:45,919
code at least give them functional and

838
00:28:42,960 --> 00:28:47,440
secure code to be copy pasting

839
00:28:45,919 --> 00:28:50,080
uh the other interesting thing that they

840
00:28:47,440 --> 00:28:51,200
found as well they did an overanalysis

841
00:28:50,080 --> 00:28:53,120
of

842
00:28:51,200 --> 00:28:54,880
of all the different results posted here

843
00:28:53,120 --> 00:28:57,039
they found the python experience the

844
00:28:54,880 --> 00:28:59,760
security background and the library

845
00:28:57,039 --> 00:29:01,760
experience actually did not change the

846
00:28:59,760 --> 00:29:04,240
success rate of

847
00:29:01,760 --> 00:29:05,840
of of these particular functional tasks

848
00:29:04,240 --> 00:29:08,320
so it doesn't really matter if you're a

849
00:29:05,840 --> 00:29:10,559
super experienced python dev have got a

850
00:29:08,320 --> 00:29:12,480
strong background in security or heaps

851
00:29:10,559 --> 00:29:14,240
of library experience you perform

852
00:29:12,480 --> 00:29:16,320
just as poorly as somebody that doesn't

853
00:29:14,240 --> 00:29:19,600
have any of that experience at all which

854
00:29:16,320 --> 00:29:21,600
is uh quite a humbling thing to hear

855
00:29:19,600 --> 00:29:24,720
and then we get into the secure results

856
00:29:21,600 --> 00:29:25,919
and it's an absolute bloodbath so in

857
00:29:24,720 --> 00:29:28,480
this particular case although the

858
00:29:25,919 --> 00:29:30,880
functional asymmetric tasks are very

859
00:29:28,480 --> 00:29:32,799
very low uh they were three times more

860
00:29:30,880 --> 00:29:35,840
likely to actually produce a secure

861
00:29:32,799 --> 00:29:37,440
result uh keys are stands out amazingly

862
00:29:35,840 --> 00:29:38,640
here you know you compare it to this one

863
00:29:37,440 --> 00:29:40,960
right where you see the functional

864
00:29:38,640 --> 00:29:43,279
results of keys are being very very low

865
00:29:40,960 --> 00:29:45,440
but for the people that managed to do it

866
00:29:43,279 --> 00:29:47,600
they had super secure results which is

867
00:29:45,440 --> 00:29:50,320
um

868
00:29:47,600 --> 00:29:51,919
curious not not good i would say

869
00:29:50,320 --> 00:29:54,720
um so yeah that on the whole the

870
00:29:51,919 --> 00:29:57,600
keysight usage was 25 times more likely

871
00:29:54,720 --> 00:29:58,559
to be secure than any other library that

872
00:29:57,600 --> 00:30:00,240
was used

873
00:29:58,559 --> 00:30:01,440
but only 10 percent of the submitted

874
00:30:00,240 --> 00:30:03,919
keys are solutions were actually

875
00:30:01,440 --> 00:30:03,919
functional

876
00:30:04,080 --> 00:30:06,480
which

877
00:30:05,279 --> 00:30:08,720
yeah this is the sort of thing that i

878
00:30:06,480 --> 00:30:10,559
cry into my pillow at night about it's

879
00:30:08,720 --> 00:30:12,799
not good not good

880
00:30:10,559 --> 00:30:14,159
um and as i mentioned at the at the top

881
00:30:12,799 --> 00:30:17,200
of the talk twenty percent of the

882
00:30:14,159 --> 00:30:19,440
submitted code was secure sorry insecure

883
00:30:17,200 --> 00:30:21,919
but the developers strongly believed

884
00:30:19,440 --> 00:30:23,520
that it actually was secure so a bit of

885
00:30:21,919 --> 00:30:24,880
dunning-kruger going on here the less

886
00:30:23,520 --> 00:30:26,399
you know about something the more

887
00:30:24,880 --> 00:30:28,720
confident you are and the greater you

888
00:30:26,399 --> 00:30:30,960
assess your competence at um compared to

889
00:30:28,720 --> 00:30:33,600
people who do know something about it

890
00:30:30,960 --> 00:30:35,279
which is uh yeah pretty sad

891
00:30:33,600 --> 00:30:37,440
so what this actually means for us

892
00:30:35,279 --> 00:30:38,799
though as tool authors and library

893
00:30:37,440 --> 00:30:40,799
authors is that libraries with

894
00:30:38,799 --> 00:30:43,200
simplified interfaces actually produce

895
00:30:40,799 --> 00:30:45,520
more secure results

896
00:30:43,200 --> 00:30:48,399
so misuse resistance is good for

897
00:30:45,520 --> 00:30:51,360
security outcomes for our users and yet

898
00:30:48,399 --> 00:30:54,080
security success rate was less than 80

899
00:30:51,360 --> 00:30:55,679
even on the simplified libraries so not

900
00:30:54,080 --> 00:30:56,480
great overall

901
00:30:55,679 --> 00:30:57,840
um

902
00:30:56,480 --> 00:30:59,279
one of the things that they found though

903
00:30:57,840 --> 00:31:01,360
was that for the libraries that did

904
00:30:59,279 --> 00:31:03,519
actually perform better on the uh on the

905
00:31:01,360 --> 00:31:06,159
functional and security side of things

906
00:31:03,519 --> 00:31:09,039
there was a lot more documentation and

907
00:31:06,159 --> 00:31:11,679
code examples of peripheral features so

908
00:31:09,039 --> 00:31:13,039
that's things like how do you do secure

909
00:31:11,679 --> 00:31:15,600
key storage or how do you do

910
00:31:13,039 --> 00:31:17,600
password-based key generation and those

911
00:31:15,600 --> 00:31:19,279
peripheral features were actually the

912
00:31:17,600 --> 00:31:20,640
things that would make or break the

913
00:31:19,279 --> 00:31:22,960
security

914
00:31:20,640 --> 00:31:25,600
or secure use of a particular library in

915
00:31:22,960 --> 00:31:27,120
fact they found that the the call

916
00:31:25,600 --> 00:31:28,640
signatures the way that these libraries

917
00:31:27,120 --> 00:31:31,039
were being invoked

918
00:31:28,640 --> 00:31:33,039
was generally secure but

919
00:31:31,039 --> 00:31:34,880
the users the developers that were going

920
00:31:33,039 --> 00:31:35,840
through these studies were actually copy

921
00:31:34,880 --> 00:31:39,279
pasting

922
00:31:35,840 --> 00:31:42,000
secure functional code but they didn't

923
00:31:39,279 --> 00:31:45,039
have the uh the knowledge or the skill

924
00:31:42,000 --> 00:31:47,440
to identify the danger because often in

925
00:31:45,039 --> 00:31:49,440
those examples those code examples it

926
00:31:47,440 --> 00:31:51,840
would have insecure inputs to these

927
00:31:49,440 --> 00:31:54,720
libraries so you can't rely on a

928
00:31:51,840 --> 00:31:57,519
developer identifying dangers

929
00:31:54,720 --> 00:31:59,279
identifying sharp spots in your apis and

930
00:31:57,519 --> 00:32:01,120
you definitely can't rely on them to

931
00:31:59,279 --> 00:32:03,760
find secure alternatives as well so if

932
00:32:01,120 --> 00:32:05,279
you you know if you your library uh

933
00:32:03,760 --> 00:32:07,519
has this base level of features and

934
00:32:05,279 --> 00:32:08,559
functionality but it requires

935
00:32:07,519 --> 00:32:11,760
inputs

936
00:32:08,559 --> 00:32:13,600
from other libraries uh or like things

937
00:32:11,760 --> 00:32:16,080
in the core language you can't actually

938
00:32:13,600 --> 00:32:18,080
rely on your users to actually use those

939
00:32:16,080 --> 00:32:18,880
properly and know when to use them as

940
00:32:18,080 --> 00:32:21,279
well

941
00:32:18,880 --> 00:32:23,279
so what this fundamentally means for us

942
00:32:21,279 --> 00:32:26,080
as library authors is that it is not

943
00:32:23,279 --> 00:32:28,320
possible to have too many code examples

944
00:32:26,080 --> 00:32:29,679
it is just like i cannot be clear about

945
00:32:28,320 --> 00:32:31,840
this like

946
00:32:29,679 --> 00:32:33,919
if you think that you have too many code

947
00:32:31,840 --> 00:32:36,240
examples in your cryptography libraries

948
00:32:33,919 --> 00:32:38,000
you don't you can just keep going in

949
00:32:36,240 --> 00:32:40,159
fact you should keep going that will get

950
00:32:38,000 --> 00:32:41,919
the best secure outcomes for your end

951
00:32:40,159 --> 00:32:43,679
users or your developers in this

952
00:32:41,919 --> 00:32:45,200
particular case

953
00:32:43,679 --> 00:32:46,880
and it just reinforces the previous

954
00:32:45,200 --> 00:32:48,159
findings as well that if devs can't find

955
00:32:46,880 --> 00:32:50,000
working code examples in your

956
00:32:48,159 --> 00:32:52,799
documentation they will go to stack

957
00:32:50,000 --> 00:32:55,200
overflow and find a foot gun so

958
00:32:52,799 --> 00:32:57,120
having great code examples

959
00:32:55,200 --> 00:32:59,360
huge impact on the overall security of

960
00:32:57,120 --> 00:33:01,200
delivered code by users

961
00:32:59,360 --> 00:33:03,440
so the main thing that you can do here

962
00:33:01,200 --> 00:33:05,919
as an author is create and maintain an

963
00:33:03,440 --> 00:33:08,480
over abundance of secure code examples

964
00:33:05,919 --> 00:33:10,399
in your libraries so like i said before

965
00:33:08,480 --> 00:33:12,720
if you think that you have too many you

966
00:33:10,399 --> 00:33:14,080
don't just keep going

967
00:33:12,720 --> 00:33:16,320
the other thing that's really important

968
00:33:14,080 --> 00:33:18,559
to do is to identify inputs and

969
00:33:16,320 --> 00:33:20,640
dependencies that your users are

970
00:33:18,559 --> 00:33:22,799
implicitly relying on so other language

971
00:33:20,640 --> 00:33:24,880
features or other libraries that they

972
00:33:22,799 --> 00:33:27,200
need to use to be able to use your

973
00:33:24,880 --> 00:33:29,760
library appropriately and correctly and

974
00:33:27,200 --> 00:33:32,960
securely and when you identify what

975
00:33:29,760 --> 00:33:34,480
those things are either provide secure

976
00:33:32,960 --> 00:33:36,559
wrappers for them

977
00:33:34,480 --> 00:33:38,240
in your library so you know if you're

978
00:33:36,559 --> 00:33:40,000
using a strongly typed language wrap

979
00:33:38,240 --> 00:33:41,440
another type for instance with some

980
00:33:40,000 --> 00:33:43,760
extra security

981
00:33:41,440 --> 00:33:45,919
security checks around it right

982
00:33:43,760 --> 00:33:47,919
and also document them exhaustively so

983
00:33:45,919 --> 00:33:50,159
make sure that all of the code examples

984
00:33:47,919 --> 00:33:52,159
that you have on your website are secure

985
00:33:50,159 --> 00:33:54,240
doesn't even matter if you have a code

986
00:33:52,159 --> 00:33:56,080
example that has a comment above it that

987
00:33:54,240 --> 00:33:58,159
says this is insecure code and you

988
00:33:56,080 --> 00:33:59,679
shouldn't use it developers will go and

989
00:33:58,159 --> 00:34:01,600
click it they will just ignore the

990
00:33:59,679 --> 00:34:03,200
comment and they will just copy the

991
00:34:01,600 --> 00:34:04,720
insecure code and they will run it and

992
00:34:03,200 --> 00:34:07,600
it will get a functional result but it

993
00:34:04,720 --> 00:34:10,240
won't be secure so it's critical that

994
00:34:07,600 --> 00:34:11,839
you have functional secure code examples

995
00:34:10,240 --> 00:34:13,520
and you have an absolute plethora of

996
00:34:11,839 --> 00:34:15,440
them

997
00:34:13,520 --> 00:34:17,520
so the tldr from this particular paper

998
00:34:15,440 --> 00:34:20,240
is that misuse resistance helps but it

999
00:34:17,520 --> 00:34:22,879
ain't enough devs do crave code examples

1000
00:34:20,240 --> 00:34:24,159
and you need to provide them

1001
00:34:22,879 --> 00:34:26,560
all right as we wrap up with the

1002
00:34:24,159 --> 00:34:28,639
research here the last thing that that's

1003
00:34:26,560 --> 00:34:31,520
worth talking about is putting runtime

1004
00:34:28,639 --> 00:34:33,119
warnings into your api for potentially

1005
00:34:31,520 --> 00:34:35,599
insecure usage

1006
00:34:33,119 --> 00:34:36,720
so again another another paper from this

1007
00:34:35,599 --> 00:34:38,560
group

1008
00:34:36,720 --> 00:34:40,000
uh and with a bunch of other researchers

1009
00:34:38,560 --> 00:34:41,919
as well actually called developers

1010
00:34:40,000 --> 00:34:43,280
deserve security warnings too and it

1011
00:34:41,919 --> 00:34:45,599
talks about the effect of integrated

1012
00:34:43,280 --> 00:34:48,240
security on cryptographic api misuse and

1013
00:34:45,599 --> 00:34:49,839
this is published back in back in 2018

1014
00:34:48,240 --> 00:34:51,440
and so the experiment in in this

1015
00:34:49,839 --> 00:34:53,760
particular study was that they had the

1016
00:34:51,440 --> 00:34:55,359
python devs that were assigned three

1017
00:34:53,760 --> 00:34:57,359
tasks with the stop codes so the

1018
00:34:55,359 --> 00:35:00,160
symmetric key generation symmetric

1019
00:34:57,359 --> 00:35:01,520
encryption and key storage and they were

1020
00:35:00,160 --> 00:35:03,599
separated into two separate groups

1021
00:35:01,520 --> 00:35:06,400
whether they were using a control or a

1022
00:35:03,599 --> 00:35:08,320
patch version of pi crypto and again ask

1023
00:35:06,400 --> 00:35:09,440
the same question is your code actually

1024
00:35:08,320 --> 00:35:11,760
secure

1025
00:35:09,440 --> 00:35:13,920
and when we say a patched version of pi

1026
00:35:11,760 --> 00:35:15,440
crypto um they just did some pretty

1027
00:35:13,920 --> 00:35:17,920
basic uh

1028
00:35:15,440 --> 00:35:19,760
pretty basic sort of overloading of like

1029
00:35:17,920 --> 00:35:22,320
the constructor

1030
00:35:19,760 --> 00:35:24,880
so that when you would instantiate a

1031
00:35:22,320 --> 00:35:27,040
potentially insecure

1032
00:35:24,880 --> 00:35:28,880
cryptographic primitive that the library

1033
00:35:27,040 --> 00:35:30,400
was exposing it would print the standard

1034
00:35:28,880 --> 00:35:32,079
out and create

1035
00:35:30,400 --> 00:35:34,480
uh and it would make sure that it would

1036
00:35:32,079 --> 00:35:35,920
uh yeah that when it did create it that

1037
00:35:34,480 --> 00:35:39,200
you knew that you were doing something

1038
00:35:35,920 --> 00:35:40,640
potentially uh potentially unsafe

1039
00:35:39,200 --> 00:35:42,800
and the results of this are really

1040
00:35:40,640 --> 00:35:44,800
fascinating so they found that for the

1041
00:35:42,800 --> 00:35:46,320
control condition and the patch

1042
00:35:44,800 --> 00:35:49,200
condition the functional success was

1043
00:35:46,320 --> 00:35:50,560
about the same right but they found that

1044
00:35:49,200 --> 00:35:52,000
uh for the developers in the patch

1045
00:35:50,560 --> 00:35:53,839
condition that through these extra

1046
00:35:52,000 --> 00:35:56,720
warnings they basically they almost had

1047
00:35:53,839 --> 00:35:58,720
twice the level of producing a secure

1048
00:35:56,720 --> 00:36:00,400
a successful secure result as well as a

1049
00:35:58,720 --> 00:36:02,960
functional result

1050
00:36:00,400 --> 00:36:04,480
so nearly twice the rate and plus they

1051
00:36:02,960 --> 00:36:06,160
also self-assess their solutions as

1052
00:36:04,480 --> 00:36:07,599
being more secure when they were given a

1053
00:36:06,160 --> 00:36:09,520
warning as well

1054
00:36:07,599 --> 00:36:11,760
they also rated the control and the

1055
00:36:09,520 --> 00:36:14,160
patch versions uh equally usable there's

1056
00:36:11,760 --> 00:36:15,599
no negligible

1057
00:36:14,160 --> 00:36:17,599
statistical difference between their

1058
00:36:15,599 --> 00:36:19,520
ratings there

1059
00:36:17,599 --> 00:36:20,800
and so yeah the other really interesting

1060
00:36:19,520 --> 00:36:22,800
finding from this particular study was

1061
00:36:20,800 --> 00:36:25,119
that developers who wrote code that

1062
00:36:22,800 --> 00:36:28,320
triggered a warning were actually 15

1063
00:36:25,119 --> 00:36:30,240
times more likely to convert it to a

1064
00:36:28,320 --> 00:36:32,480
secure solution

1065
00:36:30,240 --> 00:36:34,000
so just that simple prompting like we

1066
00:36:32,480 --> 00:36:36,079
were talking about at the very beginning

1067
00:36:34,000 --> 00:36:38,000
of this talk that is enough to be able

1068
00:36:36,079 --> 00:36:39,280
to change the mindset and get developers

1069
00:36:38,000 --> 00:36:42,320
actually building

1070
00:36:39,280 --> 00:36:44,400
uh applications with your libraries in a

1071
00:36:42,320 --> 00:36:45,920
secure and functional way

1072
00:36:44,400 --> 00:36:47,200
and natalie studied this i'm not going

1073
00:36:45,920 --> 00:36:48,560
to go into the depth of this other paper

1074
00:36:47,200 --> 00:36:49,520
but there was a follow-up study to that

1075
00:36:48,560 --> 00:36:51,119
as well

1076
00:36:49,520 --> 00:36:53,839
that actually uh did like a

1077
00:36:51,119 --> 00:36:54,800
participatory design study for like what

1078
00:36:53,839 --> 00:36:57,760
those warnings were in those

1079
00:36:54,800 --> 00:36:59,119
cryptographic apis and the the tldr the

1080
00:36:57,760 --> 00:37:02,400
thing you need to focus on is that when

1081
00:36:59,119 --> 00:37:05,040
you provide uh errors or warnings

1082
00:37:02,400 --> 00:37:06,880
you want to provide a title message

1083
00:37:05,040 --> 00:37:08,720
color the source code location where

1084
00:37:06,880 --> 00:37:10,320
that particular problem was happening a

1085
00:37:08,720 --> 00:37:11,760
link to an external resource you don't

1086
00:37:10,320 --> 00:37:13,359
have to enumerate all the different

1087
00:37:11,760 --> 00:37:15,040
security problems just you know create a

1088
00:37:13,359 --> 00:37:16,480
link off to documentation on your site

1089
00:37:15,040 --> 00:37:17,599
saying hey if you get this particular

1090
00:37:16,480 --> 00:37:19,520
error this is the sort of thing that you

1091
00:37:17,599 --> 00:37:21,680
need to think about and the last bit is

1092
00:37:19,520 --> 00:37:23,760
message classification here

1093
00:37:21,680 --> 00:37:26,320
so those are really that's it like when

1094
00:37:23,760 --> 00:37:27,760
you create these runtime errors uh also

1095
00:37:26,320 --> 00:37:29,119
like having a way for people to be able

1096
00:37:27,760 --> 00:37:30,480
to suppress them as well is sort of

1097
00:37:29,119 --> 00:37:31,440
useful i'll talk about that in a second

1098
00:37:30,480 --> 00:37:32,400
actually

1099
00:37:31,440 --> 00:37:34,240
um

1100
00:37:32,400 --> 00:37:35,520
but just these five things alone if

1101
00:37:34,240 --> 00:37:37,920
you're able to surface those two

1102
00:37:35,520 --> 00:37:40,000
developers as they are writing code that

1103
00:37:37,920 --> 00:37:41,520
will uh significantly increase the

1104
00:37:40,000 --> 00:37:43,680
likelihood that they will deliver both

1105
00:37:41,520 --> 00:37:46,400
functional and secure code to their end

1106
00:37:43,680 --> 00:37:48,640
users by using your library

1107
00:37:46,400 --> 00:37:51,760
so the really fascinating thing from

1108
00:37:48,640 --> 00:37:54,240
this is that as a library author is that

1109
00:37:51,760 --> 00:37:55,680
you don't have to change your api shape

1110
00:37:54,240 --> 00:37:57,520
you have to change the ergonomics of

1111
00:37:55,680 --> 00:37:59,920
your api to actually increase the secure

1112
00:37:57,520 --> 00:38:02,160
usage of it so just emitting runtime

1113
00:37:59,920 --> 00:38:03,280
warnings does significantly improve

1114
00:38:02,160 --> 00:38:05,839
secure

1115
00:38:03,280 --> 00:38:07,359
secure usage of your library

1116
00:38:05,839 --> 00:38:09,119
without actually changing any of the

1117
00:38:07,359 --> 00:38:11,280
parameters what the interface is to your

1118
00:38:09,119 --> 00:38:13,119
library so if you can just look at what

1119
00:38:11,280 --> 00:38:16,000
is being passed or what is being

1120
00:38:13,119 --> 00:38:17,440
instantiated and you go oh we are

1121
00:38:16,000 --> 00:38:20,079
providing this particular insecure

1122
00:38:17,440 --> 00:38:22,000
function but there are like lots of foot

1123
00:38:20,079 --> 00:38:23,359
guns with doing this you should probably

1124
00:38:22,000 --> 00:38:24,320
not do that

1125
00:38:23,359 --> 00:38:26,240
that

1126
00:38:24,320 --> 00:38:27,920
does change the developer perception

1127
00:38:26,240 --> 00:38:29,599
about whether developing and delivering

1128
00:38:27,920 --> 00:38:31,440
secure code

1129
00:38:29,599 --> 00:38:33,280
and this tallies with that's what i

1130
00:38:31,440 --> 00:38:34,480
started with right it's the psychology

1131
00:38:33,280 --> 00:38:36,800
stupid so

1132
00:38:34,480 --> 00:38:38,640
that priming that is the number one way

1133
00:38:36,800 --> 00:38:40,880
to actually improve the secure usage of

1134
00:38:38,640 --> 00:38:42,400
your of your apis and goes back to what

1135
00:38:40,880 --> 00:38:44,160
they recommended you know prime

1136
00:38:42,400 --> 00:38:47,200
developers on the spot when they need it

1137
00:38:44,160 --> 00:38:49,040
while they are coding so the two things

1138
00:38:47,200 --> 00:38:51,520
that you can do here as a library author

1139
00:38:49,040 --> 00:38:53,839
is make your library emit warnings when

1140
00:38:51,520 --> 00:38:56,000
you detect potentially insecure usage

1141
00:38:53,839 --> 00:38:57,839
and provide an obvious mechanism as well

1142
00:38:56,000 --> 00:39:00,400
to be able to silence warnings the

1143
00:38:57,839 --> 00:39:03,440
reason for that second part um and just

1144
00:39:00,400 --> 00:39:05,440
to be clear i said obvious not easy like

1145
00:39:03,440 --> 00:39:07,599
you might have to you know utter a

1146
00:39:05,440 --> 00:39:09,200
certain incantation to be able to as a

1147
00:39:07,599 --> 00:39:11,200
developer using one of these libraries

1148
00:39:09,200 --> 00:39:12,720
to do the insecure thing

1149
00:39:11,200 --> 00:39:14,560
but you want to provide an obvious way

1150
00:39:12,720 --> 00:39:15,359
to do it it doesn't have to be easy for

1151
00:39:14,560 --> 00:39:16,480
them

1152
00:39:15,359 --> 00:39:18,160
and often that's because there are

1153
00:39:16,480 --> 00:39:19,599
organizational factors that are outside

1154
00:39:18,160 --> 00:39:21,440
the control of the developers like they

1155
00:39:19,599 --> 00:39:22,320
have to use in the secure cipher suite

1156
00:39:21,440 --> 00:39:23,520
because

1157
00:39:22,320 --> 00:39:25,040
they are

1158
00:39:23,520 --> 00:39:26,240
interfacing with some other third-party

1159
00:39:25,040 --> 00:39:28,560
piece of software that they don't have

1160
00:39:26,240 --> 00:39:30,320
any control over so you know you want to

1161
00:39:28,560 --> 00:39:32,400
allow them to do the insecure thing but

1162
00:39:30,320 --> 00:39:34,240
they need to really work hard to be able

1163
00:39:32,400 --> 00:39:36,640
to do that

1164
00:39:34,240 --> 00:39:39,440
so dldr use the runtime warnings to

1165
00:39:36,640 --> 00:39:41,680
prime developers about insecure usage

1166
00:39:39,440 --> 00:39:42,640
during dev

1167
00:39:41,680 --> 00:39:45,280
so

1168
00:39:42,640 --> 00:39:48,079
the takeaways for the talk today

1169
00:39:45,280 --> 00:39:50,640
developers don't really prioritize

1170
00:39:48,079 --> 00:39:52,000
security they focus on performance and

1171
00:39:50,640 --> 00:39:54,640
functionality none of that order they

1172
00:39:52,000 --> 00:39:58,079
focus on functionality and performance

1173
00:39:54,640 --> 00:40:00,320
and really as a cryptographic api uh

1174
00:39:58,079 --> 00:40:02,000
publisher author maintainer

1175
00:40:00,320 --> 00:40:03,520
you have to do a lot of the hard work

1176
00:40:02,000 --> 00:40:05,440
for them upfront to be able to get them

1177
00:40:03,520 --> 00:40:06,960
to produce code that is both functional

1178
00:40:05,440 --> 00:40:09,280
and secure

1179
00:40:06,960 --> 00:40:11,359
you can help developers or and people

1180
00:40:09,280 --> 00:40:12,079
that are using your libraries by

1181
00:40:11,359 --> 00:40:14,480
uh

1182
00:40:12,079 --> 00:40:16,800
prioritize security by providing

1183
00:40:14,480 --> 00:40:19,200
targeted warnings in your apis

1184
00:40:16,800 --> 00:40:22,079
frictionless misuse resistant apis that

1185
00:40:19,200 --> 00:40:23,920
work and are secure and copious secure

1186
00:40:22,079 --> 00:40:25,839
code examples in your documentation

1187
00:40:23,920 --> 00:40:28,319
can't emphasize that enough like just

1188
00:40:25,839 --> 00:40:30,800
keep going with the secure examples in

1189
00:40:28,319 --> 00:40:32,640
your documentation

1190
00:40:30,800 --> 00:40:34,319
most security bugs actually happen on

1191
00:40:32,640 --> 00:40:36,079
the periphery of your crypto

1192
00:40:34,319 --> 00:40:38,160
cryptography library it's inside your

1193
00:40:36,079 --> 00:40:39,440
users applications there's a whole bunch

1194
00:40:38,160 --> 00:40:41,520
of other really interesting research

1195
00:40:39,440 --> 00:40:44,800
into that which i'll link to in a moment

1196
00:40:41,520 --> 00:40:47,359
but it's all about doing the hard work

1197
00:40:44,800 --> 00:40:49,040
to make it easy for your users to be

1198
00:40:47,359 --> 00:40:50,480
able to use your library to be able to

1199
00:40:49,040 --> 00:40:53,520
get a functional

1200
00:40:50,480 --> 00:40:55,280
secure result

1201
00:40:53,520 --> 00:40:56,560
i'm lindsey holmwood thank you very much

1202
00:40:55,280 --> 00:40:59,760
for listening and i'm really keen to

1203
00:40:56,560 --> 00:40:59,760
hear what questions you have

1204
00:41:02,000 --> 00:41:04,880
okay there

1205
00:41:03,200 --> 00:41:06,880
thank you lindsay

1206
00:41:04,880 --> 00:41:07,680
so

1207
00:41:06,880 --> 00:41:10,319
we

1208
00:41:07,680 --> 00:41:12,960
just looking i don't see any questions

1209
00:41:10,319 --> 00:41:14,560
there at the moment

1210
00:41:12,960 --> 00:41:16,640
give people a little bit of time in fact

1211
00:41:14,560 --> 00:41:18,079
while people are thinking about the sort

1212
00:41:16,640 --> 00:41:19,920
of things that they're going to ask i've

1213
00:41:18,079 --> 00:41:22,079
got a heap of other documentation and

1214
00:41:19,920 --> 00:41:23,599
references to point you at so i've i've

1215
00:41:22,079 --> 00:41:25,280
got the slides published over here at

1216
00:41:23,599 --> 00:41:26,880
cifstash.com

1217
00:41:25,280 --> 00:41:27,920
lindsay if you're interested in the sort

1218
00:41:26,880 --> 00:41:29,280
of stuff that we're doing definitely

1219
00:41:27,920 --> 00:41:32,079
come help us we we're trying to build

1220
00:41:29,280 --> 00:41:33,680
next generation cryptography um but the

1221
00:41:32,079 --> 00:41:36,160
references are over here so this is all

1222
00:41:33,680 --> 00:41:39,680
the apa citations i've got links as well

1223
00:41:36,160 --> 00:41:41,200
to uh to all of these all these papers

1224
00:41:39,680 --> 00:41:43,040
in free and open access you don't need

1225
00:41:41,200 --> 00:41:44,480
to sign up for anything

1226
00:41:43,040 --> 00:41:46,000
the other thing as well

1227
00:41:44,480 --> 00:41:47,520
this is additional reading in case

1228
00:41:46,000 --> 00:41:50,160
you're super inspired by the talk you

1229
00:41:47,520 --> 00:41:52,240
want to go learn more about it

1230
00:41:50,160 --> 00:41:54,000
there's this paper from 2011 as well

1231
00:41:52,240 --> 00:41:55,520
that's really fantastic it's an

1232
00:41:54,000 --> 00:41:56,960
empirical study of vulnerabilities in

1233
00:41:55,520 --> 00:41:58,800
cryptographic libraries it builds upon

1234
00:41:56,960 --> 00:42:00,640
that 2013 research that i covered at the

1235
00:41:58,800 --> 00:42:03,040
very beginning of the talk

1236
00:42:00,640 --> 00:42:04,400
but yeah there's a huge amount of really

1237
00:42:03,040 --> 00:42:06,000
interesting research on this and if

1238
00:42:04,400 --> 00:42:09,599
you're passionate about it i highly

1239
00:42:06,000 --> 00:42:09,599
recommend you go and read it

1240
00:42:10,800 --> 00:42:14,960
okay thank you

1241
00:42:12,800 --> 00:42:17,040
so with that i think we can close a

1242
00:42:14,960 --> 00:42:18,800
little a few minutes early

1243
00:42:17,040 --> 00:42:21,599
and

1244
00:42:18,800 --> 00:42:23,599
oh no of hearing there is one question

1245
00:42:21,599 --> 00:42:25,680
okay so what are some examples of the

1246
00:42:23,599 --> 00:42:28,960
common mistakes made by developers in

1247
00:42:25,680 --> 00:42:28,960
the use of the apis

1248
00:42:29,119 --> 00:42:33,599
yeah that's a great question um so if

1249
00:42:32,000 --> 00:42:34,880
you're interested in about in learning

1250
00:42:33,599 --> 00:42:36,720
about that

1251
00:42:34,880 --> 00:42:39,599
um it's these two papers that you want

1252
00:42:36,720 --> 00:42:41,520
to check out so api blind spots this one

1253
00:42:39,599 --> 00:42:43,680
this original study was done with java

1254
00:42:41,520 --> 00:42:44,960
the second study replicated with python

1255
00:42:43,680 --> 00:42:46,720
and java

1256
00:42:44,960 --> 00:42:50,079
so often it's

1257
00:42:46,720 --> 00:42:52,960
down to passing insecure inputs

1258
00:42:50,079 --> 00:42:54,560
mostly copied from code examples

1259
00:42:52,960 --> 00:42:55,760
and what they were able to do in these

1260
00:42:54,560 --> 00:42:57,200
particular studies was that they

1261
00:42:55,760 --> 00:42:59,920
actually enumerate a whole bunch of

1262
00:42:57,200 --> 00:43:01,760
different common um insecure usages of

1263
00:42:59,920 --> 00:43:03,680
these different libraries it mostly

1264
00:43:01,760 --> 00:43:06,160
comes down to the inputs though that the

1265
00:43:03,680 --> 00:43:08,319
developers are passing what those li to

1266
00:43:06,160 --> 00:43:10,560
those libraries so yeah highly recommend

1267
00:43:08,319 --> 00:43:12,800
you go check that out

1268
00:43:10,560 --> 00:43:16,560
okay we have a another one

1269
00:43:12,800 --> 00:43:17,280
uh the tldrs are really useful how would

1270
00:43:16,560 --> 00:43:20,960
you

1271
00:43:17,280 --> 00:43:22,400
evaluate docs uh sufficient and

1272
00:43:20,960 --> 00:43:26,079
and you are not

1273
00:43:22,400 --> 00:43:27,920
getting any more benefits

1274
00:43:26,079 --> 00:43:29,680
that's a really good question um because

1275
00:43:27,920 --> 00:43:32,079
you know at the end of the day all

1276
00:43:29,680 --> 00:43:33,440
organizations have to economize like we

1277
00:43:32,079 --> 00:43:35,520
would love to be able to sit and write

1278
00:43:33,440 --> 00:43:37,520
documentation and code examples for the

1279
00:43:35,520 --> 00:43:40,079
rest of our lives um but there is

1280
00:43:37,520 --> 00:43:42,960
obviously a point of diminishing returns

1281
00:43:40,079 --> 00:43:44,640
um my strong recommendation for that is

1282
00:43:42,960 --> 00:43:46,560
actually usability testing with your

1283
00:43:44,640 --> 00:43:48,319
users ideally you could partner up with

1284
00:43:46,560 --> 00:43:49,760
an academic institution but you know if

1285
00:43:48,319 --> 00:43:51,440
you've got some sort of product function

1286
00:43:49,760 --> 00:43:53,280
if you're in a commercial organization

1287
00:43:51,440 --> 00:43:55,359
or you've got maintainers that have got

1288
00:43:53,280 --> 00:43:57,920
that sort of responsibility i highly

1289
00:43:55,359 --> 00:43:59,119
recommend constructing tasks like the

1290
00:43:57,920 --> 00:44:01,040
sort of things that were outlines in

1291
00:43:59,119 --> 00:44:03,599
these papers to actually look at how

1292
00:44:01,040 --> 00:44:05,599
people are using it um you you're using

1293
00:44:03,599 --> 00:44:07,359
your using your libraries as well as the

1294
00:44:05,599 --> 00:44:10,480
documentation that you are that you are

1295
00:44:07,359 --> 00:44:13,040
providing um i strongly recommend just

1296
00:44:10,480 --> 00:44:14,960
doing the qualitative and quantitative

1297
00:44:13,040 --> 00:44:16,480
research um

1298
00:44:14,960 --> 00:44:18,160
uh the other thing as well that you can

1299
00:44:16,480 --> 00:44:19,200
do is check out like look at how

1300
00:44:18,160 --> 00:44:21,280
particularly if you're on the open

1301
00:44:19,200 --> 00:44:22,800
source space look at like the github

1302
00:44:21,280 --> 00:44:25,040
code search how people are actually

1303
00:44:22,800 --> 00:44:26,160
using it there um being able to you know

1304
00:44:25,040 --> 00:44:27,839
reach out to the developers that

1305
00:44:26,160 --> 00:44:30,000
originally wrote that code and find out

1306
00:44:27,839 --> 00:44:32,720
how exactly they came to that point

1307
00:44:30,000 --> 00:44:33,440
um you know at the end of the day

1308
00:44:32,720 --> 00:44:34,800
i

1309
00:44:33,440 --> 00:44:36,640
every organization is going to have to

1310
00:44:34,800 --> 00:44:37,680
make a choice about where they economize

1311
00:44:36,640 --> 00:44:40,400
and what that point of diminishing

1312
00:44:37,680 --> 00:44:41,839
returns is um but you know have a think

1313
00:44:40,400 --> 00:44:43,760
about that ahead of time rather than

1314
00:44:41,839 --> 00:44:45,359
just doing a like it has to be a the

1315
00:44:43,760 --> 00:44:46,800
other thing as well has to be a constant

1316
00:44:45,359 --> 00:44:49,119
like repetitive

1317
00:44:46,800 --> 00:44:50,960
maintainable pace and process like as

1318
00:44:49,119 --> 00:44:52,720
you ship new features

1319
00:44:50,960 --> 00:44:54,640
and provide new functionality to your

1320
00:44:52,720 --> 00:44:56,319
users um you're going to have to

1321
00:44:54,640 --> 00:44:58,160
maintain all of the code examples that

1322
00:44:56,319 --> 00:44:59,359
you've got so it's it's also

1323
00:44:58,160 --> 00:45:02,000
that classic thing of like you know

1324
00:44:59,359 --> 00:45:03,359
having so many tests that eventually

1325
00:45:02,000 --> 00:45:04,880
like the crushing weight of the test

1326
00:45:03,359 --> 00:45:06,319
actually stops you from being able to do

1327
00:45:04,880 --> 00:45:07,599
delivery so it's this interesting

1328
00:45:06,319 --> 00:45:09,440
balance that you've got to find so

1329
00:45:07,599 --> 00:45:11,280
perhaps some of the heuristics that you

1330
00:45:09,440 --> 00:45:13,200
might use to work out you know what's an

1331
00:45:11,280 --> 00:45:15,200
optimal amount of test coverage that

1332
00:45:13,200 --> 00:45:17,520
probably applies to what we're doing

1333
00:45:15,200 --> 00:45:20,160
here with like code code examples for

1334
00:45:17,520 --> 00:45:20,160
your libraries

1335
00:45:20,480 --> 00:45:25,200
okay and maybe the final one because we

1336
00:45:22,640 --> 00:45:27,839
are on at 16 30 now

1337
00:45:25,200 --> 00:45:32,720
any advice on how to improve apis when

1338
00:45:27,839 --> 00:45:35,520
you need to also maintain compatibility

1339
00:45:32,720 --> 00:45:37,839
yeah that's a really great question um

1340
00:45:35,520 --> 00:45:40,240
i will be out front and say uh i don't

1341
00:45:37,839 --> 00:45:42,800
have any good answers for that um

1342
00:45:40,240 --> 00:45:44,000
i think that providing uh

1343
00:45:42,800 --> 00:45:45,839
sort of using some of the earlier

1344
00:45:44,000 --> 00:45:47,200
techniques that i talked about

1345
00:45:45,839 --> 00:45:48,800
actually the techniques towards the end

1346
00:45:47,200 --> 00:45:50,079
about providing warnings and things like

1347
00:45:48,800 --> 00:45:51,920
that

1348
00:45:50,079 --> 00:45:52,800
a really important way to be able to do

1349
00:45:51,920 --> 00:45:54,880
that

1350
00:45:52,800 --> 00:45:56,640
i would also argue that

1351
00:45:54,880 --> 00:45:58,240
it's okay to break backwards

1352
00:45:56,640 --> 00:46:00,880
incompatibility at certain points as

1353
00:45:58,240 --> 00:46:02,880
well but if you do that it is vitally

1354
00:46:00,880 --> 00:46:04,880
important that you create

1355
00:46:02,880 --> 00:46:07,040
smooth upgrade paths to the newer

1356
00:46:04,880 --> 00:46:08,640
version of whatever you're producing um

1357
00:46:07,040 --> 00:46:10,960
so google did that relatively

1358
00:46:08,640 --> 00:46:12,560
successfully with keys are and tink

1359
00:46:10,960 --> 00:46:13,760
they've got to a point where they

1360
00:46:12,560 --> 00:46:15,200
actually got a whole bunch of the

1361
00:46:13,760 --> 00:46:16,880
feedback from these particular studies

1362
00:46:15,200 --> 00:46:18,560
and went oh no

1363
00:46:16,880 --> 00:46:19,920
we really need to improve the usability

1364
00:46:18,560 --> 00:46:22,720
here so they actually just end of life

1365
00:46:19,920 --> 00:46:24,480
keys are entirely uh i probably use that

1366
00:46:22,720 --> 00:46:25,520
as an example about how to how to sort

1367
00:46:24,480 --> 00:46:27,520
of you know

1368
00:46:25,520 --> 00:46:29,359
encourage users to upgrade to the latest

1369
00:46:27,520 --> 00:46:31,520
greatest most secure most functional

1370
00:46:29,359 --> 00:46:31,520
thing

1371
00:46:31,599 --> 00:46:36,160
okay another question has popped up

1372
00:46:33,920 --> 00:46:38,640
let's see if we can squeeze that one in

1373
00:46:36,160 --> 00:46:40,000
to what extent can static types help

1374
00:46:38,640 --> 00:46:43,520
ensure

1375
00:46:40,000 --> 00:46:46,720
secure api usage uh that is great any

1376
00:46:43,520 --> 00:46:47,839
papers that look at this aspect of api

1377
00:46:46,720 --> 00:46:50,640
design

1378
00:46:47,839 --> 00:46:52,960
yes i don't have that linked in here but

1379
00:46:50,640 --> 00:46:54,720
i'll find you in the chat after this um

1380
00:46:52,960 --> 00:46:57,359
there is lots of research into that and

1381
00:46:54,720 --> 00:46:58,720
a lot of it is contradictory

1382
00:46:57,359 --> 00:47:00,800
so

1383
00:46:58,720 --> 00:47:02,400
yeah static analysis good but i'll link

1384
00:47:00,800 --> 00:47:04,960
you to the other stuff perfectly

1385
00:47:02,400 --> 00:47:07,200
take all that on the post talk chat

1386
00:47:04,960 --> 00:47:09,040
and uh definitely lindsay have a look at

1387
00:47:07,200 --> 00:47:10,720
the comments it was quite active while

1388
00:47:09,040 --> 00:47:12,160
you were talking

1389
00:47:10,720 --> 00:47:15,640
thank you very much

1390
00:47:12,160 --> 00:47:15,640
thank you very much