1
00:00:05,239 --> 00:00:07,259
[music]

2
00:00:09,825 --> 00:00:11,845
[music]

3
00:00:14,960 --> 00:00:16,720
coming back. All right, so our next talk

4
00:00:16,720 --> 00:00:18,880
is called Whoops, I accidentally leaked

5
00:00:18,880 --> 00:00:21,359
my credentials again. So, please make

6
00:00:21,359 --> 00:00:25,480
welcome Eve Martin Jones.

7
00:00:32,399 --> 00:00:34,960
Cool. Hi everyone. I'm Eve. Um, I am a

8
00:00:34,960 --> 00:00:36,399
software engineer on the Google open

9
00:00:36,399 --> 00:00:38,719
source security team or Ghost. And today

10
00:00:38,719 --> 00:00:40,239
I'm going to talk to you about leaking

11
00:00:40,239 --> 00:00:42,480
your credentials.

12
00:00:42,480 --> 00:00:44,480
The problem of leak credentials is not a

13
00:00:44,480 --> 00:00:46,079
new one. People have been leaking

14
00:00:46,079 --> 00:00:47,600
credentials ever since there have been

15
00:00:47,600 --> 00:00:50,000
credentials to leak. Because of the

16
00:00:50,000 --> 00:00:51,360
longevity of this issue, you might

17
00:00:51,360 --> 00:00:54,399
imagine that it's a solved problem.

18
00:00:54,399 --> 00:00:56,160
But actually uh according to a bunch of

19
00:00:56,160 --> 00:00:58,079
research, leak credentials are still one

20
00:00:58,079 --> 00:01:00,239
of the most common cloud compromised

21
00:01:00,239 --> 00:01:02,320
factors. And not only are they more

22
00:01:02,320 --> 00:01:03,680
common than you might think, they're

23
00:01:03,680 --> 00:01:06,320
also extremely expensive.

24
00:01:06,320 --> 00:01:07,760
So how is it that projects and

25
00:01:07,760 --> 00:01:09,840
organizations today are still being

26
00:01:09,840 --> 00:01:12,240
compromised by leak credentials?

27
00:01:12,240 --> 00:01:15,439
Well, there are two primary reasons.

28
00:01:15,439 --> 00:01:16,960
Firstly, if you're running a modern

29
00:01:16,960 --> 00:01:19,119
software system, um developers and

30
00:01:19,119 --> 00:01:20,799
organizations have to manage a lot of

31
00:01:20,799 --> 00:01:22,799
different credentials. If you're

32
00:01:22,799 --> 00:01:24,479
developing an app, there's a good chance

33
00:01:24,479 --> 00:01:26,159
that it's running on a third party cloud

34
00:01:26,159 --> 00:01:27,920
provider um as a cluster of

35
00:01:27,920 --> 00:01:29,680
microservices run on Kubernetes

36
00:01:29,680 --> 00:01:31,040
interacting with a bunch of other

37
00:01:31,040 --> 00:01:33,920
systems like databases, pubsub, blob

38
00:01:33,920 --> 00:01:35,920
stores.

39
00:01:35,920 --> 00:01:37,759
All of these different p pieces and all

40
00:01:37,759 --> 00:01:39,119
of the people developing the different

41
00:01:39,119 --> 00:01:41,040
pieces need credentials to authenticate

42
00:01:41,040 --> 00:01:42,799
with each other. So that means you're

43
00:01:42,799 --> 00:01:44,720
probably managing hundreds or maybe even

44
00:01:44,720 --> 00:01:46,640
thousands of different credentials, each

45
00:01:46,640 --> 00:01:48,720
of which provide access to some or all

46
00:01:48,720 --> 00:01:51,759
of your infrastructure.

47
00:01:51,759 --> 00:01:53,680
Secondly, it's easier than ever to leak

48
00:01:53,680 --> 00:01:55,680
your credentials. Especially as open

49
00:01:55,680 --> 00:01:57,360
source developers, there are so many

50
00:01:57,360 --> 00:01:58,640
ways you might accidentally make your

51
00:01:58,640 --> 00:02:01,119
credentials public to the internet.

52
00:02:01,119 --> 00:02:02,880
Publishing a credential can occur when

53
00:02:02,880 --> 00:02:04,640
pushing to a publicly hosted source code

54
00:02:04,640 --> 00:02:07,119
repository like GitHub and GitLab, like

55
00:02:07,119 --> 00:02:09,599
the example here. But it can also look

56
00:02:09,599 --> 00:02:11,039
like accidentally building a leak

57
00:02:11,039 --> 00:02:12,640
credential into an open source software

58
00:02:12,640 --> 00:02:15,840
package or a Docker image.

59
00:02:15,840 --> 00:02:17,440
And that credentials are so easy to

60
00:02:17,440 --> 00:02:19,520
accidentally leak is a real worry uh

61
00:02:19,520 --> 00:02:20,879
because research shows that leak

62
00:02:20,879 --> 00:02:22,959
credentials appear um that appear in

63
00:02:22,959 --> 00:02:25,120
open source software get compromised

64
00:02:25,120 --> 00:02:26,560
literally within minutes of being

65
00:02:26,560 --> 00:02:28,720
published. Um so this is a link to a

66
00:02:28,720 --> 00:02:30,480
really interesting blog post where the

67
00:02:30,480 --> 00:02:32,319
author deployed canary tokens to a bunch

68
00:02:32,319 --> 00:02:34,000
of different artifact registries and

69
00:02:34,000 --> 00:02:36,560
took and and saw how long it took for

70
00:02:36,560 --> 00:02:38,879
someone to attempt to compromise them.

71
00:02:38,879 --> 00:02:41,519
and MPM, Pipia, and GitHub all had

72
00:02:41,519 --> 00:02:43,200
exploitation attempts in under 3

73
00:02:43,200 --> 00:02:46,080
minutes.

74
00:02:46,080 --> 00:02:47,920
Now, there is a bunch of prior art in

75
00:02:47,920 --> 00:02:49,360
secret scanning to mitigate token

76
00:02:49,360 --> 00:02:51,680
compromise, but it's mainly focused on

77
00:02:51,680 --> 00:02:53,440
scanning source code repositories. So,

78
00:02:53,440 --> 00:02:55,120
for example, GitHub and GitLab have

79
00:02:55,120 --> 00:02:56,720
secret scanning that you can enable on

80
00:02:56,720 --> 00:02:58,959
your repositories.

81
00:02:58,959 --> 00:03:00,800
That's obviously enormously valuable and

82
00:03:00,800 --> 00:03:03,040
I'm sure it's um prevented a number of

83
00:03:03,040 --> 00:03:04,480
like compromises due to leak

84
00:03:04,480 --> 00:03:06,000
credentials.

85
00:03:06,000 --> 00:03:07,680
But there are so many different types of

86
00:03:07,680 --> 00:03:09,680
open source artifacts. And this is what

87
00:03:09,680 --> 00:03:12,159
inspired Ghost, the team I work on, to

88
00:03:12,159 --> 00:03:13,840
create a pipeline for identifying and

89
00:03:13,840 --> 00:03:15,760
reporting leak credentials in a much

90
00:03:15,760 --> 00:03:17,200
broader variety of open source

91
00:03:17,200 --> 00:03:19,280
artifacts. And the results of building

92
00:03:19,280 --> 00:03:20,800
and running that pipeline is what I want

93
00:03:20,800 --> 00:03:23,760
to talk to you about today.

94
00:03:23,760 --> 00:03:25,280
So in this session, we're going to go

95
00:03:25,280 --> 00:03:27,120
over a highle overview of the pipeline

96
00:03:27,120 --> 00:03:29,120
that we built to scan the artifacts.

97
00:03:29,120 --> 00:03:30,400
Then we'll talk about some of the key

98
00:03:30,400 --> 00:03:32,000
findings we had from conducting that

99
00:03:32,000 --> 00:03:34,400
scan. And then we'll look at some useful

100
00:03:34,400 --> 00:03:36,640
strategies for mitigating um and or

101
00:03:36,640 --> 00:03:38,159
preventing credential leaks in your

102
00:03:38,159 --> 00:03:40,239
project or organization.

103
00:03:40,239 --> 00:03:41,920
And finally, we'll have a look at next

104
00:03:41,920 --> 00:03:43,519
steps uh for the credential scanning

105
00:03:43,519 --> 00:03:46,080
pipeline that we built.

106
00:03:46,080 --> 00:03:48,000
Okay, let's get started by looking at

107
00:03:48,000 --> 00:03:49,840
the pipeline.

108
00:03:49,840 --> 00:03:51,680
So this pipeline scans artifacts from a

109
00:03:51,680 --> 00:03:53,760
variety of sources. It scans artifacts

110
00:03:53,760 --> 00:03:55,280
from source code repositories like

111
00:03:55,280 --> 00:03:57,519
GitHub or GitLab. It scans artifacts

112
00:03:57,519 --> 00:04:00,400
from package registries like mpmjs,

113
00:04:00,400 --> 00:04:03,920
pippi, nougat, maven. And it also scans

114
00:04:03,920 --> 00:04:05,920
public docker image repositories like

115
00:04:05,920 --> 00:04:08,400
docker hub.

116
00:04:08,400 --> 00:04:10,239
Artifacts get pulled into the pipeline

117
00:04:10,239 --> 00:04:12,480
by an ingestion backend. And these are

118
00:04:12,480 --> 00:04:14,400
responsible for subscribing to upstream

119
00:04:14,400 --> 00:04:16,400
alerts for new artifacts, downloading

120
00:04:16,400 --> 00:04:17,759
those artifacts and their relevant

121
00:04:17,759 --> 00:04:20,000
metadata and then maintaining the state

122
00:04:20,000 --> 00:04:22,160
of them over time. Um so for example

123
00:04:22,160 --> 00:04:23,600
artifacts sometimes get deleted from

124
00:04:23,600 --> 00:04:25,680
upstream.

125
00:04:25,680 --> 00:04:27,759
They then get sent to our extraction

126
00:04:27,759 --> 00:04:29,680
backends which unpack the artifact into

127
00:04:29,680 --> 00:04:32,240
its constituent files. So for example a

128
00:04:32,240 --> 00:04:34,560
Pippi release will be unarchived. A

129
00:04:34,560 --> 00:04:36,320
source code repo will have each blob

130
00:04:36,320 --> 00:04:38,240
inspected and a dock image will be

131
00:04:38,240 --> 00:04:40,160
decomposed into its layers and then

132
00:04:40,160 --> 00:04:41,840
further decomposed into all the files in

133
00:04:41,840 --> 00:04:43,919
those layers.

134
00:04:43,919 --> 00:04:45,280
Each of these files then go to the

135
00:04:45,280 --> 00:04:46,960
scanning back end which checks for a

136
00:04:46,960 --> 00:04:48,880
variety of different credential types.

137
00:04:48,880 --> 00:04:51,680
If a match is identified, uh, it reports

138
00:04:51,680 --> 00:04:53,440
it to the reporting back end which sends

139
00:04:53,440 --> 00:04:55,840
it to the upstream credential provider

140
00:04:55,840 --> 00:04:57,360
and then they will take some remediating

141
00:04:57,360 --> 00:04:58,960
action. And this always includes

142
00:04:58,960 --> 00:05:00,639
notifying the user that their credential

143
00:05:00,639 --> 00:05:02,960
has been compromised. And depending on

144
00:05:02,960 --> 00:05:04,800
the type of credential, they might also

145
00:05:04,800 --> 00:05:06,479
immediately revoke the credential or

146
00:05:06,479 --> 00:05:07,919
schedule it for destruction in a few

147
00:05:07,919 --> 00:05:09,600
weeks.

148
00:05:09,600 --> 00:05:11,360
So let's look at how this might work in

149
00:05:11,360 --> 00:05:12,880
practice. And we can use a PPI release

150
00:05:12,880 --> 00:05:14,960
as an example.

151
00:05:14,960 --> 00:05:17,520
So the back end's monitoring pipi.org.

152
00:05:17,520 --> 00:05:20,080
sees a new release is published,

153
00:05:20,080 --> 00:05:22,800
then downloads and saves the release.

154
00:05:22,800 --> 00:05:24,720
And in this case, uh this release is a

155
00:05:24,720 --> 00:05:26,880
will file. So the pipeline unzips it and

156
00:05:26,880 --> 00:05:28,720
then inspects it to produce an index of

157
00:05:28,720 --> 00:05:31,440
each file in each directory,

158
00:05:31,440 --> 00:05:33,199
then each of the files that are unpacked

159
00:05:33,199 --> 00:05:35,680
get sent to be scanned. Most of them

160
00:05:35,680 --> 00:05:39,199
probably harmless. Um, but if there is a

161
00:05:39,199 --> 00:05:40,479
file that does contain a leak

162
00:05:40,479 --> 00:05:43,520
credential, such as a GCP API key, the

163
00:05:43,520 --> 00:05:45,600
back end detects it, passes it to the

164
00:05:45,600 --> 00:05:47,360
reporting back end, and then that back

165
00:05:47,360 --> 00:05:49,520
end is going to assemble a report. Um,

166
00:05:49,520 --> 00:05:51,199
and in this scenario, this is a Google

167
00:05:51,199 --> 00:05:52,800
Cloud API key. So, it's going to get

168
00:05:52,800 --> 00:05:54,320
reported to the Google Cloud credentials

169
00:05:54,320 --> 00:05:56,400
protection team. And then they'll take

170
00:05:56,400 --> 00:05:58,320
some steps to remediate the leak. Um,

171
00:05:58,320 --> 00:06:00,080
which I, as I mentioned, is definitely

172
00:06:00,080 --> 00:06:02,479
notifying the user and perhaps also um,

173
00:06:02,479 --> 00:06:05,680
revoking the key.

174
00:06:05,680 --> 00:06:07,840
So far, we've scanned over 6.5 billion

175
00:06:07,840 --> 00:06:09,680
files extracted from hundreds of

176
00:06:09,680 --> 00:06:12,160
millions of open source artifacts. This

177
00:06:12,160 --> 00:06:13,680
includes not just newly published

178
00:06:13,680 --> 00:06:15,680
artifacts, but also a corpus of

179
00:06:15,680 --> 00:06:18,639
historical artifacts.

180
00:06:18,639 --> 00:06:20,479
We've partnered with uh a couple of

181
00:06:20,479 --> 00:06:23,600
credential providers uh GCP, Ruby Gems,

182
00:06:23,600 --> 00:06:26,000
Pippi um to scan for a variety of

183
00:06:26,000 --> 00:06:28,240
credentials and we're very focused on

184
00:06:28,240 --> 00:06:30,479
scanning only for credentials that are

185
00:06:30,479 --> 00:06:32,560
um reportable like credentials where we

186
00:06:32,560 --> 00:06:34,080
can take an action to help remediate

187
00:06:34,080 --> 00:06:35,759
them. So for example, we're not

188
00:06:35,759 --> 00:06:37,520
interested in scanning for private keys

189
00:06:37,520 --> 00:06:38,720
um because they don't have an inherent

190
00:06:38,720 --> 00:06:41,120
provider that we can make a remediation

191
00:06:41,120 --> 00:06:42,560
response.

192
00:06:42,560 --> 00:06:44,240
We've also recently introduced scanning

193
00:06:44,240 --> 00:06:46,080
for some additional GCP credentials and

194
00:06:46,080 --> 00:06:48,240
also for Pyex tokens, but we don't have

195
00:06:48,240 --> 00:06:49,520
complete results for those yet, so

196
00:06:49,520 --> 00:06:50,800
they're going to be excluded from the

197
00:06:50,800 --> 00:06:54,000
latest slides.

198
00:06:54,000 --> 00:06:55,919
Okay, you know how the pipeline works.

199
00:06:55,919 --> 00:06:57,440
Let's talk about what we found from

200
00:06:57,440 --> 00:06:59,199
running the pipeline across our corpus

201
00:06:59,199 --> 00:07:02,319
of open source software.

202
00:07:02,319 --> 00:07:04,560
A few caveats. Um, first of all,

203
00:07:04,560 --> 00:07:06,319
different credential providers give you

204
00:07:06,319 --> 00:07:07,759
different levels of detail when you

205
00:07:07,759 --> 00:07:10,479
report a credential. So that means that

206
00:07:10,479 --> 00:07:11,919
some credential providers will tell you

207
00:07:11,919 --> 00:07:13,280
yes, you found a valid credential

208
00:07:13,280 --> 00:07:14,720
whereas others will just kind of say

209
00:07:14,720 --> 00:07:17,280
nothing. Um so that means that not all

210
00:07:17,280 --> 00:07:19,840
of the analysis here is able to include

211
00:07:19,840 --> 00:07:21,280
every credential from every provider

212
00:07:21,280 --> 00:07:22,479
because we don't have information about

213
00:07:22,479 --> 00:07:24,400
whether the credential that we found was

214
00:07:24,400 --> 00:07:26,720
valid.

215
00:07:26,720 --> 00:07:29,280
Secondly, to be as cautious as possible,

216
00:07:29,280 --> 00:07:30,560
um we're not storing a ton of

217
00:07:30,560 --> 00:07:31,919
information about the credentials once

218
00:07:31,919 --> 00:07:33,919
we've um once we've reported them. And

219
00:07:33,919 --> 00:07:36,000
that's just because don't really want to

220
00:07:36,000 --> 00:07:37,599
have a massive database of potentially

221
00:07:37,599 --> 00:07:39,160
unrevoked credentials. Um,

222
00:07:39,160 --> 00:07:40,720
[gasps and laughter]

223
00:07:40,720 --> 00:07:42,240
so that does make some type of analysis

224
00:07:42,240 --> 00:07:45,039
a bit harder though. And finally,

225
00:07:45,039 --> 00:07:47,120
because large chunks of this analysis is

226
00:07:47,120 --> 00:07:49,840
based on historical scans, it's kind of

227
00:07:49,840 --> 00:07:51,919
biased towards unrevoked credentials. So

228
00:07:51,919 --> 00:07:53,280
say you had a credential that got leaked

229
00:07:53,280 --> 00:07:55,520
in 2018, was found and reported and

230
00:07:55,520 --> 00:07:57,199
remediated. That's not going to show up

231
00:07:57,199 --> 00:07:59,120
in our scan. But say you had a

232
00:07:59,120 --> 00:08:00,960
credential in 2018 that was found,

233
00:08:00,960 --> 00:08:02,240
reported, and then the person didn't

234
00:08:02,240 --> 00:08:03,759
remediate it. that is going to show up

235
00:08:03,759 --> 00:08:05,919
in the scan. So, um you might see a bit

236
00:08:05,919 --> 00:08:07,360
how that plays out in the data that

237
00:08:07,360 --> 00:08:10,160
comes up. Okay, now that we've got that

238
00:08:10,160 --> 00:08:11,360
all out of the way, let's have a look at

239
00:08:11,360 --> 00:08:13,360
some of our results.

240
00:08:13,360 --> 00:08:16,080
So, we identified um about 20,000 unique

241
00:08:16,080 --> 00:08:18,560
valid credentials out of the 6.5 billion

242
00:08:18,560 --> 00:08:21,360
files we scanned. Um we also I've said

243
00:08:21,360 --> 00:08:22,639
unique and valid here. We also

244
00:08:22,639 --> 00:08:24,240
identified a bunch of duplicate

245
00:08:24,240 --> 00:08:26,160
credentials where someone would publish

246
00:08:26,160 --> 00:08:27,919
the same credential in like a thousand

247
00:08:27,919 --> 00:08:31,280
different MPM versions. Um, and then we

248
00:08:31,280 --> 00:08:33,120
also found a bunch of test credentials

249
00:08:33,120 --> 00:08:36,719
or uh placeholder credentials. Um, so I

250
00:08:36,719 --> 00:08:38,479
think the total amount we flagged was

251
00:08:38,479 --> 00:08:40,560
about 1.5 million and then of them

252
00:08:40,560 --> 00:08:44,880
20,000 were found to be valid.

253
00:08:44,880 --> 00:08:46,160
The majority of what we uncovered were

254
00:08:46,160 --> 00:08:48,959
GCP API keys. Um, so that's a bit funny

255
00:08:48,959 --> 00:08:51,839
because GCP API keys are not technically

256
00:08:51,839 --> 00:08:54,560
um credentials, but it is bad if you

257
00:08:54,560 --> 00:08:56,080
leak them um because they can be used

258
00:08:56,080 --> 00:08:57,760
for resource abuse. Um, so it's

259
00:08:57,760 --> 00:08:58,959
important to still find and remediate

260
00:08:58,959 --> 00:09:01,120
them. The next largest number of

261
00:09:01,120 --> 00:09:03,279
credentials were GCP service account

262
00:09:03,279 --> 00:09:05,600
keys, which make up about 13% of the

263
00:09:05,600 --> 00:09:08,080
credentials we discovered. These ones

264
00:09:08,080 --> 00:09:09,839
are a lot more serious if you leak them

265
00:09:09,839 --> 00:09:11,600
um because they have outsized privileges

266
00:09:11,600 --> 00:09:13,279
and as we'll see in a bit, they can

267
00:09:13,279 --> 00:09:15,040
grant access to entire projects or

268
00:09:15,040 --> 00:09:17,120
organizations.

269
00:09:17,120 --> 00:09:19,680
We also found small number of Pippi API

270
00:09:19,680 --> 00:09:23,760
keys and Ruby gems keys.

271
00:09:23,760 --> 00:09:27,519
Okay. Um, now let's have a look at the

272
00:09:27,519 --> 00:09:29,440
sources of the lead credentials. And I'm

273
00:09:29,440 --> 00:09:31,200
going to go through this by each

274
00:09:31,200 --> 00:09:32,880
credential type as the results are quite

275
00:09:32,880 --> 00:09:34,240
different depending on which credential

276
00:09:34,240 --> 00:09:36,880
we're looking at. And again, in this se

277
00:09:36,880 --> 00:09:37,920
section, we're talking about the number

278
00:09:37,920 --> 00:09:39,440
of unique credentials found in each

279
00:09:39,440 --> 00:09:42,000
source.

280
00:09:42,000 --> 00:09:44,000
Let's start with GCP service account

281
00:09:44,000 --> 00:09:45,920
keys. The biggest source of leaked

282
00:09:45,920 --> 00:09:47,600
service account keys here was actually

283
00:09:47,600 --> 00:09:50,880
Docker images. Um, and we can take a

284
00:09:50,880 --> 00:09:54,320
look at why this might be.

285
00:09:54,320 --> 00:09:57,120
Uh so in a source code repo and even in

286
00:09:57,120 --> 00:09:59,519
a software package it's a lot clearer to

287
00:09:59,519 --> 00:10:00,880
know what is going into the final

288
00:10:00,880 --> 00:10:03,200
artifact that you publish but because of

289
00:10:03,200 --> 00:10:05,040
their composition what goes into a

290
00:10:05,040 --> 00:10:07,760
docker image is a little bit less clear.

291
00:10:07,760 --> 00:10:10,080
Handwaving some details um a docker

292
00:10:10,080 --> 00:10:11,839
image is basically just an ordered list

293
00:10:11,839 --> 00:10:14,000
of file system layers um where each

294
00:10:14,000 --> 00:10:15,360
layer represents a diff from the

295
00:10:15,360 --> 00:10:17,360
previous layer layer of files that were

296
00:10:17,360 --> 00:10:20,160
added and files that were deleted. These

297
00:10:20,160 --> 00:10:22,240
layers get applied sequentially to

298
00:10:22,240 --> 00:10:26,000
produce the final output file system.

299
00:10:26,000 --> 00:10:28,079
So say for example in the first layer we

300
00:10:28,079 --> 00:10:30,560
add file A. In the second layer we add

301
00:10:30,560 --> 00:10:33,040
file B. But then in the third layer we

302
00:10:33,040 --> 00:10:35,200
delete file A.

303
00:10:35,200 --> 00:10:37,200
When I pull that image and apply those

304
00:10:37,200 --> 00:10:39,040
layers the final file system only

305
00:10:39,040 --> 00:10:42,560
includes file B, not file A. So you can

306
00:10:42,560 --> 00:10:43,920
probably imagine how this could cause

307
00:10:43,920 --> 00:10:45,200
some problems when it comes to leak

308
00:10:45,200 --> 00:10:46,880
credentials.

309
00:10:46,880 --> 00:10:48,480
Say we introduced a file called

310
00:10:48,480 --> 00:10:50,240
secrets.txt. txt in the first layer of

311
00:10:50,240 --> 00:10:51,200
our image [clears throat] and it's got

312
00:10:51,200 --> 00:10:53,760
all our secrets in it. But then we

313
00:10:53,760 --> 00:10:56,240
deleted it later on.

314
00:10:56,240 --> 00:10:58,240
The final file system I have doesn't

315
00:10:58,240 --> 00:10:59,360
include that file with the leak

316
00:10:59,360 --> 00:11:00,959
credentials.

317
00:11:00,959 --> 00:11:02,720
So if I just scan that final file system

318
00:11:02,720 --> 00:11:04,399
before I push it out to the world, I

319
00:11:04,399 --> 00:11:05,760
might incorrectly conclude that the

320
00:11:05,760 --> 00:11:07,360
image does not contain any leaked

321
00:11:07,360 --> 00:11:09,120
credentials.

322
00:11:09,120 --> 00:11:10,240
But we know that there are leak

323
00:11:10,240 --> 00:11:13,519
credentials hidden in that first layer.

324
00:11:13,519 --> 00:11:15,760
So instead, when uh shipping a Docker

325
00:11:15,760 --> 00:11:17,440
image or when scanning a Docker image,

326
00:11:17,440 --> 00:11:18,880
it's really important to individ

327
00:11:18,880 --> 00:11:21,279
individually unpack and scan each layer

328
00:11:21,279 --> 00:11:23,360
to check for leak credentials. And that

329
00:11:23,360 --> 00:11:25,120
way you're looking at the actual full

330
00:11:25,120 --> 00:11:27,102
set of files that are being shipped.

331
00:11:27,102 --> 00:11:28,079
[snorts]

332
00:11:28,079 --> 00:11:30,000
And this isn't a hypothetical concern.

333
00:11:30,000 --> 00:11:33,200
Um code, which is a tool for unifying

334
00:11:33,200 --> 00:11:35,600
and visualizing code coverage data, um

335
00:11:35,600 --> 00:11:38,240
and is part of a bunch of CI systems. Uh

336
00:11:38,240 --> 00:11:40,880
in 2021 they revealed that their system

337
00:11:40,880 --> 00:11:42,880
was targeted by a malicious actor who

338
00:11:42,880 --> 00:11:44,959
used an HMAC key that was discovered in

339
00:11:44,959 --> 00:11:46,880
the intermediate layer of a Docker file.

340
00:11:46,880 --> 00:11:48,720
Um sorry docker image uh and they used

341
00:11:48,720 --> 00:11:53,040
it to compromise every single code user.

342
00:11:53,040 --> 00:11:55,680
Okay. So we uh you can imagine that that

343
00:11:55,680 --> 00:11:57,760
might be a reason that we see a bunch of

344
00:11:57,760 --> 00:11:59,200
service account keys leaked in Docker

345
00:11:59,200 --> 00:12:01,920
images. Um but there's also another

346
00:12:01,920 --> 00:12:03,920
major location uh which is actually

347
00:12:03,920 --> 00:12:05,920
GitHub which was quite surprising to us

348
00:12:05,920 --> 00:12:09,600
because uh GitHub does secret scanning.

349
00:12:09,600 --> 00:12:11,839
So we have a few theories as to why this

350
00:12:11,839 --> 00:12:15,040
could be. Uh firstly GitHub is a bit

351
00:12:15,040 --> 00:12:17,120
over represented in um this data because

352
00:12:17,120 --> 00:12:19,760
of those 6.5 billion files we scanned 3

353
00:12:19,760 --> 00:12:22,123
billion of them uh came from GitHub. So

354
00:12:22,123 --> 00:12:23,120
[snorts]

355
00:12:23,120 --> 00:12:24,560
it kind of looks a bit like people are

356
00:12:24,560 --> 00:12:25,920
leaking a bunch more surface count keys

357
00:12:25,920 --> 00:12:28,000
on GitHub, but actually the GitHub files

358
00:12:28,000 --> 00:12:29,440
are just kind of over represented in our

359
00:12:29,440 --> 00:12:32,639
data set, but we still did find uh valid

360
00:12:32,639 --> 00:12:34,320
keys on GitHub. And we suspect that this

361
00:12:34,320 --> 00:12:36,160
is because uh GitHub doesn't do

362
00:12:36,160 --> 00:12:38,160
historical source code scanning. So if

363
00:12:38,160 --> 00:12:40,320
you publish your SAK to a repository

364
00:12:40,320 --> 00:12:42,240
before you enabled uh source code

365
00:12:42,240 --> 00:12:44,079
scanning, it won't get scanned and

366
00:12:44,079 --> 00:12:46,720
picked up. Um or it could be that the

367
00:12:46,720 --> 00:12:48,639
service account keys have been reported

368
00:12:48,639 --> 00:12:50,320
and the user has just chosen not to

369
00:12:50,320 --> 00:12:53,200
revoke them.

370
00:12:53,600 --> 00:12:55,680
Looking at API keys, GitHub is also the

371
00:12:55,680 --> 00:12:58,959
main source of leaked GCP API keys um

372
00:12:58,959 --> 00:13:02,480
alongside MPM. And again, that's seems a

373
00:13:02,480 --> 00:13:04,880
bit unusual because GitHub scans MPM

374
00:13:04,880 --> 00:13:06,480
packages, new MPM packages, and also

375
00:13:06,480 --> 00:13:09,680
GitHub repositories. Um but given the

376
00:13:09,680 --> 00:13:11,760
number of valid API credentials that we

377
00:13:11,760 --> 00:13:14,560
found, we kind of suspect that um it's

378
00:13:14,560 --> 00:13:16,880
those historical keys, but also

379
00:13:16,880 --> 00:13:19,040
organizations um and people maybe

380
00:13:19,040 --> 00:13:20,880
consider leaked API keys less of a

381
00:13:20,880 --> 00:13:21,839
threat than some of the other

382
00:13:21,839 --> 00:13:23,440
credentials and so maybe they're less

383
00:13:23,440 --> 00:13:24,959
likely to act on reports of those

384
00:13:24,959 --> 00:13:27,200
credentials being leaked.

385
00:13:27,200 --> 00:13:29,200
We also see leaks occurring in Docker

386
00:13:29,200 --> 00:13:31,600
images and Pipi packages which are the

387
00:13:31,600 --> 00:13:33,279
third and fourth most common sources for

388
00:13:33,279 --> 00:13:35,839
compromised API keys.

389
00:13:35,839 --> 00:13:38,160
And then for Ruby gems tokens, the only

390
00:13:38,160 --> 00:13:39,760
place we found any valid Ruby gems

391
00:13:39,760 --> 00:13:42,639
tokens was in Docker images. So our

392
00:13:42,639 --> 00:13:44,720
pipeline did identify matches that could

393
00:13:44,720 --> 00:13:46,880
have been Ruby gems tokens across GitHub

394
00:13:46,880 --> 00:13:49,120
repositories and even in a few mpm

395
00:13:49,120 --> 00:13:51,440
packages, but all of them were either

396
00:13:51,440 --> 00:13:53,200
test credentials or inactive

397
00:13:53,200 --> 00:13:54,880
credentials. So maybe ones that have

398
00:13:54,880 --> 00:13:56,720
been um previously reported and revoked

399
00:13:56,720 --> 00:13:59,360
by GitHub secret scanning.

400
00:13:59,360 --> 00:14:03,600
Um, yeah. And, uh, if there are any Ruby

401
00:14:03,600 --> 00:14:04,959
de devs in this room, I know this is

402
00:14:04,959 --> 00:14:06,720
Python, but you have the power to make

403
00:14:06,720 --> 00:14:07,920
this chart look a lot prettier and

404
00:14:07,920 --> 00:14:09,120
introduce some new colors if [laughter]

405
00:14:09,120 --> 00:14:10,800
you want.

406
00:14:10,800 --> 00:14:13,800
Um,

407
00:14:14,720 --> 00:14:16,399
okay. So, we've looked at the common

408
00:14:16,399 --> 00:14:17,839
sources of leak credentials at the

409
00:14:17,839 --> 00:14:20,079
ecosystem level. Now, let's have a look

410
00:14:20,079 --> 00:14:21,279
at the common sources of leak

411
00:14:21,279 --> 00:14:22,959
credentials within the artifacts in

412
00:14:22,959 --> 00:14:24,639
those ecosystems.

413
00:14:24,639 --> 00:14:25,839
So this section is going to be a bit

414
00:14:25,839 --> 00:14:27,760
more anecdotal um and based on our

415
00:14:27,760 --> 00:14:29,120
experience building the pipeline and

416
00:14:29,120 --> 00:14:30,639
also the results of some early

417
00:14:30,639 --> 00:14:33,040
experiments we ran.

418
00:14:33,040 --> 00:14:34,800
Uh so these are the files in Docker

419
00:14:34,800 --> 00:14:36,560
images, GitHub repos and software

420
00:14:36,560 --> 00:14:38,079
packages that we most often see

421
00:14:38,079 --> 00:14:40,720
credential leaks occur.

422
00:14:40,720 --> 00:14:42,399
This one is a really obvious one. Um

423
00:14:42,399 --> 00:14:44,639
pretty self-explanatory. We find so many

424
00:14:44,639 --> 00:14:46,959
credentials in orth files, orth.json,

425
00:14:46,959 --> 00:14:49,120
orpi, gcp.json.

426
00:14:49,120 --> 00:14:51,279
Um, I think it's pretty clear why people

427
00:14:51,279 --> 00:14:53,680
might leak um leak credentials in these

428
00:14:53,680 --> 00:14:55,760
files.

429
00:14:55,760 --> 00:14:57,360
An interesting one we found is that

430
00:14:57,360 --> 00:14:59,440
people will leak credentials uh in

431
00:14:59,440 --> 00:15:01,920
binaries. Um, and we see this a lot in

432
00:15:01,920 --> 00:15:04,880
Docker images. So, you can't find the

433
00:15:04,880 --> 00:15:06,720
credential in any plain text file or any

434
00:15:06,720 --> 00:15:08,639
source code file in the image, but it is

435
00:15:08,639 --> 00:15:10,880
like buried inside a binary, which is

436
00:15:10,880 --> 00:15:14,000
quite quite cool. Um, and we also see

437
00:15:14,000 --> 00:15:16,880
credentials leaked in Python bite code.

438
00:15:16,880 --> 00:15:18,720
And there was a reasonably high-profile

439
00:15:18,720 --> 00:15:20,160
incident that happened, I think a year

440
00:15:20,160 --> 00:15:22,240
and a half or so ago, um, involving this

441
00:15:22,240 --> 00:15:24,079
type of link, uh, leak. And I've got a

442
00:15:24,079 --> 00:15:26,959
link there. Um, so maybe this is

443
00:15:26,959 --> 00:15:28,720
something like a developer using their

444
00:15:28,720 --> 00:15:30,720
or for testing, accidentally building

445
00:15:30,720 --> 00:15:32,880
the cred into a compiled object, then

446
00:15:32,880 --> 00:15:34,800
deleting the or from the code, but they

447
00:15:34,800 --> 00:15:36,079
don't delete the binary when they upload

448
00:15:36,079 --> 00:15:39,279
the dock image.

449
00:15:39,279 --> 00:15:41,519
Okay, a couple more obvious ones. um

450
00:15:41,519 --> 00:15:43,920
config files. You might include config

451
00:15:43,920 --> 00:15:45,519
for a bunch of different things like the

452
00:15:45,519 --> 00:15:47,519
upstream registries you interact with or

453
00:15:47,519 --> 00:15:50,079
your CI/CD system in your config files.

454
00:15:50,079 --> 00:15:51,600
Um so this is another big place we find

455
00:15:51,600 --> 00:15:53,920
leak credentials.

456
00:15:53,920 --> 00:15:56,000
We also see a bunch of credentials in um

457
00:15:56,000 --> 00:15:57,600
setup and initialization files like the

458
00:15:57,600 --> 00:15:59,839
init.py file

459
00:15:59,839 --> 00:16:02,320
and this is an mpm specific one. Um but

460
00:16:02,320 --> 00:16:03,920
we found a bunch of leak credentials in

461
00:16:03,920 --> 00:16:05,920
the dist file for packages but not in

462
00:16:05,920 --> 00:16:07,839
the source file. Uh [snorts] sorry the

463
00:16:07,839 --> 00:16:09,920
source directory. Uh I'm not a

464
00:16:09,920 --> 00:16:11,120
JavaScript developer, so I didn't

465
00:16:11,120 --> 00:16:12,320
actually know what this directory was

466
00:16:12,320 --> 00:16:14,000
until we started seeing lots of leak

467
00:16:14,000 --> 00:16:16,399
credentials in it. So I believe that

468
00:16:16,399 --> 00:16:18,320
it's common pattern for mpm developers

469
00:16:18,320 --> 00:16:20,079
to store the raw code in a source

470
00:16:20,079 --> 00:16:21,759
directory and then a minified or

471
00:16:21,759 --> 00:16:24,320
concatifi concatenated version of the

472
00:16:24,320 --> 00:16:26,480
code in the disc folder and that's what

473
00:16:26,480 --> 00:16:28,639
actually gets used in production. So

474
00:16:28,639 --> 00:16:30,160
yeah, we see examples of leak

475
00:16:30,160 --> 00:16:31,839
credentials where they appear in the

476
00:16:31,839 --> 00:16:33,440
source they appear in the disc directory

477
00:16:33,440 --> 00:16:34,560
but they don't appear in the source

478
00:16:34,560 --> 00:16:37,560
directory.

479
00:16:37,600 --> 00:16:39,600
So I guess probably the main takeaway

480
00:16:39,600 --> 00:16:41,680
from this list is that leak credentials

481
00:16:41,680 --> 00:16:44,240
can occur kind of anywhere. Um they

482
00:16:44,240 --> 00:16:45,440
occur in the really really obvious

483
00:16:45,440 --> 00:16:47,680
places. Um but they can also occur in

484
00:16:47,680 --> 00:16:49,199
less expected places like your compile

485
00:16:49,199 --> 00:16:51,199
your compiled code.

486
00:16:51,199 --> 00:16:52,720
So, as a developer or as an

487
00:16:52,720 --> 00:16:54,800
organization, what I'd want to be doing

488
00:16:54,800 --> 00:16:56,959
is number one, I'd want to know exactly

489
00:16:56,959 --> 00:16:58,880
which files are going into the artifacts

490
00:16:58,880 --> 00:17:00,639
that I ship. [snorts] And then number

491
00:17:00,639 --> 00:17:02,240
two, that I'm doing something to

492
00:17:02,240 --> 00:17:04,480
proactively prevent credentials leaking

493
00:17:04,480 --> 00:17:08,679
such as scanning all the files.

494
00:17:09,280 --> 00:17:12,160
Okay, let's have a look at uh the

495
00:17:12,160 --> 00:17:14,400
characteristics of the packages that uh

496
00:17:14,400 --> 00:17:17,039
contain leaked credentials.

497
00:17:17,039 --> 00:17:18,720
So we kind of guessed based on some of

498
00:17:18,720 --> 00:17:20,559
our early experiments that credential

499
00:17:20,559 --> 00:17:22,959
leaks don't occur that often if a

500
00:17:22,959 --> 00:17:25,600
package is more popular

501
00:17:25,600 --> 00:17:27,199
and that was something that we were able

502
00:17:27,199 --> 00:17:30,559
to confirm. So packages in MPM, Pippi

503
00:17:30,559 --> 00:17:32,640
and Maven are on average transitively

504
00:17:32,640 --> 00:17:34,799
depended on by about 50 to 100 other

505
00:17:34,799 --> 00:17:37,440
packages. But the vast majority of

506
00:17:37,440 --> 00:17:39,679
packages with valid leak credentials um

507
00:17:39,679 --> 00:17:41,280
across all ecosystems have zero

508
00:17:41,280 --> 00:17:42,960
dependence. So that's zero other

509
00:17:42,960 --> 00:17:45,520
packages depending on them. And I think

510
00:17:45,520 --> 00:17:47,360
that that's quite intuitive. Um, if

511
00:17:47,360 --> 00:17:48,880
you're a package that has hundreds of

512
00:17:48,880 --> 00:17:50,960
thousands of dependents, um, the stakes

513
00:17:50,960 --> 00:17:52,240
are a lot higher. You've probably got a

514
00:17:52,240 --> 00:17:53,440
lot more people looking at the things

515
00:17:53,440 --> 00:17:55,280
that you publish. So, it's more likely

516
00:17:55,280 --> 00:17:56,799
that you would pick something up and

517
00:17:56,799 --> 00:17:58,000
then remediate it and then it wouldn't

518
00:17:58,000 --> 00:18:00,880
show up in our scan. Um, and also maybe

519
00:18:00,880 --> 00:18:02,640
you have uh more resources to be doing

520
00:18:02,640 --> 00:18:06,360
this type of scanning yourself.

521
00:18:06,559 --> 00:18:08,400
The second thing we wanted to know about

522
00:18:08,400 --> 00:18:09,840
package versions that contain leak

523
00:18:09,840 --> 00:18:12,240
credentials was uh how many of them then

524
00:18:12,240 --> 00:18:14,000
go on to be deleted from the upstream

525
00:18:14,000 --> 00:18:15,840
registries.

526
00:18:15,840 --> 00:18:18,160
So an example of remediation action you

527
00:18:18,160 --> 00:18:19,520
might want to take when your credential

528
00:18:19,520 --> 00:18:21,919
gets leaked is just delete the artifact

529
00:18:21,919 --> 00:18:24,880
um from upstream. And interestingly this

530
00:18:24,880 --> 00:18:26,880
seems to be a common practice in Pippi

531
00:18:26,880 --> 00:18:29,280
more than any other ecosystem. Um for

532
00:18:29,280 --> 00:18:31,679
example no Maven package versions that

533
00:18:31,679 --> 00:18:33,039
contained leak credentials ended up

534
00:18:33,039 --> 00:18:36,320
getting deleted later. Um, I definitely

535
00:18:36,320 --> 00:18:37,919
think uh deleting a package upstream is

536
00:18:37,919 --> 00:18:39,760
a good thing to do. Um, but you should

537
00:18:39,760 --> 00:18:41,840
also make sure that any remediation plan

538
00:18:41,840 --> 00:18:43,360
includes actually revoking the

539
00:18:43,360 --> 00:18:45,280
credentials. Uh, because obviously

540
00:18:45,280 --> 00:18:46,720
malicious actors could still have got

541
00:18:46,720 --> 00:18:48,720
access to your credentials in the time

542
00:18:48,720 --> 00:18:53,000
that it takes you to delete the package.

543
00:18:53,520 --> 00:18:56,000
Okay, speaking of remediation plans, now

544
00:18:56,000 --> 00:18:58,080
that we've got a bit more of a an idea

545
00:18:58,080 --> 00:19:00,160
of what credential leaks look like,

546
00:19:00,160 --> 00:19:02,000
let's look at some ways that you as an

547
00:19:02,000 --> 00:19:03,600
individual or an organization can help

548
00:19:03,600 --> 00:19:07,200
to mitigate leaks when they happen.

549
00:19:07,200 --> 00:19:09,440
Firstly, when issuing new credentials,

550
00:19:09,440 --> 00:19:11,280
uh, consider the principle of lease

551
00:19:11,280 --> 00:19:13,440
privilege. Users, applications, and

552
00:19:13,440 --> 00:19:15,280
processes should have only the minimum

553
00:19:15,280 --> 00:19:19,039
access necessary to perform their tasks.

554
00:19:19,039 --> 00:19:20,559
And I think that sounds like an obvious

555
00:19:20,559 --> 00:19:23,360
thing to do. Um but uh Google Cloud

556
00:19:23,360 --> 00:19:25,120
Threat Horizon's report identified that

557
00:19:25,120 --> 00:19:28,240
67% of keys used in confirmed incidents

558
00:19:28,240 --> 00:19:30,960
of customer compromise had the basic IM

559
00:19:30,960 --> 00:19:33,280
roles owner or editor which grant a

560
00:19:33,280 --> 00:19:35,600
service account thousands of unnecessary

561
00:19:35,600 --> 00:19:37,039
permissions over a project or an

562
00:19:37,039 --> 00:19:39,280
organization.

563
00:19:39,280 --> 00:19:40,480
To prevent stuff like this from

564
00:19:40,480 --> 00:19:42,720
happening um if you're deploying on

565
00:19:42,720 --> 00:19:44,720
cloud cloud providers often provide

566
00:19:44,720 --> 00:19:46,400
tools to help you evaluate permissions

567
00:19:46,400 --> 00:19:48,400
and identify which permissions can be

568
00:19:48,400 --> 00:19:50,480
safely removed.

569
00:19:50,480 --> 00:19:52,080
Cloud providers will also allow you to

570
00:19:52,080 --> 00:19:53,679
create policies within your project so

571
00:19:53,679 --> 00:19:55,440
that new credentials will always be

572
00:19:55,440 --> 00:19:56,880
created with fewer permissions by

573
00:19:56,880 --> 00:19:59,280
default.

574
00:19:59,280 --> 00:20:01,360
Another mitigation is to expire your

575
00:20:01,360 --> 00:20:04,000
credentials. So our pipeline found

576
00:20:04,000 --> 00:20:05,840
thousands of valid credentials that were

577
00:20:05,840 --> 00:20:08,320
published years and years ago. If these

578
00:20:08,320 --> 00:20:10,880
keys had reasonable expiry times, they

579
00:20:10,880 --> 00:20:12,559
wouldn't be valid anymore, even if no

580
00:20:12,559 --> 00:20:14,400
one had taken no one had detected them

581
00:20:14,400 --> 00:20:17,440
or taken any remediating action.

582
00:20:17,440 --> 00:20:20,400
Again, um some cloud platforms allow you

583
00:20:20,400 --> 00:20:22,400
to set organizational or project level

584
00:20:22,400 --> 00:20:24,799
defaults for the expiry time of your

585
00:20:24,799 --> 00:20:27,919
service count keys. Um and if you set

586
00:20:27,919 --> 00:20:29,039
that for an organization, it gets

587
00:20:29,039 --> 00:20:30,640
applied to all projects within or an

588
00:20:30,640 --> 00:20:32,640
organization. So I would definitely

589
00:20:32,640 --> 00:20:33,760
encourage you to take advantage of

590
00:20:33,760 --> 00:20:37,200
features like that.

591
00:20:37,200 --> 00:20:39,440
When a leak occurs, um you also need to

592
00:20:39,440 --> 00:20:41,120
have a plan for managing leak credential

593
00:20:41,120 --> 00:20:44,320
reports. Um and uh so a disaster

594
00:20:44,320 --> 00:20:46,640
recovery plan would recommend the first

595
00:20:46,640 --> 00:20:48,320
step is delete the artifact from

596
00:20:48,320 --> 00:20:50,720
upstream immediately. Um again we saw

597
00:20:50,720 --> 00:20:52,559
before that a lot of artifacts with leak

598
00:20:52,559 --> 00:20:53,679
credentials just get left in the

599
00:20:53,679 --> 00:20:55,679
upstream registries. Um and that kind of

600
00:20:55,679 --> 00:20:59,200
increases your exposure. Um this can be

601
00:20:59,200 --> 00:21:01,760
especially problematic if um you're not

602
00:21:01,760 --> 00:21:03,120
able to immediately revoke these

603
00:21:03,120 --> 00:21:04,799
credentials. Say it's a really important

604
00:21:04,799 --> 00:21:06,480
key that would have a significant impact

605
00:21:06,480 --> 00:21:08,080
on your production environment and you

606
00:21:08,080 --> 00:21:09,840
can't just revoke it. Um, in that case,

607
00:21:09,840 --> 00:21:11,600
it's especially important to pull down

608
00:21:11,600 --> 00:21:14,240
the artifact as soon as possible. You

609
00:21:14,240 --> 00:21:16,400
should also then rotate the key um and

610
00:21:16,400 --> 00:21:19,200
revoke the leaked key because for any

611
00:21:19,200 --> 00:21:20,559
amount of time that the artifact has

612
00:21:20,559 --> 00:21:22,080
been public, it's possible that someone

613
00:21:22,080 --> 00:21:25,200
has gotten it. It's also good to

614
00:21:25,200 --> 00:21:27,360
leverage things like audit logging to

615
00:21:27,360 --> 00:21:32,360
see where the leak key um has been used.

616
00:21:32,480 --> 00:21:34,159
Finally, uh you should also assess

617
00:21:34,159 --> 00:21:36,080
whether a longived uh credential like a

618
00:21:36,080 --> 00:21:37,679
service account key is even needed at

619
00:21:37,679 --> 00:21:39,440
all. There are a ton of different

620
00:21:39,440 --> 00:21:41,280
scenarios where you just don't require

621
00:21:41,280 --> 00:21:43,280
types like these types of credentials to

622
00:21:43,280 --> 00:21:46,320
get created um and especially exported

623
00:21:46,320 --> 00:21:48,000
and it's worth considering other options

624
00:21:48,000 --> 00:21:50,159
like service account impersonation or

625
00:21:50,159 --> 00:21:52,240
identity federation.

626
00:21:52,240 --> 00:21:54,480
Package registries um a bunch of them

627
00:21:54,480 --> 00:21:56,159
have started offering trusted publishing

628
00:21:56,159 --> 00:21:58,640
workflows. So instead of managing an API

629
00:21:58,640 --> 00:22:00,720
key to authenticate with a registry, a

630
00:22:00,720 --> 00:22:02,320
developer can configure the registry to

631
00:22:02,320 --> 00:22:04,400
trust a CI workflow that both builds and

632
00:22:04,400 --> 00:22:07,039
publishes an artifact. And CI/CD

633
00:22:07,039 --> 00:22:09,440
platforms also often provide identities

634
00:22:09,440 --> 00:22:11,440
for their CI workflows to authenticate

635
00:22:11,440 --> 00:22:13,440
with uh cloud providers without having

636
00:22:13,440 --> 00:22:15,840
to manage credentials.

637
00:22:15,840 --> 00:22:17,840
So we would encourage uh you to leverage

638
00:22:17,840 --> 00:22:21,120
those features as much as possible.

639
00:22:21,120 --> 00:22:22,640
But even better than mitigating

640
00:22:22,640 --> 00:22:24,480
credential leaks is just prevent the

641
00:22:24,480 --> 00:22:27,919
leaks from occurring in the first place.

642
00:22:27,919 --> 00:22:29,200
If you're an individual or an

643
00:22:29,200 --> 00:22:30,880
organization that publishes software,

644
00:22:30,880 --> 00:22:33,280
especially open source software, your

645
00:22:33,280 --> 00:22:34,640
workflow probably looks something like

646
00:22:34,640 --> 00:22:36,960
this. You're a developer. You write an

647
00:22:36,960 --> 00:22:38,480
awesome piece of code or you write a

648
00:22:38,480 --> 00:22:41,120
cool new software package. You push it

649
00:22:41,120 --> 00:22:42,880
to a public artifact registry like

650
00:22:42,880 --> 00:22:45,200
GitHub or Pippi.

651
00:22:45,200 --> 00:22:46,880
It then becomes available to everyone on

652
00:22:46,880 --> 00:22:49,919
the internet. That includes good actors

653
00:22:49,919 --> 00:22:51,919
like my team, the ghost team, who are

654
00:22:51,919 --> 00:22:53,280
trying to raise the alarm in case you've

655
00:22:53,280 --> 00:22:55,679
accidentally leaked your credentials.

656
00:22:55,679 --> 00:22:57,679
But it also includes malicious actors

657
00:22:57,679 --> 00:22:58,799
who are looking to abuse those

658
00:22:58,799 --> 00:23:00,880
credentials. And as we saw at the

659
00:23:00,880 --> 00:23:02,159
beginning, those malicious actors can

660
00:23:02,159 --> 00:23:03,679
work very fast within seconds or

661
00:23:03,679 --> 00:23:04,685
minutes.

662
00:23:04,685 --> 00:23:04,720
[snorts]

663
00:23:04,720 --> 00:23:06,400
So, it's really a race against time as

664
00:23:06,400 --> 00:23:07,760
to who is going to find the credentials

665
00:23:07,760 --> 00:23:09,760
first.

666
00:23:09,760 --> 00:23:11,679
The only surefire way to prevent leak

667
00:23:11,679 --> 00:23:13,600
credentials um from being abused is to

668
00:23:13,600 --> 00:23:14,960
prevent them from getting leaked in the

669
00:23:14,960 --> 00:23:17,280
first place. If you're a developer, you

670
00:23:17,280 --> 00:23:18,480
should make sure that you're scanning

671
00:23:18,480 --> 00:23:21,200
things before you publish. Um and if

672
00:23:21,200 --> 00:23:22,640
you're an organization, you should look

673
00:23:22,640 --> 00:23:24,880
into incorporating automatic pre-publish

674
00:23:24,880 --> 00:23:28,720
scanning into your developer workflows.

675
00:23:28,720 --> 00:23:30,159
There are so many ways you can do this.

676
00:23:30,159 --> 00:23:32,159
Um, you can introduce automation into

677
00:23:32,159 --> 00:23:33,760
your build system such as pre-commit

678
00:23:33,760 --> 00:23:35,280
checks to prevent keys from ever being

679
00:23:35,280 --> 00:23:37,600
checked in. GCP cloud build, for

680
00:23:37,600 --> 00:23:39,679
example, um, has build triggers and

681
00:23:39,679 --> 00:23:41,039
there are a bunch of open source

682
00:23:41,039 --> 00:23:42,880
projects such as git secrets which use

683
00:23:42,880 --> 00:23:45,360
git hooks. If you're publishing source

684
00:23:45,360 --> 00:23:47,120
code to repositories like GitHub or

685
00:23:47,120 --> 00:23:49,200
GitLab, uh, make sure you enable the

686
00:23:49,200 --> 00:23:52,720
secret scanning that they offer.

687
00:23:52,720 --> 00:23:54,240
Then there are also open source tools

688
00:23:54,240 --> 00:23:56,320
like truffle hog which you can use to do

689
00:23:56,320 --> 00:23:58,880
the scanning of your artifacts and um

690
00:23:58,880 --> 00:24:00,480
the OSV scanner project which was

691
00:24:00,480 --> 00:24:02,400
created by the ghost team also has

692
00:24:02,400 --> 00:24:06,200
support for secret scanning.

693
00:24:06,559 --> 00:24:08,880
Um another thing we'd love to see is for

694
00:24:08,880 --> 00:24:10,720
artifact registries themselves to enable

695
00:24:10,720 --> 00:24:12,799
pre-published scanning. So this is an

696
00:24:12,799 --> 00:24:14,480
added layer of protection where every

697
00:24:14,480 --> 00:24:16,480
artifact that gets uploaded to a a

698
00:24:16,480 --> 00:24:18,720
registry can be scanned for credentials

699
00:24:18,720 --> 00:24:20,400
before being published um to the

700
00:24:20,400 --> 00:24:22,799
internet. And I think that uh this is

701
00:24:22,799 --> 00:24:24,320
definitely something we'd love to see

702
00:24:24,320 --> 00:24:26,559
our artifact registries start doing and

703
00:24:26,559 --> 00:24:27,840
I think a few of them are looking into

704
00:24:27,840 --> 00:24:30,799
it.

705
00:24:30,799 --> 00:24:32,720
Okay, so with that said, let's have a

706
00:24:32,720 --> 00:24:34,720
look at what is next for our credential

707
00:24:34,720 --> 00:24:38,000
scanning pipeline that we built.

708
00:24:38,000 --> 00:24:39,919
So obviously we're super excited about

709
00:24:39,919 --> 00:24:42,799
all the credentials that we found. Um

710
00:24:42,799 --> 00:24:44,559
and I think something that's even more

711
00:24:44,559 --> 00:24:46,159
exciting is demonstrating the potential

712
00:24:46,159 --> 00:24:48,400
of a pipeline like this for scanning a

713
00:24:48,400 --> 00:24:50,799
corpus of open source software. Um, and

714
00:24:50,799 --> 00:24:52,480
we'd love to expand the scope of that

715
00:24:52,480 --> 00:24:54,320
scanning pipeline to scan for other

716
00:24:54,320 --> 00:24:57,840
things like malware.

717
00:24:57,840 --> 00:24:59,919
Um, and then finally, we also really

718
00:24:59,919 --> 00:25:01,440
want to expand the types of credentials

719
00:25:01,440 --> 00:25:02,720
that we detect and report from the

720
00:25:02,720 --> 00:25:04,400
pipeline. So those 20,000 unique

721
00:25:04,400 --> 00:25:06,000
credentials should absolutely be

722
00:25:06,000 --> 00:25:07,520
considered a lower bound to the number

723
00:25:07,520 --> 00:25:09,360
of credentials that are out there. Um,

724
00:25:09,360 --> 00:25:11,360
we're scanning for, you know, four or

725
00:25:11,360 --> 00:25:12,880
five different types of credentials out

726
00:25:12,880 --> 00:25:16,480
of thousands of different types. Um, we

727
00:25:16,480 --> 00:25:18,320
started with GCP. Um we had a great

728
00:25:18,320 --> 00:25:20,320
experience working with Pi and Ruby Gems

729
00:25:20,320 --> 00:25:22,880
um and Pyex and we're very keen to on

730
00:25:22,880 --> 00:25:24,480
board as many credential providers as

731
00:25:24,480 --> 00:25:26,559
possible um to help make open source

732
00:25:26,559 --> 00:25:28,320
more secure. So if you are a credential

733
00:25:28,320 --> 00:25:29,600
provider or you know a credential

734
00:25:29,600 --> 00:25:31,360
provider please put them in contact with

735
00:25:31,360 --> 00:25:34,799
us um this is how you can contact us.

736
00:25:34,799 --> 00:25:37,279
Also if you just want to contact us you

737
00:25:37,279 --> 00:25:39,840
can do it via that email.

738
00:25:39,840 --> 00:25:42,000
Okay thank you so much. Um happy to take

739
00:25:42,000 --> 00:25:45,217
any questions. [applause]

740
00:25:47,782 --> 00:25:49,802
[applause]

741
00:25:55,039 --> 00:25:57,919
Thanks for the talk. I'm wondering if

742
00:25:57,919 --> 00:26:00,480
you've noticed since developers are

743
00:26:00,480 --> 00:26:04,080
using more generative AI tools that the

744
00:26:04,080 --> 00:26:06,880
rate of leaked credentials has

745
00:26:06,880 --> 00:26:09,360
increased. And if so, can you talk about

746
00:26:09,360 --> 00:26:11,039
that a little bit? Thanks.

747
00:26:11,039 --> 00:26:13,760
Yes. I um I didn't manage to get it like

748
00:26:13,760 --> 00:26:15,279
in a nice enough format to put in the

749
00:26:15,279 --> 00:26:17,120
slides, but we definitely have seen the

750
00:26:17,120 --> 00:26:19,520
rate of leak credentials increasing. Um

751
00:26:19,520 --> 00:26:20,880
and in particular, I think we've seen

752
00:26:20,880 --> 00:26:23,200
the rates of Pippi

753
00:26:23,200 --> 00:26:25,360
uh tokens being leaked that are valid

754
00:26:25,360 --> 00:26:27,919
increasing um over the I think we've

755
00:26:27,919 --> 00:26:29,360
been running the pipeline for like 2 and

756
00:26:29,360 --> 00:26:32,720
1/2 years and it's going up. So yes,

757
00:26:32,720 --> 00:26:34,400
definitely. Um and I think that's a bit

758
00:26:34,400 --> 00:26:36,720
of a worry. Um, which is why I think

759
00:26:36,720 --> 00:26:38,000
it's important to have like automated

760
00:26:38,000 --> 00:26:39,600
solutions such as, you know, artifact

761
00:26:39,600 --> 00:26:41,279
registries doing scanning, developers

762
00:26:41,279 --> 00:26:44,640
and organizations having scanning.

763
00:26:44,640 --> 00:26:47,600
Eve, you talked about uh onboarding

764
00:26:47,600 --> 00:26:49,600
credential providers. What about

765
00:26:49,600 --> 00:26:52,080
onboarding more uh artifact

766
00:26:52,080 --> 00:26:54,799
repositories? Should they contact in the

767
00:26:54,799 --> 00:26:56,159
same way or is the process different?

768
00:26:56,159 --> 00:26:58,559
Yes. Yes, please. Uh, artifact providers

769
00:26:58,559 --> 00:27:00,400
contact us.

770
00:27:00,400 --> 00:27:03,360
Yep.

771
00:27:03,360 --> 00:27:07,039
Um, with the deleted packages, does your

772
00:27:07,039 --> 00:27:09,279
data get granular enough to know if

773
00:27:09,279 --> 00:27:10,240
they've been replaced with

774
00:27:10,240 --> 00:27:13,240
post-releases?

775
00:27:13,360 --> 00:27:15,200
I think it probably is, but I haven't

776
00:27:15,200 --> 00:27:16,159
personally looked at it, but that's a

777
00:27:16,159 --> 00:27:17,360
great idea. I'd be very interested to

778
00:27:17,360 --> 00:27:18,000
have a look at that.

779
00:27:18,000 --> 00:27:20,159
Yes. Okay, cool.

780
00:27:20,159 --> 00:27:23,200
Um, you talked about a few time twice,

781
00:27:23,200 --> 00:27:26,880
um, deleting the upstream artifacts, and

782
00:27:26,880 --> 00:27:28,320
I wanted to know, I guess, why you

783
00:27:28,320 --> 00:27:29,919
thought that was important when you can

784
00:27:29,919 --> 00:27:31,919
also revoke the key. Like I know of some

785
00:27:31,919 --> 00:27:34,480
ecosystems where you can't delete

786
00:27:34,480 --> 00:27:36,480
artifacts once they're upstream.

787
00:27:36,480 --> 00:27:38,080
Yeah. No, that's that's a great point.

788
00:27:38,080 --> 00:27:40,080
So I think if you have the possibility

789
00:27:40,080 --> 00:27:41,840
to delete the artifact, that's probably

790
00:27:41,840 --> 00:27:43,520
most important in a scenario where you

791
00:27:43,520 --> 00:27:45,279
can't easily revoke the key or rotate

792
00:27:45,279 --> 00:27:47,200
them. So sometimes we'll see cases where

793
00:27:47,200 --> 00:27:48,640
there's a single service account key

794
00:27:48,640 --> 00:27:49,760
being used for like every single

795
00:27:49,760 --> 00:27:51,120
production service. And if you were to

796
00:27:51,120 --> 00:27:52,880
revoke that, your whole infrastructure

797
00:27:52,880 --> 00:27:54,559
would fail. So I think in that

798
00:27:54,559 --> 00:27:56,399
situation, delete the artifact if

799
00:27:56,399 --> 00:27:58,320
possible just to like prevent more

800
00:27:58,320 --> 00:28:00,480
people from getting the artifact. But um

801
00:28:00,480 --> 00:28:02,080
and then you you give yourself a bit

802
00:28:02,080 --> 00:28:03,760
more room to try and like revoke the

803
00:28:03,760 --> 00:28:05,600
key. But yeah, absolutely. Revoking it

804
00:28:05,600 --> 00:28:06,960
is really the only way to actually

805
00:28:06,960 --> 00:28:08,720
resolve the problem because for any

806
00:28:08,720 --> 00:28:10,399
amount of time if it's been public, it's

807
00:28:10,399 --> 00:28:12,080
possible that someone has accessed it.

808
00:28:12,080 --> 00:28:14,159
Yeah, that's a great point.

809
00:28:14,159 --> 00:28:16,880
You've got uh Pi and Docker and GitHub.

810
00:28:16,880 --> 00:28:18,320
Do you have other things not on a

811
00:28:18,320 --> 00:28:21,120
diagram like crates for Rust or Laravel

812
00:28:21,120 --> 00:28:22,720
or Yep, it's all covered?

813
00:28:22,720 --> 00:28:23,600
Yep, it is. Yeah,

814
00:28:23,600 --> 00:28:24,080
good.

815
00:28:24,080 --> 00:28:26,960
Um I think I have not said them by name,

816
00:28:26,960 --> 00:28:28,880
but a bunch of them are represented in

817
00:28:28,880 --> 00:28:30,480
the logo. So if you can recognize the

818
00:28:30,480 --> 00:28:32,240
logos. Yeah, we but we do have crates.

819
00:28:32,240 --> 00:28:37,039
Um yeah, we have MPM, Pippi, Maven, um

820
00:28:37,039 --> 00:28:38,640
Go.

821
00:28:38,640 --> 00:28:42,919
Yeah, lots of them.

822
00:28:43,279 --> 00:28:45,440
I notic you were doing some uh binary

823
00:28:45,440 --> 00:28:48,000
tracking, which was great uh to see. uh

824
00:28:48,000 --> 00:28:49,919
did have you managed to talk to people

825
00:28:49,919 --> 00:28:53,200
that have uh

826
00:28:53,200 --> 00:28:55,440
credentials in binaries and like

827
00:28:55,440 --> 00:28:56,799
understand whether they believe that

828
00:28:56,799 --> 00:28:58,320
that protects them that it's just

829
00:28:58,320 --> 00:29:01,039
because it's compiled clearly uh which

830
00:29:01,039 --> 00:29:02,880
you know obviously not [laughter] uh

831
00:29:02,880 --> 00:29:05,360
strings is great uh but yeah I'm curious

832
00:29:05,360 --> 00:29:06,399
as to whether you've had those

833
00:29:06,399 --> 00:29:09,039
conversations and whether uh whether you

834
00:29:09,039 --> 00:29:10,159
got the feeling from people that they

835
00:29:10,159 --> 00:29:11,679
thought just because it was compiled

836
00:29:11,679 --> 00:29:12,880
that they were safe.

837
00:29:12,880 --> 00:29:14,240
Yeah. Um that's a great question.

838
00:29:14,240 --> 00:29:16,000
Unfortunately, we have not we don't

839
00:29:16,000 --> 00:29:17,520
really interact with any of the people.

840
00:29:17,520 --> 00:29:19,600
We leave that to the providers. So, I

841
00:29:19,600 --> 00:29:21,600
imagine hopefully someone in for

842
00:29:21,600 --> 00:29:23,120
instance Google Cloud, I know they reach

843
00:29:23,120 --> 00:29:24,320
out to users who've had compromised

844
00:29:24,320 --> 00:29:26,080
credentials. And I think that would be

845
00:29:26,080 --> 00:29:27,440
good to communicate to people that like

846
00:29:27,440 --> 00:29:29,200
this was in a binary that you uploaded

847
00:29:29,200 --> 00:29:30,960
but not in any of your um plain text

848
00:29:30,960 --> 00:29:32,720
files. So, um yeah, I think it is

849
00:29:32,720 --> 00:29:34,640
important to like let people know that

850
00:29:34,640 --> 00:29:36,240
that is a common place for compromise,

851
00:29:36,240 --> 00:29:38,080
but we don't actually interact with any

852
00:29:38,080 --> 00:29:40,159
of the people unfortunately. All

853
00:29:40,159 --> 00:29:41,360
right. And I think that's all we have

854
00:29:41,360 --> 00:29:43,200
time for. So, uh thank you very much,

855
00:29:43,200 --> 00:29:44,895
Eve.

856
00:29:44,895 --> 00:29:46,915
[applause]