1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:15,759 --> 00:00:19,520
hello everyone and welcome back to the

3
00:00:17,440 --> 00:00:22,320
kiara theater at linux conference

4
00:00:19,520 --> 00:00:24,560
australia 2022 uh next up in this slot

5
00:00:22,320 --> 00:00:27,279
we've got matthew oh i have matthew

6
00:00:24,560 --> 00:00:29,439
oliver uh who is a senior systems and

7
00:00:27,279 --> 00:00:30,960
software engineer working at nvidia

8
00:00:29,439 --> 00:00:33,040
where he primarily works in upstream

9
00:00:30,960 --> 00:00:35,040
openstack as a swift core

10
00:00:33,040 --> 00:00:36,960
today we're talking through the benefits

11
00:00:35,040 --> 00:00:39,360
and challenges of integrating open

12
00:00:36,960 --> 00:00:40,719
tracing into openstack swift now we're

13
00:00:39,360 --> 00:00:42,239
going to try and have time for questions

14
00:00:40,719 --> 00:00:44,399
but if we run out of time there will be

15
00:00:42,239 --> 00:00:46,640
follow on question time in the post chat

16
00:00:44,399 --> 00:00:48,719
post talk chat kia ora theater

17
00:00:46,640 --> 00:00:51,280
accessible by the

18
00:00:48,719 --> 00:00:53,280
left sidebar in venulis it's all yours

19
00:00:51,280 --> 00:00:55,039
matthew take it away

20
00:00:53,280 --> 00:00:57,440
thanks thanks for the introduction and

21
00:00:55,039 --> 00:00:58,719
yes i say we might need to there might

22
00:00:57,440 --> 00:01:01,120
be

23
00:00:58,719 --> 00:01:02,399
uh i tend to waffle on a bit uh golf and

24
00:01:01,120 --> 00:01:04,400
tangent for those who know me and i'm

25
00:01:02,399 --> 00:01:06,000
going to try and keep it crisp and and

26
00:01:04,400 --> 00:01:08,560
quick so

27
00:01:06,000 --> 00:01:10,159
this this talk all started uh because i

28
00:01:08,560 --> 00:01:11,280
was trying to

29
00:01:10,159 --> 00:01:13,680
um

30
00:01:11,280 --> 00:01:16,320
benchmark swift on some new i've some

31
00:01:13,680 --> 00:01:18,640
new hardware at uh at nvidia

32
00:01:16,320 --> 00:01:20,560
and the way swift is made it's all

33
00:01:18,640 --> 00:01:22,240
modular and i wanted to be able to start

34
00:01:20,560 --> 00:01:23,360
tracing and so i want to start timing

35
00:01:22,240 --> 00:01:26,159
between these middlewares we'll get all

36
00:01:23,360 --> 00:01:27,600
this in a sec but i then showed our sre

37
00:01:26,159 --> 00:01:29,360
guys i loved it and it's kind of

38
00:01:27,600 --> 00:01:32,000
snowballed into this whole

39
00:01:29,360 --> 00:01:33,520
uh tracing thing and it's upstream so we

40
00:01:32,000 --> 00:01:34,880
started working on upstream

41
00:01:33,520 --> 00:01:36,560
now i'm trying to figure out why am i

42
00:01:34,880 --> 00:01:38,240
not going to the next slide and there we

43
00:01:36,560 --> 00:01:41,119
are so like i said my name is matthew

44
00:01:38,240 --> 00:01:43,280
oliver i work at nvidia um on all the

45
00:01:41,119 --> 00:01:46,560
socials you'll see me i'm usually up as

46
00:01:43,280 --> 00:01:49,680
matt oliver you um i've been involved in

47
00:01:46,560 --> 00:01:51,600
swift uh since 2014 and i've been to

48
00:01:49,680 --> 00:01:54,720
swift court since 2015.

49
00:01:51,600 --> 00:01:55,759
so i've been doing it for a while

50
00:01:54,720 --> 00:01:58,159
uh

51
00:01:55,759 --> 00:02:00,079
i am going to tempt the demogors today

52
00:01:58,159 --> 00:02:02,000
and most might well say most like half

53
00:02:00,079 --> 00:02:06,079
my talk is going to be

54
00:02:02,000 --> 00:02:07,680
a live demo um or locally here but

55
00:02:06,079 --> 00:02:09,759
it's going to be interesting

56
00:02:07,680 --> 00:02:10,720
but to get before you guys can start

57
00:02:09,759 --> 00:02:12,560
looking

58
00:02:10,720 --> 00:02:14,080
at the fun traces

59
00:02:12,560 --> 00:02:15,280
you kind of i want you guys to

60
00:02:14,080 --> 00:02:17,680
understand what you're looking at

61
00:02:15,280 --> 00:02:19,360
otherwise it's not as exciting uh well

62
00:02:17,680 --> 00:02:20,319
it won't be as exciting as this for me

63
00:02:19,360 --> 00:02:21,920
so

64
00:02:20,319 --> 00:02:23,760
i'm going to try and bootstrap the

65
00:02:21,920 --> 00:02:25,760
audience into understanding so we're

66
00:02:23,760 --> 00:02:28,080
going to go through this really quickly

67
00:02:25,760 --> 00:02:28,959
because i want to get to the fun stuff

68
00:02:28,080 --> 00:02:31,040
so

69
00:02:28,959 --> 00:02:32,879
let's start with

70
00:02:31,040 --> 00:02:35,200
what is swift it's a fully open source

71
00:02:32,879 --> 00:02:36,800
end to end object storage solution

72
00:02:35,200 --> 00:02:37,840
it can be standalone

73
00:02:36,800 --> 00:02:40,239
and integrated into your environment

74
00:02:37,840 --> 00:02:43,920
this is what we do at nvidia or a part

75
00:02:40,239 --> 00:02:45,440
of the rest of the openstack ecosystem

76
00:02:43,920 --> 00:02:47,760
it's highly extendable it's mostly

77
00:02:45,440 --> 00:02:49,440
written in python and that's not

78
00:02:47,760 --> 00:02:50,720
it is written in python so it's very

79
00:02:49,440 --> 00:02:51,920
extendable but that's not the only

80
00:02:50,720 --> 00:02:53,840
reason why it's extended we'll get to

81
00:02:51,920 --> 00:02:55,840
more on that a little bit later as we

82
00:02:53,840 --> 00:02:57,120
look how traces work

83
00:02:55,840 --> 00:02:59,200
um

84
00:02:57,120 --> 00:03:00,959
basically it's done with whiskey so you

85
00:02:59,200 --> 00:03:02,560
can insert me to wherever you want where

86
00:03:00,959 --> 00:03:04,879
it's not written in python stuff like

87
00:03:02,560 --> 00:03:07,280
erasure coding it's done in c

88
00:03:04,879 --> 00:03:08,959
um it supports a swift api you'd hope it

89
00:03:07,280 --> 00:03:10,959
does because it's swift

90
00:03:08,959 --> 00:03:12,720
and also the s3 api

91
00:03:10,959 --> 00:03:14,159
you can do replicated or ratio coded

92
00:03:12,720 --> 00:03:16,640
policies

93
00:03:14,159 --> 00:03:18,159
it's great at scale um

94
00:03:16,640 --> 00:03:20,319
the bigger actually turns out bigger the

95
00:03:18,159 --> 00:03:22,959
better we're we're massively scaling it

96
00:03:20,319 --> 00:03:25,599
at nvidia uh in fact we've hired a bunch

97
00:03:22,959 --> 00:03:27,840
of the swift cores to make like these

98
00:03:25,599 --> 00:03:29,519
bottleneck huge scaling issues kind of

99
00:03:27,840 --> 00:03:30,799
better go away get better and better so

100
00:03:29,519 --> 00:03:32,799
they're actually a really fun time and a

101
00:03:30,799 --> 00:03:35,040
really fun community and fun problems

102
00:03:32,799 --> 00:03:36,799
we're solving

103
00:03:35,040 --> 00:03:38,400
the community upstream community is

104
00:03:36,799 --> 00:03:40,799
actually really awesome like there's a

105
00:03:38,400 --> 00:03:42,640
reason i started in 2014 and

106
00:03:40,799 --> 00:03:44,159
i've gone through

107
00:03:42,640 --> 00:03:46,879
different roles in different different

108
00:03:44,159 --> 00:03:48,560
places and stayed in the community

109
00:03:46,879 --> 00:03:51,200
because they're awesome so if you're a

110
00:03:48,560 --> 00:03:52,560
dev an operator you ride docks you're

111
00:03:51,200 --> 00:03:54,400
kind of just interested in the space you

112
00:03:52,560 --> 00:03:55,840
just want to hang it's a really it's a

113
00:03:54,400 --> 00:03:57,599
really awesome community anyway let's

114
00:03:55,840 --> 00:03:59,120
keep moving because otherwise i'll run

115
00:03:57,599 --> 00:04:01,200
out of time we don't get much time for

116
00:03:59,120 --> 00:04:02,959
questions at the end so

117
00:04:01,200 --> 00:04:04,720
let's have a brief look at the swift

118
00:04:02,959 --> 00:04:06,000
topology we need to know this because

119
00:04:04,720 --> 00:04:07,760
when you start looking at traces and

120
00:04:06,000 --> 00:04:08,640
we're jumping around through the cluster

121
00:04:07,760 --> 00:04:10,400
you kind of want to know what you're

122
00:04:08,640 --> 00:04:12,799
looking at so

123
00:04:10,400 --> 00:04:15,760
we'll talk on about the very basic paco

124
00:04:12,799 --> 00:04:17,600
installs these are what you tend to see

125
00:04:15,760 --> 00:04:19,120
in really small deployments

126
00:04:17,600 --> 00:04:22,079
where every node

127
00:04:19,120 --> 00:04:23,600
is stored has all the swift services the

128
00:04:22,079 --> 00:04:26,720
swift proxy server account service

129
00:04:23,600 --> 00:04:26,720
tennis server and object server

130
00:04:26,800 --> 00:04:30,080
but then as you stand to scale up people

131
00:04:28,800 --> 00:04:32,160
tend to start doing this they start

132
00:04:30,080 --> 00:04:34,000
splitting out the proxy from the storage

133
00:04:32,160 --> 00:04:36,320
system and this makes perfect sense the

134
00:04:34,000 --> 00:04:37,919
swift proxy is stateless

135
00:04:36,320 --> 00:04:39,440
um and so it's perfect for

136
00:04:37,919 --> 00:04:40,639
containerization

137
00:04:39,440 --> 00:04:43,280
and

138
00:04:40,639 --> 00:04:44,960
uh if you put more of them in it if you

139
00:04:43,280 --> 00:04:46,800
want more throughput and more and handle

140
00:04:44,960 --> 00:04:48,000
more connections you just scale them and

141
00:04:46,800 --> 00:04:48,960
if you want more storage you scale the

142
00:04:48,000 --> 00:04:51,840
other side that's it's pretty

143
00:04:48,960 --> 00:04:54,160
straightforward right the swift proxy uh

144
00:04:51,840 --> 00:04:56,880
is also if you're doing things like at

145
00:04:54,160 --> 00:04:58,880
rest uh encryption the swift proxy is

146
00:04:56,880 --> 00:04:59,680
the one that's going to be talking to

147
00:04:58,880 --> 00:05:01,039
uh

148
00:04:59,680 --> 00:05:02,800
is the one that's talking into your

149
00:05:01,039 --> 00:05:04,800
getting the encryption keys and stuff so

150
00:05:02,800 --> 00:05:07,039
it's also nice to have that separated

151
00:05:04,800 --> 00:05:09,280
and then you just get a encrypted stream

152
00:05:07,039 --> 00:05:11,919
hidden to the storage notes anyway

153
00:05:09,280 --> 00:05:13,360
i just told you tangence is what i do so

154
00:05:11,919 --> 00:05:15,360
i should have got someone to like ping

155
00:05:13,360 --> 00:05:17,360
me or hit me when i start going off on a

156
00:05:15,360 --> 00:05:19,360
tangent too much

157
00:05:17,360 --> 00:05:21,440
okay so at greater scale this is what we

158
00:05:19,360 --> 00:05:23,440
tend to see right so when we get even

159
00:05:21,440 --> 00:05:25,039
higher

160
00:05:23,440 --> 00:05:26,560
it's an object uh swift is an object

161
00:05:25,039 --> 00:05:30,080
storage system so

162
00:05:26,560 --> 00:05:31,120
99 of its storage is objects and so

163
00:05:30,080 --> 00:05:32,160
having

164
00:05:31,120 --> 00:05:33,919
uh

165
00:05:32,160 --> 00:05:35,680
just being able to separate that those

166
00:05:33,919 --> 00:05:37,520
object nodes away from the rest of the

167
00:05:35,680 --> 00:05:38,880
cluster i mean

168
00:05:37,520 --> 00:05:41,840
you know what i mean

169
00:05:38,880 --> 00:05:44,400
uh isolating those makes sense

170
00:05:41,840 --> 00:05:46,479
the account a container side of it is

171
00:05:44,400 --> 00:05:48,720
our metadata layer in swift um it's

172
00:05:46,479 --> 00:05:51,360
keeping track of you know acls and other

173
00:05:48,720 --> 00:05:53,120
metadata and other stats like an account

174
00:05:51,360 --> 00:05:55,680
will have a list of

175
00:05:53,120 --> 00:05:57,600
a list of containers and the containers

176
00:05:55,680 --> 00:05:59,199
will have a list of objects

177
00:05:57,600 --> 00:06:01,280
and we'll have all the acls and

178
00:05:59,199 --> 00:06:03,440
metadatas and anything else and other

179
00:06:01,280 --> 00:06:05,120
aggregating stats up the stack so you

180
00:06:03,440 --> 00:06:08,479
can think of the

181
00:06:05,120 --> 00:06:10,479
ac layer we see in this diagram as

182
00:06:08,479 --> 00:06:11,759
the metadata layer so makes sense to

183
00:06:10,479 --> 00:06:13,280
this nice approach this is what we do at

184
00:06:11,759 --> 00:06:15,120
nvidia

185
00:06:13,280 --> 00:06:16,960
there's also when we scale swift also

186
00:06:15,120 --> 00:06:18,080
knows about other availability zone

187
00:06:16,960 --> 00:06:19,680
issues too but we're not going to go

188
00:06:18,080 --> 00:06:21,360
into any of that stuff we don't have

189
00:06:19,680 --> 00:06:22,880
time so

190
00:06:21,360 --> 00:06:26,000
here's a quick look

191
00:06:22,880 --> 00:06:28,319
at what happens uh when swift

192
00:06:26,000 --> 00:06:30,080
uh we got a basic request so i think my

193
00:06:28,319 --> 00:06:31,759
mouse works i'm assuming people can see

194
00:06:30,080 --> 00:06:32,840
my mouse do stuff

195
00:06:31,759 --> 00:06:36,400
but maybe you

196
00:06:32,840 --> 00:06:38,560
can't i don't know anyway uh i'll assume

197
00:06:36,400 --> 00:06:40,639
you do so

198
00:06:38,560 --> 00:06:42,400
the request will come in and hit the

199
00:06:40,639 --> 00:06:45,120
swift proxy and the proxy this is

200
00:06:42,400 --> 00:06:48,080
obviously going to be uh with an object

201
00:06:45,120 --> 00:06:50,240
put on probably a three times replicated

202
00:06:48,080 --> 00:06:51,919
system

203
00:06:50,240 --> 00:06:52,880
um so what will happen is a request

204
00:06:51,919 --> 00:06:54,160
comes in

205
00:06:52,880 --> 00:06:57,280
the

206
00:06:54,160 --> 00:06:59,520
swift proxy will look up uh on a

207
00:06:57,280 --> 00:07:02,639
modified consistent hash ring

208
00:06:59,520 --> 00:07:04,720
for that policy of where which storage

209
00:07:02,639 --> 00:07:05,840
nodes are responsible for storing that

210
00:07:04,720 --> 00:07:07,039
object and we'll make three connections

211
00:07:05,840 --> 00:07:08,080
oh no we're going to see this in this

212
00:07:07,039 --> 00:07:09,840
trace this is the only worries when i

213
00:07:08,080 --> 00:07:11,919
bring it up right and here is a snapshot

214
00:07:09,840 --> 00:07:14,000
of trace this is not live obviously

215
00:07:11,919 --> 00:07:15,199
this is a snapshot of our trace just to

216
00:07:14,000 --> 00:07:18,240
give you an idea of what you've learned

217
00:07:15,199 --> 00:07:19,360
so far so the blue here is the proxy

218
00:07:18,240 --> 00:07:22,319
server

219
00:07:19,360 --> 00:07:25,120
and it's making an object put

220
00:07:22,319 --> 00:07:27,840
and so it's making three connections to

221
00:07:25,120 --> 00:07:30,639
one object server two object servers and

222
00:07:27,840 --> 00:07:31,599
three object servers

223
00:07:30,639 --> 00:07:33,840
now

224
00:07:31,599 --> 00:07:35,120
we can you can also follow along that

225
00:07:33,840 --> 00:07:36,479
over here you can set over here is a

226
00:07:35,120 --> 00:07:38,800
container server

227
00:07:36,479 --> 00:07:41,120
and so the object is being put into the

228
00:07:38,800 --> 00:07:42,639
object server and then it's up going to

229
00:07:41,120 --> 00:07:44,319
go update the metadata layer and so we

230
00:07:42,639 --> 00:07:45,840
can see this in our in a glance which is

231
00:07:44,319 --> 00:07:48,160
kind of cool and we're getting our

232
00:07:45,840 --> 00:07:49,759
knowledge

233
00:07:48,160 --> 00:07:51,599
now we'll quickly talk about open

234
00:07:49,759 --> 00:07:55,120
tracing

235
00:07:51,599 --> 00:07:56,240
this i know this slide goes against all

236
00:07:55,120 --> 00:07:57,360
um

237
00:07:56,240 --> 00:07:59,520
what you should and what you shouldn't

238
00:07:57,360 --> 00:08:01,120
do with with slide presentation because

239
00:07:59,520 --> 00:08:02,240
there's way too much information on this

240
00:08:01,120 --> 00:08:03,599
slide

241
00:08:02,240 --> 00:08:05,840
but i put it here in case you're

242
00:08:03,599 --> 00:08:07,440
interested and you get these um you get

243
00:08:05,840 --> 00:08:08,800
these slides and you want to go back and

244
00:08:07,440 --> 00:08:10,639
have a look at stuff

245
00:08:08,800 --> 00:08:12,560
so we'll go over very quickly so the

246
00:08:10,639 --> 00:08:14,479
project information for open tracing is

247
00:08:12,560 --> 00:08:15,599
opentracing.io it's available in nine

248
00:08:14,479 --> 00:08:17,199
languages

249
00:08:15,599 --> 00:08:18,879
i obviously use python because that's

250
00:08:17,199 --> 00:08:20,080
what swift is

251
00:08:18,879 --> 00:08:21,840
and

252
00:08:20,080 --> 00:08:23,440
it basically is an api specification

253
00:08:21,840 --> 00:08:25,039
frameworks and libraries that have that

254
00:08:23,440 --> 00:08:27,360
have implemented that specification and

255
00:08:25,039 --> 00:08:29,520
allows us as developers especially open

256
00:08:27,360 --> 00:08:31,280
source developers uh to be able to

257
00:08:29,520 --> 00:08:33,440
instrument our code

258
00:08:31,280 --> 00:08:34,880
using apis that do not fender lock us in

259
00:08:33,440 --> 00:08:37,200
no one likes vendor locking but what's

260
00:08:34,880 --> 00:08:39,519
even better is if other

261
00:08:37,200 --> 00:08:41,440
if other projects are using open tracing

262
00:08:39,519 --> 00:08:44,000
now we can start you know integrating

263
00:08:41,440 --> 00:08:45,920
with each other and start tracing

264
00:08:44,000 --> 00:08:48,160
let's not think of swift let's see the

265
00:08:45,920 --> 00:08:51,519
greater community the greater

266
00:08:48,160 --> 00:08:53,519
open open source open source environment

267
00:08:51,519 --> 00:08:55,040
anyway

268
00:08:53,519 --> 00:08:56,560
the concepts in terminology i'm only

269
00:08:55,040 --> 00:08:59,200
saying these because i will accidentally

270
00:08:56,560 --> 00:09:01,680
spout them out um and you guys won't may

271
00:08:59,200 --> 00:09:03,760
not understand what i'm talking about so

272
00:09:01,680 --> 00:09:05,120
concepts and terminology so there's a

273
00:09:03,760 --> 00:09:06,080
tracer so when you're instrumenting your

274
00:09:05,120 --> 00:09:07,200
code

275
00:09:06,080 --> 00:09:09,200
you're dealing with a tracer all the

276
00:09:07,200 --> 00:09:10,399
time and the tracer creates spans more

277
00:09:09,200 --> 00:09:12,240
on that in a sec

278
00:09:10,399 --> 00:09:14,959
um and keeps track of the active span

279
00:09:12,240 --> 00:09:17,040
it's kind of important um it also does a

280
00:09:14,959 --> 00:09:19,600
few things it injects which is really

281
00:09:17,040 --> 00:09:21,920
serialized and has a term extracts which

282
00:09:19,600 --> 00:09:24,320
is really deserializing the tracing

283
00:09:21,920 --> 00:09:25,200
context and so we can move it across

284
00:09:24,320 --> 00:09:26,080
through

285
00:09:25,200 --> 00:09:27,279
process

286
00:09:26,080 --> 00:09:29,600
boundaries

287
00:09:27,279 --> 00:09:29,600
um

288
00:09:29,920 --> 00:09:34,560
a span let's move back for a bit so

289
00:09:32,560 --> 00:09:36,959
here's this is back a slide

290
00:09:34,560 --> 00:09:39,519
going to come up you know here's a slide

291
00:09:36,959 --> 00:09:41,839
so we can we can say that this is a span

292
00:09:39,519 --> 00:09:43,040
each of these lines represents one span

293
00:09:41,839 --> 00:09:46,640
and

294
00:09:43,040 --> 00:09:49,040
this whole inject extract thing is

295
00:09:46,640 --> 00:09:51,519
when i get to the proxy swift proxy and

296
00:09:49,040 --> 00:09:53,760
i want to send a request to a different

297
00:09:51,519 --> 00:09:56,160
server or a different process boundary

298
00:09:53,760 --> 00:09:58,800
then i'll be able to inject this context

299
00:09:56,160 --> 00:10:00,399
of the trace and then send it off and

300
00:09:58,800 --> 00:10:01,760
then the the

301
00:10:00,399 --> 00:10:03,279
in this case the object server can pick

302
00:10:01,760 --> 00:10:04,640
it up and continue the trace and then we

303
00:10:03,279 --> 00:10:06,959
can knit these things all together at

304
00:10:04,640 --> 00:10:06,959
the end

305
00:10:07,839 --> 00:10:12,320
and so a span is the basic building

306
00:10:10,000 --> 00:10:14,160
block um it references other spans it

307
00:10:12,320 --> 00:10:15,680
kind of has this parent of child of if

308
00:10:14,160 --> 00:10:18,560
we go back again i'm sorry for jumping

309
00:10:15,680 --> 00:10:20,399
around so much but you can see that this

310
00:10:18,560 --> 00:10:23,920
band has a child and this child has a

311
00:10:20,399 --> 00:10:26,000
parent you know it keeps track of um

312
00:10:23,920 --> 00:10:27,519
the tree i guess

313
00:10:26,000 --> 00:10:29,760
and it's starting to finish time stamps

314
00:10:27,519 --> 00:10:30,880
and has things called tags logs and

315
00:10:29,760 --> 00:10:33,279
baggage

316
00:10:30,880 --> 00:10:36,560
all of which are key value pairs the

317
00:10:33,279 --> 00:10:39,440
only real difference is a tag is kind of

318
00:10:36,560 --> 00:10:41,200
can be filtered by or queried and so

319
00:10:39,440 --> 00:10:44,000
it's kind of it's useful for when you've

320
00:10:41,200 --> 00:10:46,000
got it when you've pulled out the trace

321
00:10:44,000 --> 00:10:47,839
they expect to use standard tags which

322
00:10:46,000 --> 00:10:49,440
makes sense stuff like http method and

323
00:10:47,839 --> 00:10:51,040
status code and stuff like that so we

324
00:10:49,440 --> 00:10:52,800
can all be

325
00:10:51,040 --> 00:10:56,320
use the same tags across

326
00:10:52,800 --> 00:10:58,079
a trace or custom so in swift we

327
00:10:56,320 --> 00:10:59,360
tag every request that goes through our

328
00:10:58,079 --> 00:11:01,440
cluster

329
00:10:59,360 --> 00:11:04,160
if every every request gets a unique

330
00:11:01,440 --> 00:11:07,360
transaction id so for in our case having

331
00:11:04,160 --> 00:11:09,760
that as a tag makes perfect sense

332
00:11:07,360 --> 00:11:11,360
logs are just key value pairs again that

333
00:11:09,760 --> 00:11:12,640
are just stored to span so you can pull

334
00:11:11,360 --> 00:11:14,399
information out

335
00:11:12,640 --> 00:11:16,320
i guess another example that i like in

336
00:11:14,399 --> 00:11:19,600
our in the swift deployment uh swift

337
00:11:16,320 --> 00:11:22,000
version of this is a proxy server

338
00:11:19,600 --> 00:11:23,760
is smart enough to when it has trouble

339
00:11:22,000 --> 00:11:25,040
talking to a node

340
00:11:23,760 --> 00:11:26,720
it can

341
00:11:25,040 --> 00:11:29,040
update it doesn't it you can error limit

342
00:11:26,720 --> 00:11:30,240
that node for for a certain time

343
00:11:29,040 --> 00:11:31,680
now when you're coming back and you're

344
00:11:30,240 --> 00:11:33,360
finding you're doing a trace because

345
00:11:31,680 --> 00:11:34,640
you're trying to find a problem that's

346
00:11:33,360 --> 00:11:36,560
the kind of information you want to know

347
00:11:34,640 --> 00:11:38,160
about your cluster so it's perfect thing

348
00:11:36,560 --> 00:11:40,399
you can pull in and get a snapshot of

349
00:11:38,160 --> 00:11:42,079
time in your trace too

350
00:11:40,399 --> 00:11:43,600
and the other thing is baggage now

351
00:11:42,079 --> 00:11:45,839
baggage

352
00:11:43,600 --> 00:11:47,040
it's funny name it it allows you to

353
00:11:45,839 --> 00:11:49,040
predict

354
00:11:47,040 --> 00:11:50,160
these key value pairs

355
00:11:49,040 --> 00:11:53,040
um

356
00:11:50,160 --> 00:11:55,040
when injected or when serialized are

357
00:11:53,040 --> 00:11:56,320
stored with that serialization so it

358
00:11:55,040 --> 00:11:57,600
means you can do some communication

359
00:11:56,320 --> 00:12:00,000
between

360
00:11:57,600 --> 00:12:01,600
traces which is between tracing entities

361
00:12:00,000 --> 00:12:02,560
between processes between whatever you

362
00:12:01,600 --> 00:12:04,560
want to call it

363
00:12:02,560 --> 00:12:06,800
okay i've probably rambled on way too

364
00:12:04,560 --> 00:12:08,880
long then

365
00:12:06,800 --> 00:12:11,360
quickly we talk about whiskey because

366
00:12:08,880 --> 00:12:13,040
it's going to make sense

367
00:12:11,360 --> 00:12:16,320
when you look at us one of these traces

368
00:12:13,040 --> 00:12:17,760
i'm going to show you um

369
00:12:16,320 --> 00:12:19,279
there's a lot of there's a lot of spans

370
00:12:17,760 --> 00:12:22,959
and most of them are because of the way

371
00:12:19,279 --> 00:12:25,600
whiskey works whiskey is a python

372
00:12:22,959 --> 00:12:27,440
rest framework um well we're going to

373
00:12:25,600 --> 00:12:29,839
delve into too much details

374
00:12:27,440 --> 00:12:32,399
pictures are worth a thousand words so

375
00:12:29,839 --> 00:12:34,240
in essence here's a very basic example

376
00:12:32,399 --> 00:12:35,519
it's built up a bunch of middlewares and

377
00:12:34,240 --> 00:12:38,240
an application at the end and as a

378
00:12:35,519 --> 00:12:40,399
request comes in it'll walk it'll work

379
00:12:38,240 --> 00:12:42,399
its way down the chain

380
00:12:40,399 --> 00:12:43,680
and then back out and send that back the

381
00:12:42,399 --> 00:12:46,079
response

382
00:12:43,680 --> 00:12:47,600
now a middleware you can interact with

383
00:12:46,079 --> 00:12:50,720
the request as it comes in

384
00:12:47,600 --> 00:12:52,560
when a response comes out or

385
00:12:50,720 --> 00:12:53,920
you can kind of i call short-circuiting

386
00:12:52,560 --> 00:12:55,839
it so you can

387
00:12:53,920 --> 00:12:56,959
in this for this this example we say

388
00:12:55,839 --> 00:12:59,360
that third middle wears an

389
00:12:56,959 --> 00:13:01,120
authentication middleware and it may

390
00:12:59,360 --> 00:13:03,440
have gone talk to your ldap server

391
00:13:01,120 --> 00:13:05,440
whatever you want and said no not

392
00:13:03,440 --> 00:13:06,399
authorized to send 401 unauthorized back

393
00:13:05,440 --> 00:13:08,880
and we don't actually have to make it

394
00:13:06,399 --> 00:13:11,360
all the way to the application

395
00:13:08,880 --> 00:13:13,760
here's a cut down version of what this

396
00:13:11,360 --> 00:13:15,760
looks like in the swift proxy server

397
00:13:13,760 --> 00:13:17,680
just so you can see that

398
00:13:15,760 --> 00:13:20,079
it's it's not nothing's rocket science

399
00:13:17,680 --> 00:13:21,680
here in fact you also notice that a lot

400
00:13:20,079 --> 00:13:25,040
of the swift features are middlewares in

401
00:13:21,680 --> 00:13:27,120
their own right so if you want to

402
00:13:25,040 --> 00:13:28,720
um not have any rate limiting i don't

403
00:13:27,120 --> 00:13:29,839
know why you don't want that but say you

404
00:13:28,720 --> 00:13:31,360
don't care about living because the

405
00:13:29,839 --> 00:13:34,160
system's all for yourself

406
00:13:31,360 --> 00:13:36,560
you could just remove that middleware

407
00:13:34,160 --> 00:13:39,360
from the pipeline definition which is a

408
00:13:36,560 --> 00:13:40,800
list of um middlewares and we wouldn't

409
00:13:39,360 --> 00:13:41,920
talk about it it would just the request

410
00:13:40,800 --> 00:13:43,680
would skip it completely so that's

411
00:13:41,920 --> 00:13:45,279
really cool um

412
00:13:43,680 --> 00:13:47,199
and earth is also where that

413
00:13:45,279 --> 00:13:48,880
extensibility comes in so you can write

414
00:13:47,199 --> 00:13:50,880
your own middleware that puts that

415
00:13:48,880 --> 00:13:53,279
inserts your own business logic

416
00:13:50,880 --> 00:13:55,519
and inserts that uh and you can send it

417
00:13:53,279 --> 00:13:57,199
anywhere you want so you can do

418
00:13:55,519 --> 00:13:58,959
something on a request as it gets coming

419
00:13:57,199 --> 00:14:01,680
into the proxy

420
00:13:58,959 --> 00:14:04,480
or when it comes back out or better yet

421
00:14:01,680 --> 00:14:06,800
it turns out the swift uh

422
00:14:04,480 --> 00:14:09,199
account server container servers and

423
00:14:06,800 --> 00:14:11,199
object servers are all whiskey and have

424
00:14:09,199 --> 00:14:12,560
their own pipelines too so you can even

425
00:14:11,199 --> 00:14:13,920
insert a middlewares to do something

426
00:14:12,560 --> 00:14:15,519
before it hits disk or when it comes out

427
00:14:13,920 --> 00:14:17,040
so swift is highly extensible it's

428
00:14:15,519 --> 00:14:19,519
pretty cool

429
00:14:17,040 --> 00:14:21,440
here's a here's another trace example

430
00:14:19,519 --> 00:14:22,880
where it shows this this is an

431
00:14:21,440 --> 00:14:24,720
authentication

432
00:14:22,880 --> 00:14:27,040
to our tempo system which is only for

433
00:14:24,720 --> 00:14:28,480
development

434
00:14:27,040 --> 00:14:30,000
but you can see

435
00:14:28,480 --> 00:14:31,600
now we look on this side we can see that

436
00:14:30,000 --> 00:14:34,240
it's walking down through the list of

437
00:14:31,600 --> 00:14:34,959
middleware and we can see actually watch

438
00:14:34,240 --> 00:14:38,800
it

439
00:14:34,959 --> 00:14:40,240
chain along going working its way down

440
00:14:38,800 --> 00:14:42,079
and then

441
00:14:40,240 --> 00:14:43,519
coming walking back up the chain never

442
00:14:42,079 --> 00:14:45,360
made it to the application never made it

443
00:14:43,519 --> 00:14:49,600
to the proxy server and walked its way

444
00:14:45,360 --> 00:14:49,600
back up and received a response

445
00:14:49,760 --> 00:14:52,959
so we're almost we're almost at demos

446
00:14:51,680 --> 00:14:54,480
this is the exciting we're going to also

447
00:14:52,959 --> 00:14:56,839
design a bit thanks for sticking with me

448
00:14:54,480 --> 00:14:58,399
we've been going for a while

449
00:14:56,839 --> 00:15:00,240
um

450
00:14:58,399 --> 00:15:02,880
so let's see so

451
00:15:00,240 --> 00:15:05,279
high level integration so like the rest

452
00:15:02,880 --> 00:15:07,120
of swift it's done

453
00:15:05,279 --> 00:15:08,480
similarly well i'm a swift developer so

454
00:15:07,120 --> 00:15:10,399
it kind of makes sense

455
00:15:08,480 --> 00:15:11,279
uh it's a basically

456
00:15:10,399 --> 00:15:13,920
uh

457
00:15:11,279 --> 00:15:16,800
middleware that we insert um and there's

458
00:15:13,920 --> 00:15:18,480
a trace mix in for uh that all the

459
00:15:16,800 --> 00:15:19,839
middle was inherent because

460
00:15:18,480 --> 00:15:21,600
it helps

461
00:15:19,839 --> 00:15:23,040
and i'll explain in a sec so the trace

462
00:15:21,600 --> 00:15:25,199
middleware as it comes in will be

463
00:15:23,040 --> 00:15:27,519
inserted in the pipeline of all the

464
00:15:25,199 --> 00:15:30,079
storage nodes too

465
00:15:27,519 --> 00:15:32,560
and when a request its sole purpose is

466
00:15:30,079 --> 00:15:35,759
when a request comes in and is to decide

467
00:15:32,560 --> 00:15:38,160
if tracing should start or not

468
00:15:35,759 --> 00:15:40,079
it's um

469
00:15:38,160 --> 00:15:42,560
it's also i guess it's also to note that

470
00:15:40,079 --> 00:15:43,920
because it starts and stops tracing that

471
00:15:42,560 --> 00:15:46,000
anything to the left of it so the

472
00:15:43,920 --> 00:15:48,720
gatekeeper here

473
00:15:46,000 --> 00:15:50,079
it won't get it won't show up in any in

474
00:15:48,720 --> 00:15:52,240
any traces so it's good to have it as

475
00:15:50,079 --> 00:15:53,600
far left as you can go uh gatekeeper

476
00:15:52,240 --> 00:15:56,320
it's always on the end and swift because

477
00:15:53,600 --> 00:15:58,639
it stops yuckiness getting out

478
00:15:56,320 --> 00:16:00,880
data you don't want to getting out so

479
00:15:58,639 --> 00:16:02,959
the request comes in and hits a trace

480
00:16:00,880 --> 00:16:04,079
now there's three ways that the trace

481
00:16:02,959 --> 00:16:07,279
that that

482
00:16:04,079 --> 00:16:08,880
that the tracer will start a trace

483
00:16:07,279 --> 00:16:11,440
um so the trace middle will start a

484
00:16:08,880 --> 00:16:13,920
trace and the first one is have you

485
00:16:11,440 --> 00:16:15,600
given it a specific hmac signature so

486
00:16:13,920 --> 00:16:18,240
the whole purpose of this

487
00:16:15,600 --> 00:16:20,560
and for our use case is

488
00:16:18,240 --> 00:16:21,680
we have a customer who's having issues

489
00:16:20,560 --> 00:16:22,959
and

490
00:16:21,680 --> 00:16:24,560
at the moment

491
00:16:22,959 --> 00:16:25,920
we can usually come tell us they're

492
00:16:24,560 --> 00:16:28,639
having issues we look at the logs and

493
00:16:25,920 --> 00:16:30,079
swift does great logging but sometimes

494
00:16:28,639 --> 00:16:31,839
there's still that level of read the

495
00:16:30,079 --> 00:16:33,759
logs come with a hypothesis and then

496
00:16:31,839 --> 00:16:36,560
they go test to see if that would make

497
00:16:33,759 --> 00:16:38,800
sure that's correct or whatever

498
00:16:36,560 --> 00:16:40,800
but with this tracing middleware

499
00:16:38,800 --> 00:16:43,360
inserted we can go give that customer

500
00:16:40,800 --> 00:16:45,600
use or whatever uh hmac signature to

501
00:16:43,360 --> 00:16:46,480
attach to the end of their query

502
00:16:45,600 --> 00:16:47,600
and

503
00:16:46,480 --> 00:16:49,680
um

504
00:16:47,600 --> 00:16:51,120
and as soon as the trace sees that it'll

505
00:16:49,680 --> 00:16:53,279
start tracing

506
00:16:51,120 --> 00:16:55,519
and it'll work its way my mask on it'll

507
00:16:53,279 --> 00:16:57,440
work its way back up this chain and then

508
00:16:55,519 --> 00:16:59,839
work your way back down and then stop

509
00:16:57,440 --> 00:16:59,839
tracing

510
00:17:00,240 --> 00:17:04,400
so that's one way of starting tracing

511
00:17:02,560 --> 00:17:07,360
the second way is a continuation from

512
00:17:04,400 --> 00:17:09,919
that so once we get to the proxy and

513
00:17:07,360 --> 00:17:12,240
we're making those additional requests

514
00:17:09,919 --> 00:17:14,480
out to storage nodes

515
00:17:12,240 --> 00:17:16,720
we enjoy like i mentioned before we

516
00:17:14,480 --> 00:17:17,839
inject the tracing context if it's

517
00:17:16,720 --> 00:17:20,079
tracing

518
00:17:17,839 --> 00:17:22,720
into the request and so that's the

519
00:17:20,079 --> 00:17:25,520
second type so if a trace middleware

520
00:17:22,720 --> 00:17:27,360
sees that there's an injected

521
00:17:25,520 --> 00:17:28,880
trace context it knows to continue

522
00:17:27,360 --> 00:17:31,200
tracing

523
00:17:28,880 --> 00:17:32,960
the third is the what i'm using

524
00:17:31,200 --> 00:17:36,160
in this demo because

525
00:17:32,960 --> 00:17:38,320
i'm lazy and that is log every

526
00:17:36,160 --> 00:17:40,080
x request in essence and i'll have it

527
00:17:38,320 --> 00:17:41,039
set as log every request because i want

528
00:17:40,080 --> 00:17:43,039
to test

529
00:17:41,039 --> 00:17:44,320
um but obviously this could be like you

530
00:17:43,039 --> 00:17:45,760
want to do a health check so you might

531
00:17:44,320 --> 00:17:48,240
want to run it every thousand or ten

532
00:17:45,760 --> 00:17:50,720
thousand or whatever you want you pick

533
00:17:48,240 --> 00:17:52,559
and so the three ways the trace mix in

534
00:17:50,720 --> 00:17:53,520
just means a whole bunch of helper

535
00:17:52,559 --> 00:17:56,720
methods are available for

536
00:17:53,520 --> 00:17:58,880
instrumentation to each middleware it um

537
00:17:56,720 --> 00:18:02,000
it automatically starts

538
00:17:58,880 --> 00:18:03,840
traces and enters and and ends the trace

539
00:18:02,000 --> 00:18:05,520
that's sorry not the trace starts the

540
00:18:03,840 --> 00:18:07,840
span

541
00:18:05,520 --> 00:18:10,080
when it enters and ends the span context

542
00:18:07,840 --> 00:18:11,840
when it finishes and just so that's why

543
00:18:10,080 --> 00:18:13,360
we just for free get that nice uh

544
00:18:11,840 --> 00:18:15,039
graphing the tracing

545
00:18:13,360 --> 00:18:16,960
and another than that you know it just

546
00:18:15,039 --> 00:18:19,039
helps oh the other thing it does is also

547
00:18:16,960 --> 00:18:21,840
it calculates the

548
00:18:19,039 --> 00:18:23,919
the response um

549
00:18:21,840 --> 00:18:25,039
the status code as it's um of the

550
00:18:23,919 --> 00:18:26,480
response

551
00:18:25,039 --> 00:18:28,720
we do this through the line because

552
00:18:26,480 --> 00:18:29,679
things like the s3 api

553
00:18:28,720 --> 00:18:32,160
they

554
00:18:29,679 --> 00:18:33,440
they might return a different response

555
00:18:32,160 --> 00:18:35,360
code for

556
00:18:33,440 --> 00:18:37,280
uh status code for different

557
00:18:35,360 --> 00:18:39,679
apis than what swift does so we it's

558
00:18:37,280 --> 00:18:41,200
nice to be able to track what it was at

559
00:18:39,679 --> 00:18:42,799
a certain point in time that's just an

560
00:18:41,200 --> 00:18:44,080
example

561
00:18:42,799 --> 00:18:45,679
so the high level integration this is

562
00:18:44,080 --> 00:18:47,919
the hmx signature i'm going to talk

563
00:18:45,679 --> 00:18:50,000
about this is the same slide as before

564
00:18:47,919 --> 00:18:51,600
but just annotated and so the trace in

565
00:18:50,000 --> 00:18:54,240
this case is going to come in hit the

566
00:18:51,600 --> 00:18:56,160
swift proxy go through that pipeline

567
00:18:54,240 --> 00:19:00,160
it's going to see this hmac signature

568
00:18:56,160 --> 00:19:02,160
which is time limited and then we'll

569
00:19:00,160 --> 00:19:04,000
start tracing the request as it's going

570
00:19:02,160 --> 00:19:05,840
to send inserts the trace context is

571
00:19:04,000 --> 00:19:07,600
pretty straightforward

572
00:19:05,840 --> 00:19:08,960
so without further ado we can do some

573
00:19:07,600 --> 00:19:11,280
demoing

574
00:19:08,960 --> 00:19:13,600
so let's see if this works well um it is

575
00:19:11,280 --> 00:19:15,440
lca so you know take it off with a grain

576
00:19:13,600 --> 00:19:18,960
of salt

577
00:19:15,440 --> 00:19:20,799
i'm using uh jaeger tracing because it

578
00:19:18,960 --> 00:19:24,160
was just a docker install away they

579
00:19:20,799 --> 00:19:26,320
implement the open tracing api so

580
00:19:24,160 --> 00:19:28,960
it just works right when i say just

581
00:19:26,320 --> 00:19:31,360
works i had to make a tracer this is a

582
00:19:28,960 --> 00:19:33,679
document that i created for

583
00:19:31,360 --> 00:19:36,880
the openstack ptg so if you want to have

584
00:19:33,679 --> 00:19:38,400
the exact same environment that i have

585
00:19:36,880 --> 00:19:41,200
i'll give you a link in fact the link to

586
00:19:38,400 --> 00:19:42,640
this google doc is in my um

587
00:19:41,200 --> 00:19:45,600
is in these slides

588
00:19:42,640 --> 00:19:46,880
so this is the whopping amount of jaeger

589
00:19:45,600 --> 00:19:49,039
tracer code

590
00:19:46,880 --> 00:19:52,679
i think it's after imports what three

591
00:19:49,039 --> 00:19:52,679
lines of code so

592
00:19:52,720 --> 00:19:55,440
yeah it was very it's very easy and

593
00:19:54,320 --> 00:19:57,440
that's because this is the whole

594
00:19:55,440 --> 00:19:58,960
advantage of open tracing

595
00:19:57,440 --> 00:20:00,559
okay so this is

596
00:19:58,960 --> 00:20:01,919
jaeger tracer and this is what we're

597
00:20:00,559 --> 00:20:04,720
going to use to show some examples but

598
00:20:01,919 --> 00:20:08,159
first we need to actually make some

599
00:20:04,720 --> 00:20:10,159
things so let me go to a terminal and

600
00:20:08,159 --> 00:20:12,559
step one this is a fresh

601
00:20:10,159 --> 00:20:14,640
swift all-in-one uh virtual swift

602
00:20:12,559 --> 00:20:16,080
all-in-one a demo environment a dev

603
00:20:14,640 --> 00:20:17,120
environment that we use so i'm just

604
00:20:16,080 --> 00:20:19,600
going to do that that's just going to

605
00:20:17,120 --> 00:20:22,080
use the swift command line tool to

606
00:20:19,600 --> 00:20:24,799
to run an authentication

607
00:20:22,080 --> 00:20:27,440
and then

608
00:20:24,799 --> 00:20:28,720
uh where i'm sorry i'm lost

609
00:20:27,440 --> 00:20:30,559
we should see

610
00:20:28,720 --> 00:20:33,280
a swift

611
00:20:30,559 --> 00:20:36,080
and if we do find traces

612
00:20:33,280 --> 00:20:38,000
where you have our first trace

613
00:20:36,080 --> 00:20:40,480
this is the transaction id

614
00:20:38,000 --> 00:20:41,760
we know what to get and if i click on it

615
00:20:40,480 --> 00:20:43,120
this should look very familiar because

616
00:20:41,760 --> 00:20:45,360
there's an authentication so you'll see

617
00:20:43,120 --> 00:20:47,039
it worked its way down to tempo so this

618
00:20:45,360 --> 00:20:49,679
screen is a little bit

619
00:20:47,039 --> 00:20:50,640
smaller so

620
00:20:49,679 --> 00:20:52,559
whatever

621
00:20:50,640 --> 00:20:53,840
uh so we see it's what we saw before but

622
00:20:52,559 --> 00:20:55,440
if we

623
00:20:53,840 --> 00:20:56,720
if we like look at the first line we can

624
00:20:55,440 --> 00:20:59,440
start looking at all these tags so

625
00:20:56,720 --> 00:21:01,039
you've got hey it was a get it got a 200

626
00:20:59,440 --> 00:21:03,600
its url was

627
00:21:01,039 --> 00:21:05,840
the temple the url

628
00:21:03,600 --> 00:21:08,320
and we have logs and stuff like that

629
00:21:05,840 --> 00:21:09,760
very exciting stuff

630
00:21:08,320 --> 00:21:11,600
but let's look at more interesting

631
00:21:09,760 --> 00:21:13,039
things

632
00:21:11,600 --> 00:21:15,520
um

633
00:21:13,039 --> 00:21:19,120
sorry be there a sec

634
00:21:15,520 --> 00:21:21,440
so let's see so next

635
00:21:19,120 --> 00:21:23,360
we'll quickly

636
00:21:21,440 --> 00:21:25,280
put in

637
00:21:23,360 --> 00:21:27,520
let's put a container because we need

638
00:21:25,280 --> 00:21:29,679
somewhere to put objects

639
00:21:27,520 --> 00:21:32,240
so i'm going to put in

640
00:21:29,679 --> 00:21:33,280
just a container called c because i'm

641
00:21:32,240 --> 00:21:34,559
lazy

642
00:21:33,280 --> 00:21:37,520
um

643
00:21:34,559 --> 00:21:40,159
and i have no imagination on naming

644
00:21:37,520 --> 00:21:43,200
and at the same time i'll put in an

645
00:21:40,159 --> 00:21:45,200
erasure come on apparently i can't

646
00:21:43,200 --> 00:21:47,520
it'd be fast if i just type this i'm

647
00:21:45,200 --> 00:21:48,880
having trouble

648
00:21:47,520 --> 00:21:49,919
copying and pasting

649
00:21:48,880 --> 00:21:50,880
and so where you can see these

650
00:21:49,919 --> 00:21:52,640
transactions happen we've got some

651
00:21:50,880 --> 00:21:55,600
transaction ids here

652
00:21:52,640 --> 00:21:56,880
four three five nine and four three six

653
00:21:55,600 --> 00:21:59,360
b

654
00:21:56,880 --> 00:22:01,120
uh if i go back here and do a fine trace

655
00:21:59,360 --> 00:22:02,240
here they both are

656
00:22:01,120 --> 00:22:04,960
first one

657
00:22:02,240 --> 00:22:07,440
has 175 spans there's a reason for that

658
00:22:04,960 --> 00:22:09,039
it's a lot one it's more in that path

659
00:22:07,440 --> 00:22:10,159
that code path is switched up a lot more

660
00:22:09,039 --> 00:22:13,760
instrumented

661
00:22:10,159 --> 00:22:15,360
than the ec put the container but um

662
00:22:13,760 --> 00:22:19,520
also because this is like i said this is

663
00:22:15,360 --> 00:22:22,000
a clean brand new um class um swift

664
00:22:19,520 --> 00:22:23,200
environment so the temp auth is going to

665
00:22:22,000 --> 00:22:24,960
create

666
00:22:23,200 --> 00:22:26,480
the account doesn't exist so it's going

667
00:22:24,960 --> 00:22:28,400
to have to create it so you see this

668
00:22:26,480 --> 00:22:29,760
really long line

669
00:22:28,400 --> 00:22:31,760
the first view are probably going to be

670
00:22:29,760 --> 00:22:33,039
like hey does this account does this

671
00:22:31,760 --> 00:22:34,080
account exist

672
00:22:33,039 --> 00:22:36,159
um

673
00:22:34,080 --> 00:22:37,760
say it's an account server and it's

674
00:22:36,159 --> 00:22:39,120
getting 404 so the account doesn't exist

675
00:22:37,760 --> 00:22:40,720
so there's a whole bunch of extra effort

676
00:22:39,120 --> 00:22:42,320
going on here because it has to create

677
00:22:40,720 --> 00:22:43,919
these things

678
00:22:42,320 --> 00:22:46,000
um if we go look at the

679
00:22:43,919 --> 00:22:48,880
towards the end we should see

680
00:22:46,000 --> 00:22:50,240
it actually putting

681
00:22:48,880 --> 00:22:51,600
so many

682
00:22:50,240 --> 00:22:53,120
we should see it actually starting to

683
00:22:51,600 --> 00:22:55,039
put this container server in and then

684
00:22:53,120 --> 00:22:57,120
it's going to go update the account

685
00:22:55,039 --> 00:22:59,360
server metadata so we can watch that

686
00:22:57,120 --> 00:23:00,320
happen and it should happen three times

687
00:22:59,360 --> 00:23:01,919
because

688
00:23:00,320 --> 00:23:04,080
my container ring is three times

689
00:23:01,919 --> 00:23:05,919
replication

690
00:23:04,080 --> 00:23:09,039
now i'll show you the same for this one

691
00:23:05,919 --> 00:23:11,280
it should be less because it's um

692
00:23:09,039 --> 00:23:14,240
still very long it looks like maybe my

693
00:23:11,280 --> 00:23:15,919
cache my memcache um

694
00:23:14,240 --> 00:23:17,280
is not is either ejecting things very

695
00:23:15,919 --> 00:23:18,640
quickly uh because you would have

696
00:23:17,280 --> 00:23:20,559
thought that would have been cached but

697
00:23:18,640 --> 00:23:22,559
hey see here's an advantage straight

698
00:23:20,559 --> 00:23:25,120
away of kind of getting to know what

699
00:23:22,559 --> 00:23:27,679
these traces look like um you can't

700
00:23:25,120 --> 00:23:29,200
start seeing a long line problem

701
00:23:27,679 --> 00:23:29,919
i guess another thing to note is what

702
00:23:29,200 --> 00:23:31,520
i'm

703
00:23:29,919 --> 00:23:33,679
the stuff that i'm putting in

704
00:23:31,520 --> 00:23:35,039
are kind of all kind of junk objects

705
00:23:33,679 --> 00:23:36,480
right like why don't put an object in

706
00:23:35,039 --> 00:23:37,600
let's let's do that

707
00:23:36,480 --> 00:23:38,720
um

708
00:23:37,600 --> 00:23:42,080
so right so

709
00:23:38,720 --> 00:23:44,000
that first container was so many so many

710
00:23:42,080 --> 00:23:45,760
spans because it said hey does it can

711
00:23:44,000 --> 00:23:47,360
exist no go get

712
00:23:45,760 --> 00:23:48,960
it doesn't so i'll go create it it

713
00:23:47,360 --> 00:23:49,760
doesn't contain exist no i'll go create

714
00:23:48,960 --> 00:23:51,279
it

715
00:23:49,760 --> 00:23:52,799
and then finally go create so there's

716
00:23:51,279 --> 00:23:54,480
lots of extra requests that usually

717
00:23:52,799 --> 00:23:56,000
shouldn't happen but

718
00:23:54,480 --> 00:23:57,679
because it's a nice

719
00:23:56,000 --> 00:24:00,000
new greenfield system that's what

720
00:23:57,679 --> 00:24:02,159
happens so let's put

721
00:24:00,000 --> 00:24:03,679
uh

722
00:24:02,159 --> 00:24:04,799
what i'm just trying to copy i'm really

723
00:24:03,679 --> 00:24:05,600
struggling with my copy and pasting

724
00:24:04,799 --> 00:24:07,120
today

725
00:24:05,600 --> 00:24:10,080
i blame

726
00:24:07,120 --> 00:24:11,440
doing this live and not in a video

727
00:24:10,080 --> 00:24:13,440
sure

728
00:24:11,440 --> 00:24:16,880
and so now i'm just putting an object uh

729
00:24:13,440 --> 00:24:18,480
into the c container which is replicated

730
00:24:16,880 --> 00:24:20,400
and while we're at it let's put one into

731
00:24:18,480 --> 00:24:21,919
the eurasian coded one which i think are

732
00:24:20,400 --> 00:24:23,120
called ec

733
00:24:21,919 --> 00:24:24,000
now

734
00:24:23,120 --> 00:24:25,360
these

735
00:24:24,000 --> 00:24:26,640
are silly objects they're four bytes

736
00:24:25,360 --> 00:24:28,880
aren't they they've just got like one

737
00:24:26,640 --> 00:24:32,679
two three four in them um

738
00:24:28,880 --> 00:24:32,679
so you know

739
00:24:32,880 --> 00:24:35,679
especially for erasure coding you

740
00:24:34,159 --> 00:24:38,799
wouldn't want to rate your code 4 bytes

741
00:24:35,679 --> 00:24:40,080
it's a little bit overkill but anyway

742
00:24:38,799 --> 00:24:42,400
let's see what happened

743
00:24:40,080 --> 00:24:44,400
so if we find these traces now we should

744
00:24:42,400 --> 00:24:46,799
see

745
00:24:44,400 --> 00:24:48,960
two new puts uh it doesn't matter really

746
00:24:46,799 --> 00:24:50,400
matter one's a bit more again

747
00:24:48,960 --> 00:24:52,559
hold on these are objects so this is

748
00:24:50,400 --> 00:24:53,679
this is correct okay so we've put us on

749
00:24:52,559 --> 00:24:56,720
object in

750
00:24:53,679 --> 00:25:00,039
it's let's see what this first one is

751
00:24:56,720 --> 00:25:00,039
it is

752
00:25:00,240 --> 00:25:04,320
um yeah okay so this should this should

753
00:25:02,240 --> 00:25:06,640
have been cached because in our memcache

754
00:25:04,320 --> 00:25:10,000
server we should already know that uh

755
00:25:06,640 --> 00:25:12,559
auth test kind of exists so

756
00:25:10,000 --> 00:25:14,240
there's a bug um although there's a mem

757
00:25:12,559 --> 00:25:16,880
cache setting issue on my dev

758
00:25:14,240 --> 00:25:18,960
environment but i'm glad i can see that

759
00:25:16,880 --> 00:25:20,960
so the important stuff here is this

760
00:25:18,960 --> 00:25:22,400
stuff then

761
00:25:20,960 --> 00:25:23,919
where we're actually putting the object

762
00:25:22,400 --> 00:25:26,880
in

763
00:25:23,919 --> 00:25:29,760
testo and again because a standard

764
00:25:26,880 --> 00:25:32,000
object uh by default i've got my default

765
00:25:29,760 --> 00:25:33,600
policy as a three times replicated we

766
00:25:32,000 --> 00:25:36,159
should have three so that's not as

767
00:25:33,600 --> 00:25:37,440
exciting what is more interesting is

768
00:25:36,159 --> 00:25:38,400
some of these other spans that i've

769
00:25:37,440 --> 00:25:41,200
created

770
00:25:38,400 --> 00:25:43,279
so i can't read it so

771
00:25:41,200 --> 00:25:45,120
this is a span

772
00:25:43,279 --> 00:25:46,880
so i've instrumented the path going

773
00:25:45,120 --> 00:25:48,960
through so there's a store object method

774
00:25:46,880 --> 00:25:50,400
a bit like flame graphs right so it's

775
00:25:48,960 --> 00:25:52,480
getting put connections it's gathering

776
00:25:50,400 --> 00:25:54,799
all the the connections to the storage

777
00:25:52,480 --> 00:25:55,679
nodes it needs to be able to put this

778
00:25:54,799 --> 00:25:57,840
um

779
00:25:55,679 --> 00:25:59,279
and there's connect to the one node this

780
00:25:57,840 --> 00:26:01,919
one that's connects to another node and

781
00:25:59,279 --> 00:26:02,799
we see the nodes interleaved in which is

782
00:26:01,919 --> 00:26:04,000
nice

783
00:26:02,799 --> 00:26:06,240
um

784
00:26:04,000 --> 00:26:09,120
down here we've got check for failure

785
00:26:06,240 --> 00:26:10,559
put connections transfer data well

786
00:26:09,120 --> 00:26:13,039
there's four bytes right so that's not

787
00:26:10,559 --> 00:26:14,960
going to do anything and then we've got

788
00:26:13,039 --> 00:26:17,039
get put responses

789
00:26:14,960 --> 00:26:19,279
um the idea is

790
00:26:17,039 --> 00:26:20,480
was that durably written if it wasn't it

791
00:26:19,279 --> 00:26:22,840
would go out make another connection

792
00:26:20,480 --> 00:26:24,640
that kind of stuff

793
00:26:22,840 --> 00:26:26,400
um

794
00:26:24,640 --> 00:26:29,440
looking at the ec one should be

795
00:26:26,400 --> 00:26:31,919
identical except it's an ec policies and

796
00:26:29,440 --> 00:26:34,159
i think the policy was a 4-2 policy

797
00:26:31,919 --> 00:26:36,320
that's four times what's our data and

798
00:26:34,159 --> 00:26:38,400
two times um

799
00:26:36,320 --> 00:26:39,840
uh parity

800
00:26:38,400 --> 00:26:44,159
so we should have six connections we

801
00:26:39,840 --> 00:26:45,440
have one two three four five six and

802
00:26:44,159 --> 00:26:47,440
we've got the same kind of stuff we're

803
00:26:45,440 --> 00:26:49,520
transferring data takes longer well

804
00:26:47,440 --> 00:26:51,520
actually is it taking longer or is it

805
00:26:49,520 --> 00:26:53,039
there's less data to send

806
00:26:51,520 --> 00:26:54,480
but there's actually more because it's

807
00:26:53,039 --> 00:26:55,679
four bytes and so there's a bit of

808
00:26:54,480 --> 00:26:57,600
overhead this is why you wouldn't use a

809
00:26:55,679 --> 00:26:59,360
razer coating on a four byte object

810
00:26:57,600 --> 00:27:00,880
um but again i don't want to go find a

811
00:26:59,360 --> 00:27:02,880
tangent and we can also see doing

812
00:27:00,880 --> 00:27:04,320
updates

813
00:27:02,880 --> 00:27:06,799
and so there it is it's doing a put to

814
00:27:04,320 --> 00:27:07,760
the ec container it got a 201

815
00:27:06,799 --> 00:27:09,840
so

816
00:27:07,760 --> 00:27:11,279
it's kind of fun to watch what it should

817
00:27:09,840 --> 00:27:13,120
look like but

818
00:27:11,279 --> 00:27:14,640
they're not all that super exciting

819
00:27:13,120 --> 00:27:17,120
because it's just putting some objects

820
00:27:14,640 --> 00:27:18,720
in i mean that is the main i guess

821
00:27:17,120 --> 00:27:21,200
reason for an object store system let's

822
00:27:18,720 --> 00:27:22,960
try and get that ec container back out

823
00:27:21,200 --> 00:27:25,279
object back out

824
00:27:22,960 --> 00:27:27,279
so for pudding we obviously had to write

825
00:27:25,279 --> 00:27:28,880
to all the

826
00:27:27,279 --> 00:27:31,760
all six nodes

827
00:27:28,880 --> 00:27:33,760
when we're getting on the other hand

828
00:27:31,760 --> 00:27:35,279
it only has to talk to four so we should

829
00:27:33,760 --> 00:27:37,520
see four

830
00:27:35,279 --> 00:27:39,200
two four four b e

831
00:27:37,520 --> 00:27:41,039
i don't know why i'm trying to make

832
00:27:39,200 --> 00:27:43,600
confirmed transaction ids they're always

833
00:27:41,039 --> 00:27:43,600
going to be at the top

834
00:27:43,919 --> 00:27:47,039
and again it seems to be doing these

835
00:27:45,520 --> 00:27:48,640
stem

836
00:27:47,039 --> 00:27:50,480
is it

837
00:27:48,640 --> 00:27:52,480
yeah it's doing it's doing auth checks

838
00:27:50,480 --> 00:27:53,919
my memcache is not working and that's

839
00:27:52,480 --> 00:27:54,799
weird

840
00:27:53,919 --> 00:27:56,000
but

841
00:27:54,799 --> 00:27:58,559
hey i guess so if you're gonna have a

842
00:27:56,000 --> 00:28:00,720
demo failure it's good to see an error

843
00:27:58,559 --> 00:28:02,399
that i could see is wrong straight away

844
00:28:00,720 --> 00:28:04,799
um i guess that's the whole point of

845
00:28:02,399 --> 00:28:04,799
tracing

846
00:28:05,600 --> 00:28:08,000
and if that's the only problem we're

847
00:28:06,559 --> 00:28:09,679
going to get then hey that's great in a

848
00:28:08,000 --> 00:28:12,559
demo

849
00:28:09,679 --> 00:28:14,240
uh so we got we're connecting to one two

850
00:28:12,559 --> 00:28:16,240
three four so only if it's connected to

851
00:28:14,240 --> 00:28:17,679
four to be able to rebuild this object

852
00:28:16,240 --> 00:28:18,880
also it doesn't have to update anything

853
00:28:17,679 --> 00:28:20,559
better data lakes we're just getting the

854
00:28:18,880 --> 00:28:22,640
object we're not like adding or deleting

855
00:28:20,559 --> 00:28:23,600
or doing like a

856
00:28:22,640 --> 00:28:25,039
create

857
00:28:23,600 --> 00:28:26,080
remove

858
00:28:25,039 --> 00:28:27,679
operation

859
00:28:26,080 --> 00:28:29,200
okay so that's all great that's all

860
00:28:27,679 --> 00:28:31,120
exciting um

861
00:28:29,200 --> 00:28:33,120
but let's do some more interesting ones

862
00:28:31,120 --> 00:28:34,240
i should probably do a time check uh

863
00:28:33,120 --> 00:28:36,000
yeah it's good

864
00:28:34,240 --> 00:28:38,159
so

865
00:28:36,000 --> 00:28:39,760
let's go back to the slides i don't know

866
00:28:38,159 --> 00:28:42,080
why my that slide didn't come back where

867
00:28:39,760 --> 00:28:42,080
are you

868
00:28:42,480 --> 00:28:46,559
have i lost my slides

869
00:28:45,200 --> 00:28:48,000
not there they are they're just not

870
00:28:46,559 --> 00:28:49,440
listening to me

871
00:28:48,000 --> 00:28:51,440
let's look at some more interesting

872
00:28:49,440 --> 00:28:53,200
features of swift now so

873
00:28:51,440 --> 00:28:55,600
first one let's do a bulk insert so bulk

874
00:28:53,200 --> 00:28:58,799
insert is a interesting

875
00:28:55,600 --> 00:29:00,960
middleware it allows you to or feature i

876
00:28:58,799 --> 00:29:02,720
guess to say what we know implement as a

877
00:29:00,960 --> 00:29:06,000
middleware

878
00:29:02,720 --> 00:29:07,840
it allows you to upload something like

879
00:29:06,000 --> 00:29:09,840
a tar file

880
00:29:07,840 --> 00:29:11,760
excuse me and

881
00:29:09,840 --> 00:29:13,520
it'll extract it

882
00:29:11,760 --> 00:29:15,039
and store all the contents objects

883
00:29:13,520 --> 00:29:17,600
inside your cluster now this is kind of

884
00:29:15,039 --> 00:29:19,440
cool um it's very serial

885
00:29:17,600 --> 00:29:22,159
it's cool because it's kind of cool the

886
00:29:19,440 --> 00:29:24,000
con the the con is it's

887
00:29:22,159 --> 00:29:25,360
it's serial and so if speed's an issue

888
00:29:24,000 --> 00:29:26,559
you wouldn't use this feature but if you

889
00:29:25,360 --> 00:29:28,159
were wanting something to run in the

890
00:29:26,559 --> 00:29:30,320
background and you're just doing it for

891
00:29:28,159 --> 00:29:32,559
like you know archival purposes then

892
00:29:30,320 --> 00:29:35,120
it's kind of cool so let's see what this

893
00:29:32,559 --> 00:29:36,640
looks like in um

894
00:29:35,120 --> 00:29:38,960
while we trace it

895
00:29:36,640 --> 00:29:41,840
so in this directory

896
00:29:38,960 --> 00:29:41,840
i'm just going to type it

897
00:29:41,919 --> 00:29:46,640
i've got a to

898
00:29:43,320 --> 00:29:49,200
extract.gz and it has

899
00:29:46,640 --> 00:29:51,440
a subfolder called extract and it's got

900
00:29:49,200 --> 00:29:53,919
a bunch for four

901
00:29:51,440 --> 00:29:56,399
files in it some text to monorail cat

902
00:29:53,919 --> 00:29:58,399
dyno and whatever what some rather

903
00:29:56,399 --> 00:29:59,600
random random text files because i'm

904
00:29:58,399 --> 00:30:01,520
lazy again

905
00:29:59,600 --> 00:30:04,399
so what we expect to happen is it will

906
00:30:01,520 --> 00:30:05,919
go and create a container called extract

907
00:30:04,399 --> 00:30:08,320
and then inside that container it'll

908
00:30:05,919 --> 00:30:10,480
have all these objects which is kind of

909
00:30:08,320 --> 00:30:13,120
neat if it works

910
00:30:10,480 --> 00:30:13,120
and i have

911
00:30:13,279 --> 00:30:16,960
bulk insert i've got it here if i can

912
00:30:15,360 --> 00:30:19,200
copy it

913
00:30:16,960 --> 00:30:21,120
and i can paste it so here's the command

914
00:30:19,200 --> 00:30:23,200
we'll do it we're saying extract archive

915
00:30:21,120 --> 00:30:25,919
it's a taji zed file so you're saying

916
00:30:23,200 --> 00:30:28,159
here's a taji zed and go do your thing

917
00:30:25,919 --> 00:30:30,080
and i'm using curl instead of any other

918
00:30:28,159 --> 00:30:31,679
things because you can see you can

919
00:30:30,080 --> 00:30:33,279
easily talk to swift

920
00:30:31,679 --> 00:30:34,640
so we can see here that it was

921
00:30:33,279 --> 00:30:37,279
successful

922
00:30:34,640 --> 00:30:38,640
we got a transaction id

923
00:30:37,279 --> 00:30:41,600
um

924
00:30:38,640 --> 00:30:43,440
it created four files

925
00:30:41,600 --> 00:30:47,039
the response code was 201 and there were

926
00:30:43,440 --> 00:30:49,360
no errors so that's great

927
00:30:47,039 --> 00:30:52,080
so if i come back here

928
00:30:49,360 --> 00:30:54,080
there's 352 spans so we know it did a

929
00:30:52,080 --> 00:30:55,840
lot we can see all the things that talk

930
00:30:54,080 --> 00:30:57,840
different servers that talk to they're

931
00:30:55,840 --> 00:30:58,799
all local on this laptop on this virtual

932
00:30:57,840 --> 00:31:00,799
environment

933
00:30:58,799 --> 00:31:03,039
but we can we can see right we can see

934
00:31:00,799 --> 00:31:05,039
that it's one of these should be go and

935
00:31:03,039 --> 00:31:08,559
create that container that extracted

936
00:31:05,039 --> 00:31:12,159
container if i scroll down to it

937
00:31:08,559 --> 00:31:14,399
it's uh put to extract yeah so we're

938
00:31:12,159 --> 00:31:16,559
we're making a uh we're putting the

939
00:31:14,399 --> 00:31:18,399
extract container

940
00:31:16,559 --> 00:31:20,080
that means these guys which one would be

941
00:31:18,399 --> 00:31:22,480
our monorail cut so it would have been

942
00:31:20,080 --> 00:31:24,080
the second one i think right so let's go

943
00:31:22,480 --> 00:31:26,960
look at this one this one in theory

944
00:31:24,080 --> 00:31:28,159
where or if it did it in serially like i

945
00:31:26,960 --> 00:31:30,080
think it does

946
00:31:28,159 --> 00:31:32,399
this should be monorailcat

947
00:31:30,080 --> 00:31:34,000
and it is it's monorailcat and so

948
00:31:32,399 --> 00:31:36,480
you can see from here that it's a pretty

949
00:31:34,000 --> 00:31:40,559
cool feature it uh it will go

950
00:31:36,480 --> 00:31:42,720
it starts extracting it and then starts

951
00:31:40,559 --> 00:31:45,039
making newer kind of sub requests out to

952
00:31:42,720 --> 00:31:46,399
all the clusters and and each of them uh

953
00:31:45,039 --> 00:31:48,799
because

954
00:31:46,399 --> 00:31:51,600
because i just put it into

955
00:31:48,799 --> 00:31:54,480
by default my default policy is

956
00:31:51,600 --> 00:31:56,880
replication is the replicated ring um

957
00:31:54,480 --> 00:31:58,240
it is

958
00:31:56,880 --> 00:32:00,080
only three times replication it's not

959
00:31:58,240 --> 00:32:02,240
erasure coding it you could obviously

960
00:32:00,080 --> 00:32:05,200
point it to a ratio uh you make it

961
00:32:02,240 --> 00:32:07,360
create a razor coded container uh to

962
00:32:05,200 --> 00:32:08,559
store objects in

963
00:32:07,360 --> 00:32:10,240
okay

964
00:32:08,559 --> 00:32:13,120
another time check for me okay we're

965
00:32:10,240 --> 00:32:14,480
getting we're getting toward the end um

966
00:32:13,120 --> 00:32:15,840
there's lots more i could talk about

967
00:32:14,480 --> 00:32:17,120
this for hours there's lots more to talk

968
00:32:15,840 --> 00:32:18,240
i could talk about

969
00:32:17,120 --> 00:32:20,320
but

970
00:32:18,240 --> 00:32:22,159
and you asked the guys at the ptg i just

971
00:32:20,320 --> 00:32:24,000
kept on going

972
00:32:22,159 --> 00:32:25,760
but if we go to the next one what did i

973
00:32:24,000 --> 00:32:26,880
go too far i don't know if i think it's

974
00:32:25,760 --> 00:32:28,559
listening to me

975
00:32:26,880 --> 00:32:30,640
no no i don't have a conclusion go back

976
00:32:28,559 --> 00:32:32,880
there so another interesting feature

977
00:32:30,640 --> 00:32:35,279
from the other side of things is

978
00:32:32,880 --> 00:32:38,720
static large objects so the idea here is

979
00:32:35,279 --> 00:32:41,279
like mtu um multi-part i'd multi-part

980
00:32:38,720 --> 00:32:43,919
things in in aws

981
00:32:41,279 --> 00:32:46,000
so you go you have your object you can

982
00:32:43,919 --> 00:32:47,279
divide it up into what we call segments

983
00:32:46,000 --> 00:32:48,640
and you upload them separately so again

984
00:32:47,279 --> 00:32:50,320
hit different swift proxies you can do

985
00:32:48,640 --> 00:32:52,320
concurrently you can do it quick if it's

986
00:32:50,320 --> 00:32:53,600
a really large item you can cut into

987
00:32:52,320 --> 00:32:55,760
thousands

988
00:32:53,600 --> 00:32:57,279
uh if it's or if you just want if it's a

989
00:32:55,760 --> 00:32:59,120
smaller item but you just want to upload

990
00:32:57,279 --> 00:33:01,440
it quicker then this is the way you

991
00:32:59,120 --> 00:33:04,159
could do it um and then it used to slide

992
00:33:01,440 --> 00:33:07,919
last it'll send a json manifest uh to

993
00:33:04,159 --> 00:33:10,480
kind of enter to stitch it all together

994
00:33:07,919 --> 00:33:11,279
so when you go get that json manifest it

995
00:33:10,480 --> 00:33:12,799
will

996
00:33:11,279 --> 00:33:15,519
it will then be able to pull these and

997
00:33:12,799 --> 00:33:17,840
stitches back together for you um so we

998
00:33:15,519 --> 00:33:19,760
can see that in action now

999
00:33:17,840 --> 00:33:21,679
i'm going to use this this time i'm not

1000
00:33:19,760 --> 00:33:23,600
going to use curl because it's a little

1001
00:33:21,679 --> 00:33:25,120
bit more um

1002
00:33:23,600 --> 00:33:26,480
involved so i'll just let the swift

1003
00:33:25,120 --> 00:33:28,480
client do its thing

1004
00:33:26,480 --> 00:33:30,880
so this command is saying

1005
00:33:28,480 --> 00:33:33,200
swift upload use slo

1006
00:33:30,880 --> 00:33:34,480
uh what's that 5ma about five mega

1007
00:33:33,200 --> 00:33:36,480
chunks

1008
00:33:34,480 --> 00:33:38,000
to my 10 megabyte image which is i think

1009
00:33:36,480 --> 00:33:39,120
i think i did it by you random so

1010
00:33:38,000 --> 00:33:41,360
nothing useful

1011
00:33:39,120 --> 00:33:41,360
um

1012
00:33:41,440 --> 00:33:44,880
and it uploaded three segments i'm

1013
00:33:42,960 --> 00:33:47,760
assuming it's five five-ish five-ish in

1014
00:33:44,880 --> 00:33:50,240
a little bit um and that last so so it's

1015
00:33:47,760 --> 00:33:53,120
made one two three requests and the last

1016
00:33:50,240 --> 00:33:55,360
request should have been inserting

1017
00:33:53,120 --> 00:33:57,519
manifest which has the name

1018
00:33:55,360 --> 00:34:00,559
10 mb dot image because that's what i

1019
00:33:57,519 --> 00:34:04,559
told it i wanted to be called

1020
00:34:00,559 --> 00:34:06,080
so if we go look at this in our trace

1021
00:34:04,559 --> 00:34:07,679
it won't be super interesting because

1022
00:34:06,080 --> 00:34:09,679
it's going to be a bunch of extra new

1023
00:34:07,679 --> 00:34:11,839
requests because it's inserting

1024
00:34:09,679 --> 00:34:13,919
one each time and we've seen inserting

1025
00:34:11,839 --> 00:34:16,800
an object the last one we'll confirm

1026
00:34:13,919 --> 00:34:18,399
that the last one is in fact um actually

1027
00:34:16,800 --> 00:34:21,280
i'll just look at the top here

1028
00:34:18,399 --> 00:34:23,440
is it is in fact uh

1029
00:34:21,280 --> 00:34:25,119
sending in the manifest file

1030
00:34:23,440 --> 00:34:28,000
well it's more this is my favorite bit

1031
00:34:25,119 --> 00:34:30,879
so what's more interesting uh is if we

1032
00:34:28,000 --> 00:34:32,560
try and get that object back out now

1033
00:34:30,879 --> 00:34:35,040
um

1034
00:34:32,560 --> 00:34:36,639
i'll just use the good old cut and paste

1035
00:34:35,040 --> 00:34:39,200
because i don't think you want 10

1036
00:34:36,639 --> 00:34:41,119
megabytes of random crap on my screen

1037
00:34:39,200 --> 00:34:42,560
or on the dev

1038
00:34:41,119 --> 00:34:43,280
so we downloaded it i don't know what

1039
00:34:42,560 --> 00:34:44,320
the

1040
00:34:43,280 --> 00:34:45,679
um

1041
00:34:44,320 --> 00:34:46,800
transaction id was but turns out it

1042
00:34:45,679 --> 00:34:48,399
doesn't matter

1043
00:34:46,800 --> 00:34:51,200
here it is

1044
00:34:48,399 --> 00:34:53,520
it doesn't seem like much but look at it

1045
00:34:51,200 --> 00:34:55,440
it's a bit of a crazy um

1046
00:34:53,520 --> 00:34:57,599
up here is the full

1047
00:34:55,440 --> 00:34:58,720
kind of time separated

1048
00:34:57,599 --> 00:34:59,760
uh

1049
00:34:58,720 --> 00:35:01,920
request

1050
00:34:59,760 --> 00:35:04,560
and it's interesting right like so what

1051
00:35:01,920 --> 00:35:06,000
it's done is it's gone to do a get of

1052
00:35:04,560 --> 00:35:08,560
the manifest this should be a get of the

1053
00:35:06,000 --> 00:35:10,720
manifest file

1054
00:35:08,560 --> 00:35:13,359
yep good i'm good i'm i do know my stuff

1055
00:35:10,720 --> 00:35:15,280
that's good um and then it's starting to

1056
00:35:13,359 --> 00:35:17,119
build so what so what whiskey does let

1057
00:35:15,280 --> 00:35:18,560
me see the time check again cool so what

1058
00:35:17,119 --> 00:35:20,640
whisky does

1059
00:35:18,560 --> 00:35:22,480
is it

1060
00:35:20,640 --> 00:35:25,680
will send back

1061
00:35:22,480 --> 00:35:27,599
um the response headers and stuff and it

1062
00:35:25,680 --> 00:35:29,599
will wrap up the body in a file like

1063
00:35:27,599 --> 00:35:32,560
iterator and

1064
00:35:29,599 --> 00:35:35,359
so it will send back

1065
00:35:32,560 --> 00:35:38,079
uh all that to the client in this case

1066
00:35:35,359 --> 00:35:40,480
curl um and then curl will start you

1067
00:35:38,079 --> 00:35:41,599
know reading the body uh to the body of

1068
00:35:40,480 --> 00:35:44,560
the request

1069
00:35:41,599 --> 00:35:46,640
so we can see it is

1070
00:35:44,560 --> 00:35:47,680
getting that manifest file at this point

1071
00:35:46,640 --> 00:35:49,200
it's actually probably going to go get

1072
00:35:47,680 --> 00:35:50,800
that first segment

1073
00:35:49,200 --> 00:35:53,680
just getting that iterator ready the

1074
00:35:50,800 --> 00:35:55,760
file like iterator ready

1075
00:35:53,680 --> 00:35:57,040
yep it's getting segment zero i'm

1076
00:35:55,760 --> 00:35:59,040
pointing at a screen that you can't see

1077
00:35:57,040 --> 00:36:00,480
me point at use the mouse mat use the

1078
00:35:59,040 --> 00:36:02,800
mouse um

1079
00:36:00,480 --> 00:36:03,839
zero first one zero based because it's

1080
00:36:02,800 --> 00:36:06,079
programming

1081
00:36:03,839 --> 00:36:08,400
because it's i t

1082
00:36:06,079 --> 00:36:10,800
and then there's this big gaps this gap

1083
00:36:08,400 --> 00:36:13,040
is caused by

1084
00:36:10,800 --> 00:36:16,000
curl so as curl reads

1085
00:36:13,040 --> 00:36:18,079
um it's it as it's dealt with the first

1086
00:36:16,000 --> 00:36:19,520
segment it reads and then gets a second

1087
00:36:18,079 --> 00:36:20,880
segment and we can actually see this

1088
00:36:19,520 --> 00:36:22,079
happening and it's kind of really cool

1089
00:36:20,880 --> 00:36:23,280
well i think it's really cool i get

1090
00:36:22,079 --> 00:36:24,320
excited about this stuff i don't know if

1091
00:36:23,280 --> 00:36:25,280
you guys do

1092
00:36:24,320 --> 00:36:27,359
um

1093
00:36:25,280 --> 00:36:30,000
i definitely do

1094
00:36:27,359 --> 00:36:32,320
let me i can do other things let me see

1095
00:36:30,000 --> 00:36:35,040
how time goes i think we've probably

1096
00:36:32,320 --> 00:36:36,480
only got what five minutes

1097
00:36:35,040 --> 00:36:37,599
before i want to try and stop for

1098
00:36:36,480 --> 00:36:39,200
questions

1099
00:36:37,599 --> 00:36:40,480
so

1100
00:36:39,200 --> 00:36:43,040
let me

1101
00:36:40,480 --> 00:36:44,880
finish the last few slides of the

1102
00:36:43,040 --> 00:36:47,359
presentation and if we want and if

1103
00:36:44,880 --> 00:36:48,800
anyone's interested i can go because

1104
00:36:47,359 --> 00:36:51,119
i've shown you all the happy path things

1105
00:36:48,800 --> 00:36:54,240
right we can also go in and like go kill

1106
00:36:51,119 --> 00:36:55,440
a server and then do requests and see

1107
00:36:54,240 --> 00:36:56,880
what the traces look like this is all

1108
00:36:55,440 --> 00:36:58,800
the fun stuff because when

1109
00:36:56,880 --> 00:37:00,000
just be honest when you're like doing a

1110
00:36:58,800 --> 00:37:01,599
trace request you're not tracing the

1111
00:37:00,000 --> 00:37:03,200
happy path you tend to be tracing

1112
00:37:01,599 --> 00:37:04,960
because an issue and we can try and

1113
00:37:03,200 --> 00:37:07,760
demonstrate that stuff

1114
00:37:04,960 --> 00:37:08,880
only more than 45 minutes

1115
00:37:07,760 --> 00:37:10,160
so

1116
00:37:08,880 --> 00:37:13,119
conclusion

1117
00:37:10,160 --> 00:37:16,560
i wrote this up about an hour ago so

1118
00:37:13,119 --> 00:37:19,760
it's a bit more verbose um

1119
00:37:16,560 --> 00:37:22,560
being able to visualize a request

1120
00:37:19,760 --> 00:37:24,000
at least to me has been amazing like in

1121
00:37:22,560 --> 00:37:25,599
my i've been involved in swift for a

1122
00:37:24,000 --> 00:37:27,200
long time i know how the code works and

1123
00:37:25,599 --> 00:37:28,960
you know build up my mental model i know

1124
00:37:27,200 --> 00:37:30,800
what's happening but to see it's like

1125
00:37:28,960 --> 00:37:32,320
seeing your baby or maybe not that far

1126
00:37:30,800 --> 00:37:33,839
but it's like

1127
00:37:32,320 --> 00:37:35,440
it's being able to see it it's actually

1128
00:37:33,839 --> 00:37:37,280
been really exciting for me uh if you

1129
00:37:35,440 --> 00:37:39,040
can't tell um

1130
00:37:37,280 --> 00:37:40,720
and i i also think that the human

1131
00:37:39,040 --> 00:37:42,560
ability to patent match means we tend to

1132
00:37:40,720 --> 00:37:45,040
spot potentially quickly

1133
00:37:42,560 --> 00:37:46,800
i i guess when i saw it before right i

1134
00:37:45,040 --> 00:37:48,160
saw that the things are not being stored

1135
00:37:46,800 --> 00:37:49,440
in memcache like they're supposed to be

1136
00:37:48,160 --> 00:37:50,800
so

1137
00:37:49,440 --> 00:37:53,920
we saw an issue

1138
00:37:50,800 --> 00:37:56,560
um but that only comes down to you need

1139
00:37:53,920 --> 00:37:58,000
to know what normal looks like so

1140
00:37:56,560 --> 00:37:59,599
there is going to be a level of having

1141
00:37:58,000 --> 00:38:01,520
to train and this is where our sres are

1142
00:37:59,599 --> 00:38:03,040
at in nvidia now that we've got some dev

1143
00:38:01,520 --> 00:38:05,040
environments we haven't rolled out to

1144
00:38:03,040 --> 00:38:06,640
full production yet um it's more in some

1145
00:38:05,040 --> 00:38:08,079
of our staging environments so we can

1146
00:38:06,640 --> 00:38:10,640
get an idea of what these things look

1147
00:38:08,079 --> 00:38:13,680
like and and get it integrated into our

1148
00:38:10,640 --> 00:38:16,000
um into our

1149
00:38:13,680 --> 00:38:20,079
manner what's the word

1150
00:38:16,000 --> 00:38:20,079
i can't think of it our pro stack our

1151
00:38:20,480 --> 00:38:23,920
monitoring there we go that's what i'm

1152
00:38:21,760 --> 00:38:24,800
looking for our monitoring stack

1153
00:38:23,920 --> 00:38:26,640
um

1154
00:38:24,800 --> 00:38:27,599
but so it first is learning what normal

1155
00:38:26,640 --> 00:38:29,119
looks like because if you don't know

1156
00:38:27,599 --> 00:38:32,000
what normal looks like then how you know

1157
00:38:29,119 --> 00:38:33,920
the pattern match doesn't work

1158
00:38:32,000 --> 00:38:36,320
can we see we can see what happened in

1159
00:38:33,920 --> 00:38:39,359
request rather than hypothesize maybe

1160
00:38:36,320 --> 00:38:40,240
maybe you guys are good at an sres and

1161
00:38:39,359 --> 00:38:42,400
you

1162
00:38:40,240 --> 00:38:43,680
sis ads out there are amazing at being

1163
00:38:42,400 --> 00:38:44,560
able to figure out what happened from

1164
00:38:43,680 --> 00:38:47,280
logs

1165
00:38:44,560 --> 00:38:49,359
and i struggle sometimes but being able

1166
00:38:47,280 --> 00:38:50,800
to actually look at what happened at the

1167
00:38:49,359 --> 00:38:52,240
time and being able to pull in

1168
00:38:50,800 --> 00:38:54,000
information and yes we're not pulling

1169
00:38:52,240 --> 00:38:55,680
all the information yet but there's the

1170
00:38:54,000 --> 00:38:57,440
code there and in that mix in there we

1171
00:38:55,680 --> 00:38:58,960
can continue to instrument the parts

1172
00:38:57,440 --> 00:39:01,760
that we need to to pull out the

1173
00:38:58,960 --> 00:39:03,200
information we need um

1174
00:39:01,760 --> 00:39:04,880
and just a nice have another way of

1175
00:39:03,200 --> 00:39:06,320
pulling out information uh get back to

1176
00:39:04,880 --> 00:39:07,920
those error limited nodes in the swift

1177
00:39:06,320 --> 00:39:10,079
proxy server

1178
00:39:07,920 --> 00:39:11,760
we do dump those to logging if you if

1179
00:39:10,079 --> 00:39:14,000
you turn your debugging up to debug but

1180
00:39:11,760 --> 00:39:16,160
in production we don't want to do that

1181
00:39:14,000 --> 00:39:18,400
especially on really really really large

1182
00:39:16,160 --> 00:39:20,000
clusters um

1183
00:39:18,400 --> 00:39:21,440
so being able to have a way of pulling

1184
00:39:20,000 --> 00:39:22,880
that data out it's kind of exciting and

1185
00:39:21,440 --> 00:39:24,160
interesting

1186
00:39:22,880 --> 00:39:25,520
there are a few interesting problems

1187
00:39:24,160 --> 00:39:27,359
that we had

1188
00:39:25,520 --> 00:39:30,000
along the way

1189
00:39:27,359 --> 00:39:31,680
um this is maybe more for people who are

1190
00:39:30,000 --> 00:39:33,599
path and developers who have had to deal

1191
00:39:31,680 --> 00:39:35,839
with eventlet and green threads and

1192
00:39:33,599 --> 00:39:37,359
stuff but weirdly i told you that the

1193
00:39:35,839 --> 00:39:40,560
the transat that

1194
00:39:37,359 --> 00:39:42,240
come on brain the um tracer

1195
00:39:40,560 --> 00:39:44,960
was in charge of keeping track of the

1196
00:39:42,240 --> 00:39:47,359
active span so anyway what i'd find was

1197
00:39:44,960 --> 00:39:50,640
once we get into an event like um thread

1198
00:39:47,359 --> 00:39:52,640
pool and now at the other side so it's

1199
00:39:50,640 --> 00:39:54,640
in an instant threading environment

1200
00:39:52,640 --> 00:39:56,480
it just had lost its um

1201
00:39:54,640 --> 00:39:58,079
active span from inside the tracer so we

1202
00:39:56,480 --> 00:39:58,960
had to like figure out ways of pinning

1203
00:39:58,079 --> 00:40:00,160
that in

1204
00:39:58,960 --> 00:40:01,200
um because

1205
00:40:00,160 --> 00:40:02,720
you know

1206
00:40:01,200 --> 00:40:05,200
suddenly tracing doesn't work and that's

1207
00:40:02,720 --> 00:40:07,839
not really useful um response iterators

1208
00:40:05,200 --> 00:40:10,319
that last one you saw with the slo

1209
00:40:07,839 --> 00:40:12,960
static clutch object getting what also

1210
00:40:10,319 --> 00:40:14,079
tends to happen is because whiskey kind

1211
00:40:12,960 --> 00:40:16,480
of

1212
00:40:14,079 --> 00:40:19,599
sends back the response with that with

1213
00:40:16,480 --> 00:40:21,440
that file like iterator it decides that

1214
00:40:19,599 --> 00:40:23,359
it kind of cleans up all the context all

1215
00:40:21,440 --> 00:40:26,720
the span contexts and stuff and so they

1216
00:40:23,359 --> 00:40:28,240
don't try and exist when we're um

1217
00:40:26,720 --> 00:40:30,640
when we're still

1218
00:40:28,240 --> 00:40:32,560
making new calls to the next segment for

1219
00:40:30,640 --> 00:40:34,560
example and so the same same kind of

1220
00:40:32,560 --> 00:40:36,560
solution to the green thread turns

1221
00:40:34,560 --> 00:40:38,000
eventlet seemed to here too it's just

1222
00:40:36,560 --> 00:40:39,200
kind of keeping those things alive

1223
00:40:38,000 --> 00:40:40,319
making sure we've got them pinned so

1224
00:40:39,200 --> 00:40:41,440
they don't just disappear when their

1225
00:40:40,319 --> 00:40:43,200
context

1226
00:40:41,440 --> 00:40:45,839
uh ends

1227
00:40:43,200 --> 00:40:48,160
um well by default um

1228
00:40:45,839 --> 00:40:49,839
and yeah because the other thing is

1229
00:40:48,160 --> 00:40:52,640
request tracing has been interesting

1230
00:40:49,839 --> 00:40:54,880
it's um

1231
00:40:52,640 --> 00:40:57,200
whiskey tends to have a lot of these

1232
00:40:54,880 --> 00:40:59,200
middlewares in memory and our request

1233
00:40:57,200 --> 00:41:01,359
coming through is really like is really

1234
00:40:59,200 --> 00:41:02,640
an event dict dictionary that kind of

1235
00:41:01,359 --> 00:41:04,880
works its way around environment

1236
00:41:02,640 --> 00:41:07,520
dictionary and because we're trying to

1237
00:41:04,880 --> 00:41:08,640
keep the scope to of tracing to a single

1238
00:41:07,520 --> 00:41:10,960
request

1239
00:41:08,640 --> 00:41:12,720
it means the kind of tracing mechanics

1240
00:41:10,960 --> 00:41:14,720
had to to some extent while the state of

1241
00:41:12,720 --> 00:41:16,160
them had to live inside these dicks and

1242
00:41:14,720 --> 00:41:17,839
that there's like

1243
00:41:16,160 --> 00:41:18,880
it's just interesting it going to

1244
00:41:17,839 --> 00:41:21,359
interesting problems because we didn't

1245
00:41:18,880 --> 00:41:23,040
want issues to bleed into other issues

1246
00:41:21,359 --> 00:41:25,040
um anyway i think i

1247
00:41:23,040 --> 00:41:26,960
am running out of time so i'll end this

1248
00:41:25,040 --> 00:41:28,800
here well actually i'll uh give you

1249
00:41:26,960 --> 00:41:31,920
these things if you want to contact me

1250
00:41:28,800 --> 00:41:33,119
uh email me at nvidia at my thing there

1251
00:41:31,920 --> 00:41:35,760
or at

1252
00:41:33,119 --> 00:41:37,599
uh oliver.edu again i'm on matt oliver

1253
00:41:35,760 --> 00:41:39,040
you on all the socials if you want to

1254
00:41:37,599 --> 00:41:41,119
come check out the swift project we all

1255
00:41:39,040 --> 00:41:42,960
hang out on ifc

1256
00:41:41,119 --> 00:41:43,920
not not quite the cool kids anymore i

1257
00:41:42,960 --> 00:41:45,920
think

1258
00:41:43,920 --> 00:41:47,520
but anyway we're on openstack swift on

1259
00:41:45,920 --> 00:41:49,119
oftc

1260
00:41:47,520 --> 00:41:50,720
there's our project website

1261
00:41:49,119 --> 00:41:52,400
swift.openstack.org

1262
00:41:50,720 --> 00:41:53,520
there is a link to the contributing

1263
00:41:52,400 --> 00:41:54,240
guide you should come and have fun with

1264
00:41:53,520 --> 00:41:56,079
us

1265
00:41:54,240 --> 00:41:58,079
um and there's some interesting links

1266
00:41:56,079 --> 00:42:00,240
there's the link to this code in garrett

1267
00:41:58,079 --> 00:42:01,040
as it stands and that google doc that

1268
00:42:00,240 --> 00:42:02,960
will

1269
00:42:01,040 --> 00:42:04,400
get you bootstrapped into doing exactly

1270
00:42:02,960 --> 00:42:06,319
what my system's doing right here if you

1271
00:42:04,400 --> 00:42:07,839
don't have a play

1272
00:42:06,319 --> 00:42:09,920
i'm really looking forward to the google

1273
00:42:07,839 --> 00:42:12,000
doc link in

1274
00:42:09,920 --> 00:42:13,440
oh yeah well i'll share this yeah yeah

1275
00:42:12,000 --> 00:42:15,280
it's not great maybe i should have like

1276
00:42:13,440 --> 00:42:17,040
shortened it but you know i don't think

1277
00:42:15,280 --> 00:42:19,440
about that until you can put the slides

1278
00:42:17,040 --> 00:42:21,760
into the text chat afterwards and okay

1279
00:42:19,440 --> 00:42:23,359
yes i'll do that i'll um i'll do

1280
00:42:21,760 --> 00:42:25,520
definitely do that

1281
00:42:23,359 --> 00:42:27,280
uh so that's it any questions

1282
00:42:25,520 --> 00:42:28,640
actually what time do we have we do have

1283
00:42:27,280 --> 00:42:30,960
a few questions we've got

1284
00:42:28,640 --> 00:42:32,240
two and a half minutes to talk right

1285
00:42:30,960 --> 00:42:33,359
um

1286
00:42:32,240 --> 00:42:35,119
yeah

1287
00:42:33,359 --> 00:42:36,720
so the first question that we've got is

1288
00:42:35,119 --> 00:42:38,160
open tracing is only application

1289
00:42:36,720 --> 00:42:40,560
specific does it use any tracing

1290
00:42:38,160 --> 00:42:42,160
mechanism from linux kernel

1291
00:42:40,560 --> 00:42:42,960
no it's just application specific it's

1292
00:42:42,160 --> 00:42:45,599
just

1293
00:42:42,960 --> 00:42:47,599
the idea is it's just an api that would

1294
00:42:45,599 --> 00:42:50,079
be cool though actually like if we could

1295
00:42:47,599 --> 00:42:51,599
do some tracing down the kernel level

1296
00:42:50,079 --> 00:42:53,280
i would like that too because there's

1297
00:42:51,599 --> 00:42:56,400
sometimes the way you know some of our

1298
00:42:53,280 --> 00:42:58,720
swift issues are not always

1299
00:42:56,400 --> 00:43:00,640
application specific right when we in

1300
00:42:58,720 --> 00:43:02,400
nvidia are looking at

1301
00:43:00,640 --> 00:43:04,160
how can we speed up transfers how can we

1302
00:43:02,400 --> 00:43:06,079
talk directly to disks can we do

1303
00:43:04,160 --> 00:43:07,760
drmaying and all this kind of fun stuff

1304
00:43:06,079 --> 00:43:09,599
so being able to kind of get under the

1305
00:43:07,760 --> 00:43:11,359
hood a bit more would be great but no

1306
00:43:09,599 --> 00:43:14,000
this is at the moment at least as far as

1307
00:43:11,359 --> 00:43:15,359
i'm aware it's um it's very application

1308
00:43:14,000 --> 00:43:17,839
specific

1309
00:43:15,359 --> 00:43:19,440
cool uh another question we have is what

1310
00:43:17,839 --> 00:43:21,839
overhead have you seen having tracing

1311
00:43:19,440 --> 00:43:23,839
going would you leave it on

1312
00:43:21,839 --> 00:43:25,359
uh this is great um because this is the

1313
00:43:23,839 --> 00:43:28,000
question i would have asked myself so

1314
00:43:25,359 --> 00:43:29,520
here we go basic benchmarks now you take

1315
00:43:28,000 --> 00:43:31,760
this with a grain of salt because this

1316
00:43:29,520 --> 00:43:34,240
is just on this death machine and i did

1317
00:43:31,760 --> 00:43:37,359
it during lca right so because that's

1318
00:43:34,240 --> 00:43:39,599
the way we do things in lca um so it's

1319
00:43:37,359 --> 00:43:42,880
very basic this is pretty much just a

1320
00:43:39,599 --> 00:43:45,040
thousand get puts and deletes it's

1321
00:43:42,880 --> 00:43:46,960
um just one k

1322
00:43:45,040 --> 00:43:49,599
object so take of the grain of salt but

1323
00:43:46,960 --> 00:43:51,920
the whole point is a similar environment

1324
00:43:49,599 --> 00:43:54,240
one there's no tracing no tracing at all

1325
00:43:51,920 --> 00:43:55,680
no code this it's like master without

1326
00:43:54,240 --> 00:43:58,640
this patch applied then we've got

1327
00:43:55,680 --> 00:44:01,040
tracing off that is the red one it is

1328
00:43:58,640 --> 00:44:03,359
got no tr it's got all the tracing code

1329
00:44:01,040 --> 00:44:05,920
enabled but it does not have not tracing

1330
00:44:03,359 --> 00:44:07,760
anything and tracing on is not how you

1331
00:44:05,920 --> 00:44:09,280
would ever run this thing it's tracing

1332
00:44:07,760 --> 00:44:10,880
every single request but you know we

1333
00:44:09,280 --> 00:44:12,480
want to test this stuff right and so

1334
00:44:10,880 --> 00:44:15,200
tracing on is the code with tracing

1335
00:44:12,480 --> 00:44:17,839
turned on every request as expected when

1336
00:44:15,200 --> 00:44:19,200
you got tracing on it gives us a

1337
00:44:17,839 --> 00:44:20,000
performance here

1338
00:44:19,200 --> 00:44:21,839
but

1339
00:44:20,000 --> 00:44:24,319
it seems at least in this small one and

1340
00:44:21,839 --> 00:44:25,920
we will be testing this um

1341
00:44:24,319 --> 00:44:27,280
as we before we roll it in production

1342
00:44:25,920 --> 00:44:29,119
too as we like test and i have bigger

1343
00:44:27,280 --> 00:44:31,440
bigger staging systems just to confirm

1344
00:44:29,119 --> 00:44:33,200
but in all my tests so far it it has

1345
00:44:31,440 --> 00:44:34,880
been negligible in a lot of the coding

1346
00:44:33,200 --> 00:44:37,599
if you have a look at the code

1347
00:44:34,880 --> 00:44:39,520
i've gone to great f efforts to try and

1348
00:44:37,599 --> 00:44:42,079
make any of the tracing up stuff if it's

1349
00:44:39,520 --> 00:44:44,960
not tracing as no op as possible

1350
00:44:42,079 --> 00:44:45,920
um so i didn't want to affect our spin

1351
00:44:44,960 --> 00:44:47,440
so

1352
00:44:45,920 --> 00:44:49,520
our latency distribution so we've got a

1353
00:44:47,440 --> 00:44:52,000
thousand puts we got a thousand uh

1354
00:44:49,520 --> 00:44:53,200
support yeah a thousand puts um and most

1355
00:44:52,000 --> 00:44:56,400
of them

1356
00:44:53,200 --> 00:44:58,319
were identical to no tracing um there's

1357
00:44:56,400 --> 00:44:59,839
slight differences as we can see so it

1358
00:44:58,319 --> 00:45:02,960
does have a does have it will have a

1359
00:44:59,839 --> 00:45:04,400
marginal effect as you'd expect um the

1360
00:45:02,960 --> 00:45:06,240
ops per second

1361
00:45:04,400 --> 00:45:08,160
we can see that

1362
00:45:06,240 --> 00:45:10,480
tracing on well yeah it's never going to

1363
00:45:08,160 --> 00:45:12,960
be great right um but tracing off for no

1364
00:45:10,480 --> 00:45:15,280
tracing of fairly similar i'd like to do

1365
00:45:12,960 --> 00:45:16,319
a bigger scale of this but thanks to

1366
00:45:15,280 --> 00:45:18,000
have asked that question because it's

1367
00:45:16,319 --> 00:45:21,280
exactly the question that i was thinking

1368
00:45:18,000 --> 00:45:21,280
which is why i preempted

1369
00:45:22,400 --> 00:45:25,040
does that answer that question well the

1370
00:45:23,920 --> 00:45:28,040
best i can answer that question right

1371
00:45:25,040 --> 00:45:28,040
now

1372
00:45:36,160 --> 00:45:38,960
is there anything else

1373
00:45:39,520 --> 00:45:42,319
uh you're muted

1374
00:45:44,160 --> 00:45:47,839
i'm muted i'm so sorry

1375
00:45:48,960 --> 00:45:53,040
uh i've got a couple more questions that

1376
00:45:50,800 --> 00:45:55,280
i'll jump into the text chat uh in the

1377
00:45:53,040 --> 00:45:58,240
press talk chat for kia ora

1378
00:45:55,280 --> 00:45:59,839
and then i will uh

1379
00:45:58,240 --> 00:46:01,920
yeah we've got afternoon tea now thank

1380
00:45:59,839 --> 00:46:02,720
you so much for your talk matthew

1381
00:46:01,920 --> 00:46:04,480
um

1382
00:46:02,720 --> 00:46:06,800
green great to have you

1383
00:46:04,480 --> 00:46:10,240
and we'll have see everyone back at 3 45

1384
00:46:06,800 --> 00:46:12,400
pm aedt for sam bishop with boldly going

1385
00:46:10,240 --> 00:46:13,680
running linux in space

1386
00:46:12,400 --> 00:46:18,520
awesome

1387
00:46:13,680 --> 00:46:18,520
thanks everyone i'll be in the chat