1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:15,679 --> 00:00:18,720
hello everyone and welcome back to the

3
00:00:17,199 --> 00:00:21,760
kia ora theater

4
00:00:18,720 --> 00:00:24,240
uh at linux conference australia 2022 uh

5
00:00:21,760 --> 00:00:25,840
next up we've got ian weinend uh ian has

6
00:00:24,240 --> 00:00:27,599
been involved with the open with

7
00:00:25,840 --> 00:00:29,279
openstack in the open devca laboratory

8
00:00:27,599 --> 00:00:31,199
for several years as an employee of red

9
00:00:29,279 --> 00:00:32,880
hat and has admin privileges for these

10
00:00:31,199 --> 00:00:35,360
projects and commit privileges for the

11
00:00:32,880 --> 00:00:37,920
broader ci ecosystem of openstack's

12
00:00:35,360 --> 00:00:39,040
devstack project and the zul prodigating

13
00:00:37,920 --> 00:00:41,040
system

14
00:00:39,040 --> 00:00:43,280
uh today he's here to talk about opendev

15
00:00:41,040 --> 00:00:44,800
a collaboratory for open source software

16
00:00:43,280 --> 00:00:46,320
development at scale and how its

17
00:00:44,800 --> 00:00:48,160
infrastructure components come together

18
00:00:46,320 --> 00:00:50,000
to run the services used by thousands of

19
00:00:48,160 --> 00:00:52,000
developers to develop key parts of the

20
00:00:50,000 --> 00:00:53,520
open source ecosystem

21
00:00:52,000 --> 00:00:54,800
ian is taking questions today so make

22
00:00:53,520 --> 00:00:56,840
sure you put them into the questions tab

23
00:00:54,800 --> 00:00:58,719
and venulis throughout

24
00:00:56,840 --> 00:01:00,640
already

25
00:00:58,719 --> 00:01:01,840
uh hello thank you for having me here

26
00:01:00,640 --> 00:01:03,359
today

27
00:01:01,840 --> 00:01:05,519
um

28
00:01:03,359 --> 00:01:06,799
will you be interested in this talk uh

29
00:01:05,519 --> 00:01:09,119
we'll be talking about sort of

30
00:01:06,799 --> 00:01:10,159
continuous integration speculative

31
00:01:09,119 --> 00:01:12,240
testing

32
00:01:10,159 --> 00:01:16,320
things that fall under the category of

33
00:01:12,240 --> 00:01:19,520
continuous delivery cd devops git ops

34
00:01:16,320 --> 00:01:20,320
uh zuul the cicd system

35
00:01:19,520 --> 00:01:22,560
uh

36
00:01:20,320 --> 00:01:23,600
not the netflix one not the ghostbusters

37
00:01:22,560 --> 00:01:25,520
one

38
00:01:23,600 --> 00:01:28,400
um

39
00:01:25,520 --> 00:01:30,159
merge trains pipelines things like that

40
00:01:28,400 --> 00:01:31,920
this will come together with a bit of

41
00:01:30,159 --> 00:01:33,360
ansible containers

42
00:01:31,920 --> 00:01:35,280
and

43
00:01:33,360 --> 00:01:37,119
we're working in a gitish environment so

44
00:01:35,280 --> 00:01:38,799
if you use any of these sort of gitish

45
00:01:37,119 --> 00:01:40,320
code review systems

46
00:01:38,799 --> 00:01:41,520
you might find something in here for you

47
00:01:40,320 --> 00:01:42,799
too

48
00:01:41,520 --> 00:01:44,799
uh

49
00:01:42,799 --> 00:01:46,479
let me pick off the last bit of the

50
00:01:44,799 --> 00:01:49,040
title of the talk first the open dev

51
00:01:46,479 --> 00:01:50,880
collaboratory what what is that

52
00:01:49,040 --> 00:01:52,479
uh many of you will be

53
00:01:50,880 --> 00:01:55,119
uh more familiar with the openstack

54
00:01:52,479 --> 00:01:57,200
project openstack's been going for

55
00:01:55,119 --> 00:01:59,360
12 years something like that it's i mean

56
00:01:57,200 --> 00:02:01,439
it's basically the way to run a sort of

57
00:01:59,360 --> 00:02:02,880
traditional cloud of virtual machines on

58
00:02:01,439 --> 00:02:03,680
top of your hardware

59
00:02:02,880 --> 00:02:05,280
uh

60
00:02:03,680 --> 00:02:06,880
open source of course

61
00:02:05,280 --> 00:02:08,399
um

62
00:02:06,880 --> 00:02:10,000
what what tended to happen with

63
00:02:08,399 --> 00:02:10,879
openstack though was over all these

64
00:02:10,000 --> 00:02:12,720
years

65
00:02:10,879 --> 00:02:14,959
uh collected a lot of projects that

66
00:02:12,720 --> 00:02:17,280
weren't sort of related to openstack

67
00:02:14,959 --> 00:02:20,319
so every problem in computer science can

68
00:02:17,280 --> 00:02:21,920
be fixed with the level of abstraction

69
00:02:20,319 --> 00:02:23,680
we started out what's called the opening

70
00:02:21,920 --> 00:02:25,680
for a foundation and split these

71
00:02:23,680 --> 00:02:27,440
separate projects out so openstack is

72
00:02:25,680 --> 00:02:30,400
now one of these projects under the

73
00:02:27,440 --> 00:02:32,080
opening for a foundation

74
00:02:30,400 --> 00:02:34,879
so what's open dev

75
00:02:32,080 --> 00:02:37,120
opendev is the uh sort of section of

76
00:02:34,879 --> 00:02:39,440
this that provides the

77
00:02:37,120 --> 00:02:42,239
infrastructure services for all these

78
00:02:39,440 --> 00:02:44,560
projects including now openstack

79
00:02:42,239 --> 00:02:47,360
that's the code hosting

80
00:02:44,560 --> 00:02:49,840
code review system garrett all the ci

81
00:02:47,360 --> 00:02:52,400
systems all the collaboration systems

82
00:02:49,840 --> 00:02:53,519
mailing lists esa pads messaging all

83
00:02:52,400 --> 00:02:57,840
that sort of stuff all of that

84
00:02:53,519 --> 00:03:00,400
infrastructure is provided by opendev

85
00:02:57,840 --> 00:03:02,480
to support these projects

86
00:03:00,400 --> 00:03:03,360
uh the way that we work

87
00:03:02,480 --> 00:03:06,159
uh

88
00:03:03,360 --> 00:03:09,040
opendev provides all these services

89
00:03:06,159 --> 00:03:10,879
to these projects which build these

90
00:03:09,040 --> 00:03:13,280
um

91
00:03:10,879 --> 00:03:15,360
uh products such as openstack and cadder

92
00:03:13,280 --> 00:03:18,480
and zul and airship

93
00:03:15,360 --> 00:03:21,440
these are then consumed by a wide range

94
00:03:18,480 --> 00:03:23,840
of companies many of these companies

95
00:03:21,440 --> 00:03:25,200
then contribute back to us

96
00:03:23,840 --> 00:03:27,920
so

97
00:03:25,200 --> 00:03:30,640
vex host for example and rackspace

98
00:03:27,920 --> 00:03:32,640
provide us with a lot of our hosting

99
00:03:30,640 --> 00:03:34,959
for hosting our services

100
00:03:32,640 --> 00:03:38,159
uh lenaro provides us with a lot of arm

101
00:03:34,959 --> 00:03:40,720
64 resources in various ways uh other

102
00:03:38,159 --> 00:03:43,519
providers provide us with uh

103
00:03:40,720 --> 00:03:46,879
hundreds of ci nodes that we used for

104
00:03:43,519 --> 00:03:48,400
running ci for the projects

105
00:03:46,879 --> 00:03:50,640
and all these companies provide

106
00:03:48,400 --> 00:03:52,879
developer time as well so

107
00:03:50,640 --> 00:03:54,720
developers that help out and help run

108
00:03:52,879 --> 00:03:56,400
the infrastructure

109
00:03:54,720 --> 00:03:57,760
we make the infrastructure better the

110
00:03:56,400 --> 00:04:00,560
projects make themselves better the

111
00:03:57,760 --> 00:04:03,040
companies get better it also works in a

112
00:04:00,560 --> 00:04:04,560
in a nice circle

113
00:04:03,040 --> 00:04:07,040
so that's what i that's what the open

114
00:04:04,560 --> 00:04:08,799
dev collaboratory is and that's

115
00:04:07,040 --> 00:04:10,879
the the services that i'm talking about

116
00:04:08,799 --> 00:04:13,760
today are the services provided by the

117
00:04:10,879 --> 00:04:15,760
open dev collaboratory

118
00:04:13,760 --> 00:04:17,840
open infrastructure what do i mean by

119
00:04:15,760 --> 00:04:19,600
open infrastructure

120
00:04:17,840 --> 00:04:21,600
uh well

121
00:04:19,600 --> 00:04:25,040
our goal is to build on open source

122
00:04:21,600 --> 00:04:26,880
technology from top to bottom that's

123
00:04:25,040 --> 00:04:28,240
that's not everybody's goal and we're

124
00:04:26,880 --> 00:04:30,320
we're sort of privileged to have this

125
00:04:28,240 --> 00:04:33,759
goal that we only use open source

126
00:04:30,320 --> 00:04:33,759
technology in our stack

127
00:04:34,320 --> 00:04:38,000
what we want with our infrastructure is

128
00:04:36,240 --> 00:04:40,080
for anybody to be able to turn up and

129
00:04:38,000 --> 00:04:42,720
contribute to our infrastructure

130
00:04:40,080 --> 00:04:45,360
without any explicit permission or

131
00:04:42,720 --> 00:04:47,840
having to join the admin group or having

132
00:04:45,360 --> 00:04:50,320
to be part of a special group that can

133
00:04:47,840 --> 00:04:51,919
can see the infrastructure

134
00:04:50,320 --> 00:04:54,000
sort of having your infrastructure out

135
00:04:51,919 --> 00:04:55,680
in the open isn't

136
00:04:54,000 --> 00:04:58,160
isn't really that common even in open

137
00:04:55,680 --> 00:05:00,639
source projects you know

138
00:04:58,160 --> 00:05:02,800
it's sort of common to maybe do this at

139
00:05:00,639 --> 00:05:04,560
a project level where maybe

140
00:05:02,800 --> 00:05:06,800
you know you have the you have the

141
00:05:04,560 --> 00:05:08,960
website checked in to get and people can

142
00:05:06,800 --> 00:05:10,800
update the website and get and

143
00:05:08,960 --> 00:05:12,639
you push a change and then you know a

144
00:05:10,800 --> 00:05:14,080
little bit later a con job pulls it in

145
00:05:12,639 --> 00:05:16,400
and updates the web page or something

146
00:05:14,080 --> 00:05:18,000
like that but all of the infrastructure

147
00:05:16,400 --> 00:05:19,759
behind that all the

148
00:05:18,000 --> 00:05:21,440
um you know

149
00:05:19,759 --> 00:05:22,479
the web services and everything behind

150
00:05:21,440 --> 00:05:24,160
that

151
00:05:22,479 --> 00:05:26,240
these aren't often sort of exposed

152
00:05:24,160 --> 00:05:27,759
publicly and

153
00:05:26,240 --> 00:05:30,639
you know everything from the

154
00:05:27,759 --> 00:05:32,560
configuration to the orchestration

155
00:05:30,639 --> 00:05:34,720
is not often not

156
00:05:32,560 --> 00:05:36,400
you know open for contribution in in a

157
00:05:34,720 --> 00:05:37,440
free manner

158
00:05:36,400 --> 00:05:40,080
um

159
00:05:37,440 --> 00:05:42,639
even some open source hosting

160
00:05:40,080 --> 00:05:45,039
providers that i've seen sort of say oh

161
00:05:42,639 --> 00:05:47,520
we'd we'd love your help to to help you

162
00:05:45,039 --> 00:05:48,720
know add admin our systems send us a

163
00:05:47,520 --> 00:05:49,919
resume

164
00:05:48,720 --> 00:05:51,520
um

165
00:05:49,919 --> 00:05:53,360
so we definitely don't want to want to

166
00:05:51,520 --> 00:05:56,160
be at this level we want to have our

167
00:05:53,360 --> 00:05:56,960
infrastructure completely open

168
00:05:56,160 --> 00:05:58,479
and

169
00:05:56,960 --> 00:06:01,199
that's the systems that we've built and

170
00:05:58,479 --> 00:06:03,840
that i'll be talking about today

171
00:06:01,199 --> 00:06:08,720
uh the only way you can really achieve

172
00:06:03,840 --> 00:06:10,000
this is to have your deployment and

173
00:06:08,720 --> 00:06:12,319
ci

174
00:06:10,000 --> 00:06:14,160
really so close that

175
00:06:12,319 --> 00:06:15,600
that they're almost the same thing we

176
00:06:14,160 --> 00:06:17,199
call it testing

177
00:06:15,600 --> 00:06:18,160
test like production

178
00:06:17,199 --> 00:06:20,240
uh

179
00:06:18,160 --> 00:06:22,479
you know we can accept sort of

180
00:06:20,240 --> 00:06:24,960
unprivileged changes coming in

181
00:06:22,479 --> 00:06:26,720
and and test them and have a really high

182
00:06:24,960 --> 00:06:28,000
level of confidence that when we commit

183
00:06:26,720 --> 00:06:28,960
these changes

184
00:06:28,000 --> 00:06:31,039
that

185
00:06:28,960 --> 00:06:33,199
we're really close to what is going to

186
00:06:31,039 --> 00:06:34,960
be pushed into production

187
00:06:33,199 --> 00:06:37,840
um

188
00:06:34,960 --> 00:06:40,240
and yeah it really allows us to run open

189
00:06:37,840 --> 00:06:42,960
infrastructure by taking this model this

190
00:06:40,240 --> 00:06:45,120
test-like production model

191
00:06:42,960 --> 00:06:47,919
um

192
00:06:45,120 --> 00:06:51,680
a few more concepts to to pick up zul

193
00:06:47,919 --> 00:06:52,720
zul is the ci system that um really

194
00:06:51,680 --> 00:06:54,000
enables

195
00:06:52,720 --> 00:06:55,840
all of this

196
00:06:54,000 --> 00:06:57,280
um hopefully some of you will be

197
00:06:55,840 --> 00:06:59,440
familiar with zul especially if you've

198
00:06:57,280 --> 00:07:02,240
ever contributed to openstack

199
00:06:59,440 --> 00:07:03,759
zuul would have run your ci jobs on the

200
00:07:02,240 --> 00:07:05,039
the change that you contributed to

201
00:07:03,759 --> 00:07:06,319
openstack

202
00:07:05,039 --> 00:07:10,080
um

203
00:07:06,319 --> 00:07:12,319
but it you know zul is a

204
00:07:10,080 --> 00:07:14,080
a continuous integration delivery and

205
00:07:12,319 --> 00:07:16,240
deployment system so it does all of

206
00:07:14,080 --> 00:07:18,960
these things for you and it works on

207
00:07:16,240 --> 00:07:20,960
project gating and

208
00:07:18,960 --> 00:07:21,759
gating is a little bit of a hard concept

209
00:07:20,960 --> 00:07:22,880
to

210
00:07:21,759 --> 00:07:26,000
exp

211
00:07:22,880 --> 00:07:27,680
um explain but we we all have it

212
00:07:26,000 --> 00:07:29,039
intuitively in our head

213
00:07:27,680 --> 00:07:30,560
and this is

214
00:07:29,039 --> 00:07:33,440
you may have the greatest testing in the

215
00:07:30,560 --> 00:07:35,360
world and if you uh commit your change

216
00:07:33,440 --> 00:07:37,360
and run your testing

217
00:07:35,360 --> 00:07:39,680
uh sorry if you propose your change and

218
00:07:37,360 --> 00:07:41,840
run your testing that's fine but if

219
00:07:39,680 --> 00:07:42,880
somebody else commits to the tree in

220
00:07:41,840 --> 00:07:45,919
front of you

221
00:07:42,880 --> 00:07:48,400
then your testing is now invalid

222
00:07:45,919 --> 00:07:49,840
and and we all sort of in intuitively

223
00:07:48,400 --> 00:07:52,160
understand this

224
00:07:49,840 --> 00:07:53,199
and the only way to safely merge our

225
00:07:52,160 --> 00:07:55,440
changes

226
00:07:53,199 --> 00:07:58,400
when we have more than one developer is

227
00:07:55,440 --> 00:08:01,840
to really put those changes into a queue

228
00:07:58,400 --> 00:08:03,680
and merge them sequentially and

229
00:08:01,840 --> 00:08:06,000
you need a system to help you do this

230
00:08:03,680 --> 00:08:08,479
you know you can't have humans sort of

231
00:08:06,000 --> 00:08:10,800
doing this it just doesn't scale

232
00:08:08,479 --> 00:08:12,639
zul essentially solves this and this is

233
00:08:10,800 --> 00:08:15,039
what we call project gating

234
00:08:12,639 --> 00:08:16,639
uh zul's been doing this for you know

235
00:08:15,039 --> 00:08:20,800
well over a decade

236
00:08:16,639 --> 00:08:23,199
uh uh other um systems now you know this

237
00:08:20,800 --> 00:08:25,039
is now called merge trains or merge

238
00:08:23,199 --> 00:08:27,440
queues in other systems as well they

239
00:08:25,039 --> 00:08:29,520
sort of catching up

240
00:08:27,440 --> 00:08:31,440
but this idea of gating that it's

241
00:08:29,520 --> 00:08:35,360
impossible for us to

242
00:08:31,440 --> 00:08:37,839
uh merge a broken change because um

243
00:08:35,360 --> 00:08:40,719
zul is merging the changes for us humans

244
00:08:37,839 --> 00:08:43,279
don't merge changes humans say this

245
00:08:40,719 --> 00:08:46,000
change is okay to merge and then what

246
00:08:43,279 --> 00:08:47,440
zul will do is put them in a queue run

247
00:08:46,000 --> 00:08:49,440
the testing

248
00:08:47,440 --> 00:08:51,360
in the most efficient way manner in the

249
00:08:49,440 --> 00:08:52,640
most efficient manner possible

250
00:08:51,360 --> 00:08:54,959
and

251
00:08:52,640 --> 00:08:55,760
push those changes in for us

252
00:08:54,959 --> 00:08:57,040
so

253
00:08:55,760 --> 00:08:59,040
um

254
00:08:57,040 --> 00:09:01,040
let's get into the the details more i've

255
00:08:59,040 --> 00:09:01,760
sort of gone through all these concepts

256
00:09:01,040 --> 00:09:04,160
but

257
00:09:01,760 --> 00:09:06,640
we need to get into some details here

258
00:09:04,160 --> 00:09:09,279
today i i just want to

259
00:09:06,640 --> 00:09:11,120
use an example of our paste system which

260
00:09:09,279 --> 00:09:12,880
is just one of our absolute simplest

261
00:09:11,120 --> 00:09:15,519
systems but it'd be interesting to just

262
00:09:12,880 --> 00:09:16,959
walk through a couple of changes here

263
00:09:15,519 --> 00:09:18,720
and probably familiar with the paste

264
00:09:16,959 --> 00:09:20,480
system you know you just put some text

265
00:09:18,720 --> 00:09:22,480
in there you paste it in there and you

266
00:09:20,480 --> 00:09:24,320
get back at url and you know you can

267
00:09:22,480 --> 00:09:26,640
share your paste is good for you know

268
00:09:24,320 --> 00:09:27,760
sharing logs and things like that

269
00:09:26,640 --> 00:09:29,920
uh

270
00:09:27,760 --> 00:09:31,600
open infrastructure you know this is all

271
00:09:29,920 --> 00:09:33,200
open you can go to this this website

272
00:09:31,600 --> 00:09:35,760
here and you can see our the paste

273
00:09:33,200 --> 00:09:38,000
service that we run

274
00:09:35,760 --> 00:09:39,760
what i i just want to pull out

275
00:09:38,000 --> 00:09:41,760
it builds a container just got a docker

276
00:09:39,760 --> 00:09:43,760
file there it builds a container nothing

277
00:09:41,760 --> 00:09:46,640
amazing builds a container based on

278
00:09:43,760 --> 00:09:48,160
python 3.8 okay fine all right let's

279
00:09:46,640 --> 00:09:50,000
just keep that in our head

280
00:09:48,160 --> 00:09:51,680
i'm going to propose a couple of changes

281
00:09:50,000 --> 00:09:53,279
here just to show you how this whole

282
00:09:51,680 --> 00:09:55,120
system works together and we're going to

283
00:09:53,279 --> 00:09:56,959
work through the whole thing here

284
00:09:55,120 --> 00:09:59,040
this is a this is uh what you would see

285
00:09:56,959 --> 00:10:00,720
on our review site review

286
00:09:59,040 --> 00:10:01,760
dot open there this is a garrett

287
00:10:00,720 --> 00:10:04,959
instance

288
00:10:01,760 --> 00:10:07,440
um we're looking at the logit um

289
00:10:04,959 --> 00:10:10,399
paste system uh

290
00:10:07,440 --> 00:10:12,079
this is the open changes for for that

291
00:10:10,399 --> 00:10:13,760
and i'm gonna propose a couple of

292
00:10:12,079 --> 00:10:14,720
changes here that we're gonna use as an

293
00:10:13,760 --> 00:10:16,959
example

294
00:10:14,720 --> 00:10:19,200
uh don't worry about the dnm in front of

295
00:10:16,959 --> 00:10:20,959
it i actually propose these to the live

296
00:10:19,200 --> 00:10:22,880
system this has all been done

297
00:10:20,959 --> 00:10:25,120
you know in the live system dnm means do

298
00:10:22,880 --> 00:10:26,320
not merge it's just an indication to

299
00:10:25,120 --> 00:10:27,760
reviewers

300
00:10:26,320 --> 00:10:28,880
you know i'm doing something don't look

301
00:10:27,760 --> 00:10:29,760
at this

302
00:10:28,880 --> 00:10:31,279
um

303
00:10:29,760 --> 00:10:33,279
so the first change we're going to

304
00:10:31,279 --> 00:10:35,600
update the

305
00:10:33,279 --> 00:10:36,880
container to python 3.9 this is

306
00:10:35,600 --> 00:10:38,480
something you definitely want to test

307
00:10:36,880 --> 00:10:41,440
right you're updating a container from

308
00:10:38,480 --> 00:10:43,279
3.8 to 3.9

309
00:10:41,440 --> 00:10:45,680
that could potentially impact your

310
00:10:43,279 --> 00:10:46,800
services you want to test that

311
00:10:45,680 --> 00:10:48,560
and

312
00:10:46,800 --> 00:10:49,760
i'm also going to put on top of that

313
00:10:48,560 --> 00:10:51,760
just

314
00:10:49,760 --> 00:10:54,000
another change just just for fun just to

315
00:10:51,760 --> 00:10:56,480
show how all this cross project testing

316
00:10:54,000 --> 00:10:58,959
works instead of saying new paste we're

317
00:10:56,480 --> 00:11:00,959
going to make it say enter new paste

318
00:10:58,959 --> 00:11:02,800
just go back see where it says new paste

319
00:11:00,959 --> 00:11:06,399
we're going to make it say enter new

320
00:11:02,800 --> 00:11:09,120
paste just just for fun

321
00:11:06,399 --> 00:11:10,720
okay so these two changes are stacked on

322
00:11:09,120 --> 00:11:14,000
top of each other

323
00:11:10,720 --> 00:11:16,959
these two changes depend on each other

324
00:11:14,000 --> 00:11:18,720
you can sort of see that in the way that

325
00:11:16,959 --> 00:11:20,640
garrett has put the changes on top of

326
00:11:18,720 --> 00:11:22,720
each other

327
00:11:20,640 --> 00:11:24,640
so i've proposed these changes i've

328
00:11:22,720 --> 00:11:26,399
pushed them into the code review system

329
00:11:24,640 --> 00:11:29,040
what's going to happen next

330
00:11:26,399 --> 00:11:30,720
this is the the zuul uh interface that

331
00:11:29,040 --> 00:11:32,720
i've extracted here

332
00:11:30,720 --> 00:11:34,240
uh you can see in the queue here i've

333
00:11:32,720 --> 00:11:36,079
just sort of

334
00:11:34,240 --> 00:11:36,959
squished it all down here

335
00:11:36,079 --> 00:11:38,560
um

336
00:11:36,959 --> 00:11:40,720
zul is going to pick up that i've pushed

337
00:11:38,560 --> 00:11:42,640
two changes and it's it's going to start

338
00:11:40,720 --> 00:11:45,360
running some testing you can see it's

339
00:11:42,640 --> 00:11:48,240
going to start running the first test

340
00:11:45,360 --> 00:11:49,360
on the base change and then it's going

341
00:11:48,240 --> 00:11:51,519
to

342
00:11:49,360 --> 00:11:53,120
put the the second change together with

343
00:11:51,519 --> 00:11:54,800
the first change and start running the

344
00:11:53,120 --> 00:11:56,160
testing on

345
00:11:54,800 --> 00:11:58,399
the second change i'm going to call that

346
00:11:56,160 --> 00:11:59,839
the api change because it you know

347
00:11:58,399 --> 00:12:02,480
changes some text which could be

348
00:11:59,839 --> 00:12:04,560
considered an api

349
00:12:02,480 --> 00:12:06,000
all right so zul's going to run its

350
00:12:04,560 --> 00:12:07,279
testing we're going to wait a few

351
00:12:06,000 --> 00:12:08,240
minutes we're going to come back to the

352
00:12:07,279 --> 00:12:11,519
change

353
00:12:08,240 --> 00:12:14,480
a little bit later this is the change

354
00:12:11,519 --> 00:12:17,040
uh i'm just calling out here that you

355
00:12:14,480 --> 00:12:17,920
know we have this relation chain again

356
00:12:17,040 --> 00:12:20,959
uh

357
00:12:17,920 --> 00:12:23,279
zul has run the testing it's verified it

358
00:12:20,959 --> 00:12:25,680
says okay all the tests pass everything

359
00:12:23,279 --> 00:12:27,360
looks good great

360
00:12:25,680 --> 00:12:28,800
and so at this point you're sort of

361
00:12:27,360 --> 00:12:30,639
going okay

362
00:12:28,800 --> 00:12:32,720
thanks ian you pushed the change and you

363
00:12:30,639 --> 00:12:34,079
ran some testing um

364
00:12:32,720 --> 00:12:36,000
we've been doing that for a long time

365
00:12:34,079 --> 00:12:37,440
that's not very exciting

366
00:12:36,000 --> 00:12:39,200
and that's

367
00:12:37,440 --> 00:12:41,120
true um

368
00:12:39,200 --> 00:12:43,440
but what we've tested here is remember

369
00:12:41,120 --> 00:12:45,360
just the the logic the the the paste

370
00:12:43,440 --> 00:12:47,519
service container here

371
00:12:45,360 --> 00:12:49,839
but this this is not what we deploy

372
00:12:47,519 --> 00:12:53,680
there's a lot more that goes

373
00:12:49,839 --> 00:12:55,680
around this service that you know has

374
00:12:53,680 --> 00:12:57,519
has not been tested at this point but we

375
00:12:55,680 --> 00:13:00,480
we we need to test all this stuff in

376
00:12:57,519 --> 00:13:02,800
blue as the open dev you know

377
00:13:00,480 --> 00:13:05,279
people providing this service to make

378
00:13:02,800 --> 00:13:06,720
sure that these changes are ready for

379
00:13:05,279 --> 00:13:08,079
production

380
00:13:06,720 --> 00:13:10,320
um

381
00:13:08,079 --> 00:13:11,440
yeah all these things in blue

382
00:13:10,320 --> 00:13:13,040
um

383
00:13:11,440 --> 00:13:15,279
i i really

384
00:13:13,040 --> 00:13:16,240
the the production side of the testing

385
00:13:15,279 --> 00:13:17,279
here

386
00:13:16,240 --> 00:13:18,320
so

387
00:13:17,279 --> 00:13:21,360
you know

388
00:13:18,320 --> 00:13:23,760
all these all these these blue bits are

389
00:13:21,360 --> 00:13:26,000
what you would call you know system

390
00:13:23,760 --> 00:13:27,120
configuration

391
00:13:26,000 --> 00:13:30,240
so

392
00:13:27,120 --> 00:13:32,560
now we come to testing and deploying

393
00:13:30,240 --> 00:13:35,519
the the two changes that we sort of

394
00:13:32,560 --> 00:13:37,760
discussed before so

395
00:13:35,519 --> 00:13:39,680
you know system configuration we

396
00:13:37,760 --> 00:13:42,240
actually have a repository where we

397
00:13:39,680 --> 00:13:43,839
store our system configuration called

398
00:13:42,240 --> 00:13:46,399
system config

399
00:13:43,839 --> 00:13:49,120
this is the the url for it and this is

400
00:13:46,399 --> 00:13:51,360
where we're going to jump in to start

401
00:13:49,120 --> 00:13:53,120
testing these changes that we've made to

402
00:13:51,360 --> 00:13:56,720
our service to make sure they're ready

403
00:13:53,120 --> 00:13:59,120
for production and ready to be pushed

404
00:13:56,720 --> 00:14:00,399
you know into service

405
00:13:59,120 --> 00:14:02,079
so

406
00:14:00,399 --> 00:14:04,480
i'm now going to propose a change to our

407
00:14:02,079 --> 00:14:05,279
system config repo

408
00:14:04,480 --> 00:14:06,959
uh

409
00:14:05,279 --> 00:14:08,959
now here's the magic

410
00:14:06,959 --> 00:14:10,959
everything's all the magic sort of comes

411
00:14:08,959 --> 00:14:12,639
from this one little line

412
00:14:10,959 --> 00:14:15,920
uh we depend on

413
00:14:12,639 --> 00:14:17,360
that previous change the api change

414
00:14:15,920 --> 00:14:20,800
that

415
00:14:17,360 --> 00:14:23,040
updates the container to python 3.9 and

416
00:14:20,800 --> 00:14:26,399
makes the makes a little change

417
00:14:23,040 --> 00:14:27,680
and what we do in this system config

418
00:14:26,399 --> 00:14:30,240
change is

419
00:14:27,680 --> 00:14:31,519
we update our testing of the paste

420
00:14:30,240 --> 00:14:33,279
service

421
00:14:31,519 --> 00:14:36,560
see you can see down there instead of

422
00:14:33,279 --> 00:14:38,160
looking for the text uh new paste

423
00:14:36,560 --> 00:14:40,399
in the in the text we're going to look

424
00:14:38,160 --> 00:14:42,720
for please enter your new paste that was

425
00:14:40,399 --> 00:14:44,399
the change we made

426
00:14:42,720 --> 00:14:46,160
we used uh

427
00:14:44,399 --> 00:14:47,440
down the bottom there we use a

428
00:14:46,160 --> 00:14:49,600
um

429
00:14:47,440 --> 00:14:51,440
it's called test infra it's it's a very

430
00:14:49,600 --> 00:14:54,720
cool uh project that

431
00:14:51,440 --> 00:14:57,199
allows you to test your infrastructure

432
00:14:54,720 --> 00:14:58,560
like write unit tests that test your

433
00:14:57,199 --> 00:15:00,320
your infrastructure

434
00:14:58,560 --> 00:15:02,240
uh check it out if that's the type of

435
00:15:00,320 --> 00:15:02,959
thing you you need to do

436
00:15:02,240 --> 00:15:05,199
so

437
00:15:02,959 --> 00:15:05,199
um

438
00:15:09,519 --> 00:15:14,320
yeah i'm sorry

439
00:15:12,399 --> 00:15:15,760
just didn't cut the window looks a

440
00:15:14,320 --> 00:15:16,880
little cut off there

441
00:15:15,760 --> 00:15:19,680
um

442
00:15:16,880 --> 00:15:20,880
so we're going to push this change

443
00:15:19,680 --> 00:15:23,440
and

444
00:15:20,880 --> 00:15:24,880
now i i kind of need to explain to you

445
00:15:23,440 --> 00:15:25,839
what's going to happen

446
00:15:24,880 --> 00:15:27,680
when

447
00:15:25,839 --> 00:15:29,519
we push this change and zul picks up

448
00:15:27,680 --> 00:15:31,759
this change

449
00:15:29,519 --> 00:15:31,759
so

450
00:15:31,839 --> 00:15:37,040
when we propose a change uh from any any

451
00:15:34,720 --> 00:15:38,720
of these services uh zuul is sitting

452
00:15:37,040 --> 00:15:41,040
there as it's listening for the changes

453
00:15:38,720 --> 00:15:44,240
that come in and it prepares what we

454
00:15:41,040 --> 00:15:45,040
call uh an executor environment

455
00:15:44,240 --> 00:15:47,040
so

456
00:15:45,040 --> 00:15:50,240
the first thing it will do is is take

457
00:15:47,040 --> 00:15:51,920
the code from uh the proposed changes

458
00:15:50,240 --> 00:15:54,320
and it will

459
00:15:51,920 --> 00:15:56,720
merge them into

460
00:15:54,320 --> 00:15:58,720
code trees that you need to test against

461
00:15:56,720 --> 00:16:01,279
so zool knows the dependencies you know

462
00:15:58,720 --> 00:16:02,639
we we set up that depends on before

463
00:16:01,279 --> 00:16:04,800
and um

464
00:16:02,639 --> 00:16:06,839
zoo will merge all the code that you

465
00:16:04,800 --> 00:16:08,639
need to test

466
00:16:06,839 --> 00:16:09,839
into uh

467
00:16:08,639 --> 00:16:12,079
the

468
00:16:09,839 --> 00:16:14,720
for you automatically

469
00:16:12,079 --> 00:16:18,720
uh the next thing it will do is set up

470
00:16:14,720 --> 00:16:21,279
uh the nodes where you uh want to test

471
00:16:18,720 --> 00:16:22,399
so these are uh ephemeral test nodes

472
00:16:21,279 --> 00:16:24,560
that they're just

473
00:16:22,399 --> 00:16:25,600
assigned and allocated to you

474
00:16:24,560 --> 00:16:27,040
and

475
00:16:25,600 --> 00:16:28,800
they're the nodes that you use for

476
00:16:27,040 --> 00:16:30,880
testing uh

477
00:16:28,800 --> 00:16:33,199
we won't talk about how those nodes are

478
00:16:30,880 --> 00:16:34,639
allocated that's it's done by node pool

479
00:16:33,199 --> 00:16:36,079
a whole separate thing that could be a

480
00:16:34,639 --> 00:16:37,759
whole separate talk we'll just take it

481
00:16:36,079 --> 00:16:40,480
for granted that we get a couple of

482
00:16:37,759 --> 00:16:42,560
nodes to test on as many as we want

483
00:16:40,480 --> 00:16:44,560
now uh the next

484
00:16:42,560 --> 00:16:46,480
at this point right we have a bunch of

485
00:16:44,560 --> 00:16:48,880
nodes waiting for

486
00:16:46,480 --> 00:16:50,639
something to happen on and

487
00:16:48,880 --> 00:16:53,199
we have our executor environment with a

488
00:16:50,639 --> 00:16:54,160
bunch of code that's ready to be tested

489
00:16:53,199 --> 00:16:57,360
so

490
00:16:54,160 --> 00:17:00,639
this is a job for ansible okay

491
00:16:57,360 --> 00:17:03,600
we set up a job infantry inventory that

492
00:17:00,639 --> 00:17:04,480
points to all the test nodes

493
00:17:03,600 --> 00:17:06,559
and then

494
00:17:04,480 --> 00:17:09,280
our ci

495
00:17:06,559 --> 00:17:11,120
jobs are just ansible playbooks they

496
00:17:09,280 --> 00:17:12,880
just run against the test nodes that

497
00:17:11,120 --> 00:17:15,120
have been assigned

498
00:17:12,880 --> 00:17:17,760
this means that there's

499
00:17:15,120 --> 00:17:20,160
your testing is written in ansible your

500
00:17:17,760 --> 00:17:20,880
deployment is written in ansible

501
00:17:20,160 --> 00:17:23,280
you

502
00:17:20,880 --> 00:17:24,640
you know everything is the same it if

503
00:17:23,280 --> 00:17:27,120
you want your testing to be a shell

504
00:17:24,640 --> 00:17:28,960
script well just make answers we'll call

505
00:17:27,120 --> 00:17:31,360
a giant shell script that that's fine

506
00:17:28,960 --> 00:17:34,080
but you can also use all the great

507
00:17:31,360 --> 00:17:37,120
features that ansible has

508
00:17:34,080 --> 00:17:39,200
you know just as easily in your

509
00:17:37,120 --> 00:17:42,720
job playbooks as well

510
00:17:39,200 --> 00:17:44,240
so uh we're at this point and it's it's

511
00:17:42,720 --> 00:17:47,440
sort of over to the testing at this

512
00:17:44,240 --> 00:17:50,720
point so the first thing that the

513
00:17:47,440 --> 00:17:53,280
test jobs do is copy the code over to

514
00:17:50,720 --> 00:17:57,200
the test nodes uh start running the test

515
00:17:53,280 --> 00:17:58,840
nodes uh the the nodes run their tests

516
00:17:57,200 --> 00:18:00,880
they do whatever they need to

517
00:17:58,840 --> 00:18:02,640
do um

518
00:18:00,880 --> 00:18:05,520
then they return their

519
00:18:02,640 --> 00:18:07,520
logs their artifacts whatever you know

520
00:18:05,520 --> 00:18:10,000
has happened on the nodes that they want

521
00:18:07,520 --> 00:18:14,320
to return for storage and

522
00:18:10,000 --> 00:18:15,919
uh reporting back is returned back

523
00:18:14,320 --> 00:18:18,720
zul takes that back

524
00:18:15,919 --> 00:18:21,039
and zul reports the change back that's

525
00:18:18,720 --> 00:18:25,039
that's the sort of high level overview

526
00:18:21,039 --> 00:18:26,880
of what zul does in response to a change

527
00:18:25,039 --> 00:18:28,799
now

528
00:18:26,880 --> 00:18:31,280
i'm gonna jump ahead and actually

529
00:18:28,799 --> 00:18:35,039
describe how we deploy

530
00:18:31,280 --> 00:18:36,559
merge changes from zul uh this will

531
00:18:35,039 --> 00:18:38,000
hopefully make this should make sense

532
00:18:36,559 --> 00:18:39,840
when i get to the next light but just

533
00:18:38,000 --> 00:18:41,280
stick with me because i know it seems

534
00:18:39,840 --> 00:18:44,960
like i'm jumping ahead

535
00:18:41,280 --> 00:18:47,679
so we now have emerged this is how we

536
00:18:44,960 --> 00:18:49,919
deploy our production changes

537
00:18:47,679 --> 00:18:51,120
we have a merged change okay so we trust

538
00:18:49,919 --> 00:18:54,240
this change this change has been

539
00:18:51,120 --> 00:18:56,640
reviewed uh it's been committed it's

540
00:18:54,240 --> 00:18:58,480
ready for deployment

541
00:18:56,640 --> 00:18:59,840
so zuul picks this up

542
00:18:58,480 --> 00:19:03,520
same thing

543
00:18:59,840 --> 00:19:06,400
zul starts up its executor environment

544
00:19:03,520 --> 00:19:08,320
zul grabs the change now this is now

545
00:19:06,400 --> 00:19:10,160
a merged change

546
00:19:08,320 --> 00:19:11,200
it puts it into into the executor

547
00:19:10,160 --> 00:19:12,160
environment

548
00:19:11,200 --> 00:19:14,799
now

549
00:19:12,160 --> 00:19:18,080
the only difference is that then before

550
00:19:14,799 --> 00:19:20,320
instead of an ephemeral test node that

551
00:19:18,080 --> 00:19:24,000
is is going to disappear

552
00:19:20,320 --> 00:19:26,559
we connect to a bastion host

553
00:19:24,000 --> 00:19:28,240
that has permission to log into our

554
00:19:26,559 --> 00:19:29,200
production hosts

555
00:19:28,240 --> 00:19:31,840
so

556
00:19:29,200 --> 00:19:34,960
we set up zul is allowed to to log into

557
00:19:31,840 --> 00:19:34,960
this bastion host

558
00:19:35,280 --> 00:19:40,799
the the code the merged changes uh are

559
00:19:38,320 --> 00:19:45,039
then pushed onto the bastion host and

560
00:19:40,799 --> 00:19:47,440
then the uh playbooks actually run

561
00:19:45,039 --> 00:19:50,000
the configuration through the bastion

562
00:19:47,440 --> 00:19:51,440
host but it end up targeting the

563
00:19:50,000 --> 00:19:54,320
production host

564
00:19:51,440 --> 00:19:55,679
and so that is how we do uh deployment

565
00:19:54,320 --> 00:19:58,240
viazul

566
00:19:55,679 --> 00:20:00,160
uh to our production services zool

567
00:19:58,240 --> 00:20:02,640
actually does our

568
00:20:00,160 --> 00:20:05,200
you know production deployment for us

569
00:20:02,640 --> 00:20:07,679
zool even deploys itself

570
00:20:05,200 --> 00:20:09,440
you know sort of circular

571
00:20:07,679 --> 00:20:10,400
reference there

572
00:20:09,440 --> 00:20:11,679
so

573
00:20:10,400 --> 00:20:13,600
this is our

574
00:20:11,679 --> 00:20:16,159
production deployment

575
00:20:13,600 --> 00:20:17,440
so when we want to test changes

576
00:20:16,159 --> 00:20:19,360
we need to

577
00:20:17,440 --> 00:20:21,120
we need to test like production what

578
00:20:19,360 --> 00:20:24,960
what are we going to do

579
00:20:21,120 --> 00:20:27,440
well it's all exactly the same except

580
00:20:24,960 --> 00:20:28,880
instead of using the actual bastion host

581
00:20:27,440 --> 00:20:31,360
that is allowed to

582
00:20:28,880 --> 00:20:33,520
speak to the actual production host

583
00:20:31,360 --> 00:20:34,640
we just turn those into ephemeral test

584
00:20:33,520 --> 00:20:37,840
nodes

585
00:20:34,640 --> 00:20:39,760
we we use um

586
00:20:37,840 --> 00:20:40,799
the the nodes that are allocated by node

587
00:20:39,760 --> 00:20:43,039
pool

588
00:20:40,799 --> 00:20:44,880
but we really do the same thing so a

589
00:20:43,039 --> 00:20:47,440
proposed change comes in from anybody

590
00:20:44,880 --> 00:20:48,960
who wants to submit it in garrett

591
00:20:47,440 --> 00:20:50,640
and

592
00:20:48,960 --> 00:20:53,919
we actually

593
00:20:50,640 --> 00:20:56,480
do the exactly the same process except

594
00:20:53,919 --> 00:20:58,320
it deploys onto two nodes that

595
00:20:56,480 --> 00:21:00,320
will disappear at the end of the test

596
00:20:58,320 --> 00:21:01,520
and we don't care about

597
00:21:00,320 --> 00:21:03,200
so

598
00:21:01,520 --> 00:21:05,039
this is our simulator production

599
00:21:03,200 --> 00:21:06,000
environment

600
00:21:05,039 --> 00:21:07,840
and

601
00:21:06,000 --> 00:21:09,120
this is really the key to how we do this

602
00:21:07,840 --> 00:21:11,520
um

603
00:21:09,120 --> 00:21:14,000
test like we're in production

604
00:21:11,520 --> 00:21:16,240
and this is why i say that our testing

605
00:21:14,000 --> 00:21:17,360
is really really close to our deployment

606
00:21:16,240 --> 00:21:18,720
um

607
00:21:17,360 --> 00:21:21,600
you know

608
00:21:18,720 --> 00:21:23,039
we tried it we really is that close that

609
00:21:21,600 --> 00:21:25,600
we have a really high level of

610
00:21:23,039 --> 00:21:27,840
confidence that if we've passed testing

611
00:21:25,600 --> 00:21:31,280
that when we push this into production

612
00:21:27,840 --> 00:21:33,440
it's all going to be okay so i want to

613
00:21:31,280 --> 00:21:34,960
dive into a little bit of of how this

614
00:21:33,440 --> 00:21:36,559
actually works give you some actual

615
00:21:34,960 --> 00:21:37,919
details

616
00:21:36,559 --> 00:21:39,039
this is actually

617
00:21:37,919 --> 00:21:42,320
uh

618
00:21:39,039 --> 00:21:45,200
the job the system config run paste this

619
00:21:42,320 --> 00:21:47,520
is the actual zool job that

620
00:21:45,200 --> 00:21:50,240
is going to be testing the logic

621
00:21:47,520 --> 00:21:52,159
container changes and tests the pace

622
00:21:50,240 --> 00:21:55,200
system um

623
00:21:52,159 --> 00:21:55,200
the job of interest

624
00:21:55,520 --> 00:21:58,640
for our situation here

625
00:21:57,760 --> 00:22:00,799
so

626
00:21:58,640 --> 00:22:02,159
this is dual config but you don't even

627
00:22:00,799 --> 00:22:04,240
need to really understand it you can

628
00:22:02,159 --> 00:22:06,559
just see what's going on you can see

629
00:22:04,240 --> 00:22:08,720
that we require the system configuration

630
00:22:06,559 --> 00:22:11,440
that that makes sense and you can see

631
00:22:08,720 --> 00:22:13,360
that we have this a node set defined

632
00:22:11,440 --> 00:22:15,039
here we need the two nodes this is the

633
00:22:13,360 --> 00:22:16,960
two nodes i've talked about we need a

634
00:22:15,039 --> 00:22:18,880
bastion host node and we need the

635
00:22:16,960 --> 00:22:21,520
production host node

636
00:22:18,880 --> 00:22:24,400
and so that's what we have defined here

637
00:22:21,520 --> 00:22:27,120
uh you know some jobs might have three

638
00:22:24,400 --> 00:22:28,960
four nodes you know they might set up a

639
00:22:27,120 --> 00:22:30,640
little test zookeeper system by

640
00:22:28,960 --> 00:22:32,559
themselves and connect up to that you

641
00:22:30,640 --> 00:22:35,039
know all sorts of things but this is

642
00:22:32,559 --> 00:22:37,120
just the absolute simplest case here of

643
00:22:35,039 --> 00:22:40,080
uh the bastion host and just one simple

644
00:22:37,120 --> 00:22:41,440
production host

645
00:22:40,080 --> 00:22:44,799
so

646
00:22:41,440 --> 00:22:46,480
i have i i've sort of

647
00:22:44,799 --> 00:22:48,880
been through this but

648
00:22:46,480 --> 00:22:51,600
it's still probably not making terribly

649
00:22:48,880 --> 00:22:52,960
much sense as to how this actually

650
00:22:51,600 --> 00:22:55,600
um

651
00:22:52,960 --> 00:22:57,760
you know works and and

652
00:22:55,600 --> 00:22:58,960
um pulls in the changes from that that

653
00:22:57,760 --> 00:23:01,520
logic

654
00:22:58,960 --> 00:23:02,799
container image so

655
00:23:01,520 --> 00:23:04,720
there's there's another little thing

656
00:23:02,799 --> 00:23:07,200
here that sort of just sits here it says

657
00:23:04,720 --> 00:23:09,280
requires logic container image

658
00:23:07,200 --> 00:23:11,600
which is um

659
00:23:09,280 --> 00:23:14,960
interesting we need to sort of pull into

660
00:23:11,600 --> 00:23:15,919
that and see how does that work

661
00:23:14,960 --> 00:23:18,799
so

662
00:23:15,919 --> 00:23:20,640
let's let's jump ship let's go back to

663
00:23:18,799 --> 00:23:22,559
logit the paste service this is where

664
00:23:20,640 --> 00:23:24,960
the jobs are defined for the logic page

665
00:23:22,559 --> 00:23:24,960
service

666
00:23:25,120 --> 00:23:29,760
uh it's rather annoying that for some

667
00:23:27,760 --> 00:23:32,080
reason

668
00:23:29,760 --> 00:23:33,520
the uh

669
00:23:32,080 --> 00:23:36,520
some of the pictures aren't showing up

670
00:23:33,520 --> 00:23:36,520
here

671
00:23:37,600 --> 00:23:40,159
we'll just have to go on

672
00:23:39,280 --> 00:23:41,520
so

673
00:23:40,159 --> 00:23:42,880
the

674
00:23:41,520 --> 00:23:45,200
this is uh

675
00:23:42,880 --> 00:23:46,320
a job called um

676
00:23:45,200 --> 00:23:48,559
the

677
00:23:46,320 --> 00:23:51,200
logit build open dev image this is what

678
00:23:48,559 --> 00:23:54,799
actually builds the

679
00:23:51,200 --> 00:23:57,279
paste service container image for us

680
00:23:54,799 --> 00:23:57,279
you can see

681
00:23:57,520 --> 00:24:01,679
this actually provides

682
00:23:59,360 --> 00:24:03,279
the logic container image so before we

683
00:24:01,679 --> 00:24:06,799
had the requires

684
00:24:03,279 --> 00:24:10,320
and here we have the provides

685
00:24:06,799 --> 00:24:11,760
what's really nice about zuul is that

686
00:24:10,320 --> 00:24:14,720
everything

687
00:24:11,760 --> 00:24:15,840
zuul has a sort of hierarchical

688
00:24:14,720 --> 00:24:19,279
job

689
00:24:15,840 --> 00:24:21,520
concept so this open dev build docker

690
00:24:19,279 --> 00:24:24,159
image is actually a

691
00:24:21,520 --> 00:24:27,360
very generic job um

692
00:24:24,159 --> 00:24:30,840
that we sort of instantiate here to to

693
00:24:27,360 --> 00:24:33,279
just build this particular image

694
00:24:30,840 --> 00:24:35,279
um you can see that this has a

695
00:24:33,279 --> 00:24:37,120
dependency on a thing called the build

696
00:24:35,279 --> 00:24:38,240
set registry

697
00:24:37,120 --> 00:24:41,200
um

698
00:24:38,240 --> 00:24:42,320
if you notice when we uh had the job up

699
00:24:41,200 --> 00:24:44,640
before

700
00:24:42,320 --> 00:24:46,880
that the opened the the build of the

701
00:24:44,640 --> 00:24:49,120
container image actually depended on

702
00:24:46,880 --> 00:24:50,480
this build set registry so it was a

703
00:24:49,120 --> 00:24:52,080
build set

704
00:24:50,480 --> 00:24:54,400
the build set is

705
00:24:52,080 --> 00:24:56,159
is this this this group

706
00:24:54,400 --> 00:24:58,400
this whole set of testing that is

707
00:24:56,159 --> 00:25:00,480
happening against this one particular

708
00:24:58,400 --> 00:25:03,360
change

709
00:25:00,480 --> 00:25:05,440
so what the build set registry is is

710
00:25:03,360 --> 00:25:07,279
actually a registry so a registry is a

711
00:25:05,440 --> 00:25:09,840
place that you can

712
00:25:07,279 --> 00:25:11,279
push container images and pull container

713
00:25:09,840 --> 00:25:13,919
images from

714
00:25:11,279 --> 00:25:15,840
it's actually a place that you can push

715
00:25:13,919 --> 00:25:17,279
a container image for a whole set of

716
00:25:15,840 --> 00:25:19,600
testing this this doesn't seem to make

717
00:25:17,279 --> 00:25:21,120
much sense for this particular case but

718
00:25:19,600 --> 00:25:24,240
he's here's something i've pulled out

719
00:25:21,120 --> 00:25:25,840
from uh where we use node pool

720
00:25:24,240 --> 00:25:28,640
so

721
00:25:25,840 --> 00:25:31,120
here's the the registry job

722
00:25:28,640 --> 00:25:32,799
we actually build a node pool image here

723
00:25:31,120 --> 00:25:34,640
in this

724
00:25:32,799 --> 00:25:36,880
and then we use that image

725
00:25:34,640 --> 00:25:39,200
one two three you know

726
00:25:36,880 --> 00:25:41,440
12 or 13 times

727
00:25:39,200 --> 00:25:43,600
afterwards to build all these these

728
00:25:41,440 --> 00:25:46,320
different platforms so

729
00:25:43,600 --> 00:25:48,799
the uh building the image relies on the

730
00:25:46,320 --> 00:25:51,200
registry once the image is built it

731
00:25:48,799 --> 00:25:53,440
pushes it into that local registry and

732
00:25:51,200 --> 00:25:56,880
then all these jobs

733
00:25:53,440 --> 00:25:58,480
uh depend on both of those and they

734
00:25:56,880 --> 00:26:01,760
actually pull

735
00:25:58,480 --> 00:26:03,840
from that local registry so when we push

736
00:26:01,760 --> 00:26:06,320
a change

737
00:26:03,840 --> 00:26:07,760
we sort of build

738
00:26:06,320 --> 00:26:09,360
the image once

739
00:26:07,760 --> 00:26:11,039
we push it into

740
00:26:09,360 --> 00:26:12,720
the local registry

741
00:26:11,039 --> 00:26:15,039
and then we can pull it as many times as

742
00:26:12,720 --> 00:26:15,919
we need with as many tests

743
00:26:15,039 --> 00:26:20,480
and

744
00:26:15,919 --> 00:26:20,480
we don't have to build it multiple times

745
00:26:22,159 --> 00:26:26,080
but

746
00:26:22,880 --> 00:26:28,320
you sort of still might be saying okay

747
00:26:26,080 --> 00:26:30,640
uh but you seem to be running

748
00:26:28,320 --> 00:26:33,760
across two separate projects here

749
00:26:30,640 --> 00:26:35,520
you've got the the logit side where i i

750
00:26:33,760 --> 00:26:37,039
sort of understand you're building

751
00:26:35,520 --> 00:26:39,760
you're building a container image with

752
00:26:37,039 --> 00:26:41,840
these changes that that's fine but how

753
00:26:39,760 --> 00:26:42,880
does this get across and and still get

754
00:26:41,840 --> 00:26:45,760
tested

755
00:26:42,880 --> 00:26:48,400
on the the configuration side

756
00:26:45,760 --> 00:26:50,640
this seems to be something missing here

757
00:26:48,400 --> 00:26:54,400
well uh

758
00:26:50,640 --> 00:26:56,640
just uh two more concepts i swear

759
00:26:54,400 --> 00:26:56,640
um

760
00:26:57,600 --> 00:27:01,440
we have a concept of an intermediate

761
00:27:00,320 --> 00:27:02,240
registry

762
00:27:01,440 --> 00:27:05,840
so

763
00:27:02,240 --> 00:27:08,080
this is actually a shared registry

764
00:27:05,840 --> 00:27:10,640
that

765
00:27:08,080 --> 00:27:13,200
we can push our changes into that sits

766
00:27:10,640 --> 00:27:16,000
that is not ephemeral that sits outside

767
00:27:13,200 --> 00:27:18,960
of our changes

768
00:27:16,000 --> 00:27:21,279
and allows us to pull

769
00:27:18,960 --> 00:27:22,320
the containers into

770
00:27:21,279 --> 00:27:25,200
uh

771
00:27:22,320 --> 00:27:28,080
other projects and to do this uh

772
00:27:25,200 --> 00:27:30,640
cross-project testing

773
00:27:28,080 --> 00:27:32,399
uh that's great that you can push a

774
00:27:30,640 --> 00:27:34,320
change into

775
00:27:32,399 --> 00:27:37,279
uh

776
00:27:34,320 --> 00:27:39,360
a sort of registry for caching but you

777
00:27:37,279 --> 00:27:40,559
actually have to be able to find that

778
00:27:39,360 --> 00:27:43,360
change again

779
00:27:40,559 --> 00:27:44,480
and so we use another part of zul which

780
00:27:43,360 --> 00:27:47,520
is the

781
00:27:44,480 --> 00:27:50,399
artifact handling which is where zul

782
00:27:47,520 --> 00:27:52,000
keeps uh meditator that jobs return

783
00:27:50,399 --> 00:27:55,039
and um

784
00:27:52,000 --> 00:27:57,760
we use that to look up the

785
00:27:55,039 --> 00:27:58,640
container in the intermediate registry

786
00:27:57,760 --> 00:28:01,279
so

787
00:27:58,640 --> 00:28:04,240
let me actually explain how this works

788
00:28:01,279 --> 00:28:05,840
in detail so we we have our proposed

789
00:28:04,240 --> 00:28:07,919
garrett change

790
00:28:05,840 --> 00:28:10,640
this was the two changes we made

791
00:28:07,919 --> 00:28:13,679
uh we built we ran this job that that

792
00:28:10,640 --> 00:28:15,679
built the the container image the build

793
00:28:13,679 --> 00:28:17,120
open dev image that we've been talking

794
00:28:15,679 --> 00:28:19,120
about

795
00:28:17,120 --> 00:28:20,799
that gets pushed to the build set

796
00:28:19,120 --> 00:28:23,200
registry that's what we've been talking

797
00:28:20,799 --> 00:28:26,000
about that can be used by

798
00:28:23,200 --> 00:28:28,320
this jobs that are being tested right

799
00:28:26,000 --> 00:28:30,720
alongside

800
00:28:28,320 --> 00:28:33,120
that change um

801
00:28:30,720 --> 00:28:34,880
but we also push that change into this

802
00:28:33,120 --> 00:28:36,559
intermediate registry

803
00:28:34,880 --> 00:28:39,440
so this is the long the long lived

804
00:28:36,559 --> 00:28:41,279
registry uh it'll sit in here it just

805
00:28:39,440 --> 00:28:43,360
doesn't like a mark sweep clean up it

806
00:28:41,279 --> 00:28:45,360
might sit in here for a couple of days

807
00:28:43,360 --> 00:28:47,440
until it's not used and then you know

808
00:28:45,360 --> 00:28:49,120
the intermediate registry

809
00:28:47,440 --> 00:28:51,120
will get rid of it an intermediate

810
00:28:49,120 --> 00:28:53,520
registry part of

811
00:28:51,120 --> 00:28:56,480
part of zul a service that that

812
00:28:53,520 --> 00:28:59,200
can be run alongside windsurf

813
00:28:56,480 --> 00:29:02,399
but then we also return an artifact from

814
00:28:59,200 --> 00:29:04,960
the job with a tag that that tells us

815
00:29:02,399 --> 00:29:07,840
where in intermediate registry this

816
00:29:04,960 --> 00:29:08,880
container is that we built

817
00:29:07,840 --> 00:29:12,640
so

818
00:29:08,880 --> 00:29:15,360
let's let's sort of have a look at this

819
00:29:12,640 --> 00:29:17,200
here's the the

820
00:29:15,360 --> 00:29:18,880
build of the

821
00:29:17,200 --> 00:29:21,039
testing image that we want to test and

822
00:29:18,880 --> 00:29:23,520
deploy

823
00:29:21,039 --> 00:29:25,679
this is the the zul interface for the

824
00:29:23,520 --> 00:29:27,120
job result

825
00:29:25,679 --> 00:29:29,440
which you can find

826
00:29:27,120 --> 00:29:31,039
via the same seoul.opendev.org

827
00:29:29,440 --> 00:29:32,640
link that i showed before

828
00:29:31,039 --> 00:29:33,919
uh you click on that you'll see

829
00:29:32,640 --> 00:29:36,320
artifacts

830
00:29:33,919 --> 00:29:39,200
and you'll see we have here a

831
00:29:36,320 --> 00:29:41,200
sort of container image artifact that

832
00:29:39,200 --> 00:29:43,520
in in this case this is really not a

833
00:29:41,200 --> 00:29:45,919
human consumable artifact so it makes

834
00:29:43,520 --> 00:29:48,159
more sense if we we just pull it

835
00:29:45,919 --> 00:29:50,399
um on the command line

836
00:29:48,159 --> 00:29:52,559
you'll see that we have this artifact

837
00:29:50,399 --> 00:29:54,640
type and you'll see that it gives us a

838
00:29:52,559 --> 00:29:56,240
reference here so that we know where to

839
00:29:54,640 --> 00:29:58,880
pull it

840
00:29:56,240 --> 00:30:01,120
from intermediate registry

841
00:29:58,880 --> 00:30:01,120
so

842
00:30:02,320 --> 00:30:07,679
this this is how we end up doing the the

843
00:30:05,200 --> 00:30:10,640
cross project testing and pulling in the

844
00:30:07,679 --> 00:30:12,399
changes from the logic container pulling

845
00:30:10,640 --> 00:30:15,360
them in testing them in our system

846
00:30:12,399 --> 00:30:17,679
configuration and making sure that we we

847
00:30:15,360 --> 00:30:19,360
know everything is is ready to go and

848
00:30:17,679 --> 00:30:20,640
ready for production

849
00:30:19,360 --> 00:30:23,600
so

850
00:30:20,640 --> 00:30:26,159
this requires logic container image this

851
00:30:23,600 --> 00:30:28,320
thing that says here what what this is

852
00:30:26,159 --> 00:30:30,399
actually saying when we unpack it in

853
00:30:28,320 --> 00:30:31,919
words it says

854
00:30:30,399 --> 00:30:34,000
zuul i

855
00:30:31,919 --> 00:30:35,760
can you provide me in in the jobs that

856
00:30:34,000 --> 00:30:38,399
i'm about to run a variable that's

857
00:30:35,760 --> 00:30:39,279
called zul artifacts

858
00:30:38,399 --> 00:30:41,440
and

859
00:30:39,279 --> 00:30:42,799
i i want that to have a list of

860
00:30:41,440 --> 00:30:45,120
artifacts

861
00:30:42,799 --> 00:30:47,440
for any job that i depend on so remember

862
00:30:45,120 --> 00:30:50,399
before we said that

863
00:30:47,440 --> 00:30:53,200
this job depends on the changes to the

864
00:30:50,399 --> 00:30:55,679
the logit container image so for any job

865
00:30:53,200 --> 00:30:57,519
that i depend on

866
00:30:55,679 --> 00:31:00,159
get me the artifacts

867
00:30:57,519 --> 00:31:02,240
any job that i depend on that provides

868
00:31:00,159 --> 00:31:05,360
logic container image

869
00:31:02,240 --> 00:31:07,840
give me their artifacts in this uh

870
00:31:05,360 --> 00:31:07,840
variable

871
00:31:09,679 --> 00:31:13,120
so

872
00:31:11,360 --> 00:31:15,519
you don't really have to look

873
00:31:13,120 --> 00:31:17,440
too much but you can it's all there if

874
00:31:15,519 --> 00:31:21,279
you want to start digging into it so

875
00:31:17,440 --> 00:31:23,360
when we run the system config

876
00:31:21,279 --> 00:31:25,919
run paste job so this is the job that is

877
00:31:23,360 --> 00:31:28,559
going to test the actual deployment of

878
00:31:25,919 --> 00:31:30,880
all the the changes in the logic

879
00:31:28,559 --> 00:31:32,399
container

880
00:31:30,880 --> 00:31:34,880
you can actually see here

881
00:31:32,399 --> 00:31:37,440
where it goes and it pulls

882
00:31:34,880 --> 00:31:39,039
uh from that url that that we were

883
00:31:37,440 --> 00:31:40,080
saying before and it's actually grabbed

884
00:31:39,039 --> 00:31:41,519
that

885
00:31:40,080 --> 00:31:43,760
um

886
00:31:41,519 --> 00:31:44,559
from the artifact so

887
00:31:43,760 --> 00:31:46,559
uh

888
00:31:44,559 --> 00:31:47,600
it does that in zul jobs i just wanted

889
00:31:46,559 --> 00:31:48,960
to sort of

890
00:31:47,600 --> 00:31:49,840
pull up the code and show you that

891
00:31:48,960 --> 00:31:51,519
there's

892
00:31:49,840 --> 00:31:52,480
you know there's no magic here you can

893
00:31:51,519 --> 00:31:54,320
sort of

894
00:31:52,480 --> 00:31:56,480
you can sort of see that

895
00:31:54,320 --> 00:31:58,720
all it does is goes through

896
00:31:56,480 --> 00:32:01,120
this is the job that pulls

897
00:31:58,720 --> 00:32:02,880
uh the container from the the separate

898
00:32:01,120 --> 00:32:05,600
project

899
00:32:02,880 --> 00:32:08,320
and it just does a loop through the zuul

900
00:32:05,600 --> 00:32:10,399
artifacts and looks for

901
00:32:08,320 --> 00:32:13,200
container image

902
00:32:10,399 --> 00:32:15,519
container images in the artifacts and it

903
00:32:13,200 --> 00:32:17,679
pulls them into

904
00:32:15,519 --> 00:32:19,440
pulls them locally so we have

905
00:32:17,679 --> 00:32:21,679
effectively pulled in

906
00:32:19,440 --> 00:32:23,600
the container with the changes from the

907
00:32:21,679 --> 00:32:26,080
other project and that's the container

908
00:32:23,600 --> 00:32:29,600
that we are now going to deploy and use

909
00:32:26,080 --> 00:32:31,440
in our system config test which

910
00:32:29,600 --> 00:32:34,240
is going to ensure that

911
00:32:31,440 --> 00:32:35,679
you know we have end to end tested

912
00:32:34,240 --> 00:32:37,360
the changes

913
00:32:35,679 --> 00:32:39,120
in the uh

914
00:32:37,360 --> 00:32:40,960
logit

915
00:32:39,120 --> 00:32:43,840
that we proposed in the

916
00:32:40,960 --> 00:32:44,160
the logic project over there

917
00:32:43,840 --> 00:32:46,080
um

918
00:32:44,160 --> 00:32:47,039
[Music]

919
00:32:46,080 --> 00:32:49,679
so

920
00:32:47,039 --> 00:32:51,600
so we will we will run our our test

921
00:32:49,679 --> 00:32:52,480
deployment here

922
00:32:51,600 --> 00:32:54,559
uh

923
00:32:52,480 --> 00:32:55,279
i just wanted to to pull out and show

924
00:32:54,559 --> 00:32:56,399
you

925
00:32:55,279 --> 00:32:58,480
that we

926
00:32:56,399 --> 00:33:00,000
as part of the results of the job will

927
00:32:58,480 --> 00:33:02,000
drag back

928
00:33:00,000 --> 00:33:05,120
basically all the logs you could need to

929
00:33:02,000 --> 00:33:07,200
debug what's going on um

930
00:33:05,120 --> 00:33:09,279
you know all the the logs from starting

931
00:33:07,200 --> 00:33:11,360
the container and starting the the

932
00:33:09,279 --> 00:33:12,799
services setting up the database all

933
00:33:11,360 --> 00:33:16,000
that sort of thing

934
00:33:12,799 --> 00:33:18,559
and you also have all the

935
00:33:16,000 --> 00:33:20,240
ansible logs that uh deployed these

936
00:33:18,559 --> 00:33:21,279
changes for you so

937
00:33:20,240 --> 00:33:24,480
um

938
00:33:21,279 --> 00:33:27,279
this is just an example of the the

939
00:33:24,480 --> 00:33:29,919
ansible steps that are deploying into

940
00:33:27,279 --> 00:33:29,919
our um

941
00:33:30,960 --> 00:33:35,120
our simulated production environment

942
00:33:33,760 --> 00:33:36,320
and it

943
00:33:35,120 --> 00:33:38,640
will go through all these steps if

944
00:33:36,320 --> 00:33:41,279
there's any problems it it'll

945
00:33:38,640 --> 00:33:42,880
flag it it'll fail you'll be able to go

946
00:33:41,279 --> 00:33:45,600
back you'll be able to debug it there's

947
00:33:42,880 --> 00:33:47,600
no problems all good you know that

948
00:33:45,600 --> 00:33:48,720
you know that the the container was

949
00:33:47,600 --> 00:33:51,679
pulled in

950
00:33:48,720 --> 00:33:52,480
it started we set up everything we set

951
00:33:51,679 --> 00:33:54,399
up

952
00:33:52,480 --> 00:33:57,440
uh everything from

953
00:33:54,399 --> 00:33:59,039
ip tables to apache to docker to

954
00:33:57,440 --> 00:34:00,559
the start scripts the stop scripts

955
00:33:59,039 --> 00:34:02,080
everything and so it's all there it's

956
00:34:00,559 --> 00:34:03,679
all been tested

957
00:34:02,080 --> 00:34:06,159
um

958
00:34:03,679 --> 00:34:07,840
it's uh yeah that's just the ansible

959
00:34:06,159 --> 00:34:09,919
bits and pieces

960
00:34:07,840 --> 00:34:12,560
um

961
00:34:09,919 --> 00:34:14,000
another kind of cool thing that we do uh

962
00:34:12,560 --> 00:34:16,639
because this is a sort of complete

963
00:34:14,000 --> 00:34:18,960
end-to-end test as part of the the paste

964
00:34:16,639 --> 00:34:22,480
test we actually start up a headless

965
00:34:18,960 --> 00:34:23,919
browser and point that at

966
00:34:22,480 --> 00:34:26,159
the service as well

967
00:34:23,919 --> 00:34:26,960
and so you'll see one of the artifacts

968
00:34:26,159 --> 00:34:29,119
is

969
00:34:26,960 --> 00:34:30,879
called screenshots and we'll actually go

970
00:34:29,119 --> 00:34:32,800
and take a screenshot of the service

971
00:34:30,879 --> 00:34:35,200
that we set up in our

972
00:34:32,800 --> 00:34:36,720
production environment uh our ephemeral

973
00:34:35,200 --> 00:34:38,960
production environment

974
00:34:36,720 --> 00:34:40,560
and you know this is the actual

975
00:34:38,960 --> 00:34:41,599
screenshot here you can see that instead

976
00:34:40,560 --> 00:34:43,280
of saying

977
00:34:41,599 --> 00:34:45,119
saying new pace before now it says

978
00:34:43,280 --> 00:34:48,879
please enter your new pace that was the

979
00:34:45,119 --> 00:34:50,480
change we made so we we have sort of 100

980
00:34:48,879 --> 00:34:52,079
confidence that

981
00:34:50,480 --> 00:34:54,879
the changes we made

982
00:34:52,079 --> 00:34:57,599
uh are deployable have worked

983
00:34:54,879 --> 00:35:00,640
uh this is running now the python 3.9

984
00:34:57,599 --> 00:35:03,839
container uh we know that our api change

985
00:35:00,640 --> 00:35:05,200
has has been made has applied

986
00:35:03,839 --> 00:35:06,480
um

987
00:35:05,200 --> 00:35:08,240
you know this is

988
00:35:06,480 --> 00:35:09,599
this is everything we want from our ci

989
00:35:08,240 --> 00:35:11,359
system we have

990
00:35:09,599 --> 00:35:12,720
great confidence that everything is

991
00:35:11,359 --> 00:35:14,800
going to work when we push this into

992
00:35:12,720 --> 00:35:16,960
production

993
00:35:14,800 --> 00:35:18,560
yeah just calling out that we

994
00:35:16,960 --> 00:35:20,160
did the main thing

995
00:35:18,560 --> 00:35:21,040
so

996
00:35:20,160 --> 00:35:22,480
you know

997
00:35:21,040 --> 00:35:25,040
deployment

998
00:35:22,480 --> 00:35:26,720
is really boring that there's

999
00:35:25,040 --> 00:35:30,000
we we run a separate job for doing

1000
00:35:26,720 --> 00:35:32,240
deployment so instead of um

1001
00:35:30,000 --> 00:35:34,480
you know system config run paste is the

1002
00:35:32,240 --> 00:35:36,560
job that runs on the unprivileged

1003
00:35:34,480 --> 00:35:39,359
changes that that come in

1004
00:35:36,560 --> 00:35:42,079
um once a change has been reviewed and

1005
00:35:39,359 --> 00:35:44,640
committed then a different job runs the

1006
00:35:42,079 --> 00:35:46,720
infraprod service paste but it

1007
00:35:44,640 --> 00:35:48,240
essentially does you know exactly the

1008
00:35:46,720 --> 00:35:50,400
same thing

1009
00:35:48,240 --> 00:35:51,599
except it targets the production machine

1010
00:35:50,400 --> 00:35:54,560
instead of the

1011
00:35:51,599 --> 00:35:54,560
ephemeral nodes

1012
00:35:54,640 --> 00:36:00,560
so this is actually the the top level

1013
00:35:57,760 --> 00:36:03,440
playbook that deploys the paste service

1014
00:36:00,560 --> 00:36:05,520
uh you know nothing very exciting there

1015
00:36:03,440 --> 00:36:07,760
you can go to our system config recapo

1016
00:36:05,520 --> 00:36:08,960
and have a look at it if you like

1017
00:36:07,760 --> 00:36:10,960
so

1018
00:36:08,960 --> 00:36:13,200
you know it's just two two branches of

1019
00:36:10,960 --> 00:36:14,560
the same thing um

1020
00:36:13,200 --> 00:36:17,359
when we're using

1021
00:36:14,560 --> 00:36:21,440
when uh unprivileged change comes in the

1022
00:36:17,359 --> 00:36:25,200
ephemeral bastion host runs this service

1023
00:36:21,440 --> 00:36:27,440
uh against the um

1024
00:36:25,200 --> 00:36:28,640
ephemeral paste node and that's what

1025
00:36:27,440 --> 00:36:29,440
happens

1026
00:36:28,640 --> 00:36:31,599
uh

1027
00:36:29,440 --> 00:36:33,440
in the production side we run exactly

1028
00:36:31,599 --> 00:36:35,599
the same playbook

1029
00:36:33,440 --> 00:36:36,400
but just against the production service

1030
00:36:35,599 --> 00:36:38,079
so

1031
00:36:36,400 --> 00:36:40,560
there's nothing different and there's

1032
00:36:38,079 --> 00:36:42,320
very little to go wrong

1033
00:36:40,560 --> 00:36:45,280
um

1034
00:36:42,320 --> 00:36:46,160
of course you know i'm i'm glossing over

1035
00:36:45,280 --> 00:36:48,400
things

1036
00:36:46,160 --> 00:36:51,119
of course production is different to

1037
00:36:48,400 --> 00:36:54,280
what we can possibly set up in our

1038
00:36:51,119 --> 00:36:58,560
um sort of ephemeral testing situation

1039
00:36:54,280 --> 00:37:00,160
a couple of things that come to mind

1040
00:36:58,560 --> 00:37:03,280
is let's encrypt

1041
00:37:00,160 --> 00:37:05,520
or basically you know ssl

1042
00:37:03,280 --> 00:37:05,520
um

1043
00:37:06,160 --> 00:37:12,000
ssl setup of services and ci you know we

1044
00:37:10,160 --> 00:37:14,000
just have self-signed certificates while

1045
00:37:12,000 --> 00:37:15,599
in production of course we have

1046
00:37:14,000 --> 00:37:18,240
um

1047
00:37:15,599 --> 00:37:20,839
authenticated certificates that we use

1048
00:37:18,240 --> 00:37:22,480
let's encrypt to get those to

1049
00:37:20,839 --> 00:37:25,040
certificates um

1050
00:37:22,480 --> 00:37:26,240
so that that sort of changes things

1051
00:37:25,040 --> 00:37:27,920
um

1052
00:37:26,240 --> 00:37:31,359
the other thing that we have to worry

1053
00:37:27,920 --> 00:37:34,560
about in production is that um

1054
00:37:31,359 --> 00:37:36,800
so the build set was the the set of

1055
00:37:34,560 --> 00:37:38,400
jobs that run against a particular

1056
00:37:36,800 --> 00:37:39,200
change that comes in

1057
00:37:38,400 --> 00:37:41,839
uh

1058
00:37:39,200 --> 00:37:43,200
we do actually have room for

1059
00:37:41,839 --> 00:37:45,839
um

1060
00:37:43,200 --> 00:37:49,760
parallel jobs to be running it

1061
00:37:45,839 --> 00:37:52,640
in wild in in in the wild so in

1062
00:37:49,760 --> 00:37:54,240
production we have um periodic jobs that

1063
00:37:52,640 --> 00:37:55,760
are running

1064
00:37:54,240 --> 00:37:57,839
for example

1065
00:37:55,760 --> 00:38:00,800
uh the the let's encrypt example

1066
00:37:57,839 --> 00:38:02,800
previously um you know let's encrypt

1067
00:38:00,800 --> 00:38:04,720
um

1068
00:38:02,800 --> 00:38:05,599
certificates expire after a couple of

1069
00:38:04,720 --> 00:38:08,240
months

1070
00:38:05,599 --> 00:38:09,599
we we have a periodic job that um goes

1071
00:38:08,240 --> 00:38:12,400
through and makes sure that all our

1072
00:38:09,599 --> 00:38:14,240
certificates are uh up to date so you

1073
00:38:12,400 --> 00:38:15,680
have to account for the fact that

1074
00:38:14,240 --> 00:38:16,640
somebody might have pushed the change

1075
00:38:15,680 --> 00:38:18,640
while

1076
00:38:16,640 --> 00:38:20,400
uh you know one of these periodic jobs

1077
00:38:18,640 --> 00:38:22,480
is running in the background to to keep

1078
00:38:20,400 --> 00:38:24,480
thing to keep things going

1079
00:38:22,480 --> 00:38:27,760
uh zuul actually

1080
00:38:24,480 --> 00:38:30,160
um provides ways for you to to deal with

1081
00:38:27,760 --> 00:38:32,400
that at the zoo level uh it gives you

1082
00:38:30,160 --> 00:38:34,720
semaphores that you can use so jobs

1083
00:38:32,400 --> 00:38:35,839
won't start running on top of each other

1084
00:38:34,720 --> 00:38:38,160
um

1085
00:38:35,839 --> 00:38:39,599
but you know this is something you need

1086
00:38:38,160 --> 00:38:41,280
to deal with um

1087
00:38:39,599 --> 00:38:42,800
in production

1088
00:38:41,280 --> 00:38:46,240
uh

1089
00:38:42,800 --> 00:38:47,440
logs is a big one so obviously in the uh

1090
00:38:46,240 --> 00:38:49,040
ephemeral

1091
00:38:47,440 --> 00:38:51,200
testing situation

1092
00:38:49,040 --> 00:38:54,000
uh where you're setting yourself up with

1093
00:38:51,200 --> 00:38:56,800
you know just just all test data and on

1094
00:38:54,000 --> 00:38:58,960
ephemeral nodes uh we grab all the logs

1095
00:38:56,800 --> 00:39:02,560
and everything's just um

1096
00:38:58,960 --> 00:39:08,400
put back in and you see it in uci jobs

1097
00:39:02,560 --> 00:39:10,320
we don't publish uh the logs for the um

1098
00:39:08,400 --> 00:39:11,280
production deployments

1099
00:39:10,320 --> 00:39:13,599
um

1100
00:39:11,280 --> 00:39:16,240
i mean we would like to but of course

1101
00:39:13,599 --> 00:39:18,480
the practicality is that you know

1102
00:39:16,240 --> 00:39:20,160
you can leak secrets fairly easily from

1103
00:39:18,480 --> 00:39:21,599
from that side of things

1104
00:39:20,160 --> 00:39:22,400
um

1105
00:39:21,599 --> 00:39:24,960
so

1106
00:39:22,400 --> 00:39:26,720
you know it if we focused on auditing

1107
00:39:24,960 --> 00:39:28,640
maybe it would be something maybe would

1108
00:39:26,720 --> 00:39:30,240
even be good because you know anything

1109
00:39:28,640 --> 00:39:31,520
you did leak you have to make sure that

1110
00:39:30,240 --> 00:39:32,800
you can rotate

1111
00:39:31,520 --> 00:39:34,720
in an instant

1112
00:39:32,800 --> 00:39:36,000
uh

1113
00:39:34,720 --> 00:39:38,800
you know

1114
00:39:36,000 --> 00:39:40,960
it would be good um but it's not

1115
00:39:38,800 --> 00:39:43,839
something we have at the moment

1116
00:39:40,960 --> 00:39:45,520
um and also not

1117
00:39:43,839 --> 00:39:47,440
not everything is sort of completely

1118
00:39:45,520 --> 00:39:49,200
fault tolerant in that

1119
00:39:47,440 --> 00:39:51,920
we can just sort of

1120
00:39:49,200 --> 00:39:53,040
commit a change and push it and have it

1121
00:39:51,920 --> 00:39:55,359
deploy

1122
00:39:53,040 --> 00:39:58,160
uh straight away

1123
00:39:55,359 --> 00:40:00,480
with zero downtime

1124
00:39:58,160 --> 00:40:03,119
some of our services like

1125
00:40:00,480 --> 00:40:03,119
gta

1126
00:40:03,280 --> 00:40:06,800
which you know runs our code browsing

1127
00:40:05,440 --> 00:40:08,800
and stuff like that

1128
00:40:06,800 --> 00:40:12,160
we do run in a fault tolerant manner

1129
00:40:08,800 --> 00:40:13,760
than that other services we don't um so

1130
00:40:12,160 --> 00:40:15,920
they have a little bit more manual

1131
00:40:13,760 --> 00:40:17,599
interaction

1132
00:40:15,920 --> 00:40:18,480
we consider that a bargain and we're

1133
00:40:17,599 --> 00:40:20,000
always

1134
00:40:18,480 --> 00:40:21,119
working to

1135
00:40:20,000 --> 00:40:22,960
basically

1136
00:40:21,119 --> 00:40:24,880
take ourselves out of the loop as much

1137
00:40:22,960 --> 00:40:26,400
as we can but

1138
00:40:24,880 --> 00:40:27,280
there definitely are still points that

1139
00:40:26,400 --> 00:40:29,119
sort of

1140
00:40:27,280 --> 00:40:30,560
need a little bit more interaction than

1141
00:40:29,119 --> 00:40:32,000
we'd like

1142
00:40:30,560 --> 00:40:34,400
um

1143
00:40:32,000 --> 00:40:37,359
and but it's all solvable i mean i think

1144
00:40:34,400 --> 00:40:39,359
that the general principle here of of

1145
00:40:37,359 --> 00:40:40,480
testing like production and and pushing

1146
00:40:39,359 --> 00:40:44,240
like this

1147
00:40:40,480 --> 00:40:45,680
is is something really um quite valuable

1148
00:40:44,240 --> 00:40:47,680
um

1149
00:40:45,680 --> 00:40:49,839
i just wanted to show an actual

1150
00:40:47,680 --> 00:40:52,079
practical example of

1151
00:40:49,839 --> 00:40:54,480
how sort of someone who came along and

1152
00:40:52,079 --> 00:40:56,319
contributed to our infrastructure uh

1153
00:40:54,480 --> 00:40:58,560
that didn't have

1154
00:40:56,319 --> 00:41:00,480
permission as such what wasn't sort of

1155
00:40:58,560 --> 00:41:02,720
part of the existing

1156
00:41:00,480 --> 00:41:04,800
group of people that generally uh work

1157
00:41:02,720 --> 00:41:07,280
on such things

1158
00:41:04,800 --> 00:41:08,960
this was just standing up at a service

1159
00:41:07,280 --> 00:41:11,359
the refstax service

1160
00:41:08,960 --> 00:41:13,359
uh you can check out the change yourself

1161
00:41:11,359 --> 00:41:14,880
uh you know we went through a lot of

1162
00:41:13,359 --> 00:41:16,880
back and forth you know there was a lot

1163
00:41:14,880 --> 00:41:18,240
of learning for everyone

1164
00:41:16,880 --> 00:41:19,599
um

1165
00:41:18,240 --> 00:41:21,440
47

1166
00:41:19,599 --> 00:41:23,599
you know different reviews

1167
00:41:21,440 --> 00:41:26,000
but you know because of the way that we

1168
00:41:23,599 --> 00:41:28,400
we set up this system you can push your

1169
00:41:26,000 --> 00:41:30,480
changes you can see the changes

1170
00:41:28,400 --> 00:41:33,359
we you we could

1171
00:41:30,480 --> 00:41:35,440
essentially you know

1172
00:41:33,359 --> 00:41:38,079
have our sort of simulated production

1173
00:41:35,440 --> 00:41:40,880
environment running these changes

1174
00:41:38,079 --> 00:41:42,960
and the developer got a lot of feedback

1175
00:41:40,880 --> 00:41:45,440
about you know how to how to build up

1176
00:41:42,960 --> 00:41:47,280
the the deployment steps

1177
00:41:45,440 --> 00:41:49,760
and then by the time we got to the end

1178
00:41:47,280 --> 00:41:51,440
after all the reviews

1179
00:41:49,760 --> 00:41:53,680
all that was really left to do was

1180
00:41:51,440 --> 00:41:56,319
commit and push the change and off it

1181
00:41:53,680 --> 00:41:58,160
went and you know popped itself into

1182
00:41:56,319 --> 00:42:00,960
production and

1183
00:41:58,160 --> 00:42:03,040
we sort of knew it would work because

1184
00:42:00,960 --> 00:42:06,800
putting it into production was really so

1185
00:42:03,040 --> 00:42:08,480
close to what we've been testing

1186
00:42:06,800 --> 00:42:10,960
that

1187
00:42:08,480 --> 00:42:13,920
it just sort of it just works

1188
00:42:10,960 --> 00:42:16,560
uh yeah this is a good example of how

1189
00:42:13,920 --> 00:42:19,040
our open infrastructure works to allow

1190
00:42:16,560 --> 00:42:20,079
people to get in and

1191
00:42:19,040 --> 00:42:21,920
help us

1192
00:42:20,079 --> 00:42:24,319
build and maintain the services that we

1193
00:42:21,920 --> 00:42:24,319
provide

1194
00:42:25,040 --> 00:42:30,480
so if you're sort of interested

1195
00:42:27,920 --> 00:42:33,040
in finding out how this all actually

1196
00:42:30,480 --> 00:42:34,880
works it you go and check out zul zulu

1197
00:42:33,040 --> 00:42:36,880
ci there's talks there

1198
00:42:34,880 --> 00:42:38,480
uh there's specific zuul talks that go

1199
00:42:36,880 --> 00:42:40,640
into a lot more detail on zuul and how

1200
00:42:38,480 --> 00:42:43,359
it all works

1201
00:42:40,640 --> 00:42:45,839
vex host is a big supporter of opendev

1202
00:42:43,359 --> 00:42:48,160
they offer a managed zool service

1203
00:42:45,839 --> 00:42:51,119
uh and the other thing to check out is

1204
00:42:48,160 --> 00:42:52,480
the um software factory project

1205
00:42:51,119 --> 00:42:55,760
from red hat

1206
00:42:52,480 --> 00:42:58,160
uh that that's not so much um

1207
00:42:55,760 --> 00:43:00,240
focused on this sort of cd side of zul

1208
00:42:58,160 --> 00:43:02,400
like this but if you think that this is

1209
00:43:00,240 --> 00:43:04,480
sort of interesting technology then

1210
00:43:02,400 --> 00:43:08,319
software software factory can really

1211
00:43:04,480 --> 00:43:10,319
help you get a garrett and a zool and

1212
00:43:08,319 --> 00:43:11,280
all these other services down there sort

1213
00:43:10,319 --> 00:43:14,000
of

1214
00:43:11,280 --> 00:43:16,000
deployed in a really all-in-one manner

1215
00:43:14,000 --> 00:43:17,040
it's a great way to sort of

1216
00:43:16,000 --> 00:43:19,380
set up a

1217
00:43:17,040 --> 00:43:21,280
software factory i guess

1218
00:43:19,380 --> 00:43:22,160
[Music]

1219
00:43:21,280 --> 00:43:24,000
so

1220
00:43:22,160 --> 00:43:24,839
that's it i'm about out of time thank

1221
00:43:24,000 --> 00:43:28,240
you

1222
00:43:24,839 --> 00:43:31,040
um yeah i just wanted to sort of give

1223
00:43:28,240 --> 00:43:32,960
you a bit of an insight into how we

1224
00:43:31,040 --> 00:43:37,440
do this open infrastructure that runs on

1225
00:43:32,960 --> 00:43:37,440
top of an open open source stack

1226
00:43:38,160 --> 00:43:43,440
how this ability to do this cross repro

1227
00:43:40,480 --> 00:43:46,720
testing and this speculative testing

1228
00:43:43,440 --> 00:43:48,240
and to test like we deploy it really um

1229
00:43:46,720 --> 00:43:49,680
allows us to have this open

1230
00:43:48,240 --> 00:43:51,040
infrastructure

1231
00:43:49,680 --> 00:43:52,720
and

1232
00:43:51,040 --> 00:43:55,200
hopefully you can see that having this

1233
00:43:52,720 --> 00:43:58,400
open string open infrastructure it's

1234
00:43:55,200 --> 00:44:00,240
just such a false multiplier that allows

1235
00:43:58,400 --> 00:44:01,200
other contributors to come in and help

1236
00:44:00,240 --> 00:44:03,599
you

1237
00:44:01,200 --> 00:44:05,760
manage your own infrastructure and for

1238
00:44:03,599 --> 00:44:08,480
the people that are sort of

1239
00:44:05,760 --> 00:44:10,640
more in the weeds of it

1240
00:44:08,480 --> 00:44:12,480
the sort of core reviewers who are going

1241
00:44:10,640 --> 00:44:15,119
to be looking at the the changes coming

1242
00:44:12,480 --> 00:44:17,599
in it just takes so much load off them

1243
00:44:15,119 --> 00:44:19,119
knowing that the ci

1244
00:44:17,599 --> 00:44:21,119
is so good that

1245
00:44:19,119 --> 00:44:22,960
it's going to be picking up

1246
00:44:21,119 --> 00:44:26,079
any of the problems

1247
00:44:22,960 --> 00:44:31,040
before they even have to look at it

1248
00:44:26,079 --> 00:44:34,160
to chat with us uh opendev on oftc irc

1249
00:44:31,040 --> 00:44:35,440
or a matrix uh there's a zuul channel on

1250
00:44:34,160 --> 00:44:37,359
matrix

1251
00:44:35,440 --> 00:44:39,520
uh you'll find plenty of people who'd be

1252
00:44:37,359 --> 00:44:42,400
happy to talk to you

1253
00:44:39,520 --> 00:44:44,240
all about it and

1254
00:44:42,400 --> 00:44:47,119
that's it

1255
00:44:44,240 --> 00:44:48,880
cool thank you so much uh ian that was a

1256
00:44:47,119 --> 00:44:50,079
really interesting talk uh we do have

1257
00:44:48,880 --> 00:44:51,680
one question which will pop into the

1258
00:44:50,079 --> 00:44:53,280
text chat afterwards if you want to join

1259
00:44:51,680 --> 00:44:54,319
that um

1260
00:44:53,280 --> 00:44:55,920
but

1261
00:44:54,319 --> 00:44:59,760
yeah thank you so much next up is our

1262
00:44:55,920 --> 00:45:01,280
lunch break uh so come back at 1 30 p.m

1263
00:44:59,760 --> 00:45:03,520
uh in this theater we've got michael

1264
00:45:01,280 --> 00:45:05,440
cohen with velociraptor digging deeper

1265
00:45:03,520 --> 00:45:07,359
in linux for dfir

1266
00:45:05,440 --> 00:45:08,800
investigation

1267
00:45:07,359 --> 00:45:10,480
thank you so much for your talking have

1268
00:45:08,800 --> 00:45:13,640
a great rest of your day

1269
00:45:10,480 --> 00:45:13,640
thank you