1
00:00:04,960 --> 00:00:19,999
[Music]

2
00:00:21,119 --> 00:00:25,199
greetings greetings and welcome back to

3
00:00:22,840 --> 00:00:26,359
the devops track sorry Dev oops track

4
00:00:25,199 --> 00:00:28,720
oops

5
00:00:26,359 --> 00:00:31,080
oops uh thank you for joining us here

6
00:00:28,720 --> 00:00:33,640
today at Pon it is great to see everyone

7
00:00:31,080 --> 00:00:36,320
here in the audience uh kicking off in

8
00:00:33,640 --> 00:00:38,320
our first session now we have Evan uh to

9
00:00:36,320 --> 00:00:39,840
talk to us about who tests the testers

10
00:00:38,320 --> 00:00:42,600
making testing

11
00:00:39,840 --> 00:00:44,480
pipelines Evan is a senior software

12
00:00:42,600 --> 00:00:46,480
engineer whose passions lie in improving

13
00:00:44,480 --> 00:00:48,840
the developer experience by enhancing

14
00:00:46,480 --> 00:00:52,359
code Health optimizing workflows and

15
00:00:48,840 --> 00:00:54,000
working towards no human errors. comom

16
00:00:52,359 --> 00:00:55,600
when he's not thinking about security

17
00:00:54,000 --> 00:00:57,399
informatics or giving talks about his

18
00:00:55,600 --> 00:01:00,199
many projects you'll find him noming on

19
00:00:57,399 --> 00:01:03,800
subway cookies or chasing bunny rabbits

20
00:01:00,199 --> 00:01:03,800
take it away Evan thank you very

21
00:01:07,479 --> 00:01:13,640
much I I love this screen saver I think

22
00:01:10,320 --> 00:01:16,400
it's very fitting for the talk um but

23
00:01:13,640 --> 00:01:20,400
yes hello everyone uh hope you're having

24
00:01:16,400 --> 00:01:23,240
a great day um so I'm I'm Evan as you

25
00:01:20,400 --> 00:01:26,320
might have heard and I want to talk

26
00:01:23,240 --> 00:01:29,079
about pipelines because while working on

27
00:01:26,320 --> 00:01:32,000
the platform team I've developed on

28
00:01:29,079 --> 00:01:35,320
pipelines as part of a gitlab repo to

29
00:01:32,000 --> 00:01:35,320
run tests for our

30
00:01:35,600 --> 00:01:41,320
codebase but I noticed there were quite

31
00:01:38,399 --> 00:01:43,840
a few oopsies that were made that could

32
00:01:41,320 --> 00:01:47,439
have been prevented if there were

33
00:01:43,840 --> 00:01:50,799
tests and it got me thinking if we as

34
00:01:47,439 --> 00:01:54,200
the platform team help developers run

35
00:01:50,799 --> 00:01:57,320
tests for their code who helps us the

36
00:01:54,200 --> 00:01:58,640
platform team write tests for running

37
00:01:57,320 --> 00:02:01,119
the

38
00:01:58,640 --> 00:02:03,119
tests so so today we'll unpack that

39
00:02:01,119 --> 00:02:06,600
topic in three easy

40
00:02:03,119 --> 00:02:09,640
steps I'll briefly introduce how to make

41
00:02:06,600 --> 00:02:12,560
a pipeline and what it

42
00:02:09,640 --> 00:02:15,720
is show how they can

43
00:02:12,560 --> 00:02:19,319
break and finally Deep dive into the

44
00:02:15,720 --> 00:02:22,560
methods to stop them from

45
00:02:19,319 --> 00:02:24,599
breaking all right so for this section

46
00:02:22,560 --> 00:02:26,640
could I please ask everyone to raise

47
00:02:24,599 --> 00:02:29,040
their

48
00:02:26,640 --> 00:02:30,280
hand okay wonderful we're going to do a

49
00:02:29,040 --> 00:02:32,200
little survey so keep keep your hand

50
00:02:30,280 --> 00:02:34,280
rais also I'm going to ask to take a

51
00:02:32,200 --> 00:02:37,440
photo because I didn't think that would

52
00:02:34,280 --> 00:02:37,440
work and I think this is really

53
00:02:38,120 --> 00:02:42,440
funny wonderful you look great Okay cool

54
00:02:41,319 --> 00:02:44,560
so keep your hands raised keep your

55
00:02:42,440 --> 00:02:46,080
hands raised okay wonderful um I I'll

56
00:02:44,560 --> 00:02:48,720
let you know when you can you can lower

57
00:02:46,080 --> 00:02:50,680
them okay so keep your hand raised if

58
00:02:48,720 --> 00:02:54,280
you know what a pipeline

59
00:02:50,680 --> 00:02:56,000
is okay so we got a few hands down don't

60
00:02:54,280 --> 00:02:57,959
put your hands down just yet but keep

61
00:02:56,000 --> 00:03:01,080
your hands raised if you've worked with

62
00:02:57,959 --> 00:03:03,560
a pipeline with r being code or work or

63
00:03:01,080 --> 00:03:07,280
whatever okay still quite a few all

64
00:03:03,560 --> 00:03:09,159
right now keep it up if you've written a

65
00:03:07,280 --> 00:03:10,599
pipeline okay so there's still quite a

66
00:03:09,159 --> 00:03:13,239
few of you that have your hands raised

67
00:03:10,599 --> 00:03:15,640
and that's great all right one last one

68
00:03:13,239 --> 00:03:18,440
keep it raised if you've had a pipeline

69
00:03:15,640 --> 00:03:21,879
that you've written fail on

70
00:03:18,440 --> 00:03:25,239
you okay none of you lowered your

71
00:03:21,879 --> 00:03:27,319
hands I was not expecting this okay

72
00:03:25,239 --> 00:03:30,000
we're in for a great talk all

73
00:03:27,319 --> 00:03:32,080
right so so that we're all in the same

74
00:03:30,000 --> 00:03:34,239
level uh for those of you who hasn't

75
00:03:32,080 --> 00:03:37,599
seen a p plane before they can look

76
00:03:34,239 --> 00:03:39,640
something like this and to put it simply

77
00:03:37,599 --> 00:03:41,519
you can think of them as a factory line

78
00:03:39,640 --> 00:03:44,239
where each step performs different

79
00:03:41,519 --> 00:03:46,680
operations to get to the final product

80
00:03:44,239 --> 00:03:48,400
but in our case our operations can look

81
00:03:46,680 --> 00:03:53,200
something like

82
00:03:48,400 --> 00:03:55,120
setup build test with our final

83
00:03:53,200 --> 00:03:57,120
operation deploying this built and

84
00:03:55,120 --> 00:04:02,360
tested

85
00:03:57,120 --> 00:04:04,480
product so why are p P lines good and

86
00:04:02,360 --> 00:04:07,760
why should you care about

87
00:04:04,480 --> 00:04:09,760
them well they allow us in one case to

88
00:04:07,760 --> 00:04:12,200
create a streamlined process that

89
00:04:09,760 --> 00:04:14,079
automates what would otherwise be very

90
00:04:12,200 --> 00:04:16,880
manual and

91
00:04:14,079 --> 00:04:19,799
tedious and in other cases they act as a

92
00:04:16,880 --> 00:04:22,479
quality control similar to a factory

93
00:04:19,799 --> 00:04:25,440
line if any stage of the factory line

94
00:04:22,479 --> 00:04:28,880
Fails Like building or quality

95
00:04:25,440 --> 00:04:31,600
assurance that item or in our case some

96
00:04:28,880 --> 00:04:34,199
code won't progress to the next stage

97
00:04:31,600 --> 00:04:36,880
and thus won't be allowed to

98
00:04:34,199 --> 00:04:38,919
merge for this presentation I'll mainly

99
00:04:36,880 --> 00:04:41,240
be talking about gitlab

100
00:04:38,919 --> 00:04:44,120
pipelines however if you are a GitHub

101
00:04:41,240 --> 00:04:47,199
user or you use any other kinds of

102
00:04:44,120 --> 00:04:49,560
pipeline tooling most of the content

103
00:04:47,199 --> 00:04:51,520
should be General enough to be applied

104
00:04:49,560 --> 00:04:53,080
and additional resources will be given

105
00:04:51,520 --> 00:04:57,320
at the

106
00:04:53,080 --> 00:05:02,160
end okay so to make a gitlab pipeline we

107
00:04:57,320 --> 00:05:04,759
first need to create a gitlab c. yaml

108
00:05:02,160 --> 00:05:06,919
file that gitlab will use as

109
00:05:04,759 --> 00:05:08,759
configuration and that's kind of a

110
00:05:06,919 --> 00:05:12,080
mouthful so I'm just going to refer to

111
00:05:08,759 --> 00:05:14,840
it as a CI config file from from here

112
00:05:12,080 --> 00:05:17,160
on and if you're new to yamal think of

113
00:05:14,840 --> 00:05:21,440
it as a way to store dictionaries and

114
00:05:17,160 --> 00:05:24,160
lists similar to Jason but more human

115
00:05:21,440 --> 00:05:26,639
friendly we'll make a simple one now

116
00:05:24,160 --> 00:05:30,280
that one can use on their python repo to

117
00:05:26,639 --> 00:05:32,479
ensure that tests pass before merging

118
00:05:30,280 --> 00:05:35,199
here we Define that we wanted the job to

119
00:05:32,479 --> 00:05:36,960
run the latest version of python within

120
00:05:35,199 --> 00:05:39,479
a Docker

121
00:05:36,960 --> 00:05:41,720
image and then we'll Define a job with a

122
00:05:39,479 --> 00:05:44,960
name test

123
00:05:41,720 --> 00:05:47,600
job and Below we'll Define script as the

124
00:05:44,960 --> 00:05:51,039
command to run for that job which for us

125
00:05:47,600 --> 00:05:55,080
will be a python file called

126
00:05:51,039 --> 00:05:56,919
test.py this is quite simple um probably

127
00:05:55,080 --> 00:05:59,120
one of the simplest pipelines we can

128
00:05:56,919 --> 00:06:01,759
create that being said it probably

129
00:05:59,120 --> 00:06:04,880
shouldn't be use as a reference as it's

130
00:06:01,759 --> 00:06:07,759
UTI utilizing a lot of what gitlab um is

131
00:06:04,880 --> 00:06:10,039
offering by default and we can get away

132
00:06:07,759 --> 00:06:13,680
with not having to install any packages

133
00:06:10,039 --> 00:06:15,520
as none are being used so if you are

134
00:06:13,680 --> 00:06:18,000
looking for a reference pipeline I'll

135
00:06:15,520 --> 00:06:21,280
have some links in the resources at the

136
00:06:18,000 --> 00:06:23,919
end that being said given how simple

137
00:06:21,280 --> 00:06:26,000
this pipeline example is it probably

138
00:06:23,919 --> 00:06:28,680
doesn't best mimic the issues that a

139
00:06:26,000 --> 00:06:31,199
real pipeline might so let's extend it

140
00:06:28,680 --> 00:06:34,120
to do so

141
00:06:31,199 --> 00:06:37,319
to do this we'll add some rules which

142
00:06:34,120 --> 00:06:40,479
decide on what conditions the job

143
00:06:37,319 --> 00:06:42,919
runs we then specify that we want the

144
00:06:40,479 --> 00:06:43,919
job to run whenever there's changes that

145
00:06:42,919 --> 00:06:48,000
have been

146
00:06:43,919 --> 00:06:50,440
made such as changes to any python

147
00:06:48,000 --> 00:06:54,280
files this is so that we can speed up

148
00:06:50,440 --> 00:06:57,080
our pipelines by not unnecessary

149
00:06:54,280 --> 00:06:59,879
unnecessarily running a costly job on

150
00:06:57,080 --> 00:07:02,160
unrelated changes to other files within

151
00:06:59,879 --> 00:07:03,960
our repo such as

152
00:07:02,160 --> 00:07:06,199
documentation another thing that we

153
00:07:03,960 --> 00:07:08,280
might want to extend on is having to run

154
00:07:06,199 --> 00:07:10,840
different types of testing jobs you

155
00:07:08,280 --> 00:07:12,479
might not want to just have a single

156
00:07:10,840 --> 00:07:16,560
test python

157
00:07:12,479 --> 00:07:19,160
job for example a database test

158
00:07:16,560 --> 00:07:22,599
job which we could set up in a similar

159
00:07:19,160 --> 00:07:25,199
way except it might uh depend on some

160
00:07:22,599 --> 00:07:28,039
test databases to have been

161
00:07:25,199 --> 00:07:30,680
created so for that we require a new

162
00:07:28,039 --> 00:07:32,560
setup job to be run

163
00:07:30,680 --> 00:07:33,639
which needs to be run under the same set

164
00:07:32,560 --> 00:07:36,280
of

165
00:07:33,639 --> 00:07:38,680
changes and additionally we need some

166
00:07:36,280 --> 00:07:41,960
way of specifying that the database test

167
00:07:38,680 --> 00:07:43,800
job can only run once the database setup

168
00:07:41,960 --> 00:07:46,400
job is

169
00:07:43,800 --> 00:07:48,560
finished to do that we use the needs

170
00:07:46,400 --> 00:07:51,319
keyword which specifies that the

171
00:07:48,560 --> 00:07:54,280
database test job needs the database

172
00:07:51,319 --> 00:07:57,440
setup job to finish successfully before

173
00:07:54,280 --> 00:08:00,159
tests can be started and completed

174
00:07:57,440 --> 00:08:02,400
correctly with the idea being is that we

175
00:08:00,159 --> 00:08:04,919
want as much running in parallel as

176
00:08:02,400 --> 00:08:07,120
possible such that our previous tests

177
00:08:04,919 --> 00:08:10,440
don't have to wait for setting up

178
00:08:07,120 --> 00:08:13,319
databases if they don't depend on

179
00:08:10,440 --> 00:08:16,240
them and so now what we've landed on is

180
00:08:13,319 --> 00:08:19,039
great it's efficient as an example and

181
00:08:16,240 --> 00:08:22,919
it works when it's merged

182
00:08:19,039 --> 00:08:26,759
in however as new PRS are made and time

183
00:08:22,919 --> 00:08:29,759
goes on one PR causes the pipeline to

184
00:08:26,759 --> 00:08:32,719
fail not because the tests themselves

185
00:08:29,759 --> 00:08:34,000
pass or fail but because the pipeline

186
00:08:32,719 --> 00:08:36,839
itself is

187
00:08:34,000 --> 00:08:39,240
incorrect gitlab will tell us that the

188
00:08:36,839 --> 00:08:42,719
database test job couldn't run because

189
00:08:39,240 --> 00:08:45,399
the database setup job didn't run now if

190
00:08:42,719 --> 00:08:47,640
you go back to our pipeline

191
00:08:45,399 --> 00:08:49,519
definition can anyone tell me why it

192
00:08:47,640 --> 00:08:51,080
didn't

193
00:08:49,519 --> 00:08:55,040
run

194
00:08:51,080 --> 00:08:58,200
PPU yes that's right there was a typo in

195
00:08:55,040 --> 00:09:00,800
the changes and so the database test job

196
00:08:58,200 --> 00:09:04,279
ran because there were changes to files

197
00:09:00,800 --> 00:09:06,079
ending in py but the database setup job

198
00:09:04,279 --> 00:09:08,040
didn't run because there weren't changes

199
00:09:06,079 --> 00:09:10,600
to files ending

200
00:09:08,040 --> 00:09:11,560
inpu and the reason I give this example

201
00:09:10,600 --> 00:09:15,000
is because this is something that's

202
00:09:11,560 --> 00:09:18,120
happened to me right it's just common

203
00:09:15,000 --> 00:09:20,760
error and so while it's great that we've

204
00:09:18,120 --> 00:09:22,240
all figured out why this happened and

205
00:09:20,760 --> 00:09:23,440
how the fix could just be a one

206
00:09:22,240 --> 00:09:26,360
character

207
00:09:23,440 --> 00:09:28,240
change in our line of work it's more

208
00:09:26,360 --> 00:09:30,519
important to understand how this was

209
00:09:28,240 --> 00:09:32,200
allowed to happen in the first place so

210
00:09:30,519 --> 00:09:34,079
it doesn't ever occur again and we don't

211
00:09:32,200 --> 00:09:36,600
have to spend time going and fixing all

212
00:09:34,079 --> 00:09:39,640
these one character

213
00:09:36,600 --> 00:09:41,959
changes so can anyone think of some

214
00:09:39,640 --> 00:09:45,680
examples for why this could have

215
00:09:41,959 --> 00:09:47,720
happened feel free to shout

216
00:09:45,680 --> 00:09:49,079
someone yes you could have had some

217
00:09:47,720 --> 00:09:51,560
careless developers you know they're

218
00:09:49,079 --> 00:09:55,839
just typing really quickly and was just

219
00:09:51,560 --> 00:09:55,839
easily missed anything else

220
00:09:59,640 --> 00:10:02,880
that's a good one I haven't thought of

221
00:10:00,920 --> 00:10:04,880
that one so the pipeline was updated to

222
00:10:02,880 --> 00:10:11,360
a new version that doesn't support the

223
00:10:04,880 --> 00:10:11,360
previous things okay cool yes set

224
00:10:12,000 --> 00:10:16,279
relies the setup job relies on

225
00:10:14,480 --> 00:10:20,320
configuration files that just happen to

226
00:10:16,279 --> 00:10:22,079
bepu you want to rerun set yeah yeah we

227
00:10:20,320 --> 00:10:26,120
don't want to keep having to rerun

228
00:10:22,079 --> 00:10:26,120
things y uh anything

229
00:10:27,440 --> 00:10:32,320
else yes lack of testing safety nets

230
00:10:37,240 --> 00:10:43,639
yes right yeah yeah so someone like

231
00:10:41,240 --> 00:10:44,680
right so I repeat it for the for the mic

232
00:10:43,639 --> 00:10:48,760
um

233
00:10:44,680 --> 00:10:49,959
someone someone ran their stuff locally

234
00:10:48,760 --> 00:10:52,320
and they had it but they hadn't

235
00:10:49,959 --> 00:10:54,560
committed it and so they wouldn't have

236
00:10:52,320 --> 00:10:56,760
noticed a problem in in the production

237
00:10:54,560 --> 00:11:00,040
yeah um some other ones could have been

238
00:10:56,760 --> 00:11:03,279
like the config is too complex maybe the

239
00:11:00,040 --> 00:11:06,839
reviewers weren't sharp enough um or we

240
00:11:03,279 --> 00:11:09,320
could have Ru in a better way to um use

241
00:11:06,839 --> 00:11:11,279
um don't repeat yourself and so forth

242
00:11:09,320 --> 00:11:14,160
but those are all good good

243
00:11:11,279 --> 00:11:17,240
points and to me I think that all of

244
00:11:14,160 --> 00:11:19,760
those are all human errors um and as you

245
00:11:17,240 --> 00:11:22,519
might have heard earlier I don't like

246
00:11:19,760 --> 00:11:24,839
human errors that could be avoided by a

247
00:11:22,519 --> 00:11:28,079
systematic change which would allow us

248
00:11:24,839 --> 00:11:30,440
to blame a system instead of a

249
00:11:28,079 --> 00:11:33,200
people so so I would say that at a

250
00:11:30,440 --> 00:11:36,120
higher level this occurred because while

251
00:11:33,200 --> 00:11:38,399
this configuration looks like code it

252
00:11:36,120 --> 00:11:43,160
isn't treated like code because we're

253
00:11:38,399 --> 00:11:46,959
used to testing code with tests but who

254
00:11:43,160 --> 00:11:50,360
tests the testing infrastructure with

255
00:11:46,959 --> 00:11:53,680
tests so now you might be thinking Evan

256
00:11:50,360 --> 00:11:56,279
this is great Insight uh but Solutions

257
00:11:53,680 --> 00:11:58,720
speak louder than problems how do I stop

258
00:11:56,279 --> 00:12:00,160
this from happening to me it's a great

259
00:11:58,720 --> 00:12:02,040
question

260
00:12:00,160 --> 00:12:03,680
to answer it there's a couple of methods

261
00:12:02,040 --> 00:12:06,920
that we'll dive

262
00:12:03,680 --> 00:12:09,639
into the first is validating our CI

263
00:12:06,920 --> 00:12:12,880
configuration using linking

264
00:12:09,639 --> 00:12:16,040
tools the second is running an endtoend

265
00:12:12,880 --> 00:12:18,440
pipeline before committing

266
00:12:16,040 --> 00:12:24,000
changes we'll then look at how we can

267
00:12:18,440 --> 00:12:26,720
run those pipeline jobs before sorry

268
00:12:24,000 --> 00:12:31,160
locally and how we can run static

269
00:12:26,720 --> 00:12:31,160
analysis testing on the CI configuration

270
00:12:31,680 --> 00:12:36,959
finally we'll look at dynamically

271
00:12:33,639 --> 00:12:39,680
generating pipelines using

272
00:12:36,959 --> 00:12:43,079
tools now I'm not going to claim that

273
00:12:39,680 --> 00:12:44,880
any single method is the best but I will

274
00:12:43,079 --> 00:12:48,360
cover the pros and cons to help you

275
00:12:44,880 --> 00:12:50,760
decide which one or combination works

276
00:12:48,360 --> 00:12:50,760
best for

277
00:12:51,279 --> 00:12:56,959
you so the first method we'll dive into

278
00:12:54,760 --> 00:13:00,040
is linting the CI config

279
00:12:56,959 --> 00:13:02,839
file linting is great because it's a

280
00:13:00,040 --> 00:13:06,320
fast method of finding problems that can

281
00:13:02,839 --> 00:13:09,160
be solved before going further and gab

282
00:13:06,320 --> 00:13:10,079
offers quite a bit here they have a vs

283
00:13:09,160 --> 00:13:12,160
code

284
00:13:10,079 --> 00:13:15,399
extension which will tell you within

285
00:13:12,160 --> 00:13:17,880
your editor whether your yaml is set up

286
00:13:15,399 --> 00:13:20,120
correctly they have a pipeline syntax

287
00:13:17,880 --> 00:13:22,560
Checker which will do the same thing as

288
00:13:20,120 --> 00:13:24,680
the extension but through a web UI and

289
00:13:22,560 --> 00:13:27,320
an API

290
00:13:24,680 --> 00:13:29,320
endpoint they also have a visualizer

291
00:13:27,320 --> 00:13:32,440
which gives a quick way of helping you

292
00:13:29,320 --> 00:13:35,079
see the dependencies between the

293
00:13:32,440 --> 00:13:36,880
jobs they have a validator which

294
00:13:35,079 --> 00:13:41,160
simulates a git push event on the

295
00:13:36,880 --> 00:13:43,839
default Branch with the rules and needs

296
00:13:41,160 --> 00:13:47,199
evaluated and finally they have a tool

297
00:13:43,839 --> 00:13:49,480
to expand your yaml into its full

298
00:13:47,199 --> 00:13:51,440
configuration which helps you see what

299
00:13:49,480 --> 00:13:54,440
the pipeline looks like with all of the

300
00:13:51,440 --> 00:13:57,120
special syntax

301
00:13:54,440 --> 00:13:59,800
expanded however all of these methods

302
00:13:57,120 --> 00:14:01,800
have their downsides too

303
00:13:59,800 --> 00:14:04,519
the VSS code extension as you might have

304
00:14:01,800 --> 00:14:07,279
guessed is only for vs

305
00:14:04,519 --> 00:14:09,360
code and even if you instead use the

306
00:14:07,279 --> 00:14:12,800
pipeline syntax Checker whether it be

307
00:14:09,360 --> 00:14:14,680
for the web UI or the endpoint they only

308
00:14:12,800 --> 00:14:18,360
tell you if there's something wrong with

309
00:14:14,680 --> 00:14:21,959
your schema not with how it

310
00:14:18,360 --> 00:14:23,639
runs the visualizer while handy doesn't

311
00:14:21,959 --> 00:14:26,079
help you understand the issues that we

312
00:14:23,639 --> 00:14:29,160
had around the pendencies with the needs

313
00:14:26,079 --> 00:14:31,399
and rules and thus won't uncover those

314
00:14:29,160 --> 00:14:34,639
issues up

315
00:14:31,399 --> 00:14:37,519
front the validator while it says it

316
00:14:34,639 --> 00:14:39,279
simulates and evaluates rules and needs

317
00:14:37,519 --> 00:14:41,720
and I could be misunderstanding here

318
00:14:39,279 --> 00:14:43,800
feel free to correct me at the end

319
00:14:41,720 --> 00:14:46,320
wasn't able to pick up on our problem

320
00:14:43,800 --> 00:14:48,600
and also doesn't have an API endpoint to

321
00:14:46,320 --> 00:14:51,920
make testing

322
00:14:48,600 --> 00:14:54,199
easier and the full configuration

323
00:14:51,920 --> 00:14:56,759
tool the issue with that is that the

324
00:14:54,199 --> 00:14:59,800
expansion is limited and won't expand

325
00:14:56,759 --> 00:15:02,399
out defaults since technically Ally each

326
00:14:59,800 --> 00:15:04,639
job has an image but we've just taken it

327
00:15:02,399 --> 00:15:07,519
here as a

328
00:15:04,639 --> 00:15:10,199
default and so while lending tools have

329
00:15:07,519 --> 00:15:12,600
their use they are only a single piece

330
00:15:10,199 --> 00:15:15,720
of the puzzle just like how we can't

331
00:15:12,600 --> 00:15:18,920
expect linters to catch logic

332
00:15:15,720 --> 00:15:21,880
errors so if lters are lacking due to

333
00:15:18,920 --> 00:15:24,720
not testing end to end why don't we

334
00:15:21,880 --> 00:15:26,639
Instead try running an endtoend pipeline

335
00:15:24,720 --> 00:15:29,079
before merging

336
00:15:26,639 --> 00:15:31,800
changes well the good news is that

337
00:15:29,079 --> 00:15:34,160
gitlab lets us run a pipeline from a

338
00:15:31,800 --> 00:15:38,120
particular Branch such as the one that

339
00:15:34,160 --> 00:15:41,720
is introducing the CI config file

340
00:15:38,120 --> 00:15:44,319
changes and it's also lets us uh specify

341
00:15:41,720 --> 00:15:47,440
variables for example overwriting those

342
00:15:44,319 --> 00:15:50,399
that would be predefined by

343
00:15:47,440 --> 00:15:53,160
gitlab this is great because it lets us

344
00:15:50,399 --> 00:15:55,720
run a p a pipeline

345
00:15:53,160 --> 00:15:58,000
identically excuse me let us run a

346
00:15:55,720 --> 00:16:01,680
pipeline identically to how one would be

347
00:15:58,000 --> 00:16:04,519
run by gitlab such as a post merge

348
00:16:01,680 --> 00:16:06,600
pipeline however there is one limitation

349
00:16:04,519 --> 00:16:09,880
being that the this pipeline depends on

350
00:16:06,600 --> 00:16:11,880
changes for that particular branch and

351
00:16:09,880 --> 00:16:14,399
therefore we won't be able to catch our

352
00:16:11,880 --> 00:16:16,399
problem as our Branch doesn't introduce

353
00:16:14,399 --> 00:16:18,319
the changes that are needed to reveal

354
00:16:16,399 --> 00:16:21,399
that

355
00:16:18,319 --> 00:16:24,040
bug however there is another way and

356
00:16:21,399 --> 00:16:27,079
that is by creating a branch off of our

357
00:16:24,040 --> 00:16:29,319
CI config file changes with the changes

358
00:16:27,079 --> 00:16:32,800
that are needed such as changing a file

359
00:16:29,319 --> 00:16:34,880
that we expect to run in the test

360
00:16:32,800 --> 00:16:36,839
directory this is great as it would

361
00:16:34,880 --> 00:16:39,440
reveal our

362
00:16:36,839 --> 00:16:41,560
problem however as you may have noticed

363
00:16:39,440 --> 00:16:44,519
this is quite an intensive and costly

364
00:16:41,560 --> 00:16:46,600
process as it's slow and manual and may

365
00:16:44,519 --> 00:16:49,120
not scale with all of the combinations

366
00:16:46,600 --> 00:16:51,399
of changes that may be introduced with

367
00:16:49,120 --> 00:16:54,480
lots of different

368
00:16:51,399 --> 00:16:56,519
rules and so if our third method we have

369
00:16:54,480 --> 00:17:00,319
something in between something that's

370
00:16:56,519 --> 00:17:03,439
not as static as linting and not as end

371
00:17:00,319 --> 00:17:06,520
to end as running entire

372
00:17:03,439 --> 00:17:10,240
pipelines for that we can use tools like

373
00:17:06,520 --> 00:17:12,600
gitlab c local and gitlab c

374
00:17:10,240 --> 00:17:14,640
local Yes you heard me right the

375
00:17:12,600 --> 00:17:16,880
difference is in a

376
00:17:14,640 --> 00:17:19,880
dash

377
00:17:16,880 --> 00:17:22,520
gitlab-ci loc is a typescript tool made

378
00:17:19,880 --> 00:17:26,160
by firal in January of

379
00:17:22,520 --> 00:17:29,039
2020 and gitlab C- local is a python

380
00:17:26,160 --> 00:17:32,880
tool made by Adrian DC in January of 201

381
00:17:29,039 --> 00:17:35,160
20 honestly it's just bizarre how

382
00:17:32,880 --> 00:17:37,240
similar these tools are both in what

383
00:17:35,160 --> 00:17:38,400
they do and when they were made and

384
00:17:37,240 --> 00:17:41,360
their

385
00:17:38,400 --> 00:17:43,480
name but anyways at the end of the day

386
00:17:41,360 --> 00:17:46,919
these are both tools that let us run

387
00:17:43,480 --> 00:17:49,640
individual jobs of our gitlab pipelines

388
00:17:46,919 --> 00:17:51,760
locally using

389
00:17:49,640 --> 00:17:53,799
containers and the core benefits of

390
00:17:51,760 --> 00:17:55,360
these tools are that they let us remove

391
00:17:53,799 --> 00:17:57,960
the dependency on

392
00:17:55,360 --> 00:17:59,960
gitlab we might not want to spin up an

393
00:17:57,960 --> 00:18:00,880
entire pipeline line just to run a

394
00:17:59,960 --> 00:18:03,640
single

395
00:18:00,880 --> 00:18:06,200
job and this is helpful as it quickens

396
00:18:03,640 --> 00:18:08,960
our feedback loop by allowing us to

397
00:18:06,200 --> 00:18:10,919
locally replicate job environments that

398
00:18:08,960 --> 00:18:13,520
might otherwise be too different if

399
00:18:10,919 --> 00:18:16,159
simply run as a

400
00:18:13,520 --> 00:18:18,200
script that's not to say that these

401
00:18:16,159 --> 00:18:20,919
tools also don't have

402
00:18:18,200 --> 00:18:22,240
downsides for one being simulators

403
00:18:20,919 --> 00:18:23,360
they're not always going to match the

404
00:18:22,240 --> 00:18:26,280
real

405
00:18:23,360 --> 00:18:28,799
thing and if we're solely testing our

406
00:18:26,280 --> 00:18:31,039
pipeline changes they don't give us any

407
00:18:28,799 --> 00:18:34,080
itic information in advance and require

408
00:18:31,039 --> 00:18:37,200
us to wait for the jobs to finish

409
00:18:34,080 --> 00:18:39,520
running and so in our examples above

410
00:18:37,200 --> 00:18:42,679
both of these tools also won't help us

411
00:18:39,520 --> 00:18:45,039
uncover our problems even though they

412
00:18:42,679 --> 00:18:47,919
might have other

413
00:18:45,039 --> 00:18:50,320
benefits and so with the fourth method

414
00:18:47,919 --> 00:18:53,159
let's explore a different kind of

415
00:18:50,320 --> 00:18:55,840
testing in this front I wasn't able to

416
00:18:53,159 --> 00:18:59,400
find any kind of static analysis testing

417
00:18:55,840 --> 00:19:02,480
tools for CI config files and thus to

418
00:18:59,400 --> 00:19:07,640
solve this problem I begrudgingly made

419
00:19:02,480 --> 00:19:10,039
my own tool CI test CI test acts as a

420
00:19:07,640 --> 00:19:12,559
missing piece of this linting endtoend

421
00:19:10,039 --> 00:19:15,039
integration testing puzzle by

422
00:19:12,559 --> 00:19:17,640
functioning as a framework for static

423
00:19:15,039 --> 00:19:20,240
analysis testing think of it as a

424
00:19:17,640 --> 00:19:22,919
toolbox to add to your shed of

425
00:19:20,240 --> 00:19:26,120
equipment by analyzing the CI config

426
00:19:22,919 --> 00:19:27,919
file and how it's changed we can gain a

427
00:19:26,120 --> 00:19:30,320
better understanding of what the

428
00:19:27,919 --> 00:19:33,720
pipeline is meant to to look

429
00:19:30,320 --> 00:19:36,280
like as an example it can it can produce

430
00:19:33,720 --> 00:19:38,520
snapshots of the pipeline by collating

431
00:19:36,280 --> 00:19:42,000
relevant information such as pairing

432
00:19:38,520 --> 00:19:44,880
together lists of jobs and

433
00:19:42,000 --> 00:19:47,240
changes and so using our example when

434
00:19:44,880 --> 00:19:50,159
the change was made we would get this

435
00:19:47,240 --> 00:19:52,559
diff this immediately lets us know that

436
00:19:50,159 --> 00:19:54,679
an additional rule set was created that

437
00:19:52,559 --> 00:19:56,559
is different from the expected rule set

438
00:19:54,679 --> 00:19:59,080
from all other

439
00:19:56,559 --> 00:20:01,280
jobs and so really we should have seen

440
00:19:59,080 --> 00:20:04,919
this instead where all the jobs

441
00:20:01,280 --> 00:20:07,600
undefined under the same set of

442
00:20:04,919 --> 00:20:10,360
changes of course like everything else

443
00:20:07,600 --> 00:20:12,400
it has its drawbacks and can't solely be

444
00:20:10,360 --> 00:20:15,400
dependent on since it doesn't run in

445
00:20:12,400 --> 00:20:18,440
integration like all the other

446
00:20:15,400 --> 00:20:21,320
methods however it will allow us to

447
00:20:18,440 --> 00:20:24,960
quickly iterate on rules with confidence

448
00:20:21,320 --> 00:20:27,880
which was something that we couldn't do

449
00:20:24,960 --> 00:20:29,840
before and that being said don't take

450
00:20:27,880 --> 00:20:32,600
that as me on the record to say that

451
00:20:29,840 --> 00:20:35,919
this will be the holy grail for all your

452
00:20:32,600 --> 00:20:37,960
pipeline tests and and so forth there is

453
00:20:35,919 --> 00:20:39,840
plenty of room for improvement and it's

454
00:20:37,960 --> 00:20:42,080
been written in a way to be highly

455
00:20:39,840 --> 00:20:44,000
extensible so if you want to help

456
00:20:42,080 --> 00:20:46,440
contribute to this or any other

457
00:20:44,000 --> 00:20:49,520
developer tooling I'm more than happy to

458
00:20:46,440 --> 00:20:52,880
help Reach Out make a pull request on

459
00:20:49,520 --> 00:20:52,880
GitHub or come to the

460
00:20:53,000 --> 00:20:58,240
Sprints all right so the last method

461
00:20:55,720 --> 00:21:00,520
that I'll cover is using tools to

462
00:20:58,240 --> 00:21:03,200
Dynamic ically generate pipelines using

463
00:21:00,520 --> 00:21:05,120
programming languages like

464
00:21:03,200 --> 00:21:08,360
python

465
00:21:05,120 --> 00:21:10,960
gitlab-ci python library is the only

466
00:21:08,360 --> 00:21:13,320
upto-date uh Library I could find to do

467
00:21:10,960 --> 00:21:15,240
this although if you are interested

468
00:21:13,320 --> 00:21:17,640
there are some typescript libraries that

469
00:21:15,240 --> 00:21:19,240
I had dismissed due to needing some more

470
00:21:17,640 --> 00:21:22,640
attention

471
00:21:19,240 --> 00:21:23,480
anyways gitlab C python library or Gip

472
00:21:22,640 --> 00:21:27,000
for

473
00:21:23,480 --> 00:21:29,440
short is a library from Thomas Stein

474
00:21:27,000 --> 00:21:31,279
buff and Daniel F Essen

475
00:21:29,440 --> 00:21:33,679
that allows you to replace your CI

476
00:21:31,279 --> 00:21:36,799
config file with

477
00:21:33,679 --> 00:21:38,159
python now you might be wondering why

478
00:21:36,799 --> 00:21:41,559
would you want

479
00:21:38,159 --> 00:21:44,120
this well if we quote the library we can

480
00:21:41,559 --> 00:21:45,960
utilize all the features of python such

481
00:21:44,120 --> 00:21:49,520
as code

482
00:21:45,960 --> 00:21:51,960
reuse variables control flows and the

483
00:21:49,520 --> 00:21:54,120
like programming

484
00:21:51,960 --> 00:21:55,919
paradigms

485
00:21:54,120 --> 00:21:58,919
libraries

486
00:21:55,919 --> 00:22:01,640
testing Packaging

487
00:21:58,919 --> 00:22:04,440
and anything else you can imagine the

488
00:22:01,640 --> 00:22:07,120
world is at your hands with

489
00:22:04,440 --> 00:22:10,200
python so the way that these generated

490
00:22:07,120 --> 00:22:12,240
pipelines work is that in our CI config

491
00:22:10,200 --> 00:22:13,400
file where we've been defining our

492
00:22:12,240 --> 00:22:16,760
testing

493
00:22:13,400 --> 00:22:18,159
jobs we instead Define a generate

494
00:22:16,760 --> 00:22:22,400
pipeline

495
00:22:18,159 --> 00:22:25,960
job which runs our python CI config

496
00:22:22,400 --> 00:22:28,640
file to create a generated config CI

497
00:22:25,960 --> 00:22:30,919
file that contains all of the previous

498
00:22:28,640 --> 00:22:34,600
jobs that we want to

499
00:22:30,919 --> 00:22:36,799
run the original CI config file had also

500
00:22:34,600 --> 00:22:39,640
defined a run pipeline

501
00:22:36,799 --> 00:22:41,200
job that has a needs on the generate

502
00:22:39,640 --> 00:22:45,120
pipeline

503
00:22:41,200 --> 00:22:48,320
job as it uses the generated CI config

504
00:22:45,120 --> 00:22:50,440
file to then create a child pipeline

505
00:22:48,320 --> 00:22:52,520
that runs all the previous jobs that we

506
00:22:50,440 --> 00:22:55,000
wanted to

507
00:22:52,520 --> 00:22:57,200
run this is great because it means we

508
00:22:55,000 --> 00:23:00,080
can replace writing all of our yaml with

509
00:22:57,200 --> 00:23:02,279
python

510
00:23:00,080 --> 00:23:05,480
okay so what does that look like if we

511
00:23:02,279 --> 00:23:07,360
want to replicate what we had before our

512
00:23:05,480 --> 00:23:11,679
config file would look like

513
00:23:07,360 --> 00:23:14,400
this we import the library create a

514
00:23:11,679 --> 00:23:17,120
pipeline specify the default

515
00:23:14,400 --> 00:23:22,000
image then we create a

516
00:23:17,120 --> 00:23:22,919
job we give it a name add a script and

517
00:23:22,000 --> 00:23:26,679
its

518
00:23:22,919 --> 00:23:30,000
rules doing the same for all other

519
00:23:26,679 --> 00:23:32,520
jobs adding our need

520
00:23:30,000 --> 00:23:34,600
before finally adding all the jobs as

521
00:23:32,520 --> 00:23:37,400
children of the

522
00:23:34,600 --> 00:23:39,240
pipeline and then finally writing that

523
00:23:37,400 --> 00:23:42,480
pipeline out to

524
00:23:39,240 --> 00:23:44,520
yaml and sure enough we can have this

525
00:23:42,480 --> 00:23:48,080
pipeline execute in our

526
00:23:44,520 --> 00:23:50,960
repo and have all the expected down

527
00:23:48,080 --> 00:23:53,640
strep jobs run

528
00:23:50,960 --> 00:23:55,840
successfully however has this caused our

529
00:23:53,640 --> 00:23:58,720
initial problem to go

530
00:23:55,840 --> 00:24:00,600
away well it doesn't give us any

531
00:23:58,720 --> 00:24:03,360
protections from the pipeline failing to

532
00:24:00,600 --> 00:24:05,400
run because again there were no reasons

533
00:24:03,360 --> 00:24:07,640
for it to run correctly since there was

534
00:24:05,400 --> 00:24:10,480
nopu

535
00:24:07,640 --> 00:24:12,919
changes and so fortunately it seems like

536
00:24:10,480 --> 00:24:16,760
generated pipelines won't directly solve

537
00:24:12,919 --> 00:24:19,440
our problem that being said they can

538
00:24:16,760 --> 00:24:22,400
help as the authors mentioned earlier

539
00:24:19,440 --> 00:24:25,200
this programmatic approach does let you

540
00:24:22,400 --> 00:24:27,679
import your pipeline and perform tests

541
00:24:25,200 --> 00:24:30,919
on it allowing us to do something

542
00:24:27,679 --> 00:24:34,120
similar like we've done with CI

543
00:24:30,919 --> 00:24:37,039
test however there is one other thing to

544
00:24:34,120 --> 00:24:39,600
consider since our pipeline is now in

545
00:24:37,039 --> 00:24:42,039
Python they do let us easily create a

546
00:24:39,600 --> 00:24:45,000
change set using a

547
00:24:42,039 --> 00:24:47,960
list that we can apply everywhere and

548
00:24:45,000 --> 00:24:50,039
thus help alleviate any rule mismatches

549
00:24:47,960 --> 00:24:52,399
as all the rules will now be the

550
00:24:50,039 --> 00:24:55,520
same this may have those of you

551
00:24:52,399 --> 00:24:57,799
knowledgeable and yamal thinking doesn't

552
00:24:55,520 --> 00:24:59,559
yamal also let you create a list that

553
00:24:57,799 --> 00:25:00,720
can be refer

554
00:24:59,559 --> 00:25:03,799
and you'd be

555
00:25:00,720 --> 00:25:06,640
correct we can create a key with an

556
00:25:03,799 --> 00:25:10,000
anchor and reference them with aliases

557
00:25:06,640 --> 00:25:12,960
to do the same thing and solve our

558
00:25:10,000 --> 00:25:15,360
problem but if we wanted to do something

559
00:25:12,960 --> 00:25:17,840
like extend this chain set it wouldn't

560
00:25:15,360 --> 00:25:19,120
be as easy as yaml doesn't allow merging

561
00:25:17,840 --> 00:25:21,720
of

562
00:25:19,120 --> 00:25:24,120
lists well that's not entirely correct

563
00:25:21,720 --> 00:25:26,679
gitlab does allow you to merge

564
00:25:24,120 --> 00:25:28,679
lists however this doesn't work for all

565
00:25:26,679 --> 00:25:32,640
cases cuz it only lets you do that

566
00:25:28,679 --> 00:25:32,640
sometimes and it won't result in valid

567
00:25:32,880 --> 00:25:39,039
yaml so are generated pipelines a good

568
00:25:36,640 --> 00:25:42,520
idea I think the authors have a good

569
00:25:39,039 --> 00:25:45,080
view on this which I'll do my best to

570
00:25:42,520 --> 00:25:47,559
summarize the cons are that it could

571
00:25:45,080 --> 00:25:49,799
allow for writing complex pipelines that

572
00:25:47,559 --> 00:25:52,120
nobody else understands when they could

573
00:25:49,799 --> 00:25:55,159
be basic things that build and test your

574
00:25:52,120 --> 00:25:58,399
code and shouldn't be in code

575
00:25:55,159 --> 00:26:00,399
too but if you're not fond of yaml your

576
00:25:58,399 --> 00:26:02,880
CI files are growing to over thousands

577
00:26:00,399 --> 00:26:05,480
of lines long or they're LED with

578
00:26:02,880 --> 00:26:07,520
complex rule sets maybe writing them in

579
00:26:05,480 --> 00:26:10,159
code is

580
00:26:07,520 --> 00:26:14,080
cleaner importantly it's a choice that's

581
00:26:10,159 --> 00:26:16,720
up to you your team and the

582
00:26:14,080 --> 00:26:18,960
situation so to close out if we look

583
00:26:16,720 --> 00:26:19,880
back at our methods and ask what is the

584
00:26:18,960 --> 00:26:22,559
best

585
00:26:19,880 --> 00:26:25,320
approach well it really depends on what

586
00:26:22,559 --> 00:26:27,919
your CI looks like if your CI config

587
00:26:25,320 --> 00:26:29,720
file has become thousands of lines long

588
00:26:27,919 --> 00:26:31,919
gener gener ating it dynamically or

589
00:26:29,720 --> 00:26:34,600
writing tests may help relieve that

590
00:26:31,919 --> 00:26:37,240
burden but at the end of the day all

591
00:26:34,600 --> 00:26:40,000
these methods are only tools to assist

592
00:26:37,240 --> 00:26:42,080
us in our Pro in our problems and if the

593
00:26:40,000 --> 00:26:43,480
underlying problem is the thousands of

594
00:26:42,080 --> 00:26:45,559
lines of CI

595
00:26:43,480 --> 00:26:46,919
configuration it might help to start

596
00:26:45,559 --> 00:26:49,799
there

597
00:26:46,919 --> 00:26:52,880
instead thus my view from all this is

598
00:26:49,799 --> 00:26:54,880
that Simplicity is key choosing one tool

599
00:26:52,880 --> 00:26:57,039
over another will still be adding

600
00:26:54,880 --> 00:26:58,760
another component to something that is

601
00:26:57,039 --> 00:27:01,159
likely already complex

602
00:26:58,760 --> 00:27:03,440
that you'll later have to deal with

603
00:27:01,159 --> 00:27:06,760
question everything that is there and

604
00:27:03,440 --> 00:27:10,240
utilize the least you can to get the job

605
00:27:06,760 --> 00:27:13,000
done as always I open to feedback as I'm

606
00:27:10,240 --> 00:27:15,799
continuously striving to be better you

607
00:27:13,000 --> 00:27:18,640
can find my contact all related repos

608
00:27:15,799 --> 00:27:21,440
tools references and sources at CIT

609
00:27:18,640 --> 00:27:23,640
test. nohum error.com

610
00:27:21,440 --> 00:27:26,320
and if you're still wondering who test

611
00:27:23,640 --> 00:27:28,279
the test is I'll be around after for

612
00:27:26,320 --> 00:27:32,039
questions

613
00:27:28,279 --> 00:27:34,559
thank you to my friends my family and uh

614
00:27:32,039 --> 00:27:37,000
you for supporting me as well as the

615
00:27:34,559 --> 00:27:38,000
open source contributors and finally to

616
00:27:37,000 --> 00:27:40,279
you for

617
00:27:38,000 --> 00:27:42,000
listening I'll leave this up now so that

618
00:27:40,279 --> 00:27:45,679
you can actually take pictures and take

619
00:27:42,000 --> 00:27:45,679
look thank you very much