1
00:00:00,000 --> 00:00:08,469
foreign

2
00:00:00,500 --> 00:00:08,469
[Music]

3
00:00:13,280 --> 00:00:17,880
talking about embedding golang in a

4
00:00:16,080 --> 00:00:20,340
kotling app

5
00:00:17,880 --> 00:00:22,859
um Charles is a senior engineer on the

6
00:00:20,340 --> 00:00:24,420
mamier team at grafana Labs we're

7
00:00:22,859 --> 00:00:26,519
particular interest in developer

8
00:00:24,420 --> 00:00:28,980
experience Automation and Cloud native

9
00:00:26,519 --> 00:00:30,960
infrastructure in today's presentation

10
00:00:28,980 --> 00:00:33,360
Charles will outline his experience when

11
00:00:30,960 --> 00:00:35,700
he tried mixing kotlin golang and why

12
00:00:33,360 --> 00:00:39,079
doing so was almost certainly a bad idea

13
00:00:35,700 --> 00:00:39,079
please welcome Charles

14
00:00:56,120 --> 00:01:01,079
round two thanks everyone

15
00:00:59,340 --> 00:01:03,180
um yeah I begin today by acknowledging

16
00:01:01,079 --> 00:01:05,220
the warranty Warrior rung people of the

17
00:01:03,180 --> 00:01:07,260
cooler Nation the traditional custodians

18
00:01:05,220 --> 00:01:09,420
of the land on which We Gather today and

19
00:01:07,260 --> 00:01:11,760
pay my respects to their Elders past and

20
00:01:09,420 --> 00:01:13,080
present I extend that respect to any

21
00:01:11,760 --> 00:01:14,700
Aboriginal and Torres Strait Islander

22
00:01:13,080 --> 00:01:16,500
people here today

23
00:01:14,700 --> 00:01:18,600
I'm here as you just heard my name is

24
00:01:16,500 --> 00:01:20,100
Charles I'm an engineer at grafina Labs

25
00:01:18,600 --> 00:01:21,600
I'm interested in a bunch of different

26
00:01:20,100 --> 00:01:23,700
things including developer tooling and

27
00:01:21,600 --> 00:01:25,020
open source I'm a maintainer of a bunch

28
00:01:23,700 --> 00:01:27,299
of different open source projects

29
00:01:25,020 --> 00:01:29,640
including protect and camel and

30
00:01:27,299 --> 00:01:30,840
contribute to many other projects uh and

31
00:01:29,640 --> 00:01:32,100
yeah while grafana has been very

32
00:01:30,840 --> 00:01:34,200
generous in allowing me to come today

33
00:01:32,100 --> 00:01:35,159
and give me time to do this

34
00:01:34,200 --> 00:01:37,140
um this is completely unrelated to

35
00:01:35,159 --> 00:01:38,700
everything I've been doing at grafana

36
00:01:37,140 --> 00:01:40,619
um for reasons that will come obvious

37
00:01:38,700 --> 00:01:42,720
shortly

38
00:01:40,619 --> 00:01:43,740
um yeah this talk is all about something

39
00:01:42,720 --> 00:01:46,320
very unusual that I've been doing

40
00:01:43,740 --> 00:01:48,420
recently uh the idea of embedding golang

41
00:01:46,320 --> 00:01:49,799
in a kotlin app and before I get into

42
00:01:48,420 --> 00:01:51,540
the inevitable question of why on Earth

43
00:01:49,799 --> 00:01:53,159
I've been trying to do that and explain

44
00:01:51,540 --> 00:01:54,840
why it's a terrible idea I'm going to

45
00:01:53,159 --> 00:01:56,340
talk about the concept in general

46
00:01:54,840 --> 00:01:57,780
break it down to a few smaller problems

47
00:01:56,340 --> 00:01:59,460
and talk about when those problems might

48
00:01:57,780 --> 00:02:01,560
actually be relevant and be a sensible

49
00:01:59,460 --> 00:02:02,759
thing to do you know in the real world

50
00:02:01,560 --> 00:02:05,040
and of course there'll be plenty of time

51
00:02:02,759 --> 00:02:06,840
for questions at the end

52
00:02:05,040 --> 00:02:08,340
so let's start at the beginning and

53
00:02:06,840 --> 00:02:10,380
let's start with the big picture of how

54
00:02:08,340 --> 00:02:12,540
this all works you know how can a kotlin

55
00:02:10,380 --> 00:02:15,020
how can I call an application call a guy

56
00:02:12,540 --> 00:02:15,020
Lang function

57
00:02:15,120 --> 00:02:20,580
the whole problem really boils down to

58
00:02:17,459 --> 00:02:22,620
one one thing the magic of C interrupt

59
00:02:20,580 --> 00:02:24,900
what do I mean by that

60
00:02:22,620 --> 00:02:27,660
both kotlin both the jvm and the native

61
00:02:24,900 --> 00:02:30,599
flavors can call C functions

62
00:02:27,660 --> 00:02:33,000
angolang can both call C functions and

63
00:02:30,599 --> 00:02:34,739
expose go functions as safe functions

64
00:02:33,000 --> 00:02:36,599
and so we can take advantage of that and

65
00:02:34,739 --> 00:02:37,620
use c as a bridge between kotlin and

66
00:02:36,599 --> 00:02:39,480
golang

67
00:02:37,620 --> 00:02:41,040
this this technique is not special it's

68
00:02:39,480 --> 00:02:42,300
not unique to these two languages like

69
00:02:41,040 --> 00:02:44,099
you can use this to bridge lots of

70
00:02:42,300 --> 00:02:45,480
different languages almost every

71
00:02:44,099 --> 00:02:46,980
function has some way to call C

72
00:02:45,480 --> 00:02:48,180
functions because of you know the fact

73
00:02:46,980 --> 00:02:50,580
that most operating systems expose

74
00:02:48,180 --> 00:02:52,319
themselves as as say functions

75
00:02:50,580 --> 00:02:54,959
um so things like Python and ruby.net

76
00:02:52,319 --> 00:02:57,239
can all call C functions

77
00:02:54,959 --> 00:02:58,980
and then many others including rust can

78
00:02:57,239 --> 00:03:00,060
expose functions as they function so

79
00:02:58,980 --> 00:03:01,500
they can be called by anything that can

80
00:03:00,060 --> 00:03:02,819
cause a function

81
00:03:01,500 --> 00:03:04,620
but yeah if it's starting to focus on

82
00:03:02,819 --> 00:03:06,300
kotlin and Garling

83
00:03:04,620 --> 00:03:08,519
so how does this work how can kotlin

84
00:03:06,300 --> 00:03:10,019
call a c function and if and therefore

85
00:03:08,519 --> 00:03:11,099
call a guideline function throughout

86
00:03:10,019 --> 00:03:13,379
this presentation I'm going to use a

87
00:03:11,099 --> 00:03:14,700
really really simple example of a golang

88
00:03:13,379 --> 00:03:16,680
function that takes in someone's name

89
00:03:14,700 --> 00:03:17,640
and gives you back a greeting for that

90
00:03:16,680 --> 00:03:20,599
person you know it's sort of a classic

91
00:03:17,640 --> 00:03:20,599
hello world kind of thing

92
00:03:20,640 --> 00:03:25,519
at a high level it looks a bit like this

93
00:03:23,099 --> 00:03:27,959
first of all we need AC function to call

94
00:03:25,519 --> 00:03:30,060
and the C definition of it looks a bit

95
00:03:27,959 --> 00:03:32,459
like this it takes in a pointer to a

96
00:03:30,060 --> 00:03:34,980
child so it's a string and returns

97
00:03:32,459 --> 00:03:36,959
another string

98
00:03:34,980 --> 00:03:39,360
as I just said before golang can expose

99
00:03:36,959 --> 00:03:42,299
going sorry go link expose Girling

100
00:03:39,360 --> 00:03:43,980
functions as a c function and so we can

101
00:03:42,299 --> 00:03:45,840
use that to provide this the C function

102
00:03:43,980 --> 00:03:47,459
that we need

103
00:03:45,840 --> 00:03:49,920
unlike what we normally do with the go

104
00:03:47,459 --> 00:03:52,019
go code and turn it into an executable

105
00:03:49,920 --> 00:03:54,180
application like a binary we can build

106
00:03:52,019 --> 00:03:56,099
that and package it up as a library so

107
00:03:54,180 --> 00:03:57,720
we can package it up as a shared object

108
00:03:56,099 --> 00:03:59,340
or a dll or a dialib depending on which

109
00:03:57,720 --> 00:04:01,019
operating system you're using

110
00:03:59,340 --> 00:04:02,459
or we can make it into a static Library

111
00:04:01,019 --> 00:04:04,819
which we can then build into our

112
00:04:02,459 --> 00:04:04,819
application

113
00:04:05,940 --> 00:04:10,319
on the other side we then need to call

114
00:04:07,319 --> 00:04:13,140
that c function from kotlin on the jvm

115
00:04:10,319 --> 00:04:14,099
we can interoperate with any c function

116
00:04:13,140 --> 00:04:15,420
through a bunch of different libraries

117
00:04:14,099 --> 00:04:16,979
I'll talk about in a bit more detail

118
00:04:15,420 --> 00:04:18,720
later on

119
00:04:16,979 --> 00:04:20,040
with kotler Native the support is even

120
00:04:18,720 --> 00:04:21,840
built in so we don't even need a library

121
00:04:20,040 --> 00:04:23,340
which is awesome and we can directly

122
00:04:21,840 --> 00:04:26,520
call any c function thanks for its

123
00:04:23,340 --> 00:04:28,500
built-in C interoperability

124
00:04:26,520 --> 00:04:29,940
so now the problem decomposes down to

125
00:04:28,500 --> 00:04:31,919
these two parts

126
00:04:29,940 --> 00:04:33,479
one is exposing Golan code as a c

127
00:04:31,919 --> 00:04:36,660
function and the other one is then

128
00:04:33,479 --> 00:04:39,240
calling that c function from from kotlin

129
00:04:36,660 --> 00:04:41,400
so let's start on the dialing side how

130
00:04:39,240 --> 00:04:43,759
do we expose a go function as a c

131
00:04:41,400 --> 00:04:43,759
function

132
00:04:43,860 --> 00:04:49,020
the first thing we need to do is Define

133
00:04:46,320 --> 00:04:51,479
our function so how many of you have go

134
00:04:49,020 --> 00:04:53,520
developers or familiar with go

135
00:04:51,479 --> 00:04:54,720
most of you all right for those of you

136
00:04:53,520 --> 00:04:55,680
who aren't I'll quickly explain what's

137
00:04:54,720 --> 00:04:56,880
going on

138
00:04:55,680 --> 00:04:58,800
um so this is a definition of a function

139
00:04:56,880 --> 00:05:01,080
it's called generate greeting it takes

140
00:04:58,800 --> 00:05:03,120
in a single parameter call of type c dot

141
00:05:01,080 --> 00:05:05,460
char and returns another parameter that

142
00:05:03,120 --> 00:05:07,740
is appointed to a CE char

143
00:05:05,460 --> 00:05:10,199
and then yeah all it does is then format

144
00:05:07,740 --> 00:05:12,060
in a message or sorry format a string

145
00:05:10,199 --> 00:05:13,880
that has a message hello from golang and

146
00:05:12,060 --> 00:05:15,720
then the name that you pass in

147
00:05:13,880 --> 00:05:17,639
yeah

148
00:05:15,720 --> 00:05:19,800
those of you who are go developers will

149
00:05:17,639 --> 00:05:21,900
instantly notice that the parameters are

150
00:05:19,800 --> 00:05:24,360
not using the ordinary go string type

151
00:05:21,900 --> 00:05:26,699
they're using this pointer to a c char

152
00:05:24,360 --> 00:05:28,080
the reason for that is that go has very

153
00:05:26,699 --> 00:05:30,240
very strict rules about memory

154
00:05:28,080 --> 00:05:32,580
management and in turn has very very

155
00:05:30,240 --> 00:05:33,960
strict rules about using strings because

156
00:05:32,580 --> 00:05:35,820
of the way that the string type is

157
00:05:33,960 --> 00:05:38,400
constructed and going and so therefore

158
00:05:35,820 --> 00:05:39,479
you have to use a special C type rather

159
00:05:38,400 --> 00:05:41,220
than the go type when you're passing

160
00:05:39,479 --> 00:05:43,820
strings back and forth and so that's

161
00:05:41,220 --> 00:05:43,820
what's going on here

162
00:05:43,860 --> 00:05:47,460
all right so we've got a function it

163
00:05:45,840 --> 00:05:49,259
will return the the value that we need

164
00:05:47,460 --> 00:05:51,900
we do need to do a couple more things to

165
00:05:49,259 --> 00:05:53,880
make this available to the world of C

166
00:05:51,900 --> 00:05:56,580
the first thing we need to do is add in

167
00:05:53,880 --> 00:05:57,840
this magic input input C this gives us

168
00:05:56,580 --> 00:05:59,580
all the c times we just use in

169
00:05:57,840 --> 00:06:01,560
particular C dot Char and the helper

170
00:05:59,580 --> 00:06:04,380
methods as well

171
00:06:01,560 --> 00:06:05,820
but it also enables cgo and cgo is the

172
00:06:04,380 --> 00:06:08,280
tool that's built into the go tool chain

173
00:06:05,820 --> 00:06:10,759
that enables interoperability with the

174
00:06:08,280 --> 00:06:10,759
world of safe

175
00:06:11,340 --> 00:06:14,580
once we've got that in place we then

176
00:06:12,900 --> 00:06:17,460
need a magic comment on this function

177
00:06:14,580 --> 00:06:19,320
that says export generate grading and

178
00:06:17,460 --> 00:06:22,440
cgo looks through any file where you

179
00:06:19,320 --> 00:06:25,259
import C like this and we'll generate

180
00:06:22,440 --> 00:06:26,880
and looks like those comments sorry

181
00:06:25,259 --> 00:06:28,380
and we'll then yeah make that available

182
00:06:26,880 --> 00:06:30,740
to the world to see it looks close it

183
00:06:28,380 --> 00:06:30,740
out for you

184
00:06:31,259 --> 00:06:34,800
the last thing we need to to make this

185
00:06:33,000 --> 00:06:37,199
work is a little bit of boilerplate so

186
00:06:34,800 --> 00:06:38,580
if you want to expose anything to C cgo

187
00:06:37,199 --> 00:06:40,979
insists of that is in the main package

188
00:06:38,580 --> 00:06:42,000
and go so we need that and if we're in

189
00:06:40,979 --> 00:06:43,620
the main package then we need a main

190
00:06:42,000 --> 00:06:45,600
function doesn't have to do anything but

191
00:06:43,620 --> 00:06:49,199
it has to be there so it's a little bit

192
00:06:45,600 --> 00:06:51,660
of extra things that we need need there

193
00:06:49,199 --> 00:06:53,280
so with all that in place we now have a

194
00:06:51,660 --> 00:06:54,539
function that can be exposed to the

195
00:06:53,280 --> 00:06:56,940
world of c and can be called anything

196
00:06:54,539 --> 00:06:58,500
that can call a c function

197
00:06:56,940 --> 00:07:01,020
the next step is to then take the source

198
00:06:58,500 --> 00:07:03,000
code and build it and so again like it's

199
00:07:01,020 --> 00:07:04,319
not a huge huge change from the The

200
00:07:03,000 --> 00:07:06,300
Ordinary World of ago where we're

201
00:07:04,319 --> 00:07:07,979
building a binary but instead we are

202
00:07:06,300 --> 00:07:09,539
going to build a library and we need to

203
00:07:07,979 --> 00:07:11,160
tell the compiler to do that

204
00:07:09,539 --> 00:07:13,380
and to do that we just pass an extra

205
00:07:11,160 --> 00:07:15,900
argument to go build so there's two

206
00:07:13,380 --> 00:07:18,479
options here you can build a shared

207
00:07:15,900 --> 00:07:20,520
Library so a dll or a dialib or a shared

208
00:07:18,479 --> 00:07:22,560
object file or you can build an archive

209
00:07:20,520 --> 00:07:25,620
which is something that you would like

210
00:07:22,560 --> 00:07:27,660
statically link into a final binary

211
00:07:25,620 --> 00:07:29,520
we can run that um ago we'll go off and

212
00:07:27,660 --> 00:07:31,259
do its thing and it will produce that

213
00:07:29,520 --> 00:07:32,819
that library for us and it will also

214
00:07:31,259 --> 00:07:34,979
produce a header file that we can then

215
00:07:32,819 --> 00:07:37,819
import into our C code to actually

216
00:07:34,979 --> 00:07:37,819
invoke that function

217
00:07:38,639 --> 00:07:42,240
so that's what it looks like on the go

218
00:07:40,740 --> 00:07:44,220
side of things let's look at what this

219
00:07:42,240 --> 00:07:46,199
looks like from the kotlin side

220
00:07:44,220 --> 00:07:47,759
I'm going to talk about both the jvm and

221
00:07:46,199 --> 00:07:48,840
kotler Native separately because it is a

222
00:07:47,759 --> 00:07:50,280
little bit different

223
00:07:48,840 --> 00:07:51,419
um but yeah most of you when you think

224
00:07:50,280 --> 00:07:55,160
of kotlin will probably think of kotlin

225
00:07:51,419 --> 00:07:55,160
jvm so I'll start there

226
00:07:55,380 --> 00:07:58,919
um everything I'm about to talk about is

227
00:07:56,940 --> 00:08:01,680
in fact not limited to kotlin on the jvm

228
00:07:58,919 --> 00:08:02,880
it's anything jvm at all so everything

229
00:08:01,680 --> 00:08:05,880
that I'm about to talk about is equally

230
00:08:02,880 --> 00:08:08,039
applicable to Java to Scala to groovy

231
00:08:05,880 --> 00:08:10,680
anything else that runs on the jvm you

232
00:08:08,039 --> 00:08:12,000
can use the same techniques for this

233
00:08:10,680 --> 00:08:13,680
and there's lots of different ways to do

234
00:08:12,000 --> 00:08:14,940
it so I'm going to go quickly through

235
00:08:13,680 --> 00:08:15,900
these and talk about each of them and

236
00:08:14,940 --> 00:08:18,419
talk about when you might want to use

237
00:08:15,900 --> 00:08:21,599
each of them as well

238
00:08:18,419 --> 00:08:23,819
the first and the original is

239
00:08:21,599 --> 00:08:25,620
jni which has been built into the jvm

240
00:08:23,819 --> 00:08:28,319
since version 1.1

241
00:08:25,620 --> 00:08:29,879
it has really great performance however

242
00:08:28,319 --> 00:08:31,979
it does have the drawback that you need

243
00:08:29,879 --> 00:08:34,380
to write a c function of your own that

244
00:08:31,979 --> 00:08:36,899
translates from the C's sorry from C to

245
00:08:34,380 --> 00:08:38,279
jvm C types and then you need to compile

246
00:08:36,899 --> 00:08:39,659
that c function for every single Target

247
00:08:38,279 --> 00:08:41,940
that you have

248
00:08:39,659 --> 00:08:44,520
um which is a pain in the ass like if

249
00:08:41,940 --> 00:08:46,500
you're you know targeting Linux and mac

250
00:08:44,520 --> 00:08:49,380
and windows and you want to Target x86

251
00:08:46,500 --> 00:08:50,820
and arm maybe doing 30 32-bit and 64-bit

252
00:08:49,380 --> 00:08:52,380
something you've got to cross compile

253
00:08:50,820 --> 00:08:54,899
this for I don't know six or eight

254
00:08:52,380 --> 00:08:56,279
different architectures it's awful

255
00:08:54,899 --> 00:08:58,620
um so yeah it's generally not what

256
00:08:56,279 --> 00:09:02,279
people do these days

257
00:08:58,620 --> 00:09:04,320
the next option is jni jna is an open

258
00:09:02,279 --> 00:09:06,060
source library and has a much nicer

259
00:09:04,320 --> 00:09:07,560
developer experience so rather than

260
00:09:06,060 --> 00:09:08,940
writing some C to translate back and

261
00:09:07,560 --> 00:09:10,140
forth between the world of c and the

262
00:09:08,940 --> 00:09:12,660
world of the jvm

263
00:09:10,140 --> 00:09:14,459
instead what you do is you write a Java

264
00:09:12,660 --> 00:09:16,920
or a kotlin interface or a class for

265
00:09:14,459 --> 00:09:18,600
structs and internally jna does the

266
00:09:16,920 --> 00:09:21,060
mapping for you

267
00:09:18,600 --> 00:09:22,560
however there's no free lunch here this

268
00:09:21,060 --> 00:09:25,800
does come at a free at a pretty

269
00:09:22,560 --> 00:09:27,779
significant performance penalty

270
00:09:25,800 --> 00:09:29,700
so then of course another thing came

271
00:09:27,779 --> 00:09:31,560
along that improved on that so jnr came

272
00:09:29,700 --> 00:09:33,540
along it is an open source Library as

273
00:09:31,560 --> 00:09:34,620
well it is quite widely used in

274
00:09:33,540 --> 00:09:36,120
particular it's used by things like J

275
00:09:34,620 --> 00:09:37,260
review so like it is it is quite battle

276
00:09:36,120 --> 00:09:38,880
tested

277
00:09:37,260 --> 00:09:40,680
it has the nice developer experience of

278
00:09:38,880 --> 00:09:42,660
jna but also has much much better

279
00:09:40,680 --> 00:09:44,880
performance as well it's still not quite

280
00:09:42,660 --> 00:09:47,959
as good as jni but it's likely good

281
00:09:44,880 --> 00:09:47,959
enough for most scenarios

282
00:09:49,320 --> 00:09:53,880
all right and then the fourth and final

283
00:09:51,300 --> 00:09:56,399
option is What's um now known as project

284
00:09:53,880 --> 00:09:58,320
Panama uh so this is something that's in

285
00:09:56,399 --> 00:10:01,320
preview for the jdk it's a replacement

286
00:09:58,320 --> 00:10:02,519
for jni it's built into the jvm

287
00:10:01,320 --> 00:10:04,260
um it's been going through the Java

288
00:10:02,519 --> 00:10:05,700
enhancement proposal process which is

289
00:10:04,260 --> 00:10:09,240
quite a mouthful for the last two years

290
00:10:05,700 --> 00:10:11,040
uh it was in preview in Java 8 19 and

291
00:10:09,240 --> 00:10:14,040
we'll be in second oh sorry is in second

292
00:10:11,040 --> 00:10:15,360
second preview now for Java 20.

293
00:10:14,040 --> 00:10:16,680
and yeah like this is a project fan

294
00:10:15,360 --> 00:10:18,120
number is really cool it combines that

295
00:10:16,680 --> 00:10:19,860
really nice developer experience that

296
00:10:18,120 --> 00:10:21,660
you get with something like jna or jnr

297
00:10:19,860 --> 00:10:23,279
with the better performance but only

298
00:10:21,660 --> 00:10:25,260
something built into the jvm can give

299
00:10:23,279 --> 00:10:26,820
you

300
00:10:25,260 --> 00:10:28,800
another really nice thing about Panama

301
00:10:26,820 --> 00:10:30,240
is that it has some really nice tooling

302
00:10:28,800 --> 00:10:32,940
support as well

303
00:10:30,240 --> 00:10:34,920
um so yeah like J J and A and J and R

304
00:10:32,940 --> 00:10:36,000
you need to write that kotlin interface

305
00:10:34,920 --> 00:10:38,820
for the Java interface to translate

306
00:10:36,000 --> 00:10:40,500
things that's still true for Panama but

307
00:10:38,820 --> 00:10:41,339
Panama also has some nice tooling to

308
00:10:40,500 --> 00:10:43,200
generate those things for you

309
00:10:41,339 --> 00:10:46,019
automatically so you don't have to do

310
00:10:43,200 --> 00:10:47,160
that that manual work

311
00:10:46,019 --> 00:10:48,600
um yeah for the example I'm going to

312
00:10:47,160 --> 00:10:50,339
show in a second I'm going to use jnr

313
00:10:48,600 --> 00:10:52,320
for two reasons one it's the thing I'm

314
00:10:50,339 --> 00:10:53,640
most familiar with and the second one is

315
00:10:52,320 --> 00:10:56,720
that it's something that you can most

316
00:10:53,640 --> 00:10:56,720
readily use today as well

317
00:10:57,180 --> 00:11:00,959
so let's see what this looks like in

318
00:10:58,800 --> 00:11:02,820
terms of some code uh the first thing we

319
00:11:00,959 --> 00:11:04,260
need to do when we want to call a c

320
00:11:02,820 --> 00:11:05,220
function is take our C function

321
00:11:04,260 --> 00:11:06,360
definition

322
00:11:05,220 --> 00:11:08,519
um so this is what it looks like to

323
00:11:06,360 --> 00:11:10,019
remind you and we need to translate that

324
00:11:08,519 --> 00:11:13,079
to an interface

325
00:11:10,019 --> 00:11:14,399
um so it looks like this in kotlin

326
00:11:13,079 --> 00:11:16,320
so

327
00:11:14,399 --> 00:11:18,060
jaina requires that every Library you

328
00:11:16,320 --> 00:11:19,260
want to call be represented in terms of

329
00:11:18,060 --> 00:11:20,339
interface for the different functions

330
00:11:19,260 --> 00:11:22,200
you want to call

331
00:11:20,339 --> 00:11:23,760
the method name the kotlin method name

332
00:11:22,200 --> 00:11:25,800
in this case generate grading has to

333
00:11:23,760 --> 00:11:27,300
match the corresponding C method um so

334
00:11:25,800 --> 00:11:28,680
generate grading has to be the same in

335
00:11:27,300 --> 00:11:29,880
both places

336
00:11:28,680 --> 00:11:31,019
the parameter names on the other hand

337
00:11:29,880 --> 00:11:32,519
don't matter

338
00:11:31,019 --> 00:11:34,380
what does matter for parameters though

339
00:11:32,519 --> 00:11:35,940
is that the order of the parameters has

340
00:11:34,380 --> 00:11:37,560
to be the same and the types of them

341
00:11:35,940 --> 00:11:38,279
have to match

342
00:11:37,560 --> 00:11:40,320
um

343
00:11:38,279 --> 00:11:41,760
what is kind of cool about jnr is that

344
00:11:40,320 --> 00:11:43,680
at least for method calls it will

345
00:11:41,760 --> 00:11:45,360
translate kotlin types or jvm types in

346
00:11:43,680 --> 00:11:47,100
general to their corresponding C types

347
00:11:45,360 --> 00:11:48,660
for you automatically so we can use an

348
00:11:47,100 --> 00:11:50,040
ordinary string here and it will

349
00:11:48,660 --> 00:11:52,640
automatically translate that to a

350
00:11:50,040 --> 00:11:52,640
pointer to a char

351
00:11:52,740 --> 00:11:57,899
so that's the translated method next

352
00:11:55,140 --> 00:12:01,260
what we need to do is tell jnr

353
00:11:57,899 --> 00:12:03,540
how to take the C library on disk

354
00:12:01,260 --> 00:12:05,399
and turn that into you know some kotlin

355
00:12:03,540 --> 00:12:06,899
code that we can actually call

356
00:12:05,399 --> 00:12:08,880
I'm not going to go into all the details

357
00:12:06,899 --> 00:12:10,019
because it is a lot and I recognize that

358
00:12:08,880 --> 00:12:11,579
I'm putting a lot of code on a slide in

359
00:12:10,019 --> 00:12:13,500
a presentation

360
00:12:11,579 --> 00:12:15,740
the the basic idea is that we need to

361
00:12:13,500 --> 00:12:18,240
take that dll or that dialib or that so

362
00:12:15,740 --> 00:12:20,220
package it up into our jar

363
00:12:18,240 --> 00:12:21,779
at build time and then at runtime pull

364
00:12:20,220 --> 00:12:23,220
it out of the jar

365
00:12:21,779 --> 00:12:25,019
um tell Jay and I where to find it on

366
00:12:23,220 --> 00:12:27,060
disk tell it what the kotlin interface

367
00:12:25,019 --> 00:12:28,200
is and then it will do the magic behind

368
00:12:27,060 --> 00:12:30,440
the scenes to map those things and make

369
00:12:28,200 --> 00:12:30,440
it work

370
00:12:31,500 --> 00:12:35,160
with the library loaded we can then use

371
00:12:33,480 --> 00:12:37,019
that library and we can call functions

372
00:12:35,160 --> 00:12:39,480
from it so for example if we wanted to

373
00:12:37,019 --> 00:12:40,800
call the generate grading method this is

374
00:12:39,480 --> 00:12:42,779
a really simple application that would

375
00:12:40,800 --> 00:12:45,540
do that we go off and load the grading

376
00:12:42,779 --> 00:12:46,980
Loop using this method above and then we

377
00:12:45,540 --> 00:12:49,380
can call it just like any other kotler

378
00:12:46,980 --> 00:12:50,820
method there's no extra sort of work

379
00:12:49,380 --> 00:12:54,019
required like what we saw with Scotland

380
00:12:50,820 --> 00:12:54,019
with them go link sorry

381
00:12:54,660 --> 00:12:57,320
so

382
00:13:00,980 --> 00:13:05,579
so that's what a method call looks like

383
00:13:04,320 --> 00:13:07,500
um just for sort of complete this I've

384
00:13:05,579 --> 00:13:08,540
also got an example of what a struct

385
00:13:07,500 --> 00:13:10,500
looks like

386
00:13:08,540 --> 00:13:11,880
so if you're passing something more

387
00:13:10,500 --> 00:13:13,680
complex than just a primitive data type

388
00:13:11,880 --> 00:13:16,079
you need a struct and see

389
00:13:13,680 --> 00:13:19,200
so let's say we've got this c-type here

390
00:13:16,079 --> 00:13:21,779
that represents a human so it takes a

391
00:13:19,200 --> 00:13:23,339
string for a name an unsigned 64-bit

392
00:13:21,779 --> 00:13:24,320
integer for age in case you're really

393
00:13:23,339 --> 00:13:27,300
old

394
00:13:24,320 --> 00:13:28,740
and a Boolean to you know whether you're

395
00:13:27,300 --> 00:13:31,079
an admin or not

396
00:13:28,740 --> 00:13:32,459
this is how that looks in kotlin with

397
00:13:31,079 --> 00:13:35,040
jaina

398
00:13:32,459 --> 00:13:36,360
so it's almost an ordinary class uh

399
00:13:35,040 --> 00:13:38,339
there are a couple of key differences

400
00:13:36,360 --> 00:13:40,320
the first one is that it has to extend

401
00:13:38,339 --> 00:13:42,240
from the special struct uh Base Class

402
00:13:40,320 --> 00:13:45,300
that jnr provides

403
00:13:42,240 --> 00:13:47,279
the properties must usually special

404
00:13:45,300 --> 00:13:48,540
types that J R provides as well like

405
00:13:47,279 --> 00:13:51,240
you'll notice these aren't the ordinary

406
00:13:48,540 --> 00:13:53,459
kotlin data types for Strings or numbers

407
00:13:51,240 --> 00:13:56,339
or even booleans and the reason for that

408
00:13:53,459 --> 00:13:57,959
is that yeah that allows jnr to do the

409
00:13:56,339 --> 00:14:00,500
translation back and forth to see

410
00:13:57,959 --> 00:14:02,339
and take it and also consider

411
00:14:00,500 --> 00:14:04,680
situations like if you've got like an

412
00:14:02,339 --> 00:14:05,639
INT in C you know handling the fact that

413
00:14:04,680 --> 00:14:08,060
is a different size on different

414
00:14:05,639 --> 00:14:08,060
platforms

415
00:14:09,180 --> 00:14:12,720
the name of the class in this case the

416
00:14:11,339 --> 00:14:15,240
name of the properties as well don't

417
00:14:12,720 --> 00:14:16,680
matter it's just the important thing is

418
00:14:15,240 --> 00:14:18,720
that the order of the properties is the

419
00:14:16,680 --> 00:14:20,760
same as the C type and the the types

420
00:14:18,720 --> 00:14:23,779
match or at least map

421
00:14:20,760 --> 00:14:23,779
um back and forth

422
00:14:24,000 --> 00:14:27,240
so um that's the jvm let's look at what

423
00:14:26,040 --> 00:14:28,920
this looks like for kotler Native as

424
00:14:27,240 --> 00:14:30,899
well

425
00:14:28,920 --> 00:14:32,519
unlike targeting the jvm with kotlin

426
00:14:30,899 --> 00:14:34,019
Native it's there's only one way to

427
00:14:32,519 --> 00:14:36,240
interoperate with c um so it's much much

428
00:14:34,019 --> 00:14:38,100
more straightforward and it's built in

429
00:14:36,240 --> 00:14:39,779
which is really cool as well

430
00:14:38,100 --> 00:14:42,000
so let's go through the same example on

431
00:14:39,779 --> 00:14:43,500
that side again we start with the c

432
00:14:42,000 --> 00:14:45,000
function definition that's the same as

433
00:14:43,500 --> 00:14:46,380
before

434
00:14:45,000 --> 00:14:48,300
um yeah like I just said given how

435
00:14:46,380 --> 00:14:49,199
common of a scenario this is with kotlin

436
00:14:48,300 --> 00:14:50,579
Native

437
00:14:49,199 --> 00:14:52,920
um cotton native doesn't require to do

438
00:14:50,579 --> 00:14:54,480
any like translation ourselves instead

439
00:14:52,920 --> 00:14:56,880
it has a tool that's built in called C

440
00:14:54,480 --> 00:14:58,260
interop that's part of copper NATO's

441
00:14:56,880 --> 00:15:00,060
built-in tooling it's built into the

442
00:14:58,260 --> 00:15:01,199
Gradle plugin and it will generate the

443
00:15:00,060 --> 00:15:03,480
method stubs and the destruct

444
00:15:01,199 --> 00:15:05,339
definitions for us automatically

445
00:15:03,480 --> 00:15:08,399
what we do do what we do need to do

446
00:15:05,339 --> 00:15:09,480
however is tell C interrupt how to do

447
00:15:08,399 --> 00:15:10,860
the translation

448
00:15:09,480 --> 00:15:13,199
so there are three bits of information

449
00:15:10,860 --> 00:15:15,600
that are really important the first is

450
00:15:13,199 --> 00:15:17,160
the kotlin package that we want the the

451
00:15:15,600 --> 00:15:18,959
methods and the the types to be

452
00:15:17,160 --> 00:15:21,360
generated in so that's that first line

453
00:15:18,959 --> 00:15:22,980
there we need to tell it which C header

454
00:15:21,360 --> 00:15:24,360
files it needs to scan to to work out

455
00:15:22,980 --> 00:15:26,459
what the methods are so that's the

456
00:15:24,360 --> 00:15:28,440
second one and the last one is telling

457
00:15:26,459 --> 00:15:30,420
it the name of the library file to

458
00:15:28,440 --> 00:15:32,579
statically link into our application so

459
00:15:30,420 --> 00:15:34,199
in this case unlike with jvm we have the

460
00:15:32,579 --> 00:15:36,120
option to statically compile a binary

461
00:15:34,199 --> 00:15:39,560
and so yeah doing that makes life much

462
00:15:36,120 --> 00:15:39,560
easier so that's what we're doing here

463
00:15:40,380 --> 00:15:42,779
all right so then with that out of the

464
00:15:42,120 --> 00:15:44,040
way

465
00:15:42,779 --> 00:15:46,620
um we can then write an application that

466
00:15:44,040 --> 00:15:48,120
calls that method just like any other

467
00:15:46,620 --> 00:15:49,800
um again it looks very very similar to

468
00:15:48,120 --> 00:15:51,899
the jvm example the only difference here

469
00:15:49,800 --> 00:15:53,699
is that similar to what we saw in golang

470
00:15:51,899 --> 00:15:55,019
kotler Native does not hide the fact

471
00:15:53,699 --> 00:15:57,060
that you're translating back and forth

472
00:15:55,019 --> 00:15:58,980
between C types and so that's why we've

473
00:15:57,060 --> 00:16:00,779
got these extra methods here on the

474
00:15:58,980 --> 00:16:03,600
string where we call C string to turn it

475
00:16:00,779 --> 00:16:04,680
into a C string and then the 2K string

476
00:16:03,600 --> 00:16:06,779
method on the end there to turn that C

477
00:16:04,680 --> 00:16:09,680
string back into a kotlin string so you

478
00:16:06,779 --> 00:16:09,680
can deal with it like normal

479
00:16:10,199 --> 00:16:14,220
so that's how you do this

480
00:16:13,019 --> 00:16:15,839
um let's talk a little bit about when

481
00:16:14,220 --> 00:16:18,120
you might actually want to do this in

482
00:16:15,839 --> 00:16:19,920
the real world

483
00:16:18,120 --> 00:16:21,240
like

484
00:16:19,920 --> 00:16:22,440
the first little hypothetical question

485
00:16:21,240 --> 00:16:24,240
is like when would you actually want to

486
00:16:22,440 --> 00:16:25,320
call a go function from kotlin code the

487
00:16:24,240 --> 00:16:26,940
answer is usually you wouldn't and

488
00:16:25,320 --> 00:16:28,860
usually you shouldn't

489
00:16:26,940 --> 00:16:30,779
uh when would you want to call a c

490
00:16:28,860 --> 00:16:32,579
function from kotlin code that's more of

491
00:16:30,779 --> 00:16:34,139
a normal thing to do

492
00:16:32,579 --> 00:16:35,820
um however I will put the caveat there

493
00:16:34,139 --> 00:16:38,759
that usually there is a really great

494
00:16:35,820 --> 00:16:41,279
Pure kotlin or pure jvm library at least

495
00:16:38,759 --> 00:16:42,480
that you can use to do most most things

496
00:16:41,279 --> 00:16:44,880
um so again like this should be an edge

497
00:16:42,480 --> 00:16:46,800
case it shouldn't be a common thing

498
00:16:44,880 --> 00:16:47,639
however like one particular scenario

499
00:16:46,800 --> 00:16:48,899
where this is a really common

500
00:16:47,639 --> 00:16:50,220
requirement is where you're dealing with

501
00:16:48,899 --> 00:16:51,899
operating system or dealing with devices

502
00:16:50,220 --> 00:16:53,279
and things like that if that isn't

503
00:16:51,899 --> 00:16:55,079
exposed to you through the standard

504
00:16:53,279 --> 00:16:56,279
libraries um then doing this is a

505
00:16:55,079 --> 00:16:57,420
perfectly normal perfectly legitimate

506
00:16:56,279 --> 00:16:58,800
thing to do

507
00:16:57,420 --> 00:17:01,579
um and the way I've just shown you is

508
00:16:58,800 --> 00:17:01,579
how you would do that

509
00:17:04,020 --> 00:17:08,579
but if you do go down this path you do

510
00:17:06,179 --> 00:17:10,980
need to be prepared for some pain and in

511
00:17:08,579 --> 00:17:12,120
particular when calling CE from kotlin

512
00:17:10,980 --> 00:17:13,679
um there are two major sources of pain

513
00:17:12,120 --> 00:17:16,679
to be aware of

514
00:17:13,679 --> 00:17:18,480
the first one is memory management so

515
00:17:16,679 --> 00:17:20,699
unlike in kotlin or many languages on

516
00:17:18,480 --> 00:17:21,660
the jvm where we generally you know

517
00:17:20,699 --> 00:17:23,939
don't need to think about memory

518
00:17:21,660 --> 00:17:24,900
management as soon as C is involved you

519
00:17:23,939 --> 00:17:26,579
need to start considering who is

520
00:17:24,900 --> 00:17:27,540
responsible for memory thinking about

521
00:17:26,579 --> 00:17:29,220
who's going to be responsible for

522
00:17:27,540 --> 00:17:30,840
freeing that memory who's gonna like

523
00:17:29,220 --> 00:17:32,400
when that's gonna happen

524
00:17:30,840 --> 00:17:33,900
um it's not super difficult I mean

525
00:17:32,400 --> 00:17:36,240
people have been doing it for decades

526
00:17:33,900 --> 00:17:37,200
but it is really error-prone and it can

527
00:17:36,240 --> 00:17:38,520
be quite difficult to reason about

528
00:17:37,200 --> 00:17:39,720
especially when you're thinking about

529
00:17:38,520 --> 00:17:41,880
like mapping back and forth between the

530
00:17:39,720 --> 00:17:43,860
two worlds

531
00:17:41,880 --> 00:17:46,140
um fun fact the example I just showed

532
00:17:43,860 --> 00:17:47,520
you actually leaks memory we need to

533
00:17:46,140 --> 00:17:48,720
free the string that we got back from

534
00:17:47,520 --> 00:17:50,100
golang

535
00:17:48,720 --> 00:17:52,740
um because yeah otherwise it will just

536
00:17:50,100 --> 00:17:54,480
sort of sit there forever

537
00:17:52,740 --> 00:17:56,280
the consequences are getting memory

538
00:17:54,480 --> 00:17:57,960
wrong can vary from simply just

539
00:17:56,280 --> 00:17:59,520
consuming too much memory um which is

540
00:17:57,960 --> 00:18:00,780
usually not the end of the world all the

541
00:17:59,520 --> 00:18:04,940
way up to things like memory corruption

542
00:18:00,780 --> 00:18:04,940
security issues and crashes as well

543
00:18:05,220 --> 00:18:09,059
the other big source of pain is the

544
00:18:07,140 --> 00:18:11,640
impact on the developer experience

545
00:18:09,059 --> 00:18:13,980
so usually you can't use a kotlin

546
00:18:11,640 --> 00:18:15,900
debugger to step into C code for example

547
00:18:13,980 --> 00:18:17,280
um so that can be quite annoying if

548
00:18:15,900 --> 00:18:18,419
you're targeting the jvm you need to

549
00:18:17,280 --> 00:18:19,860
handcraft those those interface

550
00:18:18,419 --> 00:18:21,720
definitions or those class definitions

551
00:18:19,860 --> 00:18:23,400
and that can be error prone it can be

552
00:18:21,720 --> 00:18:25,320
quite tricky to diagnose when you get

553
00:18:23,400 --> 00:18:26,400
that wrong

554
00:18:25,320 --> 00:18:27,660
um of course in kotlin Native like

555
00:18:26,400 --> 00:18:30,059
you've got that's the interrupt tooling

556
00:18:27,660 --> 00:18:31,679
and so that makes it much easier

557
00:18:30,059 --> 00:18:33,059
and finally you need to have an

558
00:18:31,679 --> 00:18:35,160
understanding of both kotlin and c and

559
00:18:33,059 --> 00:18:37,080
how those two worlds work together

560
00:18:35,160 --> 00:18:39,360
and that adds extra cognitive load of

561
00:18:37,080 --> 00:18:42,480
course so in summary two big sources of

562
00:18:39,360 --> 00:18:44,520
pain for calling C from kotlin

563
00:18:42,480 --> 00:18:47,039
when we add gulling into the mix that

564
00:18:44,520 --> 00:18:49,440
just compounds uh so memory management

565
00:18:47,039 --> 00:18:51,840
becomes more involved again because now

566
00:18:49,440 --> 00:18:53,820
we need to consider both how sorry we

567
00:18:51,840 --> 00:18:55,500
need to consider how golang NC and

568
00:18:53,820 --> 00:18:57,120
kotlin all want to work together and how

569
00:18:55,500 --> 00:18:59,900
they each manage memory and how those

570
00:18:57,120 --> 00:18:59,900
things don't quite line up

571
00:19:01,440 --> 00:19:04,980
um to give you like a concrete example

572
00:19:02,700 --> 00:19:07,980
so I mentioned before uh talking about

573
00:19:04,980 --> 00:19:09,539
the go string type that go has very very

574
00:19:07,980 --> 00:19:11,580
specific rules around what you can do

575
00:19:09,539 --> 00:19:13,919
with go manage memory and that makes

576
00:19:11,580 --> 00:19:15,360
seeing drop more more complicated there

577
00:19:13,919 --> 00:19:17,460
are certain rules about when and when

578
00:19:15,360 --> 00:19:19,679
you can and cannot pass go pointers out

579
00:19:17,460 --> 00:19:22,820
outside of go because otherwise the

580
00:19:19,679 --> 00:19:22,820
garbage collector can't work properly

581
00:19:23,280 --> 00:19:25,799
um and of course like you can't use

582
00:19:24,720 --> 00:19:28,460
things like a debugger across the

583
00:19:25,799 --> 00:19:28,460
language battery either

584
00:19:29,100 --> 00:19:32,700
but as you probably guessed from how the

585
00:19:31,020 --> 00:19:33,960
slide is laid out uh when you add

586
00:19:32,700 --> 00:19:37,080
Girling into the mix you get some

587
00:19:33,960 --> 00:19:39,960
exciting new sources of pain as well

588
00:19:37,080 --> 00:19:42,240
uh the first one is around tooling

589
00:19:39,960 --> 00:19:44,760
so when we're working like this not only

590
00:19:42,240 --> 00:19:47,039
do we need a growling tool chain and a

591
00:19:44,760 --> 00:19:49,140
Java tool a kotlin tool chain we need to

592
00:19:47,039 --> 00:19:50,700
make these two things work well together

593
00:19:49,140 --> 00:19:52,020
so that either means doing something

594
00:19:50,700 --> 00:19:54,600
like running the golang tool like

595
00:19:52,020 --> 00:19:56,220
tooling from Gradle or having some other

596
00:19:54,600 --> 00:19:58,020
alternative tool like bazel for example

597
00:19:56,220 --> 00:19:59,940
that orchestrates the whole world as

598
00:19:58,020 --> 00:20:02,520
well

599
00:19:59,940 --> 00:20:04,679
on top of that cgo needs access to a c

600
00:20:02,520 --> 00:20:06,419
compiler and a tool chain that can

601
00:20:04,679 --> 00:20:08,580
compile for each operating system and

602
00:20:06,419 --> 00:20:09,720
architecture Target that you have and

603
00:20:08,580 --> 00:20:10,919
for those of you who've ever tried to do

604
00:20:09,720 --> 00:20:12,480
that you know that's not straightforward

605
00:20:10,919 --> 00:20:15,260
and I'll talk a little bit more about

606
00:20:12,480 --> 00:20:15,260
that later on

607
00:20:16,260 --> 00:20:19,440
lastly when you're moving back and forth

608
00:20:17,880 --> 00:20:20,880
across language boundaries you're

609
00:20:19,440 --> 00:20:23,460
limited to the lowest common denominator

610
00:20:20,880 --> 00:20:24,900
of what's supported by each and this has

611
00:20:23,460 --> 00:20:27,600
a particular impact on things like data

612
00:20:24,900 --> 00:20:28,860
structure design method signatures and

613
00:20:27,600 --> 00:20:31,679
can make things like callbacks and

614
00:20:28,860 --> 00:20:34,080
things like that quite tricky as well

615
00:20:31,679 --> 00:20:35,520
so in summary there's lots of pain

616
00:20:34,080 --> 00:20:36,480
and hopefully at this point you're

617
00:20:35,520 --> 00:20:38,580
thinking like you know this has been

618
00:20:36,480 --> 00:20:39,900
interesting so far but like it sounds

619
00:20:38,580 --> 00:20:41,640
like a terrible idea like why on Earth

620
00:20:39,900 --> 00:20:43,919
did you do this

621
00:20:41,640 --> 00:20:46,200
um and the the answer is that I'm

622
00:20:43,919 --> 00:20:47,760
trading one kind of pain for another

623
00:20:46,200 --> 00:20:48,840
and to explain why I need to talk a

624
00:20:47,760 --> 00:20:50,940
little bit about an open source project

625
00:20:48,840 --> 00:20:53,400
I mentioned right at the beginning

626
00:20:50,940 --> 00:20:56,760
um called protect

627
00:20:53,400 --> 00:20:59,580
so protect is a tool for managed for

628
00:20:56,760 --> 00:21:01,980
manage for managing but dockerized local

629
00:20:59,580 --> 00:21:03,900
development environments

630
00:21:01,980 --> 00:21:05,660
it is written in kotlin for reasons you

631
00:21:03,900 --> 00:21:08,340
can read about on my blog

632
00:21:05,660 --> 00:21:10,380
and most of the docker ecosystem as many

633
00:21:08,340 --> 00:21:11,039
of you all know is written in golang

634
00:21:10,380 --> 00:21:13,320
um

635
00:21:11,039 --> 00:21:15,419
however the docker demon exposes an API

636
00:21:13,320 --> 00:21:16,740
or HTTP API

637
00:21:15,419 --> 00:21:18,660
um that in theory should be accessible

638
00:21:16,740 --> 00:21:20,940
to any language right it's HTTP it

639
00:21:18,660 --> 00:21:23,280
should be straightforward

640
00:21:20,940 --> 00:21:24,360
the reality of that however is not quite

641
00:21:23,280 --> 00:21:26,400
as nice

642
00:21:24,360 --> 00:21:28,200
parts that API are really poorly

643
00:21:26,400 --> 00:21:29,100
documented or just aren't documented at

644
00:21:28,200 --> 00:21:30,960
all

645
00:21:29,100 --> 00:21:33,059
parts of that API are implicitly use

646
00:21:30,960 --> 00:21:35,039
goaling conventions use constants from

647
00:21:33,059 --> 00:21:36,299
the standard library for example and

648
00:21:35,039 --> 00:21:39,000
those don't map nicely to operating

649
00:21:36,299 --> 00:21:40,559
system constants for example

650
00:21:39,000 --> 00:21:42,360
there are parts of the API that are

651
00:21:40,559 --> 00:21:45,780
really non-standard if I'm being polite

652
00:21:42,360 --> 00:21:49,320
or really weird if I'm being less polite

653
00:21:45,780 --> 00:21:51,299
um for example when you go to build an

654
00:21:49,320 --> 00:21:53,400
image using build kit the new image

655
00:21:51,299 --> 00:21:55,799
Builder you as the client so in this

656
00:21:53,400 --> 00:21:57,179
case protect send HTTP HTTP request to

657
00:21:55,799 --> 00:21:59,159
the demon and say hey I'd like to build

658
00:21:57,179 --> 00:22:00,299
this image the docker name says that

659
00:21:59,159 --> 00:22:02,340
sounds awesome I'm going to take that

660
00:22:00,299 --> 00:22:05,039
TCP socket you use for HTTP request and

661
00:22:02,340 --> 00:22:07,260
use it to send you grpc requests

662
00:22:05,039 --> 00:22:10,020
um and it's it's weird and of course

663
00:22:07,260 --> 00:22:11,820
yeah the most HTTP clients do not

664
00:22:10,020 --> 00:22:13,919
support that kind of weird

665
00:22:11,820 --> 00:22:15,900
having the TCP socket taken away from

666
00:22:13,919 --> 00:22:18,740
underneath you which makes integrating

667
00:22:15,900 --> 00:22:18,740
with this really tricky

668
00:22:18,840 --> 00:22:22,320
the other the other pain with the API is

669
00:22:20,820 --> 00:22:23,400
that a lot of it requires significant

670
00:22:22,320 --> 00:22:25,679
client-side logic that needs to be

671
00:22:23,400 --> 00:22:27,179
re-implemented uh so things like piping

672
00:22:25,679 --> 00:22:28,260
container i o back and forth is really

673
00:22:27,179 --> 00:22:29,820
tricky

674
00:22:28,260 --> 00:22:31,020
um dealing with registry credentials for

675
00:22:29,820 --> 00:22:32,940
pulling and pushing images is really

676
00:22:31,020 --> 00:22:34,799
tricky Building images as I just

677
00:22:32,940 --> 00:22:35,940
mentioned is really tricky all of these

678
00:22:34,799 --> 00:22:38,039
things require quite significant

679
00:22:35,940 --> 00:22:40,320
client-side logic and none of it is

680
00:22:38,039 --> 00:22:41,760
documented none of it is written down

681
00:22:40,320 --> 00:22:42,900
anywhere and so I've had to reverse

682
00:22:41,760 --> 00:22:45,380
engineer the goaling client to

683
00:22:42,900 --> 00:22:45,380
understand that

684
00:22:45,780 --> 00:22:48,780
so

685
00:22:46,860 --> 00:22:50,280
the official Docker client library for

686
00:22:48,780 --> 00:22:51,780
golang you know influence all this stuff

687
00:22:50,280 --> 00:22:54,539
it takes care of it all of it takes care

688
00:22:51,780 --> 00:22:56,400
of all of it for us nice and easily

689
00:22:54,539 --> 00:22:58,380
and there is no fully featured Docker

690
00:22:56,400 --> 00:23:00,659
client out there for kotlin or the jvm

691
00:22:58,380 --> 00:23:02,820
in general some of them do cover simple

692
00:23:00,659 --> 00:23:04,980
scenarios but none of them cover the the

693
00:23:02,820 --> 00:23:07,080
all the scenarios that I need for what I

694
00:23:04,980 --> 00:23:08,880
do and protect

695
00:23:07,080 --> 00:23:10,080
so that's what I did I ended up having

696
00:23:08,880 --> 00:23:12,480
to build my own

697
00:23:10,080 --> 00:23:13,980
um and so now or previously Patek had a

698
00:23:12,480 --> 00:23:15,780
battle tester Docker client written in

699
00:23:13,980 --> 00:23:17,760
kotlin that worked had all the features

700
00:23:15,780 --> 00:23:21,380
I needed it supports all these weird and

701
00:23:17,760 --> 00:23:21,380
wonderful things that the API does

702
00:23:21,659 --> 00:23:25,440
however maintaining that client and

703
00:23:24,000 --> 00:23:27,659
adding features to it was painful for

704
00:23:25,440 --> 00:23:28,919
all the reasons I mentioned before

705
00:23:27,659 --> 00:23:31,260
um so that poor documentation that

706
00:23:28,919 --> 00:23:33,120
implied user golang the the really

707
00:23:31,260 --> 00:23:34,440
non-standard interactions and that

708
00:23:33,120 --> 00:23:36,419
significant client-side logic that I had

709
00:23:34,440 --> 00:23:38,220
to reverse engineer

710
00:23:36,419 --> 00:23:39,720
so it'd be great if I didn't have to to

711
00:23:38,220 --> 00:23:41,340
do that in kotlin and it could could

712
00:23:39,720 --> 00:23:43,500
instead build upon the existing Garland

713
00:23:41,340 --> 00:23:44,640
client

714
00:23:43,500 --> 00:23:45,720
um and so that's what I that's what I

715
00:23:44,640 --> 00:23:47,400
ended up doing

716
00:23:45,720 --> 00:23:49,080
um I've rebuilt the tech Stocker client

717
00:23:47,400 --> 00:23:51,600
as a wrapper around the golang client

718
00:23:49,080 --> 00:23:53,760
and it's now used by protects many many

719
00:23:51,600 --> 00:23:54,960
hundreds of users every day and none of

720
00:23:53,760 --> 00:23:56,460
them have complained so that's that's a

721
00:23:54,960 --> 00:23:58,320
good sign

722
00:23:56,460 --> 00:23:59,760
um and yeah like despite all the sources

723
00:23:58,320 --> 00:24:00,720
of pain that I mentioned before this

724
00:23:59,760 --> 00:24:02,760
approach has been working really well

725
00:24:00,720 --> 00:24:04,440
for me I've traded the pain before

726
00:24:02,760 --> 00:24:05,820
documentation and the pain of implied

727
00:24:04,440 --> 00:24:08,039
used to go laying and significant

728
00:24:05,820 --> 00:24:09,059
client-side logic for more predictable

729
00:24:08,039 --> 00:24:10,140
pains that I mentioned on the previous

730
00:24:09,059 --> 00:24:12,120
slide

731
00:24:10,140 --> 00:24:13,380
so things like memory management and

732
00:24:12,120 --> 00:24:15,240
developer experience and tooling and all

733
00:24:13,380 --> 00:24:16,500
that kind of thing

734
00:24:15,240 --> 00:24:17,640
and because those pains are more

735
00:24:16,500 --> 00:24:19,020
predictable like I've been able to

736
00:24:17,640 --> 00:24:20,280
develop patterns and tooling and things

737
00:24:19,020 --> 00:24:22,860
like that to help me deal with those

738
00:24:20,280 --> 00:24:26,120
pains and make them less painful

739
00:24:22,860 --> 00:24:26,120
so I've got two examples of that

740
00:24:29,640 --> 00:24:34,200
the first is a round tooling

741
00:24:32,159 --> 00:24:35,940
so I mentioned before that you've got to

742
00:24:34,200 --> 00:24:37,559
make these two tool chains sort of work

743
00:24:35,940 --> 00:24:40,020
well together

744
00:24:37,559 --> 00:24:41,820
um I ended up building a Gradle plugin

745
00:24:40,020 --> 00:24:43,860
that can take some Girling sources and

746
00:24:41,820 --> 00:24:45,320
compile that down to static and dynamic

747
00:24:43,860 --> 00:24:48,120
libraries for each Target

748
00:24:45,320 --> 00:24:51,360
and then use Zig as the C cross compiler

749
00:24:48,120 --> 00:24:54,240
to satisfy um cgo's need for a c tool

750
00:24:51,360 --> 00:24:56,159
chain who's heard of Zeke by the way

751
00:24:54,240 --> 00:24:57,299
okay a couple of you awesome for those

752
00:24:56,159 --> 00:24:58,380
of you who haven't heard of it it is a

753
00:24:57,299 --> 00:25:00,780
really cool language in its own right

754
00:24:58,380 --> 00:25:03,120
but one of its coolest features is that

755
00:25:00,780 --> 00:25:04,500
it can act as a c cross compiler and it

756
00:25:03,120 --> 00:25:05,700
largely just works

757
00:25:04,500 --> 00:25:07,260
um which is really really cool like if

758
00:25:05,700 --> 00:25:08,460
for those of you who've ever tried to do

759
00:25:07,260 --> 00:25:11,100
cross compilation you know it's usually

760
00:25:08,460 --> 00:25:12,480
a pain Z get actually kind of works and

761
00:25:11,100 --> 00:25:15,620
it's really nice um so yeah give it a go

762
00:25:12,480 --> 00:25:15,620
if you haven't haven't seen it before

763
00:25:16,260 --> 00:25:19,559
the other thing I've spent quite a bit

764
00:25:18,120 --> 00:25:22,380
of time investing in to make my life

765
00:25:19,559 --> 00:25:24,360
easier is looking at making memory

766
00:25:22,380 --> 00:25:26,039
management easier

767
00:25:24,360 --> 00:25:27,720
and to do that what I've done is I've

768
00:25:26,039 --> 00:25:29,760
built a small code generation tool that

769
00:25:27,720 --> 00:25:31,860
takes in the definition of structs in

770
00:25:29,760 --> 00:25:34,740
yaml like this so for example here we've

771
00:25:31,860 --> 00:25:37,080
got a an arrow type that has a type and

772
00:25:34,740 --> 00:25:38,400
a message both of which are strings

773
00:25:37,080 --> 00:25:39,419
and from that single gamble definition

774
00:25:38,400 --> 00:25:40,860
that generates a bunch of different

775
00:25:39,419 --> 00:25:42,539
things that I need

776
00:25:40,860 --> 00:25:44,159
first of all it generates the

777
00:25:42,539 --> 00:25:46,559
corresponding C definition

778
00:25:44,159 --> 00:25:48,840
and it will translate the types for me

779
00:25:46,559 --> 00:25:50,880
automatically

780
00:25:48,840 --> 00:25:52,620
it will generate C functions to

781
00:25:50,880 --> 00:25:53,820
initialize and free pointers to that

782
00:25:52,620 --> 00:25:55,919
type

783
00:25:53,820 --> 00:25:58,380
um so again like it will allocate one

784
00:25:55,919 --> 00:25:59,400
using Malik and then initialize it and

785
00:25:58,380 --> 00:26:03,020
then I've got another one that I can use

786
00:25:59,400 --> 00:26:03,020
to free it when I don't need it anymore

787
00:26:03,059 --> 00:26:07,380
it will also generate go functions to

788
00:26:05,700 --> 00:26:10,260
convert from the go types to the C types

789
00:26:07,380 --> 00:26:12,779
really easily so for example here we can

790
00:26:10,260 --> 00:26:15,179
initialize a new error pointer

791
00:26:12,779 --> 00:26:17,580
um and taking the go string types so I

792
00:26:15,179 --> 00:26:20,039
can keep the sort of the messiness of C

793
00:26:17,580 --> 00:26:22,140
constraint contained in this method and

794
00:26:20,039 --> 00:26:24,740
the rest of the the go code base is

795
00:26:22,140 --> 00:26:24,740
relatively clean

796
00:26:25,679 --> 00:26:29,940
it also generates the corresponding jvm

797
00:26:28,020 --> 00:26:32,100
class definitions so similar to what we

798
00:26:29,940 --> 00:26:34,440
saw before we get a class that looks

799
00:26:32,100 --> 00:26:36,240
like this from that yaml definition

800
00:26:34,440 --> 00:26:38,340
you'll notice that one big difference

801
00:26:36,240 --> 00:26:40,140
here is that this one implements Auto

802
00:26:38,340 --> 00:26:41,460
closable as well which means that we can

803
00:26:40,140 --> 00:26:42,720
use the trial with the result resources

804
00:26:41,460 --> 00:26:43,799
pattern to make sure this is cleaned up

805
00:26:42,720 --> 00:26:45,120
properly

806
00:26:43,799 --> 00:26:46,679
um and yeah that means that it then goes

807
00:26:45,120 --> 00:26:48,720
off and calls that c method we just sort

808
00:26:46,679 --> 00:26:51,440
before to free the memory associated

809
00:26:48,720 --> 00:26:51,440
with it as well

810
00:26:51,960 --> 00:26:54,960
and it will also generate the

811
00:26:53,279 --> 00:26:57,600
corresponding jvm interface definition

812
00:26:54,960 --> 00:26:59,880
for the methods that were generated

813
00:26:57,600 --> 00:27:02,700
automatically another part of it also

814
00:26:59,880 --> 00:27:04,919
takes in the C header generated by cgo

815
00:27:02,700 --> 00:27:06,240
and translates those methods as well so

816
00:27:04,919 --> 00:27:07,679
you end up with one giant interface with

817
00:27:06,240 --> 00:27:09,059
all the different methods um and I don't

818
00:27:07,679 --> 00:27:10,260
have to write any of that myself which

819
00:27:09,059 --> 00:27:11,700
is awesome

820
00:27:10,260 --> 00:27:15,020
um and yeah that saves me a bunch from a

821
00:27:11,700 --> 00:27:15,020
bunch of tedious and error print work

822
00:27:15,179 --> 00:27:18,000
so

823
00:27:16,559 --> 00:27:18,900
um that's why I'm doing this and those

824
00:27:18,000 --> 00:27:20,220
are the things that I've had to do to

825
00:27:18,900 --> 00:27:22,320
make it work

826
00:27:20,220 --> 00:27:23,880
to to recap

827
00:27:22,320 --> 00:27:25,080
um you can use c as a bridge between

828
00:27:23,880 --> 00:27:26,700
different languages

829
00:27:25,080 --> 00:27:29,220
both flavors of kotlin can call C

830
00:27:26,700 --> 00:27:31,080
functions uh and going can both call C

831
00:27:29,220 --> 00:27:32,580
functions and expose go functions to C

832
00:27:31,080 --> 00:27:34,039
functions try to send that five times

833
00:27:32,580 --> 00:27:36,000
fast

834
00:27:34,039 --> 00:27:38,159
there are lots of ways to do this on the

835
00:27:36,000 --> 00:27:39,480
jvm each with different trade-offs I

836
00:27:38,159 --> 00:27:40,559
would recommend J R as a sensible

837
00:27:39,480 --> 00:27:42,240
starting point unless you've got

838
00:27:40,559 --> 00:27:43,140
particular constraints

839
00:27:42,240 --> 00:27:44,940
um we've got no difference really

840
00:27:43,140 --> 00:27:46,679
straightforward though

841
00:27:44,940 --> 00:27:48,799
um and yeah please don't actually do

842
00:27:46,679 --> 00:27:48,799
this

843
00:27:49,380 --> 00:27:52,559
that's all I have to say and we've got

844
00:27:50,880 --> 00:27:54,240
tons of time for questions if you're

845
00:27:52,559 --> 00:27:56,100
interested in looking at the example I

846
00:27:54,240 --> 00:27:57,000
showed you um it's up on GitHub and if

847
00:27:56,100 --> 00:27:58,740
you're interested in the docker client

848
00:27:57,000 --> 00:28:00,240
Library um I showed you what I mentioned

849
00:27:58,740 --> 00:28:01,440
before and that's available in GitHub as

850
00:28:00,240 --> 00:28:03,919
well

851
00:28:01,440 --> 00:28:03,919
thanks

852
00:28:03,930 --> 00:28:09,320
[Applause]

853
00:28:06,539 --> 00:28:09,320
yeah any questions

854
00:28:16,740 --> 00:28:20,279
thank you for your talk

855
00:28:18,360 --> 00:28:21,960
um so when you in the last part of your

856
00:28:20,279 --> 00:28:24,840
show where you were generating the the

857
00:28:21,960 --> 00:28:26,700
code from the from the yaml

858
00:28:24,840 --> 00:28:28,260
um is there a way to keep that in sync

859
00:28:26,700 --> 00:28:30,720
so if you if you want to update the the

860
00:28:28,260 --> 00:28:33,360
structs do you have to sort of copy that

861
00:28:30,720 --> 00:28:36,539
in again or is there an easy sort of

862
00:28:33,360 --> 00:28:38,039
tooling to oh so like if I was to make a

863
00:28:36,539 --> 00:28:39,120
change to that yaml does it generate

864
00:28:38,039 --> 00:28:41,520
everything else automatically is that

865
00:28:39,120 --> 00:28:43,860
yeah exactly like um or or you know vice

866
00:28:41,520 --> 00:28:46,080
versa if you if you went and made a

867
00:28:43,860 --> 00:28:48,659
change to the the

868
00:28:46,080 --> 00:28:50,159
um kotlin struct or the goaling structs

869
00:28:48,659 --> 00:28:51,900
or something like that

870
00:28:50,159 --> 00:28:54,000
um if you can get that back out into the

871
00:28:51,900 --> 00:28:55,740
m1m synchronize it to the other side

872
00:28:54,000 --> 00:28:57,900
right right

873
00:28:55,740 --> 00:28:59,820
um so the yaml is the source of Truth

874
00:28:57,900 --> 00:29:01,200
and so if you change the angle it will

875
00:28:59,820 --> 00:29:03,059
generate everything else again and

876
00:29:01,200 --> 00:29:04,440
that's built I've built that as a Gradle

877
00:29:03,059 --> 00:29:06,419
task that runs automatically as part of

878
00:29:04,440 --> 00:29:07,320
the build process so that stays in sync

879
00:29:06,419 --> 00:29:08,460
with that

880
00:29:07,320 --> 00:29:09,539
um and yeah if you made a change to any

881
00:29:08,460 --> 00:29:10,860
of the other source files that would get

882
00:29:09,539 --> 00:29:11,880
overridden

883
00:29:10,860 --> 00:29:13,080
um so you'd have to change the

884
00:29:11,880 --> 00:29:15,299
generation tool to do whatever you

885
00:29:13,080 --> 00:29:16,919
needed

886
00:29:15,299 --> 00:29:19,640
thanks

887
00:29:16,919 --> 00:29:19,640
anyone else

888
00:29:23,039 --> 00:29:27,200
I swear I haven't planted questions with

889
00:29:24,659 --> 00:29:27,200
people that I know

890
00:29:27,659 --> 00:29:33,059
just in terms of the um Docker HTTP

891
00:29:30,299 --> 00:29:34,620
versus Girling interfaces uh what do

892
00:29:33,059 --> 00:29:37,320
maintenance burden do you think you've

893
00:29:34,620 --> 00:29:38,880
traded off in terms of likely changes

894
00:29:37,320 --> 00:29:41,100
from Docker side that you'll need to

895
00:29:38,880 --> 00:29:43,460
keep supporting

896
00:29:41,100 --> 00:29:43,460
I think

897
00:29:44,039 --> 00:29:48,480
one of the big ones was buildkit and the

898
00:29:46,740 --> 00:29:50,399
like the image Builder

899
00:29:48,480 --> 00:29:52,260
um so

900
00:29:50,399 --> 00:29:53,100
the first version of Docker had first of

901
00:29:52,260 --> 00:29:54,720
you aren't familiar with history like

902
00:29:53,100 --> 00:29:56,940
the first Docker had a built-in like

903
00:29:54,720 --> 00:29:58,980
what's called The Legacy Builder now

904
00:29:56,940 --> 00:30:00,360
um and then they came up with a newer

905
00:29:58,980 --> 00:30:01,980
smarter way of building images called

906
00:30:00,360 --> 00:30:03,240
build kit

907
00:30:01,980 --> 00:30:04,740
um the trade-off with that is that it

908
00:30:03,240 --> 00:30:07,440
requires significant client-side logic

909
00:30:04,740 --> 00:30:09,059
to make that smartness work and so yeah

910
00:30:07,440 --> 00:30:10,260
one of the things that I am saving

911
00:30:09,059 --> 00:30:12,840
myself is having to implement all that

912
00:30:10,260 --> 00:30:14,760
client-side logic and as I improve that

913
00:30:12,840 --> 00:30:17,640
and continue to evolve it like I'll have

914
00:30:14,760 --> 00:30:19,880
far less to work on to keep in sync with

915
00:30:17,640 --> 00:30:19,880
that

916
00:30:21,419 --> 00:30:24,200
anyone else

917
00:30:25,679 --> 00:30:28,799
no awesome well yeah thanks so much for

918
00:30:27,480 --> 00:30:31,090
coming um and yeah I'll be around all

919
00:30:28,799 --> 00:30:33,360
day if you've got any other questions

920
00:30:31,090 --> 00:30:35,840
[Applause]

921
00:30:33,360 --> 00:30:35,840
oh