foreign
[Music]
talking about embedding golang in a
kotling app
um Charles is a senior engineer on the
mamier team at grafana Labs we're
particular interest in developer
experience Automation and Cloud native
infrastructure in today's presentation
Charles will outline his experience when
he tried mixing kotlin golang and why
doing so was almost certainly a bad idea
please welcome Charles
round two thanks everyone
um yeah I begin today by acknowledging
the warranty Warrior rung people of the
cooler Nation the traditional custodians
of the land on which We Gather today and
pay my respects to their Elders past and
present I extend that respect to any
Aboriginal and Torres Strait Islander
people here today
I'm here as you just heard my name is
Charles I'm an engineer at grafina Labs
I'm interested in a bunch of different
things including developer tooling and
open source I'm a maintainer of a bunch
of different open source projects
including protect and camel and
contribute to many other projects uh and
yeah while grafana has been very
generous in allowing me to come today
and give me time to do this
um this is completely unrelated to
everything I've been doing at grafana
um for reasons that will come obvious
shortly
um yeah this talk is all about something
very unusual that I've been doing
recently uh the idea of embedding golang
in a kotlin app and before I get into
the inevitable question of why on Earth
I've been trying to do that and explain
why it's a terrible idea I'm going to
talk about the concept in general
break it down to a few smaller problems
and talk about when those problems might
actually be relevant and be a sensible
thing to do you know in the real world
and of course there'll be plenty of time
for questions at the end
so let's start at the beginning and
let's start with the big picture of how
this all works you know how can a kotlin
how can I call an application call a guy
Lang function
the whole problem really boils down to
one one thing the magic of C interrupt
what do I mean by that
both kotlin both the jvm and the native
flavors can call C functions
angolang can both call C functions and
expose go functions as safe functions
and so we can take advantage of that and
use c as a bridge between kotlin and
golang
this this technique is not special it's
not unique to these two languages like
you can use this to bridge lots of
different languages almost every
function has some way to call C
functions because of you know the fact
that most operating systems expose
themselves as as say functions
um so things like Python and ruby.net
can all call C functions
and then many others including rust can
expose functions as they function so
they can be called by anything that can
cause a function
but yeah if it's starting to focus on
kotlin and Garling
so how does this work how can kotlin
call a c function and if and therefore
call a guideline function throughout
this presentation I'm going to use a
really really simple example of a golang
function that takes in someone's name
and gives you back a greeting for that
person you know it's sort of a classic
hello world kind of thing
at a high level it looks a bit like this
first of all we need AC function to call
and the C definition of it looks a bit
like this it takes in a pointer to a
child so it's a string and returns
another string
as I just said before golang can expose
going sorry go link expose Girling
functions as a c function and so we can
use that to provide this the C function
that we need
unlike what we normally do with the go
go code and turn it into an executable
application like a binary we can build
that and package it up as a library so
we can package it up as a shared object
or a dll or a dialib depending on which
operating system you're using
or we can make it into a static Library
which we can then build into our
application
on the other side we then need to call
that c function from kotlin on the jvm
we can interoperate with any c function
through a bunch of different libraries
I'll talk about in a bit more detail
later on
with kotler Native the support is even
built in so we don't even need a library
which is awesome and we can directly
call any c function thanks for its
built-in C interoperability
so now the problem decomposes down to
these two parts
one is exposing Golan code as a c
function and the other one is then
calling that c function from from kotlin
so let's start on the dialing side how
do we expose a go function as a c
function
the first thing we need to do is Define
our function so how many of you have go
developers or familiar with go
most of you all right for those of you
who aren't I'll quickly explain what's
going on
um so this is a definition of a function
it's called generate greeting it takes
in a single parameter call of type c dot
char and returns another parameter that
is appointed to a CE char
and then yeah all it does is then format
in a message or sorry format a string
that has a message hello from golang and
then the name that you pass in
yeah
those of you who are go developers will
instantly notice that the parameters are
not using the ordinary go string type
they're using this pointer to a c char
the reason for that is that go has very
very strict rules about memory
management and in turn has very very
strict rules about using strings because
of the way that the string type is
constructed and going and so therefore
you have to use a special C type rather
than the go type when you're passing
strings back and forth and so that's
what's going on here
all right so we've got a function it
will return the the value that we need
we do need to do a couple more things to
make this available to the world of C
the first thing we need to do is add in
this magic input input C this gives us
all the c times we just use in
particular C dot Char and the helper
methods as well
but it also enables cgo and cgo is the
tool that's built into the go tool chain
that enables interoperability with the
world of safe
once we've got that in place we then
need a magic comment on this function
that says export generate grading and
cgo looks through any file where you
import C like this and we'll generate
and looks like those comments sorry
and we'll then yeah make that available
to the world to see it looks close it
out for you
the last thing we need to to make this
work is a little bit of boilerplate so
if you want to expose anything to C cgo
insists of that is in the main package
and go so we need that and if we're in
the main package then we need a main
function doesn't have to do anything but
it has to be there so it's a little bit
of extra things that we need need there
so with all that in place we now have a
function that can be exposed to the
world of c and can be called anything
that can call a c function
the next step is to then take the source
code and build it and so again like it's
not a huge huge change from the The
Ordinary World of ago where we're
building a binary but instead we are
going to build a library and we need to
tell the compiler to do that
and to do that we just pass an extra
argument to go build so there's two
options here you can build a shared
Library so a dll or a dialib or a shared
object file or you can build an archive
which is something that you would like
statically link into a final binary
we can run that um ago we'll go off and
do its thing and it will produce that
that library for us and it will also
produce a header file that we can then
import into our C code to actually
invoke that function
so that's what it looks like on the go
side of things let's look at what this
looks like from the kotlin side
I'm going to talk about both the jvm and
kotler Native separately because it is a
little bit different
um but yeah most of you when you think
of kotlin will probably think of kotlin
jvm so I'll start there
um everything I'm about to talk about is
in fact not limited to kotlin on the jvm
it's anything jvm at all so everything
that I'm about to talk about is equally
applicable to Java to Scala to groovy
anything else that runs on the jvm you
can use the same techniques for this
and there's lots of different ways to do
it so I'm going to go quickly through
these and talk about each of them and
talk about when you might want to use
each of them as well
the first and the original is
jni which has been built into the jvm
since version 1.1
it has really great performance however
it does have the drawback that you need
to write a c function of your own that
translates from the C's sorry from C to
jvm C types and then you need to compile
that c function for every single Target
that you have
um which is a pain in the ass like if
you're you know targeting Linux and mac
and windows and you want to Target x86
and arm maybe doing 30 32-bit and 64-bit
something you've got to cross compile
this for I don't know six or eight
different architectures it's awful
um so yeah it's generally not what
people do these days
the next option is jni jna is an open
source library and has a much nicer
developer experience so rather than
writing some C to translate back and
forth between the world of c and the
world of the jvm
instead what you do is you write a Java
or a kotlin interface or a class for
structs and internally jna does the
mapping for you
however there's no free lunch here this
does come at a free at a pretty
significant performance penalty
so then of course another thing came
along that improved on that so jnr came
along it is an open source Library as
well it is quite widely used in
particular it's used by things like J
review so like it is it is quite battle
tested
it has the nice developer experience of
jna but also has much much better
performance as well it's still not quite
as good as jni but it's likely good
enough for most scenarios
all right and then the fourth and final
option is What's um now known as project
Panama uh so this is something that's in
preview for the jdk it's a replacement
for jni it's built into the jvm
um it's been going through the Java
enhancement proposal process which is
quite a mouthful for the last two years
uh it was in preview in Java 8 19 and
we'll be in second oh sorry is in second
second preview now for Java 20.
and yeah like this is a project fan
number is really cool it combines that
really nice developer experience that
you get with something like jna or jnr
with the better performance but only
something built into the jvm can give
you
another really nice thing about Panama
is that it has some really nice tooling
support as well
um so yeah like J J and A and J and R
you need to write that kotlin interface
for the Java interface to translate
things that's still true for Panama but
Panama also has some nice tooling to
generate those things for you
automatically so you don't have to do
that that manual work
um yeah for the example I'm going to
show in a second I'm going to use jnr
for two reasons one it's the thing I'm
most familiar with and the second one is
that it's something that you can most
readily use today as well
so let's see what this looks like in
terms of some code uh the first thing we
need to do when we want to call a c
function is take our C function
definition
um so this is what it looks like to
remind you and we need to translate that
to an interface
um so it looks like this in kotlin
so
jaina requires that every Library you
want to call be represented in terms of
interface for the different functions
you want to call
the method name the kotlin method name
in this case generate grading has to
match the corresponding C method um so
generate grading has to be the same in
both places
the parameter names on the other hand
don't matter
what does matter for parameters though
is that the order of the parameters has
to be the same and the types of them
have to match
um
what is kind of cool about jnr is that
at least for method calls it will
translate kotlin types or jvm types in
general to their corresponding C types
for you automatically so we can use an
ordinary string here and it will
automatically translate that to a
pointer to a char
so that's the translated method next
what we need to do is tell jnr
how to take the C library on disk
and turn that into you know some kotlin
code that we can actually call
I'm not going to go into all the details
because it is a lot and I recognize that
I'm putting a lot of code on a slide in
a presentation
the the basic idea is that we need to
take that dll or that dialib or that so
package it up into our jar
at build time and then at runtime pull
it out of the jar
um tell Jay and I where to find it on
disk tell it what the kotlin interface
is and then it will do the magic behind
the scenes to map those things and make
it work
with the library loaded we can then use
that library and we can call functions
from it so for example if we wanted to
call the generate grading method this is
a really simple application that would
do that we go off and load the grading
Loop using this method above and then we
can call it just like any other kotler
method there's no extra sort of work
required like what we saw with Scotland
with them go link sorry
so
so that's what a method call looks like
um just for sort of complete this I've
also got an example of what a struct
looks like
so if you're passing something more
complex than just a primitive data type
you need a struct and see
so let's say we've got this c-type here
that represents a human so it takes a
string for a name an unsigned 64-bit
integer for age in case you're really
old
and a Boolean to you know whether you're
an admin or not
this is how that looks in kotlin with
jaina
so it's almost an ordinary class uh
there are a couple of key differences
the first one is that it has to extend
from the special struct uh Base Class
that jnr provides
the properties must usually special
types that J R provides as well like
you'll notice these aren't the ordinary
kotlin data types for Strings or numbers
or even booleans and the reason for that
is that yeah that allows jnr to do the
translation back and forth to see
and take it and also consider
situations like if you've got like an
INT in C you know handling the fact that
is a different size on different
platforms
the name of the class in this case the
name of the properties as well don't
matter it's just the important thing is
that the order of the properties is the
same as the C type and the the types
match or at least map
um back and forth
so um that's the jvm let's look at what
this looks like for kotler Native as
well
unlike targeting the jvm with kotlin
Native it's there's only one way to
interoperate with c um so it's much much
more straightforward and it's built in
which is really cool as well
so let's go through the same example on
that side again we start with the c
function definition that's the same as
before
um yeah like I just said given how
common of a scenario this is with kotlin
Native
um cotton native doesn't require to do
any like translation ourselves instead
it has a tool that's built in called C
interop that's part of copper NATO's
built-in tooling it's built into the
Gradle plugin and it will generate the
method stubs and the destruct
definitions for us automatically
what we do do what we do need to do
however is tell C interrupt how to do
the translation
so there are three bits of information
that are really important the first is
the kotlin package that we want the the
methods and the the types to be
generated in so that's that first line
there we need to tell it which C header
files it needs to scan to to work out
what the methods are so that's the
second one and the last one is telling
it the name of the library file to
statically link into our application so
in this case unlike with jvm we have the
option to statically compile a binary
and so yeah doing that makes life much
easier so that's what we're doing here
all right so then with that out of the
way
um we can then write an application that
calls that method just like any other
um again it looks very very similar to
the jvm example the only difference here
is that similar to what we saw in golang
kotler Native does not hide the fact
that you're translating back and forth
between C types and so that's why we've
got these extra methods here on the
string where we call C string to turn it
into a C string and then the 2K string
method on the end there to turn that C
string back into a kotlin string so you
can deal with it like normal
so that's how you do this
um let's talk a little bit about when
you might actually want to do this in
the real world
like
the first little hypothetical question
is like when would you actually want to
call a go function from kotlin code the
answer is usually you wouldn't and
usually you shouldn't
uh when would you want to call a c
function from kotlin code that's more of
a normal thing to do
um however I will put the caveat there
that usually there is a really great
Pure kotlin or pure jvm library at least
that you can use to do most most things
um so again like this should be an edge
case it shouldn't be a common thing
however like one particular scenario
where this is a really common
requirement is where you're dealing with
operating system or dealing with devices
and things like that if that isn't
exposed to you through the standard
libraries um then doing this is a
perfectly normal perfectly legitimate
thing to do
um and the way I've just shown you is
how you would do that
but if you do go down this path you do
need to be prepared for some pain and in
particular when calling CE from kotlin
um there are two major sources of pain
to be aware of
the first one is memory management so
unlike in kotlin or many languages on
the jvm where we generally you know
don't need to think about memory
management as soon as C is involved you
need to start considering who is
responsible for memory thinking about
who's going to be responsible for
freeing that memory who's gonna like
when that's gonna happen
um it's not super difficult I mean
people have been doing it for decades
but it is really error-prone and it can
be quite difficult to reason about
especially when you're thinking about
like mapping back and forth between the
two worlds
um fun fact the example I just showed
you actually leaks memory we need to
free the string that we got back from
golang
um because yeah otherwise it will just
sort of sit there forever
the consequences are getting memory
wrong can vary from simply just
consuming too much memory um which is
usually not the end of the world all the
way up to things like memory corruption
security issues and crashes as well
the other big source of pain is the
impact on the developer experience
so usually you can't use a kotlin
debugger to step into C code for example
um so that can be quite annoying if
you're targeting the jvm you need to
handcraft those those interface
definitions or those class definitions
and that can be error prone it can be
quite tricky to diagnose when you get
that wrong
um of course in kotlin Native like
you've got that's the interrupt tooling
and so that makes it much easier
and finally you need to have an
understanding of both kotlin and c and
how those two worlds work together
and that adds extra cognitive load of
course so in summary two big sources of
pain for calling C from kotlin
when we add gulling into the mix that
just compounds uh so memory management
becomes more involved again because now
we need to consider both how sorry we
need to consider how golang NC and
kotlin all want to work together and how
they each manage memory and how those
things don't quite line up
um to give you like a concrete example
so I mentioned before uh talking about
the go string type that go has very very
specific rules around what you can do
with go manage memory and that makes
seeing drop more more complicated there
are certain rules about when and when
you can and cannot pass go pointers out
outside of go because otherwise the
garbage collector can't work properly
um and of course like you can't use
things like a debugger across the
language battery either
but as you probably guessed from how the
slide is laid out uh when you add
Girling into the mix you get some
exciting new sources of pain as well
uh the first one is around tooling
so when we're working like this not only
do we need a growling tool chain and a
Java tool a kotlin tool chain we need to
make these two things work well together
so that either means doing something
like running the golang tool like
tooling from Gradle or having some other
alternative tool like bazel for example
that orchestrates the whole world as
well
on top of that cgo needs access to a c
compiler and a tool chain that can
compile for each operating system and
architecture Target that you have and
for those of you who've ever tried to do
that you know that's not straightforward
and I'll talk a little bit more about
that later on
lastly when you're moving back and forth
across language boundaries you're
limited to the lowest common denominator
of what's supported by each and this has
a particular impact on things like data
structure design method signatures and
can make things like callbacks and
things like that quite tricky as well
so in summary there's lots of pain
and hopefully at this point you're
thinking like you know this has been
interesting so far but like it sounds
like a terrible idea like why on Earth
did you do this
um and the the answer is that I'm
trading one kind of pain for another
and to explain why I need to talk a
little bit about an open source project
I mentioned right at the beginning
um called protect
so protect is a tool for managed for
manage for managing but dockerized local
development environments
it is written in kotlin for reasons you
can read about on my blog
and most of the docker ecosystem as many
of you all know is written in golang
um
however the docker demon exposes an API
or HTTP API
um that in theory should be accessible
to any language right it's HTTP it
should be straightforward
the reality of that however is not quite
as nice
parts that API are really poorly
documented or just aren't documented at
all
parts of that API are implicitly use
goaling conventions use constants from
the standard library for example and
those don't map nicely to operating
system constants for example
there are parts of the API that are
really non-standard if I'm being polite
or really weird if I'm being less polite
um for example when you go to build an
image using build kit the new image
Builder you as the client so in this
case protect send HTTP HTTP request to
the demon and say hey I'd like to build
this image the docker name says that
sounds awesome I'm going to take that
TCP socket you use for HTTP request and
use it to send you grpc requests
um and it's it's weird and of course
yeah the most HTTP clients do not
support that kind of weird
having the TCP socket taken away from
underneath you which makes integrating
with this really tricky
the other the other pain with the API is
that a lot of it requires significant
client-side logic that needs to be
re-implemented uh so things like piping
container i o back and forth is really
tricky
um dealing with registry credentials for
pulling and pushing images is really
tricky Building images as I just
mentioned is really tricky all of these
things require quite significant
client-side logic and none of it is
documented none of it is written down
anywhere and so I've had to reverse
engineer the goaling client to
understand that
so
the official Docker client library for
golang you know influence all this stuff
it takes care of it all of it takes care
of all of it for us nice and easily
and there is no fully featured Docker
client out there for kotlin or the jvm
in general some of them do cover simple
scenarios but none of them cover the the
all the scenarios that I need for what I
do and protect
so that's what I did I ended up having
to build my own
um and so now or previously Patek had a
battle tester Docker client written in
kotlin that worked had all the features
I needed it supports all these weird and
wonderful things that the API does
however maintaining that client and
adding features to it was painful for
all the reasons I mentioned before
um so that poor documentation that
implied user golang the the really
non-standard interactions and that
significant client-side logic that I had
to reverse engineer
so it'd be great if I didn't have to to
do that in kotlin and it could could
instead build upon the existing Garland
client
um and so that's what I that's what I
ended up doing
um I've rebuilt the tech Stocker client
as a wrapper around the golang client
and it's now used by protects many many
hundreds of users every day and none of
them have complained so that's that's a
good sign
um and yeah like despite all the sources
of pain that I mentioned before this
approach has been working really well
for me I've traded the pain before
documentation and the pain of implied
used to go laying and significant
client-side logic for more predictable
pains that I mentioned on the previous
slide
so things like memory management and
developer experience and tooling and all
that kind of thing
and because those pains are more
predictable like I've been able to
develop patterns and tooling and things
like that to help me deal with those
pains and make them less painful
so I've got two examples of that
the first is a round tooling
so I mentioned before that you've got to
make these two tool chains sort of work
well together
um I ended up building a Gradle plugin
that can take some Girling sources and
compile that down to static and dynamic
libraries for each Target
and then use Zig as the C cross compiler
to satisfy um cgo's need for a c tool
chain who's heard of Zeke by the way
okay a couple of you awesome for those
of you who haven't heard of it it is a
really cool language in its own right
but one of its coolest features is that
it can act as a c cross compiler and it
largely just works
um which is really really cool like if
for those of you who've ever tried to do
cross compilation you know it's usually
a pain Z get actually kind of works and
it's really nice um so yeah give it a go
if you haven't haven't seen it before
the other thing I've spent quite a bit
of time investing in to make my life
easier is looking at making memory
management easier
and to do that what I've done is I've
built a small code generation tool that
takes in the definition of structs in
yaml like this so for example here we've
got a an arrow type that has a type and
a message both of which are strings
and from that single gamble definition
that generates a bunch of different
things that I need
first of all it generates the
corresponding C definition
and it will translate the types for me
automatically
it will generate C functions to
initialize and free pointers to that
type
um so again like it will allocate one
using Malik and then initialize it and
then I've got another one that I can use
to free it when I don't need it anymore
it will also generate go functions to
convert from the go types to the C types
really easily so for example here we can
initialize a new error pointer
um and taking the go string types so I
can keep the sort of the messiness of C
constraint contained in this method and
the rest of the the go code base is
relatively clean
it also generates the corresponding jvm
class definitions so similar to what we
saw before we get a class that looks
like this from that yaml definition
you'll notice that one big difference
here is that this one implements Auto
closable as well which means that we can
use the trial with the result resources
pattern to make sure this is cleaned up
properly
um and yeah that means that it then goes
off and calls that c method we just sort
before to free the memory associated
with it as well
and it will also generate the
corresponding jvm interface definition
for the methods that were generated
automatically another part of it also
takes in the C header generated by cgo
and translates those methods as well so
you end up with one giant interface with
all the different methods um and I don't
have to write any of that myself which
is awesome
um and yeah that saves me a bunch from a
bunch of tedious and error print work
so
um that's why I'm doing this and those
are the things that I've had to do to
make it work
to to recap
um you can use c as a bridge between
different languages
both flavors of kotlin can call C
functions uh and going can both call C
functions and expose go functions to C
functions try to send that five times
fast
there are lots of ways to do this on the
jvm each with different trade-offs I
would recommend J R as a sensible
starting point unless you've got
particular constraints
um we've got no difference really
straightforward though
um and yeah please don't actually do
this
that's all I have to say and we've got
tons of time for questions if you're
interested in looking at the example I
showed you um it's up on GitHub and if
you're interested in the docker client
Library um I showed you what I mentioned
before and that's available in GitHub as
well
thanks
[Applause]
yeah any questions
thank you for your talk
um so when you in the last part of your
show where you were generating the the
code from the from the yaml
um is there a way to keep that in sync
so if you if you want to update the the
structs do you have to sort of copy that
in again or is there an easy sort of
tooling to oh so like if I was to make a
change to that yaml does it generate
everything else automatically is that
yeah exactly like um or or you know vice
versa if you if you went and made a
change to the the
um kotlin struct or the goaling structs
or something like that
um if you can get that back out into the
m1m synchronize it to the other side
right right
um so the yaml is the source of Truth
and so if you change the angle it will
generate everything else again and
that's built I've built that as a Gradle
task that runs automatically as part of
the build process so that stays in sync
with that
um and yeah if you made a change to any
of the other source files that would get
overridden
um so you'd have to change the
generation tool to do whatever you
needed
thanks
anyone else
I swear I haven't planted questions with
people that I know
just in terms of the um Docker HTTP
versus Girling interfaces uh what do
maintenance burden do you think you've
traded off in terms of likely changes
from Docker side that you'll need to
keep supporting
I think
one of the big ones was buildkit and the
like the image Builder
um so
the first version of Docker had first of
you aren't familiar with history like
the first Docker had a built-in like
what's called The Legacy Builder now
um and then they came up with a newer
smarter way of building images called
build kit
um the trade-off with that is that it
requires significant client-side logic
to make that smartness work and so yeah
one of the things that I am saving
myself is having to implement all that
client-side logic and as I improve that
and continue to evolve it like I'll have
far less to work on to keep in sync with
that
anyone else
no awesome well yeah thanks so much for
coming um and yeah I'll be around all
day if you've got any other questions
[Applause]
oh