[Music]
welcome back everyone
um next up uh we have mohamed for luck
talking about uh
ebpf101 mohamed verlak is a software
engineer who is interested in how the
ebpf subsystem
can be leveraged in novel ways to enable
new use cases
uh please welcome muhammad vlad
hello so this is uh ebpf101 talk i hope
everybody can hear me all right so the
idea is that we only have 20 minutes 20
or 25 odd minutes and plus questions so
what we're going to do is we're not
going to do any programming sort of
learning
what's how to program the eppf subsystem
but it's a it's an overview from a
perspective of non-kernel programmer of
what the cbpf subsystem means because
there's been lately a lot of buzz around
the ebp subsystem
so
a little bit about me who am i i
recently finished this school book
pretending to be a linux kernel expert
pun intended
i come from a very beautiful part of the
world called srinagar
kashmir
and
yeah i am no way an expert in all of
this i just am reading it for fun and
trying to you know leverage it and use
it in my own day job and try and do
more experiments with it
so
yeah let's look at the agenda so first
of all
we are going to look at the history of
what this bpf thingy is
and then we are going to move to the
more recent the current parts of the
ebpf where the e comes from
so
without further ado let's get started
so let's go back in time probably to the
90s and assume that there's no
tcp dump and we want to design a packet
filter so
what could be our sort of
design
goals are to design a packet filter
which copies or
gives us a way to look at every packet
that goes out through the wire and
inspect it
so how could be implemented there are
generally
two
ways these are not exhaustive ways but
generally there are two ways
one could be
you copy everything that goes throughout
the wire to user space and then apply a
filter on that of whatever is
interesting to you whatever is not
the other way could be more optimal
where you write a kernel module and you
say if x is the destination port
x is the source why is the source port
blah blah blah you load that model in
the kernel it only copies the packets
what you wanted to look at were
interesting to you but again we we are
going to talk about the trade-offs
involved so what's the problem here
if we have the user space implementation
we copy everything it's it's
like
a no-brainer you copy everything what
goes on to the wire to the user space
and then
you do what you need to do with that in
the kernel space if you implemented that
module thing what would have happened
you hard code whatever you wanted
to look at what was interesting for your
use case and then only that thing gets
copied to the user space and you take a
look at it at the end of the day you
have to copy stuff to the user space the
only optimization that we are looking at
is how much do you copy
and uh if you look at if you look at the
trade-offs here in the user space thingy
it is not optimal
and the kernel
module thing
it is pretty optimal because it just
copies what you want and you're not
doing more than
more work than it's required the user
system implementation is a generic
solution you can implement it once sort
of have that switch in the driver or
whatever
and just just be done with it whenever
you want to do some tcp dump thingy you
just nudge the driver it copies
everything to your user space and then
you do the packet processing and then
apply filters for example there are 100
packets copied
you could only sort of the interesting
ones could be only three or four in the
kernel side you only copy the three
drivers but this solution is not generic
because you had
hard coded or you had written the module
and
it only would apply for that destination
and that that that that destination port
and that that whatever protocol you're
looking at
there's one more interesting case here
is the user space solution is a little
safer when we say safe i mean if there's
a bug in your code what worse could go i
mean what could go wrong
you basically could have a sick fault in
the user space that's not too bad but
nothing is going to go too bad but if
the kernel module had a bug
the whole system is going down
probably i mean the sanity of the system
is in question
so
what would be the right way to do things
here
yeah i mean what if we had best of the
both of the worlds like we had like a
generic solution
and an optimal performance so what would
that kind of a design be i mean how
would you how would you do it
so
in 1992 the psd packet filter this paper
it's a seminal paper that was published
where
the folks designed a novel architecture
to do exactly the same things of how do
we implement tcp dump and do it
optimally
so
they had their in interesting design
choice what they did is
they implemented a vm a simple vm which
resides in the kernel and
it just could not do much
it just does a bunch of loads
a bunch of stores
a little bit of jumping
jumping around very basic arithmetic
operations returns and it had like an
accumulator register and one more like x
register so you could do transfer one
instruction
i mean data from the accumulator to the
x register or maybe from the x register
to the accumulator register nothing
fancy at all so you have
basically a vm that resides in the
kernel
and you you operate that and you do
something with that which we'll come to
shortly
so
okay now you you what you've done is
you've created a vm you've dumped it
inside the kernel but how do you run
a ppf program
okay let's take a digression first i
mean how
do user space programs work
you could have a compiled program you
could have an interpreted program for
compiled programs you write the piece of
code
then you give it to the compiler
plus the linker you get a binary out of
it and then you go ahead and you run the
binary
and everything sort of you get the
output you wanted to write a hello world
program you write the source code
compile plus link it and do an a dot out
dot slash a dot out whatever and it just
prints it on the screen
you as a programmer or whoever wants to
use the program has the control on when
the program runs
in the interpreted case similar thing
you write the code you hand it over to
the interpreter it juts out the
instructions and then executes them
but how do you do it in the ppf side
because the bp vm is inside the kernel i
mean you don't have control over the
kernel you can't really really just go
and run whatever you want in the kernel
the only interfaces you have with the
kernel
is the syscall or maybe some interrupts
but generally it's just the syscall so
do you have a direct way of nudging the
kernel to run a program
probably
i mean would that make sense
we'll we'll look at it
so
how does a ppf program
run
bpf programs it's very important to note
that bpf programs unlike the normal
programs are not
dependent upon the programmer or whoever
wants to use them they do not run on his
wish
they are mostly event driven so there
are a bunch of events in the kernel that
are placed or a bunch of hook points in
the kernel and
whenever you write a ebf ebpf program or
a bpf program
you write that in the instruction set
that that simple instruction set you
target for that vm
you load the kernel wire you load the
program via a syscall into the kernel
it still will not run because you just
have loaded the program
then
you attach it to a particular hook point
so
let's let's actually look at it the bpf
programs are stateless that's an
interesting point to note that these
programs are very simple they run to
completion and you load them to the
kernel
you first of all write the filter
expression the filter expression is
pretty simple it's it's like here we are
going to go through this in a short
while but you write like
eventual answer should be true or false
you load the byte code in the kernel you
attach the loaded program to a hook
a hook could be for example every
received packet is a hook whenever
colonel receives a packet run this ppf
program
and the programs are event driven
and they run to completion there's no
sort of preemption or anything that
happens with it a pvpf event occurs the
bpf program runs to completion and at
the end it tells you yes or no i mean
and that boolean instruction at that
boolean return value can be used to do
very interesting sort of things
so let's just take a small example of
a very very simple bpf program that that
that comes via the paper i've just
copied it from the paper and this this
program here
is basically
a program that just filters out
ip packets
and which go to a particular destination
port so if you see
i'm doing a load half word
of the 12th offset in the packet so
every packet when it comes it has a
particular like the standard it has a
particular format and what you go is you
start with the packet starting and then
go to the 12th offset and see
is it ethernet protocol
and if it is then jump to l1 otherwise
l5
0 i mean i don't want to do anything
with it because it's not ethernet i
don't know what it is so get out
then you do a load byte on the 23rd
offset of the
whatever packet you had and then see if
it's tcp if it is tcp you go to the
this part of the
program otherwise you just put the
packet on the floor and then you keep on
munching a bunch of these small small
instructions that you do with it so here
for example the interesting one is you
load
if if this is an ip packet like the ip
head of the length of the ip header here
this this part is the length of the ip
header i mean it's not too interesting
of how exactly this happens but it's
it's just as a
notion here that these are very simple
instructions that you can do on a packet
receive and then
you know you return true or false and
depending on that result you could have
that if it's true i'm going to copy it
to user space if it's not i'm not going
to copy it to user space so this is
basically what's the vm and
how do you how do you write a bpf
program and then load it inside the
kernel and then make it run
uh before we move any further to the e
part of the bpf where does the e come
from we we need to look at some of the
ideas that are similar to ppf
this is interesting and of course it's
it's a little orthogonal but it helps me
to make make sense of why do we need ppf
at the first place
so think of it as the embedded lua vm in
nginx to modify behavior for example to
check certain headers if there is a
certain header in the http request
uh allow it and if there is none just
drop it on the floor
now there were two ways of doing this
kind of thing either you could pull the
nginx source code and then
add that piece of code in the c language
whatever language nginx is written in
and then compile it and whenever you
have to do anything you have to modify
that rule you always have to recompile
change and recompile
i i think that's a little unwieldy if
you want to do these kind of things and
having a lua vm embedded inside the
nginx module it helps you a lot
maybe a new of him i i don't think if i
want to extend the functionality of my
editor
i would want to pull the source code in
add that bunch of functionality inside
the source code i would rather write a
lua plug-in and then load it in the
so so these are these are some of the
things where it tells us that it's
probably easier to use the vm based
approach where you don't have to
recompile all the program and start from
zero
and
it's working out
great i mean i haven't forgotten about
emacs folks emacs
there are two ways to modify emacs
either
you modify the source code or you write
emacs lisp your init.tl whatever
so this this sort of this sort of notion
is is pretty
prevalent these days and it's pretty
pretty easy to use uh
it sometimes becomes a little difficult
to think in the sense of kernel that
okay why are we not changing the kernel
but we are
introducing a certain vm inside the
kernel and
increasing complexity but if you look at
it if you compare it with other things
it's pretty normal that you have your
own plugins and everything written a
different language than what the
original editor or whatever target you
were planning to use it on
was and it gives you a lot of
flexibility and a lot of the turnaround
time is pretty quick
so now let's move on to ebpf so
absolutely we need to introduce
ourselves to the ebp of mascot which is
the qtb
so where did this extended or e in the
bpf thingy come from alexey
sent a patch close to 2013 2014-ish
where he improved the existing bpf
infrastructure in the kernel the bpf
infrastructure was already in the kernel
and the prime users for that was tcp
dump because that's why it was sort of
gotten into the kernel it started off
from bsd but then it was very soon
ported to linux
and
if you recall
the bsd packet filter
just had hook points inside the network
stack only i mean all the hook points
were embedded there just for the network
stack
nothing else
what alexis patch did is first of all
improved the vm quality i mean earlier
if you remember we looked at the
instructions there were just a few
instructions a very small set of
registers to work with
this patch vastly improved on that
by making
uh improvements in the number of
registers you have the number of
instructions you could implement and
sort of write and overall performance so
this this makes it e this this adds the
e and then
you have hook points spread throughout
the linux kernel it is not only the
network stack that has the hook points
there is a bunch of other places where
the hook points are
so
yeah by the way if i if i was not clear
if if there are any questions please
feel free to ask them
uh and
now we have to talk a little about
uh
the c bpf or the old style bpf which is
also called as the classical bpf and
extended bpf so there are there are
differences between the c bpf the
classical
style of bpf and the extended style of
bpf the cbpf typically a very
constrainted vm will not let you do
very interesting things
while as ebpf is the extended one which
has hook points
throughout the kernel you could do a
bunch of things with it earlier you
could only do sort of packet processing
decisions
with the classical vpf but with ebpf you
could do much more for example
you could have a
hook point on every system call that's
executed and what one could do is
whenever a certain program does a system
call you could have a hook point and a
bpf program attached to that hook point
a program did a syscall
since the hook point is attached to the
cisco event it gets fired you check
whether this particular program
is allowed to make the syscall for
example a small application like a cat
like application that just has work to
get
you know file data which is the read
system call
and dump it on the screen or probably
redirect to a file does not have to do
anything with the network socket i o
calls
and
if for example for some reason you could
see malicious behavior you could deny
that so
the security aspect of it is very
interesting and you could also have hook
points in
there's there is that there are certain
points in the kernel called k probes
caret probes trace points these are
again
uh hooks where a certain function
executes there's a there's an event
attached to that so you could get a lot
of telemetry data from it and if you if
you look very
closely you could see there's this
penguin as well as the windows signs so
ebpf nowadays
runs generally on most of the platforms
that are available and windows was the
most recent edition like 2020ish
was the
thing that it started to come up and
it can run on windows of course you
can't run linux specific things
on like evp of things on windows but
generally the idea is there the vm is
there
so again the capabilities networking
absolutely it's it's where it started
from you could do a lot of networking
stuff with
ebpf
you could do a lot of security stuff you
could like we talked about you could
have a
ppf program which has a hook point and
just looks at a policy of whether a
program is allowed to do a certain
syscall or not
you could do a lot of observability like
if you have those hook points where
the trace points and the k probes you
could get you could gather a bunch of
information from the system dynamically
by attaching the ebpf program whenever
the hook fires you get that information
all the processing is done inside the
kernel and then finally you get the
answer now
why this is
fast because since you have a vm inside
the kernel you want to count how many
syscalls happened at a point x or i mean
from a to b time what you could do is
you could write that code and then have
internally the ebpa program gather all
the details do all the number crunching
and then finally when you're done it
just spits out the answer back to the
user space
uh copying data from
across the user space and kernel space
boundaries is costly so we want to do it
as little as possible and want to keep
the number crunching as local to where
the data actually is and we just only
want to look at the data
so
an interesting thing of the ebpf
verifier
and jit is
you can't really willy nilly run any
program
inside the
ebpfvm or the linux kernel i mean when
we talk about arbitrary program we talk
about like
we do that that very restricted set of
instructions that were given
that the ebp fem runs but you really
cannot do anything because
the ebpf program runs to completion now
what if a malicious user put an infinite
loop in that whenever an event fires the
cbpf program is going to go and then
just infinitely loop and since the
program runs to completion you basically
just start the cpu you did a denial of
service because now there's no way for
the system to yield the program it just
runs in the app so bpf verifier looks at
whenever you
load
the
ebpf program that you've written inside
the kernel via the bpf system call
it first of all goes to the verifier
the verifier looks at all possible
branches and whatever you have done in
your code and
first of all verifies the sanity of the
program if the program according to the
epp of verifier is same then only it
gets handed over to the jit compiler
which then emits out
native instructions for whatever
architecture you're running on and then
it moves along and does whatever it
needs to do
then you properly attach it to a
particular hook point because just
loading the program in the kernel is not
going to do anything when you load a
program in the kernel you have to attach
it to a certain
point
and uh then you do whatever you want to
do with it so let me look at the
questions
since ppf verifier evolves can we expect
that a ppf program written today for
5.16 will work in a few years oh okay
so i think
if we uh if if we look at it if
we look at the architecture of the bpf
vm it's generally very simple
so
if by what you mean is if you've written
a program today
and you have used
trace points and you have not you have
not relied yourself on
api in the kernel that changes for
example
we talked about k probes and trace
points unfortunately i do not have
enough time to talk about what those are
but trace points have a guarantee of
being more rigid like system calls
they're not going to they're going to
survive multiple kernel versions but k
probes don't give you that because
that's the internal kernel functions and
whenever the name of the function
changes the k probe changes so if you if
you you
use or leverage those kind of techniques
where you use k trace points instead of
k probes probably uh the ebpf program
should survive multiple kernel divisions
can ebpf store state between system
calls okay yeah sure that's an excellent
question i'm going to come to it
so
what are the different types of ebpf
programs recall in the classical ebpf
sense it was only sort of relatively
constrainted towards the socket filter
or the network stack
but in the extended sense
it's just
spread throughout the kernel so we're
going to talk about a little
interesting of those
some of the interesting this is not an
exhaustive list
so this bpf prog type sock filter what
is this this is basically a packet
filter
the thing that the original bpf vm
started with you apply a filter at a
particular socket that okay if this is
this destination port this source
address this
source spot blah blah blah
do something
there's another one called
xdp the bpf prog type xtp this is an
interesting one
i'm sorry this is an interesting one
because
xtp is express data path and this is a
bpf program which is attached at a hook
point which is as close to the device
driver as possible now
we have to talk a little about it so i'm
going to give one minute to it so what
happens here is whenever the packet
comes to your nic card
it first of all gets stored on your nics
memory and then from the nik and nick
network interface card and from the
network interface card it gets dmade
inside the linux kernel main memory and
then you raise a soft irq and interrupt
and then you you
tell the linux kernel that hey i got a
packet for you start processing it you
know take it into your levels of
bureaucracy of the network stack where
you first of all you know plug the
l2 header then the l3 header and then
you keep on moving it up the stack and
finally you do all sorts of you know
net filter
that kind of table manipulations and if
all is green you
give it to if it's for descent to your
process or if it was distant for some
routing
this xtp is interesting because the
moment
a packet arrives on your nic card and
it's dmade
you raise a soft irq request with the
nappy interface like the internal
network stack
you have a capability to run a program
now at that point in time
your packet is just a buffer it's not
even an sk buff yet it's just a buffer
you can do all sorts of crazy things
there you could redirect a program you
could
drop it on the floor and you'd say i
could do that with iptables why do i
need to do something there
well if you do that with iptables it
happens much higher up the lane i mean
it happens like after you've done the l2
and the l3 things so you have to
allocate a lot of space for it for a
packet that you probably did not want so
you could do that
right when you receive that packet and
if you were acting as a router or a
forwarder you could straight away ask
the bpf program
in this case xtp to you know
do something with it forward it via some
other interface
one interesting other thing that could
be done here at this point is you could
do
a kernel bypass like you've got the
buffer just
leave the kernel i don't want to go
through the network stack just straight
away deliver it to the user space and i
want to do whatever i want to do with it
like the raw socket thingy
the k probe price point sock ops are
similar and there are much more
now
recall the classical bpf was entirely
stateless ebpf as well is stateless you
can't really store state but it has the
capability to access storage which is
called bpf maps now
these maps are not not like actually a
key value pair but whenever we say ebpf
map think of it as storage ebpf storage
so a bp map is basically a generic data
structure that allows you
to pass data to and flow from the user
to the kernel and inside the kernel so
you create a bpf map by using the same
bpf syscall which is a multi-tool which
lets you do a lot of things lets you
load a program attach a program to a
particular hook point create a map
attach a map to a certain place do all
sorts of things with the map and a few
interesting map types are
a map type hash which is actually like a
key value store a map type array which
is just like a normal array a map type
prog array which stores file descriptors
of a bunch of ebf programs that you've
loaded and a bunch of other maps
recently there was a map type plume
filter that was added so you could do a
bunch of state state
stuff inside the kernel but this state
is global like
since the vf program comes in it can
access that map do whatever it needs to
do it does not store any state of its
own but it can do whatever it needs to
do in the state and then finally just
die
so i hope that answers the question
can ebp store state between calls
smoothing the input from pen uh i
i think i i talked about how how does it
store state but
probably we can take that offline
because we're pretty sure on time
and yeah what's the conclusion ebpf
programs
are not controlled by the programmer but
they run in response to events ebpm
programs run to completion now an
interesting thing here was i had put a
bracket here and i'd said that they're
not preemptive but then one friend of
mine his name is kartik he's one of the
epf developers he sort of sends patches
regularly he corrected me and said that
probably they can be preempted but not
migrated so i i just left that
and that's food for thought i myself
don't know much about it but just just
saying it
running an ebpa program is much safer
than running and maintaining a kernel
module now
what does that mean if somebody gave me
a
kernel module and said that yeah this
does an amazing solves an amazing
problem
can you run it in your production
environment i'd be very skeptical
because me running a kernel module in
production given by somebody it's it's
it's a little dangerous but on the other
hand if somebody gave me an ebf program
i would very well just try it out and
probably not in production but at least
i'll not be that hesitant because i know
the epp verifier is going to help me and
not cause any problems the entry bar to
get useful information from the kernel
is significantly reduced people like me
who don't know anything about the kernel
or probably are pretending to be kernel
experts can can sort of
know a lot about how the kernel works
and the overhead is just pay as you go
zero cost abstraction style i mean if
you're using it that's the only time
you pay for it it's it's minimal but
still you have to pay for it and
the bpf vm is already there
so
yeah
thank you and i think we are right on
time
thank you very much
um
luck
for that excellent introduction um
unfortunately we are at the end of the
time slot so we don't have time for more
questions uh if you do have more
questions feel free to uh contact for
luck offline
outside of the session and i'm sure
he'll be
more than happy to keep talking about
ebpf
um