[Music]
welcome back to the final talk today in
the yuma theater
this talk will be on nornet a unix and
open source software-based international
platform for networking research
the talk will be given by thomas
dropholtz
uh who works as a chief research
engineer for the similar metropolitan
center for digital engineering simula
met in oslo norway
thomas will endeavor to allow time for
questions at the end so please enter any
questions in the tab in venulis
take it away thomas
thank you for the introduction then
welcome to my presentation about our new
net infrastructure
first i will shortly introduce you the
motivation of our work
which is around multi-homing and
multi-pass transport and then i come to
some
interesting details about our testbed
infrastructure particularly with regard
to virtual machines containers and
multihold networking
so what's the motivation of our work
well let us look
first at classic internet communication
in the most basic property example
a browsing
when you use server browsing well you
have a pc as a web browser having an
ipv4 address
this browser connects to app server
which also has an ipv4 address
and then you have over the tcp protocol
with the 3bn check establish a
connection with 3bn ventrac and
we have classic client server
communication
this is the way most applications
in the internet
are working today
what is a quite old
way of communication
it is
likely it will work this way also for
many other applications in the future
but nowadays of course we have a bit
more than
such simple client server applications
we have internet of things
virtualization sensor networks cloud
computing vehicle networks and many
other things
and these have of course
more
strict requirements on the communication
particularly nowadays we not only have
the ipv4 protocol we have ipv6 and with
rpv6
usually we have multiple addresses for
interface and even multiple interfaces
and usually today of course we have ipv4
ipv6 dual stack devices so many
computers today have both ipv4 address
and ipv6 unless
and today of course we have also
mobility
so probably everybody of you has a
smartphone in his pocket
it's quite convenient device so if you
are at home you're using wi-fi
if you go outside the device changes
to
for example here in oslo it is uh five
three
maybe um
also 4g 3g depending on where you are
and of course the smartphone would even
have third interface
or bluetooth
same as classically
a router because the router classically
had these two interfaces
and of course also depending on what
kind of laptop computer you have we have
also multiple interfaces like ethernet
wi-fi and maybe even mobile broadband
built-in
so what can you do with multiple
interfaces
well you have the property that you are
multi-homed
if you have multiple other resistances
you are connected to multiple networks
so particularly
you can connect to
more than one internet service provider
to have redundancy
so look at this uh
user of material here
looks not that convinced about this
redundancy what's the problem with
redundancy is nice if something goes
wrong then you still have connectivity
but if nothing goes wrong
well if you are connected to two isps
each month you receive two internet
bills
for example in australia maybe an
internet bill from optos another one
from telstra
well
paying
double the amount necessary is not
really nice
obviously
so um in 99.9 something percent of the
time
nothing goes wrong and you do not
appreciate redundancy
so can you do something with a
redundancy
uh when everything is okay
there is the possibility to do
multi-pass transport if you have two rsp
connections
why not trust utilizing them both
simultaneously
then at least if everything is okay
usually
you get a higher bandwidth
this is of course nice
so
with tcp of course classical tcp
communication
is established between two ipv4
addresses this does not work but there
is an extension multi-pass sap
which supports multi-homing and
particularly multi-pass transport
and then it works
also if you are using the sctp protocol
which already has built-in support for
multi-homing
you can add an extension from current
multi-path transfer for satp cmp-sdp
to also have multi-path transport
so in either case you get multi-pass
transport
so you can utilize multiple network
connections simultaneously which is a
quite nice thing and therefore it is
also a topic in research and
standardization
so
let me shortly go into a little bit more
detail about multi-parts transport
particularly with mpdcp or cmkcp how
does it work
looking at the
osa motor stack
you have the application
protocol
in the application layer on top
of of course the kernel
which implements the transport layer
that is
mptcp or cmt-sctp as part of the
implementation of dcp or sctp
and application protocol communicates
with the transport layer over the socket
api
that is
the user space communicates with the
kernel
over this socket api
and network layer of course you have
ipv4 as well as rpv6 so as i've said
usually
you have a dual stack today
so now the transfer protocol can
establish paths or so-called subfloors
over the underlying network protocols
that is subfloors over ipv4 and ipv6 you
can even mix
some flows with ipv4 and rpv6 so you can
have
uh
mp3 connection
both using ipv4 and ipv6 in the most
extreme case
you can even start a connection in a
pure ipv4 network later at ipv6
addresses
and finally remove
the rpv4 addresses so you can upgrade
your system from ipv4 to fpv6
without breaking connections
it's of course quite extreme case but at
least it is possible of course
so now we have subflows but how to
put user data payload data onto the
surface and this decision is made by the
scheduler
also far so simple but they of course
are challenges for multi-pass transport
one of the shelves are particularly that
parts
in the network some frozen network may
overlap
particularly if they share the same
bottleneck link somewhere in the
internet
if we have multiple subfloors running
over the same share bottleneck
compared to a concurrent tcp connection
you get of course more bandwidth which
is unfair so fairness is one challenge
and the other challenge is of course
scheduling
when you think of subfloors maybe you
think of
subfloors having very similar
characteristics regarding bandwidth
delay and packet loss
but of course usually in the internet
such sub-rows are quite dissimilar
and then it makes sense to very
carefully think about
which subfloors to use and which
sufferers do not use particularly to get
a good performance performance maybe
bandwidth but also some applications are
latency sensitive
so maybe then it's not useful to use a
subfloor where you can get the highest
low port but where you have the lowest
contraption
so a lot of challenges
and of course the research for this is
necessary
but let me first introduce you how to
use mptcp
well it's defined as an rc but with some
open topics still
but with mptcp you can make a system
capable of multi-pass transport
by extending the
old tcp protocol
and particularly nice feature of mptcp
is that is quite backwards compatible so
on the wire communications and btcp
look like
normal
tcp connections so the subfloors look
like regular tcp connections that is
mptcp will also work usually over middle
boxes that particularly means network
and portables installation as well as
firewalls and such
so it's quite nice because of this
compatibility so it is easy to use
in existing networks
and the nice thing is there is a quite
good implementation for linux available
from university catholic dilemma in
belgium
and also swinburne university has some
years ago started an implementation for
freebase day
so there are two open source
implementations available
so um
how can you now use
it particularly on the linux because we
are here at the linux once
well um obviously you need an empty dcp
capable curl but is not a big deal you
can just compile one
so let me give an example
we have a computer which is connected to
the internet over an internet
open standard interface eth0
and obviously we want multi-homing and
multi-pass transport so we make a second
connection
over eth1 to a second isp
so does it now work multi-pass transport
with mptcp
no there is a bit missing yet
the first problem is of course if we
have such a setup we have
probably a default route over one of the
rsps
and
of course then the linux roding stack
uses the route which is lowest metric
which in this case would be over isp1 so
all traffic would go over isp1
while sp2 would not be utilized
of course not what we want we have to do
something in addition
and the solution under linux is routing
holes
so we in fact we have to define separate
routing tables for each isp
sb1 and the isp2
and then select for an outgoing packet
according to the source address
which routing table to use
so let me explain this in a
small example
if again our computer and we now
configure an rp address
for ethernet 0
10 1 142 as slash 24 network
and of course we define a default route
and this goes of course in the main
routing table
this is as classic network configuration
under is but now as i've said we want an
additional routing table for isb1
so we define
a new table
and since the new table number one is
empty we have also to define routing for
the local scope
and of course then at the default route
and the same we can do for ethernet one
for the second rsp in this case we use
the address 172 16 142 as again as
slash 24 network
and create a second table table number
two with the local scope as well as our
default route
so now we have in addition to the main
routing table
two additional tables table one and
table two for isb1 and the iceberg two
so now missing is just a selector so we
add
two routing holes so if a packet is
coming from the source address 10 142 we
use table number one
and if the packet is coming from 172 16
142 we use table number two
table number one and terminal number two
are numbers
if you like you can also use etc upo 2
arctic tables to give these tables names
for example if you are connected
to optos and telstra you can
name the tables optos and telstra
us look yeah we have configured now we
have the routing holds particularly the
lookup for table number one and table
number two according to the source
address
and as i've said these name tables can
also be named if it is more convenient
usually these tables just have numbers
and internally of course these
tables have just numbers but there is a
mapping of symbolical names to these
numbers
and we have of course
now
our routing table number one
as i've said with the local scope as
well as the default route and the same
for table number two with local scope
and default route in this case for isp
number two
so
and of course finally we have the main
routing table the default this is the
standard routing table used for all
other packets
not matching to one of our two new
rb holes
so now our working can work
with um
rpv4
questions of course nowadays we have
ipv6
usually in addition to ipv4 does it work
with rpg v6 as well
yes of course
you can do the same also for ipv6 that
is
get
an rgb6 address for ethernet 0
create
also the rounding table number one for
ipv6
important to mention is of course ipv4
and ipv6 are different protocols they
have both their separate routing tables
so in this case of course this is ipp6
table number one
and before we have configured everything
for ipv4 table number one
but otherwise it is the same as before
for ipv4 as we can
define our two routing tables for rb6
and also
set ipv6 routing rules to select cable
number one or table number two according
to the source address of an outgoing
packet
so now depending on the configuration we
have made now
mptcp or if you have an implementation
of cmt stp
can make use
of this new outing and fbtcp
particularly with full mesh pass manager
will then of course use all paths
simultaneously this of course can be
tested
by booting
an mptcp enabled kernel particular
useful mesh pass manager which has to be
configured in the settings of the kernel
to be compiled
and
then of course you can
test this with ping and particularly
tools like for example network meter
i'll come to this tool
later again
and particularly quite recommended if
you learn about mptcp and multi-pass
transport it's quite convenient to also
observe the packet flow
and particularly this is helpful
if something is not working as expected
for debugging
so
you can use wireshark or command to
t-shark to do this
so far the idea of multi-pass transport
particularly with mptcp
and of course
i've suffered
research around multi-path transport is
necessary
so how to do research
well um years ago i worked for a
university in germany at the university
of glensburg essen
and i my colleague wanted to do research
on multi-pass transport particularly
with cmt satp
but also with mpt sigp
so we asked our system to give us
hardware we had some old pcs
some old switches a lot of cables and
some network cards we put everything
together
and obviously
did not work
um the setup was quite cheap was in fact
for free we did not need a special
budget because the old hardware was all
literally there
but it was only treatment of paper
so particularly this hardware had
sometimes issues particularly
we observe that the network cards
sometimes switch from 100 megabits per
second to 10 million per second just for
a few milliseconds and then switch back
but the paths or some flows
were
assumed to be
having
very similar characteristics
particularly same bandwidth of course
became extremely dissimilar when a card
switched to 10 megabits per second it's
just one tenth of the bandwidth on one
pass
so um this trigger of course packs the
implementation so we did a lot of
debugging of sctp in the freebsd kernel
but
finally
we were able to find these bugs fix
these bugs
and prove that
simulations we made in advance with the
simulation model were actually quite
well modeling the reality we were able
to create in this small testbed center
and of course we
were already very committed to open
source community so we provided of
course all these bug fixes back to the
freebsd
satb community
to make the implementation in freebsd
better
and as part of this i wrote they
mentioned two networth meter to do
protocol performance comparisons between
tcp smtp
udp and dccp with the different
parameter settings so quite useful tool
which is also nowadays available
in debian linux ubuntu linux and other
distributions
well this small setup had of course a
lot of learning effects and new ideas
around multi-pass transport and research
on this
so the idea was of course then well we
did research on
internet protocols but obviously such a
testbed is an internet setup
so let's go to the internet
so we were able to convince colleagues
at two other universities
university of berlin science in book
strandford and germany as well as hanan
university in china
to also create
testbed setups
connected at each site to two different
isps
so we had an intercontinental setup with
three sites to germany to try the one in
china
and all these sites had two isps with
rb4 and
mostly also ipv6
so quite interesting we had a small
internet setup and we were able to do
more evaluation work on cmt satp as well
as mptcp
the shredder of course to a lot of many
to a lot of new ideas
but of course one of these ideas was
obviously
we had a three side setup
but still quite small compared to the
big internet
can we make this bigger
and of course yes we can
so
also the similar research about roy in
norway had
ideas around creating a
testbed setup for multi-home systems
so they
published a shop opening i applied
a few weeks later i was moving to norway
to create the nona testbed consisting of
two parts nunet core is the landline
part
this is what i in this talk today will
introduce in a bit more detail
which connects such sort of server
setups as in the figure
at the different sites with up to four
isps
and we had 21 locations with 11 in
norway and 10 aboard
and of course nowadays we all not only
have enzyme connectivity particularly we
have
mobile connectivity
so no net cause it's also of a large
part converted edge
which distributes
these embedded
systems all over norway around 200
locations
to do measurements of the performance of
the norwegian
mobile networks
will not go into too much detail about
known adria because of time constraints
that i will
present you some more details about
monet core
as i've mentioned we have in total
had in 2016 21 sites 11 in norway and 10
aboard
today it's a little bit
fewer sides abroad but still 11 in
norway with up to four isps and wherever
we were able to get ipv6
these isp connections also in the ipv4
have
rbv6 support
and particularly with such an
infrastructure oftentimes on sites you
can get a lot of
details about the internet connectivity
we have developed a tool hyper contraser
to continuously measure the routes by
tracer measurements
inside this infrastructure this is a
view over one month
from
july 2016.
particularly interesting about such
intercontinental setups is of course
that these routes in the internet
frequently change particularly between
for example europe and china so
particularly also depending on the
combinations of isp
locally and remotely you can see that
some of the connections
go westwards over united states while
other connections go eastwards or russia
or
dubai and other locations
to
east area so quite interesting
and particularly of course as mentioned
we have quite
interesting sites and particularly when
it comes to
remote systems for our net setups
well our
servers can be really remote every side
in the city of london on the island of
spitzbergen in norway
is at 78.2 degrees north
so around 1 200 kilometers just from the
north pole probably one of the
northernmost places you can place
servers
so
we deployed the site setup on a nice
spring day
was just minus 30 degree
and
sun was shining so you can imagine how
it looks like in really
wintery conditions with a lot of storm
and
cold temperatures they are also not the
only challenge the other challenges are
wild icebergs homing around this area
so obviously if you think about such a
sight if something goes wrong with the
site and somebody has to go to the site
and this is the way of transport in
longer bean it's a snowmobile so there
are no
regular roads so we have to drive a
snowmobile
so areas where there are wild icebergs
holding around so it's a challenge
so
obviously we have to think about
what can we do when something goes wrong
and things can go wrong because we have
experimental systems and there's no
response anything that can go wrong will
go wrong
so how can we avoid that experimental
software brings down
our infrastructure so that somebody has
to go to the remote side
and do some maintenance work
well the idea was then to use
virtualization
so actually our server setup consists of
a very lightweight setup of ubuntu
server lts lts is the long-term support
version of ubuntu with 5vs support
so therefore is quite stable and quite
well maintained
and everything else
goes into kernel-based virtual machines
kvm
that is particularly the routing
software goes into a
dedicated virtual machine exclusively
running on one of the servers
and all
research
software goes into virtual machines on
the other machines
but it is if something goes wrong
it's possible to log into the host
machine and do some repair work
it is how does such a physical machine
look like
as of that it's you want to serve lds
it's one customization which is
interesting to mention
we decided to use the riser fs version 3
instead of x4 because of its robustness
riser fs actually has a bit bad image
because it's named after the auto
of this file system who killed his x5
but actually his file system is quite
good so it never killed our data and
particularly it never
needed manually fighter step tracks
at boot time which are quite important
this is a quite important property
because if the remote machine wants a
manual
file system check on the console it's
quite annoying we have to the boss of
people at the remote side
which is fast but not with rhiza fs so
therefore we started with arizer fs
nowadays
after unfortunately the rise fs has been
removed from the installer of yuwundo
for new installations we use ptrfs which
became
also much more stable in the last few
years
and as i've said we are using kvm kernel
based virtual machines originally we
used virtualbox
but this was then requiring
the virtualbox is open source itself the
vnc implementation is not so required a
blob form oracle to be installed
and
we were not really satisfied with the
stability of this
so we decided then to move as soon as
possible to kvm which is quite stable
this is the physical machine setup now
coming to the router setup so called
tunnel box come to this name later
well um the tunnel box
connects
the
machine to all the isps of a site
particularly having one ipv4 and if
available one ipv6 address
per site and isp
of course these others are public
addresses but for research experiments
of course for research are nice to have
his own ip addresses
of course if you ask somebody for a
slash 24 public
ipv4 address space
that's of course not possible
those ipv4 address space is exhausted
since several years
so ipv4 addresses are quite scarce
resource
so what we had to do
was then to think about
establishing tunnels between the sites
over gre for ipv4 or ipv6 in rpv6
so that we can apply our own systematic
addressing scheme
with the private rpv4 addresses
and of course then we can give each side
a slash 24 with 10 dot provider id side
id dot node id
and for ipv6 of course we have our own
prefix and can give
for each provider inside uh full slash
64.
it is of course
we can
then allow researchers to use
own addresses for all research
experiments which is quite nice feature
and particularly of course with these
private devices direct communication
between all the sites
is possible
and for
external connectivity we had to decide
well we can of course route
external traffic
at the tunnel box of a remote site
directly
but this causes of course issues for
many hosts because for example we have
sites in germany
and they have very strange regulations
according to copyright and
file download and such
so in fact to convince somebody goes
aside in germany there is no chance
if there is a possibility that people
can do arbitrary downloads from the
internet
so therefore we route all external
traffic
over a similar site so in fact then if
somebody downloads something
from a remote site
even mistakenly for example do a video
download for testing then it goes out
over simula's address in norway and of
course in norway we have much more
relaxed regulations when it comes to
copyright and such
so therefore this was quite important
decision which was necessary
unfortunately
due to such regulations in some
countries particularly germany
well
now we have the router for doing the
routing but of course
although we have quite systematic rp
addressing scheme
it's sometimes difficult to remember all
these rp addresses we have also made our
own dns setup using behind dns server
so we can
name our devices
with the provider and site names so to
have the possibility to do lookup of the
ip addresses and particularly of course
reverse lookups of ip addresses to
symbolic names
and of course since
countries like norway or germany have
extended
a set of characters we have of course
all the support for puni gold works even
for the chinese side with chinese names
so this is one convenience and other
convenience is of course inside the dns
infrastructure we have we can also put
the ssh
keyframe prints as well as
geolocation information as sshfp and loc
resource records
and
since we route
external traffic over simula
makes of course the sense for
the package updates
to be cached by a squid proxy so also on
the tunnel box therefore there is a
screen proxy running to cache
particularly
linux installation packages so that if
multiple
virtual machines at the remote side
have to download the same package that
not all these packages have to be
downloaded via
simulator
this is the router and now let's
introduce the research node
so usually research nodes by default are
managed
as part of
a software called planet central this is
the sense of planet lapios
just with a slight modification
to have
of course an mptcp capable kernel
and
as well as a special configuration for
multi-home routing i will show an
example later this is currently based
still on fedora core 25 which is a bit
out of date now
this soon requires an update so we are
working currently on this
to do some uh
enhancements
and updates
of the software
but what this planet absentee software
does is in fact to provide
containers alexi containers
for research experiments
which have in fact adjust this fedora
core
linux installation but this planet lab
software is actually quite smart
not for each research experiment there
is a requirement to install or complete
fedora core system but in fact it just
works like having a base installation
on our ptifis file system which is then
cloned
so that is just doing copy on white so
it's
quite efficient when it comes to storage
space
and also since it's using containers
also quite efficient for memory so it's
uh
just around 2.5 gigabytes of memory it's
possible to host host multiple research
containers
uh of course besides planet app central
software it's also possible to
instantiate custom virtual machines for
example for
machines running freebastie or more
up-to-date
ubuntu system or systems as owned
kernels if this is necessary for your
search experiment but usually for many
experiments already possible
to run this planet at sample-based
software
what is our physical nodes look like
that
they run
these
simple based
nodes
as kbm virtual machines and inside these
kvm virtual machines are running lxc
containers
instantiating so-called slivers is the
name of a container
in the terminology of the plan lab
centered software so we can run multiple
of these plc nodes as separate virtual
machines
on the same physical host in addition to
possibly
further kvm based virtual machines for
example with special ubuntu
installations or frequency installations
so that is
when
the usual user logs into
one of these slippers
then
what the user sees of course is
that the machine has multiple ip
addresses one for each isp
but this is
a
slipper that is a galaxy container at
hainan university which has two isps
one is centered and one
usb is shiner unicorn
so the machine therefore has twice two
ipv4 addresses and two ipv6 addresses
so that is
in total this machine has four different
ip addresses
so outgoing this four different types
ip addresses can be used then of course
particularly with mpt sapien as i've
said the kernel of the host machine
is running mptcp and since this is a
container of course the user of this
container has the possibility to run
mptcp
as well and particularly with the two
net parameter
it's possible for the researcher to turn
on and off mptcp
or even have concurrent tcp and fbtcp
connections to do some comparisons
for the performance
of mptcp and different settings with
mptcp
and also particularly what is
installed in these containers as well as
a tool system info which displays this
information on login particularly
information about ssh key fingerprints
and
the rp addresses
and of course
these containers have
sudo permissions that is the user of
these containers as possibility to trust
with sudo
install custom software as well for
running research experiments
so
this
concludes
my presentation on new netcore
it is
showing you
the ideas of multi-homing and multi-pass
transfer and of course
introduced to the
new net testbed infrastructure
and particularly to mention of course no
net core is an open testbed
like just like landlab
so if you are interested in doing
research in onenet of course it's
possible to join monet core
and also use
this testbed infrastructure
so quite interesting thing to do
particularly if you are doing research
on networks particularly with
multi-homing but also
around routing and such
and currently we are thinking about
the future of muna particularly we want
to obtain
more funding new funding particularly to
extend the scope of no net core beyond
this basic idea of multi-pass transport
but particularly in the areas of network
function virtualization and
software-defined networking
but also around the cloud computing and
applications particularly this no net
core infrastructure can also be seen
as a distributed
multi-cloud system with several sites
so it's quite interesting for cloud
computing applications as well and
particularly if you are interested in
more details about on it you can find a
lot of information about monet
on the noirnet website
www.ntp.net
we're also very committed
to open source software development
so a lot of software coming out of no
net
is of course available
in
different web pages and particularly in
a lot of github and bitpacket
repositories so just have a look most of
the software is licensed
under
gpl
and of course we are research testbed so
we have of course also a lot of research
literature
with the results out of
experiments in onenet core available you
can find all these papers also
on the new netweb server on the
publication section
so
now we are almost over time
but we still have
a few moments for questions so thank you
for attention
and
now we have probably around two minutes
for questions left
okay thank you thomas
i'm here now to
give you the questions there is
currently one question available
and that is where is the planet lab
uh central distro available
uh it's on our no net web server there
you can find the links to all the github
repositories
so the original upstream software is
available of course
from planet lab
and our changes are in separate
repositories but they are linked on our
new net web server
and the questions or forecasts if you do
not find it you can just write me an
email then i can send you the links
thank you that's all we have in terms of
questions
would you like to wrap up that point or
do you have something else to add
uh particularly if you later have
questions just write me an email and
also will be available
at the conference in
chat rooms
excellent
okay thank you thomas
you
have a nice day
and enjoy the rest of the conference