so
welcome back to the colonel miniconf i
hope everyone had a good lunch break
um next up we have steven french
who is a principal software engineer at
microsoft azure storage
and the original author and maintainer
of the linux cifs smb3 client and he is
a member of the samba team
um he'll be talking to us today about
the uh implementation of new features
from the smb 3.1.1 protocol please
welcome steve
yes thank you
uh hopefully my volume is okay for you
guys and i'm looking forward to sharing
with you some exciting things it's been
a very interesting year not just because
of all the uh the lockdown but more
importantly because of a lot of progress
in the smb protocol and its
implementations on linux and as a member
of the samba team you know i also um
you know i'm not going to be talking a
lot about samba but i also want to
encourage you guys to keep up on the on
some of the exciting updates there as
well especially given its its history
and its origins on australia
so uh just the normal legal reminder
that although i work for microsoft may
not always be microsoft's opinion here
but i think in general it's been very
interesting working on trying to make
linux the experience of accessing the
smallest devices to the largest devices
fantastic secure fast
i'm the maintainer of the linux kernel
client one of the most active file
systems in linux
and
i'm also a member of the sama team i
wrote something about the original
original net utility with uh jim
mcdonough at susa
and i work for microsoft now
okay so i'm going to talk a little bit
about some general status of linux the
file system generally and then dive into
things relating to smb 311 the new
kernel server okay smbd
uh and then the client improvements and
then what we're expecting with the tools
and with the
what we can look forward to as we um
as we progress
like i said this has been a very
exciting time with lots going on so as
you guys know very recently we released
a turkey linux 516 gobble gobble
um i love lenas of strange names for
things
a year ago we were talking about linux
510 some
thieving octopus
so what's driving a lot of these
discussions as you guys probably know
matthew wilcox and dave howells and
amir and many of the people driving some
of the general activity what are they
looking at
uh
a lot of changes to memory management
yesterday
we just had a big pull request from dave
howells changing the offline caching
infrastructure uh and also some changes
that netfs are coming
um we see changes to containers
especially in the 512 and 513 releases
and then we see a lot of discussion
about quick apparently a large
percentage of the
the internet traffic especially um
google related is over this new
udp-based protocol instead of tcp and
windows now supports for smb windows is
the first
client and server that support this
supports quick and there's a great
presentation if you guys are curious at
the storage developer conference last
year last fall on that
this is very exciting for all kinds of
reasons that we can talk about later but
this is driving a lot of discussion
between the nfs guys the smb guys afs
team others about how we could get this
in kernel
there's a discussion about faster
encryption there are government
requirements for this and obviously a
lot of security paranoia but adding new
security features relating to better
encryption and also just the normal
stuff we have faster network rdma cards
are getting common
nvme it's getting faster
and then we have this whole new async i
o infrastructure in linux i don't know
how many you guys
are kernel geeks following this but iou
ring is the way as matthew wilcox would
say
everything async i o is dead iou ring is
the thing
right aio is not the thing iou ring is
and of course then we have the other
thing i work in azure azure has very
slow vms very fast vms
i
could be mounting to
northern europe i could be mounting to
central us and san antonio i could be
mounting to
data centers
in australia
the latencies are radically different
the speed of the vms can differ a
hundred fold this is a
real game changer for file system
features okay so let's briefly talk
about file system status
off of the year
5936 kernel change sets
similar that seven percent
of the overall kernel changes are file
system related which is pretty
impressive given
um the enormous number of drivers that
uh that occupy the vast majority of it
file systems matter
in linux and the kernel is huge right 21
million lines of code i measured that
yesterday actually
there are over 60 file systems but it's
interesting in linux
a relatively small number
btrfs sifs xfs just a small number of
file systems drive three quarters of the
activity in the file system layer
um
so the stuff i work on is one of the
more active file systems file systems
represent about five percent of source
code about a million lines of code now
to give you some comparison working at
ibm their client was
a hundred
ish thousand lines of code in one file
system
these file systems are very carefully
uh
optimized the code is stripped and
rewritten frequently so that million
lines of code is very dense and very uh
carefully reviewed very carefully
watched for most file systems
now recently the file system was the
fourth most active 356 change sets
actually might have been third i need to
check that
um
so i've got the numbers on the next
slide we can talk about that
um it's about 60 000 lines of code
it's up significantly in the last year
and that doesn't count the user space
utilities and tools and various samba
things
but the tools have grown a lot as well
and of course the samba tools are
enormous compared with all that
and interestingly the kernel server at
its current pace will also be one of the
most active components it was merged
recently
so let's talk about some overall file
system details before we dive into the
smb world
does anybody know what the most active
file system is
probably should btrfs
and xfs is number two the file system
mapping layer itself though
um has been generating over the last few
years more activity than you might think
it's not just dave howells and matthew
wilcox making changes but there have
been a lot of optimizations uh going
into the mapping layer the vfs layer
itself
1300 change sets since last
since 510 a little over a year ago
um
so
sifs was 356 nfs was 299 change sets
but as you can see the the sets client
is is
the most active of these network uh
cluster file systems and for obvious
reasons if you want to get to your max
if you want to get to azure you want to
get to the cloud if you want to get to a
low end embedded device if you want to
get to windows if you want to get to
samba
now what about servers
now linux
has multiple servers now right we have
three and a half million lines of code
in samba
it's enormous
orders of magnitude almost 100 times
bigger than than the nfs server
but now we have a kernel server
yeah so it's an interesting thing i'll
talk about the kernel server in a minute
and i think that's an exciting thing
that many of you may not uh may not be
aware of or may
uh you know may be curious about but i
also want to remind you guys that
writing file systems isn't easy
a lot of people maybe get buried in
really cool stuff going on in
analytics or in big data
but file systems you know they keep
growing and the amount of data that we
have to deal with is growing but also
the number of syscalls people think
about posix right well posix is a
fraction of the syscalls that
are exported for file system
apis these days just
you know over the last year we've added
these three
right so the number of syscalls keeps
growing file systems have over 200
syscalls
currently
so linux is not posix it's a much bigger
thing and we as file systems always have
to do that so what do i care about i
want smb to be this sort of
lingua franca this this commodity layer
that you can use to access your cluster
file system
running on linux
access the cloud and azure if you want
access your macs access a tiny embedded
device running samba or ksmbd
running some third-party camera you want
to download stuff from
so this is we want it to be the most
secure
general purpose way to get it file data
whether it's local
or whether it's in the cloud
and as linux evolves we need to keep
adding features to kernel client to
samba to ksmbd
and also to the protocol
to take advantage of these
okay so what about servers so i told you
i was going to talk a little bit about
the kernel server but first i want to
mention samba samba has these incredible
australian roots and i'm sure you've
seen andrew trigel and heard
very interesting stories about it it's
huge it's full function uh sombexp is
coming up
in about five months so you know keep
your eye out for that um
but there's now also a kernel server
which can help on some workloads like
rdma smb direct and can be easier to
optimize in some cases
and it's also been incredibly valuable
for me on the client because it's
allowed me to do more testing and more
prototyping of things like the posix
extension changes and also more
debugging quicker
it's been incredibly helpful now there
was a talk last year by nam j at samba
xp
but more recently he's provided me some
slides that i can explain
some of the recent progress for you guys
so i'd like to switch gears briefly and
talk about the kernel server progress so
you guys get a feeling for this kind of
exciting thing that
may be used on everything from very
small embedded devices because it's tiny
footprint compared with alternatives
and also it supports rdma in very fast
um very fast network transports
and also supports multi-channel so there
are certain workloads where it's going
to be incredibly fast and there are
other workloads where its small
footprint will help
but what i like about it is it's easy to
update
and
in some cases it's easier to understand
some is very large
but back to this so it was merged into
the 515 kernel not that long ago right
last three or four months so merged
august 31st
so it sat for five months in linux next
as it got various reviews and most of
these reviews function
focused on functional testing
and some very high profile developers
reviewed it that was thank you very much
um we changed i one thing i i pushed
strongly on was to make sure we try not
to use the word sifs
right because the common internet file
system from the late 90s
has been associated with some some major
security breaches and the current
dialect
smb311 doesn't have the
common internet file system cifs it's
server message block smb
so that name smb is easier to um
to associate because it matches the
dialect that's actually
included the the kernel server doesn't
support smb1 uh it's not supposed to
we're not trying to encourage insecure
dialects here so one of the things i
want to remind people though is that we
want to make sure also we don't get
confused with samba so ksmbd
is the kernel server's background demons
smbd is samba's background demon we want
to be careful not to collide between the
two and the common code has been moved
so we're trying to move over time more
and more common code between client and
server to lower the main main uh
maintenance cost
for this in the kernel and to make it
easier to review so common code has been
moving into this smb fs common directory
and it's providing some optimizations
for us
later we're going to rename the sorts
directory to smbfs client and you can
already by the way build
the sith driver
and mount
it as
smb3 instead of mounting it as sifs so
we're trying to get away from using that
old deprecated term cifs because it
implies less security but you know the
modern clients all default to
negotiating smb 311 which is highly
secure and very fast
so some some cool features um the kernel
server just like the kernel client has
added support for what some people
nicknamed military grade encryption the
strongest encryption model aes256 is now
an option
and notice the improved performance
because the kernel server can use the uh
the aes and i support in kernel notice
the megabytes per second improvement by
using the kernel drivers here provides
significant benefit to us but here you
also have the option of using the
strongest encryption available aes256
not just on the client but on the server
that will alter we can negotiate this
now
kerberos support so you want stronger
authentication um ksmb transmits the
kerberos up calls to the ksmd mount d
and uses libcare b5 for this
duplicate extents duplicate extense is a
fantastic feature uh ruffling and if
you're on a file system like btrfs that
supports reflect the linux client will
use duplicate extents for some f
allocate related operations like insert
range and the server can support it when
it's running on a file system like btrfs
that supports reflect
and fortunately the nice thing about
ksmbd is because it's in kernel it
doesn't have to deal with the sort of
vfs mapping mess
that that samba has to where it has to
have different plug-in modules for
various file system types
so it supports this transparently and
it's an example of a feature that was
kind of easier to do in the kernel
so multi-channel multi-channel is
extremely important for performance it
allows it's a feature that windows has
had for years and it's a feature that is
of incredible benefit if you put a
network adapter in a machine
the windows client and server dynam or
the windows client dynamically notices
the new adapter
and connects to it automatically so you
if you put a network adapter in
whether it's rdma or regular your
performance could double just by putting
one adapter in without doing anything
right there's no configuration needed
there's no
no magic needed you just put your
network adapter in and the protocol
supports advertising that
via the witness protocol
and some clients also poll for this but
the performance benefits of
multi-channel are huge because you're
able to distribute io
across many network cards
on the client in cases where you have
multiple network adapters or network
interfaces and it allows you to spread
io out
when your network bottlenecked
so they added the do replay retry
features as well on the kernel servers
this is another example where the kernel
server um is just some features are
easier to do in the kernel um so here is
how how it works
the client is sent the network
interfaces here you see two network
interfaces being advertised the query
network interface call sends that and
then you do a session binding request to
those sessions
then you now look at the blue versus the
yellow if you look at the blue versus
the yellow you can see that it's
interleaving requests over multiple
multiple interfaces so we're able to
spread i o out to avoid network
bottlenecks so what kind of features are
working now that are exciting some of
the things like smbdirect
there are patches and performance for
various performance improvements for the
windows client
smb direct works from both linux thank
you to longley
a couple years ago for his work he did
adding it to the support for
smb over rdma
um a few years ago to linux
but sme direct from linux and windows
works support for direct releases
works direct releases allow you to cache
metadata safely
this isn't like a timer this is just a
this is a way of safely caching metadata
and change notify which allows you to
watch
for changes in tree for example when i
if i went to a windows box or a linux
box and opened
especially on windows and mac when you
open a
explorer window
and you add a file from a different
client you'll automatically see it show
up in your window on the other clients
because change notify will have notified
you about the new file
linux unfortunately the i notify call is
local only file systems i notify isn't
supported across
network file systems but that's
something that amir and others have been
discussing fixing
adding support for rss rsf receive side
scaling allows network adapters to
offload
um
i o
for better performance um sort of
and you can see the support
the kernel server can help again here
you can see the very large performance
improvements compared to sama so there
are certain workloads where they're very
helpful um
and um
you know this smb direct improvements is
fantastic
um
so just another quick comment i wanted
to make about ksmd initially there were
some major security problems found
immediately after it was merged so in
the 515 and 516 kernel there are
significant number of security fixes
that got merged into ksmb to address
those and it was marked experimental and
disabled by default as a result
now over the next
over the the 515 release window things
improved dramatically but we are
tracking these security issues carefully
so if you have additional reviews or
comments about security please uh you
know contact the mailing list
and also
we have a wiki page to track some of
this i'm helping nam jay maintain this
co-maintaining it but nom j
and a team of four or five guys is doing
most of the work here it's a very
exciting time on the server so let's
switch gears to the client on the client
we have lots of cool progress but i
wanted to start out by mentioning that
in the smb world unlike sort of the
posix world you're used to there are
three security models you have a model
that's cheap where everyone just gets a
default owner or default group you have
an example where
you're pulling
the
mode bits
in a way that matches
what the
server is going to enforce
heckles and you also have a case where
you want the exact mode bits on the
client
but on the server they don't need to
enforce those that's mode from sid those
three cases you have to deal with all of
those
um you have to deal with all those use
cases uh in our smb world a default case
it's cheap
an ackle world where apples and mode
bits have to coexist and a third case
where the mode bits are perfect but the
client's enforcing it not the server
okay
so
a couple people
forwarded questions here and i want to
address those before we go too much
farther one of them was about rss
offload so there are network adapters
that have the capability of
copying directly to the network adapter
to avoid some of the cpu penalty so for
example um
their um
i mean a good example is we can pat in
the kernel we can pass an array of
pointers to a socket right call that
gets passed all the way down an rss
adapter what happens today in windows
for example when the windows client
detects that the server
adapter is rss it automatically sets up
two sockets so the rss capability is a
hint to the client that the server's
adapter is fast enough that it's worth
the trouble to set up two sockets not
just one
to allow for more parallelism on the
wire
now the constraints around i notify so
somebody asked about i notify so this is
a
something that amir goldstein and others
have uh have been discussing
one of the problems is that the i notify
api never calls into the network file
system to tell it
that
we're waiting on um
a notification event the network file
system has no problem notifying but it
isn't notified it isn't told there's no
sys the syscall doesn't make it down to
the
uh the file system itself it stops at
the vfs mapping layer so there's no way
for the file system to know
to notify the kernel on a callback that
a file has changed or directory has
changed
that's relatively easy to fix but it
brought up a larger discussion about you
know which features should be included
because as you might guess the smb
protocol includes more notify features
so the protocol has more features than
linux
currently advertises so is it worth
adding additional flags to linux notify
mechanism for example
you know
every client except for linux sends
notify across the network for smb
so you can see windows
being updated automatically when a file
is dropped in from another client the
directory listing changes the little
icons change
okay anyway the
the
the point i was getting out here though
about notify is these are these are
patches that have been discussed there's
no technical problem in implementing
them um but right now notify is a local
only event on linux um the server can
advertise it to remote clients but uh
there's no way for nfs or smb or
actually nfs doesn't really have
notified that way but there's no way for
smb to a
client to um
to request a notification from the
server we do have an ioctal though for
that so if you want to do this an
application can call specifically into
the cya octal on the on the linux client
to get that same feature
okay and then the other one was um
so there was a question generally about
uh performance and feature sets um
file servers
linux versus windows i think it's an
interesting thing is that there are many
cases where we expect that uh that linux
could actually be faster in some cases
than windows but let's come back to that
question about performance
um
but uh in terms of rdma i think that
what the platforms that you can take
advantage of for rdma today are
linux client
when you mount from linux though you
have to specifically request rdma on the
mount line so ask for rdma
when windows it automatically detects it
so
there are slight differences but both
windows and linux can take advantage of
it but the windows part has recently
added and there's some performance
optimizations being worked on right now
so good question to come back to so
lots of security improvements on the
client let's get back to the client here
because we're running a little bit low
on time
improvements to authentication thank you
to cheyenne there's some um the gcm
encryption now added on the client the
strongest form of encryption
uh very exciting azure for example
windows ks mbd all supported this is
fantastic you know to the cloud to have
the strongest encryption
now here's a trace if you want to look
at a wireshark example of using that
multi-channel we talked about on the
client is is much improved right at this
moment literally like
before this talk started i was working
on some
reviewing some multi-channel page
improvements for reconnect from cheyenne
now let's look at some performance
improvements we have some really cool
import
performance improvements notice that
going from the old defaults of smb3 just
smb311 got me 41 faster adding multiple
channels
uh and this new rs size we got three
times faster for something as simple as
copying a file
copying a 10 gig file got three times
faster really cool performance
improvements
so
um the other thing i wanted to mention
was there's a fantastic feature thanks
to row heath added in 513 for deferred
close this allows you to do something
like open create a temp file write to it
close open read the temp file and you
you don't end up sending any of those
reads because the file is briefly
deferred the close
uh and then when sees the re the second
open it doesn't have to to re-read the
same network as you can see this is you
know three times faster for something as
simple as you know create a file write
it read it close it read it
so this is really cool also metadata
caching is much more granular now it's
possible to separate the metadata
caching for directories and files we
have much better debugging we have
dynamic trace points ebpf is the thing
you know i think you saw microsoft and
many others are joining together to try
to make ebpf even better but i'm trying
to add events to make it easier to debug
and optimize 87 of them now we added the
shutdown support in 513. now for you
guys to look at later i've included a
pretty detailed list of all of the
features
by release
so if you guys want to review that later
you can see you know each kernel now red
hat and sousa and others have been very
aggressive about backboarding this some
of the ubuntu kernels have also
backported
but i think what you're going to see
here is lots of cool fixes and if you're
you know want to go through release by
release you can see what's been added
you know by downloading the slides after
the presentation
so
i was very excited in 514 about some f
allocate
improvements to be able to
handle
sparse files and to be able to handle
some new tests uh some new features of
uh
uh f allocate that uh that were not
supported before thanks to ronnie at uh
some of his work at red hat we also in
the 515 kernel had a kind of
controversial thing we removed the
weakest forms of authentication ntlmv1
from
you know 1996 and land man from the
early 90s
which has provoked some controversy
removing these very old
uh very very old security algorithms
obviously ntlmv2 has been around since
the late 90s
and it's much stronger but
you know it breaks some very old devices
and smb has worked for years with some
devices that are 30 years old 25 years
old
the 516
had some really cool improvements for
compounding some reconnect improvements
and now what are we dealing with we're
dealing with a big patch set for fs
cache and for offline caching and nfs
and some multi-channel patches but what
i'm most excited about is looking
forward things like
quick and
because quick solves many problems but
it also gives you another choice for
encryption to work around the port 445
problem for example i wasn't able to
save my presentation as easily into the
cloud because port 445 is blocked by my
provider but isn't blocked by the
provider
in the next city over from me
so quick avoids that problem giving you
encrypted traffic
but it also has some neat parallelism
features um also i want to add
compression support for smb1 311 we've
made a start at that
and also um there is uh some work to
improve metadata performance by more
generally allowing using direct releases
the windows client does a much better
job than linux the mac client does metro
the linux client is much worse than
windows the mac client
does much better than linux as well in
some of these areas and these are areas
that we can really improve
also we need to improve id mapping
choices there are many cases where we
can't use rfc 2307 um
also invented by an australian by the
way
where we need other id mapping choices
to be able to map sanely a global
identity to a local uid
okay we're running fairly low on on time
and
i don't want to overrun my presentation
too much here but i want to remind you
there's some great tooling improvements
in the user space tools you can now view
alternative data streams for example
and we have some general configuration
advice on the later slides here
but i also want to say our biggest focus
right now is preventing regressions and
improving testing we added more than 40
tests to our test group we have this
automated thing called the buildbot
um
thanks to work by ronnie and irelian and
others and every single pull request we
send
we make sure that we run the buildbot on
and that has improved the quality
enormously so if you have additional
tests you want to include these leverage
xfs tests in many cases but they also
run things like the get functional test
please send them to us please work with
us on improving test coverage we want to
make these releases the most stable ever
and every single pull request we're
running an exhaustive huge set of tests
anyway i thank you very much for your
time
and uh you know if we have time for any
last you know one minute of questions
that's great
um but otherwise feel free to contact me
at some french at gmail or on the linux
sift's mailing list
the linux mailing list you can see on
one of these slides here at the end
okay
thank you very much steve um
unfortunately we have reached the end of
the time slot so if you've got further
questions um you can contact steve uh
outside of this session and i'm sure
he'll be happy to answer questions
yeah very much so thank you very much
you