[Music]
hello everyone and welcome to the kiara
uh theater at linux conference australia
2022. uh first up we've got nicola de
fran who is a principal multimedia
engineer at collabora
and is actively involved with both the
g-streamer and linux media communities
to help create a solid support for codex
on linux
and he's here to present bring webm
alpha support to gstreamer introducing
video transition retro alpha plane
support a workaround and we'll dive into
the architecture he's implemented to
support this kind of video now we are
going to have time for questions at the
end
uh so put your questions into the
questions tab and venueless as we go
and try to put them in with plenty of
time because we are operating on a delay
but yeah take it away nicola
thank you caitlyn
so hello thanks uh thank you very much
lca
team for organizing this uh virtual
conference i understand how difficult it
can be it's amazing organization we have
a lot of support so a big thanks
and it's also part of our effort to not
be the problem and solve the problem
so
as i gotta introduce i'm nicolas dufrein
i'm just drummer contributor
actually i've been contributing for over
10 years
um
i'm quite lucky
because i'm allowed to actually
contribute to this project in my
day-to-day job at collabora i've been at
calabra for 11 years this
now and
i've been able to learn and contribute
to
so many things especially nice for
someone like me that likes to learn new
things and being a consultant is just
about that jumping into a new project
solving new problems
and getting the work
the job done and speaking of a job
uh the
implementation of webm alpha inside
gstreamer was actually
mandated by one of our customers in the
streaming industry
so they wanted to use g streamer on some
sort of a
pro streamer handheld that would
basically
allow on the road
let's see more professional streaming
capabilities but one of the big missing
features in this rumor for them was the
ability to play back 10 years
and then
yeah
but to be honest i had just no idea what
his tenure was so first thing i did is
look at wikipedia which says um well a
stinger is an opaque
organ or body part found in various
animals typically
arthropods that usually deliver some
kind of venom it wasn't very useful
but after some more research
because i really wanted to know where
that word came from
i actually found that the words tenures
is used in
american radio station or was used in
american radio station
as a little sound that would be fairly
unique to the station and that would
allow the audience to actually recognize
the the channel that they're that they
are listening to
so and that definition actually well
it's not really a definition but that
definition actually fits very well to
what streamers today actually uses as an
audio videos tenure
which is a way to personalize their
streaming and their streaming transition
or and it could be an entrance a
transition or an ending in order to have
a very specific signature
so what they are they are personal
animation
for transition they usually start with a
transparent video that transit toward an
opaque
middle point where the transition take
place and then it would go back to
transparent and then it would go away in
the montage
and
these are while typically altered with
tools like after effects but it looks
like it's feasible these days to to make
them actually using open source tool
there's a demonstration i've seen
recently about
producing a stinger with [ __ ] enscape
and also with shotgun video editor which
natively support
with with ffmpeg the ability to to
encode and export uh those video
uh test tenures actually
could have been in other format web than
webm you could have actually used uh one
of the pro codec like a pro res 444 that
has native support for
for that or some higher profile in hevc
but the truth is that it's rarely
supported by your hardware it's not
accelerated on everybody's computer
so you need a more expensive setup to
use them
and that's where the wet em the webm
team which is a google team actually
came up with a great idea which is webm
alpha
and they wanted and they actually
managed to do that using
their
royalty free codec vp8 and vp9
but that there's a but there's a
there's a problem because neither vp8 or
vp9 have support for an alpha channel
they only support three channels the
colors
the crew and the two chromas the two
color plane
the idea they had is that what if we use
two streams
and we map the alpha from the second
stream
as
we map the luma as from the second
screen the the brightness
as the alpha and then it just ignore
the uv plane
and that's exactly what they did so
typically that's the simplest form and
that's what the software live vpx would
produce
uh
vp8 or vp9 decode to
three planes the y the u and the v and
then they would decode twice and then
they would map the y as a and the
the x's there are the u and the v and
they are actually set to something
uh
uniform so always the same value with
the idea that uh during compression
because it's always the same value it's
going to compress very well so it's not
going to introduce as much overhead
um this is also a bit unlike if you were
to do that with h.264 where you can
actually
have
monochrome encoding you could have used
monochrome encoding instead of
of this workaround
now that
that by itself
is not sufficient so
uh they actually needed to
store and attach the the two frames
together somewhere and with bit stream
like vp8 and vp9 which are our turn i
mean they're done you can't actually
extend them there's no way to add new
new information to it
so their idea was to use
a container format instead so containers
are the name we give to
things like webm mp4
uh you'll find transport stream in the
stream in the the broadcast industry
and
they would extend webm which is a subset
of matroska which is well known in the
uh in the multimedia community
in order to support this
now what you need to know about webm to
keep things simple it's implemented on
top of
uh
next enable on ebml which is extendable
binary markup language
which
what is nice is that you can actually
express it in xml in order to represent
the content
so to make it short and simplify it a
bit
a webm file is split into
two groups the segment
is what describes the data that is
stored in the container and then you get
a cluster which has block group and
blocks and block additions
to to actually um carry the data that
you're you're streaming
and
the way they
decide they they manage to add
this alpha channel this extra second
string is through what we call the block
addition which is an abstract
piece of binary that you can add
attached to a frame per frame and block
addition have no meaning unless you
actually add a tag in the track entry so
that's what they did so they added a tag
to the video track
which says that we are in alpha mode one
why one why not true or false because
they probably intended to be able to add
other modes in the future
i don't see what it would be but they
want to be safe
so whenever alpha mode is set it means
that there's going to be a block
addition for each frame and that this
block addition is an alpha channel
so enough of the
introduction here
uh
how do we map this into gstreamer and
what i really want to show you is the
journey of a gstreamer developer after
learning a technology and trying to map
it back
into gstreamer technology
just just read me well this is entirely
supported in ffmpeg it has been there
for a while you can both encode and
decode in ffmpeg so we're a bit in the
catch-up mode in the digital world in
this regard because we didn't have any
use case so far
actually
that being said
there was prior to we'll see later that
there was already some some work done
we also had to keep in consideration we
had to know actually what we needed to
consider before we started so
um one of the thing i noticed is that
non-alpha decoder the existing vp8 and
vp9 decoder would work just fine
ignoring alpha with the black background
and that should keep working for sure
another thing we didn't want to
introduce
any wrapper for the case where you
actually decode uh
vp9 data because this can be
pretty big 4k 60fps with hdr and
everything these days
with netflix and and not netflix but
with uh
with with google youtube actually using
that a lot
uh it should also work with auto plugger
that what it means basically it should
work in your favorite media player
without having to code something
and of course it would it should not
break the ability to to do to actually
disable stuff to disable hardware
decoding which is a relatively new
feature
and that would have been very easy to to
ignore
now about the prior arts so before i
started um
just like what i i do usually and i i
would strongly recommend if you're part
of a community
i've talked about the project i wanted
to do
uh over the irc channel of g streamer
and
and basically give some layout of the
the design i would like to to to use to
make sure that i'm not totally off track
that i'm not doing something that would
never work
and uh that was really good because uh
tim philip mirrar and matthias poncher
actually had an old branch which was
implementing that for software decoding
there was a bunch of little issues that
actually led to not them making it
upstream
basically it was only software decoder
and anytime you would have actually a
hardware decoder in your system
it would actually prefer the hardware
decoder over the the software decoder
but the other decoder might not have
alpha support so that the auto plugging
decision was not correct and was quite
surprising
they were also using a feature called q
data which is a
basically string identifier for extra
data on on
on buffers in gcstreamer
to store the block audition i thought it
was not very elegant but it was not a
bug it was it was fine but it to me that
was not the most elegant way so i've
decided to use a different approach
there
um
but in the end it was a big time saver
because they had solved all the the
things that i knew the last which was
ebml uh parsing and
and and handling so they had the metro
rosca de max part in place all i had to
do was to map
so speaking of mapping so
um in order to make the
codec alpha stream identifiable i
decided to add a new field to the to the
caps that would be exposed by the
demuxers so i would map the alpha mode
from the ebml to a gstreamer caps codec
alpha it's a boolean because i don't
believe we'll have a new mode in the
future
and
for the older codecs they would remain
with no fields this is something very
that you you would have to know if you
were to do development in gstreamer uh
especially regarding the caps
that if a field is not in the caps it
basically
mean that it supports any value for
other field
and
so and it it's it's nice because in our
case it's a way to not break
compatibility it means that the old uh
the oldy color will be compatible with
the streams that have codec equal alpha
set into as a field
if i had created a new format a new
media type actually like video x vp9
dash alpha i would have broken this
implementation
the second part of the mapping was where
do we store the block addition as i said
it was attached to the gst buffer
already so i just keep that but the way
i decided to attach it was to
provide in the video library
what we call gst meta which is a little
object that we can attach to every
buffer in order to carry some extra data
and that meta which is called gsd video
codec alpha meta actually will holds a
reference to the gst buffer that
represents the alpha channel
now
we got the mapping and how do we process
this
uh in order to decode
so
my main idea was to avoid actually
modifying the vp9 decoders to support
alpha
the reason is that well the prior art
from tim and matthew was was done this
way and it would
highly increase the complexity of the
of the decoders and i really didn't want
that
and just remember is a very powerful
graph processing tool and i i thought
that hey we should create a graph for it
so that's what i end up that was my
draft actually which i kept till the end
except i renamed few things
well that's the final name sorry
so i
basically to to support this new graph i
basically need two new tools that would
allow me to basically split the streams
then i can decode and then i need to
combine those frames to form
the alpha
so the first element i had to write
is codec alpha demax
it's
it was a simple element it would have
one input stream it would read for from
the input
the gst video codec alpha meta it would
remove the codec alpha field from the
caps and then it would push everything
toward two pads so there's the color the
the the base pad which is the colors
and then it would push the the gc buffer
from the codec alpha meta on the alpha
pad and that's it you now you have two
streams that you can put into cues run
into decoders and get two parallel
decoder running
now out of the decoders
and now we have 2d colors running on
their own threads so there's a bit more
of a challenge when it comes times to
combine the alpha
so what
so i ended up implementing alpha combine
as a fairly simple element in the sense
that it doesn't do
time zone
synchronization or timeout like an
aggregator would do because i thought it
was a bit of an overhead to introduce
yet another thread
in this case
and because we know that there's always
going to be one frame uh
one frame for for one alpha frame for
all color frames and in fact if there's
one missing we can actually signal it
with the gap
we have gap mechanism and gstreamer for
that
so we'll wait until the color and the
alpha buffer are ready then we would
append the memory
together you have to know that gst
buffers are our containers for gst
murmur json memory are blocks of data
and we basically represent a virtual
single memory
group
so it doesn't matter if your data is one
gst buffer or multiple gst buffer
except there's some restriction you
cannot have multiple gst memory for
for the same plane in the streamer
should be the same
and then i can just happen them together
and ignore the uv and then i get
a new format uh fortunately for the
software decoder uh the decoder produce
i420 which is exactly which i described
at the beginning three separate planes
and then i would transform this in an
already supported format called a420
which is the same format but at the end
there's an alpha plane that is added
exactly what we needed
the beauty of alpha combine
is that despite its name it's actually
not doing any
binary operations so it's totally zero
copy it's just a per symbol like
manipulation of the the memory
inside the gstreamer system
now that's great but part of my work was
to support hardware decoders and to be
honest at today's hardware decoder they
will all support nv12 or some variation
of it
so
in nv12
the the format is basically
okay but the format of nv12 is not that
different it keeps the luma the the
brightness actually
uh separate and it will interleave the u
and the v in the same buffer
so that basically means we could do
exactly the same thing if such a format
existed
but the truth is that i didn't find any
such a format
but that didn't really block me so
i mean first time in 10 years i decided
to create my own format so i created a
v12
arguably i think i found some people
that might be using that on some
proprietary tools so i might not be the
creator but anyway it's the first made
up format i added to gstreamer
and
that serves very well the purpose so
it's exactly that it's nv12 and then you
can add an extra
alpha format at the end in order to
carry the alpha and you ignore the uv uh
the uv plane and they have the same
features that when you set the uv you
want to set them to
a uniform value in order to compress
very well
so in the end
we added bunch of elements into
gstreamer so the there's a new plugin
called codec alpha which have the
helpers
um they're the helpers i mentioned and
then to make all this works we actually
need decoders so actually we need bins
so we need a container that will
actually abstract the graph i've showed
earlier
and i ended up exposing some decoders so
there's vp8 alpha decode bin and vp9
alpha decode bin and these are strictly
used to decode webm alpha with software
decoders and then for the v4 l2 sl vpl
alpha in the code bin it's very long
before l2 sl just just a side note sl is
for stateless it's a new
kernel api
to support decoding with
hardware that is called to be stateless
which doesn't actually do the parsing
and the state management for your codex
this is something i've been working on
outside of gstreamer a lot on these apis
and reviewing this api upstream
so this is also another subject i work
on and we wanted to support these of
course we can add as many wrappers as we
need as long as we have the ability to
do the zero copy alpha combine
or
another mean so some some
some tool that would allow this in the
context
um i prepared couple of demos actually
of this uh so this is basically i
thought it was much more interesting to
to see what these things these i just
want to mention before
we actually started demo that i'm
streaming i'm doing
some encoding over the internet and my
little laptop may suffer a little bit
it's very warm
so some of the demo that worked great
before the presentation actually can
stutter a little bit i noticed this
during my
my tech check
so
let's start with the first evo
how that works
first thing let's check what we have new
so i said we have a new
codec alpha
plugin
so you can see that's gc inspect that
we have the new alpha combine element
the new codec alpha demux element and
then we have the 2d code bin
and then if
yeah i won't see the v4l1 on my laptop
because i don't have such hardware
in there and if i look into let's say
vp9
of
the code bin
you can see that
let's start from here
you can see that we have a codec decoder
that is for video
it's a wrapper bin for vp9
you can see is rank so
one of the
one of the issue we had with was that
due to how the auto plugger compared the
the capabilities in order to determine
what is a fit and what is not a fit and
the fact that we use the same media type
the easiest way to make the alpha
decoder work was basically to limit
their use to alpha decoding so they
don't support a standard vp9 and make
their rank so rank is just to order
basically the list of elements when it
comes time still to plug to something
higher so it's primary plus 10 in this
case
and then uh if it was a hardware wrapper
bin it would be a little higher again so
that's just made so it's ir any vp9
implementation
and it
it would be looked at first or discarded
by the uh the auto plugger
and then there's not much um property
it's all the default stuff but you can
see all the child elements the the
combiner the decoders the queues and
the uh
the codic alpha demox in there
now
how do you play this let's take a
stinger because we we talk about stainer
so here we're going to use a little
terminal player command line player
i'm going to use spleeventry because
it's generally work better for this type
of application
and
this is a stainer i've downloaded on the
internet that was available for free i
just forgot where it is from
but thanks for this so let's
play this demo
let's play but it did play in the
background
there we go
so when i'm screen sharing i discovered
that
uh window don't pops
over they actually stay behind so i need
to remember so that that's what it
looked like you can you can see that
oh i'm missing a bit of it you can see
that initially it's uh it's transparent
oops and then it goes away pretty hard
to
catch right
let's seek to zero put the window in the
front
there we go
hope you saw it so that's using
basically
um
i'm basically using the player and i'm
using a gl image sync and i'm telling
the
the display sync to avoid ignoring the
alpha channel so you can see through the
window
now stinger is not the only thing we
could play there's also
this this uh this is about transparent
videos in general so you can you can
play all kind of content so i created
another late
demo which i found which i'll show you
on the web also later
and in this case we're just playing a
different movie so let's run demo
overlay
there we go
this one stutter a little bit
but you got a phone actually that is uh
going around on top of my desktop
and it's all semi-transparent
and
one of the surprises i
i kind of thought it would help but i
didn't i didn't know it was going to
just work so this effort also helped the
webkit community in order to
fill in some some of the gap and some of
the missing feature because browser are
mandated to support these streams but uh
grimmer bay's webkit implementation
didn't support that yet
uh so that's helping uh webkit gtk and
web webkit wp
and i'll show you what it looked like
before
actually
before my changes
here we go
so it's going to be a bit stuttery but
basically you had the phone
going around it's a website
and you have the phone going around and
a black background so it's clearly not
the appropriate render for this web page
so for them it was actually
some test failure
let me show you how i was running this
so
demo epiphany before
so basically i've used one of the new
features which allow you to adjust the
rank of element using an environment and
i've disabled vp9 alpha decode bin so it
was no longer visible to epiphany which
is another well to webkit which is using
anatom plugger to choose the element and
then i'm visiting this web page over
here
now let's do the same
but we're not going to disable the alpha
decoder
the other one
there we go
so
i was really surprised i was really
impressed
when i was done when i it was all
debugging everything i just launched
epiphany the chrome web browser and it
literally just worked there was a little
uh problem around the edge of that phone
because there's a gradient of
semi-transparency and they had a
mishandling of the
pre-multiplied alpha
in this case because videos are straight
we say straight so they they have a
full color information and the alpha
while in
in rendering usually you want to
pre-multiply the color as soon as
possible because if you're going to
render the same object it's it's much
faster
to do the blending it optimize the
blending you also want to do this for
scaling purpose uh but as they never
tested it it was quite uh expected to
have small small little bugs
but they they they fixed it the same
basically the same day i've published
the code so thanks to egalia crew
who did all this nice work
and now as i said i needed to support a
hardware acceleration
and the hardware i'm going to show you
today i'm not going to be able to show
it to the camera there's too many wires
there so i'm going to unplug it if i do
that
it's actually a
renegade inlet from liber computer it
actually hosts a rockchip
rk3399 it's a chip that has been
very popular from i would say 2015 2019
on chromebooks arm chromebooks
uh it's pretty powerful
uh our target was rk
3566 but that's two early samples and
two unstable for me to show you so
what i'm going to do to be able to show
you that i'm going to use another
distribu
pipeline so it's the rk33 viewer
here
which actually runs a v4l2 source
because it's a uvc camera
and
i just select the format because my
little
capture box likes to convert and then
i'll send it to gle matching so this
should if it works
now we're on target display western oh
it's in the background
okay there we go so now we have weston
i need a terminal to actually launch a
demo there
so i'm actually on the device i've
cross-built g-streamer on my machine
and i'm actually i can show you how but
in my g-streamer 3 i'm running a script
called gstm i just asked to strip off
the path from my nfs export
and this way i don't need to install g
streamer every time i make a
modification so it's my development
environment on target
it's a bit slow to load over nfs there's
a lot of files to look at
but it works
then i'll go to the demo folder
uh i wanted to show you the same demos
but turns out my os on this target is
too old so i got a two old mezza with
two old pan frost uh to hold everything
basically to show you the browser
running but be aware this is all
mainline linux
it's a 516 rc something
with
stock mezza running
from fedora and
it's quite good it's quite good i'm
quite impressed these days of what you
can do
with just just open source just mainline
uh
tool so the demo that we're history and
that i'm going to show you is the
overlay the little overlay phone
that's been
there we go so it's starting
there we go so we're on rk3399 we got a
hardware accelerated vp9
alpha transform
one of the one of the thing that you
have to consider when you want to use
hardware acceleration
on these
is that
in order to decode this stream which is
just a simple 30 fps vp9 video it's
1080p
i need 60 fps power in my hardware in
order to do that because i need to
decode two concurrent stream and my my
system need to be well optimized if you
want to go in um streaming standards and
go to 60 fps you need a 120 fps
decoder in alert to do that and if
you're streaming in 4k that could
effectively be a challenge to find
hardware even laptops that supports that
you probably need a big a big gpu to
actually handle it
but keeping that in mind there's always
a way
to do that
there we go
i'm quite impressed but the be aware
that the vp9 decoder on that platform is
a 4k 60 capable so having 60 fps 1080p
1080p should be fine
and voila that's it for my demos
now uh
okay let me keep my there we go so i
like everything else um
it's quite rare that we managed to do
all the features we would like in a go
especially when they're not sponsored
but there's many addition that could be
done so if you want to contribute or you
want to encourage some contributions
feel free to
to to to express your your needs uh one
of the thing i looked at is uh
what other hardware accelerated
accelerator could be used uh to do that
um
i got from uh
okay so i got that it should be fairly
easy with uh direct3d 11 uh decoders to
do that and dark
and version 12 also
uh it's just about someone to do it um
about va api it looks like we need some
more infrastructure around drm pixel
format with the modifiers and all these
things in place before we can actually
do something that is
truly efficient because
um even though they do nv12 they do a
bit of a variation of of nv12 so the
decoder would produce nv12 but the data
is actually whether compressed tiled or
something which is specific and
undocumented well not really
undocumented but it's specific let's say
to intel or md and we need to carry that
modifier that that extra representation
until we get to the gpu in order to
process and color convert the yu yuv to
rgb the the data so until we have this
infrastructure in place it's going to be
hard but i think as soon as this is in
place the wrapper shouldn't be much more
complex than what we have for software
decoder and the v4l decoder i forgot to
list it but i also
i basically forgot to do that i will
probably do that in the next weeks
before release
but i forgot the v4l stateful decoders
which also have vp8 and vp9 support
we're also missing the apple codec
support that would be used on ios and
os x
and we're entirely
missing the encoding and maxing so in
order to encode though there was a
discussion ongoing on the issue tracker
which has been very useful and we
believe that
we need basically the reverse
tool so we need an alpha split that
would actually
take a an a yuv or an av12 buffer and
make two nv12 or two i420 buffers
and we would need the kodak alpha max
which would basically attach the um
the the alpha gsd buffer on the gst
buffer of the colors and finally we
would need to mux it i believe that
maxing will be uh
much easier basically uh we know from
the we will know ahead of time from the
caps that we need to add the alpha
the the alpha tag in the segment
so that's a very good fit and then we
just need to pick the
the the gst meta and wrap it as a block
addition in the stream so that should be
very straightforward i believe
oops
no focus
voila um
oh that's pretty good so thank you ah
that was it and uh we're ready for the
questions
excellent hello i'm back oh i need to
build my headset sorry oh yeah you don't
um we've got some good questions that
was a really interesting talk thank you
so much for that nichola thanks
um the first question that we're gonna
go for is do the resolutions have to
match on the main stream and the alpha
stream could he use a lower resolution
on the alpha stream and upscale
right now they must match
in this it's actually defined by the
webml specification
and it would also be a limitation
because alpha combine
the element that actually takes the two
uh the two streams and combine them
assume that they actually match
so we don't have metadata to describe an
alpha plane which would be um would be a
bit some sample i would say if it was
like a half of a fourth of the plane so
yes
they need to match
okay and we've got a very similar
question as well likewise do the frame
rates have to match on the main and
alpha streams could you use a lower
frame rate for the alpha stream and
interpolate at playback
so the spec is incomplete it doesn't
define everything
but when you look at the streams being
produced
the rules are the following you need
your first frame to have an alpha frame
that's mandatory so when you start you
need to provide the initial data for
your alpha even if it's fully
transparent
and this is to satisfy hardware decoders
basically because they need special
buffers and crafting those buffer with a
cpu is uh it's slow so it would just
delay the start
and increase your latency
and then you're allowed to skip
the alpha channel uh anytime actually
the data does not change
that being said it's a lot of effort uh
for something for for very low gain
because
uh
the
the vp it's something you need to know
about the vpx bitstream which is from
on2
it was highly patented before it was
released by google so thanks for that
it's probably the best
the best compressed bitstream out there
it actually do some statistics uh on the
previous decoded frame in order to
better compress your your parameters so
if if a parameter is always the same
value it's going to use less bits
and if you have
like the same frame it actually can go
up to just couple of bytes to say please
just render a previous frame that's part
of the codex in vp8 and vp9 this is a
very powerful feature so usually it gets
so small
just a couple of bites that it's
probably too much effort to to do that
but you can yes there's no problem with
that
that's real cool
um
i think that is all the questions that
we've got in chat for now yep i must
admit this was an excellent question
thank you
yeah
that's really cool to know though the
encoding like that's really interesting
um
thank you so much for coming and talking
to us this was a really interesting talk
um about some stuff that i don't often
hear about which is really cool
um
i'll just mention one thing uh i had to
do
um there's job opening at collabora so
if
you're looking for a new opportunity
please visit call call.la slash carriers
if you find something that suits you
just you can contact
directly or one of us if you know one of
us we're everywhere on irc on
slack on whatever wherever we're pretty
much all over the place
thank you
cool thank you so much
um
our next talk in this room is the let's
make a game with paris spotfield addison
and that's 11 40 am adt so that's in
like 15 minutes if i can do maths
correctly
um
great
i'll see you later thank you very much
bye