hello welcome back and welcome all the
way from california jotika singh
she's the vp of data science at icx
where she and her team work on natural
language processing future engineering
and all kinds of machine learning
jotika has a master's degree from the
university of california los angeles
where among other research topics she
worked on signal and speech processing
and developed new approaches to remove
noise from speech
she shares her findings via her open
source projects on github such as pi
youtube analysis and pi audio processing
and that's what jotika is talking about
today
audio
what is audio data
how to build features and classification
models on audio how to solve these
problems in python
now is where we find out from the author
of pi audio processing herself
so please join me in a hand of virtual
applause for jotika singh and
classifying audio into types using
python
thank you so much for the wonderful
introduction um like you mentioned i'll
be talking about classifying audio into
types using python
before diving right in i just quickly
want to introduce myself i work as a vp
of data science as i6 media it's a
content and audience intelligence
company based in washington dc um
attaching my social media handles there
because i'm going to be posting the
slide deck on social media as well on my
twitter account
after the talk also in case anybody has
any questions that you are unable to ask
me during the conference uh you can
shoot a note out to me on twitter
uh also attaching my linkedin and github
accounts for reference i'm also going to
be
you know there's an upcoming book in the
summer or to fall 2022 that i'm working
on that i'm working on authoring it's on
natural language processing in the real
world which contains um
descriptions about how natural language
processing is used across several
industry verticals and actually how to
implement it using python
so without further ado this entire talk
will contain a few sections starting
from what is audio
machine learning at a high level
audio features
tools and using some of these tools and
then classification examples that
classify audio into different types
across different genres uh towards the
end using the tools that we have
discussed previously
so what is audio uh it's essentially a
signal that vibrates in the audible
frequency range what does that mean well
when i'm talking now and you can hear me
through the speakers
uh those sounds basically create air
pressure waves that is then received by
our ear and these pressure signals is
converted uh to some responses that our
brain can understand and finally
recognize the audio as a particular
meaning
there are so many great matlab tools for
just digital signal processing speech
processing and audio processing
a lot of research that goes on you know
the first thing where we can actually
see the effects of those research is in
matlab
given that machine learning is the
language of choice for
building classification models uh uh
python is the language of choice there
uh there's there's little few gaps that
i noticed in the community when i was
trying to build audio classification
models and i needed to extract
particular features so in that attempts
uh there are some open source libraries
that are created to do audio processing
in python
uh and one of them is also pi audio
processing which i'll be talking about
the usage of it in a little bit
so what is machine learning at a high
level uh it's essentially we can imagine
this as divided into three phases
there's a data phase
there's a training phase and an
evaluation phase the data phase has
everything to do with data from data
collection whether you are scraping data
or you have data from some resource or
you're actually leveraging publicly
available data sets to then cleaning of
the data because oftentimes the data is
not in the perfect shape uh that it's
ready for feature extraction but once
you have cleaned the data then we
transform the data so that it is now in
a numerical representation which goes as
an input to the machine learning model
that you're training and then the
evaluation of the model further
influences what else you can do in the
data phase do you need more data do you
need to clean it differently uh do you
need to use other data transformation
techniques
so as you mentioned features are
numerical representation of data
the usable machine learning models but
they really highly depend on the data
type
for instance
if you have a text corpus using word to
web to represent the phrases or the
words within the text corpus as
numerical representations works
perfectly
but if you were to pass just random
numbers through word to work we don't
really expect to get anything meaningful
so there are different feature
generation methods that are suitable for
different types of data
this gets us to audio features now there
are so many different audio features and
we are not going to talk about all of
them but i wanted to mention everything
or not everything but a lot of the
things on one slide so if anybody is
curious and wants to look up other
things for reference
it is there in one place
let's start with
two important things when we're talking
about audio features uh spectrum and
kepstrum what is spectrum
when the audio signal is passed through
a fourier transform what results is a
spectrum but what is it essentially
it is the audio signal in the frequency
domain
how we compute that is using a fourier
transform
if people are aware about fourier series
even if not it is just a way to
represent your signals in terms of sines
and cosines
uh and that is for your transform and
that helps us get the signal in the
frequency domain that is called spectrum
now if we take the log magnitude of the
spectrum to reduce amplitude differences
and then take the inverse fourier
transform what results is a kepstrum
now kepstrom is neither in the time
domain nor in the frequency domain
why is it not in the time domain even
though we took inverse fourier transform
is because of the log magnitude step
uh and because we took inverse fourier
transform it's not in the frequency
domain and oftentimes people refer to
this as a quifferency domain
so to show a representation of how these
uh different representations look
visually uh there's a wave form of a
simple wobble and then the spectrum
followed by the capstone
and then the first 20 capstone
coefficients now one thing great about
the kepstrum is that the first few
sometimes 13 sometimes 20 coefficients
form as great features to build machine
learning models
but why do we care about the frequency
domain right why is it that we are
talking about spectrum and kepstrom and
uh features associated with that now the
inspiration is biology especially if you
consider uh how we even see an image you
know what it looks to the eye is certain
thing but for a computer to process that
uh when our eyes see something we see
something a color like blue or green
uh but for a computer they can represent
it as the pixel values associated behind
the image
right similarly when we hear there's a
whole process that goes on and inspired
by that is how we generate features from
audio
there's the spiral
fluid fill structure in the ear it's
called the cochlea it has thousands of
tiny hair that are of different lengths
what happens is the longer hair resonate
with sounds of lower frequencies and the
shorter hair resonate with higher
frequencies
so it's almost considered because of the
way uh the signal is processed like our
ear is a natural fourier transform
analyzer and this is why uh spectrum
kepstrum is of great interest to us
coming from the kepstrum there are a few
features that form like of great
importance and a lot of
machine learning applications it's
called the mel frequency kept
coefficients
behind it is what we call the mel filter
bank which is just those triangles as
you see on the screen
now you see the triangles keep getting
wider and this is because our human ear
is a breast less frequency selective
after one kilohertz so we want to grab
less and less as we go forward now this
filter the aim is to represent closely
how the human hearing works
and how it's mathematically produced is
the spectrum of the signal passes
through a male scale filter bank which
is those that the filter that you see on
the screen and then a log magnitude
followed by a discrete cosine transform
which results in mfcc features
the discrete cosine transform is uh also
finds application in things like jpeg
compression because the job of the
discrete cosine transform is to get the
shape of the signal rather than the
sharper peaks because it's known that
those sharper smaller peaks are just
noise
another capsule coefficient is called
gamma tone frequency capsule
coefficients it's a little bit different
from mfcc as you see the filter is now
uh with us with a softer slope
and soft edges
now the inspiration here is again how
the human hair works hearing works and
this filter gamma tone
filter bank is known to be the
stimulation of the front edge of the
cochlea so it's again more closely
representing how we hear
and the computation is very similar as
in the spectrum passes through this
filter bank and then there are steps for
down sampling and loudness compressions
followed by discrete cosine transform
and that gets us to our gfcc features
visually mfcc and gfcc look like this as
you can see on the screen for same audio
signal
now we see that you know there are like
mid processing features and they do look
different they convey different
information so it's not like one is a
derivative of the other but it is uh
produced differently through different
processes and each of them convey a
great deal of information when using
machine learning models
they have been used individually as well
as you know a combination of those two
features can be used to build machine
learning models as well
there's some other features that are
proved to be of great importance
especially in applications of speech
processing that is linear prediction
capsule coefficients bark frequency kept
through coefficients power normalized
schedule coefficients
uh and then there's some other which are
related to the spectrum of the signal
like spectrum entropy flux uh how many
times is cross is zero
and then chroma features which
essentially represents the tonal content
of musical audio signal
uh so that's what it it represents so it
can be a very useful feature when
considering classifying
music related content
the many tools that one can be leveraged
in python for audio processing and
bundling audio machine learning
classifiers and i just wanted to list
all of them in one place so if anybody
is interested for different types of
audio processing you please check these
out
coming to the library pi audio
processing it's essentially a python
library for audio analysis and
classification there are a bunch of
different functions that can be
performed using this library starting
from audio format conversion because
you'll see a lot of conversion methods
out there work on wave format but your
audio can be on very different formats
as well so converting it from the
different formats to wave
uh audio visualization you know
sometimes you may want to visualize your
audio with or without building the model
so that's something that we can do with
this library as well there's some audio
cleaning techniques that help you remove
any silence or low activity segments
from your signal before you pass it into
any further processing
uh then there are audio feature
extractions uh for mfcc that was spoken
about
gfcc spectral features and the chroma
features as well
i want to say particularly when i was uh
working on this project where i wanted
to use gfcc i was having a hard time
finding a python implementation
and that's what motivated me to create
this library is when i used the matlab
code that i had and converted that to
python
and that's where this comes from
furthermore when once you have built
your features you can use existing
cyclone classifiers with auto hyper
parameter tuning using this library
if you want you can also use it without
the cyclic loan classifiers if you want
to use it with their own custom backend
uh furthermore there are three
pre-trained models of classification
audio models that are provided with this
library that can help you establish a
baseline if you're working on similar
problems of classifying audio
we remember looking at this particular
diagram in the beginning which talked
about machine learning at a high level
so just converting that to machine
learning for audio signals and how
different components uh can be related
to what we've spoken about
so in terms of data collection you can
use your own data set if you have one
but if you don't there are many publicly
available data sets and that i've
attached in a resources slide that will
be towards the end like i said i'll be
sharing the slide deck so
uh you can free free to access that
resource there
secondly uh we have data cleaning uh
which can be used uh it can be
constituent of converting audio formats
but also cleaning and removing the
silence
segments from the audio so that can be
done using pi audio processing as well
and then transformation is the feature
formation which can be done using pi
audio processing using the extract
features module
these features can be extracted to use
with your own back end or
it can also be used with existing
circuit learn models using the run
classification module which can help you
train and classify your signals and also
give you statistics like confusion
matrix and essentially how your
classifications have run on your
evaluation data set
if you're thinking of starting with such
a problem uh let's talk about the flow
what kind of questions you would ask if
you want to
do something with audio analyze it or
create a classification model so let's
say you have an audio
does it need to be converted to wave
if so
yes we can do that with a module present
in pi audio processing does it need to
be cleaned
if so we can use pi audio processing
clean module does it need to uh do you
need to build a circuit learn classifier
that can be done as well using train and
classify
if not do you want to just extract the
features to use with your own custom
model that can be done as well using
extract features module
and then if not you just want to use a
pre-trained model to just classify audio
that you have
that can be done as well and there are
instructions in the readme to how to
exactly do that
if none of that if you just want to
visualize your audio that can be done as
well using pi audio processing plot
module
and if none of that
please help us by creating an issue on
github and mention all these things that
you want to do in python for audio and
that you're not able to do please create
these issues please feel free to
contribute uh in terms of working on
some of the existing issues as well it's
an open source project and we very much
welcome everybody's input the
communities input
now coming to audio classification we
have talked about the library we've
talked about some features we've talked
about what audio is and let's get it all
together by actually discussing some of
the audio classification examples that
have been built using pi audio
processing
so the first one is the audio type
classification
in this problem we'll be classifying
audio into three possible classes speech
music birds
so the first thing we do is of course
the data that is considered so in in
this case we are using 50 samples each
so total of 150 samples for training
and then for testing there are 14
samples for each class
the first thing we do is train an mfcc
model and keeping the classifier
constant at svm
so the mfcc generated the training
confusion matrix that can be seen on the
top and it looks like it's doing pretty
good
when we pass the test data through this
this model that was created using mfcc
feature we see music is getting
classified 13 out of 14 correctly and
speech and births is 14 out of 14. so
this is a good model and it looks like
mfcc has definitely
got
parts to it that help the machine
learning model to really decipher
between these three classes
this is what a representation looks like
of the features of a speech music and
birth signal so we can see how the
feature looks and there are like
different patterns
associated with the different types of
signals
now just for experimentation purposes
let's try a gfcc feature model and see
if that makes any difference to the
model so the training confusion matrix
still looks good
and when we test it the testing results
are also pretty good we have 14 out of
14 for music and for birds but 12 out of
14 for
uh speech which is a little bit
different from what we had when we were
training with the mfcc feature
so it looks like standalone gfcc is also
contributing something to the model that
helps it decipher clear patterns between
how music speech and birds look
distinctly
here's a comparison between the mfcc
feature and the gfcc feature for speech
music bird sample and we can see it's
relaying different information from the
plots that we can see in front of us
now because they're relaying different
information one last experiment that i
wanted to do was combine these two
features together so use them in
conjunction together and again the
training confusion matrix looks good the
testing one also looks good
again it's more pretty much similar to
how mfcc was performing so further
testing uh could be used on using
further new samples for speech music
bird to evaluate these
but this is mainly to show off
capability of the features itself while
keeping the classifier constant if one
wanted to create a even further invest
into this model and create an even
better model can look at more samples
the quality of data the quantity of data
and then different classification
backends
the second example that i want to talk
about is the music genre classification
now in this case we have 10 music genres
it's pop metal disco blues reggae
classical rock hip hop country and jazz
there are 80 samples for training the
model
per class and then 20 samples per class
for testing uh there's a paper that was
published in 2002 that used mfcc
features for doing this performing this
essential classification and i've linked
that in the resources slide as well so
let's just use mfcc feature again
keeping classifier constant as svm to
see how this performs so it can be seen
with the confusion matrix of the
training side that some of the classes
like metal uh classical they are doing
showing good numbers doing well but then
there are some like country uh disco
that are not doing that great
and when we run our testing samples
through this classifier that was trained
using mfcc feature we see uh
it's again mixed we have classical 18
out of 20 correctly classified but you
know disco blues rock
reggae they're all uh
lower numbers
so let's see if we add features now now
earlier one was just mfcc and now we do
mfcc gfcc spectral as well as chroma
features
now that improves our training confusion
matrix significantly all the numbers
have gone up and same thing we can
notice for the testing 20 samples each
but now we see that pop has gone up by
five more correctly classified disco by
nine um and country by seven so
everything has improved so certainly
adding these features added something to
our model that helped decipher between
these classes better again the
classifier was kept constant at svm if
the classifier is experimented with as
well the model can be further improved
to further see where their model is
failing or not the testing data can be
further elaborated using confusion
matrices and getting the precision
recall in f1 score so this helps you see
which class is going wrong and exactly
where
for example in this case we see a
particularly reggae is really getting
incorrectly classified as hip-hop
a lot so if that was uh somewhere you
know we want to invest time in checking
the data samples that exist the data
quantity the data quality that could be
done and it really depends on also what
your goal is if your goal is mainly to
be able to classify pop and metal and
maybe let's say classical then you
already have a model that does a decent
job for those particular classes
this is just to showcase the capability
of extracting features and using
classifiers but for the things that
could be tried are experimenting with
the data quantity and seeing what the
data sizes look like the data quality in
particular whether there is any noisy
samples of other features that can be
used also another consideration factor
would be some of these uh genres have
music with vocals so maybe some sort of
detection that way and then also other
classifier back ends can be experimented
with
lastly i'm going to discuss a location
name classification problem so
classifying audio that is spoken
location names and seeing if we are able
to decipher it so considering a very
very basic example two spoken uh
location names london and boston now
these have similarities the number of
characters that form these words the
number of cons uh
syllables that form these words
uh is london and boston so we see in
this representation right in front of us
there is something that looks different
in the plot for london in boston so that
makes us feel like okay well easy right
uh it looks very differentiable why
won't a model be able to do that but
then when we compare
three different london spoken
representations in three different
boston basically everything on your left
is london and everything on your right
is boston so these charts very quickly
look different because you know they're
spoken by different people they're
different styles that one can say the
same name different accents different
pause locations so there's a lot of
variety here that goes on in how one
speaks itself
so here we're conducting two experiments
one is we're training only on female
voice samples and then testing on male
voice samples for training we have 23
samples for london and 23 samples for
boston and then for testing we have 17
samples of each for london and for
boston but we're testing on only male
voice and training on only female voice
so let's see if our model can do that
we try an mfcc feature we get a
confusion matrix
we see uh the testing and we have 9 8 9
out of 17 for boston correctly
classified 8 out of 17 for london
let's see if we can improve that so we
tried gfcc a feature entry in the model
and now we have 13 out of 17 correctly
classified for uh london and same for
boston
further trying to improve it adding
spectral features with gfcc now we have
15 out of 17 correctly classified for
london and 14 out of 17 for boston
now there's so much going on here in the
sense our training is only female voice
and testing is only male voice and there
is a difference because males and
females have different lengths of the
vocal tracts which leads to voices in
different pitches so there is that
difference as well
that we are hoping that that model is
able to still pick up on the spoken
representations and get past the
differences in the training in the
testing sample
now if we combine these samples and
shuffle it up and now if you're training
on female and male voice and also
testing on female and male voice
now our data gets more representative
when we're training the model so we see
even mfcc is now doing better than what
it was before when we were just training
on female samples and then testing on
male voice samples
so now we see with with this different
representation all the models are doing
better the training confusion matrix
looks much better because of the
representation that we have
and all both of these experiments were
done using the svm classifier as well
just to keep the classifier constants we
were able to compare the effects of
different features
here's the very much promised resource
slide it has several links to features
that we did not talk about and also some
of the papers that i mentioned
uh the music genre classification
and the audio data sets where you can
find publicly available open source data
sets
finally i want to thank everyone for
tuning in it's been a pleasure thank you
so much
thank you jotika and we have some
questions for you um
the first one is do you think this
toolkit could be useful for other
vibrational analysis like um the example
they give is seismic signal processing
earthquakes
that is a very very interesting uh
thought so because this library deals
with mainly audio if you're talking
about any audio effects of these signals
or any patterns that are audible that
you can hear then i think it's
definitely worth a try
i have personally not uh you know heard
of that uh
application before but it sounds very
interesting and if there's any audible
component to it i would definitely give
it a shot
there's another question that asks do
you think this toolkit um so is my audio
processing language agnostic or is it
just for english
oh so well it's actually it doesn't
matter so
the pi audio processing library is
essentially written in python but what
language or audio is in it does not
matter because it's going to train on
the samples you provide so if your
samples are in english it's going to
train on that data so it really depends
on the data that you pass in rather than
the library the library should be
expected to do similar
and how do you deal with noisy data so
is there um have you considered
automated noise classification and
cleaning to any degree
yeah that's a very good question noise
is a very ill i would say frustrating
component of just dealing with anything
classification and dealing with data
that's a good question and i think one
thing about noise some components are
very basic and simple where you just
have a signal where the interest
is in a very particular segments and the
audio itself is long so that way uh it's
more
simpler solutions like removing of the
silent segments but then there are other
aspects removing noise like spectral
subtraction
in which you just take the pull audio
and take the less
less i would say component filled
portions
see what the signal looks like there and
just subtract that from the entire
signal to remove noises such as
something going on in the background you
know the
like train noise car noise is something
going on in the background so i think
it's a very interesting application and
there has been the use of mfcc features
a bunch to even remove noise from data
as features in gfcc as well especially
in speaker identification when they're
noisy samples provided these features
help you clean them up as well
we have
a lot of questions on venueless and the
next one is
you've mentioned
mfcc and gfcc features uh what are the
implications for classifying speech
features in people like accents the age
of the speaker and maybe other features
very good question as well so there are
a lot of applications of these features
itself in like gender classification
uh
and so on gfcc specifically is very
useful for speaker identification as
well so if you want to have a task where
you want to differentiate between people
in terms of in different age groups as
well i would definitely give that a shot
and then there's pncc
bfcc there are other capstone
coefficients that really help uh break
down your signal and classify it into
types as such especially when it's
considered when it has speaker
information associated
um what's the visualization plotting
stack uh based on for pi audio
processing here's another question from
the audience
can you repeat that sorry what's the
visualization and plotting stack based
on for power processing yes it's a good
question so it's mainly uh scikit-learn
and matplotlib
and essentially if you have your
features or any data that you want to
visualize once you have the data you
know visualizing is uh can be done with
anything like matt gottlieb seabourn any
of any of your favorite visualization
libraries
and the final question we still have
time for it so people asking where the
music samples entire songs or were you
only clipping part of the song and if
you did both does the length of the
sample have
an effect on model performance that's a
very good question the length of the
sample unless it's like really short and
does not convey much information would
not have a very significant impact
because you're windowing the signal and
that's where you're extracting features
and then averaging it out for your
signal the data set specifically i used
was entire audios and it's called the gt
zan data set and i've also attached a
link in the resources slide to this data
set which contains all these genres so
if you want to explore the particular
data set it's it's going to be right
there
thank you so much jotika singh thank you
for speaking at pike on a you 2021.
thank you so much it's been a pleasure
thank you very much for organizing the
whole thing
and for our audience now at home we have
a bit of a break and we'll be back at
let me check at 1 30 melbourne time with
graph data science and paco nathan see
you then