Welcome back. Before I introduce our next speakers, please remember that the
conference close is going to be in the Curlyboi theatre, that's the
room just above this one in Venueless. So don't go anywhere, except for that room after Q&A,
and we'll have the conference room and a nice time there.
Next up, we've got Setting up a machine translation service for Timor-Leste
with Raphael Merx and Mel Mistica.
>> Hi everyone, thanks for coming to our talk. We'll be around for
Q&A after the talk, and also we'll be online as well so please ask questions.
Thanks very much.
>> Allright, hello everyone. Today we're going to talk to how
we setup a machine translation service for Timor-Leste. So machine translation service
is a fancy natural language process term to mean something like Google Translate. So a service
that would translate sentences and whole articles. And we setup a new one for a language which is not
supported by Google Translate or Bing Translate or any other free online machine translation service.
This was entirely setup as a volunteer project. You can check out the project on the website.
We have apps for Android and iOS and there is about 10,000 monthly active users and it is
growing quickly. A little about us. I am Raphael. I am the head of governance and transparency at
a technology for the development sector NGO. Then we have Mel who is our resident NLP
expert and she is a language data scientist at the University of Queensland.
We will start with background, what is the language and why it is important. Mel going to
take us through the project fundamentals and how we can approach it. Then I am going to talk about
how we did it in practice, gathered the data, trained the model and published it. Mel is going
to conclude with future interesting work that we can setup and some of the usage statistics.
All right. Little bit of background. Timor-Leste is a half island nation. The other half of the
island is part of Indonesia. It has around 1.3 million inhabitants and got independence in 2002.
It has a variety of language and people might peek Tetun or Indonesian.
One of the languages standing out is Tetun and Tetun is the langua franca of the country.
A lot of people don't share the same local language will speak Tetun with each other.
Tetun is the most spoken language in the media, in politics, Etc. It is definitely a language of high
importance. Despite that, like I said, it was not on Google translate or Bing translate or any other
translation service. For a person trying to learn Tetun I can find words in individual dictionaries
but if I find a sentence I don't have an easy way to translate into English. A Tetun speaker
that doesn't yet master English, if they find a whole article in English they can't translate it
easily into Tetun. If they have a sentence they are most comfortable speaking in Tetun and want
to translate that to English there is no easy way to do this until we setup this new service.
All right. I am going to leave you to Mel who will take us through the fundamentals of our project.
>> Mel: Thanks for a great summary of the context in which Tetun
is used, Raphael. Given there diversity of the linguistic back grund -- background of the country
we thought it would be great there was a machine learning project for anybody to use.
We asked ourselves a couple of questions when we embarked on this. What resources do
we have or what resources could we quickly and easily develop ourselves? And what was
required to build an MT system. Obviously the resources we needed would depend on the kind of
system we wanted to build. For example, some of the earlier systems were symbolic systems that
were rule-based and required a bunch of rules and lexicons.
Here these kinds of systems were made up of three components. A set of rules for analysis, a set of
rules for transfer or correspondence rules, and a set of rules to generate the source language.
And actually, there were very few resources that were required to just get it up and running.
One of the very first systems from the '50s only had a handful of rules and no more than a couple
of hundred words. These kinds of systems were great for a very limited domain and they were very
quick to get up and going but beyond the rules, the few rules that were developed,
and the very limited number of words that it could analyze, it was not a very -- these were not
very robust systems, not to a wide variety ofm inputs. They were quite brittle.
From the 19'90s onwards, statistical machine learning translation was the made
mode. These models comprised of the translation and language model.
The language model here in purple is trained on vast amounts of monolingual data.
It is the part of the model that's responsible for a fluent sounding output. The translation model is
learned from parallel text. Often millions of sentences. It is very resource hungry.
It is not shown here in this formula. But part of building this kind of system and developing this
kind of translation model, also required modeling the best word alignment between the source
and the target language. The source is denoted by X and Y is the target output. In order to
learn a good translation model, we required lots and lots of parallel data. After 2014,
we then experienced the next big seismic change in machine translation with the use of neural models.
We will talk about neural models a bit more later but first I want to return to our resource issue.
We had only started with an idea of building an MT system with no other resources.
We had no available parallels and having a brittle system was not that appealing to us.
It would require, building this kind of MT system, a symbolic MT system, would require the help of
many linguists who knew how to code and who were possibly proficient in both languages.
But going down the SMT path meant building our own corpus. This bought -- brought other issues.
If we created a corpus for an SMT project, we wanted to be able to have a kind of translation,
the kind of translation system that reflected standard Tetun. This would limit what kinds of
text we could include. It meant that we couldn't include some sources
that were too formal. The other issue we had was that even though you had text that were
bilingual these texts were not completely parallel. It would have to be thrown away.
OK. Popping back to talking about the different types of MT systems. In 2014, we were introduced
to a new way of doing MT using neural networks. Before 2017, we had sequence to sequence models or
seek to seek models. These models were end to end models comprised of an encoder and decoder.
The encoder handles the input sequence and produced one encoding or one representation of the
whole entire input sentence. A decoder was then responsible for generating an output based on the
encoded input. It is a conditional language model that is conditioned on the encoding of the input.
OK. Cut to 2017 where we had attention-based neural networks
introduced to us. This brought about significant improvements in MT. -- in NMT.
We were not relying on how well the encoder represented this input.
The decoder could focus on specific parts of the input that it was currently translating
so it wasn't as though the encoder was responsible to just create this one representation.
The decoder was conditioned, was a language model, that was conditioned on the input but at different
time steps. In essence, it was a more elegant solution to SMTs world alignment problem. So given
these advances in NMT we thought we would try our hand at imelementing an attention-based model
from preexisting and publicly available tools and Raphael will go through -- implementing -- this
and let you know how we went about that process. >> Raphael: Thank you, Mel. Now that Mel has given
us the theory fundamentals of how we will approach building our machine translation model I will take
you through what we did in practice. The first step is gathering parallel English to Tetun text.
What you find in the wild is not parallel sentences which is what you want to train
a model but you will find parallel articles. In our example, there is a wealth of websites
that publish content in Tetun and English. The government website, U.N., NGOs, think tanks,
and some educational books as well. We set out to scrape this data, gather it and try after in
the next step to create parallel sentences from it. This is the example you see on the right-hand
side of the slide. This is an article published on the official government of Tetun website
both in English and Tetun about receiving 65,000 doses of the vaccine from Australia.
Once we have this parallel article we need to find parallel sentences.
The same paragraph in English and Tetun might have a different sentence count. Sometimes
one paragraph in English is never translated to Tetun and the other way around. The kind of tool
used is a sentence aligner. What you do is fit to align parallel article and
a dictionary of what Tetun word corresponds to the English word and it will use this to guess
the sentences that correspond. You throw away sentences. There is a confidence score for each
pair and we try to adjust the threshold that we used here so we would not discard
too many sentences but at the same time the quality of the sentence pairs was pretty good.
Now we have our initial set of sentence pairs, we are still going to have to clean them out before
we fit them to the model. This step is actually fairly important because you could have different
Unicode representations of the same text. You will need to remove long long
printing characters that might confuse your model. Another thing of high importance was
standardizing spelling. This was especially true for Tetun and also English for example like
UK versus American versus Australian spelling. For Tetun, even though there is a government mandated
spelling sometimes people won't use it. This is the case for the word on the right that means
which or where in Tetun and the bottom is the government mandated forms. The forms at the top
are used in the mild so we need to standardize these. There is two goals. One is because we want
our model to use the correct form of Tetun and the second goal is because it will make our model
smaller because it will not confuse the model to see the different forms of the same word
and try to have to guess between which form is the right one. OK. After this,
we apply tokenization. Separating each one of the words by a space and also the punctuation has to
be separated by a space. We spit the corpus into a training set, validation and test set and are
ready to create a first model. What did we use the train the model? This is going to be a
neural network and so we will need a GPU to train our model. In our case, we used Google
pro. It gives you a notebook and if you pay for the service you get a fairly good GPU. We got P100
which were more than good enough. I think the whole model takes half a day to be trained.
The library we used is called fair seq. It is a toolkit built by Facebook AI on top of Pytorch
that is specifically for sequence to sequence models. Machine translation models have a sentence
a sequence of tokens and the output is a sentence which is also a sequence of tokens.
Fairseq has examples on how you can use it to create our own model and made architectures
depending on what you want to do and on the size of your training data. All right. After this, we
created a first model, the quality was pretty poor but it was clear that if we improved it it would
be more than enough to be useful. At this stage, we just iterate. Iterating means getting feedback
from actual real Tetun linguists. Letting them try out the model and they can point out this is
not working like it should. This is fine. Trying to track progress during training using tensor
board. Is the validation set overfitting or the training set overfitting ect. Like I said before,
more spell checks and standardization. In our case it made the model a lot better.
All right. So by the end of the model training, I think there was something like 15-20 iterations we
had a model that was fairly good quality. Google translates for moderately well resourced language
have a BLU score around 40. This is what we had for our model where we had I think 38 BLU score
from English to Tetun and 40 in Tetun to English on the test set. We were pretty happy with it.
Trying it out on various articles in both directions we could see it
was useful and getting the gist of the article and people could actually use
to try to get the meaning of especially a language that isn't theirs. Now that we have a model of
fairly decent quality, we serve it using a Django API. The model is loaded by fairseq
but the load is happening inside a Django end point. Django rest framework, one end point,
three parameters and the text which is a series of sentences, the source language code and target
language code and everything it does is split the text into a bunch of sentences. For each sentence,
we applied cleaning up the corpus so the spell checks and translating each sentence individually
using our first model and responding to the request with that translated text.
That API is used by the front end. A website and mobile app called Tetun.org.
I had created them before for just a simple English Tetun dictionary and
a fair number of people were using it. This is a translate feature added to the
existing front ends. You can see it all here on the right-hand side. It is essentially like
a Google translate. You enter a sentence in English, click translate and it gives you the
sentence in Tetun. >> Mel:
As Raphael mentioned the site hosting the current service used to be a bilingual service
and now core number of users doubled. We get around two and half thousand translations a day
and 10,000 active users per month. The people who normally use the service are people who
translate reports or students of the language translating everyday interactions. We are now
thinking of the next steps and how we improve the quality of the translations.
We are able to translate reports sufficiently but we fall down translating everyday conversational
language because the corpus wasn't trained on this kind of language at all.
One way to remedy this of course is to include this kind of conversational style
as training. In terms of different techniques to improve the overall translation quality, we wanted
to explore ways that we could improve the quality of translation without the reliance of manually
constructing parallel corpra because this is expensive task. One avue -- avenue we want today
pursue was using more monoling u -- monolingual data. We are creating a synthetic parallel corpus.
Please keep in touch with us via the website so you can check how we are fairing in this endeavor.
Thanks very much for attending our presentation. We hope you enjoyed us talking you on our journey
and creating this NMT service for Tetun. Thanks. MODERATOR: Thank you for the talk. Before we
welcome you back to the do the Q&A after the session, just reminding people, after this
Q&A there is a conference close session in the Curlyboi theater above Platypus Hall so don't go
anywhere except that room. That went well. We have several questions with votes on them. We will go
for the first question first. Which is what sort of sizes of corpus do you need for language model
and translation model that you require? >> Mel do you want to answer that?
>> You may need to unmute. >> Do you mean if we went for
the SMT rather than neural? You usually need millions and millions. Raphael we
only had about 120,000 sentences? Is that right? >> Yeah, yeah, that's right. For Tetun it was
enough. It depends on the size of the vocabulary. Depending on whether languages have a very wide
vocabulary or not. In the case of Tetun, you know, each word has usually only one form
apart from some words that come from port -- Portugese. It happened to work fine for us.
>> That's interesting. Cool work. Have you tried using embedded based metrics EG bert score?
>> We haven't. >> No, we haven't. But maybe -- yeah. We could
rather than using bloom we could use berts skull. That's really what it was invented for. Thanks
for that suggestion. This was kind of done on the shoe string as well. Actually, Raphael and I met
via somebody that I met via Facebook. We actually, Raphael, and I have never actually met in-person.
We just kind thought it would be really nice to have this resource and it wasn't available.
>> There is a thousand things that we could do and try if we had unlimited time. This is one
of them. You know, there is so many others like back translation Etc. It is just not enough time.
MODERATOR: We have another question which is how much do you estimate the entire system costs?
>> Raphael: Django is on digital ocean droplet and I think that's $10 per month
and that's it. The front end using netlify. IOS app is costing me a $100 a year because of the
Apple developer fee. The usage is low enough that it is still free and there is the domain. You can
say all together maybe $250 per year. >> But it also cost a lot of love.
We put a lot of love in it too. >> We did. It cost way more time than
it cost money that's for sure. >> Yes. [Laughter]
MODERATOR: Absolutely. I am not sure if we answered this in the talk.
What help do you need going forward? >> It is really just time. Actually,
we have to try to figure out if we want to make sure that we can accommodate this conversational
sort of language. Like we kind of have to be a bit picky about how we are going to construct
that corpus because we don't -- like we said, we wanted to have, you know, more standard language
but, you know, conversational language is really quite informal so how can we stick to standard
language and still kind of, you know, have sort of that type of conversational style.
You know, with blogs sometimes you are going to have spelling that we don't want.
Raphael started looking at more informal kind of corpu that we decided to throw
out. Did you want to talk a bit about that? >> This is something we would need to look for.
For example, we don't have any data coming from places that have informal speak like Facebook.
It exists out there. There is a bunch of people posting on Facebook using Tetun
and probably use more informal language than say a government website. This is what I would
start looking into. We haven't had the time to do this. Trying to find sources that have more
informal language and trying to integrate that and setup a test corpus of this is informal sentences
and we know the translation. Probably the model doesn't perform well on them now. Try to, yeah,
like feed more informal sentences to the model and see how it adapts. Hopefully it doesn't lose
quality and gains quality on the informal speak. It is high priority. Apart from that there is many
things that would be interesting to do. Trying to have a speech to text transcription for the
Tetun language is not something anybody has. It would be useful to a bunch of people trying to,
yeah, like, use data. Add more languages apart from English. I regularly have people reaching
out over the Facebook passenger we have for the service saying can you add Indonesian and I have
tried and the quality wasn't good and I gave up for lack of time. These are things if we added I
know there is people out there who would use it. MODERATOR: We have several questions and only
time for one more question. I think we will pick the next highest voted one which is
what is needed to get this translation service added to Google translate or other platforms?
>> I reached out to the product manager from Google translate before we started working on this
saying hey, you know, we are -- it would be really, really awesome for the country if you
could add this language to Google translate. I was in touch with the ministry of education and
they were the people saying hey, can you figure out some way to reach out to Google. Anyway,
reached out to Google, and then product manager responded saying, you know, we are
essentially focusing on the languages we always support and trying to improve the
quality of translation. Even if they wanted to add languages to their list, I don't have a ranging
of languages by a number of speakers but I am assuming if they have a hundred languages now,
the hundredth language has many more speakers than Tetun. Even if they added another hundred, it is
possible Tetun still wouldn't be in that batch. Essentially, yeah. I mean Google has priorities
and they put a high priority on languages that have a lot of speakers so it isn't their priority
at the moment. He did mention if we have to the order of one million parallel sentences,
they could start looking into it. But I mean, yeah. We are one order of magnitude lower than
this so I am not sure we can find that many more. MODERATOR: That's rough. Thank you so much. We
have more questions so I will put them into the platypus text chat. As Jack said, please,
don't go anywhere. The conference close is in Curlyboi theater after we finish. Head over to
the Curlyboi theater as soon as we finish up here and we will have lovely closing up and
thank yous and recaps of the day. Thank you so much for being here at PyConline AU 2021.
>> Thanks, everyone. >> Bye.