research-article

A study on the use of sequence-to-sequence neural networks for automatic translation of brazilian portuguese to LIBRAS

Authors:

Vinícius Veríssimo,

Cecília Silva,

Manuella Aschoff,

Thaís GaudêncioAuthors Info & Claims

WebMedia '19: Proceedings of the 25th Brazillian Symposium on Multimedia and the Web

Pages 101 - 108

https://doi.org/10.1145/3323503.3360292

Published: 29 October 2019 Publication History

Abstract

The World Health Organization estimates that approximately 466 million people have some level of hearing loss. This significant portion of the world population faces several challenges in accessing information. The main problem is that the languages in which the deaf community can perceive and produce in a natural way are sign languages (SL). An alternative to dealing with this would be the translation of the content from an oral language to SL. However, when it comes to accessing online content, it is necessary to consider translating SL not only for audio or video content, but also for more complex text on websites. This is already a difficult task by itself for the volume involved, and it also addresses some additional challenges, related to the high cost of human interpreter service and the great dynamism of Internet content. In this context, one of the most promising approaches to such scenarios is the use of machine translation applications from oral to sign language. This work evaluates the use of neural network models usually used in natural language processing for the production of LIBRAS glosses from texts in Portuguese. Using a 2^k factorial experiment design, we evaluated the impact of several aspects such as database size, types of models and training parameters in the quality of automatic translation obtained. The results of the experiments were very promising and point to an initial superiority of the LightConv model in most of the evaluated scenarios.

References

[1]

T. M. U. Araújo. 2012. Uma soluçao para geraçao automática de trilhas em lıngua brasileira de sinais em conteúdos multimıdia. 2012. 197 f. Ph.D. Dissertation. Tese (Doutorado em Engenharia Elétrica e de Computaçao)-Universidade Federal. ...

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR (2015).

[3]

Mu-Yen Chen, Min-Hsuan Fan, Young-Long Chen, and Hui-Mei Wei. 2013. Design of experiments on neural network's parameters optimization for time series forecasting in stock markets. Neural Network World 23, 4 (2013), 369.

[4]

Mohamed Amine Chéragui. 2012. Theoretical overview of machine translation. Proceedings ICWIT (2012), 160.

[5]

Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103--111, (2014).

[6]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724--1734 (2014).

[7]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS (2014).

[8]

Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7784--7793.

[9]

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language Modeling with Gated Convolutional Networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR, 933--941.

Digital Library

[10]

Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. 2012. Learning where to attend with deep architectures for image tracking. Neural computation 24, 8 (2012), 2151--2184.

[11]

Cláudia Freitas, Paulo Rocha, and Eckhard Bick. 2008. Floresta sintá (c) tica: bigger, thicker and easier. In International Conference on Computational Processing of the Portuguese Language. Springer, 216--219.

Digital Library

[12]

Jennifer J Gago, Valentina Vasco, Bartek Łukawski, Ugo Pattacini, Vadim Tikhanoff, Juan G Victores, and Carlos Balaguer. 2019. Sequence-to-Sequence Natural Language to Humanoid Robot Sign Language. arXiv preprint arXiv:1907.04198 (2019).

[13]

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 1243--1252.

Digital Library

[14]

Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. 2017. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv:1706.03059 (2017).

[15]

Benson Kituku, Lawrence Muchemi, and Wanjiku Nganga. 2016. A review on machine translation approaches. Indonesian Journal of Electrical Engineering and Computer Science 1, 1 (2016), 182--190.

[16]

Manuella ACB Lima, Tiago MU de Araújo, and Erickson S de Oliveira. 2015. Incorporation of syntactic-semantic aspects in a libras machine translation service to multimedia platforms. In Proceedings of the 21st Brazilian Symposium on Multimedia and the Web. ACM, 133--140.

Digital Library

[17]

Manuella Aschoff Cavalcanti Brandão Lima et al. 2015. Tradução automática com adequação sintático-semântica para LIBRAS. (2015).

[18]

Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412--1421 (2015).

[19]

MD Okpor. 2014. Machine translation approaches: issues and challenges. International Journal of Computer Science Issues (IJCSI) 11, 5 (2014), 159.

[20]

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.

[21]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 311--318.

[22]

Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In 2014 14th International Conference on Frontiers in Handwriting Recognition. IEEE, 285--290.

[23]

Rubén San-Segundo, Juan Manuel Montero, R Córdoba, Valentin Sama, F Fernández, LF D'Haro, Verónica López-Ludeña, D Sánchez, and Antonio García. 2012. Design, development and field evaluation of a Spanish into sign language translation system. Pattern Analysis and Applications 15, 2 (2012), 203--224.

Digital Library

[24]

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715--1725 (2016).

[25]

Daniel Stein. 2018. Machine translation: Past, present and future. Language technologies for a multilingual Europe 4 (2018), 5.

[26]

William C Stokoe, Dorothy C Casterline, and Carl G Croneberg. 1976. A dictionary of American Sign Language on linguistic principles. Linstok Press.

[27]

Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. 2018. Sign Language Production using Neural Machine Translation and Generative Adversarial Networks. In BMVC. 304.

[28]

Marianne Rossi Stumpf. 2000. Lingua de Sinais: escrita dos surdos na Internet. In V Congresso Ibero-Americano de Informâtica na Educação-RIBIE-Chile.

[29]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104--3112.

[30]

Valerie Sutton. 1995. Lessons in sign writing. SignWriting.

[31]

LILIANE CORREIA TOSCANO, BRITO DIZEU, and SUELI APARECIDA CAPORALI. 2005. A língua de sinais constituindo o surdo como sujeito. Educ. Soc 26, 91 (2005), 583--597.

[32]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.

[33]

Krzysztof Wolk. 2019. Machine Learning in Translation Corpora Processing. CRC Press.

[34]

Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay Less Attention with Lightweight and Dynamic Convolutions. arXiv preprint arXiv:1901.10430 (2019).

[35]

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).

Cited By

Zappatore MRuggieri G(2024)Adopting machine translation in the healthcare sectorComputer Speech and Language10.1016/j.csl.2023.10158284:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.csl.2023.101582
Alves VRibeiro JRomero LFaria PCosta ÂFerreira V(2023)A Gloss Based Translation From European Portuguese to Portuguese Sign Language2023 30th International Conference on Systems, Signals and Image Processing (IWSSIP)10.1109/IWSSIP58668.2023.10180304(1-4)Online publication date: 27-Jun-2023
https://doi.org/10.1109/IWSSIP58668.2023.10180304
Alves VRibeiro JFaria PRomero L(2022)Neural Machine Translation Approach in Automatic Translations between Portuguese Language and Portuguese Sign Language Glosses2022 17th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI54924.2022.9820212(1-7)Online publication date: 22-Jun-2022
https://doi.org/10.23919/CISTI54924.2022.9820212
Show More Cited By

Index Terms

A study on the use of sequence-to-sequence neural networks for automatic translation of brazilian portuguese to LIBRAS
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Accessibility
    1. Accessibility systems and tools
    2. Accessibility technologies

Recommendations

Analysis of rule-based machine translation and neural machine translation approaches for translating portuguese to LIBRAS
WebMedia '19: Proceedings of the 25th Brazillian Symposium on Multimedia and the Web

In this paper, we propose a rule-based machine translator for Brazilian Portuguese to Brazilian Sign Language translation. This translator was implemented using the part of speech tagging and lemmatization techniques with implementations in Aelius and ...
AnyLanguage-To-LIBRAS: Evaluation of an Machine Translation Service of Any Oralized Language for the Brazilian Sign Language
WebMedia '17: Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web

Deaf people communicate naturally through visual-gesture languages, called sign languages (LS). As a result, they have a great deal of difficulty absorbing oral content, either in written or spoken form, even in the oral language (LO) of their native ...
Word Reordering for Translation into Korean Sign Language Using Syntactically-guided Classification

Machine translation aims to break the language barrier that prevents communication with others and increase access to information. Deaf people face huge language barriers in their daily lives, including access to digital and spoken information. There ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

WebMedia '19: Proceedings of the 25th Brazillian Symposium on Multimedia and the Web

October 2019

537 pages

ISBN:9781450367639

DOI:10.1145/3323503

General Chairs:
Joel dos Santos
CEFET/RJ
,
Débora Christina Muchaluat Saade
UFF
,
Maria da Graça C. Pimentel
University of Sao Paulo, Brazil
,
Alessandra Alaniz Macedo
University of Sao Paulo, Brazil

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WebMedia '19

WebMedia '19: Brazilian Symposium on Multimedia and the Web

October 29 - November 1, 2019

Rio de Janeiro, Brazil

Acceptance Rates

Overall Acceptance Rate 270 of 873 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
133
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zappatore MRuggieri G(2024)Adopting machine translation in the healthcare sectorComputer Speech and Language10.1016/j.csl.2023.10158284:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.csl.2023.101582
Alves VRibeiro JRomero LFaria PCosta ÂFerreira V(2023)A Gloss Based Translation From European Portuguese to Portuguese Sign Language2023 30th International Conference on Systems, Signals and Image Processing (IWSSIP)10.1109/IWSSIP58668.2023.10180304(1-4)Online publication date: 27-Jun-2023
https://doi.org/10.1109/IWSSIP58668.2023.10180304
Alves VRibeiro JFaria PRomero L(2022)Neural Machine Translation Approach in Automatic Translations between Portuguese Language and Portuguese Sign Language Glosses2022 17th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI54924.2022.9820212(1-7)Online publication date: 22-Jun-2022
https://doi.org/10.23919/CISTI54924.2022.9820212
Lobeiro MVaz RAlves L(2022)A Proposal to Apply SignWriting in IMSC1 Standard for the Next-Generation of Brazilian DTV Broadcasting SystemProceedings of the Brazilian Symposium on Multimedia and the Web10.1145/3539637.3557053(230-233)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3539637.3557053

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents