Elamur Research⌕
☰
home / papers / 2021 / Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas

Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas

Summary

The AmericasNLP 2021 shared task drew broader NLP interest to automatically translating the native languages spoken across the Americas. These languages have very little data, inconsistent spelling, unclear word boundaries and strong dialect differences. The task paired ten languages from eight families with Spanish. Organizers supplied publicly available parallel corpora for training, a small transformer baseline taken from the FLORES setup, and new development and test sets from AmericasNLI, a manual translation of Spanish XNLI.

The task was "open," so teams could add any external data, dictionaries or pretrained models. Track 1 allowed training on the development set; Track 2 did not. ChrF set the official ranking, BLEU was also reported, and a small manual evaluation rated adequacy and fluency.

Eight teams sent 214 submissions, and most clearly beat the baseline. The strongest systems relied on careful data cleaning, normalization, domain adaptation and compact multilingual models. Statistical MT also stayed competitive. These results give a benchmark and practical guidance for low-resource, morphologically rich languages.

Key points

  • The ten languages translated from Spanish were Asháninka, Aymara, Bribri, Guarani, Nahuatl, Otomí, Quechua, Rarámuri, Shipibo-Konibo and Wixarika. Training data ranged from about 3.9k sentence pairs (Asháninka) to about 125k (Quechua), and each dev/test set had roughly 1,000 sentences.
  • ChrF was the main metric because many of these languages are polysynthetic and lack standardized word boundaries. Averaged across languages, the best systems scored 12.97 ChrF points above the baseline.
  • The AmericasNLI dev/test data stayed private until the competition ended, so it could not leak into training. It often differed from the training corpora in dialect, orthography and domain; for example, Nahuatl training data is close to Classical orthography, while the test set is modern Huasteca Nahuatl.
  • For the manual evaluation, the translator who built the test sets rated 100 random sentences each for Shipibo-Konibo and Otomí. Ratings used separate 5-point adequacy and fluency scales and covered the baseline plus the three top systems: Helsinki, Tamalli and UTokyo.
  • Compact multilingual NMT models beat massively multilingual ones. SMT systems outperformed some larger pretrained submissions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.americasnlp-1.23

@inproceedings{mager2021findings,
  title = {{Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas}},
  author = {Mager, Manuel and Oncevay, Arturo and Ebrahimi, Abteen and Ortega, John and Rios, Annette and Fan, Angela and Gutierrez-Vasques, Ximena and Chiruzzo, Luis and Giménez-Lugo, Gustavo and Ramos, Ricardo and Ruiz, Ivan Vladimir Meza and Coto-Solano, Rolando and Palmer, Alexis and Mager-Hois, Elisabeth and Chaudhary, Vishrav and Neubig, Graham and Vu, Ngoc Thang and Kann, Katharina},
  booktitle = {Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas},
  year = {2021},
  doi = {10.18653/v1/2021.americasnlp-1.23},
  url = {https://doi.org/10.18653/v1/2021.americasnlp-1.23}
}

References

56 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić and Ivan Vulić. 2019. JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204–3210, Florence, Italy. Association for Computational Linguistics. find our card
  2. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Sant… find our card
  3. El Moatez Billah-Nagoudi, Wei-Rui Chen, Muhammad Abdul-Mageed, and Hasan Cavusoglu. 2021. IndT5: A Text-to-Text Transformer for 10 Indigenous Languages. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas, Online. Association for Co… find
  4. Marcel Bollmann, Rahul Aralikatte, Héctor Murrieta-Bello, Daniel Hershcovich, Miryam de Lhoneux, and Anders Søgaard. 2021. Moses and the character-based random babbling baseline: CoAStaL at AmericasNLP 2021 shared task. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation f… find
  5. David Brambila. 1976. Diccionario rarámuri-castellano (tarahumar). Obra Nacional de la buena Prensa. find
  6. Luis Chiruzzo, Pedro Amarilla, Adolfo Ríos, and Gustavo Giménez Lugo. 2020. Development of a Guarani - Spanish parallel corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2629–2633, Marseille, France. European Language Resources Association. find
  7. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Language resources and evaluation, 49(2):375–395. find our card
  8. Matthew Coler and Petr Homola. 2014. Rule-based machine translation for Aymara, pages 67–80. Cambridge University Press. find
  9. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… find
  10. Adolfo Constenla, Feliciano Elizondo, and Francisco Pereira. 2004. Curso Básico de Bribri. Editorial de la Universidad de Costa Rica. find
  11. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… find
  12. Maximiliano Duran. 2010. Lengua general de los incas. http://quechua-ayacucho.org/es/index_es.php. Accessed: 2021-03-15. ↗
  13. Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir, Gustavo A. Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando A. Coto Solano, Ngoc Thang Vu, and Katharina Kann. 2021. AmericasNLI:… find
  14. Isaac Feldman and Rolando Coto-Solano. 2020. Neural machine translation models with back-translation for the extremely low-resource indigenous language Bribri. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3965–3976, Barcelona, Spain (Online). International… find
  15. Dayana Iguarán Fernández, Ornela Quintero Gamboa, Jose Molina Atencia, and Oscar Elías Herrera Bedoya. 2013. Design and implementation of an “Web API” for the automatic translation Colombia’s language pairs: Spanish-Wayuunaiki case. In Communications and Computing (COLCOM), 2013 IEEE Colombian Confe… find
  16. Sofía Flores Solórzano. 2017. Corpus oral pandialectal de la lengua bribri. http://bribri.net. ↗
  17. Ana-Paula Galarreta, Andrés Melgar, and Arturo Oncevay. 2017. Corpus creation and initial SMT experiments between Spanish and Shipibo-konibo. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 238–244, Varna, Bulgaria. INCOMA Ltd. find
  18. Candace Kaleimamoowahinekapu Galla. 2016. Indigenous language revitalization, promotion, and education: Function of digital technology. Computer Assisted Language Learning, 29(7):1137–1151. find
  19. Héctor Erasmo Gómez Montoya, Kervy Dante Rivas Rojas, and Arturo Oncevay. 2019. A continuous improvement framework of machine translation for Shipibo-konibo. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pages 17–23, Dublin, Ireland. European Association for Ma… find
  20. Ximena Gutierrez-Vasques. 2015. Bilingual lexicon extraction for a distant language pair using a small parallel corpus. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 154–160. find
  21. Ximena Gutierrez-Vasques, Gerardo Sierra, and Isaac Hernandez Pompa. 2016. Axolotl: a web accessible parallel corpus for Spanish-Nahuatl. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4210–4214, Portorož, Slovenia. European Language Resour… find
  22. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho… find our card
  23. Diego Huarcaya Taquiri. 2020. Traducción automática neuronal para lengua nativa peruana. Bachelor’s thesis, Universidad Peruana Unión. find
  24. Cesar Iter and Zenobio Ortiz-Cárdenas. 2019. Runasimita yachasun. Método de quechua, 1 edition. Lima. find
  25. Instituto Francés de Estudios Andinos, Carla Victoria Jara Murillo. 2018a. Gramática de la Lengua Bribri. EDigital. find
  26. Carla Victoria Jara Murillo. 2018b. I Ttè Historias Bribris, second edition. Editorial de la Universidad de Costa Rica. find
  27. Carla Victoria Jara Murillo and Alí García Segura. 2013. Se’ ttö’ bribri ie Hablemos en bribri. EDigital. find
  28. Rebecca Knowles, Darlene Stewart, Samuel Larkin, and Patrick Littell. 2021. NRC-CNRC Machine Translation Systems for the 2021 AmericasNLP Shared Task. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas, Online. Association for Comp… find
  29. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for… find
  30. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… find our card
  31. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  32. James Loriot, Erwin Lauriault, Dwight Day, and Peru. Ministerio de Educación. 1993. Diccionario Shipibo-Castellano. find
  33. Manuel Mager, Diónico Carrillo, and Ivan Meza. 2018a. Probabilistic finite-state morphological segmenter for wixarika (huichol) language. Journal of Intelligent & Fuzzy Systems, 34(5):3081–3087. find
  34. Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza-Ruiz. 2018b. Challenges of language technologies for the indigenous languages of the Americas. In Proceedings of the 27th International Conference on Computational Linguistics, pages 55–69, Santa Fe, New Mexico, USA. Association f… find
  35. Manuel Mager and Ivan Meza. 2018. Hacia la traducción automática de las lenguas indígenas de méxico. In Proceedings of the 2018 Digital Humanities Conference. The Association of Digital Humanities Organizations. find
  36. Enrique Margery. 2005. Diccionario Fraseológico Bribri-Español Español-Bribri, second edition. Editorial de la Universidad de Costa Rica. find
  37. Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel bible corpus. Oceania, 135(273):40. find our card
  38. Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In Proceedings of the 12th Language Resources and Evaluation Conference, p… find our card
  39. Elena Mihas. 2011. Añaani katonkosatzi parenini, El idioma del alto Perené. WI:Clarks Graphics. find
  40. Oscar Moreno. 2021. The REPU CS’ spanish–quechua submission to the AmericasNLP 2021 shared task on open machine translation. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas, Online. Association for Computational Linguistics. find
  41. Sebastian Nordhoff and Harald Hammarström. 2012. Glottolog/Langdoc:Increasing the visibility of grey literature for low-density languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3289–3294, Istanbul, Turkey. European Language Resour… find
  42. John Ortega, Richard Alexander Castro-Mamani, and Jaime Rafael Montoya Samame. 2020. Overcoming resistance: The normalization of an Amazonian tribal language. In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, pages 1–13, Suzhou, China. Association for Computational… find
  43. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  44. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  45. Shantipriya Parida, Subhadarshi Panda, Amulya Dash, Esau Villatoro-Tello, A. Seza Doğruöz, Rosa M. Ortega-Mendoza, Amadeo Hernández, Yashvardhan Sharma, and Petr Motlicek. 2021. Open Machine Translation for Low Resource South American Languages (AmericasNLP 2021 Shared Task Contribution). In Procee… find
  46. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  47. Prokopis Prokopidis, Vassilis Papavassiliou, and Stelios Piperidis. 2016. Parallel Global Voices: a Collection of Multilingual Corpora with Citizen Media Stories. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 900–905, Portorož, Slovenia. E… find
  48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21:1–67. find
  49. Romano, Rubén and Richer, Sebastián. 2008. Ñaantsipeta asháninkaki birakochaki. www.lengamer.org/publicaciones/diccionarios/. find
  50. Lane Schwartz, Francis Tyers, Lori Levin, Christo Kirov, Patrick Littell, Chi-kiu Lo, Emily Prud’hommeaux, Hyunji Hayley Park, Kenneth Steimel, Rebecca Knowles, et al. 2020. Neural polysynthetic language modelling. arXiv preprint arXiv:2005.05477. — [arXiv:2005.05477](https://arxiv.org/abs/2005.05477) find
  51. Michel Simard and Atsushi Fujita. 2012. A poor man’s translation memory using machine translation evaluation metrics. In Proceedings of the 10th Biennial Conference of the Association for Machine Translation in the Americas (AMTA). find
  52. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA). find
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  54. Raúl Vázquez, Yves Scherrer, Sami Virpioja, and Jörg Tiedemann. 2021. The Helsinki submission to the AmericasNLP shared task. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas, Online. Association for Computational Linguistics. find
  55. Shiyue Zhang, Benjamin Frey, and Mohit Bansal. 2020. ChrEn: Cherokee-English machine translation for endangered language revitalization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 577– 595, Online. Association for Computational Linguistic… find
  56. Zheng, Francis, Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2021. Low-Resource Machine Translation Using Cross-Lingual Language Model Pretraining. In Proceedings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas, Online. Association f… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mager-2021-findings-americasnlp-2021-shared-task/