Elamur Research⌕
☰
home / papers / 2018 / Challenges of language technologies for the indigenous languages of the Americas

Challenges of language technologies for the indigenous languages of the Americas

Paper · 2018 · Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra et al.

Summary

The Americas have roughly 900 indigenous languages spoken by about 28 million people. Many are endangered, yet language technology has largely ignored them. Mager and colleagues address this gap with a survey rather than a new system. They first describe typological traits that make standard NLP pipelines struggle: polysynthetic and agglutinative morphology, tonal contrasts, strong dialectal variation and missing spelling standards. They then catalogue the digital corpora, dictionaries, speech data and tools they could find. A task-by-task review of published work follows, covering machine translation, morphological analysis and segmentation, POS tagging, parsing, spell checking, OCR and language identification. They found NLP work on only about 35 languages. It concentrates on North American languages, the Uto-Aztecan family and the largest South American languages, with morphology and MT dominating and activity rising markedly after 2013. The authors argue that subword-aware, low-resource methods look most promising. Progress would test how general NLP models really are and could benefit speaker communities. The paper comes with a continuously updated public list of resources.

Key points

  • Scale of diversity: about 140 language families exist worldwide, and close to 40% of them are native to the Americas. The authors identified NLP contributions for only ~35 of the continent's languages.
  • Parallel corpora are scarce and often built from non-digital sources. Examples include Axolotl (Spanish–Nahuatl, with dialectal, diachronic and orthographic variation), a downloadable Wixarika–Spanish corpus of Andersen tales, and a Spanish–Shipibo-konibo corpus. Shipibo-konibo also has a POS-tagged corpus and an online dictionary.
  • Morphological datasets include:
    • CoNLL-SIGMORPHON 2017, which covers Haida (7,040 words), Navajo (12,000) and Quechua (12,000).
    • An Oto-Manguean inflectional class database with over 13,000 verb entries from twenty languages.
    • The Kann et al. (2018) segmentation set of 4,468 words from Mexicanero, Nahuatl, Wixarika and Yorem Nokki.
  • Other resources and tasks:
    • A Guarani speech corpus of 1,000 phrases from 110 speakers (not publicly released).
    • Monolingual corpora for 16 Peruvian languages, plus language identification across 17 languages that tolerates non-standard spelling.
    • Parsing experiments using the Quechua-Spanish Treebank.
  • The paper includes a Nahuatl version of its title and abstract. Its resource list is maintained at github.com/pywirrarika/naki.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 166 works

Cite

Cite as Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra and Ivan Meza-Ruiz. 2018. Challenges of language technologies for the indigenous languages of the Americas. Proceedings of the 27th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mager2018challenges,
  title = {{Challenges of language technologies for the indigenous languages of the Americas}},
  author = {Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra and Ivan Meza-Ruiz},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1006/}
}

References

94 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, James Cowie, Steve Helmreich, Wanying Jin, Maria Pilar Milagros, Bill Ogden, Hamid Mansouri Rad, and Ron Zacharski. 2006. Guarani: a case study in resource development for quick ramp-up mt. In Proceedings of the 7th Conference of the Association for Machine Translation in the America… find
  2. Carlo Alva and Arturo Oncevay. 2017. Spell-checking based on syllabification and character-level graphs for a peruvian agglutinative language. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 109–116. find
  3. Antti Arppe, Marie-Odile Junker, and Delasie Torkornoo. 2017. Converting a comprehensive lexical database into a computational model: The case of East Cree verb inflection. ComputEL-2, page 52. find
  4. Ehsaneddin Asgari and Hinrich Schütze. 2017. Past, present, future: A computational investigation of the typology of tense in 1000 languages. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 113–124. find
  5. Alicia Alexandra Assini. 2014. Natural language processing and the Mohawk language: creating a finite state morphological parser of Mohawk formal nouns. Scholars’ Press. find
  6. Emily M Bender. 2011. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6(3):1–26. find
  7. Christian Bentz, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardzic. 2016. A comparison between morphological complexity measures: typological data vs. language corpora. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 142–153. find
  8. Dustin Bowers, Antti Arppe, Jordan Lachler, Sjur Moshagen, and Trond Trosterud. 2017. A morphological parser for Odawa. ComputEL-2, page 1. find
  9. Joan Bresnan, Ash Asudeh, Ida Toivonen, and Stephen Wechsler. 2015. Lexical-functional syntax, volume 16. John Wiley & Sons. find
  10. Malgorzata Cavar, Damir Cavar, and Hilaria Cruz. 2016. Endangered language documentation: Bootstrapping a Chatino speech corpus, forced aligner, asr. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo,… find
  11. Indhira Castro Cavero and Jaime Farfán Madariaga. 2007. Traductor morfológico del castellano y quechua. Revista I+i, 1(1). find
  12. Matt Coler and Petr Homola. 2014. Rule-based machine translation for aymara. Endangered Languages and New Technologies, page 67. find
  13. Marta R Costa-jussà and José AR Fonollosa. 2016. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 357–361.
  14. Rolando Coto-Solano and Sofı́a Flores Solórzano. 2016. Alineación forzada sin entrenamiento para la anotación automática de corpus orales de las lenguas indı́genas de costa rica. Káñina, 40(4):175–199. find
  15. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  16. The SIGMORPHON 2016 shared task—morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22. find
  17. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Gėraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, et al. 2017. find
  18. CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30. find
  19. Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. In Workshop on Morphological and Phonological Learning. find
  20. Mathias Creutz and Krista Lagus. 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology. find
  21. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and improving morphological learning in the neural machine translation decoder. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volu… find
  22. Andrés de Olmos, Ascensión H de León-Portilla, and Miguel León Portilla. 2002. Arte de la lengua mexicana, volume 9. UNAM. find
  23. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  24. Joel Dunham, Gina Cook, and Joshua Horner. 2014. Lingsync & the online linguistic database: New models for the collection and management of data for language communities, linguists and language learners. In Proceedings of the 2014 Workshop on the Use of Computational Methods in the Study of Endanger… find
  25. Alexandra Espichán-Linares and Arturo Oncevay-Marcos. 2017. A low-resourced peruvian language identification model. In Proceedings of the SIMBig 2017 Track on Applied Natural Language Processing, ANLP 2017. find
  26. Benoit Farley. 2012. The Uqailaut project. URL http://www.inuktitutcomputing. ca. ↗
  27. Dayana Iguarán Fernández, Ornela Quintero Gamboa, Jose Molina Atencia, and Oscar Elı́as Herrera Bedoya. 2013. Design and implementation of an “web api” for the automatic translation colombia’s language pairs: Spanish-Wayuunaiki case. In Communications and Computing (COLCOM), 2013 IEEE Colombian Co… find
  28. Mikel L Forcada, Mireia Ginestı́-Rosell, Jacob Nordfalk, Jim O’Regan, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martı́nez, Gema Ramı́rez-Sánchez, and Francis M Tyers. 2011. Apertium: a free/open-source platform for rule-based machine translation. Machine translation, 25(2):127–… find
  29. Ana Paula Galarreta, Andrés Melgar, and Arturo Oncevay. 2017. Corpus creation and initial SMT experiments between Spanish and Shipibo-konibo. In Proceedings of RANLP. find
  30. Dan Garrette and Hannah Alpert-Abrams. 2016. An unsupervised model of orthographic variation for historical document transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 467–472. find
  31. John Goldsmith. 2001. Unsupervised learning of the morphology of a natural language. Computational Linguistics, 27(2):153–198. find
  32. José González Hernández. 2016. Herramienta de traduccin automática de wayuunaiki a españkol. Caso de estudio: sintagmas nominales y verbales simples. Ph.D. thesis, Universidad de Zulia. find
  33. Ximena Gutierrez-Vasques, Gerardo Sierra, and Hernandez Isaac. 2016. Axolotl: a web accessible parallel corpus for Spanish-Nahuatl. In International Conference on Language Resources and Evaluation (LREC). find
  34. Ximena Gutierrez-Vasques. 2015. Bilingual lexicon extraction for a distant language pair using a small parallel corpus. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 154–160. find
  35. Ximena Gutierrez-Vasques. 2017. Exploring bilingual lexicon extraction for Spanish-Nahuatl. In ACL Workshop in Women and Underrepresenting Minorities in Natural Language Processing. find
  36. Atticus G Harrigan, Katherine Schmirler, Antti Arppe, Lene Antonsen, Trond Trosterud, and Arok Wolvengrey. 2017. Learning from the computational modelling of Plains Cree verbs. Morphology, 27(4):565–598. find
  37. Armin Hoenen. 2016. Wikipedia titles as noun tag predictors. In International Conference on Language Resources and Evaluation (LREC). find
  38. Petr Homola. 2011. Parsing a polysynthetic language. In RANLP, pages 562–567. find
  39. Carolina Huenchullan. 2000. Data collection and language technologies for Mapudungun. In International Conference on Language Resources and Evaluation (LREC). find
  40. Mans Hulden. 2009. Foma: a finite-state compiler and library. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics: Demonstrations Session, pages 29–32. Association for Computational Linguistics. find
  41. Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 62–70. find
  42. Katharina Kann and Hinrich Schütze. 2017. The LMU system for the CoNLL-SIGMORPHON 2017 shared task on universal morphological reinflection. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 40–48. find
  43. Katharina Kann, Manuel Mager, Ivan Meza, and Hinrich Shütze. 2018. Fortification of neural morphological segmentation models for polysynthetic minimal-resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  44. Kimmo Kettunen. 2014. Can type-token ratio be used to show morphological complexity of languages? Journal of Quantitative Linguistics, 21(3):223–245. find
  45. Ben King and Steven Abney. 2013. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1110–1119. find
  46. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints.
  47. Oskar Kohonen, Sami Virpioja, and Krista Lagus. 2010. Semi-supervised learning of concatenative morphology. In ACL Special Interest Group on Computational Morphology and Phonology. find
  48. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  49. Paul Lamere, Philip Kwok, Evandro Gouvea, Bhiksha Raj, Rita Singh, William Walker, Manfred Warmuth, and Peter Wolf. 2003. The CMU SPHINX-4 speech recognition system. In IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP 2003), Hong Kong, volume 1, pages 2–5. find
  50. Ariadna Font Llitjós, Roberto Aranovich, and Lori Levin. 2005. Building machine translation systems for indigenous languages. In Second Conference on the Indigenous Languages of Latin America (CILLA II), Texas, USA. find
  51. Manuel Mager and Ivan Meza. 2018. Hacia la traducción automática de las lenguas indı́genas de méxico. In Proceedings of the 2018 Digital Humanities Conference. The Association of Digital Humanities Organizations. find
  52. Manuel Mager, Diónico Carrillo, and Ivan Meza. 2018. Probabilistic finite-state morphological segmenter for the Wixarika (Huichol) language. Journal of Intelligent & Fuzzy Systems, 34(5):3081–3087. find
  53. Jesus Manuel Mager Hois, Carlos Barron Romero, and Ivan Vladimir Meza Ruı́z. 2016. Traductor estadı́stico wixarika - español usando descomposición morfológica. COMTEL, (6). find
  54. Jesus Manuel Mager Hois. 2017. Traductor hı́brido wixárika - español con escasos recursos bilingües. Master’s thesis, Universidad Autónoma Metropolitana. find
  55. Peter Makarov, Tatiana Ruzsics, and Simon Clematide. 2017. Align and copy: Uzh at sigmorphon 2017 shared task for morphological reinflection. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 49–57. find
  56. Diego Manuel Maldonado, Rodrigo Villalba Barrientos, and Diego P Pinto-Roa. 2016. Eñe’e: Sistema de reconocimiento automático del habla en guaranı́. In Simposio Argentino de Inteligencia Artificial (ASAI 2016)-JAIIO 45 (Tres de Febrero, 2016). find
  57. Michael Maxwell and Aric Bills. 2017. Endangered data for endangered languages: Digitizing print dictionaries. ComputEL-2, page 85. find
  58. Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel bible corpus. Oceania, 135(273):40.
  59. Norman A McQuown. 1955. The indigenous languages of latin america. American Anthropologist, 57(3):501– 570. find
  60. A Medina Urrea and M Alvarado Garcı́a. 2006. Un experimento de reconocimiento automático de la derivación léxica en el ralámuli. La lengua y la antropologı́a para un conocimiento global del hombre. find
  61. Alfonso Medina-Urrea. 2007. Affix discovery by means of corpora: Experiments for Spanish, Czech, Ralámuli and Chuj. In Aspects of Automatic Text Analysis, pages 277–299. Springer. find
  62. Alfonso Medina-Urrea. 2008. Affix discovery based on entropy and economy measurements. Texas Linguistics Society, pages 99–112. find
  63. Jeffrey C Micher. 2017. Improving coverage of an inuktitut morphological analyzer using a segmental recurrent neural network. ComputEL-2, page 101. find
  64. Marianne Mithun. 2001. The languages of native North America. Cambridge University Press. find
  65. Marianne Mithun. 2017. Polysynthesis in north america. In The Oxford Handbook of Polysynthesis. find
  66. Christian Monson, Lori Levin, Rodolfo M Vega, Ralf D Brown, Ariadna Font-Llitjos, Alon Lavie, Jaime G Carbonell, Eliseo Cañulef, and Rosendo Huisca. 2004. Data collection and analysis of Mapudungun morphology for spelling correction. Computer Science Department, page 300. find
  67. Christian Monson, Ariadna Font Llitjós, Roberto Aranovich, Lori Levin, Ralf Brown, Eric Peterson, Jaime Carbonell, and Alon Lavie. 2006. Building NLP systems for two resource-scarce indigenous languages: mapudungun and quechua. Strategies for developing machine translation for minority languages, p… find
  68. Sebastian Nordhoff, Harald Hammarström, Robert Forkel, and Martin Haspelmath. 2013. Glottolog 2.0. Max Planck Institute for Evolutionary Anthropology. find
  69. Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, and Anna Korhonen. 2016. Survey on the use of typological information in natural language processing. arXiv preprint arXiv:1610.03349. find
  70. E.L. Palancar and T. Feist. 2015. Oto-Manguean Inflectional Class Database. University of Surrey. find
  71. José Pereira-Noriega, Rodolfo Mercado-Gonzales, Andrés Melgar, Marco Sobrevilla-Cabezudo, and Arturo Oncevay-Marcos. 2017. Ship-lemmatagger: Building an NLP toolkit for a peruvian native language. In International Conference on Text, Speech, and Dialogue, pages 473–481. Springer. find
  72. Maja Popovic and Hermann Ney. 2004. Towards the use of word stems and suffixes for statistical machine translation. In International Conference on Language Resources and Evaluation (LREC). find
  73. Andrés Osvaldo Porta. 2010. The use of formal language models in the typology of the morphology of amerindian languages. In Proceedings of the ACL 2010 Student Research Workshop, pages 109–113. Association for Computational Linguistics. find
  74. Katharina Probst, Ralf D Brown, Jaime G Carbonell, Alon Lavie, Lori Levin, and Erik Peterson. 2001. Design and implementation of controlled elicitation for machine translation of low-density languages. Computer Science Department. find
  75. Gilbert Ramı́rez and Lustig Wolf. 1996. Interactive guarani dictionary. https://www.uni-mainz.de/ cgi-bin/guarani2/dictionary.pl. Accessed: 2018-03-16. ↗
  76. Annette Rios, Anne Göhring, and Martin Volk. 2008. A Quechua-Spanish parallel treebank. LOT Occasional Series, 12:53–64. find
  77. Annette Rios. 2016. A basic language technology toolkit for quechua. find
  78. H. Santiago. 2017. Método para la alieación automática de textos entre los idiomas mixteco y español. Master’s thesis, Centro Nacional de Investigación y Desarrollo Tecnológico. find
  79. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715–1725.
  80. Gary F Simons and Charles D Fennig. 2017. Ethnologue: Languages of the world. SIL, Dallas, Texas. find
  81. Conor Snoek, Dorothy Thunder, Kaidi Loo, Antti Arppe, Jordan Lachler, Sjur Moshagen, and Trond Trosterud. 2014. Modeling the noun morphology of Plains Cree. In Proceedings of the 2014 Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 34–42. find
  82. Sofı́a Flores Solórzano. 2017. Desarrollo de un analizador automático de estados finitos para la lengua bribri. http://morphology.bribri.net/. ↗
  83. Miguel Soriano. 2018. Traductor Automático español –purépecha mediante OpenNMT. Master’s thesis, Universidad de Guadalajara, Mexico. find
  84. Marc Thouvenot. 2005. Gran diccionario náhuatl. http://www.gdn.unam.mx/. ↗
  85. Marc Thouvenot. 2011. Chachalaca en cen, juntamente. In Compendio Enciclopedico del Nahuatl, DVD. INAH. find
  86. Jörg Tiedemann. 2018. Emerging language spaces learned from massively multilingual corpora. arXiv preprint arXiv:1802.00273. — [arXiv:1802.00273](https://arxiv.org/abs/1802.00273) find
  87. Francis M Tyers, Mikel L Forcada, and Gema Ramırez-Sánchez. 2009. The Apertium machine translation platform: Five years on. In Proc. of the First Intl. Workshop on Free/Open-Source Rule-Based Machine Translation, pages 3–10.
  88. Alfonso Medina Urrea, José Abel Herrera Camacho, and Maribel Alvarado Garcıa. 2009. Towards the speech synthesis of raramuri: A unit selection approach based on unsupervised extraction of suffix sequences. Advances in Computational Linguistics, page 243. find
  89. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2016–2027, Vancouver, Canada, July. Association for Computational Linguistics. find
  90. Calderon Vilca, Hugo David, Cárdenas Mariñó, Flor Cagniy, and Edwin Fredy Mamani Calderon. 2012. Analizador morfológico de la lengua quechua basado en software libre helsinkifinite-statetransducer (hfst). COM-TEL. find
  91. Sami Virpioja, Jaakko J Väyrynen, Mathias Creutz, and Markus Sadeniemi. 2007. Morphology-aware statistical machine translation based on morphs induced in an unsupervised manner. Machine Translation Summit XI, 2007:491–498. find
  92. Claudio Wagner. 2016. Las lenguas indı́genas de américa (lenguas amerindias). Revista Documentos Lingüı́sticos y Literarios UACh, (17):30–37. find
  93. JCW Walters, M Mirten, P Hernández, E Pérez, and CH Upton. 2002. Diccionario náhuatl de los municipios de mecayapan y tatahuicapan de juárez. Veracruz. 2 a edición electrónica (cuerpo), Instituto Lingüı́stico de Verano, AC www. sil. org/mexico/nahuatl/istmo/G020a-DiccNahIst-NAU. htm. find
  94. H Christoph Wolfart and Francis Pardo. 1973. Computer-assisted linguistic analysis. University of Manitoba Anthropology Papers, (6). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mager-2018-challenges-language-technologies-indigenous-languages/