Elamur Research
☰
home / papers / 2020 / Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages

Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.findings-emnlp.195.pdf

Summary

Machine translation still serves mainly a handful of well-resourced languages. The authors argue that the gap for African languages reflects social and historical inequities as well as a shortage of data, so MT researchers cannot close it alone. They map the MT pipeline onto roles (content creators, translators, curators, language technologists, evaluators, and stakeholders) and show that several of these roles are usually missing in low-resourced settings. Their remedy is participatory research, carried out through the Masakhane community, in which speakers of the languages build datasets, train models and judge the output. Many of these participants have no formal scientific training. The project produced new translation datasets, JW300-based benchmarks for more than 30 African target languages, and human post-editing evaluations for about a third of them on out-of-domain text (a COVID-19 survey and TED talks). These evaluations show that in-domain JW300 BLEU tends to overstate real quality. For many of these languages, this is the first human evaluation of MT, and the work offers a reusable template for community-driven NLP.

Key points

  • Benchmarks cover translation from English into 30+ African languages and are trained on JW300, a corpus of religious and missionary content. Data, models, code and evaluations are released at github.com/masakhane-io/masakhane-mt.
  • Eleven volunteers produced 707 evaluated translations in 10 days. The languages were Igbo, Nigerian Pidgin, Shona, Luo, Hausa (two annotators), Kiswahili, Yoruba, Fon and Dendi.
  • The authors chose post-editing over direct assessment and scored outputs with HTER, BLEU and ChrF against the corrected versions.
    • The test material was a 39-item COVID-19 survey plus Multitarget TED subsets of 30–120 sentences per language.
    • These subsets add languages to TED that had no prior translations.
  • JW300 BLEU overestimated quality for every evaluated language except Yoruba, and training-set size predicted generalization better.
    • Training data ranged from 6,937 pairs for Dendi to 875,558 for Kiswahili.
    • The two Hausa annotators showed modest agreement (Spearman ρ = 0.56).
  • Igbo and Yoruba have similar data sizes and JW300 scores but very different COVID results. The authors attribute this to dialect inconsistency in the Igbo data and outputs. Table 1 also shows how scarce online text is: Igbo has about 27M speakers but only 1,487 Wikipedia articles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.195

@inproceedings{nekoto2020participatory,
  title = {{Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages}},
  author = {Nekoto, Wilhelmina and Marivate, Vukosi and Matsila, Tshinondiwa and Fasubaa, Timi and Fagbohungbe, Taiwo and Akinola, Solomon Oluwole and Muhammad, Shamsuddeen and Kabenamualu, Salomon Kabongo and Osei, Salomey and Sackey, Freshia and Niyongabo, Rubungo Andre and Macharm, Ricky and Ogayo, Perez and Ahia, Orevaoghene and Berhe, Musie Meressa and Adeyemi, Mofetoluwa and Mokgesi-Selinga, Masabata and Okegbemi, Lawrence and Martinus, Laura and Tajudeen, Kolawole and Degila, Kevin and Ogueji, Kelechi and Siminyu, Kathleen and Kreutzer, Julia and Webster, Jason and Ali, Jamiil Toure and Abbott, Jade and Orife, Iroro and Ezeani, Ignatius and Dangana, Idris Abdulkadir and Kamper, Herman and Elsahar, Hady and Duru, Goodness and Kioko, Ghollah and Espoir, Murhabazi and Biljon, Elan van and Whitenack, Daniel and Onyefuluchi, Christopher and Emezue, Chris Chinenye and Dossou, Bonaventure F. P. and Sibanda, Blessing and Bassey, Blessing and Olabiyi, Ayodele and Ramkilowan, Arshath and Öktem, Alp and Akinfaderin, Adewale and Bashir, Abdallah},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.195},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.195}
}

References

68 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lishan Adam. 1997. Content and the web for african development. Journal of information science, 23(1):91–97. find
  2. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 937–947, Val… find
  3. Željko Agić and Ivan Vulić. 2019. JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204–3210, Florence, Italy. Association for Computational Linguistics. find our card
  4. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884… find our card
  5. Neville Alexander. 2009. Evolving african approaches to the management of linguistic diversity: The acalan project. Language Matters, 40(2):117–132. find
  6. Vamshi Ambati and Stephan Vogel. 2010. Can crowds build parallel corpora for machine translation systems? In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 62–65, Los Angeles. Association for Computational Linguistics. find
  7. Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. 2010. Active learning-based elicitation for semi-supervised word alignment. In Proceedings of the ACL 2010 Conference Short Papers, pages 365–370, Uppsala, Sweden. Association for Computational Linguistics. find
  8. Antonios Anastasopoulos and Graham Neubig. 2019. Should all cross-lingual embeddings speak english? find
  9. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs… find our card
  10. Amittai Axelrod, Diyi Yang, Rossana Cunha, Samira Shaikh, and Zeerak Waseem, editors. 2019. Proceedings of the 2019 Workshop on Widening NLP. Association for Computational Linguistics, Florence, Italy. find
  11. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translatio… find our card
  12. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  13. Luisa Bentivogli, Mauro Cettolo, Marcello Federico, and Federmann Christian. 2018. Machine translation human evaluation: an investigation of evaluation based on post-editing and its relation with direct assessment. In 15th International Workshop on Spoken Language Translation 2018, pages 62–69. find
  14. Andrew Caines. 2019. The geographic diversity of nlp conferences. find
  15. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), pages 261–268, Trento, Italy. find
  16. Colin Cherry, Greg Durrett, George Foster, Reza Haffari, Shahram Khadivi, Nanyun Peng, Xiang Ren, and Swabha Swayamdipta, editors. 2019. Proceedings of the 2nd Workshop on Prasad, Evan Crew, Chieu Nguyen, and Françoise Beaufays. 2019. Writing across the world’s languages: Deep internationalization… — [arXiv:1912.01218](https://arxiv.org/abs/1912.01218) find
  17. Deep Learning Approaches for Low-Resource NLP (DeepLo 2019). Association for Computational Linguistics, Hong Kong, China. find
  18. Christopher Cieri, Mike Maxwell, Stephanie Strassel, and Jennifer Tracey. 2016. Selection criteria for low resource language programs. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 4543–4549, Portorož, Slovenia. European Language Resour… find
  19. Guy De Pauw, Peter Waiganjo Wagacha, and Gilles-Maurice de Schryver. 2009. The SAWA corpus: A parallel corpus English - Swahili. In Proceedings of the First Workshop on Language Technologies for African Languages, pages 9– 16, Athens, Greece. Association for Computational Linguistics. find
  20. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon… find our card
  21. Bonaventure FP Dossou and Chris C Emezue. 2020. Ffr v1. 0: Fon-french neural machine translation. arXiv preprint arXiv:2003.12111. — [arXiv:2003.12111](https://arxiv.org/abs/2003.12111) find
  22. Kevin Duh. 2018. The multitarget ted talks task. http://www.cs.jhu.edu/˜kevinduh/a/ multitarget-tedtalks/ . ↗
  23. David M Eberhard, Gary F. Simons, and Charles D. Fenning. 2019. Ethnologue: Languages of the worlds. twenty-second edition. find
  24. Herman Engelbrecht. 2018. The deep learning indaba report. ACM SIGMultimedia Records, 9(3):5–5. find
  25. PB English, MJ Richardson, and Catalina Garzón-Galvis. 2018. From crowdsourcing to extreme citizen science: participatory research for environmental health. Annual review of public health, 39:335–350. find
  26. Daan van Esch, Elnaz Sarbar, Tamar Lucassen, Jeremy O’Brien, Theresa Breiner, Manasa Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the… find our card
  27. Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas. Associa… find
  28. ∀, Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp Öktem, Wole Akin, G… find
  29. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… find
  30. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long… find our card
  31. Adriana Guevara-Rukoz, Isin Demirsahin, Fei He, Shan-Hui Cathy Chu, Supheakmungkol Sarin, Knot Pipatsrisawat, Alexander Gutkin, Alena Butryna, and Oddur Kjartansson. 2020. Crowdsourcing Latin American Spanish for low-resource text-to-speech. In Proceedings of The 12th Language Resources and Evaluati… find
  32. James Jowi, Charles Ochieng Ong’ondo, and Mulu Nega. 2018. Building phd capacity in sub-saharan africa. find
  33. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali– english and sinhala–english. In Proceedings of the 2019 Conference on Empirical Meth… find our card
  34. Yunsu Kim, Yingbo Gao, and Hermann Ney. 2019. Effective cross-lingual transfer of neural machine translation models without shared vocabularies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1246–1257, Florence, Italy. Association for Computational… find
  35. Reza Haffari, Colin Cherry, George Foster, Shahram Khadivi, and Bahar Salehi, editors. 2018. Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP. Association for Computational Linguistics, Melbourne. find
  36. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080. find our card
  37. Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. 2019. Unsung challenges of building and deploying language technologies for low resource language communities. In Sixteenth Inte… find
  38. Pratik M. Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. Lucie-Aimée Kaffee, Hady ElSahar, Pavlos Vougiouklis, Christophe Gravier, Frédérique Laforest, Jonathon Hare, and Elena Simperl. 2018. Mind the (language) gap: generation of multilingual wikipedia summaries from… find
  39. Julia Kreutzer, Jasmijn Bastings, and Stefan Riezler. 2019. Joey NMT: A minimalist NMT toolkit for novices. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demon… find
  40. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210. find
  41. Vukosi Marivate, Tshephisho Sefara, Vongani Chabalala, Keamogetswe Makhaya, Tumisho Mokgonyane, Rethabile Mokoena, and Abiodun Modupe. 2020. Investigating an approach for low resource language dataset creation, curation and classification: Setswana and sepedi. arXiv preprint arXiv:2003.04986. find
  42. Laura Martinus, Jason Webster, Joanne Moonsamy, Moses Shaba Jnr, Ridha Moosa, and Robert Fairon. 2020. Neural machine translation for south africa’s official languages. arXiv preprint arXiv:2005.06609. — [arXiv:2005.06609](https://arxiv.org/abs/2005.06609) find
  43. Arya McCarthy. 2017. The new digital divide: Language is the impediment to information access. https://hilltopicssmu. wordpress.com/2017/04/08/the-new-digital-divide-language-is-the-impediment-to-information-access/ . find
  44. Cindy A. McKellar. 2014. An english to xitsonga statistical machine translation system for the government domain. In Proceedings of the 2014 PRASA, RobMech and AfLaT International Joint Symposium, pages 229–233. find
  45. Rajend Mesthrie. 1995. Language and social history: Studies in South African sociolinguistics. New Africa Books. find
  46. Alice Millour and Karën Fort. 2018. Toward a lightweight solution for less-resourced languages: Creating a POS tagger for alsatian using voluntary crowdsourcing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Langu… find
  47. Alp Öktem, Mirko Plitt, and Grace Tang. 2020. Tigrinya neural machine translation with transfer learning for humanitarian response. arXiv preprint arXiv:2003.11523. find
  48. Iroro Orife. 2020. Towards neural machine translation for edoid languages. arXiv preprint arXiv:2003.10704. — [arXiv:2003.10704](https://arxiv.org/abs/2003.10704) find
  49. Iroro Orife, David I Adelani, Timi Fasubaa, Victor Williamson, Wuraola Fisayo Oyewusi, Olamilekan Wahab, and Kola Tubosun. 2020. Improving yor\\ub\\’a diacritic restoration. arXiv preprint arXiv:2003.10564. — [arXiv:2003.10564](https://arxiv.org/abs/2003.10564) find
  50. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  51. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. find our card
  52. Maja Popović. 2015. chrF: character n-gram fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find
  53. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics. find our card
  54. Matt Post, Chris Callison-Burch, and Miles Osborne. 2012. Constructing parallel corpora for six Indian languages via crowdsourcing. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 401–409, Montréal, Canada. Association for Computational Linguistics. find
  55. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  56. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. find
  57. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. find our card
  58. Alex S Taylor, Siân Lindley, Tim Regan, David Sweeney, Vasillis Vlachokyriakos, Lillie Grainger, and Jessica Lingel. 2015. Data-in-place: Thinking through the relations between data and community. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 2863–287… find
  59. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214–2218. find
  60. Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1661– 1670, Berlin, Germany. Association for Com… find
  61. Elan Van Biljon, Arnu Pretorius, and Julia Kreutzer. 2020. On optimal transformer depth for low-resource language translation. arXiv preprint arXiv:2004.04418. — [arXiv:2004.04418](https://arxiv.org/abs/2004.04418) find
  62. Ngugi Wa Thiong’o. 1992. Decolonising the mind: The politics of language in African literature. East African Publishers. find
  63. Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. find
  64. Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  65. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual bert? arXiv preprint arXiv:2005.09093. find our card
  66. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Online. find our card
  67. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. find
  68. Language Domain Size Nigerian Pidgin COVID TED TED COVID TED COVID TED COVID TED COVID TED COVID COVID 39 100 30 39 80 78 120 39 50 39 55 39 39 Luo Yoruba Hausa Igbo Fon Swahili Shona Dendi *Figure 2: Education (a) and occupation (b) of a subset of 37 participants as indicated in a voluntary survey… find