Elamur Research
☰
home / papers / 2022 / Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.tacl-1.9.pdf

Summary

Indic languages have more than a billion speakers, yet machine translation for them has lagged because little parallel text is publicly available. Existing web-mined collections such as CCMatrix cover only a few of these languages. The authors built Samanantar, a parallel corpus pairing English with 11 Indic languages from two language families. First, they gathered and deduplicated existing public datasets, including OPUS collections, PMIndia and CVIT-PIB. Second, they mined new sentence pairs from three kinds of sources: scanned documents processed with OCR, multilingual news and education websites, and the monolingual IndicCorp collection. Alignment relied on multilingual sentence embeddings (LaBSE) and approximate nearest neighbor search. Human annotators judged the mined pairs to be of high quality. The authors also derived Indic-to-Indic pairs by pivoting through English. They trained a multilingual model, IndicTrans, on the data. It beat every open-source system tested and, on many benchmarks, beat Google and Microsoft. This makes Samanantar a key open resource for Indic NMT.

Key points

  • Samanantar contains about 49.7M English–Indic sentence pairs. Of these, 12.4M come from existing sources and 37.4M were newly mined, roughly a fourfold increase. Sizes range from 141K pairs (en-as) to 10.1M pairs (en-hi).
  • The corpus includes Hindi (hi), Bengali (bn), Marathi (mr), Gujarati (gu), Punjabi (pa), Odia (or), Assamese (as), Tamil (ta), Telugu (te), Kannada (kn) and Malayalam (ml). IndicCorp was the largest single source of the mined data, at about two-thirds.
  • Pivoting through English produced 83.4M sentence pairs across all 55 Indic–Indic language pairs.
  • For quality checks, 38 annotators rated 9,566 sentence pairs. These cross-lingual similarity judgments were released together with the corpus.
  • IndicTrans outperformed OPUS-MT, mBART50 and CVIT-ILMulti. It still trails the commercial systems on FLORES. Its largest gains appear for low-resource languages such as or and kn, especially when translating into English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

166 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 161 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00452

@article{ramesh2022samanantar,
  title = {{Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages}},
  author = {Ramesh, Gowtham and Doddapaneni, Sumanth and Bheemaraj, Aravinth and Jobanputra, Mayank and AK, Raghavan and Sharma, Ajitesh and Sahoo, Sujit and Diddee, Harshita and J, Mahalakshmi and Kakwani, Divyanshu and Kumar, Navneet and Pradeep, Aswin and Nagaraj, Srihari and Deepak, Kumar and Raghavan, Vivek and Kunchukuttan, Anoop and Kumar, Pratyush and Khapra, Mitesh Shantadevi},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00452},
  url = {https://doi.org/10.1162/tacl_a_00452}
}

References

62 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anvita Abbi. 2012. Languages of India and India and as a Linguistic Area. http://www .andamanese.net/LanguagesofIndiaand Indiaasalinguisticarea.pdf. Retrieved November 15, 2015. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find our card
  3. Željko Agić and Ivan Vulić. 2019. JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204–3210. Florence, Italy. Association for Computational Linguistics. find our card
  4. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval… find our card
  5. Loı̈c Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, S… find our card
  6. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federman… find our card
  7. Resources and Evaluation, 49(2):375–395. https://doi.org/10.1007/s10579-014 -9287-y, PubMed: 26321896 — [doi:10.1007/s10579-014](https://doi.org/10.1007/s10579-014) ↗
  8. Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. 2017. An empirical study of language relatedness for transfer learning in neural machine translation. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pages 282–286. The National University (Phillippines). find
  9. Yoshua Bengio, J. Louradour, Ronan Collobert, and J. Weston. 2009. Curriculum learning. In ICML ’09. https://doi.org/10.1145 /1553374.1553380 find our card
  10. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural LanguageProcessing(EMNLP), pages 5960–5969, Online. Association for Computational… find our card
  11. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the… find our card
  12. Murray B. Emeneau. 1956. India as a lingustic area. Language. https://doi.org/10 .2307/410649 find
  13. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. find our card
  14. Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoı̂t Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedr… find our card
  15. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875,… find our card
  16. Markus Freitag and Orhan Firat. 2020. Complete multilingual neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 550–560, Online. Association for Computational Linguistics. find
  17. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. CoRR, abs/2106.03193. find
  18. Vikrant Goyal, Anoop Kunchukuttan, Rahul Kejriwal, Siddharth Jain, and Amit Bhagwat. 2020. Contact relatedness can help improve Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: The bible in 100 languages. Language multilingual NMT: Microsoft STCI-MT @ WMT20. In Proc… find
  19. Papers, pages 888–895, Belgium, Brussels. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18 -6478 — [doi:10.18653/v1/w18](https://doi.org/10.18653/v1/w18) ↗
  20. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning. find
  21. Herve Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128. https://doi.org/10.1109 /TPAMI.2010.57, PubMed: 21088323 find
  22. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English. In Proceedings of the 2019 Conference on Empirical Metho… find our card
  23. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computation… find
  24. Tom Kocmi and Ondřej Bojar. 2018. Trivial transfer learning for low-resource neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 244–252, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18 -6325 ↗
  25. Barry Haddow and Faheem Kirefu. 2020. Pmindia – a collection of parallel corpora of languages of India. find
  26. Tahmid Hasan, Abhik Bhattacharjee, Kazi Samin, Masum Hasan, Madhusudan Basak, M. Sohel Rahman, and Rifat Shahriyar. 2020. Not low-resource anymore: Aligner ensembling, batch filtering, and new datasets for Bengali-English machine translation. https://doi.org/10 .18653/v1/2020.emnlp-main.207 find
  27. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17 -3204 ↗
  28. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017a. Billion-scale similarity search with GPUS. arXiv preprint arXiv:1702.08734. find our card
  29. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017b. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the A… find
  30. Anoop Kunchukuttan. 2020. The IndicNLP Library. https://github.com/anoopkunchukuttan /indic nlp library/blob/master/docs /indicnlp.pdf. ↗
  31. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. find
  32. Pierre Lison and Jörg Tiedemann. 2016. Open-Subtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of no… find
  33. Jerin Philip, Shashank Siripragada, Vinay P. Namboodiri, and C. V. Jawahar. 2020. Revisiting low resource status of Indian languages in machine translation. 8th ACM IKDD CODS and 26th COMAD. https://doi.org/10 .1145/3430984.3431026 find
  34. Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, Sadao Oda, and Yusuke Kurohashi. 2021. Overview of the 8th workshop on Asian translation… find
  35. Matt Post, Chris Callison-Burch, and Miles Osborne. 2012. Constructing parallel corpora for six indian languages via crowdsourcing. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 401–409, Montréal, Canada. Association for Computational Linguistics. find
  36. Loganathan Ramasamy, Ondřej Bojar, and Zdeněk Žabokrtský. 2012. Morphological processing for English-Tamil statistical machine translation. In Proceedings of the Workshop on Machine Translation and Parsing in Indian Languages (MTPIL-2012), pages 113–122. find
  37. Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, and Sadao Kurohashi. 2020. Overview of the 7th workshop on Asian translation. In Proceedings of the 7th Workshop on Asian Transl… find
  38. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813. https://doi.org/10 .18653/v1/2020.emnlp-main.365 find our card
  39. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In International Joint Conference on Natural Language Processing. find our card
  40. Annette Rios, Mathias Müller, and Rico Sennrich. 2020. Subword segmentation and a single bridge language affect zero-shot neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 528–537, Online. Association for Computational Linguistics. find
  41. Pedro Javier Ortiz Suarez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora, pages 9–16, Mannheim. Leibniz-Institut für Deutsche Spra… find
  42. H. Riza, M. Purwoadi, Gunarso, T. Uliniansyah, A. A. Ti, S. M. Aljunied, L. C. Mai, V. T. Thang, N. P. Thai, V. Chea, R. Sun, S. Sam, S. Seng, K. M. Soe, K. T. Nwet, M. Utiyama, and C. Ding. 2016. Introduction of the Asian language treebank. In 2016 Conference of The Oriental Chapter of Internationa… — [doi:10.1109/icsda.2016.7918974](https://doi.org/10.1109/icsda.2016.7918974) find
  43. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  44. Shantipriya Parida, Satya Ranjan Dash, Ondřej Bojar, Petr Motlicek, Priyanka Pattnaik, and Debasish Kumar Mallick. 2020. OdiEnCorp 2.0: Odia-English parallel corpus for machine translation. In Proceedings of the WILDRE5– 5th Workshop on Indian Language Data: Resources and Evaluation, pages 14–19, M… find
  45. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019a. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from Wikipedia. find our card
  46. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2019b. Ccmatrix: Mining billions of high-quality parallel sentences on the WEB. CoRR, abs/1911.04944. find our card
  47. Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  48. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2020. Ccmatrix: Mining billions of high-quality parallel sentences on the Web. https://doi.org/10.18653/v1 /2021.acl-long.507 find our card
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… find
  50. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pre-training and finetuning. find
  51. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Li… find
  52. Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1342–1348,… find
  53. Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 175–182, Riga, Latvia. Northern European Association for Language Technology (NEALT). find
  54. Jörg Tiedemann and Santhosh Thottingal. 2020a. OPUS-MT -- building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisbon, Portugal. European Association for Machine Translation. find
  55. P. Shah and V. Bakrola. 2019. Neural machine translation system of indic languages—an attention based approach. In 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP), pages 1–5. https://doi.org /10.1109/ICACCP.2019.8882969 — [doi:10.1109/icaccp.2019.8882969](https://doi.org/10.1109/icaccp.2019.8882969) ↗
  56. Jörg Tiedemann and Santhosh Thottingal. 2020b. OPUS-MT—Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT). Lisbon, Portugal. find
  57. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12). Istanbul, Turkey. European Language Resources Association (ELRA). find
  58. Kārumūri V. Subbārāo. 2012. South Asian Languages: A Syntactic Typology. Cambridge University Press. https://doi.org/10 .1017/CBO9781139003575 find
  59. Suhas Jayaram Subramanya, F. Devvrit, H. V. Simhadri, R. Krishnawamy, and R. Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node. In Advances in Neural Information Processing Systems, volume 32, pages 13771–13781. find
  60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  61. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  62. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. find