Elamur Research⌕
☰
home / papers / 2019 / Generalized Data Augmentation for Low-Resource Translation

Generalized Data Augmentation for Low-Resource Translation

Paper · 2019 · Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos et al.

Summary

Neural MT for low-resource languages (LRLs) is hampered by tiny parallel corpora. Ordinary back-translation also helps little here, because the reverse model is itself trained on scarce data. Xia et al. propose a unified augmentation framework that draws on every available resource: English monolingual text, parallel data from a closely related high-resource language (HRL, e.g. Turkish for Azerbaijani), and monolingual HRL/LRL corpora. Beyond back-translating English into the LRL, they also back-translate English into the HRL. Their main contribution is turning HRL–ENG data into pseudo-LRL–ENG data through a two-stage pivot. First, HRL words are replaced with LRL equivalents taken from a dictionary induced by aligning word embeddings. Second, a modified unsupervised MT system (M-UMT) rewrites these substituted sentences so they look more like genuine LRL text. On four low-resource language pairs under extreme data scarcity, the pivot-generated data outperforms supervised back-translation by about 1.5–8 BLEU. The method is cheap and independent of model architecture, which makes it useful whenever a minority language has a well-resourced relative.

Key points

  • Four augmentation routes are compared: ENG→LRL and ENG→HRL back-translation, HRL→LRL conversion of real HRL–ENG data, and ENG→HRL→LRL conversion of back-translated data.
  • The dictionary is learned by solving the Procrustes problem via SVD, using identical words as the seed lexicon and CSLS for similarity. Keeping only mutual nearest neighbours (bidirectional) generally beats unidirectional induction, with BEL as the exception.
  • Word substitution followed by M-UMT exceeds plain unsupervised HRL→LRL translation by more than 2–10 BLEU. Combining English monolingual data with HRL–ENG data adds up to 1.1 BLEU more.
  • The share of rare test words that augmentation "addresses" correlates strongly (Pearson) with BLEU. This suggests the gains come largely from better rare-word coverage.
  • Code: github.com/xiamengzhou/DataAugForLRL (ACL 2019, Carnegie Mellon University).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 166 works

Cite

Cite as Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos and Graham Neubig. 2019. Generalized Data Augmentation for Low-Resource Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1579

@inproceedings{xia2019generalized,
  title = {{Generalized Data Augmentation for Low-Resource Translation}},
  author = {Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos and Graham Neubig},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1579},
  url = {https://doi.org/10.18653/v1/p19-1579}
}

References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations.
  3. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. arXiv preprint arXiv:1902.01382. find
  4. Hany Hassan, Mostafa Elaraby, and Ahmed Tawfik. 2017. Synthetic data for neural machine translation of spoken-dialects. arXiv preprint arXiv:1707.00079. find
  5. Victor Chahuneau, Eva Schlinger, Noah A Smith, and Chris Dyer. 2013. Translating into morphologically rich languages with synthetic phrases. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1677–1687. find
  6. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational L… find
  7. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1925–1935. find
  8. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. pages 67–72.
  9. Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, and Wei Xu. 2017. Joint training for pivot-based neural machine translation. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3974–3980. find
  10. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 385–391.
  11. Trevor Cohn and Mirella Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 728–735.
  12. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. find
  13. Surafel M Lakew, Quintino F Lotito, Matteo Negri, Marco Turchi, and Marcello Federico. 2018. Improving zero-shot translation of low-resource languages. arXiv preprint arXiv:1811.01389. find
  14. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
  15. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In International Conference on Learning Representations. find
  16. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148–156.
  17. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018c. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049. find
  18. Adrià De Gispert and Jose B Marino. 2006. Catalan-english statistical machine translation without parallel corpus: bridging through spanish. In Proc. of 5th International Conference on Language Resources and Evaluation (LREC), pages 65–68. Citeseer. find
  19. Tomer Levinboim and David Chiang. 2015. Supervised phrase table triangulation with neural word embeddings for low-resource languages. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1079– 1083, Lisbon, Portugal. Association for Computational Linguisti… find
  20. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P…
  21. Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In The 57th Annual Meeting of the Association…
  22. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. arXiv preprint arXiv:1805.00879. — [arXiv:1805.00879](https://arxiv.org/abs/1805.00879) find
  23. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  24. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
  25. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proc. IJCNLP, volume 2, pages 296–301.
  26. Yuta Nishimura, Katsuhito Sudoh, Graham Neubig, and Satoshi Nakamura. 2018. Multi-source neural machine translation with data augmentation. find
  27. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua…
  28. Shuo Ren, Zhirui Zhang, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Unsupervised neural machine translation with SMT as posterior regularization. arXiv preprint arXiv:1901.04112. — [arXiv:1901.04112](https://arxiv.org/abs/1901.04112) find
  29. Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. 2019. Multilingual neural machine translation with soft decoupled encoding. In International Conference on Learning Representations. find
  30. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 369–379.
  31. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011, D…
  32. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1934– 1945. find
  33. Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation. arXiv preprint arXiv:1808.07894.
  34. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575. find
  35. Peter H Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10. find
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 86–96. find
  37. Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2018. Multiple-attribute text style transfer. CoRR, abs/1811.00552. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2019-generalized-data-augmentation-low-resource/