Elamur Research⌕
☰
home / papers / 2019 / Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Summary

Pre-trained encoders such as BERT transfer poorly when a task's training data is in one language and its test inputs are in another. Multilingual BERT and XLM partly address this, but XLM relies on a single cross-lingual objective, translation language modeling. Unicoder keeps XLM's architecture and shared BPE vocabulary but pre-trains on five tasks. Alongside masked and translation language modeling, it adds three new cross-lingual objectives, with training data taken from existing machine translation corpora:

  • Cross-lingual word recovery: an attention-based word alignment objective.
  • Cross-lingual paraphrase classification: detecting whether two sentences in different languages share a meaning.
  • Cross-lingual masked language model: masked language modeling over documents that alternate between languages.

The authors also propose Multi-language Fine-tuning, which trains jointly on source-language data and machine-translated data for all target languages. Against an XLM baseline, Unicoder gains 1.8 accuracy points on average across the 15 XNLI languages, a new state of the art. It gains 5.5 points on French and German in XQA, a cross-lingual question answering dataset the authors built. The work shows that richer cross-lingual supervision and multilingual fine-tuning improve zero-resource transfer, even for English.

Key points

  • Hard negatives for paraphrase classification: a lightweight DAN model trained on random negatives selects similar but incorrect translations. Positive and negative examples are balanced 1:1.
  • Cross-lingual documents: these are built from document-aligned MT corpora such as MultiUN and OpenSubtitles. Every even-indexed sentence is replaced by its translation, and the result is truncated to 256 tokens.
  • More fine-tuning languages generally help: fine-tuning on 15 languages always beats single-language TRANSLATE-TRAIN. Fine-tuning on English alone gives 85.1 on English XNLI with a 75.3 average across languages. Pairing English with French raises these to 85.4 and 76.8.
  • Pairwise effects vary: adding Vietnamese or Urdu to English lowers average accuracy. Vietnamese and Thai noticeably hurt English, while French and Spanish help it.
  • Transfer patterns across languages: Russian training data generalizes best, outperforming even English, and Urdu generalizes worst. Transfer among English, Spanish and French is easiest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1252

@inproceedings{huang2019unicoder,
  title = {{Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks}},
  author = {Huang, Haoyang and Liang, Yaobo and Duan, Nan and Gong, Ming and Shou, Linjun and Jiang, Daxin and Zhou, Ming},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1252},
  url = {https://doi.org/10.18653/v1/d19-1252}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Proceedings of the Seventh conference on International Language Resources and Evaluation, pages 2868–2872. European Language Resources Association (ELRA). find
  2. Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel M Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yunhsuan Sung, Brian Strope, et al. 2018. Effective parallel corpus mining using bilingual sentence embeddings. arXiv: Computation and Language, pages 165–176. find our card
  3. Dan Hendrycks and Kevin Gimpel. 2017. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv: Learning. find
  4. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L… find our card
  5. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. international conference on learning representations. find
  6. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925. find our card
  7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callisonburch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Corbett Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. The Association for Computational Linguistics. find
  8. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464. find our card
  9. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2017. The iit bombay english-hindi parallel corpus. language resources and evaluation. find our card
  10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  11. Pichuan Chang, Michel Galley, and Christopher D Manning. 2008. Optimizing chinese word segmentation for machine translation performance. pages 224–232. find our card
  12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  14. Andreas Eisele and Yu Chen. 2010. Multiun: A multilingual corpus from united nation documents. In Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. find
  15. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Hervé Jégou, et al. 2018. Word translation without parallel data. find
  16. Liyuan Liu, Jingbo Shang, Frank F Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. national conference on artificial intelligence, pages 5253–5260. find our card
  17. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv: Computation and Language. find
  18. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  19. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv: Computation and Language. find
  20. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. find
  21. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  22. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. ↗
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715–1725. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. pages 353–355. find our card
  26. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. north american chapter of the association for computational linguistics, 1:1112–1122. find
  27. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. arXiv preprint arXiv:1904.09077. find our card
  28. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2019-unicoder-universal-language-encoder-pre/