Elamur Research⌕
☰
home / papers / 2020 / Pre-training via Paraphrasing

Pre-training via Paraphrasing

Summary

Almost all large-scale NLP pre-training relies on masked language modeling, where parts of a text are hidden and then restored. This paper presents MARGE (Multilingual Autoencoder that Retrieves and Generates), a sequence-to-sequence model that learns from a different signal. For each target document, MARGE finds related documents, often in other languages, and must regenerate the target using only those documents. A retrieval component scores how relevant each evidence document is to the target. Those scores bias the decoder's cross-attention, so the reconstruction loss trains both retrieval and generation together, starting from random weights. Pre-training uses news and Wikipedia, with metadata-based sharding to keep similarity search tractable. The resulting objective loosely resembles paraphrasing, translation, multi-document summarization and retrieval. Without any task-specific training, the model reaches up to 35.8 BLEU on document translation. After fine-tuning, it is competitive with masked models such as XLM-R across many discriminative and generative tasks and languages. This matters because it shows that masking is not the only workable pre-training objective, and that objectives closer to real tasks can enable zero-shot use.

Key points

  • Relevance scoring: relevance is the cosine similarity between document embeddings. These come from a 4-layer Transformer encoder whose weights are shared with the first four layers of the reconstruction encoder. Documents are chunks of up to 512 tokens.
  • Attention bias: the reconstruction model encodes each evidence document separately and concatenates the results. A trainable scalar β multiplies the relevance score, which is added to the decoder's cross-attention logits.
  • Data sharding: news articles are grouped by publication date. Wikipedia is split into 1000 shards, so that chunks of one article and its other-language versions land in the same shard.
  • Re-indexing: every 10k updates, pairwise similarities within sampled shards are recomputed and only the top-k pairs are kept.
  • Batching: batches are built greedily from the evidence–target graph, with cross-language links weighted 100 versus 1 for same-language links.
  • Results: zero-shot document translation reaches up to 35.8 BLEU, and cross-lingual summarization transfer beats strong baselines.
  • Encoder size: the encoder has 12 layers, dimension 1024, and feed-forward size 4096.
  • Uneven retrieval across languages: related languages retrieve each other more often (Bulgarian–Russian, Chinese–Japanese). Isolated languages such as Telugu, Thai and Indo-Iranian ones get few cross-lingual links.

Interesting ideas

  • I1. Retrieval as the noise source. Instead of corrupting text with masks, the model must rebuild a document from related but different texts. This idea could carry over to any domain with naturally overlapping documents, such as trade reports, news on one event, or descriptions of one place.
  • I2. Learning retrieval through attention bias. Adding a weighted relevance score to cross-attention logits makes the retriever trainable from the generation loss alone. This is cheaper than marginalizing over each retrieved document separately, and the trick is easy to reuse in other multi-source encoder-decoders.
  • I3. Using a random encoder at the start. A shared, randomly initialized encoder already places lexically overlapping texts near each other. That is enough to bootstrap the retrieval loop, which suggests self-training pipelines may not need a pre-trained retriever.
  • I4. Cheap metadata constraints. Grouping documents by date or by interlanguage links shrinks the search space and improves retrieval accuracy. This is a practical tip for anyone mining loosely parallel Ukrainian–other-language corpora.
  • I5. Diagnosing retrieval by language. Measuring how often each language retrieves documents in each other language reveals feedback loops, where weak languages get retrieved less and less. This diagnostic is worth running for any multilingual model that includes lower-resource languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

68 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 58 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.15020 · arXiv 2006.15020

@misc{lewis2020pre,
  title = {{Pre-training via Paraphrasing}},
  author = {Lewis, Mike and Ghazvininejad, Marjan and Ghosh, Gargi and Aghajanyan, Armen and Wang, Sida and Zettlemoyer, Luke},
  year = {2020},
  doi = {10.48550/arxiv.2006.15020},
  eprint = {2006.15020},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.15020}
}

References

38 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610, 2019. find our card
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041, 2017. ↗ our card
  3. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://www.aclweb.org/anthology/N19-1423. ↗
  6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197, 2019. ↗ our card
  7. Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv preprint arXiv:1711.05217, 2017. ↗ our card
  8. Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437–450, 2018. find our card
  9. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. ↗
  10. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080, 2020. ↗ our card
  11. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019. find our card
  12. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351, 2017. find
  13. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016. ↗ our card
  14. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. ↗
  15. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019. ↗
  16. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. ↗
  17. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043, 2017. ↗ our card
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019a. ↗ our card
  19. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475, 2019b. ↗ our card
  20. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401, 2020. ↗
  21. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. arXiv preprint arXiv:2002.11794, 2020. ↗
  22. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. ↗ our card
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. ↗
  24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210, 2020. ↗
  25. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305, 2017. find our card
  26. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hierarchical attention networks. arXiv preprint arXiv:1809.01576, 2018. ↗ our card
  27. Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771, 2018. ↗ our card
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. ↗
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. ↗ our card
  30. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327, 2020. ↗ our card
  31. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. Ccmatrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944, 2019. ↗ our card
  32. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Mlsum: The multilingual summarization corpus. arXiv preprint arXiv:2004.14900, 2020. ↗ our card
  33. Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. arXiv preprint arXiv:1909.00437, 2019. ↗ our card
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. find
  35. John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. arXiv preprint arXiv:1901.10444, 2019. ↗
  36. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv preprint arXiv:1908.11828, 2019a. ↗ our card
  37. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019b. ↗ our card
  38. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Using Comparable Corpora, pages 39–42, 2018. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/