Elamur Research
☰
home / papers / 2019 / ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1907.12412

Summary

Pre-trained language models such as BERT and XLNet mostly learn from word and sentence co-occurrence. They leave aside other signals in raw text, including named entities, sentence structure and discourse relations. The Baidu authors propose ERNIE 2.0, a framework that lets new self-supervised or weakly supervised pre-training tasks be added over time. Each time a task is introduced, the model starts from its earlier parameters and trains the new task jointly with the previous ones. This "continual multi-task learning" spreads a fixed budget of iterations per task across training stages, which helps the model avoid losing what it learned before. All tasks share one Transformer encoder, and a task embedding is added to the input. Each task uses either a token-level or a sentence-level loss. The model was pre-trained on encyclopedia, book, news, dialog, search-relevance and discourse data, then fine-tuned on English GLUE and nine Chinese benchmarks. It beat BERT on all of them and generally outperformed XLNet. The work shows that piling up diverse auxiliary objectives, rather than relying on a few, yields stronger general-purpose representations.

Key points

  • Seven pre-training tasks are grouped into three families:
    • word-aware: Knowledge Masking, Capital Prediction, Token-Document Relation
    • structure-aware: Sentence Reordering, Sentence Distance
    • semantic-aware: Discourse Relation, IR Relevance
  • On the GLUE test set, ERNIE 2.0 LARGE scores 83.6, which is 3.1% above BERT LARGE. ERNIE 2.0 BASE reaches 80.6 and beats BERT BASE on all 10 tasks. On dev, LARGE tops XLNet LARGE everywhere except MNLI-m.
  • For Chinese, the model is tested on nine datasets: LCQMC, BQ Corpus, NLPCC-DBQA, ChnSentiCorp, XNLI, MSRA-NER, DuReader, DRCD and CMRC 2018. ERNIE 2.0 beats BERT BASE on all nine, and the LARGE version sets new state-of-the-art results.
  • An ablation compared training strategies under equal budgets (4 tasks, 50k steps each, base-size model):
    • Continual multi-task learning scored best, with 79.0 on MNLI-m, 87.8 on SST-2 and 84.0 on MRPC.
    • Plain sequential continual learning was weakest, which is consistent with forgetting.
  • Code and pre-trained models are released at github.com/PaddlePaddle/ERNIE. The paper was published at AAAI 2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis

Cited by

211 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 202 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.12412 · arXiv 1907.12412

@misc{sun2019ernie,
  title = {{ERNIE 2.0: A Continual Pre-training Framework for Language Understanding}},
  author = {Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Tian, Hao and Wu, Hua and Wang, Haifeng},
  year = {2019},
  doi = {10.48550/arxiv.1907.12412},
  eprint = {1907.12412},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.12412}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. find
  2. Chen, Z., and Liu, B. 2018. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 12(3):1–207. find
  3. Chen, J.; Chen, Q.; Liu, X.; Yang, H.; Lu, D.; and Tang, B. 2018. The bq corpus: A large-scale domain-specific chinese corpus for sentence semantic equivalence identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4946– 4951. find
  4. Conneau, A.; Lample, G.; Rinott, R.; Williams, A.; Bowman, S. R.; Schwenk, H.; and Stoyanov, V. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. find
  5. Cui, Y.; Liu, T.; Xiao, L.; Chen, Z.; Ma, W.; Che, W.; Wang, S.; and Hu, G. 2018. A span-extraction dataset for chinese machine reading comprehension. CoRR abs/1810.07366. find
  6. Dai, Z.; Yang, Z.; Yang, Y.; Cohen, W. W.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. find
  7. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. He, W.; Liu, K.; Liu, J.; Lyu, Y.; Zhao, S.; Xiao, X.; Liu, Y.; Wang, Y.; Wu, H.; She, Q.; et al. 2017. Dureader: a chinese machine reading comprehension dataset from real-world applications. arXiv preprint arXiv:1711.05073. find our card
  9. Lample, G., and Conneau, A. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  10. Levow, G.-A. 2006. The third international chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, 108– 117. find
  11. Liu, X.; Chen, Q.; Deng, C.; Zeng, H.; Chen, J.; Li, D.; and Tang, B. 2018. Lcqmc: A large-scale chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, 1952–1962. find
  12. Liu, X.; He, P.; Chen, W.; and Gao, J. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. find our card
  13. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  14. Parisi, G. I.; Kemker, R.; Part, J. L.; Kanan, C.; and Wermter, S. 2019. Continual lifelong learning with neural networks: A review. Neural Networks. find our card
  15. Pennington, J.; Socher, R.; and Manning, C. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1532–1543. find
  16. Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  17. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  18. Sang, E. F., and De Meulder, F. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. find
  19. Shao, C. C.; Liu, T.; Lai, Y.; Tseng, Y.; and Tsai, S. 2018. Drcd: a chinese machine reading comprehension dataset. arXiv preprint arXiv:1806.00920. find
  20. Sileo, D.; Van-De-Cruys, T.; Pradel, C.; and Muller, P. 2019. Mining discourse markers for unsupervised sentence representation learning. arXiv preprint arXiv:1903.11850. find
  21. Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A.; and Potts, C. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 1631–1642. find
  22. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; and Wu, H. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. find our card
  23. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008. find
  24. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  25. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card