Elamur Research
☰
home / papers / 2019 / ERNIE: Enhanced Representation through Knowledge Integration

ERNIE: Enhanced Representation through Knowledge Integration

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.09223

Summary

Masked language models like BERT hide individual tokens, which for Chinese means single characters. A model can then often fill a gap from neighbouring characters inside the same word or name, without learning how entities relate to each other. ERNIE, developed at Baidu, changes what gets masked instead of adding explicit knowledge-graph embeddings. Pre-training runs in three stages: random character masking, then masking entire phrases located with segmentation and chunking tools, and finally masking complete named entities. The Transformer encoder is the same size as BERT-base. It is trained on a mixed Chinese corpus of encyclopedic, news and forum text. An additional Dialogue Language Model (DLM) objective uses query–response threads from the forum data. ERNIE outperforms BERT on all five Chinese benchmarks tested, usually by 0.5–2 points. In cloze probes it also recovers the correct entity names more often. The work shows that masking larger linguistic units is a cheap way to build factual knowledge into pretrained representations. Code and models were released in PaddlePaddle LARK.

Key points

  • Architecture: The encoder stacks 12 layers with a hidden size of 768 and uses 12 self-attention heads, the same configuration as BERT-base. The shared vocabulary has 17,964 Unicode characters, and traditional characters are converted to simplified.
  • Pre-training data (sentences): Chinese Wikipedia 21M, Baidu Baike 51M, Baidu news 47M and Baidu Tieba 54M. Tieba dialogues feed the DLM task, which uses dialogue embeddings to mark speaker roles across multiple turns.
  • Test-set gains over BERT:
    • XNLI: 77.2 → 78.4
    • LCQMC: 87.0 → 87.4
    • MSRA-NER F1: 92.6 → 93.8
    • ChnSentiCorp: 94.3 → 95.4
    • nlpcc-dbqa: MRR 94.6 → 95.1 and F1 80.8 → 82.7
  • Masking ablation (10% of the data, XNLI test): character masking alone gives 76.8%. Adding phrase masking raises this to 77.3%, and adding entity masking to 77.6%. Training on the full corpus reaches 78.4%.
  • Cloze probes: these cover entities such as the capital of Australia and the founder of relativity. BERT often produced non-words or copied names from the context, while ERNIE usually gave the correct entity. Where ERNIE was wrong, it still produced an entity of the right type, for example Melbourne instead of Canberra.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

227 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 213 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.09223 · arXiv 1904.09223

@misc{sun2019ernie,
  title = {{ERNIE: Enhanced Representation through Knowledge Integration}},
  author = {Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Chen, Xuyi and Zhang, Han and Tian, Xin and Zhu, Danxiang and Tian, Hao and Wu, Hua},
  year = {2019},
  doi = {10.48550/arxiv.1904.09223},
  eprint = {1904.09223},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.09223}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155. find
  2. Daniel Cer, Yinfei Yang, Sheng Yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, and Chris Tar. 2018. Universal sentence encoder. find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  4. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. find
  5. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  6. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  7. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. find our card
  8. Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. Lcqmc: A large-scale chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1952–1962. find
  9. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  12. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  13. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  16. Wilson L Taylor. 1953. cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  18. Ellen M Voorhees. 2001. Overview of the trec 2001 question answering track. In TREC, pages 42–51. find
  19. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  20. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  21. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun Hsuan Sung, and Brian Strope. 2018. Learning semantic textual similarity from conversations. find