Elamur Research⌕
☰
home / papers / 2019 / UER: An Open-Source Toolkit for Pre-training Models

UER: An Open-Source Toolkit for Pre-training Models

Paper · 2019 · Zhe Zhao, Hui Chen, Jinbin Zhang et al.

Summary

Pre-training models such as ELMO, GPT and BERT differ in their training corpus, encoder architecture, pre-training target and fine-tuning strategy. No single configuration suits every downstream task or compute budget, yet most open-source code covers only one or a few models. The authors present UER (Universal Encoder Representations), a modular PyTorch toolkit that builds models from interchangeable parts in four components: subencoder, encoder, target, and downstream fine-tuning. With a few lines of code, users can rebuild established models like BERT or combine modules into designs nobody has tried yet. Several targets can be weighted together, and distributed training is supported. Using UER, the team released a Chinese model zoo covering varied corpora, encoders and targets. Their BERT reimplementation performs on par with the HuggingFace and ERNIE implementations. Choosing in-domain corpora, suitable targets or lighter encoders can beat or nearly equal the BERT baseline. The toolkit therefore makes it cheaper to reproduce published models and to explore new pre-training choices.

Key points

  • Modules:
    • Encoders: LSTM, GRU, CNN, GatedCNN, AttentionNN, Transformer, and their stacks.
    • Targets: LM, MLM (cloze), autoencoder, NSP, NMT, and CLS.
    • Chinese subword features: radicals and pinyin, with each character treated as a word.
  • GLUE dev set: UER's BERT reimplementation matches HuggingFace overall. It is higher on RTE (66.8 vs 61.0) and QNLI (91.5 vs 89.0) and slightly lower on SST-2 (92.4 vs 93.0).
  • ERNIE benchmark: UER is comparable to ERNIE and better on MSRA-NER (93.6 vs 92.6).
  • Amazon review corpus pre-training: Starting from BERT-base Chinese, further pre-training with the CLS target raised ChnSentiCorp from 94.3 to 95.8 and Douban from 87.5 to 88.3.
  • Lighter encoder: A pre-trained 2-layer LSTM (embedding size 512, hidden size 1024) reached 96.9 on Online shopping and 94.5 on ChnSentiCorp, slightly above BERT (96.3 and 94.3). It scored 86.5 on Douban versus BERT's 87.5. Code and model zoo: github.com/dbiir/UER-py.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢TencentShenzhen · China134 works · 19,743 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju and Xiaoyong Du. 2019. UER: An Open-Source Toolkit for Pre-training Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-3041

@inproceedings{zhao2019uer,
  title = {{UER: An Open-Source Toolkit for Pre-training Models}},
  author = {Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju and Xiaoyong Du},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3041},
  url = {https://doi.org/10.18653/v1/d19-3041}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  3. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. EMNLP.
  5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
  6. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. find
  7. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  9. Jie Yang and Yue Zhang. 2018. Ncrf++: An open-source neural sequence labeling toolkit. arXiv preprint arXiv:1806.05626.
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8). find
  11. Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in chinese, english, japanese and korean? find
  12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. find
  13. Zhe Zhao, Tao Liu, Shen Li, Bofang Li, and Xiaoyong Du. 2017. Ngram2vec: Learning improved word representations from ngram co-occurrence statistics. EMNLP. find
  14. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
  15. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In ACL., volume 2.
  16. Yoon Kim. 2014. Convolutional neural networks for sentence classification. EMNLP.
  17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2019-uer-open-source-toolkit-pre/