Elamur Research
☰
home / papers / 2019 / Pre-Training with Whole Word Masking for Chinese BERT

Pre-Training with Whole Word Masking for Chinese BERT

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.08101

Summary

Most strong pre-trained language models were built for English, and training them is expensive. This leaves Chinese NLP without solid, reusable baselines. The authors address this in two ways. First, they adapt whole word masking (wwm) to Chinese. Because WordPiece does not break Chinese text into sub-character pieces, they use the LTP segmenter to find word boundaries and then mask every character of a selected word together. Input tokenization stays the same as in the original BERT. Second, they introduce MacBERT, a RoBERTa-style model whose pre-training task is MLM as correction (Mac). Instead of inserting an artificial [MASK] symbol, Mac substitutes a similar word, so pre-training inputs look more like the text seen during fine-tuning. MacBERT also combines wwm with N-gram masking and uses sentence-order prediction. The team trains a family of Chinese baselines (BERT, RoBERTa, ELECTRA and compact RBT models) and evaluates them on ten Chinese tasks, from sentence-level to document-level. MacBERT reaches state-of-the-art scores on many of these tasks. All models are publicly released, which makes them a practical starting point for Chinese and other non-English NLP.

Key points

  • Chinese wwm uses LTP word segmentation to choose masking units. Masking percentage and other settings follow the original wwm code unchanged.
  • MacBERT's Mac objective swaps masked words for similar words rather than [MASK], aiming to reduce the mismatch between pre-training and fine-tuning. It also uses WWM/N-gram masking and SOP instead of NSP.
  • The released models include BERT-wwm, RoBERTa-wwm, ELECTRA, MacBERT and the small RBT series, all available at github.com/ymcui/Chinese-BERT-wwm.
  • Evaluation covers ten Chinese NLP datasets, including machine reading comprehension and text classification, along with ablations explaining where the gains come from.
  • All models are derived from BERT or ELECTRA and keep the same input format, so they can be swapped for one another during fine-tuning without code changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

303 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 292 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/taslp.2021.3124365 · arXiv 1906.08101

@misc{cui2019pre,
  title = {{Pre-Training with Whole Word Masking for Chinese BERT}},
  author = {Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing},
  year = {2019},
  doi = {10.1109/taslp.2021.3124365},
  eprint = {1906.08101},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/taslp.2021.3124365}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Cui, W. Che, T. Liu, B. Qin, Z. Yang, S. Wang, and G. Hu, “Pre-training with whole word masking for chinese bert,” arXiv preprint arXiv:1906.08101, 2019. — [arXiv:1906.08101](https://arxiv.org/abs/1906.08101) find
  2. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find
  3. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 784–7… find our card
  4. S. Reddy, D. Chen, and C. D. Manning, “Coqa: A conversational question answering challenge,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 249–266, 2019. find our card
  5. E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer, “QuAC: Question answering in context,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp.… find our card
  6. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: a benchmark for question answering research,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019. find our card
  7. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “Race: Large-scale reading comprehension dataset from examinations,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017, pp. 796– 805. [Online]. Available: http://www… find
  8. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul.… find
  9. S. Zhang, H. Zhao, Y. Wu, Z. Zhang, X. Zhou, and X. Zhou, “Dual co-matching network for multi-choice reading comprehension,” arXiv preprint arXiv:1901.09381, 2019. — [arXiv:1901.09381](https://arxiv.org/abs/1901.09381) find
  10. Q. Ran, P. Li, W. Hu, and J. Zhou, “Option comparison network for multiple-choice reading comprehension,” arXiv preprint arXiv:1903.03033, 2019. — [arXiv:1903.03033](https://arxiv.org/abs/1903.03033) find
  11. Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, X. Tian, D. Zhu, H. Tian, and H. Wu, “Ernie: Enhanced representation through knowledge integration,” arXiv preprint arXiv:1904.09223, 2019. find our card
  12. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” arXiv preprint arXiv:1906.08237, 2019. find our card
  13. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. find
  14. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,” arXiv preprint arXiv:1907.10529, 2019. find our card
  15. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942, 2019. find
  16. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “ELECTRA: Pre-training text encoders as discriminators rather than generators,” in ICLR, 2020. [Online]. Available: https://openreview.net/pdf?id=r1xMH1BtvB ↗
  17. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017, pp. 5998–6008. find
  18. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016. find
  19. L. Kong, C. de Masson d’Autume, L. Yu, W. Ling, Z. Dai, and D. Yogatama, “A mutual information maximization perspective of language representation learning,” in International Conference on Learning Representations, 2020. [Online]. Available: https: //openreview.net/forum?id=Syx79eBKwr find
  20. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc.,… find
  21. W. Che, Z. Li, and T. Liu, “Ltp: A chinese language technology platform,” in Proceedings of the 23rd International Conference on Computational Linguistics: Demonstrations. Association for Computational Linguistics, 2010, pp. 13–16. find
  22. Y. Levine, B. Lenz, O. Lieber, O. Abend, K. Leyton-Brown, M. Tennenholtz, and Y. Shoham, “{PMI}-masking: Principled masking of correlated spans,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=3Aoft6NWFej ↗
  23. H. Wang and Y. Hu, “Synonyms,” 2017. [Online]. Available: https://github.com/huyingxi/Synonyms ↗
  24. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates,… find
  25. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. find
  26. Y. You, J. Li, J. Hseu, X. Song, J. Demmel, and C.-J. Hsieh, “Reducing bert pre-training time from 3 days to 76 minutes,” arXiv preprint arXiv:1904.00962, 2019. find
  27. Y. Cui, T. Liu, W. Che, L. Xiao, Z. Chen, W. Ma, S. Wang, and G. Hu, “A Span-Extraction Dataset for Chinese Machine Reading Comprehension,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  28. C. C. Shao, T. Liu, Y. Lai, Y. Tseng, and S. Tsai, “Drcd: a chinese machine reading comprehension dataset,” arXiv preprint arXiv:1806.00920, 2018. find
  29. X. Duan, B. Wang, Z. Wang, W. Ma, Y. Cui, D. Wu, S. Wang, T. Liu, T. Huo, Z. Hu et al., “Cjrc: A reliable human-annotated benchmark dataset for chinese judicial reading comprehension,” in China National Conference on Chinese Computational Linguistics. Springer, 2019, pp. 439–451. find
  30. S. Tan and J. Zhang, “An empirical study of sentiment analysis for chinese documents,” Expert Systems with applications, vol. 34, no. 4, pp. 2622–2629, 2008. find
  31. J. Li and M. Sun, “Scalable term selection for text categorization,” in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007. find
  32. L. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan, “CLUE: A Chinese lang… find our card
  33. A. Conneau, R. Rinott, G. Lample, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “Xnli: Evaluating cross-lingual sentence representations,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. find
  34. X. Liu, Q. Chen, C. Deng, H. Zeng, J. Chen, D. Li, and B. Tang, “Lcqmc: A large-scale chinese question matching corpus,” in Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 1952–1962. find
  35. J. Chen, Q. Chen, X. Liu, H. Yang, D. Lu, and B. Tang, “The BQ corpus: A large-scale domain-specific Chinese corpus for sentence semantic equivalence identification,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computa… find
  36. H. Hu, K. Richardson, L. Xu, L. Li, S. Kuebler, and L. Moss, “Ocnli: Original chinese natural language inference,” in Findings of EMNLP, 2020. [Online]. Available: https://arxiv.org/abs/2010.05444 ↗
  37. Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, “Ernie 2.0: A continual pre-training framework for language understanding,” arXiv preprint arXiv:1907.12412, 2019. find our card
  38. J. Wei, X. Ren, X. Li, W. Huang, Y. Liao, Y. Wang, J. Lin, X. Jiang, X. Chen, and Q. Liu, “NEZHA: Neural Contextualized Representation for Chinese Language Understanding,” arXiv e-prints, p. arXiv:1909.00204, Aug. 2019. — [arXiv:1909.00204](https://arxiv.org/abs/1909.00204) find
  39. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016, pp. 265–283. find
  40. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2016, pp. 2383– 2392. [Online]. Available: http://www… find our card
  41. Y.-H. Tseng, L.-H. Lee, L.-P. Chang, and H.-H. Chen, “Introduction to SIGHAN 2015 bake-off for Chinese spelling check,” in Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing. Beijing, China: Association for Computational Linguistics, Jul. 2015, pp. 32–37. [Online]. Available: h… find