Elamur Research⌕
☰
home / papers / 2026 / Entropy-aware Masking for Masked Language Modeling

Entropy-aware Masking for Masked Language Modeling

Summary

Standard masked language modeling chooses which tokens to hide uniformly at random. This can waste training signal on tokens the model already predicts well. This short *SEM 2026 paper instead selects masks by entropy: the uncertainty of a model's predictive distribution at each position decides which 15% of tokens get masked. Entropy comes from one of two sources. In teacher-masking, it is computed by a fixed pretrained reference model. In self-masking, it is computed by the model under training, after a teacher-masking warm-up that avoids a cold start. The authors compare five selection rules by pretraining a 12-layer BERT and a 4-layer BERTlet on wikitext-103 with the MLM objective only, then fine-tuning on seven GLUE tasks. Masking the most uncertain tokens works best. Warm-started self-masking lifts the GLUE average by about 3–4 points over random masking, and combining it with knowledge distillation gives the strongest result. The method is cheap to adopt and especially helps small encoders. However, it has only been tested on English data and models up to about 110M parameters.

Key points

  • Models and setup: BERT (109M parameters) and BERTlet (28.8M parameters) were pretrained on wikitext-103 for 25 epochs and evaluated on CoLA, MRPC, QNLI, QQP, RTE, SST-2 and MNLI. Hugging Face BERT checkpoints served as reference models.
  • Teacher-masking: High-entropy selection raised the GLUE average from 73.27 to 75.02 for BERT and from 70.93 to 74.07 for BERTlet. Mid-entropy selection was harmful, dropping BERT to 57.56.
  • Self-masking:
    • Without a teacher warm-up, it collapsed (BERT: 57.41).
    • With the warm-up, it reached 76.47 for BERT and 75.15 for BERTlet.
    • With a frozen encoder, the BERTlet gain was still about 3%.
  • Scaling and distillation:
    • Pretraining BERTlet on bookcorpus gave 77.09, compared with 71.66 for a randomly masked baseline.
    • Adding knowledge distillation only after the warm-up scored 77.19, versus 76.04 when distillation was applied from the start.
  • Masking ratio: Raising the ratio to 40% brought no clear gain. Masking a single token per sequence cost about 1 GLUE point.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.27

@inproceedings{srinivasagan2026entropy,
  title = {{Entropy-aware Masking for Masked Language Modeling}},
  author = {Srinivasagan, Gokul and Hartung, Kai and Georges, Munir},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.27},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.27}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  2. Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train no evil: Selective masking for task-guided pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966–6974, Online. Association for Computational Lingui… find
  3. Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. 2022. Using selective masking as a bridge between pre-training and fine-tuning. arXiv preprint arXiv:2211.13815. — [arXiv:2211.13815](https://arxiv.org/abs/2211.13815) find
  4. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  5. Quentin Lhoest, Albert Villanova Del Moral, Yacine Jernite, Abhishek Thakur, Patrick Von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, and 1 others. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 conference on empiri… find our card
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  7. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. find our card
  8. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. find our card
  9. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, Online. Association for Computatio… find our card
  10. Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985– 3000, Dubrovnik, Croatia. Association for Computational Lingu… find
  11. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf… find our card
  12. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2):1–124. find our card
  13. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srinivasagan-2026-entropy-aware-masking-masked-language/