Elamur Research
☰
home / papers / 2020 / Masked Language Model Scoring

Masked Language Model Scoring

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.240.pdf

Summary

Masked language models such as BERT are usually finetuned before use, and they lack an obvious way to assign probabilities to whole sentences. This makes them seem unsuitable for tasks that rank candidate outputs, unlike left-to-right models like GPT-2. Salazar et al. address this with pseudo-log-likelihood (PLL). Each token in a sentence is hidden in turn, the model predicts it from the surrounding context on both sides, and the log probabilities of these predictions are added together. The authors use PLLs from pretrained BERT, RoBERTa and XLM to rerank n-best hypotheses from end-to-end speech recognition (LibriSpeech) and neural translation (TED Talks, IWSLT 2015, WMT 2014). They also test grammaticality judgments on BLiMP without any supervision. In addition, they define pseudo-perplexity for evaluating a model on a corpus, and they show that a finetuned model can estimate PLL in one forward pass instead of one pass per token. The masked models match or beat GPT-2, which suggests that pretrained MLMs can be dropped into existing systems as sentence scorers.

Key points

  • Speech recognition: Rescoring with RoBERTa large lowers WER by about 30% relative on LibriSpeech test-clean and 18% on test-other. Domain adaptation brings further gains.
  • Translation: Low-resource TED Talks pairs, such as Galician→English with roughly 10k sentence pairs, improve by up to +1.7 BLEU. A single 15-language XLM improves translations into several target languages at once.
  • Grammaticality (BLiMP): BERT and RoBERTa beat GPT-2 by up to 3.9% absolute, with roughly +10 points on island effects and NPI licensing.
  • Faster scoring: Regression on the [CLS] token lets a model approximate PLL without masking. This replaces the cost of one pass per token with a single pass.
  • Code: The scoring library is released at github.com/awslabs/mlm-scoring.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

253 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 240 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.240

@inproceedings{salazar2020masked,
  title = {{Masked Language Model Scoring}},
  author = {Salazar, Julian and Liang, Davis and Nguyen, Toan Q. and Kirchhoff, Katrin},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.240},
  url = {https://doi.org/10.18653/v1/2020.acl-main.240}
}

References

58 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In NAACL-HLT. find our card
  2. Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. 2015. Bidirectional recurrent neural network language models for automatic speech recognition. In ICASSP. find
  3. Julian Besag. 1975. Statistical analysis of non-lattice data. The Statistician, 24(3):179–195. find
  4. Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  5. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, R Cattoni, and Marcello Federico. 2015. find
  6. The IWSLT 2015 evaluation campaign. Technical report, FBK and KIT. find
  7. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP. find
  8. Xie Chen, Anton Ragni, Xunying Liu, and Mark JF Gales. 2017. Investigating bidirectional recurrent neural network language models for speech recognition. In INTERSPEECH. find
  9. Noam Chomsky. 1957. Syntactic structures. Mouton. find
  10. Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019. On the use of BERT for neural machine translation. In WNGT. find
  11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. Deep Learning Workshop, NeurIPS. find our card
  12. Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018. Large margin neural language model. In EMNLP. find
  13. Frederick Jelinek, Lalit Bahl, and Robert Mercer. 1975. Design of a linguistic statistical decoder for the recognition of continuous speech. IEEE Trans. Inf. Theory, 21(3):250–256. find
  14. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In EMNLP. find our card
  15. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. find
  16. Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. 2019. Jasper: An end-to-end convolutional neural acoustic model. In INTERSPEECH. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  18. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In WMT. find
  19. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In EMNLP. find our card
  20. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS. find
  21. Toan Q Nguyen and David Chiang. 2018. Improving lexical choice in neural machine translation. In NAACL-HLT. find
  22. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL. find
  23. Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT. find
  24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  25. Andrew Finch and Eiichiro Sumita. 2009. Bidirectional phrase-based statistical machine translation. In EMNLP. find
  26. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085. find our card
  27. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. 2018. Parallel WaveNet: Fast high-fidelity speech synthesis. In ICML. find
  28. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. find our card
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In ICASSP. find
  30. Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, et al. 2020. GluonCV and GluonNLP: Deep learning in computer vision and natural language processing. JMLR, 21(23):1–7. find
  31. Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In NAACL-HLT. find our card
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find
  33. Anton Ragni, Edgar Dakin, Xie Chen, Mark JF Gales, and Kate M Knill. 2016. Multi-language neural network language models. In INTERSPEECH. find
  34. Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In EMNLP. find
  35. Brian Roark, Murat Saraclar, Michael Collins, and Mark Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In ACL. find
  36. Carson T Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Language Science Press. find
  37. Yangyang Shi, Martha Larson, Pascal Wiggers, and Catholijn M Jonker. 2013. Exploiting the succeeding words in recurrent neural network language models. In INTERSPEECH. find
  38. Joongbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using BERT for speech recognition. In ACML. find
  39. Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold fusion: Training seq2seq models together with language models. In INTER-SPEECH. find
  40. Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018. Simple fusion: Return of the language model. In WMT. find
  41. Chenguang Wang, Mu Li, and Alexander J Smola. 2019. Language models with transformers. arXiv preprint arXiv:1904.09408. — [arXiv:1904.09408](https://arxiv.org/abs/1904.09408) find
  42. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: An efficient data augmentation algorithm for neural machine translation. In EMNLP. find
  43. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. TACL. find
  44. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. TACL, 7:625–641. find
  45. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018. ESPnet: End-to-end speech processing toolkit. In INTERSPEECH. find
  46. Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card
  47. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  48. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS. find
  49. Deyi Xiong, Min Zhang, and Haizhou Li. 2011. Enhancing language models in statistical machine translation with backward n-grams and mutual information triggers. In ACL. find
  50. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2018. Cross entropy of neural language models at infinity– a new bound of the entropy rate. Entropy, 20(11):839. find
  51. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation. In EMNLP. find
  52. Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. 2018. A comparison of techniques for language model integration in encoder-decoder speech recognition. In SLT. find
  53. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS. find our card
  54. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL. find our card
  55. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. find
  56. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In ICLR. find
  57. Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In ICLR. find
  58. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In NeuralGen. find our card