Elamur Research
☰
home / papers / 2019 / BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1902.04094

Summary

BERT is pretrained to fill in masked tokens using context on both sides, so it is not obvious how to use it as an ordinary language model that scores or generates text. Wang and Cho reinterpret BERT as a language model defined by a Markov random field (MRF-LM) over fixed-length sequences. In this view, its masked-LM objective is a form of (stochastic) pseudo log-likelihood training. The framing gives two uses. Sentences can be ranked by their unnormalized log-probabilities, and text can be sampled with Gibbs sampling. The sampler starts from an all-[MASK] sequence and repeatedly picks a random position, masks it and refills it from the model's top-k predictions, with no extra parameters or training.

The authors use the pretrained BERT base and large checkpoints, trained on Toronto Book Corpus and Wikipedia, and compare them against OpenAI GPT. They measure quality with corpus-BLEU, perplexity under an external LM and human fluency ratings, and diversity with self-BLEU and unique n-grams. BERT's outputs are more varied but somewhat less fluent than GPT's. The work matters because it turns an encoder used mainly for fine-tuning into a generative model and opens it to MCMC-based generation.

Key points

  • Setup: Each model produced 1000 uncased sequences of 40 tokens. Sampling used the non-sequential Gibbs scheme with top-k = 100, and the implementation was the HuggingFace PyTorch BERT. GPT, which is roughly the size of BERT base and trained on TBC, served as the baseline. A left-to-right sequential variant of the sampler was also tried.
  • Diversity (self-BLEU, lower is more diverse): BERT large scored 9.43 and BERT base 10.06, versus 40.02 for GPT. Real data scored 9.80 for WT103 and 12.51 for TBC.
  • Quality (perplexity, external LM): Perplexity was measured with a Gated Convolutional LM trained on WikiText-103. It was 331.47 for BERT large, 279.10 for BERT base and 154.29 for GPT. For comparison, it was 314.28 on real TBC sentences and 54.00 on WT103 sentences.
  • GPT overlap: GPT had corpus-BLEU of 30.75 against TBC, higher than TBC sampled against itself (23.05). Together with its low share of unique n-grams, this suggests GPT relies heavily on generic sentences.
  • Human fluency and code: Human fluency on a 4-point Likert scale (100 samples per model, 3 annotators each) averaged 2.37 for BERT large, 2.65 for BERT base and 2.80 for GPT. All means fall within one standard deviation of each other, and BERT large's ratings are bimodal. Code is released at github.com/nyu-dl/bert-gen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

121 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.04094 · arXiv 1902.04094

@misc{wang2019bert,
  title = {{BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model}},
  author = {Wang, Alex and Cho, Kyunghyun},
  year = {2019},
  doi = {10.48550/arxiv.1902.04094},
  eprint = {1902.04094},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.04094}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julian Besag. 1977. Efficiency of pseudolikelihood estimation for simple gaussian fields. Biometrika, pages 616–618. find
  2. John S Bridle. 1990. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pages 227–236. Springer. find
  3. KyungHyun Cho, Tapani Raiko, and Alexander Ilin. 2010. Parallel tempering is efficient for learning restricted boltzmann machines. In Neural Networks (IJCNN), The 2010 International Joint Conference on, pages 1–8. IEEE. find
  4. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. Language modeling with gated convolutional networks. arXiv preprint:1612.08083. find our card
  5. Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau. 2010. Tempered markov chain monte carlo for training of restricted boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 145–152. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint:1810.04805. find
  7. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint:1805.04833. find
  8. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. find our card
  9. Yacine Jernite, Alexander Rush, and David Sontag. 2015. A fast variational approach for learning markov random field language models. In International Conference on Machine Learning, pages 2209–2217. find
  10. Nikita Kitaev and Dan Klein. 2018. Multilingual constituency parsing with self-attention and pre-training. find our card
  11. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv e-prints, page arXiv:1901.07291. find
  12. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. find our card
  13. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint:1301.3781. find
  14. Radford M Neal. 1993. Probabilistic inference using markov chain monte carlo methods. find
  15. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. arXiv preprint:1901.04085. find our card
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  18. Ruslan R Salakhutdinov. 2009. Learning in markov random fields using tempered transitions. In Advances in neural information processing systems, pages 1598–1606. find
  19. Robert H Swendsen and Jian-Sheng Wang. 1986. Replica monte carlo simulation of spin-glasses. Physical review letters, 57(21):2607. find
  20. Pascal Vincent. 2011. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674. find
  21. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(Dec):3371–3408. find
  22. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852–2858. find
  23. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. arXiv preprint:1802.01886. find our card
  24. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint:1506.06724. find our card