Elamur Research⌕
☰
home / papers / 2020 / Enabling Language Models to Fill in the Blanks

Enabling Language Models to Fill in the Blanks

Summary

Standard left-to-right language models generate text well but can only use preceding context, so they cannot fill gaps in the middle of a document. Bidirectional models such as BERT and SpanBERT see both sides but need the span length in advance. Donahue, Lee and Liang propose infilling by language modeling (ILM). Spans of a complete text are randomly replaced with blank tokens. The model is then trained to read this masked text, followed by a separator and only the removed spans, each ending in an answer token. Any off-the-shelf LM can learn this format; the authors fine-tune GPT-2 on three domains: short STORIES, CS paper ABSTRACTS and song LYRICS. ILM matches the perplexity of a much costlier baseline that regenerates the whole document, while keeping sequences almost as short as ordinary language modeling. Starting from pre-trained weights consistently beat training from scratch. In a human study on stories, ILM-infilled sentences were the hardest to tell apart from human writing. This makes the method a practical, architecture-agnostic basis for writing-assistance tools.

Key points

  • Low overhead: For k blanks, an ILM training example is only 2k + 1 tokens longer than the original text, with k averaging about 2. The approach also adds just three special tokens to the vocabulary. The LM-All baseline, by contrast, makes sequences about 1.81× as long, versus about 1.01× for ILM.
  • Masking scheme: The mask function walks a granularity hierarchy (document, paragraph, sentence, n-gram, word) and masks whole subtrees with 3% probability. This yields roughly 15% of tokens masked, echoing BERT. Each granularity has its own blank token, which gives users coarse control over span length.
  • Experimental comparison: All strategies use GPT-2 "small" with identical hyperparameters. ILM is compared with three baselines:
    • LM (past context only);
    • LM-Rev (future context only);
    • LM-All (masked plus full text, following Fedus et al., 2018).
  • Human evaluation: Annotators saw five-sentence STORIES with one machine-infilled sentence and had to identify it. Each system received 100 responses, and the maximum achievable score is effectively 80%. ILM outperformed BERT, the self-attention model of Zhu et al. (2019) (SA) and plain LM. Unexpectedly, plain LM also beat both BERT and SA.
  • Released resources: Code, data and trained models are on GitHub (chrisdonahue/ilm) and CodaLab, and an interactive web demo is also available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.225

@inproceedings{donahue2020enabling,
  title = {{Enabling Language Models to Fill in the Blanks}},
  author = {Donahue, Chris and Lee, Mina and Liang, Percy},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.225},
  url = {https://doi.org/10.18653/v1/2020.acl-main.225}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI21. 2019. HAIM: A modest step towards controllable text generation. AI21 Labs Blog. find
  2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  3. Yannis Assael, Thea Sommerschield, and Jonathan Prag. 2019. Restoring ancient text using deep learning: a case study on greek epigraphy. arXiv:1910.06262. find
  4. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  5. N. Chambers and D. Jurafsky. 2008. Unsupervised learning of narrative event chains. In Human Language Technology and Association for Computational Linguistics (HLT/ACL). find
  6. R. Rudinger, P. Rastogi, F. Ferraro, and B. V. Durme. 2015. Script induction as language modeling. In Empirical Methods in Natural Language Processing (EMNLP). find
  7. Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018. Neural text generation in stories using entity representations as context. In Association for Computational Linguistics: Human Language Technologies. find
  8. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? arXiv:1909.10705. find
  9. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171– 4186. find
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  11. T. Deyes. 1984. Towards an authentic ‘discourse cloze’. Applied Linguistics, 5(2):128–137. find
  12. A. Fan, M. Lewis, and Y. Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. find
  13. W. Fedus, I. Goodfellow, and A. M. Dai. 2018. Maskgan: Better text generation via filling in the. In International Conference on Learning Representations (ICLR). find
  14. J. Gu, Q. Liu, and K. Cho. 2019. Insertion-based decoding with automatically inferred generation order. arXiv preprint arXiv:1902.01370. find
  15. D. Ippolito, D. Grangier, C. Callison-Burch, and D. Eck. 2019. Unsupervised hierarchical story infilling. In NAACL Workshop on Narrative Understanding, pages 37–43. find
  16. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. find our card
  17. D. Liu, J. Fu, P. Liu, and J. Lv. 2019. TIGS: An inference algorithm for text infilling with gradient search. arXiv preprint arXiv:1905.10752. find
  18. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). find
  19. Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. arXiv:2002.03079. find
  20. Y. Shih, W. Chang, and Y. Yang. 2019. XL-Editor: Post-editing sentences with xlnet. arXiv preprint arXiv:1910.10479. — [arXiv:1910.10479](https://arxiv.org/abs/1910.10479) find
  21. W. L. Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. find
  22. A. Wang and K. Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094. find our card
  23. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card
  24. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  25. L. Yao, N. Peng, R. Weischedel, K. Knight, D. Zhao, and R. Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Association for the Advancement of Artificial Intelligence (AAAI). find
  26. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS. find our card
  27. W. Zhu, Z. Hu, and E. Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158. — [arXiv:1901.00158](https://arxiv.org/abs/1901.00158) find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/donahue-2020-enabling-language-models-fill-blanks/