Elamur Research⌕
☰
home / papers / 2022 / Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Paper · 2022 · Kushal Arora, Layla El Asri, Hareesh Bahuleyan et al.

Summary

Neural text generators often produce repetitive, dull, incoherent or hallucinated output. A common explanation is exposure bias: models trained with teacher forcing always see gold contexts during training, but at generation time they condition on their own earlier predictions. Whether this mismatch actually harms generation has been disputed. The authors address the question by framing generation as sequential decision-making. The language model acts as a policy, contexts are states, and next tokens are actions. Under a KL-style per-step loss, teacher forcing is equivalent to behavior cloning. This equivalence lets them reuse regret bounds from imitation learning to define exposure bias as accumulated error during decoding. In their experiments, error grows super-linearly with sequence length. They explain, both analytically and empirically, why perplexity does not reveal this compounding. Comparing decoding strategies, they find that methods with less error accumulation also produce better text. The work gives a principled, measurable definition of exposure bias that can be used to evaluate new training and decoding methods.

Key points

  • Behavior cloning with a per-step loss of KL divergence from the oracle matches the teacher-forcing (MLE) objective up to a constant, so both yield the same model.
  • The borrowed regret bound runs from linear in T (no compounding) to quadratic in T (worst case), where T is the maximum sequence length. Observed error accumulation is super-linear, which confirms that exposure bias matters.
  • Perplexity is computed on gold contexts, so it cannot capture errors that arise from the model's own generated contexts.
  • A moderate amount of stochastic sampling lowers error accumulation and yields more coherent, oracle-like text than deterministic decoding.
  • The paper appeared in Findings of ACL 2022 (pp. 700–710), and code is released at github.com/kushalarora/quantifying_exposure_bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓McGill UniversityMontreal · Canada51 works · 10,476 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 145 works

Cite

Cite as Kushal Arora, Layla El Asri, Hareesh Bahuleyan and Jackie Cheung. 2022. Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.58

@inproceedings{arora2022why,
  title = {{Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation}},
  author = {Kushal Arora, Layla {El Asri}, Hareesh Bahuleyan and Jackie Cheung},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.58},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.58}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. An actor-critic https://www.calculquebec.ca 4 https://www.computecanada.ca algorithm for sequence prediction. In 5th International Conference on Learning Representation… ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs, stat]. ArXiv: 1409.0473.
  3. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1171–1179. Curran… find
  4. Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020. Reinforcement learning based graph-to-sequence model for natural question generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  5. Byung-Ju Choi, Jimin Hong, David Park, and Sang Wan Lee. 2020. F2̂-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9167–9182, Online. Association for Computational… find
  6. Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. 2020. Residual energy-based models for text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  7. Wanyu Du and Yangfeng Ji. 2019. An empirical comparison on imitation learning and reinforcement learning for paraphrase generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMN… find
  8. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
  9. Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James R. Glass. 2019. Quantifying exposure bias for neural language generation. CoRR, abs/1905.10617. — [arXiv:1905.10617](https://arxiv.org/abs/1905.10617) find
  10. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. arXiv:1904.09751 [cs]. ArXiv: 1904.09751. find
  11. Rémi Leblond, Jean-Baptiste Alayrac, Anton Osokin, and Simon Lacoste-Julien. 2018. SEARNN: training rnns with global-local losses. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. find
  12. Stéphane Ross and Drew Bagnell. 2010. Efficient reductions for imitation learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pages 661–668. find
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461 [cs, stat]. 00007 arXiv: 1910.13461…
  14. Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 627–635. find
  15. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
  16. Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, and Haizhou Li. 2019. Teacher-student training for robust tacotron-based TTS. CoRR, abs/1911.02839. — [arXiv:1911.02839](https://arxiv.org/abs/1911.02839) find
  17. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  18. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs]. 00218 arXiv: 1609.07843.
  19. Florian Schmidt. 2019. Generalization in Generation: A closer look at Exposure Bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 157–167, Hong Kong. Association for Computational Linguistics. 00002. find
  20. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. arXiv:1704.04368 [cs]. ArXiv: 1704.04368. find
  21. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. arXiv:1507.04808 [cs]. ArXiv: 1507.04808.
  22. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–1692, Berlin, Germany. Associatio…
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 [cs]. ArXiv: 1706.03762.
  24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. page 67.
  26. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  27. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y.-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. arXiv:2004.13637 [cs].
  28. Ashwin K. Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models.
  29. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. CoRR, abs/2005.03642.
  30. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural Text Generation with Unlikelihood Training. arXiv:1908.04319 [cs, stat]. 00000 arXiv: 1908.04319.
  31. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280. find
  32. Yifan Xu, Kening Zhang, Haoyu Dong, Yuezhou Sun, Wenlong Zhao, and Zhuowen Tu. 2019. Rethinking exposure bias in language modeling. CoRR, abs/1910.11235. — [arXiv:1910.11235](https://arxiv.org/abs/1910.11235) find
  33. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 2852–2858. AAAI Press. find
  34. Amy Zhang, Adam Lerer, Sainbayar Sukhbaatar, Rob Fergus, and Arthur Szlam. 2018. Composable Planning with Attributes. arXiv:1803.00512 [cs]. ArXiv: 1803.00512. — [arXiv:1803.00512](https://arxiv.org/abs/1803.00512) find
  35. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
  36. Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019b. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long P…
  37. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Ling…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arora-2022-why-exposure-bias-matters-imitation/