Elamur Research
☰
home / papers / 2019 / Pragmatically Informative Text Generation

Pragmatically Informative Text Generation

What it is about

Conditional generators often under-inform: a restaurant description may leave out attributes of its input meaning representation. The paper applies two pragmatic methods from the Rational Speech Acts framework to generation. The reconstructor method uses a listener that tries to recover the input from the output and rescores candidates. The distractor method makes the speaker, token by token, prefer words that separate the true input from a distractor. On the E2E restaurant task the reconstructor-based speaker raises BLEU from 66.52 to 68.60 over its base model and beats the best E2E challenge result on BLEU, NIST and CIDEr. On CNN/Daily Mail, both methods add 0.2 to 0.5 ROUGE points. A coverage ratio (how often each input value appears in the output) shows that much of the gain comes from mentioning more attributes.

Ideas

For Mova

Chapters: languages/en/33-implicature-and-presupposition.md (the quantity maxim), languages/en/30-pragmatics-overview.md. Code: prag/src/rsa.rs (the RSA module), world/src/knowledge.rs (school facts), en/src/lin.rs (candidates). Measure: coverage ratio per attribute type on the dev lessons; the test set is used once per release.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1410.pdf

Summary

Neural conditional text generators often omit parts of their input. For example, a restaurant description may skip the price or the area. This paper uses Rational Speech Acts–style computational pragmatics to address that. The idea is that a speaker should produce text from which a listener could recover the original input.

The authors add two pragmatic layers on top of strong existing base speakers (S0):

  • Reconstructor-based (S1R): a separately trained listener scores beam-search candidates by how well the input can be rebuilt from them.
  • Distractor-based (S1D): following Cohn-Gordon et al., decoding works word by word and favors tokens that separate the true input from a constructed distractor input.

They test both on two tasks:

  • E2E: generation from meaning representations, with the Puzikov and Gurevych system as S0.
  • CNN/Daily Mail: abstractive summarization, building on the extract-then-rewrite model of Chen and Bansal.

Both methods improve on their base systems, reaching state-of-the-art BLEU on E2E and competitive ROUGE on summarization. The work shows that pragmatic reasoning, previously used mostly in grounded settings, also reduces underinformativeness in standard NLG.

Key points

  • E2E results: BLEU rises from 66.52 (S0) to 68.60 with S1R and 67.76 with S1D. S1R also achieves the best CIDEr (2.37).
  • CNN/Daily Mail results: the pragmatic systems add about 0.2–0.5 ROUGE and 0.2–1.8 METEOR over S0. Here S1D outperforms S1R.
  • Listeners and distractors:
    • For E2E, the reconstructor is a per-attribute multi-class classifier.
    • For summarization, the reconstructor is an ABS model trained in reverse, from summary sentence to source sentence.
    • For E2E, the distractor masks the attributes present in the input and fills absent attributes with their most frequent values.
    • For summarization, the distractor is the previous extracted sentence.
  • Coverage analysis: a coverage-ratio analysis on E2E shows that S1R mentions every attribute type more often than S0. Masking a chosen attribute in S1D's distractor usually boosts coverage of that attribute, which offers a way to control the output.
  • Code: released at github.com/sIncerass/prag_generation.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1410 · arXiv 1904.01301

@inproceedings{shen2019pragmatically,
  title = {{Pragmatically Informative Text Generation}},
  author = {Shen, Sheng and Fried, Daniel and Andreas, Jacob and Klein, Dan},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1410},
  eprint = {1904.01301},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1410}
}

References

38 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  2. Michael C Frank, Noah D Goodman, Peter Lai, and Joshua B Tenenbaum. 2009. Informative communication in word production and word learning. In Proceedings of the 31st annual conference of the cognitive science society, pages 1228–1233. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗ our card
  4. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  5. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 1662–1675, New Orleans, Louisiana. Association for Computational Linguistics. find
  6. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 conference on Empirical Methods in Natural Language Processing, pages 410–419. Association for Computational Linguistics. find
  7. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  8. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 484– 494, Berlin, Germany. Association for Computational Linguistics. find our card
  9. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1724–1734, Doha, Qatar. Association for Computational Linguistics. find
  10. Reuben Cohn-Gordon, Noah Goodman, and Chris Potts. 2018. Pragmatically informative image captioning with character-level reference. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
  11. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138–145. Morgan Kaufmann Publishers Inc. find
  12. Ondrej Dušek and Filip Jurčı́ček. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In ACL. find
  13. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG challenge. In Proceedings of the International Conference on Natural Language Generation. find
  14. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. ↗ our card
  15. Heng Gong. 2018. Technical report for E2E NLG challenge. In E2E NLG Challenge System Descriptions. find
  16. Herbert P Grice. 1970. Logic and conversation. find
  17. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. find our card
  18. Laurence Horn. 1984. Toward a new taxonomy for pragmatic inference: Q-based and r-based implicature. Meaning, form, and use in context: Linguistic applications, 11:42. find
  19. Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 152–162, New Orleans, Louisiana. Association for Computational Linguistics. find
  20. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic. Association for Computational Linguistics. find our card
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. find
  22. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2015. Generation and comprehension of unambiguous object descriptions. arXiv preprint arXiv:1511.02283. ↗ our card
  23. Will Monroe, Robert XD Hawkins, Noah D Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics. find our card
  24. Will Monroe, Jennifer Hu, Andrew Jong, and Christopher Potts. 2018. Generating bilingual pragmatic color references. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
  25. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam. ILLC. find
  26. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Meeting of the Association for the Advancement of Artificial Intelligence. find our card
  27. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In SIGNLL Conference on Computational Natural Language Learning, pages 280– 290, Berlin, Germany. Association for Computational Linguistics. find
  28. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics. find
  29. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. find
  30. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the Conference on Computer Vision and Pattern Recognition. find our card
  31. Biao Zhang, Jing Yang, Qian Lin, and Jinsong Su. 2018. Attention regularized sequence-to-sequence learning for E2E NLG challenge. In E2E NLG Challenge System Descriptions. find
  32. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2–3):103–233. find
  33. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue. find our card
  34. Naho Orita, Eliana Vornov, Naomi Feldman, and Hal Daumé III. 2015. Why discourse affects speakers’ choice of referring expressions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1639– 1649. find
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  36. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the International Conference on Learning Representations, volume abs/1705.04304. ↗ our card
  37. Yevgeniy Puzikov and Iryna Gurevych. 2018. E2E NLG challenge: Neural models vs. templates. In Proceedings of the International Conference on Natural Language Generation. find
  38. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. find