Elamur Research
☰
home / papers / 2018 / Learning to Write with Cooperative Discriminators

Learning to Write with Cooperative Discriminators

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1152.pdf

Summary

RNN language models write fluent sentences, but over whole paragraphs their output tends to be bland, repetitive and occasionally self-contradictory. Holtzman et al. address this by pairing a base RNN generator with a committee of discriminatively trained scorers, each tied to one of Grice's maxims. The scorers target repetition, contradiction or redundancy (via an NLI classifier), relevance to the context, and writing style. Their outputs are added to the language model log-probability in a weighted decoding objective applied during beam search. The weights start uniform and are updated by comparing how the system scores its own best continuation against the gold one. Because the scorers are trained on prefixes, they can rerank partial hypotheses. The resulting system, L2W, was evaluated on text continuation for BookCorpus and TripAdvisor. Human judges clearly preferred it over baselines such as Seq2Seq and SeqGAN for coherence, style and informativeness. Automatic metrics also showed less repetition and greater diversity. The approach matters because it improves long-form generation in a domain-agnostic way without retraining the underlying model.

Key points

  • The decoding objective adds learned-weight discriminator scores to the RNN LM log-probability, which forms a Product of Experts when the scores are log probabilities.
  • The four communication models are:
    • Repetition: an RNN over cosine similarities of words within a recent window.
    • Entailment: the decomposable attention model trained on SNLI and MultiNLI (over 940,000 examples), which rewards "neutral" sentence pairs.
    • Relevance: distinguishes the true continuation from a randomly sampled one.
    • Style: distinguishes human text from LM output.
  • All discriminators use 300-dimensional GloVe embeddings in their first layer.
  • In the ablation, the repetition module gave the largest gain and the entailment module the smallest. The authors attribute the weak entailment result to domain mismatch and to scores that update only at sentence boundaries.
  • On BookCorpus, crowd workers preferred L2W even over the original human continuation. Remaining weaknesses include subtle self-paraphrasing, loosely connected sentences, and pronoun inconsistency (e.g., switching between "we" and "I").

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1152 · arXiv 1805.06087

@inproceedings{holtzman2018learning,
  title = {{Learning to Write with Cooperative Discriminators}},
  author = {Holtzman, Ari and Buys, Jan and Forbes, Maxwell and Bosselut, Antoine and Golub, David and Choi, Yejin},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1152},
  eprint = {1805.06087},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1152}
}

References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In product reviews from attributes. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 623–632, Valencia, Spain.… find our card
  2. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182. Association for Computational Linguistics. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. find
  5. Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-likelihood augmented discrete generative adversarial networks. CoRR, abs/1702.07983. — [arXiv:1702.07983](https://arxiv.org/abs/1702.07983) find
  6. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 218–226, Boulder, Colorado. Associa… find
  7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111. find our card
  8. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98, San Diego,… find our card
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05,… find
  10. Michael Denkowski and Alon Lavie. 2010. Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 250–253. find
  11. Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. 2017. Learning to generate Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
  12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680. find
  13. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2016. Efficient softmax approximation for gpus. arXiv preprint arXiv:1609.04309. find
  14. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017. Improving neural language models with a continuous cache. In International Conference on Learning Representations. find
  15. H Paul Grice, Peter Cole, Jerry Morgan, et al. 1975. Logic and conversation. 1975, pages 41–58. find
  16. Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771–1800. find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  18. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. CoRR, abs/1602.02410. find
  19. Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 329–339. find
  20. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of the Association of Computational Linguistics. find our card
  21. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, pages 48–54. Association for Computational Lingu… find
  22. Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In Proceedings of the Conference on Computer Vision and Pattern Recognition. find
  23. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego,… find our card
  24. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Austin, Texas. Association for Computational Linguisti… find our card
  25. Zachary Chase Lipton, Sharad Vikram, and Julian McAuley. 2015. Capturing meaning in product reviews with character-level generative text models. CoRR, abs/1511.03683. — [arXiv:1511.03683](https://arxiv.org/abs/1511.03683) find
  26. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the Conference on Empirical Methods in Natural Langua… find
  27. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330. find our card
  28. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing lstm language models. ICLR. find our card
  29. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. find
  30. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan. Association for Computational Linguistics. find
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  32. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2249–2255. Association for Computational Linguistics. find our card
  33. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning (ICML), pages 1310–1318. find
  34. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. CoRR, abs/1705.04304. find our card
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  36. Allen Schmaltz, Alexander M. Rush, and Stuart Shieber. 2016. Word ordering without syntax. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2319– 2324, Austin, Texas. Association for Computational Linguistics. find
  37. Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2210–2219. Associ… find
  38. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575. find our card
  39. Ashwin K Vijayakumar, Michael Cogswell, Ram-prasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424. find our card
  40. Hongning Wang, Yue Lu, and ChengXiang Zhai. 2010. Latent aspect rating analysis on review text data: a rating regression approach. In SIGKDD Conference on Knowledge Discovery and Data Mining. find
  41. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 764–… find
  42. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. CoRR, abs/1704.05426. find
  43. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  44. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  45. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017a. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Association for the Advancement of Artificial Intelligence, pages 2852–2858. find
  46. Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2017b. The neural noisy channel. In International Conference on Learning Representations. find
  47. Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017c. A joint speaker-listener-reinforcer model for referring expressions. In Proceedings of the Conference on Computer Vision and Pattern Recognition, volume 2. find
  48. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724. find our card