Elamur Research⌕
☰
home / papers / 2020 / Assessing Discourse Relations in Language Generation from GPT-2

Assessing Discourse Relations in Language Generation from GPT-2

Summary

Large pre-trained generators such as GPT-2 produce fluent text, but it has been unclear whether the discourse connectives they emit (e.g., but, because) signal relations that actually hold between the linked clauses. Ko and Li present the first discourse-level analysis of GPT-2 output. They test two settings: GPT-2 medium fine-tuned for dialogue on PersonaChat, and "organic" continuation, where the untuned model completes an utterance prefix ending in a connective. Crowd workers saw generated sentences with the connective hidden and chose the PDTB top-level relation they believed fit best, or NONE. GPT-2's connective was then checked against the majority choice. GPT-2 often selects inappropriate connectives. It does well when the relation is obvious to humans but poorly when more inference is needed, and the fine-tuned model matches human judgments more closely. As a remedy, the authors propose a post-processing step that uses a BERT-based connective classifier to swap in a better connective. This raises agreement with annotators in both settings. The findings argue for modeling discourse explicitly in generation.

Key points

  • Setup: GPT-2 medium (345M) was fine-tuned on 122,499 PersonaChat (ConvAI2) prompt–response pairs.
    • Fine-tuned decoding used top-k sampling (k=10); organic decoding used nucleus sampling (p=0.9).
    • Annotators on Amazon Mechanical Turk preferred the fine-tuned model's responses over SpaceFusion, 45.5% vs 16.9%.
  • Connectives studied: 11 connectives were extracted heuristically.
    • About 11% of human PersonaChat responses contain one, compared with 26% of fine-tuned outputs and 15% of organic outputs.
    • The distribution is dominated by and and but.
  • Annotation: About 1.2K sentences per scenario were labeled, each by 5 MTurk workers.
    • A majority relation emerged for 89.7% of fine-tuned sentences (Krippendorff's alpha 0.508) and 83.5% of organic sentences (alpha 0.382).
  • Agreement with humans: GPT-2's connectives matched the majority relation in 81.5% of fine-tuned and 74.9% of organic sentences.
    • Accuracy exceeded 92% when all five annotators agreed, but fell to 64.3% (fine-tuned) and 53.2% (organic) when only three agreed.
  • Fix: The connective predictor is BERT-base-uncased trained on 1M Reddit sentences plus 300K synthetic NONE pairs.
    • Replacing GPT-2's connectives with its predictions raised accuracy (≥3 annotators agreeing) to 82.9% for fine-tuned and 80.4% for organic output.
    • The gain was largest in the organic setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.inlg-1.8 · arXiv 2004.12506

@inproceedings{ko2020assessing,
  title = {{Assessing Discourse Relations in Language Generation from GPT-2}},
  author = {Ko, Wei-Jen and Li, Junyi Jessy},
  booktitle = {Proceedings of the 13th International Conference on Natural Language Generation},
  year = {2020},
  doi = {10.18653/v1/2020.inlg-1.8},
  eprint = {2004.12506},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.inlg-1.8}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Or Biran and Kathleen McKeown. 2015. Discourse planning with an n-gram model of relations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1973– 1977. \#\# Conclusion This work presents an assessment of discourse relations in organic and fine-tuned lang… find
  3. Chloé Braud and Pascal Denis. 2016. Learning connective-based word representations for implicit discourse relation identification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 203–213. find our card
  4. Charles B. Callaway. 2003. Integrating discourse markers into a pipelined natural language generation architecture. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 264–271. find
  5. Alex Lascarides and Nicholas Asher. 2008. Segmented discourse representation theory: Dynamic semantics with discourse structure. In Computing meaning, pages 87–124. Springer. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Mingyu Derek Ma, Kevin Bowden, Jiaqi Wu, Wen Cui, and Marilyn Walker. 2019. Implicit discourse relation identification for open-domain dialogues. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 666– 672. find our card
  8. Bonnie JF Meyer and Roy O Freedle. 1984. Effects of discourse type on recall. American Educational Research Journal, 21(1):121–143. find
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898. find
  10. Keith K Millis, Arthur C Graesser, and Karl Haberlandt. 1993. The impact of connectives on the memory for expository texts. Applied Cognitive Psychology, 7(4):317–339. find
  11. Xiang Gao, Sungjin Lee, Yizhe Zhang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2019. Jointly optimizing diversity and relevance in neural response generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find
  12. Allen Nie, Erin Bennett, and Noah Goodman. 2019. Dissent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510. find our card
  13. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  14. Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf. 2019. Large-scale transfer learning for natural language generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6053–6058. find
  15. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In Proceedings of the Eighth International Conference on Learning Representations. find
  16. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In Language Resources and Evaluation Conference. find our card
  17. Rosalind Horowitz. 1987. Rhetorical structure in discourse processing. In Comprehending oral and written language, pages 117–160. Academic Press. find
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
  19. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse-driven language models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 332–3… find
  20. Ted JM Sanders and Leo GM Noordman. 2000. The role of coherence relations and their linguistic markers in text processing. Discourse processes, 29(1):37–60. find
  21. Merel Scholman and Vera Demberg. 2017. Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task. In Proceedings of the 11th Linguistic Annotation Workshop, pages 24–33. find
  22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the Third International Conference on Learning Representations. find
  23. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 843–861. find
  24. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  25. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. In NeurIPS 2018 CAI Workshop. find
  26. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning-Shared Task, pages 1–16. find our card
  27. Frances Yung, Vera Demberg, and Merel Scholman. 2019. Crowdsourcing discourse relation annotations by a two-step connective insertion task. In Proceedings of the 13th Linguistic Annotation Workshop, pages 16–25. find our card
  28. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213. find our card
  29. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DialoGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu… find our card

↑ back to top

This page in Markdown →