Elamur Research⌕
☰
home / papers / 2020 / QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines

QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines

Summary

Annotating discourse relations, meaning how two propositions within a text connect (causally, temporally, contrastively and so on), has traditionally required trained experts working with schemes such as PDTB or RST. That makes such annotation slow and expensive. This paper extends the QA-based semantic annotation approach (as in QASRL and QAMR) to discourse. The authors encode an intra-sentential relation as a question–answer pair. The question prefix (e.g., "Despite what…?", "In what case…?") signals the relation sense, and the question body and the answer hold the two discourse units. The prefixes are mapped onto most of the PDTB 3.0 sense taxonomy and can capture both explicit and implicit relations. Through a dedicated crowdsourcing interface, non-expert workers produced the QADiscourse dataset. The authors also built a pipeline parser as a baseline. It achieves fairly high unlabeled QA alignment, but it picks the correct relation prefix only about half the time. The work matters because it suggests discourse annotation can be scaled up with laymen and adapted to other languages and domains.

Key points

  • The QADiscourse dataset covers roughly 9,000 sentences and over 16,600 QA pairs, and the authors released it openly.
  • The question templates follow a subset of PDTB 3.0 senses:
    • Pragmatic variants (Belief, SpeechAct) are merged into their general senses.
    • Purpose is folded into the two causal questions.
    • Three Expansion senses are left out, including the very frequent Expansion.Conjunction, because they assert no informational proposition.
  • The baseline is a pipeline with three stages:
    • Prefix and target detection.
    • Question-body generation with a COPYNET/pointer-generator model over BERT encodings, implemented in ALLENNLP.
    • Answer extraction with BERT fine-tuned on SQuAD, then further tuned on 5,004 aligned instances.
  • Full-pipeline test results:
    • UQA F1 of 83.69 (precision 80.79, recall 86.8).
    • LQA accuracy of 66.59.
    • Prefix accuracy of only 49.94.
  • In a manual check of 50 predicted QAs, 22 were correct and 2 more were valid but absent from the gold data. Other errors included ungrammatical-but-sensible and nonsensical outputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

44 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 35 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.224 · arXiv 2010.02815

@inproceedings{pyatkin2020qadiscourse,
  title = {{QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines}},
  author = {Pyatkin, Valentina and Klein, Ayal and Tsarfaty, Reut and Dagan, Ido},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.224},
  eprint = {2010.02815},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.224}
}

References

40 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
  2. Or Biran and Kathleen McKeown. 2015. Pdtb discourse parsing as a tagging task: The two taggers approach. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 96–104. find our card
  3. Lynn Carlson, Mary Ellen Okurowski, and Daniel Marcu. 2002. RST discourse treebank. Linguistic Data Consortium, University of Pennsylvania. find
  4. Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale qa-srl parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060. find our card
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6. find our card
  8. Barbara J Grosz and Candace L Sidner. 1986. Attention, intentions, and the structure of discourse. Computational linguistics, 12(3):175–204. find our card
  9. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640. find our card
  10. Hangfeng He, Qiang Ning, and Dan Roth. 2020. Quase: Question-answer driven sentence encoding. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  11. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 643– 653. find our card
  12. R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL). find
  13. Daisuke Kawahara, Yuichiro Machida, Tomohide Shibata, Sadao Kurohashi, Hayato Kobayashi, and Manabu Sassano. 2014. Rapid development of a corpus with discourse annotations using two-stage crowdsourcing. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Te… find
  14. Yudai Kishimoto, Shinnosuke Sawada, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2018. Improving crowdsourcing-based annotation of japanese discourse relations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find our card
  15. René Knaebel, Manfred Stede, and Sebastian Stober. 2019. Window-based neural tagging for shallow discourse argument labeling. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 768–777. find
  16. I-Ta Lee and Dan Goldwasser. 2019. Multi-relational script learning for discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4214–4226. find our card
  17. Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 137–147. find
  18. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  19. Wanqiu Long, Xinyi Cai, James Reid, Bonnie Webber, and Deyi Xiong. 2020. Shallow discourse annotation for chinese ted talks. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1025–1032. find
  20. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  21. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Sho… find our card
  22. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. Citeseer. find our card
  23. Rashmi Prasad, Bonnie Webber, Alan Lee, Sameer Pradhan, and Aravind Joshi. 2015. Bridging sentential and discourse-level semantics through clausal adjuncts. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 64–69. find
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
  25. Wei Shi and Vera Demberg. 2019. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM… find our card
  26. Jan Van Kuppevelt. 1995. Discourse structure, topicality and questioning. Journal of linguistics, pages 109–147. find
  27. Linh Van Ngo, Khoat Than, Thien Huu Nguyen, et al. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4201–4207. find our card
  28. Bonnie Webber, Markus Egg, and Valia Kordoni. 2012. Discourse structure and language technology. Natural Language Engineering, 18(4):437–490. find
  29. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2016. A discourse-annotated corpus of conjoined vps. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 22–31. find
  30. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. find
  31. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5:6–1. find
  32. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. Ted-q: Ted talks and the questions they evoke. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1118–1127. find
  33. Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher NL Clark, Annie Louis, and Bonnie Webber. 2016. Filling in the blanks in understanding discourse adverbials: Consistency, conflict, and context-dependence in a crowdsourced elicitation task. In Proceedings of the 10th Linguistic Annotation… find
  34. Matthijs Westera and Hannah Rohde. 2019. Asking between the lines: Elicitation of evoked questions in text. In Proceedings of the Amsterdam Colloquium. find
  35. Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257–2267. find our card
  36. Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Crowdsourcing a high-quality gold standard for qa-srl. In ACL 2020 Proceedings, forthcoming. Association for Computational Linguistics. find
  37. Merel Scholman and Vera Demberg. 2017. Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task. In Proceedings of the 11th Linguistic Annotation Workshop, pages 24–33. find
  38. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning-Shared Task, pages 1–16. find our card
  39. Frances Yung, Merel CJ Scholman, and Vera Demberg. 2019. Crowdsourcing discourse relation annotations by a two-step connective insertion task. In The 13th Linguistic Annotation Workshop, page 16. find our card
  40. Deniz Zeyrek, Amalia Mendes, and Murathan Kurfalı. 2018. Multilingual extension of pdtb-style annotation: The case of ted multilingual discourse bank. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC2018). find

↑ back to top

This page in Markdown →