Elamur Research⌕
☰
home / papers / 2024 / QUDSELECT: Selective Decoding for Questions Under Discussion Parsing

QUDSELECT: Selective Decoding for Questions Under Discussion Parsing

Paper · 2024 · Ashima Suvarna, Xiao Liu, Tanmay Parekh et al.

Summary

Questions Under Discussion (QUD) parsing links each sentence of a text to an earlier "anchor" sentence through an implicit question that the later sentence answers. A good QUD must meet three criteria: the answer sentence should actually answer it (answer compatibility), it should introduce no concepts unknown from prior context (givenness), and it should relate to its anchor (anchor relevance). Earlier parsers ran anchor detection and question generation as separate steps, and their outputs often broke at least one criterion. QUDSELECT fine-tunes an instruction-following LLM to produce the anchor and the question together. At inference it samples many candidate pairs and keeps the one rated highest by simple scorers, one per criterion, that need neither training nor reference answers. Experiments on DCQA with LLaMA2-7B, Mistral-7B and GPT-4 show gains over baselines of about 9% in human judgments and about 4% under learned automatic evaluators. The work shows that cheap, theory-driven reranking can steer a generative discourse parser toward linguistically valid structures.

Key points

  • Data: DCQA, about 22k English questions over 606 news articles. Each sentence after the first, which serves as the tree root, receives one ⟨anchor, question⟩ edge.
  • Decoding works in two beam stages (k = 10 by default). The model first proposes k anchors, and these are deduplicated. It then generates k questions for each remaining anchor. The candidate with the highest summed criterion score is kept.
  • The scorers work as follows:
    • Answer compatibility is the entailment probability from bart-large-mnli.
    • Givenness is the share of the question's content-word lemmas that also occur in the prior context.
    • Anchor relevance is the lemma overlap between the anchor and the question's largest noun phrase, which is treated as its focus.
  • Human evaluation covered 100 questions from 8 articles, judged by 3 MTurk annotators, with 68.3% agreement. QUDSELECT (Mistral-7B) produced:
    • 67% of questions answered directly,
    • 78% of questions introducing no new concepts,
    • 68% fully grounded questions.
  • Results improve as k rises from 1 to 20, with a slight loss in givenness. Even at k = 20, fewer than 3 distinct anchors appear on average, so cost grows roughly linearly. GPT-4 tends to put answer content into its questions ("answer leakage"), and reranking reduces this.

Interesting ideas

  • I1. Generate many candidates, then pick one by summing several cheap criterion scores in [0,1] that need no references; this enforces theoretical constraints without retraining the model.
  • I2. Givenness can be measured as the fraction of content lemmas in a generated item that already appear in the preceding text, a transparent test for invented material.
  • I3. An implicit question can label the relation between a sentence and an earlier anchor, giving discourse structure as a tree of questions rather than a fixed relation inventory.
  • I4. The decoder diversifies the structural choice first (which anchor), deduplicates it, and only then varies the surface content (which question); this yields varied candidates while keeping cost manageable.
  • I5. Locating a question's focus in its largest noun phrase and checking that focus against the anchor gives a simple, explainable relevance test.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when the witness parsers or LLM teachers disagree, rank their outputs with explicit, explainable validity checks before accepting silver data.
  • I2 (world): flag generated answers or inferred facts that mention entities or predicates absent from the world of the text, as a guard against hallucination.
  • I3 (frames): add QUD-style edges between sentence frames so that each new sentence records which earlier frame it elaborates and what it answers.
  • I4 (back): in graph-to-text generation, first pick alternative graph orderings or focus nodes, then realize each one, rather than sampling whole sentences blindly.
  • I5 (answer): check that a reader's answer matches the noun-phrase focus of the test question in the UD tree before accepting it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Ashima Suvarna, Xiao Liu, Tanmay Parekh, Kai-Wei Chang and Nanyun Peng. 2024. QUDSELECT: Selective Decoding for Questions Under Discussion Parsing. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.76

@inproceedings{suvarna2024qudselect,
  title = {{QUDSELECT: Selective Decoding for Questions Under Discussion Parsing}},
  author = {Ashima Suvarna, Xiao Liu, Tanmay Parekh, Kai-Wei Chang and Nanyun Peng},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.76},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.76}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. ↗
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  4. Anton Benz and Katja Jasinskaja. 2017. Questions under discussion: From sentence to discourse. ↗
  5. Kordula De Kuthy, Madeeswaran Kannan, Haemanth Santhi Ponnusamy, and Detmar Meurers. 2020. Towards automatically generating questions under discussion to link information and discourse structure. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5786–5798. find
  6. Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. Qud-based annotation of discourse structure and information structure: Tool and evaluation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). ↗
  7. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. arXiv preprint arXiv:1603.08887. ↗
  8. Christoph Hesse, Anton Benz, Maurice Langner, Felix Theodor, and Ralf Klabunde. 2020. Annotating quds for generating pragmatically rich texts. In Proceedings of the Workshop on Discourse Theories for Text Planning, pages 10–16. find
  9. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  10. Nan Hu, Zirui Wu, Yuxuan Lai, Xiao Liu, and Yansong Feng. 2022. Dual-channel evidence fusion for fact verification over texts and tables. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5232–5242. ↗
  11. Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, and Heng Ji. 2024. TextEE: Benchmark, reevaluation, reflections, and future challenges in event extraction. In Findings of the Association for Computational Linguistics ACL 2024, pages 12804– 12825, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. find
  12. AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b (2023). arXiv preprint arXiv:2310.06825. ↗
  13. Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Discourse comprehension: A question answering framework to represent sentence connections. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11752–11764, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  14. Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srinivas, Greg Durrett, and Junyi Jessy Li. 2023. Discourse analysis via questions and answers: Parsing dependency structures of questions under discussion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11181–11195, Toronto, Canada. Association for Computational Linguistics. ↗
  15. Xiao Liu, Yansong Feng, and Kai-Wei Chang. 2024. Casa: Causality-driven argument sufficiency assessment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5282–5302. ↗
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  17. Mingyu Derek Ma, Alexander Taylor, Wei Wang, and Nanyun Peng. 2023. DICE: Data-efficient clinical event extraction with generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15898–15917, Toronto, Canada. Association for Computational Linguistics. ↗
  18. Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1869–1881, Online. Association for Computational Linguistics.
  19. Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. Conditional generation with a question-answering blueprint. Transactions of the Association for Computational Linguistics, 11:974–996. ↗
  20. Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, and Kyle Lo. 2023. A question answering framework for decontextualizing user-facing snippets from scientific documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3194–3212. find
  21. Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, and Nanyun Peng. 2023. GENEVA: Benchmarking generalizability for event argument extraction with hundreds of event types and argument roles. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3664–3686, Toronto, Canada. Association for Computational Linguistics. find
  22. Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, and Nanyun Peng. 2024a. Contextual label projection for cross-lingual structured prediction. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5738–5757, Mexico City, Mexico. Association for Computational Linguistics. ↗
  23. Tanmay Parekh, Anh Mac, Jiarui Yu, Yuxuan Dong, Syed Shahriar, Bonnie Liu, Eric Yang, Kuan-Hao Huang, Wei Wang, Nanyun Peng, and Kai-Wei Chang. 2024b. Event detection from social media for epidemic prediction. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5758–5783, Mexico City, Mexico. Association for Computational Linguistics. ↗
  24. Arndt Riester. 2019. Constructing qud trees. In Questions in discourse, pages 164–193. Brill. ↗
  25. Arndt Riester, Lisa Brunetti, and Kordula De Kuthy. 2018. Annotation guidelines for questions under discussion and information structure. Information structure in lesser-described languages: Studies in prosody and syntax, pages 403–443. ↗
  26. Arndt Riester, Amalia Canes Nápoles, and Jet Hoek. 2021. Combined discourse representations: Coherence relations and questions under discussion. In Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation, pages 26–30.
  27. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and pragmatics, 5:6–1. ↗
  28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ↗
  29. Jan Van Kuppevelt. 1995. Discourse structure, topicality and questioning. Journal of linguistics, 31(1):109– 147. ↗
  30. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705. ↗
  31. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1118–1127, Marseille, France. European Language Resources Association.
  32. Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. 2023. QUDeval: The evaluation of questions under discussion discourse parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5344– 5363, Singapore. Association for Computational Linguistics.
  33. Liyan Xu, Jiangnan Li, Mo Yu, and Jie Zhou. 2024. Graph representation of narrative context: Coherence dependency via retrospective questions. arXiv preprint arXiv:2402.13551. ↗
  34. Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4031–4047. find
  35. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suvarna-2024-qudselect-selective-decoding-questions-under/