Elamur Research⌕
☰
home / papers / 2024 / Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues

Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues

Paper · 2024 · Maria Francis, Julius Steuer, Dietrich Klakow et al.

Summary

The paper looks at the hidden assumptions built into questions in courtroom cross-examination. Leading and loaded questions often force a witness to accept, or openly deny, something they never agreed to. The authors define two tasks. The first is Presupposition as Natural Language Inference (PNLI), which adds a "presupposition" label to the usual entailment/contradiction/neutral labels. The second is presupposition generation: given a question, produce the list of propositions it takes for granted. They build two datasets: PECaN for classification and PGen for generation. To collect the data they combine manual annotation, prompts to GPT 3.0, and examples augmented from existing NLI corpora. They then fine-tune BERT and T5. On PNLI, T5 clearly beats BERT, especially on the presupposition class. On generation, T5 produces reasonable presuppositions that follow systematic trigger-based patterns, but it does not yield complete lists. The work matters because it gives a resource for testing whether models capture implicit, non-asserted content. That ability is relevant to dialogue analysis and to handling questions that cannot be answered as asked.

Key points

  • The two datasets are PECaN (Presupposition, Entailment, Contradiction and Neutral) for PNLI classification and PGen (Presupposition Generation) for list generation.
  • On PNLI, T5 reaches 84.35% overall accuracy against 71.85% for BERT. On the presupposition label alone, T5 scores 93% and BERT 73%.
  • Presupposition generation with T5 reaches a ROUGE score of 43.47 despite little training data. The generated lists are typically incomplete.
  • Question types in the annotated cross-examinations were tagged partly with ISO 24617-2 dialogue-act categories. Their frequencies are:
    • leading 35.6%
    • loaded 27.8%
    • propositional 15.1%
    • check 11.1%
    • set 8.8%
    • imperatives 1.6%
  • Observed triggers are factive and aspectual verbs, it-clefts, wh-clefts, quantifiers, focus particles, definites and possessives. These are grouped into existential, lexical and structural presuppositions.

Interesting ideas

  • I1. Adding presupposition as its own label next to entailment, contradiction and neutral forces a model to tell backgrounded content apart from asserted consequences, instead of merging both into "entailment".
  • I2. Presuppositions are tied to concrete triggers: definites and possessives yield existence, factive and aspectual verbs yield lexical presuppositions, and clefts yield structural ones. These triggers are recognizable from surface or syntactic cues.
  • I3. Presuppositions survive when the trigger is negated, modalised, questioned or put in a conditional, so a question still commits its asker to background facts.
  • I4. A loaded question carries an assumption the addressee never accepted, so checking presuppositions against known facts tells you when a question should be challenged rather than answered.
  • I5. The training data mixes hand annotation, LLM-prompted examples and items recycled from existing corpora to cover a rare phenomenon cheaply.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could mark each inference it derives as asserted, presupposed or implicated, so that the explanation shows why a proposition holds.
  • I2 (tree): trigger detection could be written as UD-pattern rules over the dependency tree, such as a cleft construction, a factive head with a ccomp, or a det/poss on a nominal, with no learned black box.
  • I3 (world): when reading questions or negated sentences, the world of the text could still add the presupposed entities and states, flagged as speaker-assumed rather than asserted.
  • I4 (answer): before answering a reader question, Mova could check its presuppositions against the text world and reply "the story never says that" when one fails.
  • I5 (learn): an LLM teacher tier could propose presupposition lists that rule-based trigger detection then confirms, keeping only agreed items as silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksImplicature · 138 works

Cite

Cite as Maria Francis, Julius Steuer, Dietrich Klakow and Volha Petukhova. 2024. Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{francis2024who,
  title = {{Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues}},
  author = {Maria Francis, Julius Steuer, Dietrich Klakow and Volha Petukhova},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/2kek9o75nhcr},
  url = {https://doi.org/10.63317/2kek9o75nhcr}
}

References

33 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Langshaw Austin. 1975. How to do things with words, volume 88. Oxford university press. find
  2. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. ↗
  3. Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. CoRR, abs/1705.02364. https://lct-master.org/ ↗
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Gisli H Gudjonsson. 1984. Interrogative suggestibility: Comparison between ‘false confessors’ and ‘deniers’ in criminal trials. Medicine, Science and the Law, 24(1):56–60. find
  7. Maosheng Guo, Yu Zhang, Dezhi Zhao, and Ting Liu. 2017. Generating textual entailment using residual LSTMs. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data: 16th China National Conference, CCL 2017, and 5th International Symposium, NLP-NABD 2017, Nanjing, China, October 13-15, 2017, Proceedings 16, pages 263–272. Springer. find
  8. ISO. 2020. Language resource management – Semantic annotation framework – Part 2: Dialogue acts. ISO 24617-2, Second Edition. ISO Central Secretariat, Geneva. find
  9. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IM-Plicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find
  10. Jiyou Jia. 2008. The generation of textual entailment with NLML in an intelligent dialogue system for language learning CSIEC. In 2008 International Conference on Natural Language Processing and Knowledge Engineering, pages 1– 8. IEEE.
  11. Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In Proceedings of the 29th International Conference on Computational Linguistics, pages 779–785, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  12. Kathy Kellermann. 2007. Persuasive question asking: how question wording influences answers. In Annual Meeting of the State Bar Association of California, Anaheim, CA. find
  13. Layth Muthana Khaleel. 2010. An analysis of presupposition triggers in english journalistic texts. Journal of College of Education for Women, 21(2):523–551. find
  14. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. arXiv preprint arXiv:2101.00391. ↗
  15. Vladyslav Kolesnyk, Tim Rocktäschel, and Sebastian Riedel. 2016. Generating natural language inference chains. arXiv preprint arXiv:1606.01404. ↗
  16. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957–966. PMLR. find
  17. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question-answering. Natural Language Engineering, 7(4):343–360. ↗
  18. CA Logogye. 2016. Syntactic and pragmatic analysis of cross-examination in ghanaian law courts. Journal of Literature, Languages and Linguistics, 26:24. find
  19. Kenneth J Melilli. 2003. Leading questions on direct examination: A more pragmatic approach. Am. J. Trial Advoc., 27:155. find
  20. Zuzana Nevěřilová. 2014. Paraphrase and textual entailment generation in Czech. Computación y Sistemas, 18(3):555–568. ↗
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. ↗
  22. Lucas M Seuren. 2019. Questioning in court: The construction of direct examinations. Discourse Studies, 21(3):340–357. ↗
  23. Robert Stalnaker. 1975. Presuppositions. In Contemporary Research in Philosophical Logic and Linguistic Semantics: Proceedings of a Conference Held at the University of Western Ontario, London, Canada, pages 31–41. Springer. find
  24. William B Swann, Toni Giuliano, and Daniel M Wegner. 1982. Where leading questions can lead: The power of conjecture in social interaction. Journal of Personality and Social Psychology, 42(6):1025. ↗
  25. Douglas Walton. 1999. The fallacy of many questions: On the notions of complexity, loadedness and unfair entrapment in interrogative theory. Argumentation, 13:379–383. ↗
  26. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. ↗
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. 8. ↗
  28. Language Resource References Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  29. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. ↗
  30. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. ↗
  31. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IM-Plicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find
  32. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 1–8.
  33. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1112–– 1122.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/francis-2024-who-did-you-blame-when/