Elamur Research⌕
☰
home / papers / 2026 / Don't ‘Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

Don't ‘Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

Paper · 2026 · Shenran Wang, Vered Shwartz and Hila Gonen

Summary

The paper examines false-presupposition QA (FPQA), where a model should either answer a question or point out and correct a mistaken assumption inside it. The authors argue that current evaluation is skewed. Benchmarks contain far more flawed questions than real usage does, and methods are scored mainly on those questions, while performance on ordinary questions with true presuppositions (TPQs) is rarely checked.

They test many FPQA approaches on four benchmarks: (QA)², CREPE, Syn-QA², and Cancer-Myth. The approaches include direct prompting, GEPA prompt tuning, decompose-then-fact-check pipelines, attention-head ablation (FAITH), and LoRA fine-tuning. These run on several model families and under different retrieval settings. Responses are scored by an LLM judge using mirrored rubrics for false-presupposition questions (FPQs) and TPQs.

The central finding is a consistent tradeoff: methods that catch more false presuppositions wrongly "correct" more true ones. Using a realistic share of flawed questions, plain Direct QA scores best overall. An ablation on gold presuppositions traces the problem to the fact-checking step, which tends to reject claims it cannot confirm. This matters because FPQA gains measured on skewed benchmarks may make real assistants worse.

Key points

  • Benchmarks: (QA)², CREPE, single-hop Syn-QA², and Cancer-Myth.
  • Models: Gemma, Meta-Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, OLMo-3-7B-Instruct, and Gemini-3-Flash. Fact checking is done by the LLM itself or by MiniCheck, plus Llama3-Med42-8B on Cancer-Myth.
  • Realistic FPQ rate: in 100 hand-labelled WildChat questions, only about 13% were FPQs. This compares with 21% (Kim et al., 2021), 25% in CREPE, and up to 100% in synthetic sets.
  • Weighted scoring: an expected score is computed as 0.13·V_F + 0.87·V_T. Under it, Direct QA ranks first and FP Identification ranks last on every dataset.
  • Fact-checking accuracy: on gold presuppositions, FPQ accuracy is near-perfect, but TPQ accuracy on Cancer-Myth stays below 50%. Gemini with web search reaches about 76% on Syn-QA² TPQs, still well below its FPQ accuracy.
  • Retrieval effects:
    • Open-weight models do best with top-4 passages.
    • Feeding all passages can hurt them and causes more gibberish in smaller models.
    • Gemini does better with no passages or with web search.
  • Released resource: manual presupposition annotations for the 150 TPQs of Cancer-Myth.

Interesting ideas

  • I1. Every "detect the flaw" test gets a mirrored control set of flawless items, scored on a symmetric rubric that penalises unnecessary corrections, so over-triggering becomes visible.
  • I2. Per-class scores are combined using a base rate measured by hand-labelling a small sample of real traffic, instead of trusting the benchmark's own class balance.
  • I3. The pipeline's components are diagnosed in isolation by feeding gold presuppositions straight to the verifier, which shows that the fact-checker, not the extractor, is the weak link.
  • I4. The core failure is treating "cannot prove" as "false", so a verifier should keep "unsupported" distinct from "contradicted".
  • I5. Converting a question into a declarative statement before checking it gives a clean, atomic claim to verify.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Add true-premise twins to each false-premise or trick question in the evaluation suite, as negative controls that catch a repair rule or teacher tier that starts "correcting" everything.
  • I2 (answer): When reporting reader accuracy on fables or school tests, weight sub-scores by how often each question type actually occurs, so rare adversarial cases do not dominate tuning decisions.
  • I3 (world): Check the world-of-text consistency module directly on gold presupposition statements, separate from parsing and frame extraction, so errors can be attributed to the correct stage.
  • I4 (world): When checking a question's presuppositions against entities and states, use three outcomes (supported, contradicted, unknown) and only flag a false premise on an explicit contradiction.
  • I5 (frames): Turn question frames into declarative frames, including the presupposition carried by wh-words and definites, and pass each one as a separate checkable claim to the world layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🔬Canadian Institute for Advanced ResearchToronto · Canada28 works · 16,725 citations🔬Vector InstituteToronto · Canada33 works · 4,535 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 441 worksImplicature · 138 works

Cite

Cite as Shenran Wang, Vered Shwartz and Hila Gonen. 2026. Don't ‘Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance. arXiv:2608.06539.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.06539 · arXiv 2608.06539

@misc{wang2026don,
  title = {{Don't ‘Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance}},
  author = {Shenran Wang, Vered Shwartz and Hila Gonen},
  year = {2026},
  doi = {10.48550/arxiv.2608.06539},
  eprint = {2608.06539},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.06539}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations. ↗
  2. Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. 2021. Think you have solved direct-answer question answering? try arc-da, the direct-answer ai2 reasoning challenge. Preprint, arXiv:2102.03315. ↗
  3. Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for LLM-as-a-judge: How to statistically justify replacing human annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16051– 16081, Vienna, Austria. Association for Computational Linguistics.
  4. Myra Cheng, Robert D. Hawkins, and Dan Jurafsky. 2026. Accommodation and epistemic vigilance: A pragmatic account of why LLMs fail to challenge harmful beliefs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16181– 16203, San Diego, California, United States. Association for Computational Linguistics. ↗
  5. Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical llms. Preprint, arXiv:2408.06142. ↗
  6. Ashwin Daswani, Rohan Sawant, and Najoung Kim. 2024. Syn-qa2: Evaluating false assumptions in long-tail questions with synthetic qa datasets. Preprint, arXiv:2403.12145. ↗
  7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  8. Ruohao Guo, Wei Xu, and Alan Ritter. 2025. How to protect yourself from 5G radiation? investigating LLM responses to implicit misinformation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28842– 28861, Suzhou, China. Association for Computational Linguistics. find
  9. Wookje Han, Jinsol Park, and Kyungjae Lee. 2023. Pre-WoMe: Exploiting presuppositions as working memory for long form question answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8312–8322, Singapore. Association for Computational Linguistics. find
  10. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  11. Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. Won’t get fooled again: Answering questions with false premises. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5626–5643, Toronto, Canada. Association for Computational Linguistics. find
  12. Haeji Jung and Hila Gonen. 2026. Phantombench: Benchmarking the non-existential threat of language models. Preprint, arXiv:2606.11105. ↗
  13. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. 2023. (QA) 2 : Question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8466–8487, Toronto, Canada. Association for Computational Linguistics. ↗
  14. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3932–3945, Online. Association for Computational Linguistics. ↗
  15. Clara Lachenmaier, Judith Sieker, and Sina Zarrieß. 2025. Can LLMs ground when they (don’t) know: A study on direct and loaded political questions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14956–14975, Vienna, Austria. Association for Computational Linguistics. find
  16. Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, and 50 others. 2026. Olmo 3. Preprint, arXiv:2512.13961. ↗
  17. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. ↗
  18. Mohammadamin Shafiei, Hamidreza Saffari, and Nafise Sadat Moosavi. 2025. MultiHoax: A dataset of multi-hop false-premise questions. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10169–10187, Vienna, Austria. Association for Computational Linguistics. find
  19. Anthony Sicilia, Mert Inan, and Malihe Alikhani. 2025. Accounting for sycophancy in language model uncertainty estimation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7866–7881, Albuquerque, New Mexico. Association for Computational Linguistics. find
  20. Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. 2025. Trust me, I’m wrong: LLMs hallucinate with certainty despite knowing the answer. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14665– 14688, Suzhou, China. Association for Computational Linguistics. find
  21. Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818–8847, Miami, Florida, USA. Association for Computational Linguistics.
  22. Maksym Taranukhin and Vered Shwartz. 2026. Legal llm hallucination should be evaluated as failure of legal warrant. In AI for Law Workshop @ ICML. ↗
  23. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  24. Benedikt J. Wagner. 2026. Two axes of llm abstention: Answer correctness and question answerability. Preprint, arXiv:2607.08456. ↗
  25. Rui Wang, Hongru Wang, Fei Mi, Boyang Xue, Yi Chen, Kam-Fai Wong, and Ruifeng Xu. 2024. Enhancing large language models against inductive instructions with dual-critique prompting. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5345–5363, Mexico City, Mexico. Association for Computational Linguistics. ↗
  26. Zijie Wang and Eduardo Blanco. 2025. Identifying and answering questions with false assumptions: An interpretable approach. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24069–24087, Suzhou, China. Association for Computational Linguistics. ↗
  27. Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. CREPE: Open-domain question answering with false presuppositions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10457–10480, Toronto, Canada. Association for Computational Linguistics. ↗
  28. Hongbang Yuan, Pengfei Cao, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao. 2024. Whispers that shake foundations: Analyzing and mitigating false premise hallucinations in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2670–2683, Miami, Florida, USA. Association for Computational Linguistics. find
  29. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. Preprint, arXiv:2506.05176. ↗
  30. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. Preprint, arXiv:2405.01470. ↗
  31. Wang Bill Zhu, Tian qi Chen, Xinyan Velocity Yu, Ching Ying Lin, Jade Law, Mazen Jizzini, Jorge J. Nieva, Ruishan Liu, and Robin Jia. 2026. Cancer-myth: Evaluating large language models on patient questions with false presuppositions. In The Fourteenth International Conference on Learning Representations.
  32. Yanxu Zhu, Jinlin Xiao, Yuhang Wang, and Jitao Sang. 2025. KG-FPQ: Evaluating factuality hallucination in LLMs with knowledge graph-based false premise questions. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10472–10490, Abu Dhabi, UAE. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-don-t-well-actually-me/