Elamur Research⌕
☰
home / papers / 2026 / Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Paper · 2026 · Manar Ali, Judith Sieker, Sina Zarrieß et al.

Summary

The paper asks whether language models can act like cooperative listeners: notice when they are unsure what a speaker means and ask about it instead of guessing. Open dialogue makes this hard to measure, because nobody can say for sure when a question was needed. The authors therefore use reference games, where a listener must pick a described target from a small fixed set and failure is obvious. They use the color-grid dataset of McDowell and Goodman (2019), in which three 3×3 color grids are shown at difficulty levels far, split and close. Three VLMs act as listeners: Qwen2.5-VL-7B, Qwen2.5-VL-72B and GPT-5-mini. The models are tested in three ways. A baseline measures resolution accuracy, with confidence taken from agreement across five samples. A second run allows clarification questions. A third run has a human answer Qwen-72B's questions. GPT-5-mini is accurate and asks more often when uncertain. The Qwen models' questioning is largely unrelated to their uncertainty, and answering their questions barely helps. Even in this simple, controlled task, models seldom turn uncertainty into useful clarification.

Key points

  • Data: 197 games of 60 rounds each. Only the speaker's first description is used. GPT-5-mini was tested on 500 rounds because of API cost, and the Qwen models were tested on all rounds. Humans reach 0.92 accuracy overall (close 0.91, split 0.93, far 0.97).
  • Baseline accuracy on the 500-round subset was 0.53 for Qwen-7B, 0.77 for Qwen-72B and 0.91 for GPT-5-mini. Mean confidence from sampling agreement was high for all three (0.88 / 0.91 / 0.99).
  • Clarification request rates differed sharply:
    • Qwen-7B almost never asks (about 0.002 on the full data).
    • Qwen-72B asks in about 24% of cases, and this rate is flat across difficulty levels.
    • GPT-5-mini asks in 13% of cases, more on harder items (0.17 for close) than on easy ones (0.06 for far).
  • Three scores evaluate the clarification run:
    • CR-Rate: how often the model asks.
    • Accuracy: correctness on the items where the model answered rather than asked.
    • Relaxed accuracy: counts a request as a non-error.
  • In the human-in-the-loop study on Qwen-72B's 116 questions, confidence rose (0.871 → 0.902). Accuracy on those items dropped slightly (0.776 → 0.741), with a gain only in the close condition (+0.081).

Interesting ideas

  • I1. Confidence is estimated by sampling an answer several times and taking the share that agrees with the majority, which gives a simple, model-agnostic uncertainty signal without access to logits.
  • I2. A closed candidate set makes "clarification was needed" a checkable fact rather than a judgment call, so asking behavior can be scored against ground truth.
  • I3. Relaxed accuracy treats an honest "I need more information" as better than a wrong answer, separating abstention quality from raw accuracy.
  • I4. Graded difficulty conditions (far/split/close) reveal whether a system's uncertainty and questioning actually rise as the task gets harder, which is a cheap calibration check.
  • I5. The usefulness of a clarification question is tested by annotating it as task-relevant or not, answering it, and re-running the model to measure the change in accuracy. This shows that asking alone proves nothing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Agreement among witness parsers on each arc or label can serve as a graded confidence score for silver data. Low-agreement items can then be routed to an LLM teacher instead of being trusted.
  • I2 (world): When a referring expression matches several entities in the world of the text, Mova knows the candidate set explicitly. It can flag genuine referential ambiguity and test whether its resolver's uncertainty tracks that ambiguity.
  • I3 (answer): School-test and fable-QA readers could be scored with both strict and relaxed metrics. This rewards explainable abstention when the text truly underdetermines the answer, instead of encouraging confident guessing.
  • I4 (learn): Evaluation sets could be split into controlled difficulty tiers, such as ambiguous vs. unambiguous attachment. This would show whether parser and coreference confidence rises and falls with real difficulty.
  • I5 (back): Clarification questions generated from the meaning graph should point at the specific open variable, such as which of two candidate antecedents is meant. Their value can be checked by whether the answer actually resolves the graph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bielefeld UniversityBielefeld · Germany35 works · 1,196 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Manar Ali, Judith Sieker, Sina Zarrieß and Hendrik Buschmeier. 2026. Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.76

@inproceedings{ali2026reference,
  title = {{Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests}},
  author = {Manar Ali, Judith Sieker, Sina Zarrieß and Hendrik Buschmeier},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.76},
  url = {https://doi.org/10.18653/v1/2026.gem-main.76}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966. ↗
  2. David L. Bimler, John Kirkland, and Kimberly A. Jameson. 2004. Quantifying variations in personal color spaces: Are there sex differences in color vision? Color Research & Application, 29(2):128–134. ↗
  3. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6):1482–1493. ↗
  4. Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge, UK. find
  5. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. ↗
  6. Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. 2023. A diachronic perspective on user trust in AI under uncertainty. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5567– 5580, Singapore. ACL. ↗
  7. Mark Dingemanse and N.J. Enfield. 2023. Interactive repair and the foundations of language. Trends in Cognitive Sciences, 28:30–42. ↗
  8. Mark Dingemanse, Seán G. Roberts, Julija Baranova, Joe Blythe, Paul Drew, Simeon Floyd, Rosa S. Gisladottir, Kobin H. Kendrick, Stephen C. Levinson, Elizabeth Manrique, Giovanni Rossi, and Nick J. Enfield. 2015. Universal principles in the repair of communication problems. PLoS ONE, 10:e0136100. ↗
  9. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. ↗
  10. Simeon Junker, Manar Ali, Larissa Koch, Sina Zarrieß, and Hendrik Buschmeier. 2025. Are multimodal large language models pragmatically competent listeners in simple reference resolution tasks? In Findings of the Association for Computational Linguistics: ACL 2025, pages 24101–24109, Vienna, Austria. ACL. find
  11. Baber Khalid, Malihe Alikhani, and Matthew Stone. 2020. Combining cognitive modeling and reinforcement learning for clarification in dialogue. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4417–4428, Barcelona, Spain (Online). ICLL. find
  12. Clara Lachenmaier, Judith Sieker, and Sina Zarrieß. 2025. Can LLMs ground when they (don’t) know: A study on direct and loaded political questions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 14956– 14975, Vienna, Austria. ACL. find
  13. Chris Madge, Matthew Purver, and Massimo Poesio. 2025. Referential ambiguity and clarification requests: Comparing human and LLM behaviour. In Proceedings of the 8th Workshop on Computational Models of Reference, Anaphora and Coreference, pages 1–11, Suzhou, China. ACL.
  14. Eric Mayor, Lucas M. Bietti, and Adrian Bangerter. 2025. Can large language models simulate spoken human conversations? Cognitive Science, 49(9):e70106. ↗
  15. Bill McDowell and Noah Goodman. 2019. Learning from omission. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 619–628, Florence, Italy. ACL. find
  16. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. ↗
  17. Judith Sieker, Simeon Junker, Ronja Utescher, Nazia Attari, Heiko Wersing, Hendrik Buschmeier, and Sina Zarrieß. 2024. The illusion of competence: Evaluating the effect of explanations on users’ mental models of visual question answering systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19459–19475, Miami, FL, USA. ACL. find
  18. Holger Mitterer and Jan Peter de Ruiter. 2008. Recalibrating color categories using world knowledge. Psychological Science, 19(7):629–634. find
  19. Judith Sieker, Clara Lachenmaier, and Sina Zarrieß. 2025. LLMs struggle to reject false presuppositions when misinformation stakes are high. In Proceedings of the 47th Annual Meeting of the Cognitive Science Society, pages 832–839, San Francisco, CA, USA. ↗
  20. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. ↗
  21. Elizabeth Stokoe, Saul Albert, Hendrik Buschmeier, and Wyke Stommel. 2024. Conversation analysis and conversational technologies: Finding the common ground between academia and industry. Discourse & Communication, 18(6):837–847. ↗
  22. OpenAI. 2025. GPT-5 System Card. find
  23. Alberto Testoni and Raquel Fernández. 2024. Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 258–275, St. Julian’s, Malta. ACL. ↗
  24. Ildikó Pilán, Laurent Prévot, Hendrik Buschmeier, and Pierre Lison. 2024. Conversational feedback in scripted versus spontaneous dialogues: A comparative analysis. In Proceedings of the 25th Meeting of the Special Interest Group on Discourse and Dialogue, pages 440–457, Kyoto, Japan. ACL. find
  25. Ole Pütz and Elena Esposito. 2024. Performance without understanding: How ChatGPT relies on humans to repair conversational trouble. Discourse & Communication, 18(6):859–868. ↗
  26. Qwen Team. 2025. Qwen2.5-VL technical report. Preprint, arXiv:2502.13923. ↗
  27. Neil Rathi, Dan Jurafsky, and Kaitlyn Zhou. 2025. Humans overrely on overconfident language models, across languages. In Second Conference on Language Modeling, Montreal, Canada. ↗
  28. Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. 2024. Grounding gaps in language model generations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6279–6296, Mexico City, Mexico. ACL. ↗
  29. Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. 2025. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 58(3):1–38. ↗
  30. Alberto Testoni, Barbara Plank, and Raquel Fernández. 2025. RAcQUEt: Unveiling the dangers of overlooked referential ambiguity in visual LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23627–23647, Suzhou, China. ACL. find
  31. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transactions of the Association for Computational Linguistics, 13:220–248. ↗
  32. Edda Weigand. 1999. Misunderstanding: The standard case. Journal of Pragmatics, 31:763–785. ↗
  33. Michael J. Q. Zhang and Eunsol Choi. 2025. Clarify when necessary: Resolving ambiguity through interaction with LMs. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5526–5543, Albuquerque, NM, USA. ACL. find
  34. Chenglei Si, Navita Goyal, Tongshuang Wu, Chen Zhao, Shi Feng, Hal Daumé Iii, and Jordan Boyd-Graber. 2024. Large language models help humans verify truthfulness – Except When They Are Convincingly Wrong. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1459–1474, Mexico City, Mexico. ACL. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2026-reference-games-as-testbed-alignment/