📄 Referential ambiguity and clarification requests: comparing human and LLM behaviour

Summary

This paper asks whether ambiguous references actually lead people to ask clarification questions, and whether LLMs behave the same way. The authors build on the Minecraft Dialogue Corpus, in which an Architect directs a Builder in a voxel world. They combine two existing annotation efforts into one aligned MMAX resource: MDC-R, which marks reference and referential ambiguity, and MSDC, which marks SDRT discourse relations, including clarification and confirmation requests. They then ask several LLMs to produce the Builder's turn at points of possible uncertainty and compare these turns with what the original human Builders said.

The link between referential ambiguity and human clarification turns out to be weak. People rarely question an ambiguous reference. They usually act on a guess and later check task-level details. LLMs show the reverse pattern, and they agree poorly with humans. Models built for reasoning, and older models given Chain of Thought prompts, ask more questions and more relevant ones. This suggests that the ability to clarify depends on simulated reasoning, a link the authors connect to metacognition and Theory of Mind.

Key points

• Source corpus: MDC contains 509 dialogues set in an 11 × 9 × 11 Malmo-based world.

  • MDC-R covers 100 of these dialogues, with reference annotated by experts under the ARRAU guidelines.
  • MSDC provides exhaustive SDRT annotation.

• Merging the corpora: a token-level script aligned the two corpora. It adds a new MMAX layer called SDRT. Each markable in this layer is an Elementary Discourse Unit, and a to attribute holds the ID of the related markable.

• Ambiguity types studied:

  • discourse deixis, which points back into the conversation;
  • spatial deixis, which points into the voxel world.

  A clarification question appears in roughly 77% of dialogues, while a confirmation question appears in roughly 75%.

• Models compared:

  • llama2:13b in three versions: plain, zero-shot CoT (COTZERO) and one-shot CoT (COTONE);
  • deepseek-r1:8b;
  • llama3.2:3b;
  • llama3.3:70b.

• Relevance result: relevance was scored only on spatial-deixis ambiguity.

  • Human Builders asked no relevant questions there.
  • Plain llama2 also asked none.
  • CoT variants and newer models asked a few, with llama3.3:70b asking the most (5).

Interesting ideas

• I1. Two independently made annotation layers on the same text (reference and discourse relations) are merged into one format through token-level alignment, so phenomena can be studied in relation to each other.
• I2. The paper separates uncertainty about what a phrase refers to from uncertainty about what the task requires, and finds that humans react to the second far more than the first.
• I3. Relevance is judged only on the most literal category, spatial deixis, because there it can be checked objectively whether a question targets the annotated ambiguity.
• I4. People often act first and then ask for confirmation, rather than asking before acting, so the reference behaviour is a strategy and not simply "ask when unsure".
• I5. Zero-shot or one-shot CoT prompting let an older model approach the questioning behaviour of reasoning-tuned models.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): coreference chains and speech-act or discourse links could share one token-anchored store, which would let prag check whether an unresolved referent coincides with a clarification move.
• I2 (world): the world builder could record competing referent candidates as an explicit ambiguity state instead of silently choosing one, and say which kind of uncertainty it is.
• I3 (learn): when evaluating a fuzzy behaviour such as asking, abstaining or flagging, start with the subset where correctness is mechanically checkable and use it as a trustworthy anchor metric.
• I4 (answer): readers could commit to the best reading but attach a confirm/verify note naming the assumed referent, keeping the answer explainable.
• I5 (learn): when an LLM teacher labels referents, it could be prompted to reason step by step and to state when it cannot decide, giving silver data with honest "ambiguous" labels instead of forced guesses.