📄 ClarVis: A Dataset of Clarification Requests and Grounding in Collaborative Data Visualization Dialogues

Summary

Clarification requests (CRs) are how dialogue partners repair or check shared understanding. Most existing CR resources, however, come from search, forum or QA settings that are asynchronous or single-turn and lack a shared, changing task context. The authors build ClarVis from 29 unscripted sessions in which pairs of users explored a COVID-19 dataset together while a conversational assistant produced charts and maps on a shared screen. Following Benotti and Blackburn's grounded view of CRs, annotators worked in two passes. They first marked each utterance as CR or Rest by its function rather than its surface form. They then assigned each CR a grounding modality: Auditory, Visual or Kinesthetic. Two benchmark tasks are defined: CR identification and grounding-modality classification. Both are tested with classic ML baselines and Llama-3.2-3B-Instruct in zero-shot and few-shot settings, with and without preceding turns. The labels can be partly learned from text, and Visual CRs are the easiest. Auditory and Kinesthetic CRs remain hard, which suggests that visual and interaction state is needed beyond the transcript.

Key points

• The data are 29 dialogues with over 6.3K user utterances (6.9K turns in total), taken from an earlier user study. They are released on GitHub with transcripts, visualization captions, labels and train/validation/test splits.
• Modalities are A (mishearing, ASR errors), V (what the display shows) and K (actions, system capabilities, whether data exist). The socioperceptual category from the SCARE scheme is dropped. When cues are genuinely tied, the precedence rule K > V > A applies.
• Questions that introduce a new analytic goal or ask for a new chart are labeled Rest, not CR.
• Llama-3.2-3B-Instruct reaches very high recall but low precision on CR detection, because it reacts to words like "what", "how" and "can". For modality classification, its macro F1 is about 0.36–0.41 on gold CRs, and Kinesthetic F1 is often near zero in the pipeline setting.
• The error analysis identifies three recurring problems: overlap between Visual and Kinesthetic readings, vague deictic references such as "this one?", and rare, minimal Auditory cues such as "What?".

Interesting ideas

• I1. Each CR is labeled by which channel the misunderstanding comes from (hearing, seeing, acting), and a fixed priority order resolves genuine ties, which keeps the labels consistent.
• I2. The paper scores modality prediction twice: once on gold CRs and once as a full pipeline after automatic CR detection, which separates upstream detection errors from downstream classification errors.
• I3. CRs are defined by their function, and goal-introducing questions are deliberately treated as a negative class, which exposes models that fire on wh-words alone.
• I4. The authors suggest building contrastive few-shot pairs from near-identical utterances that carry different labels; for instance, "What does this map show?" is labeled V, whereas "Can it show poverty by region?" is labeled K.
• I5. The study adds context step by step (no context, +Prev1, +Prev2) and finds that more history does not always help.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): A clarification speech act in prag could carry a "source" slot (utterance, displayed or world state, or action/capability), with a documented tie-break order that keeps the label explainable.
• I2 (learn): Every stage, such as tree → frames → world, could be evaluated both on gold input and on predicted input, so that a regression can be attributed to the step that caused it.
• I3 (frames): The question/CR detector should check whether the utterance points back to prior content, for example through anaphora to an earlier turn, rather than trigger on interrogative surface form; Rest-style questions make good negative controls.
• I4 (learn): LLM teacher prompts and silver-data checks could use minimal pairs of this kind to sharpen boundaries between frames or speech-act types.
• I5 (world): Elliptical repairs like "What's the same?" should be resolved against the world-of-text state rather than a raw window of earlier turns, and the useful context depth should be measured, not assumed.