📄 Are You Asking the Right Questions? Teaching Machines to Ask Clarification Questions

Summary

The paper tackles automatically choosing a good clarification question for an underspecified request, using technical help forums as a testbed. The author mines StackExchange (askubuntu, unix, superuser) to build about 37K triples. Each triple pairs an original post, a question someone left in its comments, and the later edit in which the author supplied the missing detail. The model borrows the decision-theoretic idea of expected value of perfect information (EVPI): a question is worth asking if the answers it is likely to receive would make the post more complete. Two jointly trained neural parts implement this, both using LSTM encoders over GloVe embeddings. An answer model estimates how plausible each candidate answer is for a question. A utility model scores how complete the post becomes once an answer is added. Candidates come from ten similar posts retrieved with Lucene. EVPI beats random, bag-of-ngrams and feed-forward baselines, but the task remains hard even for people. The work matters because it treats question asking as reasoning about missing information rather than generating text from a template.

Key points

• Dataset construction. The dataset contains ~37K (post, question, answer) triples from askubuntu, unix and superuser. A comment is linked to the edit that answers it by cosine similarity of averaged word embeddings.
• Candidate retrieval. Lucene retrieves 10 similar posts. Their comment questions become the question candidates, and their follow-up edits become the answer candidates.
• Answer model. Answer probability decays exponentially with the squared distance between a candidate answer and a predicted answer vector.
• Utility model. It is a sigmoid feed-forward classifier. The true answer counts as a positive, and nine retrieved answers count as negatives.
• Hard setting results. With Lucene-chosen distractors, EVPI reaches 23.3 accuracy, 43.4 MRR, 51.0 R@3 and 70.3 R@5. The feed-forward baseline scores 17.4 accuracy and the bag-of-ngrams baseline 11.6.
• Easy setting results. With random distractors, EVPI reaches 61.1 accuracy, and here bag-of-ngrams (54.9) beats feed-forward (49.0).
• Human comparison. 14 CS graduate students labelled 50 hard examples and reached only 36% accuracy. On items where they reported high confidence, their accuracy rose to 42%.

Interesting ideas

• I1. EVPI question selection: Rank possible questions by the usefulness of their likely answers, weighted by how probable each answer is, instead of scoring the question text alone. This gives a principled way to decide what to ask when information is missing.
• I2. Edit history as free supervision: Revision logs naturally pair a question with the exact text that resolved it, which yields aligned training data without manual annotation.
• I3. Hard versus random distractors: Reporting results with retrieved look-alike negatives and with random negatives side by side shows how much of a score comes from easy surface cues. In this paper, the two settings even reversed the order of the baselines.
• I4. Soft targets from paraphrased questions: The answer loss also pulls the prediction toward answers of near-duplicate questions, weighted by how similar those questions are. This lets one reformulation borrow supervision from the others.
• I5. Slot templates filled from context: The proposed generator clusters questions, strips topic words to form templates, and refills the slots with terms from the current post. A related variant switches at each step between ordinary vocabulary and post-specific words.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): When the world of a text has unfilled slots, such as an unknown quantity or an unresolved owner, Mova could pick the clarification whose expected answers would settle the most downstream states. It could also record the expected-value calculation as the explanation for that choice.
• I2 (learn): Wiki or forum revision histories could supply silver pairs of "gap → filled information" for training a detector of underspecified sentences.
• I3 (learn): Evaluation could routinely pair each test with retrieval-based hard negatives as a negative control. This would expose readers that succeed only on lexical overlap.
• I4 (answer): Training for QA on fables could share credit across paraphrased school-test questions so the reader does not overfit a single wording.
• I5 (back): Generation from the meaning graph could use frame-level question templates whose slots are filled from entities in the world of the text, which keeps every output traceable.