📄 DP: A Detector for Presuppositions in survey questions

Summary

Survey questions often take something for granted. For example, asking when someone graduated assumes they did. Respondents for whom the assumption is false may skip the question or give invented answers, and this distorts survey statistics. The authors built DP, a module of the QUAID questionnaire-critiquing tool, to flag such hidden assumptions for survey designers. From a corpus of 550 U.S. Census Bureau questions, expert raters identified 22 problematic items, and a matched set of 22 clean items served as a contrast. Analysing these produced two lists of surface cues: one signals a presupposition, the other signals its absence. The cues include specific words, POS tags, and partial syntactic structures from the Brill tagger and Abney's SCOL/CASS chunk parser. A decision tree combines the cues and labels the presupposition's type, such as habit, possession or event. Against human ratings, DP agreed about as well as the best-agreeing pair of humans. After a revision, it reached perfect recall on questions all raters judged problematic. Detection is deliberately left as advice, and the human decides whether the presupposition is valid.

Key points

• The development data came from a 550-question U.S. Census Bureau corpus. Three expert raters identified 22 problematic questions, and a random contrast sample of 22 unproblematic ones was added to prune cues that did not discriminate.
• The processing pipeline has three stages:
  • Brill's rule-based tagger assigns POS tags.
  • Abney's SCOL/CASS cascaded, bottom-up chunk parser builds partial structures.
  • A decision tree applies the indicator rules, and an exception list handles possessive NPs.
• For evaluation, three raters scored about 90 out-of-context questions from 12 questionnaires on a 4-point scale, which was collapsed to Boolean.
  • Human agreement was weak. The best pair reached r = 0.41, kappa = 0.36 and 67% agreement.
  • DP matched those two raters at r = 0.32/0.31, with 63%/66% agreement.
• Scoring used two gold standards: Pmaj (at least two raters say yes) and Pcomp (all three say yes). The updated DP scored as follows:
  • Pmaj: hit rate 0.75, false alarms 0.44, recall 0.84, precision 0.75, d' 0.8.
  • Pcomp: hit rate 0.90, false alarms 0.52, recall 1.00, precision 0.46, d' 1.3.
• In the revised decision tree, presence cues override absence cues whenever the two conflict. This trades precision for recall, because a human reviews every flag anyway.

Interesting ideas

• I1. Keep two cue lists, one for evidence that a phenomenon is present and one for evidence that it is absent, and make conflict resolution an explicit, tunable priority rather than a learned blur.
• I2. Each detection comes with a typed explanation: the cue fires, it maps to a presupposition category, and the relevant span is inserted into a message template, giving output like "presupposes a possession (child)".
• I3. Score the system against two gold standards, majority agreement and unanimous agreement. If performance is much better on the unanimous items, the cues are capturing the salient, uncontroversial cases.
• I4. Build the cue set from positive examples, then test it on a matched negative sample and drop any cue that fires on both.
• I5. The proposed next step is to suppress a flag when earlier discourse has already established the presupposed entity, which requires looking back an unknown distance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): The prag crate could use paired trigger and anti-trigger lists for presupposition and implicature, with a documented rule deciding which list wins.
• I2 (frames): Presupposition frames could record the trigger, the category and the filled slot, so every inference can be explained.
• I3 (learn): When silver data comes from several witness parsers, results could be reported separately for full-agreement and majority-agreement items, to tell real errors apart from disputed cases.
• I4 (learn): This is a simple negative control for self-induced repair rules: a rule is kept only if it stays silent on a contrast set.
• I5 (world): Before raising a presupposition, the world of the text could be checked for an existing entity or state. If one exists, the presupposition is satisfied; if not, Mova should flag or accommodate it.