📄 A Corpus of Scope-disambiguated English Text

Summary

Earlier attempts to build data for statistical quantifier scope disambiguation were narrow. They looked only at sentences with exactly two quantifiers drawn from a fixed list, and they ignored definites, bare plurals, proper nouns, negation and other operators. The authors build the first moderately sized corpus with complete scope information. It uses a domain where scoping is intuitive and matters for understanding: natural-language descriptions of plain-text editing tasks, of the kind handled by SED, AWK or EMACS.

Sentences come from three sources: web one-liners, online tutorials and CS graduate students. The set was then expanded through Mechanical Turk, where workers described a task after seeing two input/output examples. Texts were parsed with the Stanford PCFG parser and chunked, and the chunks were corrected by hand. Annotators then marked pairwise outscoping, coreference and distributivity for every scopal element.

Agreement is measured by comparing transitive closures of scope DAGs. It reaches kappa 75% at the constraint level, while prior work reported 52%. This gives statistical scope resolution a realistic training and test resource.

Key points

• The corpus has about 500 core sentences plus about 2000 collected via Mechanical Turk, roughly 2500 in total. Only the core 500 were annotated at the time of the paper.
• Sentences average 3.9 scopal terms, and 95% of these are NPs. The remainder are negations (marked N#) and other operators such as "or" and "possibly" (marked O#).
• Annotation has three parts:
  • pairwise outscoping (i>j);
  • coreference (i=j);
  • no interaction, which is left unmarked.
• Plurals get an extra distributive index "id", which can apply in some constraints but not in others.
• A subset of 100 sentences was labelled by three annotators. Agreement was:
  • kappa 75.0% at the constraint level and 66% at the sentence level;
  • 92.3% and 89% under the relaxed κ-EZ measure, which treats NI as compatible with any label.
• The three labels WS, NS and NI are close to evenly distributed, about 33% each. By contrast, Higgins and Sadock (2003) had 890 WSJ sentences with more than 60% no-interaction labels and 52% IAA.

Interesting ideas

• I1. Showing crowd workers two input/output examples rules out trivial descriptions, and a wrong-but-valid description is kept as a new task, so the corpus grows by bootstrapping.
• I2. Annotators are told to leave pairs unscoped when both orders are logically equivalent, which removes spurious disagreement caused by intuition rather than meaning.
• I3. Scope is compared by turning each scoping into a DAG, taking its transitive closure, and scoring every chunk pair, with a relaxed variant that accepts NI against anything.
• I4. Collective versus distributive reading is decided per constraint rather than per sentence, with an implicit universal attached to each plural.
• I5. Choosing a domain where getting scope right is required to finish the task makes human judgments consistent.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver data for the maths word-problem readers could be gathered by showing an LLM teacher or humans two worked instances and harvesting alternative descriptions as new items.
• I2 (frames): When the scope or order between two existentials or two universals is irrelevant, frames should mark it "no interaction" instead of choosing arbitrarily, so witness comparisons don't count false conflicts.
• I3 (learn): The same closure-based pairwise score, strict and relaxed, could evaluate quantifier/negation ordering in the meaning graph against gold data or between witness parsers.
• I4 (world): A plural entity in the world could be stored as a set with an optional distributive quantifier, used only in the relations that need it, e.g. "lines that don't start with a digit".
• I5 (answer): Maths word problems and instruction-like texts could serve as a first testbed for scope handling, because the correct answer reveals the intended reading.