📄 Syntactic Testsuites and Textual Entailment Recognition

Summary

Standard RTE benchmarks mix many phenomena: anaphora, temporal reasoning, world knowledge and more. Because of this, they cannot show how well a system handles any single one. The authors isolate entailments that depend only on syntax and propose an evaluation method built on two parts.

The first part is a testsuite generated automatically. Hand-written XML syntactic patterns and a small base lexicon produce sentence variants such as active, passive, relativised, nominalised and coordinated forms. Each sentence carries predicate–argument triples, and the entailment label is computed by comparing these triples. Every sentence also keeps tags recording the constructions used to build it.

The second part applies error mining, a technique borrowed from grammar debugging. It ranks the tag pairs that most often co-occur with the system's mistakes.

Applied to the Afazio system on 10,000 pairs, the method gives 73.2% accuracy and reveals many missed true entailments. The mining points to noun/verb alternations and coordination as the main weaknesses. The broader value is a diagnostic tool that tells developers what to fix, not just a single score.

Key points

• Testsuite design:
  • 10 sub-suites of 1,000 items each, with true and false entailments balanced in each.
  • Sentences have up to three clauses.
  • Predicates cover 10 subcategorisation types, combined differently in each sub-suite.
• Afazio pipeline: the Stanford parser, then cascaded rewrite modules that normalise paraphrases (voice changes, redistributions, noun/verb variants) into one semantic form, then automated first-order reasoning.
• Overall results:
  • 7,317 correct, 2,515 incorrect, and 168 cases where the reasoner gave no answer.
  • On gold-true pairs, 2,482 were wrong, roughly half. On gold-false pairs, only 33 were wrong.
• Scoring method: errors are ranked with the suspicion rate of Sagot and de La Clergerie (2006), weighted by the log of the number of error pairs. Failures on true pairs and on false pairs are mined separately.
• Top finding: a passive relativised verb in H paired with an active coordinated verb in T fails in about 61% of its 65 cases, an expected gain of about 37 pairs if fixed. The rare false-pair failures cluster around relativised and PP arguments, pointing to PP attachment and relative pronoun resolution.

Interesting ideas

• I1. Generate entailment pairs from a pattern grammar plus a lexicon, and derive gold labels by comparing predicate–argument triples, so that labels are computed rather than annotated by hand.
• I2. Have every generated item carry tags naming the constructions it contains (voice, relativisation, nominal vs verbal predicate, coordination), so that failures can later be traced to specific phenomena.
• I3. Apply grammar-debugging error mining (suspicion rate) to a downstream semantic task, ranking feature pairs from text and hypothesis that co-occur with failures.
• I4. Rank suspects by suspicion times the log of the error count, which balances how strongly a feature predicts failure against how much fixing it would gain.
• I5. Mine the two error directions (missed entailments vs spurious ones) separately, because system biases make them qualitatively different.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): A similar generator could build controlled "who did what to whom" question sets in which only the syntactic form varies, testing whether readers answer the same way across forms.
• I2 (learn): Evaluation sets could carry such construction tags, letting regressions be explained by phenomenon rather than by a single aggregate number.
• I3 (frames): Mining mismatched frame outputs by the pair of UD constructions involved would show which alternations, such as passive vs active or nominalisation vs verb, the frame builder mishandles.
• I4 (learn): When choosing which self-induced repair rule to write next, this score prioritises problems that are both likely real and worth fixing.
• I5 (world): Checks on world-of-text inference could count missed facts and hallucinated facts separately, since each points to different broken components.