📄 Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner

Summary

Building systems for Recognizing Textual Entailment (RTE) is hampered by two things. The evaluation conditions change from one RTE Challenge to the next, so experience does not carry over well. Developers also have no easy way to judge how good their training-set results really are before submitting. The authors extend the open-source EDITS package with EDITS-GA. It treats system configuration as a search problem and uses a genetic algorithm (via JGAP) to explore combinations of components: algorithms, cost schemes, rules and optimizers. Each candidate is trained and scored on the development data with a user-chosen fitness measure. They ran it on the "Main" task data of RTE1–RTE6. The automatically found configurations beat EDITS's own hand-picked official submissions by up to 4.51% and always scored above the participants' average. Each search took minutes for most datasets and under two hours for the largest. The result is a cheap, fast baseline, or "sparring partner", that developers can use to place their own system's numbers in context.

Key points

• Search space: configurations are encoded as 18 boolean switches, giving 2^18 = 262,144 combinations.

  • Exhaustive training was estimated at about 4 days even on the small datasets.
  • The genetic algorithm evaluates around 400 configurations per dataset, with a maximum of 20 iterations.

• Fitness measure: accuracy for RTE1–RTE5 and F-measure for RTE6, following the official metric of each challenge.

• Preprocessing:

  • Lucene's tokenizer and stemmer are applied automatically to raw data.
  • The Stanford parser plugin supplies trees for the syntax-based algorithms, such as tree edit distance.

• EDITS-GA results (rank among participants):

  Dataset	Score	Rank
  RTE1	0.5787	3rd
  RTE2	0.6225	5th
  RTE3	0.6875	4th
  RTE4	0.595	10th
  RTE5	0.6233	9th
  RTE6	0.4673	3rd

  • Relative gains over the official EDITS runs were +2.89% to +4.51%; RTE3 has no official EDITS entry.

• What changed on RTE6: the hand-picked run used lexical overlap with a Levenshtein-based cost for unmatched hypothesis words. The search instead chose an IDF-based cost scheme and turned on stop-word filtering, which gave the 4.51% gain.

Interesting ideas

• I1. Represent every optional component and parameter of a pipeline as an on/off gene, then let a genetic search over training-set scores pick the combination; this beats both exhaustive search and expert guessing when the space is large.
• I2. An automatically tuned, freely available system serves as a "sparring partner": a quick reference score telling you whether your own sophisticated approach is actually better than a well-configured simple one.
• I3. Small, seemingly minor configuration choices (cost scheme, stop-word filtering) caused gaps of several points, so hand-chosen "reasonable" settings should not be trusted without checking.
• I4. The fitness function is left to the user and should follow the target evaluation metric (accuracy vs. F-measure), so the search optimizes what will actually be judged.
• I5. Making the search cheap (lightweight default preprocessing, bounded iterations) keeps tuning feasible under deadlines.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's many optional repair rules, witness-parser votes and feature toggles could be encoded the same way and tuned automatically against a dev set instead of by intuition.
• I2 (learn): every new reader or rule set could be compared against an auto-tuned shallow baseline on the same data, as a sanity check and a kind of negative control.
• I3 (frames): role-assignment and coreference heuristics with thresholds or cost choices should be swept systematically, since tiny switches may move accuracy noticeably.
• I4 (answer): when tuning the school-test or fable-QA readers, the search objective should match each test's real scoring, not a generic proxy.
• I5 (tree): parser hyperparameter or dialect-conversion option searches could use a fast tokenize-and-lemmatize path and a fixed generation budget so that tuning runs finish in minutes.