📄 Machine Learning Based Semantic Inference: Experiments and Observations at RTE-3

Summary

Li et al. describe a Georgia Tech entry to the RTE-3 challenge. The task is to decide whether a hypothesis (H) follows from a text (T). Each T-H pair is turned into seven features, and a WEKA classifier learns from them.

The features are:

• WordNet-based lexical semantic similarity weighted by IDF (LSS)
• named-entity coverage (NE)
• coverage of dependent content word pairs (DEP)
• average gap between mapped tokens (DIST)
• negation parity (NG)
• task label (TK)
• text length (LEN)

The pipeline draws on OAK, WordNet::SenseRelate, WordNet::Similarity, the C&C parser and a MEAD IDF dictionary. A J48 decision tree beat the other learners tried. The two official runs reached accuracy 0.64 and 0.6488, and LSS and NE carried most of the signal.

The authors also show that a trivial classifier using the per-task label distribution already scores 0.545, so RTE-3 is easier than earlier rounds. They argue that the official Average Precision measure is unstable when confidences are tied and is not comparable across datasets. They propose a tie-aware ranking procedure and a normalized variant. The analysis matters as much as the system: it shows how much of the reported progress strong baselines and sound metrics can explain.

Key points

• Data: the RTE-3 development and test sets each hold 800 pairs, tagged by task (IE, IR, QA, SUM) and by length (short/long).
• Baselines: answering "YES" for everything gives 0.5125. Matching the per-task (or per-task-and-length) YES/NO distribution gives 0.5450.
• Single features: LSS alone gets 0.6263 and NE alone 0.6250. NE alone reaches 0.78 on QA pairs.
• Feature combinations: the best set was LSS+NE+TK+LEN+DIST (run 2, 0.6488). Using all features dropped accuracy to 0.6275.
• Weakest task: IE pairs were hardest, scoring barely above the 0.525 IE baseline. High word overlap pushed the system to call almost all of them entailed.
• Metric fixes: on the RTE-3 test set, Average Precision can fall no lower than 0.3172. The authors propose subtracting this bound and rescaling to get a normalized AvgP.

Interesting ideas

• I1. Measure how much of the hypothesis is covered by the text: the share of H's named entities or head–dependent content-word pairs found in T. An entity counts as matched if a T entity contains it, and an empty set scores 1, which makes this a cheap directional check for missing information.
• I2. Build dummy baselines that know only the label distribution per task and length bucket, and report every result against them. On RTE-3 such a baseline was already well above 50%.
• I3. When ranking by confidence, place negatives ahead of positives within tied groups so ties earn no lucky credit, then rescale Average Precision between its worst-case floor and 1. This makes scores stable and comparable across test sets with different positive rates.
• I4. Score alignment looseness by the average gap between consecutive matched tokens in T, rather than the total count of unmatched tokens. Long appositives then do not by themselves rule out entailment.
• I5. A feature that beats the baseline alone can still hurt inside a combination (DEP cost 1.5% when added to LSS+NE+TK+LEN). Ablations therefore need to be run on combinations, not only on single features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): a reader could test, before answering, whether each entity and each head–dependent link in a candidate answer is actually grounded in the world built from the text, and refuse when coverage is incomplete.
• I2 (learn): add per-category majority-label scorers as negative controls in evaluation suites, so gains on school tests or fable QA are not mostly skew in the label distribution.
• I3 (learn): apply tie-pessimistic ordering and floor-normalized AvgP whenever comparing confidence-ranked outputs, such as candidate parses or repair rules, across differently balanced evaluation sets.
• I4 (frames): when matching a question's frame against text frames, measure distance along the dependency path but skip appositive and parenthetical subtrees, so inserted descriptions do not break role matching.
• I5 (learn): when adding a new witness parser or repair rule, measure it by leave-one-out against the full system rather than in isolation before keeping it.