📄 Experiments of UNED at the Third Recognising Textual Entailment Challenge

Summary

The UNED team took part in the third PASCAL Recognising Textual Entailment challenge (RTE-3). Their main question was how much named entities help in deciding whether a text entails a hypothesis. Named entities matter a lot here: 82.6% of the test hypotheses contain at least one. The system is built on shallow lexical and syntactic analysis. Freeling supplies lemmas and named-entity tags, and Minipar produces dependency trees. The system then measures n-gram overlap, how much of the hypothesis's dependency branches can be found in the text, and whether every hypothesis entity is covered by some entity in the text. An SVM makes the yes/no decision, and a separate model is trained for each RTE task (IE, IR, QA, SUM). The two submitted runs reached 62.38% and 63.12% overall accuracy. QA was the strongest task at 72%, and IE was the weakest at about 53%. The authors conclude that entity checks are a reliable signal for QA pairs. IE pairs need more than that: the system must also check the relations that link the entities.

Key points

• Entity entailment rules:
  • A proper noun entails another if its string contains the other's string.
  • A time expression entails another if its time range lies inside the other's range.
  • A number entails another if its range is enclosed by the other's range.
  • When these rules fail, entities whose Levenshtein distance is under 20% still count as matching (e.g. Yasser/Yaser/Yasir).
• Dropping the NER type labels entirely, so that any entity can match any other by string containment, did slightly better on the RTE-2 test set (0.614 vs 0.610). It also avoids errors where the tagger labels "1990" as a date in the text but as a number in the hypothesis.
• Training one SVM per task beat a single shared model:
  • Cross-validation on the RTE-3 development set: 0.67/0.66 vs 0.64/0.62.
  • RTE-3 test set: 0.62/0.64 vs 0.59/0.60.
• Results by task:
  • QA reached 72% in both runs. In run 2 this came from the entity-coverage feature alone.
  • In SUM pairs, 75.5% of hypothesis nodes fall inside matching branches, against roughly 6–8.5% for the other tasks. This explains why adding the dependency features helped SUM.
• Most IE and QA hypotheses contain named entities (98.5% and 97%). Even so, entity coverage did not help IE. In the failing IE pairs every entity is matched, but the relation between them differs, for example "took the side of" vs "was an opponent of", or an active/passive swap.

Interesting ideas

• I1. Each entity type gets its own entailment test: substring containment for names, interval inclusion for dates, range enclosure for numbers. This turns entity matching into a directional check rather than a similarity score.
• I2. When a tagger's type labels are noisy, the authors ignored them and matched entities on their strings alone. Because text and hypothesis are close to each other, the same expression should behave the same way in both.
• I3. A "matching branch" is a root-to-leaf path in the hypothesis tree where every node is lexically entailed by the text. The share of hypothesis nodes covered by such branches gives an explainable measure of how much of one tree is included in the other.
• I4. The authors compared tasks by how often the hypothesis tree is included in the text tree. This showed that syntactic matching only pays off when text and hypothesis are phrased in parallel (SUM).
• I5. The error analysis shows that matching all the entities while missing the relation between them produces false positives. Typical cases are an active/passive swap or a different predicate.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Dates and quantities in the world of the text could be stored as ranges, so that "in 1986" is answered by checking that the event's interval lies inside the year.
• I2 (tokens): When the hypothesis and the text disagree on the category of the same surface string, the system could reconcile the tags instead of trusting each tag separately.
• I3 (answer): A reader judging a yes/no question could report which branches of the question tree are supported by the story tree and which branches broke the match.
• I4 (learn): Evaluation sets could be split by the degree of paraphrase between text and question, so that a tree-matching component is not credited or blamed on data where it cannot apply.
• I5 (frames): Entailment should be checked over who-did-what-to-whom frames with semantic roles, so that "X defeated Y" never matches "X was defeated by Y".