📄 BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

Summary

The BUAP team took part in SemEval 2014 Task 1, which asks systems to judge pairs of full sentences in two ways. The first is a graded relatedness score. The second is an entailment label. Their system turns each word into several kinds of evidence:

• dependency partners (object, subject, property);
• character n-grams and skip-grams;
• an LSA vector trained in R on a home-made corpus built from Europarl, Project Gutenberg and the Open Office Thesaurus.

For each of nine word-level similarity measures, every word of the first sentence is matched to its most similar word in the second sentence, and these best scores are summed. This yields a nine-dimensional feature vector per pair. Weka's logistic regression predicts relatedness. For entailment, the vector is extended with synonym, antonym and negation-cue flags and fed to an SVM.

Relatedness came out slightly below the task average, at rank 12 of 17. Entailment did better, at rank 7 of 18. The authors credit this gain to the lexical-relation and negation features. The work shows that cheap, hand-built similarity features can be competitive on sentence-pair tasks.

Key points

• Features f1–f3 compare dependency partners (object/subject/property). Each takes the best of six NLTK WordNet metrics: Leacock & Chodorow, Lesk, Wu & Palmer, Resnik, Lin and Jiang & Conrath.
• Features f4–f5 apply cosine similarity to character n-grams (n=2–5) and skip-grams (n=2–5, skip r=2–5). Features f6–f9 compare LSA vectors using the Mihalcea measure, cosine, Euclidean distance and Chebyshev distance.
• Relatedness results with logistic regression (Weka defaults) were Pearson 0.697, Spearman 0.645 and MSE 0.528. The averages across all runs were 0.719, 0.672 and 0.639.
• For entailment, the SVM (Weka defaults) added several binary features:
  • WordNet synonyms;
  • antonyms collected from Wikipedia;
  • the negation cues "no", "not", "nobody" and "none".
• The entailment run placed 7th of 18 teams. The authors plan to reuse the entailment-only features for relatedness.

Interesting ideas

• I1. Scoring a sentence pair by greedily aligning each word to its best partner and summing gives an easy alignment-based similarity that can work with any word-level metric.
• I2. Instead of committing to one WordNet metric, take the maximum over six of them, so whichever metric fits a given word pair wins.
• I3. Comparing words through their dependency partners (subject, object, property) brings sentence structure into the similarity, beyond what lexical overlap captures.
• I4. A handful of negation-cue and antonym flags noticeably improved entailment over purely similarity-based features, because related sentences can still contradict each other.
• I5. The gap between the two subtasks acts as an informal ablation, isolating what the lexical-relation and negation features contribute.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Comparing a question with candidate answer sentences could use this greedy alignment, but over UD tree nodes rather than raw words, so each matched pair stays inspectable as an explanation.
• I2 (world): When the WordNet-derived ontology is queried for concept similarity, an ensemble-max over several path- and information-content-based measures could be cached alongside the record of which measure produced the value.
• I3 (frames): Frame matching between two texts could score role fillers (agent, patient, attribute) against each other separately, which mirrors these f1–f3 features in a frame-level form.
• I4 (answer): Readers judging whether a statement follows from a fable should explicitly check polarity and antonymy between aligned frames before trusting a high similarity score.
• I5 (learn): Evaluations could routinely run the same pipeline with and without a feature group on paired tasks, which treats the gap as evidence of what that group contributes.