📄 UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features

Summary

The paper describes UdL, a system for Track 5 of SemEval-2017 Task 1. The track asks for a 0–5 semantic similarity score for each English sentence pair, and systems are scored by Pearson correlation against hidden human ratings on 250 test pairs. The authors trained on the STS datasets released since 2012, plus aggregated data shared by an earlier team. The pipeline has two stages. A pairwise transformer first turns each sentence pair into a small set of comparison features: word-vector alignment of tokens sharing the same PoS or NE tag, cosine similarity of tf–idf vectors built from character n-grams, a numeric-content difference, a length difference, and an optional domain label. A random forest regressor then maps these features to a score. The best run reached 0.8004, above the track baseline (0.7278) and median (0.7775). The authors present the comparator-style setup as a first step toward learning sentence representations from pairs. They also report that the shallow character n-gram feature carried most of the model's predictive weight.

Key points

• Features and their importance in the random forest:
  • Aligned PoS tags (17 tags): 0.113
  • Aligned NE tags (10 tags): 0.003
  • tf–idf char 2–3-gram BoW cosine: 0.847
  • Number-sum difference: 0.006
  • Character-length difference: 0.032
• Regressor choice: Random forest (1024 trees, max depth 8) beat Lasso CV (512 iterations) in development, with a mean of 0.81 versus 0.79 on images, answers-students and headlines 2016.
• Official runs:
  • Run1 (all features except domain class): 0.8004
  • Run2 (with the domain feature, all test pairs labeled "Image-captions"): 0.7805
  • Run3 (hashtags removed from tweet-news training pairs): 0.7901
  • Best system in the track: 0.8547
• Tooling: polyglot and spaCy for tagging, 300-dimensional GloVe and spaCy vectors, a scikit-learn pipeline, and code released at github.com/natsheh/sensim.
• Data cleaning: Tab separators that also appeared inside sentences required manual repair. A spell-correction step with Enchant was tested but not used in the submissions.

Interesting ideas

• I1. Tokens are aligned only when they carry the same PoS or NE tag and their vector similarity passes a threshold, with competing matches settled by the higher score; restricting matches by category keeps the comparison structured rather than a loose overlap of words.
• I2. Spelled-out numbers are first converted to numeric values (e.g. "sixty-five" to 65), and the pair is then compared on numeric content as its own signal.
• I3. The feature-importance analysis revealed that a shallow character n-gram cosine dominated the more linguistic features, which shows how easily surface overlap can explain most of a benchmark score.
• I4. Learning a comparator over pairs is proposed as a route to good single-item representations, much as face verification is trained on same/different pairs, because pairs are far more plentiful than labeled examples.
• I5. Adding a domain feature whose test value had to be guessed by hand lowered the score (0.7805 vs 0.8004), a reminder that metadata unavailable or unreliable at test time can hurt more than help.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Two frames could be compared by aligning their arguments only within matching semantic roles, with conflicts resolved greedily by similarity, to give an explainable paraphrase check.
• I2 (world): Quantities in the world of a text could be normalized to values before comparison, which helps consistency checks in maths word problems and fable QA.
• I3 (learn): A char n-gram similarity baseline could serve as a negative control, so that structured tree/frame comparisons must show gains beyond what surface overlap already predicts.
• I4 (learn): Self-play could generate many sentence pairs (paraphrase versus perturbed variant) to tune which tree and frame features matter for meaning equivalence.
• I5 (answer): Readers should not depend on genre or source tags that will be missing or guessed at inference, or should be evaluated with those tags withheld.