📄 UBC: Cubes for English Semantic Textual Similarity and Supervised Approaches for Interpretable STS

Summary

The paper reports UBC's two entries in SemEval 2015 Task 2. For English STS (scoring sentence pairs from 0 to 5), the authors build a "cube": an N×M matrix over the parse-tree nodes of both sentences, covering words and phrases, with one layer per similarity source. The layers are Collobert and Weston vectors composed with a Recursive Autoencoder, averaged Mikolov vectors, and PPDB paraphrase scores. The cube is collapsed by taking the maximum across layers, and the final score comes from the IDF-weighted formula of Mihalcea et al. This single run placed only 54th of 74.

For the Interpretable STS pilot, they built a pipeline:

• chunking, with rule-based fixes to the chunker output;
• token alignment with the Sultan et al. aligner;
• conversion of token links into chunk links;
• Hungarian-Munkres matching, extended to detect 1:N "context" alignments;
• an SVM for relation tags and cube-based regression for scores.

This supervised configuration achieved the top result on the images dataset. The work shows that a modular, explainable alignment pipeline can be competitive.

Key points

• The cube had three layers. The Collobert and Weston layer used RAE for phrase nodes. The Mikolov layer used word vectors summed and divided by phrase length. The PPDB XXXL layer was sparse, averaged both conditional directions, and added extra rows and columns for PPDB phrases that match no tree node.
• The layers were merged by taking the maximum per cell. The authors judged these resources more prone to false negatives than false positives, so a high score from any one layer was trusted.
• In the SYS scenario, the ixa-pipes-chunker (an OpenNLP maximum-entropy model) was post-processed with four regex rules. These rules merge chunks, for example a prepositional phrase with the noun phrase after it, or noun phrases split by punctuation or conjunctions.
• Training data was 750 pairs each for images and headlines; test data was 375 image pairs and 378 headline pairs. The SVM (LIBSVM in Weka) used 21 features, tuned with 5-fold cross-validation and grid search. The features covered Jaccard overlap, segment length, WordNet similarity and depth of chunk heads, and cube values.
• On images test data, run2 reached F1 Type+Score of 0.6159 (GS) and 0.5643 (SYS), beating the baseline by about 0.18–0.19. On headlines SYS, the naive run1 slightly beat run2 (0.5019 vs 0.4705).

Interesting ideas

• I1. Keeping several similarity sources as parallel layers over the same word/phrase grid lets new evidence be added without changing the scorer, and every cell stays traceable to its source.
• I2. Combining layers by maximum is justified by an explicit error assumption: the sources miss more true matches than they invent false ones. The proposed fix for the obvious weakness is to add separate dissimilarity layers for negation and antonymy.
• I3. Token links are summed inside chunk regions to form a chunk-to-chunk weight matrix. Optimal assignment then picks the 1:1 matches, and the runner-up links are kept as "context" alignments, which represents 1:N relations within a 1:1 format.
• I4. A cheap majority-style tagger acts as the baseline, and the learned model only refines the cases the baseline labels "equivalent". This keeps the learned part narrow and easy to check against the default.
• I5. A handful of regex rules that merge chunks (a PP with its NP, NPs joined by conjunctions or punctuation) noticeably improved the task metric. This shows that segment granularity should be tuned to the downstream task, not taken directly from a tagger.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Compiled knowledge (WordNet, FrameNet, Wikidata, measures) could fill separate layers of a mention-to-mention matrix, so any similarity judgment can report which resource supported it.
• I2 (answer): When readers compare an answer option with the text world, a match from any resource could count as support, while negation and antonymy are handled on a separate channel so they cannot be averaged away.
• I3 (frames): Frame arguments in two sentences (or a question and the text) could be aligned this way, with secondary links recorded instead of discarded, which gives an explainable account of paraphrase.
• I4 (learn): Self-induced repair rules or learned classifiers could be scoped to override a deterministic default only in one well-defined case, and evaluated by how much they gain over that default.
• I5 (tree): Phrase spans derived from UD trees could be regrouped by small, auditable rules before alignment or frame building, with each rule's effect measured downstream.