📄 scaLAR SemEval-2024 Task 1: Semantic Textual Relatednes for English

Summary

The paper describes an English submission to SemEval-2024 Task 1, which asks systems to score how related two sentences are. Relatedness here is broader than similarity. Two sentences can be related by topic, viewpoint, time of origin, or because one elaborates on the other, not only through paraphrase or entailment. The authors used the task's English data: 5,500 scored training pairs and 250 test pairs, with both sentences stored in a single text field. They encoded each sentence with the Sentence-Transformers model bert-base-nli-mean-tokens. For the unsupervised track (Track B), the score was simply the cosine of the two embeddings. For the supervised track (Track A), they briefly fine-tuned the same model on the training pairs. Performance was measured with Spearman rank correlation against human scores. Fine-tuning reached 0.803, a little below the official baseline of 0.830. The untuned cosine approach reached 0.693, which the authors report as better than the unsupervised baseline. The work matters mainly as a cheap, reproducible reference point that ran entirely on free Google Colab.

Key points

• Data: the SemEval-2024 STR English set has 5,500 training pairs (PairID, Text, Score) and 250 test pairs (PairID, Text), with the two sentences in each pair split by a newline marker.
• Encoder: bert-base-nli-mean-tokens, which mean-pools token vectors into 768-dimensional sentence embeddings and was trained on NLI data.
• Fine-tuning setup: batch size 16, a single epoch, CosineSimilarityLoss, and the Adam optimizer.
• Results (Spearman): 0.803 for supervised Track A versus the 0.830 baseline, and 0.693 for unsupervised Track B using cosine with no tuning.
• The authors themselves note that this encoder is outdated because of weak embeddings. They suggest LLM zero-shot scoring and extra training data as next steps.

Interesting ideas

• I1. Running the exact same encoder twice, once frozen and once lightly tuned, isolates how much the task labels contribute, here roughly 0.11 Spearman from one epoch.
• I2. The task separates relatedness (same topic, same perspective, same time frame, one sentence continuing another) from strict similarity (paraphrase or entailment). This gives a checklist of distinct relation types rather than a single fuzzy score.
• I3. Spearman correlation compares only rankings, so a system can be judged against human scores without its outputs sharing their scale.
• I4. Mean-pooled token embeddings with cosine are essentially a dense bag-of-words, yet they still reach 0.693. That makes them a natural floor for any structure-aware method.
• I5. Sentence pairs packed into one field and split on a delimiter is a quirk of the task data that every system must handle before modeling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Any learned component should be reported both before and after training on the same split, so the gain from silver or teacher data is measured against an untrained control.
• I2 (world): Relatedness between two sentences could be computed explicitly from the text world, through shared entities, overlapping events, temporal order, and elaboration links, with each contributing factor shown as the explanation.
• I3 (answer): A graph-based relatedness score from Mova could be checked against the STR 2024 human scores directly, with no need for calibration.
• I4 (learn): This pipeline can serve as a control baseline, so that frame- and tree-based relatedness must clearly beat it to show that structure adds value.
• I5 (tokens): The en crate's input readers should treat such pair-in-one-field formats explicitly, so the sentence boundaries are not left to the sentence splitter.