📄 Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation

Summary

The paper describes the Lotus team's entry to SemEval-2021 Task 2 (MCL-WiC). The task asks whether a target word carries the same sense in two sentences, framed as a binary T/F decision. The authors worked only on the English multilingual subset: 8000 training pairs, 1000 development pairs and 1000 test pairs. Their base system fine-tunes a pre-trained BERT encoder from TensorFlow Hub on the two sentences joined into one input, with the target word wrapped in a new "[TGT]" token and the [CLS] output used for classification.

They tried three additions on top of this base:

• WordNet lexnames (supersenses), ranked by how close each candidate synset sits to the surrounding context words;
• full paraphrases produced by back-translation through Spanish, fed in as extra input vectors;
• only the words the paraphrase changed, appended to the original sentence.

Each configuration was run five times. Sentence-level paraphrases gave the best mean accuracy, about 85.8% against 84.6% for plain BERT. The WordNet features changed almost nothing. The study suggests that cheap paraphrase augmentation can help sense comparison, while the way knowledge-base information is encoded matters a great deal.

Key points

• Data: MCL-WiC English, with 8000 train, 1000 dev and 1000 test sentence pairs. The dev set was used only to pick the number of fine-tuning iterations, via early stopping on validation loss.

• Input format and training setup:

  • Input: [CLS] sentence1 [SEP] sentence2 [SEP], with [TGT] added to the vocabulary and placed on both sides of the target word.
  • Training: Adam with weight decay and warm-up, batch size 32, at most 10 epochs.

• WordNet variant:

  • Each target synset gets a score that sums NLTK shortest-path distances to the synsets of context words, weighted inversely by positional distance. Stop words, punctuation and numbers are removed first.
  • The synsets' lexnames are then appended in order of relevance, separated by "[LX]".

• Paraphrase variants:

  • Paraphrases come from English→Spanish→English back-translation with the Microsoft Translator API.
  • The sentence variant adds three extra input vectors (six instead of three), whose embeddings are summed.
  • The word variant aligns each changed word to an original word with fastText similarity and joins the two with "[S]".

• Mean test accuracy over 5 runs (std in parentheses):

  Variant	Mean accuracy	Std
  BERT	84.62	0.688
  +WordNet	84.78	0.366
  +paraphrase sentences	85.78	0.571
  +paraphrase words	84.76	1.378

Interesting ideas

• I1. Wrapping the word of interest in a dedicated marker token on both sides lets one sentence-pair classifier focus on a single position without any architecture change.
• I2. Synsets of a target can be ranked by summing their WordNet path distances to the senses of nearby content words, discounted by how far those words are in the sentence, which gives a transparent sense-relevance score.
• I3. Round-trip machine translation through a pivot language is a cheap source of second views of the same sentence, and these views helped the sense decision more than added lexicon features did.
• I4. Keeping only the changed words of a paraphrase failed partly because it broke the context and partly because aligning words by fastText similarity was unreliable (for example, "work" replacing "play" was not matched back).
• I5. Reporting mean and standard deviation over several random initialisations showed that the paraphrase-words variant was the least stable, which a single run would have hidden.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): when gparse or a classifier must decide whether two predicate occurrences share a VerbNet class, marking the predicate tokens gives a simple, inspectable way to say which word the decision concerns.
• I2 (world): the WordNet ontology layer could use this distance-weighted score as an explainable first-pass sense ranking before choosing an entity type or verb class, logging which context words drove the choice.
• I3 (learn): back-translated variants could serve as an extra witness when building silver trees and frames, keeping only analyses that stay stable across the paraphrase.
• I4 (back): when the text → graph → text compressor is checked against its input, alignment should go through the graph or tree rather than raw embedding similarity, and whole sentences should be compared rather than isolated diffs.
• I5 (learn): evaluations of new parser or repair-rule variants should be repeated over several seeds, with the spread reported next to negative controls, before a gain is accepted.