📄 ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD

Summary

The paper tackles Arabic Word Sense Disambiguation (WSD), which is hard because Arabic has no WordNet-style sense inventory with labeled examples. The authors built such a resource from the Arabic Ontology and Birzeit University's lexicographic database (about 400K glosses from roughly 150 lexicons). They parsed definitions into separate glosses and example contexts, chose one lexicon's sense list per lemma, generated negative pairs, and found the target word in each context. That last step is difficult because the word usually appears inflected and may occur more than once with different meanings.

The result is about 167K context-gloss pairs. WSD is cast as binary sentence-pair classification: given a context and a gloss, decide whether the gloss is the right sense. Three Arabic BERT models were fine-tuned, and several ways of marking the target word were compared. AraBERTv02 performed best at about 84% accuracy. Target markers added only around 1%. The work also stresses a clean evaluation: no context appears in both train and test. The authors note that an earlier Arabic result was inflated because of exactly this kind of overlap.

Key points

• Dataset size: 167,207 pairs in total, of which 60,323 are True and 106,884 are False. They cover 26,169 undiacritized lemmas, 32,839 glosses and 60,272 contexts.

• Data selection rules:

  • Lemmas were kept only if every gloss had at least one example.
  • One-word glosses and contexts were removed.
  • When a lemma appeared in several lexicons, the lexicon with the most glosses was used, to avoid duplicate senses.

• Models and setup: AraBERTv02, QARiB (base) and CAMeLBERT were fine-tuned with BertForSequenceClassification on the [CLS] token. Settings were learning rate 2e-5, 1412 warmup steps, batch size 16, 4 epochs, on a single Tesla P100.

• Results with single-quote markers:

  Model	Accuracy	F1 (True)	F1 (False)
  AraBERTv02	84%	72	89
  CAMeLBERT	82%	—	—
  QARiB	80%	—	—

• Marker variants: four were tested on AraBERTv02: no marker, single quotes, [UNUSED0] on both sides, and [UNUSED0]/[UNUSED1] as open/close. All marked variants also start the gloss with the target word and a colon. Gains over no marker were about 1%. The dataset and models are released by Birzeit.

Interesting ideas

• I1. Instead of a per-word sense classifier, each (context, candidate gloss) pair is scored separately, so one model covers every lemma and the chosen sense comes with its human-readable definition.
• I2. The authors re-ran a prior Arabic WSD experiment and found its high F1 came from test contexts that had also been seen in training, so they split by context.
• I3. When merging many overlapping dictionaries, they keep the single source with the richest sense list for each lemma rather than taking the union, which avoids near-duplicate senses.
• I4. A lemma enters the data only if all of its senses have examples, so negative pairs always contrast real, attested alternatives.
• I5. The target occurrence is highlighted with delimiters, and the word is also prefixed to the gloss; this helps a little even for a strong encoder.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): WordNet sense selection could score context-gloss pairs, and the winning gloss would serve directly as the explanation for the ontology link.
• I2 (learn): silver-data train/test splits should be made by source sentence, and evaluations should include a leakage check as a standard negative control.
• I3 (world): when combining WordNet, FrameNet and Wikidata entries for a lemma, choosing one primary inventory per lemma could prevent inflated, redundant sense sets.
• I4 (learn): when building sense or frame training sets, dropping incompletely attested lemmas keeps the negatives meaningful instead of skewed toward senses that lack examples.
• I5 (frames): marking the predicate token, for example with reserved tokens, when classifying VerbNet roles or frames could cheaply focus the model on the right verb in sentences with repeated words.