📄 Abed at KSAA-RD Shared Task: Enhancing Arabic Word Embedding with Modified BERT Multilingual

Summary

The paper describes a system submitted to KSAA-RD, the ArabicNLP 2023 shared task on Arabic reverse dictionaries, which is modeled on SemEval 2022 CODWOE. The system receives a dictionary gloss and has to produce the vector of the word it defines. There are two tracks: Arabic glosses (RD) and English glosses mapped to Arabic targets (CLRD). The author fine-tunes BERT Multilingual Cased and adds a multi-head attention block plus a linear layer. These regress directly onto the target embeddings, which are 256-dimensional SGNS vectors or 300-dimensional Electra vectors. Training uses MSE loss and AdamW. Because the training set is small, the author expands it with simple text augmentation and stops after two epochs. On the official metrics (cosine, MSE, ranking), the system beats the task baselines on dev and test, and Electra targets come out ahead of SGNS. The work shows that a lightly modified multilingual encoder combined with cheap augmentation is competitive in a low-resource lexical-semantics setting. The gains on SGNS targets are smaller.

Key points

• Model: BERT Multilingual (pretrained on 106 languages), followed by multi-head attention and a linear projection to the embedding size. Training uses MSE loss, AdamW with lr 2e-5, and only 2 epochs, because validation loss rose after that.
• Arabic augmentation: only random deletion and random swap. Each gloss longer than two words gets five variants, and punctuation is removed.
• English (cross-lingual) augmentation: synonym replacement and random insertion using WordNet via NLTK.
• RD test results:
  • Electra: cosine 62.5%, ranking 28.5%, against a benchmark of 48.85% and 31.28%.
  • SGNS: cosine 53.8%, MSE 3.1%, ranking 29.1%, against a benchmark cosine of 35.61% and ranking of 38.52%.
• CLRD test results:
  • Electra: cosine 56.5%, ranking 28.1%, against a benchmark of 54.09% and 36.22%.
  • SGNS: improves only marginally, with cosine 27.0% vs 26.23% and ranking 45.2% vs 50.17%.
  • For Electra, ranking on test was worse than on dev in both tracks.

Interesting ideas

• I1. The augmentation is matched to the resources available for each language: synonym-based operations appear only where a lexical database exists (English), and Arabic gets only resource-free deletion and swap.
• I2. The same encoder is trained against two different target vector spaces (SGNS vs Electra), and the results show that some targets are much easier to reach from glosses than others.
• I3. The paper reports cosine, MSE and ranking together, and they sometimes disagree; for example, Electra's cosine improved while its test ranking slipped.
• I4. On tiny data, training stops as soon as validation loss turns upward, after just two epochs, instead of running a fixed schedule.
• I5. The proposed future work suggests adding POS prediction as an auxiliary task and feeding in gloss length and POS as features to improve gloss representations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when growing silver data, apply WordNet-driven paraphrase perturbations only where the lemma and sense are reliable, and fall back to structure-free noise otherwise.
• I2 (world): when linking WordNet glosses to compiled knowledge, compare several candidate target representations before committing to one, since the choice of target alone can change accuracy a lot.
• I3 (learn): evaluate any gloss-to-concept or retrieval component with both a similarity score and a rank-based score, so that apparent gains on a single metric are not trusted blindly.
• I4 (tree): when fine-tuning gparse on small UD dialect or converted treebanks, use dev-loss-triggered early stopping to avoid overfitting.
• I5 (frames): a definition encoder used for VerbNet or WordNet sense matching could share a POS/morphology head, keeping representations tied to explicit, explainable categories.