📄 MISSION at KSAA-CAD 2024: AraT5 with Arabic Reverse Dictionary

Summary

This paper is a system description for the reverse dictionary track of the KSAA-CAD 2024 shared task. Given an Arabic gloss, the system must produce a vector close to the embedding of the defined word, so that the word itself or a near synonym can be retrieved. The data come from three contemporary Arabic dictionaries stored in the ISO LMF format, where each lemma carries a gloss, a POS tag and examples. Target vectors are supplied in three contextualized spaces: AraELECTRA, AraBERTv2 (Farasa-segmented) and camelBERT-MSA.

The author takes AraT5v2-base-1024 and adds a linear projection head. One copy is fine-tuned per target space with an MSE objective, and the checkpoint with the best validation cosine is kept. On the test set, cosine similarity reaches about 0.55 for the AraELECTRA space, 0.72 for camelBERT-MSA and 0.77 for AraBERTv2.

The contribution is a simple, reproducible baseline with public notebooks and checkpoints. Its main lesson is that the choice of target embedding space strongly affects how learnable the gloss-to-word mapping is.

Key points

• Data. Three dictionaries are used: Omar's Contemporary Arabic Language Dictionary, Mu'jam Arriyadh and Al Wassit LMF. The splits are 31,372 train, 3,921 dev and 3,922 test entries. Target embeddings are labelled Electra (AraELECTRA), bertSEG (AraBERTv2) and bertMSA (camelBERT-MSA).
• Model. The encoder is AraT5v2-base-1024 with a final linear layer that emits the embedding.
• Training. Three separate fine-tunings are run with AdamW (lr 1e-4, betas 0.9/0.999, weight decay 1e-6) and batch size 32 on a Colab A100. Training lasts 20 epochs for Electra and 10 for the other two spaces.
• Test results (normalized rank / MSE / cosine).
  • Electra: 0.248 / 0.230 / 0.551
  • bertMSA: 0.332 / 0.322 / 0.722
  • bertSEG: 0.417 / 0.078 / 0.773
  • Dev scores are almost identical to test scores.
• Caution. The paper's own analysis says bertSEG has higher MSE, but its table shows bertSEG with the lowest MSE. Also, scores computed in different embedding spaces are not strictly comparable. Notebooks are on GitHub (thxa/KSAA-CAD2024) and models on Hugging Face (7H4M3R/AraT5-multi-word-embedding).

Interesting ideas

• I1. Treat a reverse dictionary as regression into a fixed embedding space rather than classification over the vocabulary, so unseen words can still be retrieved by nearest neighbour.
• I2. Train one shared encoder recipe against several alternative target spaces, then compare how learnable each space is.
• I3. Score retrieval by building a similarity matrix of all predictions against all gold targets, then counting how many targets beat the true one, normalized by set size.
• I4. Optimize one criterion (MSE) but pick checkpoints by a different one that better reflects use (validation cosine).
• I5. Dictionary entries with lemma, gloss, POS and examples form ready-made, aligned description-to-word pairs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): A description such as "the animal that carries a shell" could be projected into a vector space of WordNet synsets, letting the world model attach an entity to a concept even when no lemma is given.
• I2 (learn): When choosing which lexical representation feeds frames or the ontology, the same probe could be trained against each candidate to see which one a model can actually reach from text.
• I3 (learn): This gives a cheap, label-free evaluation for any vector Mova emits, such as concept or entity embeddings, and the shuffled-target version doubles as a negative control.
• I4 (learn): Parser or frame training could pick checkpoints by downstream answer accuracy or LAS on held-out data instead of raw training loss.
• I5 (answer): WordNet or Wiktionary glosses could become test items of the "what word means…" type for school-course readers, with the gold lemma as the answer.