📄 Baleegh at KSAA-CAD 2024: Towards Enhancing Arabic Reverse Dictionaries

Summary

Arabic has far weaker reverse dictionary (RD) tools than English or Chinese. An RD takes a description of a meaning and returns matching words. This paper is the Baleegh team's entry to the KSAA-CAD 2024 shared task, where each dictionary gloss must be mapped onto a target word embedding. The authors made two changes to the usual approach. First, they replaced the BERT-family encoders common in earlier Arabic RD work with AraT5 V2. Second, they added example contexts automatically to every training entry, pulled from Arabic Wikipedia through semantic vector search. The model encodes the input with AraT5, pools the result into one vector, and passes it through a linear layer that regresses onto three target spaces (electra, bertseg, bertmsa). The gloss-only model came close to the baselines on bertseg and bertmsa but was weaker on electra. The Wikipedia examples slightly reduced performance. The work matters as an early test of T5 for Arabic RD and as an honest negative result on retrieval-based data enrichment.

Key points

• The KSAA-CAD dataset has 39,214 word–gloss entries split 31,372 / 3,921 / 3,922 (train/dev/test). Each entry has word, gloss and pos fields plus three target embeddings: electra, bertseg and bertmsa.
• Enrichment drew on Cohere's Arabic Wikipedia Embeddings (3.1M entries, embedded with multilingual-22-12). Each "word:gloss" string was embedded with that same model, and the top 5 passages by cosine similarity were kept, giving 156,860 examples.
• The model is the AraT5 V2 encoder (AraT5-base-1024), followed by mask-weighted mean pooling and a linear layer. It was trained with MSE loss, Adam at learning rate 3e-5, and only 2 epochs because longer training overfit.
• Gloss-only test cosine was 0.5678 for electra, 0.7168 for bertmsa and 0.7739 for bertseg. Adding examples lowered cosine and gave clearly worse rank scores. Rank was much worse than the CamelBERT and MARBERT baselines in all settings.
• The authors blame several factors: noisy or off-topic retrieved passages, words missing from Wikipedia, and targets built from definitions alone.

Interesting ideas

• I1. Use a single embedding model for both the query and the corpus, and phrase the query as "word:gloss" so the gloss pins down which sense of the word to retrieve contexts for; this cheaply turns a dictionary into a sense-tagged example bank.
• I2. The enrichment failed partly because the training targets came only from definitions, so the added context worked against what the model was graded on; augmentation needs to match how the targets were built.
• I3. The metrics told different stories: cosine scores were competitive while rank scores were much worse, so judging on one similarity measure would have been misleading.
• I4. Regressing the same encoder onto several target spaces showed which spaces it fits easily (bertseg, bertmsa) and which it fits poorly (electra), which the authors link to how similar the vocabularies are.
• I5. Treating a reverse dictionary as "description in, word out" is a direct model for choosing words from meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Querying a corpus with lemma plus WordNet gloss could collect sense-specific example sentences for the ontology, which would help explain why a particular sense was chosen.
• I2 (learn): Before adding auto-retrieved context to silver data, compare it against a no-augmentation control and check that the gold labels can actually reflect the extra signal.
• I3 (learn): Report both closeness and ranking/retrieval metrics for any embedding-based component, so that a near-miss vector is not mistaken for picking the correct item.
• I4 (tokens): Fitting our lemma representations to several external embedding spaces could serve as a probe of tokenizer and vocabulary compatibility before we pick one.
• I5 (back): During generation, a frame's semantic description could be mapped to a ranked list of candidate lemmas, with the score for each recorded as an explanation of the lexical choice.