📄 Methodical Evaluation of Arabic Word Embeddings

Summary

The paper tackles the lack of a proper way to judge the quality of Arabic word embeddings. Earlier work mostly machine-translated English analogy sets, which leads to errors, irrelevant questions, and missing Arabic-specific relations. The authors had an Arabic linguist define nine relation types, such as capital, plural, past tense, and the Arabic dual ("pair"). Two native speakers then filled them with word pairs, and the resulting analogy test bank of more than 100,000 questions was released together with an evaluation tool. They extend the classic analogy test in two ways: a relation can be represented by averaging ten example pairs, and an answer counts as correct if it appears among the five nearest words. The second change matters because Arabic clitics produce many surface variants of the same word. Four existing embeddings are compared: CBOW, Skip-gram and GloVe trained on the Zahran et al. corpus, and the Arabic Polyglot vectors. Extrinsic tests use document classification and NER. CBOW leads on the analogy test, the three corpus-based models are close on the downstream tasks, and Polyglot trails everywhere.

Key points

• Each of the benchmark's 9 relations contains more than 100 word pairs. They cover Arabic grammar (Plural, Pair, Past Tense, Comparative), word meaning (Opposite, Male-Female) and world knowledge (Nationality, Currency, Capital). It is available from oma-project.com.
• Questions missing from an embedding's vocabulary are scored as wrong, so coverage is part of the result.
• On overall analogy accuracy, CBOW scores 16.3% (top-1, two pairs), 17.4% (top-1, 11 pairs), 26.6% (top-5, two pairs) and 31.1% (top-5, 11 pairs). Polyglot stays at about 1–3.4%.
• On document classification (F-measure), Skip-gram scores 0.954, CBOW 0.948, GloVe 0.946 and Polyglot 0.882. On NER with Columbia's Arabic NER corpus, GloVe scores 0.816, CBOW 0.800, Skip-gram 0.799 and Polyglot 0.649.
• Both downstream tasks use the same LSTM model with two dense layers and a softmax, trained for 150 epochs. Classification uses 8,000/1,000/1,000 Wikipedia articles; NER uses 3,852 training and 963 test sentences.

Interesting ideas

• I1. Native test sets beat translated ones: a translated benchmark carries over questions that don't fit the target language and misses relations unique to it, so tests should be written by speakers from the language's own categories.
• I2. Lenient top-k matching tolerates inflected variants: accepting the gold word anywhere in the top five avoids penalizing answers that differ only by an article or possessive affix.
• I3. Relations are better defined by many examples than by one: averaging the offsets of ten pairs gives a steadier picture of a relation than a single pair and raised accuracy for every model.
• I4. Missing coverage counts as failure: scoring out-of-vocabulary questions as wrong stops a small resource from looking good on the few items it happens to know.
• I5. Pair intrinsic and extrinsic tests: analogy accuracy and downstream F-scores ranked the models differently, so a single kind of evaluation would have been misleading.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation suites for non-English or dialectal data should be authored natively rather than translated from English sets, so they don't reward English-shaped behaviour.
• I2 (tokens): Lemma or morphology checks could score a prediction as a near-hit when it differs from gold only by inflection, keeping those cases separate from true errors.
• I3 (world): When inducing relation types for the knowledge layer, such as plural-of or capital-of, a prototype built from several confirmed examples would be more robust than generalizing from one instance.
• I4 (learn): Mova's evaluations should report unparsed or unknown inputs as errors rather than dropping them, so recall gaps stay visible next to precision.
• I5 (answer): Component-level metrics such as LAS should always be checked against end-task results on the school tests and fable QA, since the two can disagree.