📄 Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset

Summary

Arabic script usually leaves out short-vowel and gemination marks. Proper nouns in Arabic Wikipedia therefore often have more than one reading, as with the same spelling for "Oman" and "Amman". Foreign names are the hardest case. The authors found that 99.45% of their entries carried no diacritics at all. They treat the problem as a mix of transliteration, diacritization and lemmatization. Given an undiacritized Arabic name plus its English Wikipedia title, the goal is the fully diacritized Arabic lemma. They built the resource from part of the CP-WIKI list (CamelProp), which originally came from Wikidata. Annotators diacritized and lemmatized each name by hand, following written guidelines. The English gloss guided vowel choices, and names with several valid readings received several entries. GPT-4o, prompted with both forms, reached 73% exact-match accuracy. A manual error review showed that many mismatches were still plausible variants. The released data is the first open benchmark of its kind, and the results show how far current models still are from reliable use.

Key points

• Dataset size: 3,000 unique Arabic Wikipedia proper nouns (people, places, organizations), with 3,362 Arabic–English pairs in total.
• Source: CP-WIKI, the Wikidata-derived portion of CamelProp from Khairallah et al. (2024).
  • About 63K single-word entries from a 14-Mar-2023 dump.
  • It originally shipped without diacritics.
• Lemmatization steps:
  • remove the article Al- only when it is a determiner rather than part of the name;
  • reduce plural demonyms to a singular base form;
  • restore missing Hamzas.
• Benchmark result: GPT-4o reached 73% accuracy. The error classes were:
  • diacritic-only mismatches;
  • added or dropped long-vowel letters (AWY), where the model mostly dropped them;
  • consonant or ta-marbuta swaps (10 cases);
  • several changes at once (25 cases).
• Comparison and release:
  • The data and guidelines are released at github.com/CAMeL-Lab/CamelProp.
  • The closest earlier work, Mubarak et al. (2009), used only 500 training and 200 test items.

Interesting ideas

• I1. The English gloss serves as a side channel that resolves ambiguity in the Arabic form, turning context-free diacritization into partial transliteration.
• I2. When a name has more than one valid reading, the benchmark stores several gold entries instead of forcing a single answer.
• I3. The errors are sorted into named categories (diacritics, vowel letters, consonant swaps, multiple changes), and plausible-but-non-gold outputs are reviewed rather than simply counted wrong.
• I4. Proper-noun lemmatization is defined by a few explicit transformations: stripping the article, reducing demonym plurals and fixing spelling.
• I5. The gold resource was built on top of an existing name list that lacked key annotations, rather than collected from scratch.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): When normalizing entity names, Mova could use a Wikidata label in another language or script as an extra witness for spelling and pronunciation.
• I2 (learn): Evaluation sets for lemmas or entity forms could keep several accepted gold answers, so that legitimate variants are not counted as failures.
• I3 (learn): Mova's evaluation reports could split errors into similar categories and flag "acceptable variant" cases for separate review by a teacher model.
• I4 (tokens): The en lemmatizer could apply a small explicit rule set to name-derived words, such as reducing "Venezuelans" to "Venezuelan", and log which rule fired so the step stays explainable.
• I5 (world): Mova's ontology could take existing Wikidata-derived name lists and add the missing layers, such as canonical lemma and entity type, by targeted manual or silver annotation.