📄 Rule-and Dictionary-based Solution for Variations in Written Arabic Names in Social Networks, Big Data, Accounting Systems and Large Databases

Summary

Arabic personal names often appear in several spellings. Causes include omitted vowels, accent-driven letter swaps, hamza handling, final-letter alternations, and old 7-bit mainframes that could not store some characters. A search for one spelling therefore misses records stored under another. Exact matching forces users to type every variant, and plain approximate matching returns unrelated names.

Jordan's government office for civil registration and passports supplied the authors with about 75,000 distinct first names covering 9.9 million living and deceased Jordanian citizens. Their pipeline had three stages:

• Offline, pair every name with others at Levenshtein distance one or less.
• Filter those pairs with a subset of Alshamsan's catalogue of reasons why Arabic names diverge.
• Correct the result by hand.

The automatic stage alone produced about 7% false acceptances and 7.9% false rejections, so manual editing took over 500 hours. The final dictionary links each name to its alternatives and marks the most frequent form as the standard spelling. It supports two uses: normalizing names when records are entered, and expanding a search query to all variants. This matters for banking, accounting, social networks and security watch lists.

Key points

• Source data: about 75,000 distinct raw first names belonging to 9.9 million Jordanian citizens, living and dead.
• The final dictionary holds 17,992 names, and each has between 1 and 13 alternative spellings.
• The rule-based construction alone gave at least 7% erroneous acceptances and 7.9% erroneous rejections, which were fixed by more than 500 hours of manual editing.
• Search cost:
  • A dictionary lookup costs O(log k).
  • Querying the target database for all variants costs O(m log n).
  • Since k and m are effectively constant, the total is about O(log n), so the expansion barely slows normal search.
• Female names have more variants than male ones because of their typical feminine endings (ى, ا, ءا, ة/ه). Legacy 7-bit encodings are named as a cause not covered in Alshamsan's catalogue.

Interesting ideas

• I1. Cheap generator plus linguistic filter. Gather candidate variant pairs with a cheap, recall-oriented step (edit distance ≤ 1), then keep only pairs that a known cause of variation explains, so each link carries a reason.
• I2. Precompile instead of expanding online. The authors argue that applying rules at query time causes a combinatorial blow-up and yields non-names, so they freeze vetted variants into an indexed table.
• I3. Most frequent form as canonical. Each variant cluster gets a standard form chosen by corpus frequency rather than by prescriptive choice.
• I4. Report false acceptances and false rejections separately. The two error types are measured apart, showing that the automatic stage both over-links and under-links.
• I5. One resource for both writing and reading. The same mapping normalizes names at insertion time and expands them at query time.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Spelling-variant links (colour/color, judgement/judgment, archaic forms in tales) could be built the same way, with each accepted pair tagged by the rule that licenses it, keeping lemma normalization explainable.
• I2 (world): Compiled knowledge such as alias tables for entities and measures could be generated and checked offline once, leaving runtime with only a cheap indexed lookup.
• I3 (tokens): When several surface forms map to one lemma or entity name, picking the most attested one as canonical gives a reproducible, data-justified default.
• I4 (learn): Self-induced repair rules and silver-data merges could be scored on both error directions, so a rule that boosts coverage cannot hide the spurious links it adds.
• I5 (back): The variant table used to normalize input could also guide generation, so output uses the canonical spelling that understanding mapped to.