📄 A Corpus-Aligned Uthmani-to-Standard Quranic Word Mapping and a Deterministic Recitation Validator

Summary

Quranic text circulates in two byte-distinct spellings: the Uthmani script used in printed mushafs and in Tanzil, and Standard (Imla'i) Arabic, which mainstream Arabic NLP tools and ASR systems expect. Most of the gap comes from one character, U+0670 (superscript alef). Standard spelling sometimes replaces it with a full alef and sometimes drops it, so no single character rule is correct. PyArabic and CAMeL Tools either keep it or delete it across the board. The author aligns all 6,236 verses in both forms and derives a 2,290-pair word-level Uthmani-to-Standard map, then builds a seven-step normalizer around it. After normalization, 90.9% of verses become identical across the two forms, and the remaining divergence is catalogued rather than hidden. On top of this sits a deterministic recitation validator with no LLM. It finds the recited verse from ASR text through four increasingly lenient search layers and grades the recitation by word error rate in five tiers. It passes 122 of 124 author-written test cases and identified the right verse for all 34 real ASR transcripts. All artifacts except the deployment data are released and reproducible.

Key points

• How the map is built. Construction has four automatic passes and no manual curation:
  • positional alignment for the 5,813 verses whose word counts match;
  • stripping of the ornamental mark U+06DE (found in 199 verses);
  • difflib.SequenceMatcher plus a dynamic program for the other 423 verses, mapping one Uthmani word to 1–3 Standard words, which yields 58 one-to-many entries such as the vocative particle written joined in Uthmani;
  • a majority vote for the two keys that had conflicting forms.
• Map coverage. The 2,290 pairs come from 8,881 aligned occurrences of U+0670 and cover 2,290 of 2,291 distinct forms. One form that could not be aligned consistently was omitted on purpose instead of guessed.
• Baseline failure. Deleting U+0670, as strip_diacritics and dediac_ar do, gives the correct Standard word in only 359 of 2,290 cases (15.7%). Standard has a full alef in 1,335 of these words and nothing in 362.
• Normalizer residue. 5,669 verses match after normalization. The 567 that still differ contain 628 divergent word pairs across 263 types, and the ten most frequent types account for 30%. They fall into known classes: assimilated lam of the article, letters missing in Uthmani, and medial hamza.
• Validator design and results.
  • The search layers are: exact token AND-match; lemma/root expansion via the Quranic Arabic Corpus; at least 60% token coverage for queries of three or more tokens; and fuzzy Ratcliff/Obershelp matching with threshold 0.45.
  • It scores 98.4% (122/124). Both failures arise because a single substituted word turns another verse into an exact match.
  • 16.5% of verses are inherently ambiguous from text alone.
  • There is no held-out split, and the map covers only the Hafs narration.

Interesting ideas

• I1. Build a cross-convention mapping by aligning two parallel versions of the same corpus, store only the tokens that actually need mapping, settle conflicts by majority vote, and leave unalignable items out instead of guessing; this keeps the resource small, derived, and auditable.
• I2. Test a normalizer by running both variants of every item through it and counting how often they converge, then list the residue by type and frequency instead of claiming the problem is solved.
• I3. Run a cascade of matchers from strict to lenient (exact, lemma/root, coverage threshold, fuzzy) and stop at the first layer that succeeds, so cheap and confident matches never reach the noisy layers.
• I4. Separate ambiguity that is inherent in the data from engineering errors with a full-corpus census, so the error rate is not blamed on the system for cases no text-only method could resolve.
• I5. Map error levels onto graded confidence tiers and decline to answer when the error is high, because a confident wrong correction does more harm than an abstention.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): The same recipe could induce word- or relation-level conversion tables between UD dialects or treebank versions from parallel annotations of the same sentences, instead of hand-written conversion rules.
• I2 (tokens): Normalizing American/British spellings or tokenizer variants of the same text and itemizing the non-converging pairs would give the en crate a concrete, ranked repair list.
• I3 (answer): When a QA reader must locate the sentence in a fable that supports a question, an explicit exact → lemma → partial → fuzzy cascade keeps every match explainable by the layer that produced it.
• I4 (learn): Evaluation could first count sentences with genuinely ambiguous attachment or coreference and report accuracy on the remaining cases separately, much like a negative control.
• I5 (answer): Readers answering school tests could emit tiered confidence derived from how degraded the parse or frame match was, abstaining instead of producing an authoritative-looking wrong answer.