📄 Transliterating Urdu for a Broad-Coverage Urdu/Hindi LFG Grammar

Summary

The authors want a single LFG grammar to serve both Urdu and Hindi. The two languages share nearly all syntax and vocabulary but are written in different scripts: an Arabic-based script for Urdu and Devanagari for Hindi. Their solution is to convert each script into one shared, case-sensitive Roman transliteration scheme, so that one finite-state morphology and one XLE grammar can sit downstream of both. This paper covers the Urdu side. It is a cascade of standalone C++ modules:

• Unicode normalization to composed characters
• dictionary-based restoration of the usually omitted vowel diacritics (aerabs)
• conversion to the UZT encoding standard
• context-sensitive transliteration rules compiled with XFST

The rules deal with letters that act as either consonant or vowel, with nasalization and aspiration markers, and with Arabic/Persian loan letters that share one sound. The authors tested on 1000 frequent words. Accuracy was 99.5% on fully diacritized text and 92.5% when undiacritized and foreign words were included. The transliterator is wired into the Urdu ParGram pipeline, which makes later Hindi support cheap.

Key points

• The pipeline has four stages: Normalization → Diacritization → Unicode-to-UZT (Urdu Zabta Takhti) → Transliteration, with the last stage compiled as an XFST finite-state machine. Each stage can also be used as a separate tool.
• Diacritization looks words up in the CRULP lexicon of about 80,000 diacritized Urdu words. When a word has several entries, the first one is taken, which can lose the correct reading.
• Loan letters that all sound like /z/ are kept apart as z, z2, z3, z4, with the most frequent letter getting plain z. Case marks the dental/retroflex contrast (t vs T). N marks nasalization and H marks aspiration. The vowel inventory of the scheme comprises 3 short vowels and 8 long ones.
• The test set was 1000 unique high-frequency words from an 18-million-word news corpus (Jang, BBC Urdu), checked by hand. Results were 99.5% on fully diacritized input and 92.5% on input with non-diacritized and foreign words.
• In XLE, the output then goes through tokenization, the XFST morphology (e.g. gARI+Noun+Fem+Sg) and the ParGram-style C- and F-structures. A known limitation is that the C++ cascade only runs one way, so generation is blocked. The proposed fix is to reimplement it as an FST.

Interesting ideas

• I1. Map two different scripts onto one shared abstract representation so that a single morphology and grammar can serve both languages, instead of building two parallel analyzers.
• I2. When several written forms collapse to one sound, keep them distinct with numbered variants (z, z2, z3) rather than merging them. The default letter stays readable and the original spelling can still be recovered.
• I3. Normalize all input to one canonical form (composed Unicode) before any rules run, so no rule has to be written twice for equivalent encodings.
• I4. The authors note that picking the first lexicon match loses information. Their suggested improvement is to carry all candidate forms forward and let morphology and syntax choose between them.
• I5. Build the cascade as finite-state transducers, which run in both directions, so the same rules can analyze and generate.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): translation through the meta-world could use the same principle, with surface-specific front ends feeding one shared intermediate layer so that downstream reasoning is written only once.
• I2 (tokens): when normalizing spelling variants or Unicode look-alikes, keep a variant index on the normalized token so lemmas stay clean but the original surface form can be restored exactly for explanations or generation.
• I3 (tokens): apply NFC-style normalization plus quote, dash and apostrophe canonicalization before the tokenizer, so tokenizer and morphology rules see a single form.
• I4 (tree): pass ambiguous lemma or tag lattices into the parser rather than committing early, and record which later stage settled the ambiguity, keeping the decision explainable.
• I5 (back): writing normalization and morphology as invertible mappings would let the text → graph → text compressor reuse analysis rules for surface realization.