📄 The Conundrum of New Online Languages: Translating Arabic Chat

Summary

This is a short abstract for a government-user talk by Joel Ross of Basis Technology, not a full paper with experiments. It addresses Arabizi, a way of writing Arabic in Latin letters. Writers mix in digits to stand for sounds the Latin alphabet lacks, and they add non-Arabic words. Arabizi became common in online chat across the Middle East during the recent upheavals. Because the form is so irregular, agencies could not process it automatically. They relied on scarce, tech-literate human translators who could not keep up with the volume of incoming messages. The talk presents Basis Technology's Arabic chat translator, which collapses many romanized spellings of a word into one standard Arabic-script form. Once normalized, the text can be passed to existing Arabic tools such as search, linguistic analysis, entity extraction, name transliteration and machine translation into English. Arabizi writers tend to write in their own dialect rather than Modern Standard Arabic, so the system can also estimate where a writer probably comes from. The abstract reports no quantitative evaluation.

Key points

• Venue: AMTA 2012, Government MT User Program. The submitter is Basis Technology and the presenter is Joel Ross.
• Arabizi is defined as Arabic written with Roman characters plus numerals, often mixed with non-Arabic words.
• Example: the variants mo2amrat, mo2amaraat and mou'amret are all mapped to a single conventional Arabic-script word.
• After conversion, the output feeds a downstream toolchain: indexing/search, linguistic analysis, entity extraction, name transliteration/translation and MT into English or other languages.
• As a side benefit, dialect cues in Arabizi let the system infer a writer's probable geographic origin. No accuracy figures are given.

Interesting ideas

• I1. Rather than building new analyzers for a noisy script, first normalize it into the standard written form so the whole existing toolchain can be reused unchanged.
• I2. Treat many inconsistent spellings as one canonical item, with digits read as stand-ins for missing sounds rather than as numbers.
• I3. Nonstandard writing is not only noise to remove: it also carries information about the writer's dialect and likely origin.
• I4. Use automation as a triage pass so that limited human experts handle only the material that matters.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A front-end normalizer could map chat-style English (digit and letter shortcuts, phonetic spellings) to standard tokens before tagging, keeping the original span so each step stays explainable.
• I2 (tokens): The lemma and morphology layer could hold a variant-to-canonical table, so that "2" or "4" inside a word is not mis-tagged as NUM.
• I3 (world): Detected variety or register markers could be stored as attributes of the narrator or speaker entity in the world of the text, instead of being discarded during normalization.
• I4 (learn): Routine inputs could be resolved by the agreeing witness parsers, with only disputed or unusual sentences escalated to the expensive LLM teacher tiers.