📄 Conventional Orthography for Dialectal Arabic

Summary

Dialectal Arabic (DA) is the everyday spoken Arabic of the Arab world, and it is increasingly written online in blogs, forums, SMS and email. It has no agreed spelling standard, so native writers and the builders of DA resources spell words in inconsistent ways, which hurts computational processing. The authors propose CODA, a conventional orthography for dialectal Arabic built mainly to support NLP modeling rather than linguistic transcription. CODA uses Arabic script and stays close to MSA orthographic habits. It abstracts away from sub-dialectal phonological variation and aims to be one shared convention that keeps differences between dialects small. The paper sets out the design principles and then gives detailed guidelines for Egyptian Arabic (EGY). These cover consonant spelling, Hamza, clitics, Ta-Marbuta, nunation and a list of lexical exceptions. A preliminary annotation of more than 110K words showed that a substantial share of raw dialectal text needs normalization. The work matters because a consistent written target makes it feasible to annotate DA data and to build tools that automatically convert text into CODA.

Key points

• CODA replaces the authors' earlier CCO (COLABA Conventional Orthography). CCO tried to encode dialect-specific phonology and morphology. Annotators found it hard to learn, described it as "foreign", and agreed poorly, especially on short vowels.
• CODA draws on the LDC transcription guidelines for Levantine and Iraqi Arabic and on CallHome Egyptian. Unlike those, it targets general-purpose writing rather than phonological transcription.
• EGY consonant rules:
  • A few consonants keep their etymological MSA spelling. For example, the letter q is kept even when it is pronounced as a glottal stop.
  • Most other sound differences are spelled as pronounced.
  • Hamza is omitted in words where EGY does not pronounce it.
  • The common Egyptian habit of writing word-final y as Alif-Maqsura is not allowed.
• Clitics and spacing:
  • EGY-only clitics are written attached to the word. These include progressive bi+, future Ha+ and negation +š.
  • Indirect-object enclitics and the negation particle mA are written as separate words.
  • A word list handles ad hoc spellings, such as dah 'this' and barDuh 'also'.
• Preliminary annotation results:
  • Four annotators worked on more than 110K words.
  • 15.1% of words were changed.
  • Among the changes, 12.1% were merges (removing a wrong space) and 13.2% were splits (adding a missing space).

Interesting ideas

• I1. Design for annotators, not just linguists. The failure of CCO shows that a phonetically faithful scheme can be too hard to learn and annotate consistently. Choosing conventions that writers find familiar is a useful lesson for any project that normalizes non-standard text, such as Surzhyk or dialectal Ukrainian.
• I2. One convention across many varieties. CODA picks spellings that reduce differences among dialects instead of recording each local pronunciation. This makes resources more reusable across varieties.
• I3. Etymological spelling only where it helps. Some sounds are spelled after the standard-language cognate, while most differences are written as pronounced. This hybrid balances readability and data sparsity against faithfulness to speech.
• I4. A separate list for exceptions. Irregular or competing spellings of frequent words go into an explicit word list instead of complicating the general rules. This keeps the rules simple and the treatment of high-frequency items consistent.
• I5. Measure the edits. Reporting the share of words changed, split by merge and split operations, gives a cheap first estimate of how far raw text is from the normalized standard. It also shows what an automatic normalizer (here, "CODAfication") would need to learn.