📄 AIDA: Automatic Identification and Glossing of Dialectal Arabic

Summary

AIDA, a demo system from Columbia University's Center for Computational Learning Systems, tackles the problem of telling Modern Standard Arabic (MSA) apart from dialectal Arabic (DA) in written text. The task is hard for several reasons. Arabic is diglossic. DA mostly appears in informal genres and has no standard spelling. Many words also look identical across varieties while meaning different things (faux amis). AIDA combines dictionaries and language models into one pipeline that works at both the token and sentence level. For each token it reports four things: a variety label, a named-entity tag, an MSA gloss for dialectal words, and an English gloss. MSA candidates are ranked by how often they occur in Arabic Gigaword. The paper gives no evaluation figures. Instead it presents the tool, which users can run through a configurable web interface or as an offline pipeline. Its intended value is to help Arabic machine translation cope with informal dialectal input by first marking dialect material and mapping it to standard equivalents.

Key points

• Each token gets a CLASS of MSA, DA or UNK. Dialectal tokens are further split into Egyptian or Other, and tokens that fit both varieties can be marked DA/MSA.
• Named entities are tagged as Person, Organization, GeoPolitical entity or Location. In the paper's example, a multi-token span ("mSr fy") is labelled GPE.
• MSA glosses are sorted by frequency in Arabic Gigaword (LDC2009T30), and an English gloss is supplied alongside.
• Examples are shown in Buckwalter transliteration, and the user can choose the output encoding and which fields to display.
• The system is available as a web interface and as a packaged offline pipeline. The demo paper reports no accuracy numbers.

Interesting ideas

• I1. Instead of forcing a binary standard-vs-dialect decision, the label set includes UNK and a shared DA/MSA tag, so genuinely ambiguous tokens stay visible.
• I2. Dialectal tokens are glossed into standard-variety equivalents ranked by corpus frequency, which creates a normalization layer before translation.
• I3. The paper treats faux amis, words shared across varieties with different meanings, as a central difficulty that requires context rather than lookup.
• I4. Dialect labels, NE classes and glosses all sit in one per-token record, and multi-token NE spans are grouped.
• I5. A linearized English gloss gives a word-by-word interlinear view of the input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The English tagger could mark tokens as standard, colloquial ("gonna", "ain't") or shared, so later stages know which forms need normalization.
• I2 (tree): Before parsing, nonstandard English forms could be mapped to ranked standard lemmas while keeping the originals, which would help UD dialect conversion and reduce parser errors on informal text.
• I3 (learn): Faux-ami-style tokens (same spelling, register-dependent sense) could be collected as negative controls to check that the system is not just doing dictionary lookup.
• I4 (world): Entity typing could be attached to the same explainable token record that feeds entity creation, so every world entity traces back to its tagged span.
• I5 (back): An interlinear gloss layer could serve as a debugging trace for translation through the meta-world, showing per-token correspondences next to the generated output.