📄 Simplified guidelines for the creation of Large Scale Dialectal Arabic Annotations

Summary

Arabic speakers routinely mix Modern Standard Arabic (MSA) with their native dialect (DA), and this mixing is common in online forums. NLP tools trained on MSA perform poorly on such text, and before this work no annotated resource marked where the switches happen. Elfardy and Diab treat switch-point detection as word-level dialect identification. They propose a deliberately coarse and simplified annotation scheme, derived from the finer-grained dialectalness guidelines of Habash et al. (2008).

Each token gets one of seven tags: MSA, DA, Both, Named Entity, Foreign, Typo or Unknown. Annotators judge every token after reading the entire post, and the author's handle stays visible to them. The data was crawled from Egyptian, Levantine and Iraqi forums, cleaned automatically, and ranked by estimated dialectalness. Three Egyptian (EGY) and three Levantine (LEV) sets were annotated.

Annotators broadly confirmed the automatic post ranking. Most disagreement involved the Both and Named Entity classes. The scheme is a practical basis for large-scale code-switching corpora.

Key points

• The seven token labels are M, D, B, F, N, T and U. Phonology counts only when it shows up in spelling or meaning. For example, active and passive readings of an undiacritized verb are treated as different words.
• Rules for assigning D:
  • a purely dialectal lexeme is D even when it carries MSA affixes or clitics;
  • a shared stem with dialectal clitics is D;
  • systematic dialectal sound shifts of MSA cognates are D;
  • "faux amis" (same spelling as MSA, different meaning in context) are D.
• Agreement rates in Set A:
  • DA tokens: 3,018 in EGY (63% agreement) and 2,094 in LEV (66%);
  • B tokens: 3,147 in EGY (65%) and 2,771 in LEV (74%).
• B was the most frequent label overall. Foreign tokens were rare because the post ranking penalized posts with much Latin script.
• Planned next steps:
  • a minimum of 3 annotators for each dialect, with their disagreements resolved through adjudication;
  • an extra layer that names the specific dialect of each DA token (for example EGY or LEV).

Interesting ideas

• I1. Instead of forcing a binary choice, the scheme adds an explicit "Both" class for tokens that are identical in form and meaning across the two varieties, so shared vocabulary does not inflate either side.
• I2. The tag is decided in a fixed order: lexical identity comes first and overrides affixes, and inflection decides only when the stem is shared. This gives annotators a deterministic procedure instead of an intuition.
• I3. The guidelines separate context-independent judgments (pure dialect lexemes) from context-dependent ones (homographs whose meaning shifts with the surrounding text). Each case is then handled by its own type of rule.
• I4. Keeping the post author's handle visible helped annotators recognize handles reused inside posts as names rather than ordinary adjectives or nouns.
• I5. Ranking candidate data automatically before annotation, and then inspecting a full per-class confusion matrix, exposed exactly which labels (B and N) the guidelines leave unclear.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): when converting between UD dialects or English registers, tag constructions as variety-specific or shared, so that conversion rules fire only on genuinely divergent material.
• I2 (tokens): the morphology layer could assign register or origin features with the same rule, stem first and then affixes, and record which rule fired so the decision stays explainable.
• I3 (tokens): the tagger and lemmatizer could flag a small list of known "faux amis" homographs that always need sentence context, and resolve all other words by lookup.
• I4 (frames): coreference on dialogues or forum-like text could register speaker or author names as entities in advance, so mentions of them are not misparsed as common words.
• I5 (learn): when building silver data from witness parsers, compute label-pair confusion matrices to find the categories where rules or teacher prompts need refinement, instead of relying on a single agreement score.