📄 Tweet Conversation Annotation Tool with a Focus on an Arabic Dialect, Moroccan Darija

Summary

Historically unwritten languages such as Moroccan Darija have very few textual resources, but they increasingly appear on Twitter. There they are often written in more than one script and mixed with other languages, even inside a single message. The authors built the Dialect Annotation Tool (DATOOL), a graphical annotator for tweets. It can pull in the whole conversation around each tweet, either from a user's timeline or via Apollo, so annotators see the surrounding discourse.

Version 1.0 supports tweet-level and segment-level language marking and shows summary statistics. Version 2.0 adds a built-in Maximum Entropy dialect classifier that pre-labels tweets and shows its confidence, so the annotator corrects labels instead of starting from nothing. Annotations can be exported as GEXF graphs for GEPHI, which supports network and code-switching analysis. A native speaker labeled about 3,000 tweets at roughly 250 per hour. Adding LDA-derived features raised classifier recall from 24.1% to 44.1% with precision near 97%. The work matters for building corpora when few native speakers are available.

Key points

• A native Darija speaker annotated 3,013 tweets from 3 users, about 1,000 per user, at about 250 tweets/hour. Of the 1,400 Arabic-script tweets, 1,013 contained Darija.
• The classifier separates Darija, Egyptian, Gulf, Levantine and MSA. It is a MaxEnt model trained with LIBLINEAR on the Zaidan and Callison-Burch data, extended with new Darija material:
  • 399 jokes from noktazwina.com;
  • 874 web pages found with Darija-specific queries;
  • 2,745 entries selected by semi-supervised expansion from 37,538 hespress.com comments.
• The semi-supervised cutoff was checked by sampling 131 items at intervals of 45 down the ranked list. A further spot check found 118 of 122 entries really contained Darija.
• Results on the annotated tweets:
  • baseline: 96.9% precision, 24.1% recall;
  • with LDA cluster features: 97.2% precision, 44.1% recall.
  • Most apparent false positives turned out to be errors in the gold labels.
• GEXF export offers four views: social network, information network, social-information network, and the "plus" view. Visualizations showed that one user wrote mostly Romanized Darija, often mixed with French. The current classifier works only on Arabic-script text and only at the tweet level.

Interesting ideas

• I1. When self-training grows a set of examples, the authors checked a ranked list at fixed intervals and cut it at the first wrong sample, then spot-checked inside each block. This gives a cheap, documented estimate of how clean the selected data is.
• I2. They abstracted each token into a "shape" by mapping letter classes to placeholder symbols, which lets the model generalize across spellings in noisy text.
• I3. Automatic labels are shown with their confidence, and a user-set threshold hides the weak ones, so humans only verify plausible suggestions.
• I4. Annotators label a message while seeing the full conversation around it, because a message's meaning and language often depend on earlier turns.
• I5. One set of annotations is exported as several graph projections: who addresses whom, message order, and both combined. This shows different patterns for analysis.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): silver data agreed by witness parsers could be ranked by agreement and audited this way, setting an explicit acceptance cutoff instead of a guessed threshold.
• I2 (tokens): shape signatures for unknown or misspelled English words could back up the tagger and morphology when the lexicon has no entry.
• I3 (learn): LLM-teacher or parser outputs could go to review filtered by confidence, which focuses human effort and makes each acceptance traceable.
• I4 (frames): speech-act and coreference decisions could explicitly use the previous turns of a dialogue, recording which turn justified each decision.
• I5 (world): the world of a text could be rendered as separate entity-interaction and event-order graphs, which would help debugging and explaining answers.