📄 Introducing A large Tunisian Arabizi Dialectal Dataset for Sentiment Analysis

Summary

Tunisians often write their dialect online in Latin letters mixed with digits, a style the authors call Tunizi. Earlier sentiment resources for it were small, at roughly 9k sentences or fewer, which is too little for deep learning. The paper releases a corpus of 100k social-media comments drawn from public posts on many topics, including sport, politics, comedy, TV and music. The comments were filtered to keep only Latin-script text and cleaned of links, emoji and punctuation. Ten native speakers then labelled each comment as positive, negative or neutral using short guidelines aimed at frequent ambiguous cases. The data is split 7:1:2 into train, validation and test sets. As baselines, the authors fine-tune mBERT directly and also feed mBERT embeddings into a CNN. On three classes, the CNN variant reaches about 80% accuracy, but macro-F1 is only 58% because neutral comments are rare. On a positive/negative subset, plain mBERT reaches 82.9%. The dataset is the largest Tunizi sentiment resource, it is freely available on Zenodo, and it can also support dialect identification or NER.

Key points

• Label counts: 52,711 positive, 43,767 negative and 3,522 neutral comments. The corpus has 996,782 tokens and 189,257 distinct words.
• Tunizi's romanization follows French habits ("ch", "gh", "kh"), whereas Egyptian Arabizi follows English habits. Digits such as 3, 5, 7 and 9 stand for Arabic sounds that have no Latin letter.
• A cited study (Abidi, 2019) of 1.2M Tunisian comments found that 53% use Latin script, 34% Arabic script and 13% switch between scripts.
• Three-class results:
  • mBERT+CNN: accuracy and micro-F1 of 0.803, macro-F1 of 0.580.
  • mBERT alone: 0.79 and 0.558.
• Binary results (96,478 comments, neutral dropped): mBERT scores 0.829 on all metrics, ahead of mBERT+CNN at 0.822.

Interesting ideas

• I1. Showing macro-F1 next to micro-F1 and accuracy reveals that a model with a respectable headline score is nearly ignoring the rare class, so averaged metrics alone can mislead.
• I2. When one class is too sparse, the authors rerun the experiments on a balanced subset without it, which separates "the task is hard" from "the class is starved of data."
• I3. The same word gets spelled differently depending on which European language the writers lean on (French versus English conventions), so spelling variation follows the community rather than occurring at random.
• I4. Instead of exhaustive annotation manuals, annotators received short guidelines focused only on the most frequent ambiguous cases, which kept the work fast while protecting agreement.
• I5. A cheap script filter plus removal of links, emoji and punctuation before annotation keeps the labelled data in one consistent form.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation reports should always include per-class or macro scores, so that rare frame types or rare dependency relations are not hidden by frequent ones.
• I2 (learn): When a parser or frame classifier underperforms, an ablation that drops sparse labels can show whether more silver data is needed or whether the model itself is at fault.
• I3 (tokens): Normalizing English spelling variants such as British/American forms or informal respellings could use conventions grouped by community rather than one flat lookup table.
• I4 (learn): Prompts for the LLM teacher tiers could list just the recurring hard cases, such as particle attachment or irony, rather than long rulebooks.
• I5 (tokens): A pre-tokenizer pass that strips URLs and isolates emoji would keep noisy web text from distorting the tagger and lemma statistics.