📄 ANLPers at MAHED2025: From Hate to Hope: Boosting Arabic Text Classification

Summary

The paper describes a system for Sub-task 1 of the MAHED2025 shared task. The task is to sort short Arabic posts, in MSA or dialect, into hate, hope or not_applicable. The provided corpus has 6,890 texts and is skewed toward not_applicable, which makes up about 54%.

The authors compared three ways of using pre-trained Arabic BERT models:

• a frozen encoder whose mean-pooled embeddings feed a deep feed-forward classifier;
• full end-to-end fine-tuning with a single linear head;
• fine-tuning with two extra fully connected layers on top.

Each architecture was tested on raw text, on text passed through a cleaning pipeline, and on cleaned text plus paraphrases generated by an OpenAI model. The cleaning pipeline drops Latin characters, symbols, emojis and diacritics, and normalizes Unicode and whitespace. The paraphrases were meant to preserve labels and to balance and diversify the training data.

The best run reached macro-F1 0.6747 on the official test set. It used BERT-base-AraBERTv02 + NN with cleaning and generated data.

A manual audit then found that about a fifth of the training labels were wrong. This suggests that annotation quality, not model design, is the main limit on performance.

Key points

• Dataset: 6,890 instances, split into hate 1,301, hope 1,892 and not_applicable 3,697, with a majority/minority ratio of about 2.84. Texts average roughly 22 words.
• Training splits: the authors trained on balanced subsets. Without augmentation this meant 1,000 per class for training and 250 per class for validation. With GPT4-mini paraphrases it grew to 4,000 per class for training and 300 per class for validation.
• Best result: AraBERTv02 + NN with cleaning, generated data, LR 2e-5 and dropout 0.3 reached test macro-F1 0.6747 (validation 0.6394).
• Effect of augmentation: the same setup without generated data, at dropout 0.7, scored 0.6736, so the paraphrases added only a small gain.
• Other configurations: plain fine-tuned AraBERTv02 reached 0.6504. Frozen embeddings with an 8-layer network reached 0.6235.
• Label audit: a manual review of 100 random training samples found 22 mislabeled, with errors in every class. The per-class numbers in the paper's table and text do not fully agree.

Interesting ideas

• I1. Hand-check a small random sample of training labels to estimate how noisy the data is before blaming the model; this audit gives a rough ceiling on achievable scores.
• I2. Use an LLM to paraphrase training items while keeping the label fixed, which adds lexical and structural variety and can rebalance rare classes.
• I3. Run frozen-encoder-plus-head, full fine-tuning and fine-tuning-plus-extra-layers side by side under identical settings, to see where the gain actually comes from.
• I4. Downsample to equal class counts for training and validation, so macro-averaged metrics reflect performance on the minority classes rather than the dominant one.
• I5. Keep the text normalization explicit and step-by-step: strip symbols, normalize Unicode and collapse whitespace as separate, inspectable operations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Periodically audit around 100 random silver-data items where the witness parsers agreed, to measure residual noise and decide whether repair rules or relabeling should come before more training.
• I2 (answer): Generate meaning-preserving rewrites of school test questions and fable QA items, then check that Mova's answer stays the same, as a robustness test for the readers.
• I3 (tree): Apply the same three-way comparison to gparse, freezing the transformer versus fine-tuning it, to learn whether the parser's quality depends on encoder adaptation or on the scoring head.
• I4 (frames): Evaluate speech-act or irony detection on class-balanced sets, so that rare acts like threats or sarcasm are not hidden behind the frequent neutral statements.
• I5 (tokens): Log each normalization step in the en crate as its own traceable transform, so any tokenization error can be traced back to a specific cleaning rule.