📄 Classifying Arabic Crisis Tweets using Data Selection and Pre-trained Language Models

Summary

The paper addresses classifying Arabic crisis tweets from a new disaster when no labelled tweets for that disaster exist yet. Classifiers trained on pooled data from earlier crises often degrade on a new event, because topics, entities, formality and Arabic dialects differ between events. The authors propose an unsupervised domain adaptation method based on data selection. Every unlabelled tweet from the target crisis is embedded with multilingual Sentence-BERT. A K-nearest-neighbours search by cosine similarity then retrieves its closest labelled examples from the multi-event source pool. Only this selected subset is used to fine-tune a BERT classifier.

The method is evaluated on two tasks: relevancy detection and multi-label information-type classification. Fine-tuning on the selected subset beats fine-tuning on all source data, and it generally outperforms self-training adaptation. Adding self-training on top of the selection gives no further gain.

The approach matters because it needs only a small amount of unlabelled target text. That makes it usable in the first hours of an emergency. The authors present it as the first data-selection adaptation study for Arabic cross-crisis tweet classification.

Key points

• A domain is defined as one crisis-specific dataset collected from social media, so each disaster counts as its own domain. Arabic dialect variation by location adds to the shift between domains.
• The embeddings come from multilingual S-BERT (Reimers and Gurevych, 2020), because no monolingual Arabic S-BERT was available. The classifier itself is a fine-tuned BERT model.
• In Algorithm 1, only relevant source tweets are searched for each target tweet's k nearest neighbours. Duplicates are then removed, and all irrelevant source tweets are added back unchanged.
• Off-topic labels that were actually about another crisis were filtered out of the training data. For example, Kuwait flood posts were marked irrelevant in the Jordan flood data. The model therefore learns "crisis vs. non-crisis" rather than "this specific event".
• Two tasks are covered: relevancy detection, which suffers from class imbalance, and multi-label information classification (e.g., infrastructure damage, caution). Selection beats using all source data and beats self-training, while combining selection with self-training does not help.

Interesting ideas

• I1. Select training data per target instance: for each unlabelled target item, pull its k nearest labelled neighbours in a sentence-embedding space, so the training set is tailored to what will actually be processed.
• I2. Apply selection asymmetrically: filter the positive class by similarity but keep the whole negative class, so the model still sees a broad view of "what is not relevant".
• I3. Clean labels that are only true relative to one event: a post about a different flood is "irrelevant" to one dataset but still a crisis, and such cases were removed to avoid contradictory supervision.
• I4. Treat each source collection as its own domain and measure how well training transfers to a held-out one, rather than shuffling everything together.
• I5. Check whether stacking two adaptation methods actually helps: here self-training on top of selection gave nothing extra, a useful negative result.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before retraining the parser or tagger for a new genre (fables, maths problems), pick treebank sentences nearest to the target texts instead of training on everything.
• I2 (learn): When building silver data for a frame or speech-act detector, retrieve target-like positives and keep a full, unfiltered pool of negatives as a control.
• I3 (frames): Check for annotations that are context-relative (e.g., a role or event judged against one story only) before reusing them as general training signal.
• I4 (learn): Evaluate parsers with leave-one-genre-out splits across treebanks, which shows how much a domain shift hurts.
• I5 (learn): Test each added repair or self-play loop against the simpler baseline alone, and keep it only if it adds measurable gain.