📄 Kawarith: an Arabic Twitter Corpus for Crisis Events

Summary

Arabic is heavily used on Twitter, yet before this work there was no openly available Arabic corpus covering many different crises. Alharbi and Lee built Kawārith, a multi-dialect collection of over one million Arabic tweets from 22 Middle Eastern emergencies between October 2018 and September 2020. The events span floods, shootings, bombings, wildfires, sandstorms, explosions and the COVID-19 pandemic.

Tweets were gathered through the Twitter search API with an iterative, precision-oriented keyword and hashtag procedure. The authors explored the content to find the dominant information types. They then had annotators hand-label a subset from seven events with a multi-label scheme; COVID-19 tweets were labelled only for relevance.

As a benchmark, they fine-tuned a separate AraBERT base model for each event. Even with small per-event training sets, the models reached solid accuracy and low Hamming loss, and macro-F1 stayed above 70% in nearly every case without any rebalancing. The resource supports situational-awareness tools, crisis detection and opinion analysis for a low-resource, dialect-rich language.

Key points

• Kawārith holds more than 1M Arabic tweets from 22 crises (2018–2020), released with the query terms and collection dates.
• The gold set has 12,446 labelled tweets from seven events, including Jordan floods, Cairo bombing, Dragon storms, Beirut explosion and COVID-19.
  • Average Krippendorff's alpha is about 0.7.
  • Only 4.4% of items carry more than one label.
  • About 15% of the COVID-19 sample is irrelevant.
• The authors also publish 405 domain-independent, multi-dialect Arabic stop words.
• AraBERT base setup:
  • per-event stratified 80/20 split;
  • dropout 0.2 before a linear layer, binary cross-entropy;
  • learning rate 5e-5, maximum length 60, 5 epochs;
  • batch sizes 8 and 32, each run with three seeds.
• Results:
  • COVID-19 relevance classification reached about 94% accuracy.
  • Multi-label Hamming loss stayed roughly between 0.05 and 0.08.
  • With fewer than about 600 training examples, batch size 8 beat 32.

Interesting ideas

• I1. Collection grows from seed hashtags, adds newly found tags, and stops once no fresh relevant ones appear; it prefers event-specific terms over generic hazard words, trading recall for cleaner data.
• I2. Disagreements go to a third annotator, majority wins, and a tweet is dropped entirely when all three disagree, rather than forcing a label.
• I3. Labels are multi-label (a tweet can be both an update and a prayer), and scoring adds Hamming loss alongside macro and micro F1.
• I4. The authors note that irrelevant posts were often confused with the opinions class, and propose a binary relevance filter before fine-grained categorisation.
• I5. Every experiment is repeated with three seeds, and a smaller batch size proved better in the low-data regime.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): silver corpora for rare constructions could be built the same way, expanding seed patterns only while they keep pulling in genuine examples, so the training set stays clean.
• I2 (learn): when several witness parsers produce three different analyses, the sentence could be discarded from silver data instead of resolved by a tie-breaking rule.
• I3 (frames): speech-act and implicature tags could be allowed to co-occur, with Hamming loss showing partially correct label sets that exact-match accuracy would hide.
• I4 (answer): a cheap gate that checks whether the input is actually a question about the text could run before the readers, so off-topic inputs do not get forced into some answer type.
• I5 (learn): parser and classifier evaluations could report means over several seeds, and small-treebank training could default to smaller batches.