📄 Towards a Corpus of Violence Acts in Arabic Social Media

Summary

The authors build a resource to help human rights organisations find reports of abuse in Arab social media. Their sources are often in conflict zones, and most posts are written in dialects rather than MSA. Data collection had two stages. First, a keyword stream from the Twitter API used a hand-built list of 237 Arabic violence stems, expanded with orthographic variants and affixes; it gathered 557,576 tweets. Second, an advanced filter removed emoticon-bearing posts, very short posts, sexual adverts and near-duplicates, leaving 76,619 tweets. Of these, 20,151 were labelled on CrowdFlower. Each tweet received at least five judgements under a scheme of violence classes (HRA, crime, conflict, violence, accident, crisis, political opinion, plus "other"), whose definitions were refined in a pilot and reviewed by Minority Rights Group experts. Each label gets a confidence score weighted by annotator trust. Fleiss' kappa rises from 0.50 over all tweets to 0.85 for tweets with confidence ≥ 0.8. Agreement stays poor for Conflict and, below high confidence, for Crime. The corpus is freely released.

Key points

• Expanding the keyword list:
  • The 237 stems were multiplied by reversing Darwish's (2002) letter normalization, generating every spelling with the easily confused alef, final ya and final ta-marbuta/ha forms.
  • Common prefixes and suffixes were attached.
  • Malformed generated forms were then pruned by hand.
• Advanced filter steps:
  • drop tweets containing emoticons;
  • drop tweets with fewer than 5 tokens;
  • remove sexual adverts using a 15-word list of advertiser terms and hashtags;
  • deduplicate by normalizing text (strip hashtags, line breaks and separators, apply letter normalization) and hashing it with SHA-1.
• Annotation scheme development:
  • A pilot used 5 classes, 31 test questions, 1,025 tweets and 3 judgements per tweet.
  • The final job used the expanded class set, 206 test questions and at least 5 judgements per tweet, with more raters added when agreement was low.
• Aggregate label distribution:
  • other 38%, opinion 21%, conflict 16%, HRA 12% (2,367 tweets), crime 7%;
  • accident, violence and crisis each 1–3%;
  • aggregate and individual-judgement distributions differ by only about ±2%.
• Kappa by confidence threshold:
  • Overall kappa is 0.50 (all tweets), 0.78 (CS ≥ 0.7), 0.85 (≥ 0.8) and 0.99 (≥ 0.9).
  • Conflict never exceeds 0.37.
  • Crime jumps to 0.91 only at CS ≥ 0.9.
  • Two releases exist: the raw job report with duplicates and a cleaned set of unique tweets.

Interesting ideas

• I1. Run a normalization rule backwards to enumerate all plausible surface spellings of a lexicon entry, so an exact-match filter can catch variants it would otherwise miss.
• I2. Score each candidate label as the trust-weighted share of the raters who chose it, then measure agreement separately in confidence buckets instead of reporting one global number.
• I3. Detect near-duplicates by stripping cosmetic material (hashtags, spacing, letter variants) before hashing, so trivially edited copies collapse to one key.
• I4. Run a small pilot with seeded gold questions, study which questions annotators missed and their objections, then revise the class inventory before the full job.
• I5. The HRA vs. Conflict confusion shows that the same verb ("killed", "martyr") gets different event labels depending on who the agent is (state forces vs. combatants) and whether the victim is a civilian.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Inverting our spelling normalizers (British/American, hyphenation, apostrophe variants) could generate lookup keys for lexicon and gazetteer matching without a fuzzy matcher.
• I2 (learn): Weighting witness parsers by their measured reliability and reporting agreement per confidence band would show where silver data is safe to train on and where it is not.
• I3 (learn): A normalize-then-hash pass over training and test sentences would guard evaluation against leakage from lightly reformatted duplicates.
• I4 (learn): Seeding LLM-teacher batches with known-answer items and revising label schemes after a pilot would act as a negative control on teacher quality.
• I5 (frames): Event-type labels should be derived from role fillers (agent's authority status, victim type) in the frame rather than from the trigger word, which keeps the decision explainable.