Elamur Research⌕
☰
home / papers / 2019 / Crisis Detection from Arabic Tweets

Crisis Detection from Arabic Tweets

Paper · 2019 · Alaa Alharbi and Mark Lee

Summary

The paper looks at automatically separating flood-related posts from unrelated ones in Arabic Twitter. Arabic social media is hard to process because of dialect variety and unstandardised spelling, and no public Arabic crisis corpus existed. The authors built one themselves. They crawled tweets about ten Middle Eastern floods from autumn 2018 and annotated four of them: Jordan, Kuwait, Qurayyat and Al-Lith. Each tweet received a relevance label and an information-type label. They then compared two classical classifiers (linear SVM, multinomial NB) against four neural models (CNN, LSTM, CLSTM, BiLSTM) in two setups. In the first, training and testing used the same event. In the second, models were trained only on earlier events and tested on a later one. Within a single event, recurrent models scored highest, reaching about 95% accuracy. Across events, accuracy fell to roughly 61–78%, though neural models usually stayed ahead. This setup matters in practice because labelled data from a crisis that is still unfolding is rarely available in time.

Key points

  • The gold-standard corpus contains 4,037 labelled tweets from four 2018 floods in different regions, so several dialects are covered. About 24.69% of the tweets are irrelevant.
  • Each tweet was labelled by two native speakers, and a third person resolved disagreements.
    • Cohen's Kappa was ≈0.82 for relevance and ≈0.9 for information type.
    • Information types follow the scheme of Olteanu et al. (2015): affected people, infrastructure damage, donations, caution/advice, sympathy, other useful information, and not applicable.
  • Same-event results (5-fold CV):
    • BiLSTM was best on Jordan (92.14) and Qurayyat (95.48).
    • LSTM was best on Kuwait (91.91) and Al-Lith (94.08).
    • SVM was competitive, while NB was clearly weakest.
  • Cross-event results:
    • The best score was 77.95 (LSTM), achieved by training on all three earlier events and testing on Al-Lith.
    • LSTM beat CNN in 9 of 11 train/test pairs.
  • Adding more training events did not always help. For the DNNs, Kuwait+Qurayyat gave lower scores on Al-Lith than Qurayyat alone.

Interesting ideas

  • I1. The test is chronological across events: models learn only from floods that happened earlier and are scored on a later one, which mimics real deployment better than random splits.
  • I2. Pooling more training sources is not reliably better; specific combinations of events lowered accuracy compared with a single well-matched event.
  • I3. Relevance and informativeness are labelled separately, and error analysis showed that on-topic but uninformative posts were often mistaken for off-topic ones.
  • I4. Before sampling, retweets are collapsed to their original text and exact duplicates are removed, which prevents the same message from leaking into both training and test.
  • I5. Disagreements between two annotators go to a third judge, rather than being averaged or dropped.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Hold out whole sources or genres in time order as a test set, so parser and reader scores show transfer and not memorised style.
  • I2 (learn): When merging treebanks or silver sets, ablate each added source and keep only those that measurably help, instead of assuming bigger is better.
  • I3 (frames): Keep "is this about the event" apart from "what act does this utterance perform", so that sympathy or prayer utterances are tagged as expressive acts and not discarded as noise.
  • I4 (learn): Deduplicate sentences across silver and evaluation pools before splitting, so agreement scores among witness parsers are not inflated by repeats.
  • I5 (learn): When two witness parsers disagree on an arc, escalate only that case to a third witness or an LLM teacher tier, which keeps adjudication cheap and targeted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksCommunication · 97 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Alaa Alharbi and Mark Lee. 2019. Crisis Detection from Arabic Tweets. Proceedings of the 3rd Workshop on Arabic Corpus Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alharbi2019crisis,
  title = {{Crisis Detection from Arabic Tweets}},
  author = {Alaa Alharbi and Mark Lee},
  booktitle = {Proceedings of the 3rd Workshop on Arabic Corpus Linguistics},
  year = {2019},
  url = {https://aclanthology.org/W19-5609/}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fabian Abel, Claudia Hauff, Geert-Jan Houben, Richard Stronkman, and Ke Tao. 2012. Twitcident: fighting fire with information from social web streams. In Proceedings of the 21st International Conference on World Wide Web, pages 305–308. ACM. find
  2. Waleed Alabbas, Haider M al Khateeb, Ali Mansour, Gregory Epiphaniou, and Ingo Frommholz. 2017. Classification of colloquial arabic tweets in real-time to detect high-risk floods. In 2017 International Conference On Social Media, Wearable And Web Analytics (Social Media), pages 1–8. IEEE. find
  3. Mohamed Arafa and Crystal Armstrong. 2016. ” facebook to mobilize, twitter to coordinate protests, and youtube to tell the world”: New media, cyberac-tivism, and the arab spring. Journal of Global Initiatives: Policy, Pedagogy, Perspective, 10(1):6. find
  4. Zahra Ashktorab, Christopher Brown, Manojit Nandi, and Aron Culotta. 2014. Tweedr: Mining twitter to inform disaster response. In ISCRAM. find
  5. Fabricio Benevenuto, Gabriel Magno, Tiago Rodrigues, and Virgilio Almeida. 2010. Detecting spammers on twitter. In Collaboration, electronic messaging, anti-abuse and spam conference (CEAS), volume 6, page 12. find
  6. Mark A Cameron, Robert Power, Bella Robinson, and Jie Yin. 2012. Emergency situation awareness from twitter for crisis management. In Proceedings of the 21st International Conference on World Wide Web, pages 695–698. ACM. find
  7. Cornelia Caragea, Adrian Silvescu, and Andrea H Tapia. 2016. Identifying informative messages in disaster events using convolutional neural networks. In International Conference on Information Systems for Crisis Response and Management, pages 137– 147. find
  8. David Chiang, Mona Diab, Nizar Habash, Owen Rambow, and Safiullah Shareef. 2006. Parsing arabic dialects. In 11th Conference of the European Chapter of the Association for Computational Linguistics. ↗
  9. Yarin Gal. 2016. Uncertainty in deep learning. Ph.D. thesis, PhD thesis, University of Cambridge. find
  10. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2010. Part-of-speech tagging for twitter: Annotation, features, and experiments. Technical report, Carnegie-Mellon Univ Pittsburgh Pa School of Computer Science. ↗
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  12. Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2018. Processing social media messages in mass emergency: Survey summary. In Companion of the The Web Conference 2018 on The Web Conference 2018, pages 507–511. International World Wide Web Conferences Steering Committee. find
  13. Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013a. Extracting information nuggets from disaster-related messages in social media. In Iscram. find
  14. Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013b. Practical extraction of disaster-relevant information from social media. In Proceedings of the 22nd International Conference on World Wide Web, pages 1021–1024. ACM. find
  15. Abdullah Talha Kabakus and Resul Kara. 2017. A survey of spam detection methods on twitter. International Journal of Advanced Computer Science and Applications, 8(3). ↗
  16. Habibul Haque Khondker. 2011. Role of the new media in the arab spring. Globalizations, 8(5):675– 679. ↗
  17. Jooho Kim, Juhee Bae, and Makarand Hastak. 2018. Emergency information diffusion on online social media during storm cindy in us. International Journal of Information Management, 40:153–165. find
  18. Jooho Kim and Makarand Hastak. 2018. Social network analysis: Characteristics of online social networks after a disaster. International Journal of Information Management, 38(1):86–96. ↗
  19. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗
  20. Kirill Kireyev, Leysia Palen, and Kenneth Anderson. 2009. Applications of topics models to analysis of disaster-related twitter data. In NIPS Workshop on applications for topic models: text and beyond, volume 1. Canada: Whistler. find
  21. Venkata Kishore Neppalli, Cornelia Caragea, and Doina Caragea. 2018. Deep neural networks versus naı̈ve bayes classifiers for identifying informative tweets during disasters. find
  22. Dat Tien Nguyen, Kamela Ali Al Mannai, Shafiq Joty, Hassan Sajjad, Muhammad Imran, and Prasenjit Mitra. 2017. Robust classification of crisis-related data on social networks using convolutional neural networks. In Eleventh International AAAI Conference on Web and Social Media. find
  23. Alexandra Olteanu, Sarah Vieweg, and Carlos Castillo. 2015. What to expect when the unexpected happens: Social media communications across crises. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, pages 994–1009. ACM. find
  24. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  25. Sasa Petrovic, Miles Osborne, Richard McCreadie, Craig Macdonald, Iadh Ounis, and Luke Shrimpton. 2013. Can twitter replace newswire for breaking news? In Seventh international AAAI conference on weblogs and social media. find
  26. Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, pages 851–860. ACM. find
  27. Jyoti Prakash Singh, Yogesh K Dwivedi, Nripendra P Rana, Abhinav Kumar, and Kawaljeet Kaur Kapoor. 2017. Event classification and location prediction from tweets during disasters. Annals of Operations Research, pages 1–21. ↗
  28. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  29. Zeynep Tufekci and Christopher Wilson. 2012. Social media and the decision to participate in political protest: Observations from tahrir square. Journal of communication, 62(2):363–379. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alharbi-2019-crisis-detection-arabic-tweets/