Elamur Research⌕
☰
home / papers / 2016 / Arabic Corpora for Credibility Analysis

Arabic Corpora for Credibility Analysis

Paper · 2016 · Ayman Al Zaatari, Rim El Ballouli, Shady ELbassouni et al.

Summary

The paper addresses a resource gap: credibility classifiers for social media need human-labelled training data, and before this work no public Arabic corpus of blogs or tweets had credibility annotations. The authors built two corpora. The first contains 2,708 Arabic tweets about two heavily discussed Syrian news topics, labelled credible or non-credible by three qualified annotators. Each annotator saw the tweet, the author's full profile, and Google results restricted to the days around the tweet's posting. The second contains 175 Arabic blog posts, a mix of news and opinion, rated as credible, fairly credible, or non-credible. The blog posts were also annotated for reasonability, bias, objectivity, and sentiment, four dimensions chosen through a pilot study. The authors argue that tweets and blogs need different annotation schemes, because the evidence for judging them sits in different places. A blog classifier reached about 73% accuracy on the original data and 80% after SMOTE balancing, well above trivial baselines. Both corpora are publicly released through the OMA project site.

Key points

  • About 36 million tweets were streamed over two weeks and indexed with Apache Lucene, then queried with Arabic news topics. The two largest topics were annotated: "forces of the Syrian government" (1,131 credible / 510 non-credible) and "Syrian revolution" (439 credible / 628 non-credible).
  • Five candidates took a tutorial and qualification test, and three passed. Gold and repeated tweets were mixed in to monitor annotation quality. Fleiss' kappa was 0.43, which counts as moderate agreement, and final labels were set by majority vote.
  • The blog corpus has 175 posts (90 news, 85 opinion) collected through Google Blogs search on trending topics. These included the Lebanese elections, the Syrian crisis, the Special Tribunal for Lebanon, the FIFA World Cup 2014, and phones.
  • The four extra blog features came from a 25-blog pilot study. Features were ranked by Information Gain in WEKA, and the pilot annotators confirmed the choice.
  • On the blogs, the CAB classifier was tested with Decision Table and Naïve Bayes learners. It scored roughly 71–73% accuracy and 70–72% F-measure unbalanced, and around 80% with SMOTE. Stratified, uniform, and majority-class baselines were far lower.

Interesting ideas

  • I1. Hidden gold items plus silently repeated items check annotators for both accuracy and self-consistency, without them knowing which items are tests.
  • I2. Annotation dimensions were picked empirically: a small pilot with many candidate features, then Information Gain ranking, keeping only those that actually predict the target label.
  • I3. Evidence shown to annotators was time-boxed, with search restricted to ±5 days around the post, so judgments reflect what was knowable at that moment.
  • I4. The annotation scheme follows where the evidence lives: tweets are judged with external context such as the author profile and web results, while blogs are judged mainly from their own text.
  • I5. Results are reported against stratified, uniform, and majority-class baselines, before and after class balancing, which shows how much of the score comes from label skew.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): mix known-answer and duplicate sentences into LLM-teacher or witness-parser batches, and down-weight sources that contradict themselves or miss the gold.
  • I2 (frames): before committing to new pragmatic labels (e.g. irony or speech-act subtypes), pilot-annotate a few texts and keep only the labels that carry information about the final judgment.
  • I3 (world): attach a time anchor to facts pulled from compiled knowledge, so a claim in a text is checked against the state of the world at the text's date rather than today.
  • I4 (answer): tag each test question by whether it is answerable from the text alone or needs background knowledge, and score the readers separately on the two kinds.
  • I5 (learn): always report these trivial baselines next to evaluation numbers as negative controls, especially on skewed label sets like rare dependency relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ayman Al Zaatari, Rim El Ballouli, Shady ELbassouni, Wassim El-Hajj, Hazem Hajj, Khaled Shaban, Nizar Habash and Emad Yahya. 2016. Arabic Corpora for Credibility Analysis. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaatari2016arabic,
  title = {{Arabic Corpora for Credibility Analysis}},
  author = {Ayman {Al Zaatari}, Rim {El Ballouli}, Shady ELbassouni, Wassim El-Hajj, Hazem Hajj, Khaled Shaban, Nizar Habash and Emad Yahya},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/3rq52ttyazqb},
  url = {https://doi.org/10.63317/3rq52ttyazqb}
}

References

22 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Eidan, R. M. B., Al-Khalifa, H. S., & Al-Salman, A. S. (2009). Towards the measurement of arabic weblogs credibility automatically. Proceedings of the 11th International Conference on Information Integration and Web-Based Applications & Services, pp. 618-622. find
  2. Al-Eidan, R., Al-Khalifa, H., & Al-Salman, A. (2010). Measuring the credibility of arabic text content in twitter. Digital Information Management (ICDIM), 2010 Fifth International Conference On, pp. 285-291. find
  3. Badaro, G., Baly, R., Hajj, H., Habash, N., & El-Hajj, W. (2014). A large scale arabic sentiment lexicon for arabic opinion mining. ANLP 2014, 165
  4. Białecki, A., Muir, R., & Ingersoll, G. (2012). Apache lucene 4. SIGIR 2012 Workshop on Open Source Information Retrieval, pp. 17-24. find
  5. Castillo, C., Mendoza, M., & Poblete, B. (2011). Information credibility on twitter. Proceedings of the 20th International Conference on World Wide Web, pp. 675-684. find
  6. Flanagin, A. J., & Metzger, M. J. (2000). Perceptions of internet information credibility. Journalism & Mass Communication Quarterly, 77(3), 515-540. ↗
  7. Fogg, B., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003). How do users evaluate the credibility of web sites?: A study with over 2,500 participants. Proceedings of the 2003 Conference on Designing for User Experiences, pp. 1-15. find
  8. Gayo-Avello, Panagiotis Takis Metaxas, Eni Mustafaraj, Markus Strohmaier, Harald Schoen and Peter Gloor, Daniel, Castillo, C., Mendoza, M., & Poblete, B. (2013). Predicting information credibility in time-sensitive social media. Internet Research, 23(5), 560-588. find
  9. Gupta, A., & Kumaraguru, P. (2012). Credibility ranking of tweets during high impact events. Proceedings of the 1st Workshop on Privacy and Security in Online Social Media, pp. 2. find
  10. Gupta, A., Kumaraguru, P., Castillo, C., & Meier, P. (2014). TweetCred: A real-time web-based system for assessing http://oma-project.com/res/tweet_corpus http://oma-project.com/res/blog_corpus credibility of content on twitter. arXiv Preprint arXiv:1405.5490, Gupta, M., Zhao, P., & Han, J. (2012). Evaluating event credibility on twitter. SDM, pp. 153-164. ↗
  11. Habash, N. (2010). Introduction to Arabic Natural Language Processing, Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  12. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1), 10-18. ↗
  13. Juffinger, A., Granitzer, M., & Lex, E. (2009). Blog credibility ranking by exploiting verified content. Proceedings of the 3rd Workshop on Information Credibility on the Web, pp. 51-58. find
  14. Metzger, M. J. (2007). Making sense of credibility on the web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58(13), 2078-2091. ↗
  15. Nakamura, A., Suzuki, Y., & Ishikawa, Y. (2013). Clustering editors of wikipedia by editor's biases. Web Intelligence (WI) and Intelligent Agent Technologies (IAT), 2013 IEEE/WIC/ACM International Joint Conferences On, , 1. pp. 351-358. find
  16. Olteanu, A., Peshterliev, S., Liu, X., & Aberer, K. (2013). Web credibility: Features exploration and credibility prediction. Advances in information retrieval (pp. 557-568) Springer. find
  17. Pasha, A., Al-Badrashiny, M., Kholy, A. E., Eskander, R., Diab, M., Habash, N., et al. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland, Schwarz, J., & Morris, M. (2011). Augmenting web pages and search results to support credibility assessment. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1245-1254.
  18. Stoyanov, V., & Cardie, C. (2008). Annotating topics of opinions. find
  19. LREC, Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010). Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12), 2544-2558. ↗
  20. Ulicny, B., Baclawski, K., & Magnus, A. (2007). New metrics for blog mining. Defense and Security Symposium, pp. 65700I-65700I-12. ↗
  21. Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Fam Med, 37(5), 360-363. find
  22. Weerkamp, W., & de Rijke, M. (2012). Credibility-inspired ranking for blog post retrieval. Information Retrieval, 15(3-4), 243-277. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaatari-2016-arabic-corpora-credibility-analysis/