Elamur Research⌕
☰
home / papers / 2017 / CAT: Credibility Analysis of Arabic Content on Twitter

CAT: Credibility Analysis of Arabic Content on Twitter

Paper · 2017 · Rim El Ballouli, Wassim El-Hajj, Ahmad Ghandour et al.

Summary

The paper addresses automatic credibility judgment for individual Arabic tweets. Unreliable or spam posts degrade downstream tasks such as sentiment analysis and rumor detection, and Arabic had received little attention here. The authors collected about 17 million hashtagged Arabic tweets over two weeks and filtered out religious quotations, ads, emoticon-only posts, obscene posts and retweets. They then sampled a topic-independent set of 9,000 tweets. Each tweet received three crowd judgments on an in-house platform. The CAT classifier is binary and combines 26 content-based features with 22 user-based ones. Some of the user features summarize the author's past timeline, and others come from profile-picture analysis via Google Cloud Vision. CAT beat several baselines and the TweetCred system by a sizable margin in weighted average F-measure. The paper gives 21% in the abstract and 16.7% in the introduction. A feature-reduction study found that compact subsets outperform the full set. URL-related signals, negative sentiment and timeline habits proved most predictive. The annotated corpus is released publicly.

Key points

  • Dataset: 9,000 Arabic tweets, built by taking a random 10% from each hashtag cluster so that topics are mixed.
  • Annotation: six paid judges gave three labels per tweet (credible / non-credible / can't decide). Judges saw both the tweet page and the author's full profile.
  • Features: 26 content-based (sentiment, text cues, URLs, punctuation) and 22 user-based (follower counts, expertise, timeline statistics, face detection on the profile image).
  • Signal strength of URLs: about 80% of tweets containing a URL were credible, versus about 40% without one. For authors with versus without a profile URL, the figures were 74% vs 47%.
  • Feature selection: a graph-search selection stopped after 100 non-improving expansions and evaluated roughly 1,000–4,000 subsets. The best subsets were under half the original size yet scored higher. Features such as day of week, user mentions and character counts turned out irrelevant.

Interesting ideas

  • I1. Examining the complement of each selected feature subset reveals which features carry no information (e.g., weekday), which is a cheap way to confirm irrelevance rather than just rank importance.
  • I2. Topic independence was enforced by drawing an equal fraction from every hashtag cluster instead of sampling the raw stream, which would be dominated by trending topics.
  • I3. Aggregating a speaker's history (average URLs, hashtags, retweet share, posting gaps) gives a reliability prior that a single message cannot supply.
  • I4. Annotators were given a "can't decide" label and rich context (profile, prior posts), which acknowledges that the task is subjective and keeps forced guesses out of the gold data.
  • I5. Feature-subset search with a "stale state" cutoff (stop after N expansions without improvement) makes exhaustive selection tractable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): run evaluation on the inverse of selected parser or repair-rule features as a negative control, checking that leftovers really add nothing.
  • I2 (learn): build silver and test sets by sampling proportionally from source or genre clusters, so scores are not inflated by one dominant domain.
  • I3 (world): keep per-source statistics as entity attributes in the text world, so claims can be weighted by who asserted them.
  • I4 (learn): let LLM teacher tiers and witness parsers abstain explicitly, and drop abstained items from silver data instead of taking a majority guess.
  • I5 (tree): apply the same bounded search to prune perceptron parser feature templates while tracking LAS on held-out treebanks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon34 works · 2,561 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓Université Joseph FourierGrenoble · France3 works · 127 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Rim El Ballouli, Wassim El-Hajj, Ahmad Ghandour, Shady Elbassuoni, Hazem Hajj and Khaled Shaban. 2017. CAT: Credibility Analysis of Arabic Content on Twitter. Proceedings of the Third Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1308

@inproceedings{ballouli2017cat,
  title = {{CAT: Credibility Analysis of Arabic Content on Twitter}},
  author = {Rim {El Ballouli}, Wassim El-Hajj, Ahmad Ghandour, Shady Elbassuoni, Hazem Hajj and Khaled Shaban},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1308},
  url = {https://doi.org/10.18653/v1/w17-1308}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmad A Al Sallab, Ramy Baly, Gilbert Badaro, Hazem Hajj, Wassim El Hajj, and Khaled B Shaban. 2015. Deep learning models for sentiment analysis in arabic. In ANLP Workshop 2015, page 9. ↗
  2. Aditi Gupta, Ponnurangam Kumaraguru, Carlos Castillo, and Patrick Meier. 2014. Tweetcred: A real-time web-based system for assessing credibility of content on twitter. arXiv Preprint arXiv:1405.5490. ↗
  3. Ahmad A Al Sallab, Ramy Baly, Gilbert Badaro, Hazem Hajj, Wassim El Hajj, and Khaled B Shaban. in press 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing. find
  4. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10– 18. ↗
  5. Roula Hobeica, Hazem Hajj, and Wassim El Hajj. 2011. Machine reading for notion-based sentiment mining. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on, pages 75– 80. IEEE. find
  6. Ayman Al Zaatari, Reem El Ballouli, Shady ELbassouni, Wassim El-Hajj, Hazem Hajj, Khaled Bashir Shaban, and Nizar Habash. 2016. Arabic corpora for credibility analysis. Proceedings of the Language Resources and Evaluation Conference (LREC), pages 4396–4401.
  7. Byungkyu Kang, John O’Donovan, and Tobias HÃűllerer. 2012. Modeling topic specific credibility on twitter. ACM International Conference on Intelligent User Interfaces. find
  8. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Technical report, National Bureau of Economic Research. ↗
  9. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  10. Amal Abdullah AlMansour. 2016. CREDIBILITY AS-SESSMENT FOR ARABIC MICRO-BLOGS USING NOISY LABELS. Ph.D. thesis, King’s College London. find
  11. M. Maamouri, D. Graff, B. Bouziri, S. Krouna, and S. Kulick. 2010. Ldc standard arabic morphological analyzer (sama) v. 3.1. LDC Catalog no.LDC2010L01.ISBN, pp. 1-58563, 2010. find
  12. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale arabic sentiment lexicon for arabic opinion mining. ANLP.
  13. G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. 1990. Introduction to wordnet: An online lexical database. International Journal of Lexicography, vol. 3, pp. 235-244. find
  14. Gilbert Badaro, Ramy Baly, Rana Akel, Linda Fayad, Jeffrey Khairallah, Hazem Hajj, Wassim El-Hajj, John ODonovan, Byungkyu Kang, Georg Meyer, Tobias Hollerer, and Sibel Adalii. 2012. Credibility in context: An analysis of feature distributions in twitter. Privacy, Security, Risk and Trust (PASSAT), International Conference on Social Computing (SocialCom). find
  15. A. Pasha, M. Al-Badrashiny, A. E. Kholy, R. Eskander, M. Diab, N. Habash, M. Pooleery, O. Rambow, and R. Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. Proceedings of the 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland.
  16. Mohammad Bin Sultan, Hend AlKhalifa, and Abdul-Malik AlSalman. 2010. Measuring the credibility of arabic text content in twitter. Digital Information Management (ICDIM). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ballouli-2017-cat-credibility-analysis-arabic-content/