Elamur Research⌕
☰
home / papers / 2014 / An Arabic Twitter Corpus for Subjectivity and Sentiment Analysis

An Arabic Twitter Corpus for Subjectivity and Sentiment Analysis

Paper · 2014 · Eshrag Refaee and Verena Rieser

Summary

Sentiment work on Arabic lags far behind English. Part of the reason is the language's rich morphology. Another part is that annotated social-media data was rarely shared. Refaee and Rieser address this gap by building a hand-annotated collection of Arabic tweets for analysing subjectivity and sentiment (SSA).

They collected tweets through the Twitter Search API using topic-oriented queries about products, public figures, sport, social issues and similar subjects. The data come from two separate time windows, giving a development set and a test set. Two native speakers labelled each tweet as positive, negative, mixed or neutral and reached κ = 0.816.

Each tweet also carries automatically extracted features:

  • morphology from MADA+TOKAN, plus a flag marking whether analysis succeeded,
  • word n-grams,
  • hits in a merged sentiment lexicon,
  • emoticons,
  • social-signal cues.

Comparing the two time slices shows strong topic drift: frequent content words barely overlap. In related work, classifiers trained on one slice generalised poorly to the other. The corpus matters as an early publicly released Arabic SSA resource and as evidence that Twitter's shifting topics undermine word-based models.

Key points

  • The corpus has 8,868 gold tweets. The development set holds 7,503 tweets (20 Jan–21 Feb 2014) and the test set 1,365 (6–15 Nov 2013). Observed agreement was 91.74%, with weighted κ = 0.816.
  • Polar tweets are those labelled positive or negative. Any tweet that even one annotator marked other/uncertain was discarded, as were ads and redundant tweets given the skip label.
  • The combined sentiment lexicon has 4,422 entries, built from three sources:
    • ArabSenti,
    • MPQA machine-translated into Arabic and hand-corrected (2,627 entries),
    • a new 484-word dialectal lexicon, which is released.
  • Morphological features come from MADA+TOKAN v3.2. Because that tool covers only MSA, a "has morphological analysis" feature marks dialectal or misspelled words it cannot handle.
  • The release goes through ELRA in CSV and ARFF formats. Tweets appear as IDs only, for privacy reasons. The training portion has 96,493 tokens and 26,724 unique word types.

Interesting ideas

  • I1. Annotators had to state a reason for each label, and those reasons were used to refine the scheme in rounds on a small subset; this also shows when people agree on a label but for different reasons.
  • I2. Instead of silently trusting an analyser outside its domain, a separate feature records whether it produced any analysis at all, so downstream models know when the morphology is unreliable.
  • I3. Training and test data were drawn from separate time windows, and comparing their frequent content words revealed topic drift that random splits would hide.
  • I4. When translating an English polarity lexicon, the authors corrected entries whose context-free polarity changed in translation, for instance a phrase that is negative in English but neutral in Arabic.
  • I5. An error analysis found that emoticon-based distant labels became noisy because emoticon direction is ambiguous in right-to-left text; even "obvious" weak signals depend on the writing conventions of the text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers and human checks could return a short justification with each silver label, so label-level agreement can be separated from reason-level agreement before data enters training.
  • I2 (tokens): the en tagger and lemmatizer could expose an explicit "no analysis / out-of-lexicon" flag per token, letting later stages lower confidence instead of acting on a guessed lemma.
  • I3 (learn): evaluation sets could be split by source or period rather than at random, with a vocabulary-overlap check acting as a negative control against topic memorisation.
  • I4 (world): when compiling value or attitude knowledge from external resources, entries should be checked for polarity that depends on context, so the ontology records default evaluations rather than connotations of a particular usage.
  • I5 (frames): irony and speech-act cues such as emoticons or laugh tokens should be normalised for orientation and treated as weak evidence, never as decisive labels for stance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heriot-Watt UniversityEdinburgh · United Kingdom15 works · 5,365 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Eshrag Refaee and Verena Rieser. 2014. An Arabic Twitter Corpus for Subjectivity and Sentiment Analysis. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{refaee2014arabic,
  title = {{An Arabic Twitter Corpus for Subjectivity and Sentiment Analysis}},
  author = {Eshrag Refaee and Verena Rieser},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/3zem5qb5stkg},
  url = {https://doi.org/10.63317/3zem5qb5stkg}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M. and Diab, M. (2012a). Awatif: A multi-genre corpus for modern standard arabic subjectivity and sentiment analysis. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
  2. Abdul-Mageed, M. and Diab, M. (2012b). Awatif: A multi-genre corpus for modern standard arabic subjectivity and sentiment analysis. In Proceedings of the Eight International Conference on Language Resources and  Example Ð Q m ' (respect) Gen: m mod: i num: per: stt: s 3 na vox: a Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
  3. Abdul-Mageed, M., Diab, M. T., and Korayem, M. (2011). Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 587–591, Stroudsburg, PA, USA. Association for Computational Linguistics.
  4. Al-Sabbagh, R. and Girju, R. (2012). Yadac: Yet another dialectal arabic corpus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
  5. Bifet, A. and Frank, E. (2010). Sentiment knowledge discovery in twitter streaming data. In Discovery Science, pages 1–15. Springer. find
  6. Carletta, J. (1996). Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2):249–254. ↗
  7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  8. Diab, M., Habash, N., Rambow, O., Altantawy, M., and Benajiba, Y. (2010). COLABA: Arabic dialect annotation and processing. In Proceedings of the LREC Workshop on Semitic Language Processing). European Language Resources Association (ELRA). find
  9. Eisenstein, J. (2013). What to do about bad language on the internet. In Proceedings of NAACL-HLT, pages 359– 369.
  10. Habash, N. and Rambow, O. (2005). Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  11. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  12. Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. ↗
  13. Mourad, A. and Darwish, K. (2013). Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs. WASSA 2013, page 55.
  14. Nizar Habash, O. R. and Roth, R. (2009). MADA+TOKAN: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Choukri, K. and Maegaard, B., editors, Proceedings of the Second International Conference on Arabic Language Resources and Tools, Cairo, Egypt, April. The MEDAR Consortium. find
  15. Refaee, E. and Rieser, V. (2014a). Can we read emotions from a smiley face? Emoticon-based distant supervision for subjectivity and sentiment analysis of arabic twitter feeds. In 5th International Workshop on Emotion, Social Signal, Sentiment & Linked Open Data. find
  16. Refaee, E. and Rieser, V. (2014b). Subjectivity and sentiment analysis of arabic twitter feeds with limited resources. In 9th International Conference on Language Resources and Evaluation (LREC’14). find
  17. Wilson, T., Wiebe, J., and Hoffmann, P. (2009). Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Computational Linguistics, 35(3):399–433. ↗
  18. Zaidan, O. F. and Callison-Burch, C. (2013). Arabic dialect identification. Computational Linguistics. ↗
  19. Zhang, L., Ghosh, R., Dekhil, M., Hsu, M., and Liu, B. (2011). Combining lexiconbased and learning-based methods for twitter sentiment analysis. HP Laboratories, Technical Report HPL-2011, 89. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/refaee-2014-arabic-twitter-corpus-subjectivity-sentiment/