Elamur Research⌕
☰
home / papers / 2014 / LABR: A Large Scale Arabic Sentiment Analysis Benchmark

LABR: A Large Scale Arabic Sentiment Analysis Benchmark

Paper · 2014 · Mahmoud Nabil, Mohamed Aly and Amir Atiya

Summary

Arabic sentiment analysis has lacked large, open, comparable data. Earlier Arabic corpora held a few thousand examples at most, were mostly not released, and had no fixed train/test division. The authors built LABR by harvesting book reviews from Goodreads, cleaning them and keeping only purely Arabic texts. The result is 63,257 reviews, each carrying a 1–5 star rating. They describe the corpus statistics and define two tasks: positive/negative polarity and full star-rating prediction. For both tasks they publish fixed training, validation and test splits, in balanced and unbalanced variants. They then run a broad set of standard text classifiers as baselines; SVM and logistic regression come out strongest. They also derive a domain-specific sentiment lexicon from the training portion, containing single words and multi-word expressions. That lexicon reaches competitive accuracy while using only a small share of the features. The dataset and reproduction scripts are public, so LABR can serve as a common benchmark for Arabic opinion mining.

Key points

  • Collection: over 220,000 reviews were crawled from goodreads.com in March 2013, taken from the first 2,143 titles of a "Best Arabic Books" list. More than 70% turned out to be non-Arabic and were discarded.
  • Cleaning: the authors stripped HTML, links, newlines and decorative Unicode, and collapsed repeated dots. Any review containing characters outside Arabic letters, digits and punctuation was then dropped.
  • Final corpus statistics:
    • 63,257 reviews, 16,486 users and 2,131 books.
    • About 4.13M tokens and 342,199 sentences.
    • Median review length is 33 tokens, the mean is 65 and the maximum is 3,736.
  • The ratings are heavily skewed toward 4–5 stars. The labels are also noisy, and 3-star reviews are ambiguous in polarity.
  • Comparison with earlier Arabic sets: OCA, AWATIF, TAGREED, TAHRIR and MONTADA range from 500 to about 3,100 items, so LABR is roughly an order of magnitude larger.

Interesting ideas

  • I1. Releasing fixed splits for each task, in both balanced and natural-skew versions, separates "can the model discriminate" from "does it survive real class priors."
  • I2. A sentiment lexicon mined from training data can match a full feature set while using far fewer features. This makes the lexicon a compact, inspectable stand-in for opaque models.
  • I3. The lexicon includes compound expressions, because a phrase's sentiment often differs from that of its parts.
  • I4. The authors note that star ratings and review text often disagree, as with sarcasm or positive scores attached to complaints. Such cases are a natural pool of irony or noise examples.
  • I5. A strict character-whitelist filter is a cheap, transparent way to keep a crawled corpus monolingual before any modelling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation sets could ship in a skewed and a balanced form. This would show whether a reader's gains come from real understanding or from leaning on majority classes.
  • I2 (world): polarity values for words and phrases could be compiled from a corpus into the knowledge layer. Each value would carry provenance, so evaluative states attached to entities stay explainable.
  • I3 (frames): evaluative meaning should be computed over multiword units or subtrees, not summed word by word. This fits Mova's no-bag-of-words principle and helps implicature handling.
  • I4 (frames): paired signals like rating versus text could flag candidate ironic utterances for the irony detector. They could also serve as negative controls when testing it.
  • I5 (tokens): a similar script whitelist could gate silver input text, keeping mixed-script noise away from the tokenizer and tagger.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Mahmoud Nabil, Mohamed Aly and Amir Atiya. 2014. LABR: A Large Scale Arabic Sentiment Analysis Benchmark. arXiv:1411.6718.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1411.6718 · arXiv 1411.6718

@misc{nabil2014labr,
  title = {{LABR: A Large Scale Arabic Sentiment Analysis Benchmark}},
  author = {Mahmoud Nabil, Mohamed Aly and Amir Atiya},
  year = {2014},
  doi = {10.48550/arxiv.1411.6718},
  eprint = {1411.6718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1411.6718}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Abbasi, H. Chen, and A. Salem. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):12, 2008. find
  2. M. Abdul-Mageed and M. Diab. Toward building a large-scale arabic sentiment lexicon. In Proceedings of the 6th International Global WordNet Conference, pages 18–22, 2012a. find
  3. M. Abdul-Mageed and M. T. Diab. Subjectivity and sentiment annotation of modern standard arabic newswire. In Proceedings of the 5th Linguistic Annotation Workshop, pages 110–118. Association for Computational Linguistics, 2011.
  4. M. Abdul-Mageed and M. T. Diab. Awatif: A multi-genre corpus for modern standard arabic subjectivity and sentiment analysis. In LREC, pages 3907–3914, 2012b.
  5. M. Abdul-Mageed, M. T. Diab, and M. Korayem. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 587–591. Association for Computational Linguistics, 2011.
  6. M. Abdul-Mageed, M. Diab, and S. Kübler. Samar: Subjectivity and sentiment analysis for arabic social media. Computer Speech & Language, 28(1):20–37, 2014. ↗
  7. M. Aly and A. Atiya. Labr: Large scale arabic book reviews dataset. In Meetings of the Association for Computational Linguistics (ACL), Sofia, Bulgaria, 2013. find
  8. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In NIPS, volume 4, page 2, 2007. find
  9. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive-aggressive algorithms. The Journal of Machine Learning Research, 7:551–585, 2006. find
  10. X. Ding, B. Liu, and P. S. Yu. A holistic lexicon-based approach to opinion mining. In Proceedings of the 2008 International Conference on Web Search and Data Mining, pages 231–240. find
  11. S. R. El-Beltagy and A. Ali. Open issues in the sentiment analysis of arabic social media: A case study. In Innovations in Information Technology (IIT), 2013 9th International Conference on, pages 215–220. find
  12. S. R. El-Beltagy and A. Rafea. Kp-miner: A keyphrase extraction system for english and arabic documents. Information Systems, 34(1):132–144, 2009. ↗
  13. S. R. El-Beltagy and A. Rafea. An accuracy-enhanced light stemmer for arabic text. ACM Transactions on Speech and Language Processing (TSLP), 7(2):2, 2011. ↗
  14. M. Elhawary and M. Elfeky. Mining arabic business reviews. In Data Mining Workshops (ICDMW), 2010 IEEE International Conference on, pages 1108–1113. find
  15. H. ElSahar and S. R. El-Beltagy. A fully automated approach for arabic slang lexicon extraction from microblogs. In Computational Linguistics and Intelligent Text Processing, pages 79–91. Springer, 2014. find
  16. H. ElSahar and S. R. El-Beltagy. Building large arabic multi-domain resources for sentiment analysis. In Computational Linguistics and Intelligent Text Processing, pages 23–34. Springer, 2015. find
  17. S. Gindl, A. Weichselbraun, and A. Scharl. Cross-domain contextualisation of sentiment lexicons. 2010. find
  18. D. Gräbner, M. Zanker, G. Fliedl, and M. Fuchs. Classification of customer reviews based on sentiment analysis. In Information and Communication Technologies in Tourism 2012, pages 460–470. ↗
  19. L. Jiang, M. Yu, M. Zhou, X. Liu, and T. Zhao. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 151–160. Association for Computational Linguistics, 2011. find
  20. A. Kennedy and D. Inkpen. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22(2):110–125, 2006. ↗
  21. E. Kouloumpis, T. Wilson, and J. Moore. Twitter sentiment analysis: The good the bad and the omg! ICWSM, 11:538– 541, 2011. find
  22. C. D. Manning and H. Schütze. Foundations of statistical natural language processing. MIT press, 1999. find
  23. D. Maynard, K. Bontcheva, and D. Rout. Challenges in developing opinion mining tools for social media. Proceedings of@ NLP can u tag# user_generated_content, 2012. find
  24. A. McCallum, K. Nigam, et al. A comparison of event models for naive bayes text classification. In AAAI-98 workshop on learning for text categorization, volume 752, pages 41–48. Citeseer, 1998.
  25. A. Pak and P. Paroubek. Twitter as a corpus for sentiment analysis and opinion mining. In LREC, 2010.
  26. B. Pang and L. Lee. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1– 135, 2008. ↗
  27. M. Rushdi-Saleh, M. T. Martín-Valdivia, L. A. Ureña-López, and J. M. Perea-Ortega. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology, 62(10):2045–2054, Oct. 2011. ISSN 1532-2882. doi: 10.1002/asi.21598. ↗
  28. A. Shoukry and A. Rafea. Preprocessing egyptian dialect tweets for sentiment mining. In The Fourth Workshop on Computational Approaches to Arabic Script-based Languages, page 47, 2012.
  29. M. Taboada, J. Brooke, M. Tofiloski, K. Voll, and M. Stede. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307, 2011. ↗
  30. K. Voll and M. Taboada. Not all words are created equal: Extracting semantic orientation as a function of adjective relevance. In AI 2007: Advances in Artificial Intelligence, pages 337–346. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nabil-2014-labr-large-scale-arabic-sentiment/