Elamur Research⌕
☰
home / papers / 2016 / iLab-Edinburgh at SemEval-2016 Task 7: A Hybrid Approach for Determining Sentiment Intensity of Arabic Twitter Phrases

iLab-Edinburgh at SemEval-2016 Task 7: A Hybrid Approach for Determining Sentiment Intensity of Arabic Twitter Phrases

Paper · 2016 · Eshrag Refaee and Verena Rieser

Summary

The paper describes the system that won the Arabic Twitter subtask of SemEval-2016 Task 7. The task gives a word or short phrase from Arabic tweets and asks for a real-valued sentiment intensity (SI) score between 0 and 1. Scores below 0.5 mean negative and scores above mean positive. Earlier Arabic work mostly predicted polarity only.

The authors build a two-stage pipeline:

  • Stage 1: a bagged ensemble of 10 WEKA linear regression models, trained on lemma unigrams from SI-scored lexica, produces an initial score.
  • Stage 2: hand-written rules adjust that score using a separate, polarity-only combined lexicon. Lexicon hits move the score into the matching half of the scale, and negators shift it by a fixed amount.

Preprocessing normalises Twitter artefacts, interchangeable letters and character lengthening, then lemmatises with MADAMIRA. The submitted version, trained on labMT1.0, reached a Kendall score of 0.536. Comparing training lexica shows that a smaller, human-rated lexicon beats larger automatically built ones. This points to lexicon quality, not size, as the deciding factor.

Key points

  • The official submission scored Kendall τ 0.5362 and Spearman ρ 0.680 on the 1,166-instance SemEval test set. A 200-instance gold development set was used to compare system versions.
  • Phase 1 was trained on three lexica with SI scores:
    • labMT1.0: 10k frequent words, rated by 50 MTurk annotators on a 9-point scale and rescaled to [0,1].
    • Ar-tweet: 9,785 unigrams weighted by SVM coefficients learned from 9k tweets.
    • SLSA v1.0: about 35k MSA lemmas, scored by linking to SentiWordNet.
  • Results by training lexicon (Kendall τ):
    • Ar-tweet alone collapsed to 0.0243.
    • SLSA alone reached 0.5244.
    • No combination beat labMT alone: 0.5261 with Ar-tweet, 0.5256 with SLSA, 0.5141 with all three.
    • Without the rules, labMT+SLSA was best at 0.377.
  • The rule phase uses ArabSenti (1,492 words), a Google-translated and manually filtered MPQA (2,627 entries) and a 489-word dialectal lexicon. It improved results by 14% on average.
  • Bagging 10 predictors cut LR training on 10k instances from over 48 hours to 12.6 minutes.

Interesting ideas

  • I1. A learned model gives a prior score, explicit lexicon rules then correct it, and the model's score is kept unchanged when no lexicon entry fires; each layer stays inspectable on its own.
  • I2. Negation shifts a score by a fixed step, but scores in a narrow band around neutral (0.45–0.55) are exempt, so weakly evaluative phrases are not flipped into strong opinions.
  • I3. An SVM-induced lexicon absorbed topical correlates such as a politician's name as strongly negative. The authors read this as context leakage that is likely to go stale over time, and propose filtering out weak weights.
  • I4. Systematic ablation of each knowledge source showed that adding a larger but noisier resource lowered results compared with the best clean one alone.
  • I5. Expressive lengthening is reduced by capping any run of a repeated letter at two, and interchangeable letter forms are mapped to one canonical character before lemmatisation. This raises lexicon coverage on noisy text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): evaluative frames in prag could take a statistical polarity prior and then apply logged lexical and negation rules, so every change to an attitude judgement has a stated reason.
  • I2 (frames): when computing polarity under negation or implicature, a neutral dead zone would stop "not bad"-style cases on bland predicates from producing spurious strong stances.
  • I3 (learn): self-induced lexicons or repair rules could be checked with a negative control (named entities, topical nouns) to catch features that are merely co-occurring rather than meaning-bearing.
  • I4 (world): when merging compiled knowledge (WordNet, FrameNet, Wikidata, induced lists), each source should be ablated separately and its provenance tracked, instead of assuming more coverage helps.
  • I5 (tokens): the en tokenizer could normalise lengthened words ("sooooo" → "soo") before lemma lookup, keeping the original form as an emphasis cue for later stages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heriot-Watt UniversityEdinburgh · United Kingdom15 works · 5,365 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Eshrag Refaee and Verena Rieser. 2016. iLab-Edinburgh at SemEval-2016 Task 7: A Hybrid Approach for Determining Sentiment Intensity of Arabic Twitter Phrases. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1077

@inproceedings{refaee2016ilab,
  title = {{iLab-Edinburgh at SemEval-2016 Task 7: A Hybrid Approach for Determining Sentiment Intensity of Arabic Twitter Phrases}},
  author = {Eshrag Refaee and Verena Rieser},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1077},
  url = {https://doi.org/10.18653/v1/s16-1077}
}

References

31 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2014. SANA: A large scale multi-genre, multi-dialect lexicon for Arabic subjectivity and sentiment analysis. In Proceedings of the Language Resources and Evaluation Conference (LREC).
  2. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 587–591, Stroudsburg, PA, USA. Association for Computational Linguistics.
  3. Muhammad Abdul-Mageed, Sandra Kuebler, and Mona Diab. 2012. SAMAR: A system for subjectivity and sentiment analysis of Arabic social media. In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pages 19–28. Association for Computational Linguistics.
  4. Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of Twitter data. In Proceedings of the Workshop on Languages in Social Media, pages 30–38. Association for Computational Linguistics.
  5. Shehab Ahmed, Michel Pasquier, and Ghassan Qadah. 2013. Key issues in conducting sentiment analysis on Arabic social media text. In Innovations in Information Technology (IIT), 2013 9th International Conference on, pages 72–77. IEEE. find
  6. Silvio Amir, Wang Ling, Ramón Astudillo, Bruno Martins, Mario J. Silva, and Isabel Trancoso. 2015. INESC-ID: A regression model for large scale twitter sentiment lexicon induction. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 613–618, Denver, Colorado, June. Association for Computational Linguistics. find
  7. Alexandra Balahur, Rada Mihalcea, and Andrés Montoyo. 2014. Computational approaches to subjectivity and sentiment analysis: Present and envisaged methods and applications. Computer Speech & Language, 28(1):1–6. ↗
  8. Michele Banko and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th annual meeting on association for computational linguistics, pages 26–33. Association for Computational Linguistics.
  9. Albert Bifet and Eibe Frank. 2010. Sentiment knowledge discovery in Twitter streaming data. In Discovery Science, pages 1–15. Springer. find
  10. Leo Breiman. 1996. Bagging predictors. Machine learning, 24(2):123–140.
  11. Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, Karine Megerdoomian, Matthew T. McMahon, Brian F. Tivnan, and Christopher M. Danforth. 2015. Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences, 112(8):2389– 2394. ↗
  12. Samhaa R El-Beltagy and Ahmad Ali. 2013. Open issues in the sentiment analysis of Arabic social media: A case study. In Innovations in Information Technology (IIT), 2013 9th International Conference on, pages 215–220. IEEE. find
  13. Ramy Eskander and Owen Rambow. 2015. SLSA: A sentiment lexicon for Standard Arabic. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2545–2550, Lisbon, Portugal, September. Association for Computational Linguistics.
  14. Andrea Esuli and Fabrizio Sebastiani. 2006. SentiWord-Net: A publicly available lexical resource for opinion mining. In Proceedings of LREC, volume 6, pages 417–422. find
  15. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, pages 1–12. find
  16. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene selection for cancer classification using support vector machines. Machine learning, 46(1-3):389–422. ↗
  17. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological analysis and disambiguation for dialectal Arabic. In HLT-NAACL, pages 426–432.
  18. Svetlana Kiritchenko, Saif M. Mohammad, and Mohammad Salameh. 2016. Semeval-2016 task 7: Determining sentiment intensity of English and Arabic Phrases. In Proceedings of the International Workshop on Semantic Evaluation, SemEval’16, San Diego, California, June.
  19. Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The good the bad and the OMG! In Fifth International AAAI Conference on Weblogs and Social Media (ICWSM), 11:538–541. find
  20. Fawaz HH Mahyoub, Muazzam A Siddiqui, and Mohamed Y Dahab. 2014. Building an Arabic sentiment lexicon using semi-supervised learning. Journal of King Saud University-Computer and Information Sciences, 26(4):417–424. ↗
  21. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of Modern Standard Arabic and Arabic microblogs. WASSA 2013, page 55.
  22. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
  23. Eshrag Refaee and Verena Rieser. 2014a. An Arabic twitter corpus for subjectivity and sentiment analysis. In 9th International Conference on Language Resources and Evaluation (LREC’14).
  24. Eshrag Refaee and Verena Rieser. 2014b. Evaluating distant supervision for subjectivity and sentiment analysis on Arabic twitter feeds. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).
  25. Eshrag Refaee and Verena Rieser. 2014c. Subjectivity and sentiment analysis of Arabic twitter feeds with limited resources. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools (OS-ACT). find
  26. Sara Rosenthal, Preslav Nakov, Alan Ritter, and Veselin Stoyanov. 2014. Semeval-2014 task 9: Sentiment analysis in twitter. Proc. SemEval.
  27. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307. ↗
  28. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, 63(1):163–173. ↗
  29. Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In Proceedings of the SIGDIAL 2013 Conference, page 423–432, Metz, France, August. ACL, Association for Computational Linguistics. ↗
  30. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 347–354. Association for Computational Linguistics.
  31. Ian H Witten, Eibe Frank, and Mark A Hall. 2013. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/refaee-2016-ilab-edinburgh-semeval-2016-task/