Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 5: Aspect Based Sentiment Analysis

SemEval-2016 Task 5: Aspect Based Sentiment Analysis

Summary

This paper reports on the third edition of SemEval's aspect-level sentiment analysis challenge, known as ABSA. ABSA means detecting which entities and attributes a review talks about and what sentiment is expressed toward each. Earlier editions (2014, 2015) were English-only and sentence-focused. SE-ABSA16 extended the SE-ABSA15 annotation framework in three ways: to seven more languages (Arabic, Chinese, Dutch, French, Russian, Spanish, Turkish), to new domains (cameras, phones, museums, telecom tweets), and to a new text-level subtask that summarises the opinions of a whole review.

The organisers released 39 manually annotated datasets with a shared evaluation procedure and baselines. Participants received 245 runs from 29 teams. Most systems beat the baselines, and runs that used external resources scored above constrained ones. Two contrasts stood out:

  • Detecting aspects was easier at the text level than at the sentence level.
  • Assigning polarity was harder at the text level, because reviews often mix sentiments about the same aspect.

The resource matters because it is a multilingual, multi-domain ABSA benchmark built under common guidelines, which enables cross-lingual work.

Key points

  • The task released 19 training and 20 test datasets covering 8 languages and 7 domains. Of these, 25 target sentence-level ABSA and 14 target text-level ABSA. There are 70,790 tuples in total: 47,654 at sentence level and 23,136 at text level (6 languages, 3 domains).
  • Subtask 1 (sentence-level) has three slots:
    • Slot1: an aspect category, written as an entity#attribute pair (E#A) from fixed inventories.
    • Slot2: the Opinion Target Expression (OTE), given by character offsets, or "null" when the target is implicit.
    • Slot3: polarity, one of positive, negative or neutral.
  • Subtask 2 (text-level) outputs {category, polarity} pairs per review and adds a "conflict" label. Subtask 3 (out-of-domain) offered only French museum data and received no submissions.
  • Teams could submit constrained runs (provided training data only) and unconstrained runs (any external resources). English drew by far the most interest, with 27 teams and 156 runs.
  • The best Slot1 and Slot2 scores were often similar. Joint {Slot1, Slot2} scores were clearly lower, and en/rest gave the highest results overall.

Interesting ideas

  • I1. Linked opinion tuples: Each opinion is stored as one record tying category, target span and polarity together. This lets the evaluation reward correct linking, not just isolated labels, and the format would suit other structured extraction tasks.
  • I2. Text-level labels built from sentence-level ones: Annotators derived review-level labels from sentence-level annotations, picking the dominant sentiment rather than simply aggregating. When no sentiment dominated, they used a "conflict" label. Making aggregation an explicit judgment like this is useful wherever document labels are derived from fragments.
  • I3. Constrained vs. unconstrained tracks: Running two parallel tracks separates the gains from better modelling from the gains due to extra lexicons or data. This makes leaderboards easier to interpret.
  • I4. Shared guidelines across languages: Applying one schema across languages and domains makes datasets directly comparable. It also opens the way to cross-lingual or language-agnostic methods, a practical template for building resources in a new language such as Ukrainian.
  • I5. Out-of-domain track: Test domains were hidden until evaluation began, which gives an honest measure of robustness to unseen domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

375 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 361 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1002

@inproceedings{pontiki2016semeval,
  title = {{SemEval-2016 Task 5: Aspect Based Sentiment Analysis}},
  author = {Pontiki, Maria and Galanis, Dimitris and Papageorgiou, Haris and Androutsopoulos, Ion and Manandhar, Suresh and AL-Smadi, Mohammad and Al-Ayyoub, Mahmoud and Zhao, Yanyan and Qin, Bing and Clercq, Orphée De and Hoste, Véronique and Apidianaki, Marianna and Tannier, Xavier and Loukachevitch, Natalia and Kotelnikov, Evgeniy and Bel, Nuria and Jiménez-Zafra, Salud María and Eryiğit, Gülşen},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1002},
  url = {https://doi.org/10.18653/v1/s16-1002}
}

References

32 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Table 7: Results for SB2.* level subtask. Future work will address the creation of datasets in more languages and domains and the enrichment of the annotation schemas with other types of SA-related information like topics, events and figures of speech (e.g., irony, metaphor). \#\# Acknowledgments The… find
  2. Erik Cambria, Björn W. Schuller, Yunqing Xia, and Catherine Havasi. 2013. New Avenues in Opinion Mining and Sentiment Analysis. IEEE Intelligent Systems, 28(2):15–21. find
  3. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM TIST, 2(3):27. find
  4. Judith Chevalier and Dina Mayzlin. 2006. The effect of word of mouth on sales: Online book reviews. J. Marketing Res, pages 345–354. find
  5. Orphée De Clercq and Véronique Hoste. 2016. Rude waiter but mouthwatering pastries! An exploratory study into Dutch Aspect-Based Sentiment Analysis. In Proceedings of the 10th International Conference on Language Resources and Evaluation. find
  6. Gülsen Eryigit, Fatih Samet Cetin, Meltem Yanık, Turkcell Global Bilgi, Tanel Temel, and Ilyas Ciçekli. 2013. TURKSENT: A Sentiment Annotation Tool for Social Media. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse. find
  7. Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009. Beyond the Stars: Improving Rating Predictions using Review Text Content. In Proceedings of WebDB. find
  8. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation, Denver, Colorado. find
  9. Salud M. Jiménez-Zafra, Giacomo Berardi, Andrea Esuli, Diego Marcheggiani, María Teresa Martín-Valdivia, and Alejandro Moreo Fernández. 2015. A Multilingual Annotated Dataset for Aspect-Oriented Opinion Mining. In Proceedings of Empirical Methods in Natural Language Processing, pages 2533–2538. find
  10. Roman Klinger and Philipp Cimiano. 2014. The USAGE Review Corpus for Fine Grained Multi Lingual Opinion Analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, Reykjavik, Iceland. find
  11. Patrik Lambert. 2015. Aspect-Level Cross-lingual Sentiment Classification with Constrained SMT. In Proceedings of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2015, Beijing, China, pages 781–787. find
  12. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. find
  13. Morgan & Claypool Publishers. Natalia Loukachevitch, Pavel Blinov, Evgeny Kotelnikov, Yulia Rubtsova, Vladimir Ivanov, and Elena Tutubalina. 2015. SentiRuEval: Testing Object-oriented Sentiment Analysis Systems in Russian. In Proceedings of International Conference Dialog. find
  14. Julian J. McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning Attitudes and Attributes from Multi-aspect Reviews. In 12th IEEE International Conference on Data Mining, ICDM 2012, Brussels, Belgium, December 10-13, 2012, pages 1020–1025. find
  15. Margaret Mitchell. 2013. Overview of the TAC2013 Knowledge Base Population Evaluation English Sentiment Slot Filling. In Proceedings of the 6th Text Analysis Conference, Gaithersburg, Maryland, USA. find
  16. Saif M Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation, San Diego, California. find our card
  17. Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 Task 2: Sentiment Analysis in Twitter. In Proceedings of the 7th International Workshop on Semantic Evaluation, Atlanta, Georgia. find our card
  18. Stelios Piperidis. 2012. The META-SHARE Language Resources Sharing Infrastructure: Principles, Challenges, Solutions. In Proceedings of the 8th International Conference on Language Resources and Evaluation. find
  19. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Ireland. find our card
  20. Maria Pontiki, Dimitrios Galanis, Harris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 Task 12: Aspect Based Sentiment Analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation, Denver, Colorado. find our card
  21. Diego Reforgiato Recupero and Erik Cambria. 2014. Eswc’14 challenge on concept-level sentiment analysis. In Semantic Web Evaluation Challenge - SemWebEval 2014 at ESWC 2014, Anissaras, Crete, Greece, pages 3–20. find
  22. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Ireland. find
  23. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 Task 10: Sentiment Analysis in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation, Denver, Colorado. find our card
  24. Josef Ruppenhofer, Roman Klinger, Julia Maria Struß, Jonathan Sonntag, and Michael Wiegand. 2014. IG-GSA Shared Tasks on German Sentiment Analysis (GESTALT). In Workshop Proceedings of the 12th Edition of the KONVENS Conference, pages 164–173. find
  25. Yohei Seki, David Kirk Evans, Lun-Wei Ku, Hsin-Hsi Chen, Noriko Kando, and Chin-Yew Lin. 2007. Overview of Opinion Analysis Pilot Task at NTCIR-6. In Proceedings of the 6th NTCIR Workshop, Tokyo, Japan. find
  26. Yohei Seki, David Kirk Evans, Lun-Wei Ku, Le Sun, Hsin-Hsi Chen, and Noriko Kando. 2008. Overview of Multilingual Opinion Analysis Task at NTCIR-7. In Proceedings of the 7th NTCIR Workshop, Tokyo, Japan. find
  27. Yohei Seki, Lun-Wei Ku, Le Sun, Hsin-Hsi Chen, and Noriko Kando. 2010. Overview of Multilingual Opinion Analysis Task at NTCIR-8: A Step Toward Cross Lingual Opinion Analysis. In Proceedings of the 8th NTCIR Workshop, Tokyo, Japan, pages 209–220. find
  28. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of Empirical Methods in Natural Language Processing, pages 1631–1642, Stroudsburg, P… find
  29. Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: A Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of the European Chapter of the Association for Computational Linguistics, pages 102–107. find our card
  30. Ezgi Yıldırım, Fatih Samet Çetin, Gülşen Eryiğit, and Tanel Temel. 2015. The impact of nlp on turkish sentiment analysis. TÜRKİYE BİLİŞİM VAKFI BİLGİSAYAR BİLİMLERİ ve MÜHENDİSLİĞİ DERGİSİ, 7(1 (Basılı 8). find
  31. Kyung Hyan Yoo and Ulrike Gretzel. 2008. What Motivates Consumers to Write Online Travel Reviews? J. of IT & Tourism, 10(4):283–295. find
  32. Yanyan Zhao, Bing Qin, and Ting Liu. 2015. Creating a Fine-Grained Corpus for Chinese Sentiment Analysis. IEEE Intelligent Systems, 30(1):36–43. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pontiki-2016-semeval-2016-task-5-aspect/