Elamur Research⌕
☰
home / papers / 2015 / SemEval-2015 Task 12: Aspect Based Sentiment Analysis

SemEval-2015 Task 12: Aspect Based Sentiment Analysis

Summary

This shared task paper describes the second SemEval edition on Aspect Based Sentiment Analysis (ABSA). The goal is to move past whole-text polarity and detect opinions about particular entities and their facets. Compared with SE-ABSA14, the organizers redefined an aspect category as a pairing of an entity type E with an attribute type A, both drawn from a domain ontology. They also linked category, opinion target expression (OTE) and polarity into a single tuple.

Annotation was done on complete reviews rather than isolated sentences, so labels could depend on the surrounding context. Data covered laptops and restaurants (Subtask 1, in-domain) and hotels (Subtask 2). The hotels domain had no training data, and only polarity had to be predicted for it.

The task drew 93 submissions from 16 teams. Polarity accuracy reached about 79% in-domain, and the best hotel score was 85.84%. Jointly extracting targets and categories was much harder, with a best F1 of 42.90%. The resulting framework and datasets supply a structured, application-oriented benchmark for building opinion summaries from reviews.

Key points

  • Data sizes:
    • Laptops: 277 training reviews (1739 sentences) and 173 test reviews (761 sentences).
    • Restaurants: 254 training reviews (1315 sentences) and 96 test reviews (685 sentences).
    • Hotels: test only, 30 reviews (266 sentences).
  • Label inventories:
    • Entity types: 22 for laptops, 6 for restaurants, 7 for hotels.
    • Attribute types: 9 for laptops, 5 for restaurants, 8 for hotels.
  • Target and joint extraction (restaurants only):
    • OTE extraction: EliXa scored best at 70.05 F1, against a baseline of 48.06.
    • Joint Slot1&2: NLANGP scored 42.90 F1, against a baseline of 34.44, by simply merging its per-slot outputs.
  • Polarity in-domain: Sentiue was best in both domains, at 79.34% on laptops and 78.69% on restaurants.
    • It used a MaxEnt classifier with n-grams, POS tags, lemmas, negation words and lexica (MPQA, Bing Liu's lexicon, AFINN).
  • Hotels (out-of-domain): lsislif scored 85.84% with a Liblinear model built for restaurants, versus a majority baseline of 71.68%. LT3 came second with 80.53% using an SVM trained on restaurant data.

Interesting ideas

  • I1. Splitting aspects into entity and attribute (E#A): Each aspect is separated into what is being judged and which property of it is judged. This yields a compact, combinable label space that could carry over to other domains, such as Ukrainian product or trade reviews.
  • I2. Labeling sentences with review-level context: Annotators assign labels per sentence but read the whole review. A sentence that sounds positive can therefore be marked negative when later text overturns it. This is a cheap way to capture discourse effects without full document modeling.
  • I3. Narrowing "neutral" and dropping "conflict": Neutral is reserved for mildly positive or mildly negative opinions, and the "conflict" label is removed because the fine-grained schema makes it rare. Clearer label semantics like these can reduce annotator disagreement.
  • I4. Isolating polarity in a hidden domain: For hotels, gold categories and targets were supplied, so only polarity was tested. This separates domain-transfer ability for sentiment from errors in the extraction steps.
  • I5. Checking class balance between splits: The organizers explained weaker restaurant results by the different share of positive labels in training and test data. Comparing class distributions across splits is a simple diagnostic worth doing on any benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

297 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 284 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s15-2082

@inproceedings{pontiki2015semeval,
  title = {{SemEval-2015 Task 12: Aspect Based Sentiment Analysis}},
  author = {Pontiki, Maria and Galanis, Dimitris and Papageorgiou, Haris and Manandhar, Suresh and Androutsopoulos, Ion},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2082},
  url = {https://doi.org/10.18653/v1/s15-2082}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bing Liu. 2006. Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data, 2006 and 2011: Springer. find
  2. Samuel Brody and Noemie Elhadad. 2010. An unsupervised aspect-sentiment model for online reviews. In Proceedings of NAACL, pages 804–812, Los Angeles, California. find our card
  3. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-- 27:27. find
  4. Gayatree Ganu, Noemie Elhadad, and Amelie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In Proceedings of WebDB, Providence, Rhode Island, USA. find
  5. Minqing Hu and Bing Liu. 2004a. Mining and summarizing customer reviews. In Proceedings of KDD, pages 168–177, Seattle, WA, USA. find
  6. Minqing Hu and Bing Liu. 2004b. Mining opinion features in customer reviews. In Proceedings of AAAI, pages 755–760, San Jose, California. find
  7. Yohan Jo, and Alice H Oh. 2011. Aspect and sentiment unification model for online review analysis. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 815-824. find
  8. Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, pages 375–384. find
  9. Julian McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning attitudes and attributes from multi-aspect reviews. In Proceedings of the 12th IEEE International Conference on Data Mining, ICDM ’12, pages 1020–1025, Brussels, Belgium. find
  10. Margaret Mitchell. 2013. Overview of the TAC2013 Knowledge Base Population Evaluation: English Sentiment Slot Filling. In Proceedings of the Text Analysis Conference (TAC), Gaithersburg, MD, USA. find
  11. John Pavlopoulos. 2014. Aspect based sentiment analysis. PhD thesis, Dept. of Informatics, Athens University of Economics and Business, Greece. find
  12. Stelios Piperidis. 2012. The META-SHARE language resources sharing infrastructure: Principles, challenges, solutions. In Proceedings of LREC-2012, pages 36–42, Istanbul, Turkey. find
  13. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. find our card
  14. Josef Ruppenhofer, Roman Klinger, Julia Maria Struß, Jonathan Sonntag, and Michael Wiegand. 2014. IGGSA Shared Tasks on German Sentiment Analysis (GESTALT). In Workshop Proceedings of the 12th Edition of the KONVENS Conference, pages 164-173. find
  15. Yohei Seki, David Kirk Evans, Lun-Wei Ku, Hsin-Hsi. Chen, Noriko Kando, and Chin-Yew Lin. 2007. Overview of opinion analysis pilot task at ntcir-6. In Proceedings of NTCIR-6 Workshop Meeting, pages 265– 278. find
  16. Yohei Seki, David Kirk Evans, Lun-Wei Ku, Le Sun, Hsin-Hsi Chen, and Noriko Kando. 2008. Overview of multilingual opinion analysis task at NTCIR-7. In Proceedings of the 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering, and Cross-… find
  17. Yohei Seki, Lun-Wei Ku, Le Sun, Hsin-Hsi Chen, and Noriko Kando. 2010. Overview of Multilingual Opinion Analysis Task at NTCIR-8: A Step Toward Cross Lingual Opinion Analysis. In Proceedings of the 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Qu… find
  18. Josef Steinberger, Tomáš Brychcín and Michal Konkol. 2014. Aspect-Level Sentiment Analysis in Czech. In Proceedings of the 5th workshop on computational approaches to subjectivity, sentiment and social media analysis, Association for Computational Linguistics, pages 24–30, Baltimore, Maryland. find
  19. Pontus Stenetorp, Sampo Pyysalo, Goran Topi´c, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. In Proceedings of EACL, pages 102–107, Avignon, France. find our card
  20. Tun Thura Thet, Jin-Cheon Na, and Christopher S. G. Khoo. 2010. Aspect-based sentiment analysis of movie reviews on discussion boards. J. Information Science, 36(6):823–848. find
  21. Lei Zhang and Bing Liu. 2014. Aspect and Entity Extraction for Opinion Mining", book chapter in Data Mining and Knowledge Discovery for Big Data: Methodologies, Challenges, and Opportunities, Springer, 2014. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pontiki-2015-semeval-2015-task-12-aspect/