SemEval-2014 Task 4: Aspect Based Sentiment Analysis
Summary
Most sentiment analysis work assigns a single polarity to a whole sentence or document and ignores which parts of the reviewed item the opinion is about. This shared task targeted aspect-based sentiment analysis (ABSA), where systems must find what features of a product or venue are discussed and how the reviewer feels about each. The organisers released English review sentences from two domains: restaurants (3041 training and 800 test sentences) and laptops (3045 and 800). The restaurant training data reused and extended the Ganu et al. (2009) corpus. Annotation was done in BRAT under a two-annotator scheme with adjudication. Four subtasks were defined: extracting aspect terms, classifying their polarity, detecting coarse aspect categories, and classifying category polarity. The last two were offered only for restaurants. The task drew 163 submissions from 32 teams. CRF-based extractors and SVM polarity classifiers using sentiment lexicons performed best. Laptops proved harder than restaurants. The datasets and the shared evaluation protocol became a common benchmark for later ABSA research.
Key points
- Four subtasks: SB1 aspect term extraction, SB2 aspect term polarity, SB3 aspect category detection, SB4 aspect category polarity. The polarity labels are positive, negative, neutral and conflict.
- Data totals 7686 sentences: Restaurants 3841 and Laptops 3845, each with an 800-sentence test set. Restaurant sentences additionally carry categories such as FOOD, PRICE, SERVICE and ANECDOTES/MISCELLANEOUS.
- SB1: IHS RD. reached the top laptop F1 of 74.55% with a CRF using NER, POS, parsing and semantic features. Restaurant scores were about 10 points higher than laptop scores overall.
- SB2 (26 teams per domain): DCU and NRC-Canada tied on laptops at 70.48% accuracy. On restaurants, DCU scored 80.95% and NRC-Canada 80.15%.
- SB4 (20 teams): NRC-Canada was best with 82.92% accuracy. It used an SVM whose SB2 features were extended with category-specific information.
Interesting ideas
- I1. Separate "conflict" label. When a sentence praises and criticises the same aspect, it gets its own "conflict" label instead of a forced dominant polarity. This sidesteps a judgment annotators struggle to agree on, and it suits any opinion or stance corpus where mixed verdicts are common.
- I2. Pilot round before full annotation. One annotator first labelled a 300-sentence pilot per domain, and an expert reviewed it. The observed disagreement types were then turned into extra guidelines before the rest of the data was annotated. This is a cheap way to stabilise guidelines on a new annotation project.
- I3. Explicit boundary rules. Coordinated phrases such as "noodle and rices dishes" are tagged as one maximal noun phrase. Adjectives are included only when they belong to the item's name. Misspellings are kept as written. Rules like these reduce span disagreements in any extraction dataset.
- I4. Two task views. The subtasks cover both an open-vocabulary view (free aspect terms) and a closed-inventory view (predefined categories). This lets the benchmark serve settings with and without a fixed taxonomy, a design that transfers well to new domains or languages lacking an ontology.
- I5. Domain-adapted sentiment lexicons. The top polarity systems relied heavily on lexicons, and NRC-Canada's ablation showed lexicon features mattered most. DCU hand-pruned non-sentiment words (e.g., "really") and added domain words (e.g., "mouthwatering"). Lexicon adaptation is a cheap, effective lever for lower-resource languages.
Cited by
- Aspect-Category-Opinion-Sentiment Quadruple Extraction with Implicit Aspects and Opinions
- Recurrent Attention Network on Memory for Aspect Sentiment Analysis
- It's absolutely divine! Can fine-grained sentiment analysis benefit from coreference resolution?
- An Interactive Multi-Task Learning Network for End-to-End Aspect-Based Sentiment Analysis
- A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis
- NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews
- Transformation Networks for Target-Oriented Sentiment Classification
- Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings
- Arabic aspect based sentiment analysis using bidirectional GRU based models
- Arabic aspect sentiment polarity classification using BERT
- SemEval-2016 Task 6: Detecting Stance in Tweets
- Sentiment Analysis and Language Models for Kwanyama
- SemEval-2015 Task 12: Aspect Based Sentiment Analysis
- SemEval-2016 Task 5: Aspect Based Sentiment Analysis
- Entity-Level Sentiment Analysis with Sentence Relevance Detection
- SemEval-2014 Task 9: Sentiment Analysis in Twitter
- SemEval-2017 Task 4: Sentiment Analysis in Twitter
- Learning Named Entity Tagger using Domain-Specific Dictionary
- Extending Czech Aspect-Based Sentiment Analysis with Opinion Terms: Dataset and LLM Benchmarks
- Aspect-Level Sentiment Analysis Via Convolution over Dependency Tree
- Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence
- Aspect Level Sentiment Classification with Deep Memory Network
- Attention-based LSTM for Aspect-level Sentiment Classification
- Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis
- Relational Graph Attention Network for Aspect-based Sentiment Analysis
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
- Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction
- Position-Aware Tagging for Aspect Sentiment Triplet Extraction
- A Unified Generative Framework for Aspect-based Sentiment Analysis
- Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks
- Aspect Sentiment Quad Prediction as Paraphrase Generation
- Sentiment Analysis in the Era of Large Language Models: A Reality Check
Cite
@inproceedings{pontiki2014semeval,
title = {{SemEval-2014 Task 4: Aspect Based Sentiment Analysis}},
author = {Pontiki, Maria and Galanis, Dimitris and Pavlopoulos, John and Papageorgiou, Harris and Androutsopoulos, Ion and Manandhar, Suresh},
booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
year = {2014},
doi = {10.3115/v1/s14-2004},
url = {https://doi.org/10.3115/v1/s14-2004}
}References
- Samuel Brody and Noemie Elhadad. 2010. An unsupervised aspect-sentiment model for online reviews. In Proceedings of NAACL, pages 804–812, Los Angeles, California. find our card
- Michael Gamon, Anthony Aue, Simon Corston-Oliver, and Eric K. Ringger. 2005. Pulse: Mining customer opinions from free text. In IDA, pages 121–132, Madrid, Spain. find
- Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009. Beyond the stars: Improving rating predictions using review text content. In Proceedings of WebDB, Providence, Rhode Island, USA. find
- Minqing Hu and Bing Liu. 2004a. Mining and summarizing customer reviews. In Proceedings of KDD, pages 168–177, Seattle, WA, USA. find
- Minqing Hu and Bing Liu. 2004b. Mining opinion features in customer reviews. In Proceedings of AAAI, pages 755–760, San Jose, California. find
- Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
- Chong Long, Jie Zhang, and Xiaoyan Zhu. 2010. A review selection approach for accurate feature rating estimation. In Proceedings of COLING (Posters), pages 766–774, Beijing, China. find
- Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP, pages 79–86, Philadelphia, Pennsylvania, USA. find our card
- John Pavlopoulos and Ion Androutsopoulos. 2014a. Aspect term extraction for sentiment analysis: New datasets, new evaluation measures and an improved unsupervised method. In Proceedings of LASM-EACL, pages 44–52, Gothenburg, Sweden. find
- John Pavlopoulos and Ion Androutsopoulos. 2014b. Multi-granular aspect aggregation in aspect-based sentiment analysis. In Proceedings of EACL, pages 78–87, Gothenburg, Sweden. find
- Stelios Piperidis. 2012. The META-SHARE language resources sharing infrastructure: Principles, challenges, solutions. In Proceedings of LREC-2012, pages 36–42, Istanbul, Turkey. find
- Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In Proceedings of HLT/EMNLP, pages 339–346, Vancouver, British Columbia, Canada. find our card
- Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. In Proceedings of EACL, pages 102–107, Avignon, France. find our card
- Tun Thura Thet, Jin-Cheon Na, and Christopher S. G. Khoo. 2010. Aspect-based sentiment analysis of movie reviews on discussion boards. J. Information Science, 36(6):823–848. find
- Ivan Titov and Ryan T. McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of ACL, pages 308–316, Columbus, Ohio, USA. find our card
- Peter Turney. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In Proceedings of ACL, pages 417–424, Philadelphia, Pennsylvania, USA. find our card
This page in Markdown → https://md.research.elamur.ai/free/en/papers/pontiki-2014-semeval-2014-task-4-aspect/