Elamur Research⌕
☰
home / papers / 2017 / OMAM at SemEval-2017 Task 4: English Sentiment Analysis with Conditional Random Fields

OMAM at SemEval-2017 Task 4: English Sentiment Analysis with Conditional Random Fields

Paper · 2017 · Chukwuyem Onyibe and Nizar Habash

Summary

The paper describes an English system entered in every subtask of SemEval-2017 Task 4, which concerns Twitter sentiment. The subtasks cover three-way polarity (A), two-way and five-way polarity toward a topic (B, C), and estimating label distributions per topic (D, E). The authors use CRF++, a Conditional Random Fields toolkit, with a small set of lexical features. Training pools all SemEval 2016 data with 4,000 extra tweets from SentiStrength. To combine sources that use different labels, they convert between label scales by folding labels together or by duplicating and relabeling examples. Topics are deliberately ignored, even in the topic-based subtasks.

Tuning happens per subtask in two stages. First they search every combination of seven features, then they grid-search the CRF++ parameters c and f. Results land mid-table overall, but the system ranked first on the micro-averaged MAE for subtask C. Unigrams and SentiStrength scores were present in every winning configuration. The work shows how far a topic-blind lexical CRF can go and gives a reproducible baseline.

Key points

  • Features: seven candidates were tested.

    • unigrams, which were always included
    • bigrams
    • tweet length (twtlen)
    • binned tweet length (twtlenbin): LOW ≤11 words, MID 12–22, HIGH ≥23
    • SentiStrength scores (senti): the negative score, the positive score, their sum, and a value mapped onto −2..+2
    • URL removal (rurl)
    • stopword removal (stpwrd)

  • Tuning: c was searched from 0.5 to 10.0 in steps of 0.5, and f from 1 to 4. Each subtask was optimized against its own official metric: AvgR, MAE^M, KL or EMD.

  • Label conversion:

    • For three-point data, ±2 were folded into ±1.
    • For two-point data, neutral tweets were either removed or duplicated as one positive and one negative copy.
    • For five-point data, each positive tweet was duplicated as +1 and +2, each negative as −1 and −2, and neutrals were duplicated to keep the classes balanced.

  • Official scores:

    SubtaskScoreRank
    AAvgR 0.59024 of 38
    BAvgR 0.77915 of 23
    CMAE^M 0.89510 of 15
    CMAE^µ 0.4751
    DKL 0.16412 of 15
    EEMD 0.3507 of 12

  • Topic handling: following TwiSE, the winner of SemEval-2016 4C, topics were dropped and one multi-class classifier was used. Quantification for D and E came from a simple aggregation script that SemEval had supplied.

Interesting ideas

  • I1. Corpora annotated on different label scales can be merged by mapping labels down, or by cloning an example once for each finer label it could carry, so no data is wasted.
  • I2. Train and predict in a finer label space, then collapse to the coarse target. If the model picks a class the target lacks (neutral), fall back to its next most probable label.
  • I3. A deliberately topic-blind system still scored mid-table on topic-based tasks. This shows how much a benchmark can be solved without the information it claims to test.
  • I4. Tune in two stages: an exhaustive feature-subset search with default hyperparameters, then a hyperparameter grid on the winner, each done against the task's own metric.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): the same duplication and relabeling could pool silver data from treebanks or annotation tiers of different granularity, without discarding the coarser sources.
  • I2 (frames): a speech-act or irony classifier could predict over a rich inventory and map down to whatever coarser set a consumer needs, keeping the ranked alternatives as an explainable fallback.
  • I3 (learn): run a "target-ablated" model as a negative control, so we can verify that frame or role components actually use the argument they are supposed to reason about.
  • I4 (tree): the perceptron parser's feature templates could be selected this way per treebank and evaluated directly on LAS, keeping the search cheap and the chosen configuration easy to explain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Chukwuyem Onyibe and Nizar Habash. 2017. OMAM at SemEval-2017 Task 4: English Sentiment Analysis with Conditional Random Fields. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2111

@inproceedings{onyibe2017omam,
  title = {{OMAM at SemEval-2017 Task 4: English Sentiment Analysis with Conditional Random Fields}},
  author = {Chukwuyem Onyibe and Nizar Habash},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2111},
  url = {https://doi.org/10.18653/v1/s17-2111}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology 61(12):2544–2558. ↗
  2. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis. Association for Computational Linguistics, pages 1–9.
  3. Xiaolong Wang, Furu Wei, Xiaohua Liu, Ming Zhou, and Ming Zhang. 2011. Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. In Proceedings of the 20th ACM international conference on Information and knowledge management. ACM, pages 1031–1040. find
  4. Georgios Balikas and Massih-Reza Amini. 2016. Twise at semeval-2016 task 4: Twitter sentiment classification. arXiv preprint arXiv:1606.04351 . ↗
  5. Changhua Yang, Kevin Hsin-Yih Lin, and Hsin-Hsi Chen. 2007. Emotion classification using web blog corpora. In Web Intelligence, IEEE/WIC/ACM International Conference on. IEEE, pages 275–278. find
  6. Ramy Baly, Gilbert Badaro, Ali Hamdi, Rawan Moukalled, Rita Aoun, Georges El-Khoury, Ahmad Al Sallab, Hazem Hajj, Nizar Habash, Khaled Shaban, and Wassim El-Hajj. 2017. Omam at semeval-2017 task 4: Evaluation of english state-of-the-art sentiment analysis models for arabic and a new topic-based model. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, pages 601– 608. http://www.aclweb.org/anthology/S17-2099. ↗
  7. Erik Cambria, Björn Schuller, Yunqing Xia, and Catherine Havasi. 2013. New avenues in opinion mining and sentiment analysis. IEEE Intelligent Systems 28(2):15–21. ↗
  8. Adam Faulkner. 2014. Automated classification of stance in student essays: An approach using stance target information and the wikipedia link-based measure. Science 376(12):86. find
  9. Taku Kudo. 2005. Crf++: Yet another crf toolkit. Software available at http://taku910.github.io/crfpp/ . ↗
  10. Preslav Nakov, Alan Ritter, Sara Rosenthal, Veselin Stoyanov, and Fabrizio Sebastiani. 2016. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation. Association for Computational Linguistics, San Diego, California, SemEval ’16.
  11. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In LREC.
  12. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval 2(1–2):1–135. ↗
  13. Ashwin Rajadesingan and Huan Liu. 2014. Identifying users with opposing opinions in twitter debates. In International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction. Springer, pages 153–160. find
  14. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, pages 501–516. http://www.aclweb.org/anthology/S17-2088.5 674 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/onyibe-2017-omam-semeval-2017-task-4/