Elamur Research⌕
☰
home / papers / 2023 / SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning

SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning

Paper · 2023 · Kiana Kheiri and Hamid Karimi

Summary

Tweet sentiment classification remains hard. Posts are short and noisy, and they carry sarcasm, irony, emojis and abbreviations that classical classifiers often misread. Kheiri and Karimi (Utah State University) test whether OpenAI's GPT family can do better on the Twitter benchmark of SemEval 2017 Task 4. They compare three ways of using GPT:

  • prompting GPT-3.5 Turbo with task instructions;
  • fine-tuning GPT models on the labelled tweets;
  • training a separate classifier on GPT embeddings.

The study asks three questions: how GPT compares with existing machine-learning systems, how individual GPT products differ, and whether GPT handles subtle cues such as emotion, emojis and mixed sentiment. The authors report that the GPT-based approaches beat the previous best results on this benchmark by over 22% in F1-score. A qualitative analysis suggests these models also cope better with context and sarcasm. The work is early evidence that general-purpose LLMs can outperform task-specific sentiment models, although it rests on a single English Twitter dataset.

Key points

  • Benchmark: SemEval 2017 Task 4 (Twitter sentiment analysis). The prior systems compared include SemEval-2017 entries such as DataStories, BB_twtr, TSA-INF, XJSA and ELiRF-UPV, plus BERTweet, judging by the references.
  • Three GPT strategies were tested: prompt engineering with GPT-3.5 Turbo, fine-tuned GPT models, and classification on GPT embeddings.
  • The headline result is an F1-score gain of more than 22% over the previous state of the art. Per-model scores were not in the provided excerpt.
  • The authors include an error analysis of linguistic nuances (context, sarcasm, emojis, mixed sentiment), contrasting the best GPT model with earlier state-of-the-art systems.
  • Code is released at github.com/DSAatUSU/SentimentGPT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utah State UniversityLogan · United States2 works · 56 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 works

Cite

Cite as Kiana Kheiri and Hamid Karimi. 2023. SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning. arXiv:2307.10234.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.10234 · arXiv 2307.10234

@misc{kheiri2023sentimentgpt,
  title = {{SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning}},
  author = {Kiana Kheiri and Hamid Karimi},
  year = {2023},
  doi = {10.48550/arxiv.2307.10234},
  eprint = {2307.10234},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.10234}
}

References

97 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Karimi, C. VanDam, L. Ye, and J. Tang, “End-to-end compromised account detection,” in 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 2018, pp. 314–321. find
  2. C. VanDam, P.-N. Tan, J. Tang, and H. Karimi, “Cadet: A multi-view learning framework for compromised account detection on twitter,” in 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 2018, pp. 471–478. find
  3. H. Karimi, P. Roy, S. Saba-Sadiya, and J. Tang, “Multi-source multi-class fake news detection,” in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 1546–1557. find
  4. A. Brookhouse, T. Derr, H. Karimi, H. R. Bernard, and J. Tang, “Road to the white house: Analyzing the relations between mainstream and social media during the us presidential primaries,” in Proceedings of the 32nd ACM Conference on Hypertext and Social Media, 2021, pp. 57–66. find
  5. H. Karimi and J. Tang, “Learning hierarchical discourse-level structure for fake news detection,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019, pp. 3432–3442. find
  6. H. Karimi, T. Derr, A. Brookhouse, and J. Tang, “Multi-factor congressional vote prediction,” in Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 2019, pp. 266–273. find
  7. A. Patino, D. A. Pitta, and R. Quinones, “Social media’s emerging importance in market research,” Journal of consumer marketing, vol. 29, no. 3, pp. 233–237, 2012. find
  8. H. Karimi, T. Derr, K. T. Torphy, K. A. Frank, and J. Tang, “A roadmap for incorporating online social media in educational research,” Teachers College Record, vol. 121, no. 14, pp. 1–24, 2019. find
  9. ——, “Towards improving sample representativeness of teachers on online social media: A case study on pinterest,” in Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21. Springer International Publishing, 2020, pp.… find
  10. S. Solanki, K. Kheiri, M. A. Tsugawa, and H. Karimi, “Leveraging social media analytics in engineering education research,” in American Society for Engineering Education (ASEE) Annual Conference and Exposition Mining, 2023. find
  11. H. Karimi, K. T. Torphy, T. Derr, K. A. Frank, and J. Tang, “Understanding and promoting teacher connections in online social media: A case study on pinterest,” in TALE 2020, 2020. find
  12. H. Karimi, K. T. Torphy, and K. A. Frank, “An analysis of diffusion of teacher-curated resources on pinterest,” in Proceedings of The 16th International Conference on Educational Data Mining (EDM), 2023. find
  13. K. T. Knake, H. Karimi, S. Hu, K. A. Frank, and J. Tang, “Educational research in the twenty-first century: Leveraging big data to explore teachers’ professional behavior and educational resources accessed within pinterest,” The Elementary School Journal, vol. 122, no. 1, pp. 86–111, 2021. find
  14. H. Karimi, J. Tang, X. Weiss, and J. Huang, “Automatic identification of teachers in social media using positive unlabeled learning,” in 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021, pp. 643–652. find
  15. H. Karimi, K. T. Knake, and K. A. Frank, “Teachers in social media: A gender-aware behavior analysis,” in 2022 IEEE International Conference on Big Data (Big Data). IEEE, 2022, pp. 1842–1849. find
  16. C. Reuter, S. Stieglitz, and M. Imran, “Social media in conflicts and crises,” Behaviour & Information Technology, vol. 39, no. 3, pp. 241– 251, 2020. find
  17. L. Braghieri, R. Levy, and A. Makarin, “Social media and mental health,” American Economic Review, vol. 112, no. 11, pp. 3660–3693, 2022. find
  18. Y. Lin and S. Geertman, “Can social media play a role in urban planning? a literature review,” Computational Urban Planning and Management for Smart Cities 16, pp. 69–84, 2019. find
  19. F. Z. Xing, F. Pallucchini, and E. Cambria, “Cognitive-inspired domain adaptation of sentiment lexicons,” Information Processing & Management, vol. 56, no. 3, pp. 554–564, 2019. find
  20. N. Tsapatsoulis, “Opinion mining from social media short texts: Does collective intelligence beat deep learning?” Frontiers in Robotics and AI, 2019. find
  21. A. Balahur, “Sentiment analysis system adaptation for multilingual processing: The case of tweets,” Inf. Process. Manag., 2015. find
  22. H. Wang, “Generative models for sentiment analysis and opinion mining,” 2017. find
  23. G. Biondi, “Web-based similarity for emotion recognition in web objects,” 2016 IEEE/ACM 9th International Conference on Utility and Cloud Computing (UCC), 2016. find
  24. M. Axhiu, “The usage of sentiment analysis for hearing the voice of the customer and improving businesses,” Journal of Educational and Social Research, 2014. find
  25. E. Xia, “Tweet sentiment analysis of the 2020 u.s. presidential election,” Companion Proceedings of the Web Conference 2021, 2021. find
  26. E. M. G. Younis, “Sentiment analysis and text mining for social media microblogs using open source tools: An empirical study,” International Journal of Computer Applications, 2015. find
  27. R. Bose, “Sentiment analysis on online product reviews,” Information and Communication Technology for Sustainable Development, 2019. find
  28. I. Aattouchi, “Sentiment analysis of health care: Review,” E3S Web of Conferences, 2021. find
  29. B. Zhang, “Home-based digital assessments with applied sentiment & emotion ai capture improved quality-of-life in asthma patients,” 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2021. find
  30. S. Liu, “Public attitudes toward covid-19 vaccines on english-language twitter: A sentiment analysis,” Vaccine, 2021. find
  31. D. G. S. Bhawna Tiwari, “Sentiment analysis extensively to harness every aspect of primary school education,” 2021. find
  32. C. Sharma, “Stress analysis for students in online classes,” 2021 Grace Hopper Celebration India (GHCI), 2021. find
  33. P. Sudhir, “Comparative study of various approaches, applications and classifiers for sentiment analysis,” Global Transitions Proceedings, 2021. find
  34. M. Wankhade, A. C. S. Rao, and C. Kulkarni, “A survey on sentiment analysis methods, applications, and challenges,” Artificial Intelligence Review, vol. 55, no. 7, pp. 5731–5780, 2022. find
  35. E. Omuya, “Sentiment analysis on social media using machine learning approach,” 2021. find
  36. S. Witteveen and M. Andrews, “Paraphrasing with large language models,” arXiv preprint arXiv:1911.09661, 2019. find
  37. B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” arXiv preprint arXiv:2301.07069, 2023.
  38. X. Liu, H. Cheng, P. He, W. Chen, Y. Wang, H. Poon, and J. Gao, “Adversarial training for large neural language models,” arXiv preprint arXiv:2004.08994, 2020. find
  39. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, “A watermark for large language models,” arXiv preprint arXiv:2301.10226, 2023.
  40. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  41. S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu, “Plug and play language models: A simple approach to controlled text generation,” arXiv preprint arXiv:1912.02164, 2019.
  42. L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines, vol. 30, pp. 681–694, 2020. find
  43. M. L. Siddiq, J. Santos, R. H. Tanvir, N. Ulfat, F. A. Rifat, and V. C. Lopes, “Exploring the effectiveness of large language models in generating unit tests,” arXiv preprint arXiv:2305.00418, 2023. — [arXiv:2305.00418](https://arxiv.org/abs/2305.00418) find
  44. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” arXiv preprint arXiv:2201.11903, 2022.
  45. Y. Fu, H. Peng, A. Sabharwal, P. Clark, and T. Khot, “Complexity-based prompting for multi-step reasoning,” arXiv preprint arXiv:2210.00720, 2022.
  46. S. Rosenthal, N. Farra, and P. Nakov, “Semeval-2017 task 4: Sentiment analysis in twitter,” arXiv preprint arXiv:1912.00741, 2019.
  47. M. Syamala and N. Nalini, “A deep analysis on aspect based sentiment text classification approaches,” International Journal of Advanced Trends in Computer Science and Engineering, vol. 8, no. 5, pp. 1795– 1801, 2019. find
  48. J. Hartmann, J. Huppertz, C. Schamp, and M. Heitmann, “Comparing automated text classification methods,” International Journal of Research in Marketing, vol. 36, no. 1, pp. 20–38, 2019. find
  49. O. Netzer, A. Lemaire, and M. Herzenstein, “When words sweat: Identifying signals for loan default in the text of loan applications,” Journal of Marketing Research, vol. 56, no. 6, pp. 960–980, 2019. find
  50. A. Esuli and F. Sebastiani, “Sentiwordnet: a high-coverage lexical resource for opinion mining,” Evaluation, vol. 17, no. 1, p. 26, 2007. find
  51. A. Pak, P. Paroubek et al., “Twitter as a corpus for sentiment analysis and opinion mining.” in LREc, vol. 10, no. 2010, 2010, pp. 1320–1326.
  52. M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas, “Sentiment strength detection in short informal text,” Journal of the American society for information science and technology, vol. 61, no. 12, pp. 2544–2558, 2010. find
  53. T. Wilson, J. Wiebe, and P. Hoffmann, “Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis,” Computational linguistics, vol. 35, no. 3, pp. 399–433, 2009.
  54. K. Kenyon-Dean, “Sentiment analysis: It’s complicated!” 2018. find
  55. M. Birjali, M. Kasri, and A. Beni-Hssane, “A comprehensive survey on sentiment analysis: Approaches, challenges and trends,” Knowledge-Based Systems, vol. 226, p. 107134, 2021. find
  56. T. Mullen and N. Collier, “Sentiment analysis using support vector machines with diverse information sources,” in Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp. 412–418. find
  57. P. D. Turney, “Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews,” arXiv preprint cs/0212032, 2002.
  58. B. Pang and L. Lee, “A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts,” arXiv preprint cs/0409058, 2004.
  59. B. Guthier, “Language-independent data set annotation for machine learning-based sentiment analysis,” 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017. find
  60. E. Kouloumpis, T. Wilson, and J. Moore, “Twitter sentiment analysis: The good the bad and the omg!” in Proceedings of the international AAAI conference on web and social media, vol. 5, no. 1, 2011, pp. 538–541. find
  61. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  62. R. Dale, “Gpt-3: What’s it good for?” Natural Language Engineering, vol. 27, no. 1, pp. 113–118, 2021. find
  63. M. Pota, M. Ventura, R. Catelli, and M. Esposito, “An effective bert-based pipeline for twitter sentiment analysis: A case study in italian,” Sensors, vol. 21, no. 1, p. 133, 2020. find
  64. A. Chiorrini, C. Diamantini, A. Mircoli, and D. Potena, “Emotion and sentiment analysis of tweets using bert.” in EDBT/ICDT Workshops, vol. 3, 2021. find
  65. S. Praveen and V. Vajrobol, “Understanding the perceptions of healthcare researchers regarding chatgpt: a study based on bidirectional encoder representation from transformers (bert) sentiment analysis and topic modeling,” Annals of Biomedical Engineering, pp. 1–3, 2023. find
  66. A. Bello, S.-C. Ng, and M.-F. Leung, “A bert framework to sentiment analysis of tweets,” Sensors, vol. 23, no. 1, p. 506, 2023. find
  67. C. Shaib, M. L. Li, S. Joseph, I. J. Marshall, J. J. Li, and B. C. Wallace, “Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success),” arXiv preprint arXiv:2305.06299, 2023. find
  68. S. Cohen, D. Presil, O. Katz, O. Arbili, S. Messica, and L. Rokach, “Enhancing social network hate detection using back translation and gpt-3 augmentations during training and test-time,” Information Fusion, p. 101887, 2023. find
  69. N. Kim, R. Patel, A. Poliak, A. Wang, P. Xia, R. T. McCoy, I. Tenney, A. Ross, T. Linzen, B. Van Durme et al., “Probing what different nlp tasks teach machines about function word comprehension,” arXiv preprint arXiv:1904.11544, 2019.
  70. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  71. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  72. N. Kolt, “Predicting consumer contracts,” Berkeley Tech. LJ, vol. 37, p. 71, 2022. find
  73. K.-L. Chiu, A. Collins, and R. Alexander, “Detecting hate speech with gpt-3,” arXiv preprint arXiv:2103.12407, 2021. — [arXiv:2103.12407](https://arxiv.org/abs/2103.12407) find
  74. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  75. Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” arXiv preprint arXiv:1908.08345, 2019.
  76. V. Kieuvongngam, B. Tan, and Y. Niu, “Automatic text summarization of covid-19 medical research articles using bert and gpt-2,” arXiv preprint arXiv:2006.01997, 2020. — [arXiv:2006.01997](https://arxiv.org/abs/2006.01997) find
  77. Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan, “Dialogpt: Large-scale generative pre-training for conversational response generation,” arXiv preprint arXiv:1911.00536, 2019.
  78. R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022. find
  79. https://platform.openai.com/docs/guides/gpt/chat-completions-api. ↗
  80. “Openai models,” https://platform.openai.com/docs/models/gpt-3-5. ↗
  81. V. Liu and L. B. Chilton, “Design guidelines for prompt engineering text-to-image generative models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–23. find
  82. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023.
  83. D. M. E.-D. M. Hussein, “A survey on sentiment analysis challenges,” Journal of King Saud University-Engineering Sciences, vol. 30, no. 4, pp. 330–338, 2018. find
  84. L. Jia, “The effect of negation on sentiment analysis and retrieval effectiveness,” Proceedings of the 18th ACM conference on Information and knowledge management, 2009. find
  85. S. Bharti, “Sarcasm detection in textual data: A supervised approach,” 2019. find
  86. https://platform.openai.com/docs/guides/embeddings/ what-are-embeddings. ↗
  87. U. Gain and V. Hotti, “Low-code automl-augmented data pipeline–a review and experiments,” in Journal of Physics: Conference Series, vol. 1828, no. 1. IOP Publishing, 2021, p. 012015. find
  88. C. Baziotis, N. Pelekis, and C. Doulkeridis, “DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Vancouver, Canada: Association for Computational…
  89. A. A. Deshmane and J. Friedrichs, “TSA-INF at SemEval-2017 task 4: An ensemble of deep learning architectures including lexicon features for Twitter sentiment analysis,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Vancouver, Canada: Association for Comput… find
  90. M. Cliche, “Bb twtr at semeval-2017 task 4: Twitter sentiment analysis with cnns and lstms,” arXiv preprint arXiv:1704.06125, 2017. find
  91. Y. Hao, Y. Lan, Y. Li, and C. Li, “XJSA at SemEval-2017 task 4: A deep system for sentiment classification in Twitter,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 728–731. [Onli… find
  92. D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, and B. Qin, “Learning sentiment-specific word embedding for twitter sentiment classification,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2014, pp. 1555– 1565.
  93. J.-Á. González, F. Pla, and L.-F. Hurtado, “ELiRF-UPV at SemEval-2017 task 4: Sentiment analysis using deep learning,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 723–727. [Onl… find
  94. D. Q. Nguyen, T. Vu, and A. T. Nguyen, “Bertweet: A pre-trained language model for english tweets,” 2020.
  95. G. E. Hinton and S. Roweis, “Stochastic neighbor embedding,” Advances in neural information processing systems, vol. 15, 2002. find
  96. H. Liu, W. Jin, H. Karimi, Z. Liu, and J. Tang, “The authors matter: Understanding and mitigating implicit bias in deep text classification,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 74–85. find
  97. https://platform.openai.com/docs/guides/embeddings/limitations-risks. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kheiri-2023-sentimentgpt-exploiting-gpt-advanced-sentiment/