Elamur Research⌕
☰
home / papers / 2024 / Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English

Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English

Paper · 2024 · Dipankar Srirag, Jordan Painter, Aditya Joshi et al.

Summary

Most sentiment benchmarks are built around Standard American English, so little is known about how well models handle other national varieties. The authors describe their work toward a sentiment classification benchmark for Australian (en-AU), Indian (en-IN) and British (en-UK) English, with a New York City subset serving as an en-US reference. They gathered Google Places reviews through location-based filtering and used star ratings as automatic labels. They then tested how dataset construction choices affect difficulty. These choices were which ratings count as positive or negative, how long the reviews are, and how densely they contain polarity words.

They fine-tuned BERT-base, DistilBERT-base and RoBERTa-base on each subset and observed large swings in scores. Pairing adjacent-ish ratings (2 vs 4) made the task much harder than pairing extreme ratings (1 vs 5). Reviews rich in sentiment words were also harder. The drops were steepest for en-IN, while the inner-circle varieties scored higher overall.

The main contribution is practical guidance on sampling and label design, which matters for anyone building robustness tests across dialects.

Key points

  • Data collection.
    • Reviews come from the Google Places API, with cities chosen by population thresholds: 10,000 in Australia, 100,000 in India and 50,000 in the UK.
    • 29,512 NYC reviews serve as the en-US baseline.
    • Non-English text was removed using fastText language probabilities, and places tagged as tourist attractions were dropped.
  • Locale validation. Two native-speaker authors (British and Indian English) relabelled a sample by locale.
    • Their agreement with the locale tags for en-IN was 0.64 and 0.88.
    • Cohen's kappa between the two annotators was only 0.191, and en-AU and en-UK overlapped heavily.
  • Label and sampling design.
    • Two label configurations were compared: SIMPLE (1★ vs 5★) and HARD (2★ vs 4★).
    • Nested quartile subsets were built by review length (LEN-75/50/25).
    • Further nested subsets were built by the share of polarity words, counted with spaCy's TextBlob extension (SENT-75/50/25).
  • Training setup. Each subset was split 70/15/15, and models were trained and tested within the same sample. Results are reported as macro P/R/F1.
  • Main results.
    • SIMPLE en-AU reached the best average F1 (97.2), and en-US had the lowest (93.5).
    • Moving to HARD cost 13.7 F1 points on average and 19.4 points for en-IN.
    • SIMPLE-SENT-75 en-AU reached 97.5, while HARD-SENT-25 fell to 64.8.
    • Longer reviews generally helped the models.

Interesting ideas

  • I1. Control task difficulty by choosing how far apart the source labels sit: extreme ratings give an easy split and neighbouring ratings a hard one, both from the same data.
  • I2. Make the test harder by keeping texts dense in polarity words of either sign, so a model cannot win by spotting a single cue word.
  • I3. Derive variety labels from posting location, then audit them with native speakers and inter-annotator kappa, which exposed that inner-circle varieties are hard to tell apart.
  • I4. Build nested quartile subsets (top 75/50/25%) along one property, such as length, to get a simple difficulty curve rather than a single score.
  • I5. Use metadata that users supply anyway (star ratings) as free weak labels, while noting their noise and planning manual annotation later.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): We could build paired easy/hard evaluation sets from one graded source, for example confidence-scored silver parses, to see whether gains hold on the borderline cases.
  • I2 (frames): Irony and implicature test sets in prag could deliberately favour sentences mixing positive and negative lexis, checking that frame-level reasoning, not word counts, decides the reading.
  • I3 (tree): When testing the parsers on Indian or Australian English, we should validate any "dialect" tags with a small human audit before trusting variety-specific error analyses.
  • I4 (learn): Evaluation reports could slice results by sentence length or clause count quartiles, revealing whether repair rules or teacher tiers help mainly on long inputs.
  • I5 (learn): Naturally occurring signals, such as answer keys in school tests, can seed silver data, with a small hand-checked slice kept as a quality control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia34 works · 1,509 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Dipankar Srirag, Jordan Painter, Aditya Joshi and Diptesh Kanojia. 2024. Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English. arXiv:2410.11216.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.11216 · arXiv 2410.11216

@misc{srirag2024experiences,
  title = {{Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English}},
  author = {Dipankar Srirag, Jordan Painter, Aditya Joshi and Diptesh Kanojia},
  year = {2024},
  doi = {10.48550/arxiv.2410.11216},
  eprint = {2410.11216},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.11216}
}

References

32 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Abdulmohsen Al-Thubaity, Qubayl Alqahtani, and Abdulaziz Aljandal. 2018. Sentiment lexicon for sentiment analysis of saudi dialect tweets. Procedia computer science, 142:301–307. find
  3. Verena Blaschke, Christoph Purschke, Hinrich Schuetze, and Barbara Plank. 2024. What do dialect speakers want? a survey of attitudes towards language technology for German dialects. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 823–841, Bangkok, Thailand. Association for Computational Linguistics. find
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗
  5. Asian English speakers. In Findings of the Association for Computational Linguistics ACL 2024, pages 4067–4081, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. find
  6. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2024. Natural language processing for dialects of a language: A survey. Preprint, arXiv:2401.05632. ↗
  7. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics. ↗
  8. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51–57. find
  9. ElMehdi Boujou, Hamza Chataoui, Abdellah El Mekki, Saad Benjelloun, Ikram Chairi, and Ismail Berrada. 2021. An open access nlp dataset for arabic dialects : Data collection, labeling, and model construction. Preprint, arXiv:2102.11000. ↗
  10. Jiacheng Li, Jingbo Shang, and Julian McAuley. 2022. UCTopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6159–6169, Dublin, Ireland. Association for Computational Linguistics. find
  11. Steven Coats. 2022. The corpus of Australian and New Zealand spoken English: A new resource of naturalistic speech transcripts. In Australasian Language Technology Association Workshop, pages 1–5.
  12. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  13. Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315–2338, Online. Association for Computational Linguistics. ↗
  14. Isa Maks and Piek Vossen. 2013. Sentiment analysis of reviews: Should we analyze writer intentions or reader perceptions? In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 415–419, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BUL-GARIA. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  16. Salima Mdhaffar, Fethi Bougares, Yannick Esteve, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of tunisian dialects: Linguistic ressources and experiments. In Third Arabic natural language processing workshop (WANLP), pages 55–61.
  17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  18. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  19. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa’id Ahmad, Meriem Beloucif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. 2023. Afrisenti: A twitter sentiment analysis benchmark for african languages. Preprint, arXiv:2302.08956. ↗
  20. Faye Holt, William Held, and Diyi Yang. 2024. Perceptions of language technology failures from South Ahmed Oussous, Fatima-Zahra Benjelloun, Ayoub Ait Lahcen, and Samir Belfkih. 2020. Asa: A framework Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. Dialectbench: A nlp benchmark for dialects, varieties, and closely-related languages. arXiv preprint arXiv:2403.11009. for arabic sentiment analysis. Journal of Information Science, 46(4):544–559. ↗
  21. An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. 2023. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 2251–2255, New York, NY, USA. Association for Computing Machinery. find
  22. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics. ↗
  23. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.
  24. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108. ↗
  25. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-value: A framework for cross-dialectal english nlp. In The 61st Annual Meeting Of The Association For Computational Linguistics.
  26. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics. ↗
  27. Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. 2023. Dialect-robust evaluation of generated text. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6010–6028, Toronto, Canada. Association for Computational Linguistics. ↗
  28. Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. 2015. Sentiment embeddings with applications to sentiment analysis. IEEE transactions on knowledge and data Engineering, 28(2):496–509. ↗
  29. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537. ↗
  30. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461. ↗
  31. Suge Wang, Deyu Li, Lidong Zhao, and Jiahao Zhang. 2013. Sample cutting method for imbalanced text sentiment classification based on brc. Knowledge-Based Systems, 37:451–461. ↗
  32. Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4569–4586, Seattle, United States. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srirag-2024-experiences-creating-benchmark-sentiment-classification/