Elamur Research⌕
☰
home / papers / 2024 / A Corpus for Sentence-Level Subjectivity Detection on English News Articles

A Corpus for Sentence-Level Subjectivity Detection on English News Articles

Summary

The authors study how to automatically determine whether an individual sentence is subjective. This is useful for fact-checking, sentiment analysis and bias detection. The usual approach relies on lexicons of "subjective" words, but these are tied to a specific language and domain. Instead, the authors created language-independent annotation guidelines following the prescriptive paradigm. A sentence counts as subjective if its information cannot be taken at face value: it contains the author's opinion, sarcasm, exhortations, derogatory language or rhetorical figures. Quotations from third parties and the author's description of their own emotions count as objective. The guidelines were refined over two pilot rounds and then used to build the NewsSD-ENG corpus from British news on controversial political topics. The authors also re-annotated the Italian SubjectivITA. On both corpora, they compared SVM, logistic regression, BERT and SBERT in monolingual, multilingual and cross-lingual settings. Training on data from both languages gave the best or equal results. SBERT in the zero-shot setting did as well as the monolingual variant. The guidelines can therefore be transferred to other languages without lexicons or machine translation.

Key points

  • NewsSD-ENG contains 1,049 sentences from 23 articles in eight British outlets, including The Guardian, The Economist and The Spectator. There are 638 objective and 411 subjective sentences. The train/dev/test split is 731/99/219 sentences (16/3/4 articles).
  • Seven annotators took part. Each sentence was labelled by two of them, disagreements were discussed, and a third annotator was brought in when needed. Agreement (Krippendorff's α) rose from 0.51 to 0.83 after discussion. A third annotator was needed in fewer than 10% of cases, and these 89 disputed sentences are marked separately in the corpus.
  • The second pilot tested annotation that took the article context into account. Agreement was lower: 0.38 versus 0.53 without context. The final corpus was therefore annotated on isolated sentences.
  • The lexicon-based tool TextBlob barely agrees with the human annotation (mean α −0.21). This shows that a simple dictionary-based approach is not suitable for the task.
  • Models trained on the English data together with the re-annotated SubjectivITA perform as well as or better than monolingual ones. Cross-lingual SBERT achieved results comparable to the monolingual model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2305.18034

@inproceedings{antici2024corpus,
  title = {{A Corpus for Sentence-Level Subjectivity Detection on English News Articles}},
  author = {Antici, Francesco and Ruggeri, Federico and Galassi, Andrea and Korre, Katerina and Muti, Arianna and Bardi, Alessandra and Fedotova, Alice and Barrón-Cedeño, Alberto},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/38y82q5xd7db},
  eprint = {2305.18034},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/38y82q5xd7db}
}

References

63 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, and Alexandra Uma, editors. 2022. Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022. European Language Resources Association, Marseille, France. find
  2. Desislava Aleksandrova, François Lareau, and Pierre André Ménard. 2019. Multilingual sentence-level bias detection in Wikipedia. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 42–51, Varna, Bulgaria. INCOMA Ltd. find
  3. Ida Amini, Samane Karimi, and Azadeh Shakery. 2019. Cross-lingual subjectivity detection for resource lean languages. In Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 81–90, Minneapolis, USA. Association for Computational Linguistics. find
  4. Francesco Antici, Luca Bolognini, Matteo Antonio Inajetovic, Bogdan Ivasiuk, Andrea Galassi, and Federico Ruggeri. 2021. Subjectivita: An italian corpus for subjectivity detection in newspapers. In CLEF, volume 12880 of LNCS, pages 40–52. Springer. find
  5. Carmen Banea, Rada Mihalcea, and Janyce Wiebe. 2010. Multilingual subjectivity: Are more languages better? In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 28–36, Beijing, China. Coling 2010 Organizing Committee. find
  6. Carmen Banea, Rada Mihalcea, and Janyce Wiebe. 2014. Sense-level subjectivity in a multilingual setting. Comput. Speech Lang., 28(1):7– 19. ↗
  7. Alberto Barrón-Cedeño, Firoj Alam, Tommaso Caselli, Giovanni Da San Martino, Tamer Elsayed, Andrea Galassi, Fatima Haouari, Federico Ruggeri, Julia Maria Struß, Rabindra Nath Nandi, Gullal S. Cheema, Dilshod Azizov, and Preslav Nakov. 2023a. The CLEF-2023 checkthat! lab: Checkworthiness, subjectivity, political bias, factuality, and authority. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III, volume 13982 of Lecture Notes in Computer Science, pages 506–517. Springer. find
  8. Alberto Barrón-Cedeño, Firoj Alam, Andrea Galassi, Giovanni Da San Martino, Preslav Nakov, Tamer Elsayed, Dilshod Azizov, Tommaso Caselli, Gullal S. Cheema, Fatima Haouari, Maram Hasanain, Mücahid Kutlu, Chengkai Li, Federico Ruggeri, Julia Maria Struß, and Wajdi Zaghouani. 2023b. Overview of the CLEF-2023 checkthat! lab on checkworthiness, subjectivity, political bias, factuality, and authority of news articles and their source. In CLEF, volume 14163 of Lecture Notes in Computer Science, pages 251–275. Springer. find
  9. Farah Benamara, Baptiste Chardon, Yannick Mathieu, and Vladimir Popescu. 2011. Towards context-based subjectivity analysis. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1180–1188, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find
  10. Cristina Bosco, Leonardo Allisio, Valeria Mussa, Viviana Patti, Giancarlo Francesco Ruffo, Manuela Sanguinetti, and Emilio Sulis. 2014. Detecting happiness in italian tweets: Towards an evaluation dataset for sentiment analysis in felicitta. In ES 3 LOD@LREC, pages 56–63. ELRA. find
  11. Cristina Bosco, Viviana Patti, and Andrea Bolioli. 2013. Developing corpora for sentiment analysis: The case of irony and senti-tut. IEEE Intell. Syst., 28(2):55–63. ↗
  12. Daniel Braun. 2023. I beg to differ: how disagreement is handled in the annotation of legal machine learning data sets. Artificial Intelligence and Law. ↗
  13. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In AAAI, pages 6860–6868. AAAI Press. find our card
  14. Iti Chaturvedi, Erik Cambria, Roy E. Welsch, and Francisco Herrera. 2018. Distinguishing between facts and opinions for sentiment analysis: Survey and challenges. Inf. Fusion, 44:65–77. ↗
  15. Phillipa Chong. 2019. Valuing subjectivity in journalism: Bias, emotions, and self-interest as tools in arts reporting. Journalism, 20(3):427–443. find
  16. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273–297. ↗
  17. J.S. Cramer. 2004. The early origins of the logit model. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 35(4):613–626. ↗
  18. Nilanjana Das and Santwana Sagnika. 2020. A subjectivity detection-based approach to sentiment analysis. In Machine Learning and Information Processing, pages 149–160, Singapore. Springer Singapore. ↗
  19. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguistics, 10:92–110. ↗ our card
  20. Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2022. Make the best of cross-lingual transfer: Evidence from POS tagging with over 100 languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7676–7685, Dublin, Ireland. Association for Computational Linguistics. find
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  22. Andrea Esuli and Fabrizio Sebastiani. 2006. Determining term subjectivity and term orientation for opinion mining. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 193–200, Trento, Italy. Association for Computational Linguistics. find
  23. Michael Färber, Victoria Burkard, Adam Jatowt, and Sora Lim. 2020. A multidimensional dataset based on crowdsourcing for analyzing and detecting news bias. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, page 3007–3014, New York, NY, USA. Association for Computing Machinery. find
  24. Christiane Fellbaum. 2010. WordNet, pages 231– 243. Springer Netherlands, Dordrecht. find
  25. Edward Finegan. 1995. Subjectivity and subjectivisation: an introduction. Subjectivity and subjectivisation: Linguistic perspectives, pages 1–15. ↗
  26. Andrea Galassi, Federico Ruggeri, Alberto Barrón-Cedeño, Firoj Alam, Tommaso Caselli, Mücahid Kutlu, Julia Maria Struß, Francesco Antici, Maram Hasanain, Juliane Köhler, Katerina Korre, Folkert Leistra, Arianna Muti, Melanie Siegel, Mehmet Deniz Türkmen, Michael Wiegand, and Wajdi Zaghouani. 2023. Overview of the CLEF-2023 checkthat! lab: Task 2 on subjectivity detection. In CLEF (Working Notes), volume 3497 of CEUR Workshop Proceedings, pages 236– 249. CEUR-WS.org. find
  27. Chengguang Gan and Tatsunori Mori. 2023. Sensitivity and robustness of large language models to prompt template in japanese text classification tasks. In PACLIC, pages 1–11. Association for Computational Linguistics. ↗
  28. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1161– 1166. Association for Computational Linguistics. find
  29. Lucas Graves. 2016. Deciding what’s true: The rise of political fact-checking in American journalism. Columbia University Press. find
  30. Shohreh Haddadan, Elena Cabrio, and Serena Villata. 2019. Yes, we can! mining arguments in 50 years of US presidential campaign debates. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4684–4690, Florence, Italy. Association for Computational Linguistics. find
  31. Kuan-Hao Huang, Wasi Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021. Improving zero-shot cross-lingual transfer learning via robust training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1684–1697, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  32. Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, pages 195–203. ACM. find
  33. Caio Libânio Melo Jerônimo, Leandro Balby Marinho, Cláudio E. C. Campelo, Adriano Veloso, and Allan Sales da Costa Melo. 2019. Fake news classification based on subjective language. In Proceedings of the 21st International Conference on Information Integration and Web-based Applications & Services, iiWAS 2019, Munich, Germany, December 2-4, 2019, pages 15–24. ACM. find
  34. Klaus Krippendorff. 2011. dorff’s alpha-reliability. Computing krippen-Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics. find our card
  35. Rada Mihalcea, Carmen Banea, and Janyce Wiebe. 2012. Multilingual subjectivity and sentiment analysis. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, page 4, Jeju Island, Korea. Association for Computational Linguistics. find
  36. Arianna Muti and Alberto Barrón-Cedeño. 2022. A checkpoint on multilingual misogyny identification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 454–460, Dublin, Ireland. Association for Computational Linguistics. find
  37. An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. 2017. Aggregating and predicting sequence labels from crowd annotations. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 299–309, Vancouver, Canada. Association for Computational Linguistics. find
  38. Bo Pang and Lillian Lee. 2004a. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278, Barcelona, Spain. ↗
  39. Bo Pang and Lillian Lee. 2004b. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278, Barcelona, Spain. ↗
  40. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics. find our card
  41. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  42. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics. find our card
  43. Ellen Riloff and Janyce Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 105–112. find
  44. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. ↗ our card
  45. Samir Rustamov, Elshan Mustafayev, and Mark Clements. 2013. Sentence-level subjectivity detection using neuro-fuzzy models. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 108–114, Atlanta, Georgia. Association for Computational Linguistics. find
  46. Abel Salinas and Fred Morstatter. 2024. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. CoRR, abs/2401.03729. ↗
  47. Elena Savinova and Fermin Moscoso Del Prado. 2023. Analyzing subjectivity using a transformer-based regressor trained on naïve speakers’ judgements. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 305–314, Toronto, Canada. Association for Computational Linguistics. find
  48. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. CoRR, abs/2310.11324. ↗
  49. Adam Stepinski and Vibhu O. Mittal. 2007. A fact/opinion classifier for news articles. In SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007, pages 807–808. ACM. find
  50. Milagro Teruel, Cristian Cardellino, Fernando Cardellino, Laura Alonso Alemany, and Serena Villata. 2018. Increasing argument annotation reproducibility by using inter-annotator agreement to improve guidelines. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  51. University of Adelaide. 2014. Objective language. find
  52. Francielle Vargas, Kokil Jaidka, Thiago Pardo, and Fabricio Benevenuto. 2023. Predicting sentence-level factuality of news and bias of media outlets. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023), pages 1197–1206, Varna, Bulgaria. find
  53. Kisfalvi Veronika. 2006. Subjectivity and emotions as sources of insight in an ethnographic case study: A tale of the field. M@n@gement, 9(3):117–135. ↗
  54. Lucas Lima Vieira, Caio Libânio Melo Jerônimo, Cláudio E. C. Campelo, and Leandro Balby Marinho. 2020. Analysis of the subjectivity level in fake news fragments. In WebMedia ’20: Brazillian Symposium on Multimedia and the Web, São Luís, Brazil, November 30 - December 4, 2020, pages 233–240. ACM. find
  55. Julio Villena-Román, Janine García-Morera, Miguel Ángel García Cumbreras, Eugenio Martínez-Cámara, María Teresa Martín-Valdivia, and Luis Alfonso Ureña López. 2015. Overview of TASS 2015. In Proceedings of TASS 2015: Workshop on Sentiment Analysis at SEPLN co-located with 31st SEPLN Conference (SEPLN 2015), Alicante, Spain, September 15, 2015, volume 1397 of CEUR Workshop Proceedings, pages 13–21. CEUR-WS.org. find
  56. Svitlana Volkova, Theresa Wilson, and David Yarowsky. 2013. Exploring demographic language variations to improve multilingual sentiment analysis in social media. In EMNLP, pages 1815–1827. ACL. find
  57. Karin Wahl-Jorgensen. 2013. Subjectivity and story-telling in journalism. Journalism Studies, 14(3):305–320. ↗
  58. Janyce Wiebe, Rebecca F. Bruce, and Thomas P. O’Hara. 1999a. Development and use of a gold-standard data set for subjectivity classifications. In ACL, pages 246–253. ACL. find
  59. Janyce Wiebe and Ellen Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. In Computational Linguistics and Intelligent Text Processing, pages 486–497, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  60. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Lang. Resour. Evaluation, 39(2-3):165–210. ↗
  61. Janyce M. Wiebe, Rebecca F. Bruce, and Thomas P. O’Hara. 1999b. Development and use of a gold-standard data set for subjectivity classifications. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 246–253, College Park, Maryland, USA. Association for Computational Linguistics. find
  62. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 602–608, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  63. Hong Yu and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, EMNLP ’03, page 129–136, USA. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/antici-2024-corpus-sentence-level-subjectivity-detection/