Elamur Research⌕
☰
home / papers / 2026 / The Impact of Highlighting Subjective Language on Perceived News Trustworthiness

The Impact of Highlighting Subjective Language on Perceived News Trustworthiness

Paper · 2026 · Mohammad Shokri, Vivek Sharma, Emily Klapper et al.

Summary

The paper asks whether pointing out subjective language in news makes readers trust it less. Instead of detecting misinformation automatically, it studies reader perception. The authors took true COVID-19 vaccination articles from the CoAID dataset and filtered them with a fine-tuned RoBERTa subjectivity classifier. Three annotators then labelled the remaining sentences by hand as subjective or objective. Each article was shown as plain text in two versions: unchanged, and with the subjective sentences marked in yellow. Crowdworkers on Prolific rated both versions in a within-subjects design with counterbalanced order.

Highlighting caused a small but statistically significant drop in trust, and the size of the drop varied widely across articles and readers. A linear regression on article-level linguistic and rhetorical features plus reader vaccine stance showed that the text itself explains most of this variation. The authors conclude that interventions at the moment of reading could complement fact-checking, because readers often form trust judgments before any verification reaches them.

Key points

  • Article pipeline: 100 sampled CoAID articles were cut to 26 by keeping those with a subjectivity ratio above 10%. Manual annotation reduced the set to 17, with Krippendorff's α = 0.73 and majority-vote labels.
  • Participants: 150 U.S.-born native English speakers aged 18–65 were recruited on Prolific. After attention checks and removing completions under 6 minutes, 136 participants and 1,334 paired ratings remained, about 75 per article.
  • Rating task: each participant judged 10 articles on a 4-point trust scale. Trust was defined as how fair and unbiased the author appears.
  • Effect of highlighting: mean trust fell from 2.68 (original) to 2.47 (highlighted), while the median stayed at 3.
  • Drivers of trust change: in the ∆trust regression, article features had an aggregate coefficient magnitude of 0.34, against 0.039 for the reader feature (vaccine stance). The main article features were subjectivity density and propaganda-style devices such as Loaded Language and Name Calling or Labeling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓City University of New YorkNew York · United States17 works · 2,564 citations🎓Hunter CollegeNew York · United States9 works · 1,245 citations🎓The Graduate Center, CUNYNew York · United States13 works · 207 citations🎓John Jay College of Criminal JusticeNew York · United States2 works🎓New York City College of TechnologyBrooklyn · United States2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 worksCommunication · 97 works

Cite

Cite as Mohammad Shokri, Vivek Sharma, Emily Klapper, Shweta Jain, Elena Filatova and Sarah Ita Levitan. 2026. The Impact of Highlighting Subjective Language on Perceived News Trustworthiness. The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.wassa-1.6

@inproceedings{shokri2026impact,
  title = {{The Impact of Highlighting Subjective Language on Perceived News Trustworthiness}},
  author = {Mohammad Shokri, Vivek Sharma, Emily Klapper, Shweta Jain, Elena Filatova and Sarah Ita Levitan},
  booktitle = {The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.wassa-1.6},
  url = {https://doi.org/10.18653/v1/2026.wassa-1.6}
}

References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francesco Antici, Federico Ruggeri, Andrea Galassi, Katerina Korre, Arianna Muti, Alessandra Bardi, Alice Fedotova, and Alberto Barrón-Cedeño. 2024. A corpus for sentence-level subjectivity detection on english news articles. In Proceedings of the 2024 Joint International Conference on Computational…
  2. Rudolf Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221–233. find
  3. Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning about readers’ reactions to news headlines. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find
  4. Mahmoudreza Babaei, Juhi Kulshrestha, Abhijnan Chakraborty, Elissa M Redmiles, Meeyoung Cha, and Krishna P Gummadi. 2021. Analyzing biases in perception of truth in news stories and their implications for fact checking. IEEE Transactions on Computational Social Systems, 9(3):839–850. find
  5. Diego Garusi and Sergio Splendore. 2023. Advancing a qualitative turn in news media trust research. Sociology compass, 17(4):e13075. find
  6. Matyas Bohacek, Michal Bravansky, Filip Trhlík, and Václav Moravec. 2023. Czech-ing the news: Article trustworthiness dataset for czech. In Proceedings of the 13th Workshop on Computational Approaches Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slon… find
  7. Chen Han, Wenzhen Zheng, and Xijin Tang. 2025. Debate-to-detect: Reformulating misinformation detection as a real-world debate with large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15125–15140, Suzhou, China. Association for Comp… find
  8. Roger C. Mayer, James H. Davis, and F. David Schoorman. 1995. An integrative model of organizational trust. Academy of Management Review, 20(3):709– 734. find
  9. D Harrison McKnight, Larry L Cummings, and Norman L Chervany. 1998. Initial trust formation in new organizational relationships. Academy of Management review, 23(3):473–490. find
  10. Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance detection for mis-and disinformation identification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1259–1277.
  11. Arkadiusz Modzelewski, Witold Sosnowski, Tiziano Labruna, Adam Wierzbicki, and Giovanni Da San Martino. 2025. PCoT: Persuasion-augmented chain of thought for detecting fake news and social media disinformation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic… find
  12. Hendrik Heuer and Andreas Breiter. 2018. Trust in news on social media. In Proceedings of the 10th Nordic conference on human-computer interaction, pages 137–147. find
  13. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research… find
  14. OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. Technical report. find
  15. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th international conference on computational linguistics, pages 3391–3401.
  16. Castulus Kolo, Joschka Mütterlein, and Sarah Anna Schmid. 2022. Believing journalists, ai, or fake news: The role of trust in media. In HICSS, pages 1–10. find
  17. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 231–240.
  18. Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology, 4 edition. Sage Publications. find
  19. Seung Yeop Lee and Sang Woo Lee. 2023. Normative or effective? the role of news diversity and trust in news recommendation services. International Journal of Human–Computer Interaction, 39(6):1216–1229. find
  20. Anku Rani, Dwip Dalal, Shreya Gautam, Pankaj Gupta, Vinija Jain, Aman Chadha, Amit Sheth, and Amitava Das. 2025. Sepsis: I can catch your lies–a new paradigm for deception detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Researc… find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  22. Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake news detection through multi-perspective speaker profiles. In Proceedings of the eighth international joint conference on natural language processing (volume 2: Short papers), pages 252–256. find
  23. Francisco-Javier Rodrigo-Ginés, Jorge Carrillo-de Albornoz, and Laura Plaza. 2024. A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect it. Expert Systems with Applications, 237:121641. find
  24. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. Timelms: Diachronic language models from twitter. arXiv preprint arXiv:2202.03829.
  25. Victoria L Rubin, Niall Conroy, Yimin Chen, and Sarah Cornwell. 2016. Fake news or truth? using satirical cues to detect potentially misleading news. In Proceedings of the second workshop on computational approaches to deception detection, pages 7–17. find
  26. Xiaoxiao Ma, Yuchen Zhang, Kaize Ding, Jian Yang, Jia Wu, and Hao Fan. 2024. On fake news detection with LLM enhanced semantics mining. In Proceedings of the 2024 Conference on Empirical Methods Jukka Ruohonen. 2024. A comparative study of online disinformation and offline protests. SN Social Scienc… find
  27. Elena Savinova and Fermin Moscoso Del Prado. 2023. Analyzing subjectivity using a transformer-based regressor trained on naïve speakers’ judgements. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 305–314. find
  28. William Yang Wang. 2017. " liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648.
  29. Andrea Wenzel, Anthony Nadler, Melissa Valle, and Marc Lamont Hill. 2018. Listening is not enough: Mistrust and local news in urban and suburban philly. Columbia Journalism Review. find
  30. Vivek Sharma, Mohammad Mahdi Shokri, Sarah Ita Levitan, Elena Filatova, and Shweta Jain. 2025. Analysis of propaganda in tweets from politically biased sources. arXiv preprint arXiv:2507.08169. — [arXiv:2507.08169](https://arxiv.org/abs/2507.08169) find
  31. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational linguistics, 30(3):277–308.
  32. Mohammad Shokri, Vivek Sharma, Elena Filatova, Shweta Jain, and Sarah Levitan. 2024. Subjectivity detection in English news using large language models. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 215–226, Bangkok, Thaila…
  33. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83. find
  34. Ines Zelch, Matthias Hagen, Benno Stein, and Johannes Kiesel. 2025. Reproducing the argument quality prediction of project debater. In Proceedings of the 12th Argument mining Workshop, pages 181–188. find
  35. Kilian Sprenkamp, Daniel Gordon Jones, and Liudmila Zavolokina. 2023. Large language models for propaganda detection. arXiv preprint arXiv:2310.06422. — [arXiv:2310.06422](https://arxiv.org/abs/2310.06422) find
  36. Jingrong Tong. 2024. From content to context: A qualitative case study of factors influencing audience perception of the trustworthiness of covid-19 data visualisations in uk newspaper coverage. Journalism, 25(7):1481–1499. find
  37. Zhao Tong, Yimeng Gu, Huidong Liu, Qiang Liu, Shu Wu, Haichao Shi, and Xiao-Yu Zhang. 2025. Generate first, then sample: Enhancing fake news detection with llm-augmented reinforced sampling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  38. Pramukh Nanjundaswamy Vasist, Debashis Chatterjee, and Satish Krishnan. 2024. The polarizing impact of political disinformation and hate speech: A cross-country configural narrative. Information Systems Frontiers, 26(2):663–688. find
  39. Aswathy Velutharambath, Amelie Wührl, and Roman Klinger. 2024. How entangled is factuality and deception in german? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9538–9554. find
  40. Nitin Verma, Kenneth R Fleischmann, and Kolina S Koltai. 2018. Demographic factors and trust in different news sources. Proceedings of the Association for Information Science and Technology, 55(1):524–533. find
  41. Svitlana Volkova and Jin Yea Jang. 2018. Misleading or falsification: Inferring deceptive strategies and types in online news and social media. In Companion Proceedings of the The Web Conference 2018, pages 575–583. find
  42. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. science, 359(6380):1146–1151. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shokri-2026-impact-highlighting-subjective-language-perceived/