Elamur Research⌕
☰
home / papers / 2025 / Using Sentiment Analysis to Investigate Peer Feedback by Native and Non-Native English Speakers

Using Sentiment Analysis to Investigate Peer Feedback by Native and Non-Native English Speakers

Paper · 2025 · Brittney Exline; Melanie Duffin; Brittany Harbison; Chrissa da Gomez; and David Joyner

Summary

The study asks whether being a native or non-native English speaker changes how students experience peer review in English-language online graduate CS courses. The authors drew on archival data from three classes of a large, low-cost online master's program (Fall 2018–Summer 2022). They combined LMS survey answers (perceived impact of peer feedback, sex, age range, English fluency) with comments from the program's in-house peer review tool. After removing students without the needed surveys, 7,101 remained. A random 500 of them were kept, together with about 41,000 comments they wrote or received. Each comment was scored for sentiment with the Twitter-roBERTa base model, and the scores were related to language status, controlling for sex and age.

Native speakers rated peer feedback less favorably. Non-native speakers wrote more positive reviews but received less positive ones. Significant interactions with sex and age suggest the effect is modest and tangled with other factors. The authors argue that designers of scalable peer feedback activities should take these differences into account.

Key points

  • Three research questions: perceived impact of feedback (RQ1), sentiment of feedback authored (RQ2), and sentiment of feedback received (RQ3), each compared between native and non-native speakers.
  • Sample: 500 students randomly drawn from 7,101, with about 41,000 comments. Sampling avoided the cost of scoring the full pool of over 500,000 comments.
  • Fluency had four survey options, but over 98% chose "Native speaker" or "Fully fluent non-native speaker", so only those two groups were analyzed. Gender was likewise reduced to Male/Female.
  • Checked against human reviewers, the sentiment tool reached roughly 70% accuracy. Its most frequent error was labelling negative sentences as neutral, which likely pushes scores upward.
  • Context:
    • The program costs under 7,000 USD and is about 32% non-U.S. residents.
    • Peer review is non-anonymous and does not affect grades.
    • Students' grades instead depend on how well they write their reviews.

Interesting ideas

  • I1. Before trusting an off-the-shelf model trained on another genre (tweets) on new text, the authors had humans check a sample, then reported both the accuracy and the dominant error direction. Knowing the bias direction tells you how conclusions may be skewed.
  • I2. The paper notes that averaging sentence-level polarity gives the same score to an all-neutral text and to an evenly mixed positive/negative text. Keeping the distribution, not just the mean, avoids this.
  • I3. Sentiment is split by direction: what a person writes versus what that person receives. This reveals asymmetries that a single per-person score would hide.
  • I4. The data were restricted to a pre-2023 window, so the texts were almost certainly human-written, without AI assistance. This makes provenance an explicit design choice.
  • I5. The cited literature reports that non-native reviewers use more hedges and cognitive verbs (think, feel) when softening criticism. These are surface cues for how criticism is negotiated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when an LLM teacher or witness parser is applied to a new domain, log a per-label confusion profile on a hand-checked slice, not just one accuracy number.
  • I2 (frames): when sentence-level evaluative speech acts are rolled up into a text-level stance, keep counts per polarity so that mixed praise-and-criticism is distinguishable from flat neutrality.
  • I3 (world): store evaluations as directed edges (evaluator → target with polarity), so the answer layer can explain who praised or criticized whom, and whether it was reciprocated.
  • I4 (learn): tag corpora by date or source to separate human-authored from likely LLM-influenced text, and use the human-only portion as a cleaner control set.
  • I5 (frames): treat hedges and cognitive-verb framing as signals of mitigated criticism in implicature detection, rather than reading such sentences as neutral.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksComputer Science Applications · 131 worksEducation · 212 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSentiment & emotion · 1,035 works

Cite

Cite as Brittney Exline; Melanie Duffin; Brittany Harbison; Chrissa da Gomez; and David Joyner. 2025. Using Sentiment Analysis to Investigate Peer Feedback by Native and Non-Native English Speakers. arXiv:2507.22924.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.22924 · arXiv 2507.22924

@misc{gomez2025using,
  title = {{Using Sentiment Analysis to Investigate Peer Feedback by Native and Non-Native English Speakers}},
  author = {Brittney Exline; Melanie Duffin; Brittany Harbison; Chrissa {da Gomez;} and David Joyner},
  year = {2025},
  doi = {10.48550/arxiv.2507.22924},
  eprint = {2507.22924},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.22924}
}

References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Danial Baharudin and Abu Bakar Razali. 2021. A Review of Literature on the Potentials and Problems of Face-to-Face and Online Peer Feedback and the Patterns of Interaction among ESL/EFL Learners in a Peer Feedback Environment. 3L The Southeast Asian Journal of English Language Studies 27, 4 (2021), 114–128. doi:10.17576/3l-2021-2704-09 ↗
  2. Zachariah J. Beasley, Alon Friedman, and Paul Rosen. 2021. Through the Looking Glass: Insights Into Visualization Pedagogy Through Sentiment Analysis of Peer Review Text. IEEE Computer Graphics and Applications 41, 6 (2021), 59–70. doi:10.1109/mcg.2021.3115387 ↗
  3. Alex Duncan, Bobbie Eicher, and David A. Joyner. 2020. Enrollment Motivations in an Online Graduate CS Program. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. ACM, Portland, OR, USA, 1241–1247. doi:10.1145/3328778.3366848 ↗
  4. Mario Gielen and Bram De Wever. 2015. Structuring peer assessment: Comparing the impact of the degree of structure on peer feedback content. Computers in Human Behavior 52 (2015), 315–325. doi:10.1016/j.chb.2015.06.019 ↗
  5. Changqin Huang, Yaxin Tu, Zhongmei Han, Fan Jiang, Fei Wu, and Yunliang Jiang. 2023. Examining the relationship between peer feedback classified by deep learning and online learning burnout. Computers & Education 207 (2023), 104910. doi:10.1016/j.compedu.2023.104910 ↗
  6. Bart Huisman, Nadira Saab, Paul Van den Broek, and Jan Van Driel. 2018. The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis. Assessment & Evaluation in Higher Education 44, 6 (2018), 863–880. doi:10.1080/02602938.2018.1545896 ↗
  7. Ratanond Koonchanok, Yanling Pan, and Hyeju Jang. 2024. Public attitudes toward chatgpt on twitter: sentiments, topics, and occupations. Social Network Analysis and Mining 14, 1 (2024), 106. doi:10.1007/s13278-024-01260-7 ↗
  8. Karen Kreutel. 2007. "I’m not agree with you." ESL Learners’ Expressions of Disagreement. Teaching English as a Second or Foreign Language 11, 3 (2007), 1–35. find
  9. Jacqueline Ng Lane, Bruce Ankenman, and Seyed Iravani. 2018. Insight into Gender Differences in Higher Education: Evidence from Peer Reviews in an Introductory STEM Course. Service Science 10, 4 (2018), 442–456. doi:10.1287/ serv.2018.0224 ↗
  10. Campbell Leaper and Melanie M. Ayres. 2007. A Meta-Analytic Review of Gender Variations in Adults. Personality and Social Psychology Review 11, 4 (2007), 328–363. doi:10.1177/1088868307302221 ↗
  11. Hedy McGarrell. 2010. Native and non-native English speaking student teachers engage in peer feedback. Canadian Journal of Applied Linguistics 13, 1 (2010), 71–90. find
  12. Brian McKenzie, Enyu Zhou, and Alessandro Regio. 2023. Graduate Enrollment and Degrees: 2012 to 2022 Report. https://cgsnet.org/wp-content/uploads/2023/10/2022-Graduate-Enrollment-and-Degrees-Final-Report.pdf. Retrieved January 15, 2025. ↗
  13. Marion Neumann and Robin Linzmayer. 2021. Capturing Student Feedback and Emotions in Large Computing Courses: A Sentiment Analysis Approach. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 541–547. doi:10.1145/3408877.3432403 ↗
  14. Heather Newman and David Joyner. 2018. Sentiment Analysis of Student Evaluations of Teaching. In Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science, vol 10948, C. Penstein Rosé, R. Martínez-Maldonado, H. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, and B. du Boulay (Eds.). Springer, Cham, Cham, Switzerland, 246–250. doi:10.1007/978-3-319-93846-2_45 ↗
  15. NFAP. 2021. International Students in Science and Engineering. https://nfap.com/wp-content/uploads/2021/09/International-Students-in-Science-and-Engineering.NFAP-Policy-Brief.August-2021.pdf. Retrieved January 15, 2025. ↗
  16. Kathryn Nielsen. 2014. On class, race, and dynamics of privilege: Supporting generation 1.5 writers across the curriculum. In WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices. Parlor Press, Anderson, SC, 129–150. find
  17. Omid Noroozi, Maryam Alqassab, Nafiseh Taghizadeh Kerman, Seyyed Kazem Banihashem, and Ernesto Panadero. 2024. Does perception mean learning? Insights from an online peer feedback setting. Assessment & Evaluation in Higher Education 50, 1 (2024), 83–97. doi:10.1080/02602938.2024.2345669 ↗
  18. José Carlos Ocampo, Ernesto Panadero, David Zamorano, Iván Sánchez-Iglesias, and Fernando Diez Ruiz. 2023. The effects of gender and training on peer feedback characteristics. Assessment & Evaluation in Higher Education 49, 4 (2023), 539–555. doi:10.1080/02602938.2024.2345669 ↗
  19. Ernesto Panadero and Anastasiya A. Lipnevich. 2022. A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review 35 (2022), 100416. doi:10.1016/j.edurev.2021.100416 ↗
  20. Melinda Reichelt. 2021. Linguistic Bias against ESL Writing? In Linguistic Discrimination in US Higher Education. Routledge, New York, NY, 20–37. ↗
  21. Donald L. Rubin and Melanie Williams-James. 1997. The impact of writer nationality on mainstream teachers. Journal of Second Language Writing 6, 2 (1997), 139–154. doi:10.1016/s1060-3743(97)90031-x ↗
  22. Anna Wärnsby, Asko Kauppinen, Laura Aull, Djuddah Leijen, and Joe Moxley. 2018. Affective Language in Student Peer Reviews: Exploring Data from Three Institutional Contexts. Journal of Academic Writing 8, 1 (2018), 28–53. doi:10. 18552/joaw.v8i1.429 ↗
  23. Jiming Zhou, Yongyan Zheng, and Joanna Hong-Meng Tai. 2019. Grudges and gratitude: the social-affective impacts of peer assessment. Assessment & Evaluation in Higher Education 45, 3 (2019), 345–358. doi:10.1080/02602938.2019. 1643449 ↗
  24. Wei Zhu. 2001. Interaction and feedback in mixed peer response groups. Journal of Second Language Writing 10, 4 (2001), 251–276. doi:10.1016/s1060-3743(01) 00043-1 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/exline-2025-using-sentiment-analysis-investigate-peer/