Elamur Research⌕
☰
home / papers / 2025 / LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives

LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives

Paper · 2025 · Ratna Kandala and Katie Hoemann

Summary

The paper asks whether recent Dutch-adapted LLMs estimate emotional valence in everyday Flemish writing better than classic lexicon tools used in psychology. The authors used an ambulatory-assessment corpus of 24,854 free-text entries from 102 participants in Belgium, gathered over 70 days. Four times a day, participants described what was happening and how they felt, and rated their own valence on a –50 to +50 scale. Those self-ratings, rather than annotator labels, served as ground truth. Three open-weight Dutch LLMs, given a single English prompt asking for a 1–7 rating, were compared with LIWC and Pattern.nl using Pearson and polyserial correlations. The lexicon tools scored almost every text. The LLMs often failed to return a usable rating. Pattern.nl was the most reliable tool overall. On the texts they did score, ChocoLlama and GEITje correlated somewhat better with self-reports. The work matters for low-resource variants because coverage and domain fit can outweigh model sophistication when tools are applied to real, colloquial narratives.

Key points

  • Data: 24,854 Dutch/Flemish entries from 102 participants (ages 18–65) over 70 days. Each entry was a short text or a 1-minute voice note, paired with a self-rated valence from –50 to +50.
  • Models and tools: ChocoLlama-8B-Instruct (Llama-3-8B, LoRA), GEITje-7B-ultra (Mistral 7B, full pretraining), and Reynaerde-7B-chat (Llama-2-7B, QLoRA) were compared with LIWC (Dutch dictionary of 6,614 words) and Pattern.nl (3,304 lemmas, mostly adjectives).
  • Coverage:
    • LIWC and Pattern.nl: 99.9%
    • ChocoLlama: 69.9%
    • GEITje: 38.0%
    • Reynaerde: 1.8%
  • Correlations with self-reports (Pearson / polyserial):
    • Pattern.nl: 0.31 / 0.31
    • LIWC PosEmo: 0.21 / 0.23
    • LIWC NegEmo: –0.23 / –0.23
    • ChocoLlama: 0.35 / 0.40 on its subset, significantly above Pattern there (t = 4.02)
    • GEITje: 0.35 / 0.44, not significantly different from Pattern
    • Reynaerde: 0.18 / 0.24
  • Prompt variations: Switching between Dutch and English prompts made no notable difference, and neither did zero-shot versus few-shot prompting. The authors attribute LLM weakness to a mismatch between formal training corpora and casual first-person narratives.

Interesting ideas

  • I1. Using the writer's own rating as the gold label avoids annotator guesswork about inner states, which suits any task where the author is the true authority on meaning.
  • I2. Reporting coverage next to accuracy, and comparing systems on the same subset, shows that a model which silently fails on many inputs can look stronger than it is.
  • I3. Pattern.nl beat raw word counting by scoring chunks and applying negation and intensifier rules before aggregating, so even simple compositional structure adds signal.
  • I4. A lexicon built from product reviews ends up treating out-of-domain emotional words as neutral, so the origin of a resource quietly shapes its errors.
  • I5. Forcing an LLM into a strict output format can fail massively, as Reynaerde's 1.8% usable answers show, so output parsing is itself a failure point to measure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Where possible, Mova could score its emotion or attitude readings against author-supplied labels, keeping these separate from judgments inferred by outside annotators or LLM teachers.
  • I2 (learn): Every evaluation, especially of LLM teacher tiers, should log abstentions and malformed outputs and measure scores on the intersection of inputs, so selection bias cannot inflate the results.
  • I3 (frames): Polarity could be computed over the dependency tree, flipping under negation and scaling under degree modifiers, giving an explainable attitude value per frame instead of a bag-of-words tally.
  • I4 (world): Compiled value and sentiment knowledge should record its source domain, letting Mova flag when a text's vocabulary falls outside what the resource was built on.
  • I5 (learn): LLM teacher outputs should be passed through a validated parser, with per-model failure rates tracked so that unreliable teachers are down-weighted rather than trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 works

Cite

Cite as Ratna Kandala and Katie Hoemann. 2025. LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives. arXiv:2511.07641.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.07641 · arXiv 2511.07641

@misc{kandala2025llms,
  title = {{LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives}},
  author = {Ratna Kandala and Katie Hoemann},
  year = {2025},
  doi = {10.48550/arxiv.2511.07641},
  eprint = {2511.07641},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.07641}
}

References

40 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandridis, G., Varlamis, I., Korovesis, K., Caridakis, G., and Tsantilas, P. (2021). A survey on sentiment analysis and opinion mining in Greek social media. Information, 12:331. ↗
  2. Ali, H., Hashmi, E., Yildirim, S. Y., and Shaikh, S. (2024). Analyzing Amazon products sentiment: a comparative study of machine and deep learning, and transformer-based techniques. Electronics, 13(7):1305. ↗
  3. Augustyniak, Ł., Woźniak, S., Gruza, M., Gramacki, P., Rajda, K., Morzy, M., and Kajdanowicz, T. (2023). Massively multilingual corpus of sentiment datasets and multi-faceted sentiment classification benchmark. arXiv preprint arXiv:2306.07902. ↗
  4. Blitzer, J., Dredze, M., and Pereira, F. (2007). Biographies, Bollywood, boom-boxes: domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pp. 440–447. find
  5. Boot, P., Zijlstra, H., and Geenen, R. (2017). The Dutch translation of the Linguistic Inquiry and Word Count (LIWC) 2007 dictionary. Dutch Journal of Applied Linguistics, 6(1):65–76. ↗
  6. Bordia, S. and Bowman, S. R. (2019). Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 7–15. Association for Computational Linguistics. https://aclanthology.org/N19-3002/. ↗
  7. Chun, J. (2021). SentimentArcs: a novel method for self-supervised sentiment analysis of time series shows SOTA transformers can struggle finding narrative arcs. arXiv preprint arXiv:2110.09454. ↗
  8. Dashtipour, K., Gogate, M., Adeel, A., Larijani, H., and Hussain, A. (2021). Sentiment analysis of Persian movie reviews using deep learning. Entropy, 23:596. ↗
  9. De Smedt, T. and Daelemans, W. (2012). Pattern for Python. Journal of Machine Learning Research, 13(1):2063–2067. ↗
  10. Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., Jones, M., Krettek-Cobb, D., Lai, L., Mitchell, N. J., Ong, D. C., Dweck, C. S., Gross, J. J., and Pennebaker, J. W. (2023). Using large language models in psychology. Nature Reviews Psychology, 2(11):688–701. ↗
  11. Fiok, K., Karwowski, W., Gutierrez, E., and Wilamowski, M. (2021). Analysis of sentiment in tweets addressed to a single domain specific Twitter account: comparison of model performance and explainability of predictions. Expert Systems with Applications, 186:115771. ↗
  12. Frisina, P. G., Borod, J. C., and Lepore, S. J. (2004). A meta-analysis of the effects of written emotional disclosure on the health outcomes of clinical populations. The Journal of Nervous and Mental Disease, 192(9):629–634. ↗
  13. Gatti, L., and van Stegeren, J. (2020). Improving Dutch sentiment analysis in Pattern. Computational Linguistics in the Netherlands Journal, 10:73–89. find
  14. Hutto, C. J. and Gilbert, E. (2014). VADER: a parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media, 8(1):216–225. find
  15. Li, Z., Shi, Y., Liu, Z., Yang, F., Payani, A., Liu, N., and Du, M. (2024). Quantifying multilingual performance of large language models across languages. arXiv preprint arXiv:2404.11553. ↗
  16. Ligthart, A., Catal, C., and Tekinerdogan, B. (2021). Systematic reviews in sentiment analysis: a tertiary study. Artificial Intelligence Review, 54:4997–5053. ↗
  17. Liu, Z., Zhang, T., Yang, K., Thompson, P., Yu, Z., and Ananiadou, S. (2023). Emotion detection for misinformation: a review. arXiv preprint arXiv:2311.00671. ↗
  18. Liu, Z., Yang, K., Xie, Q., Zhang, T., and Ananiadou, S. (2024). EmoLLMs: a series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5487–5496. find
  19. Martínez, G., Molero, J. D., González, S., Conde, J., Brysbaert, M., and Reviriego, P. (2025). Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57:5. https://doi.org/10.3758/s13428-024-02515-z ↗
  20. Meeus, M., Rathé, A., Rémy, F., Delobelle, P., Decorte, J.-J., and Demeester, T. (2024). ChocoLlama: lessons learned from teaching llamas Dutch. arXiv preprint arXiv:2412.07633. ↗
  21. Mishra, K., Priya, P., Burja, M., and Ekbal, A. (2023). e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13952–13967. find
  22. Mohammad, S. M. and Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  23. Muhammad, S., Abdulmumin, I., Ayele, A., Ousidhoum, N., Adelani, D., Yimam, S., Ahmad, I., Beloucif, M., Mohammad, S., Ruder, S., Hourrane, O., Jorge, A., Brazdil, P., Ali, F., David, D., Osei, S., Shehu-Bello, B., Lawan, F., Gwadabe, T., Rutunda, S., Belay, T., Messelle, W., Balcha, H., Chala, S., Gebremichael, H., and Opoku, B. (2023). AfriSenti: a Twitter sentiment analysis benchmark for African languages. In Proceedings of EMNLP 2023. ↗
  24. Obradovich, N., Khalsa, S. S., Khan, W. U., et al. (2024). Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry Neuroscience, 2:8. ↗
  25. Pataranutaporn, P., Liu, R., Finn, E., et al. (2023). Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5:1076–1086. ↗
  26. Pennebaker, J. W. (1997). Writing about emotional experiences as a therapeutic process. Psychological Science, 8(3):162–166. ↗
  27. Pennebaker, J. W., Francis, M. E., and Booth, R. J. (2001). Linguistic Inquiry and Word Count: LIWC 2001. Lawrence Erlbaum Associates, Mahwah, NJ. find
  28. Pennebaker, J. W., Booth, R. J., and Francis, M. E. (2007). Linguistic Inquiry and Word Count: LIWC2007: operator’s manual. Austin, TX: LIWC.net. find
  29. Pennebaker, J. W., Booth, R. J., Boyd, R. L., and Francis, M. E. (2015). Linguistic Inquiry and Word Count: LIWC2015. Austin, TX: Pennebaker Conglomerates. https://www.liwc.net ↗
  30. Qiu, J., Lam, K., Li, G., et al. (2024). LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6:1418–1420. ↗
  31. Radaideh, M. I., Hwang, O. H., and Radaideh, M. I. (2024). Fairness and social bias quantification in large language models for sentiment analysis. SSRN. ↗
  32. ReBatch. (2024). Reynaerde-7B-Chat. Hugging Face. find
  33. Reusens, M., Reusens, M., Callens, M., Vanden Broucke, S., and Baesens, B. (2022). Benchmark study for Flemish Twitter sentiment analysis. SSRN. https://ssrn.com/abstract=4096559 ↗
  34. Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., and Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4):1064–1082. https://doi.org/10.1111/ajps.12103 ↗
  35. Shteynberg, G., Halpern, J., Sadovnik, A., et al. (2024). Does it matter if empathic AI has no empathy? Nature Machine Intelligence, 6:496–497. ↗
  36. Tausczik, Y. R. and Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1):24–54. ↗
  37. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. https://arxiv.org/abs/2307.09288 ↗
  38. Vanroy, B. (2024). GEITje 7B Ultra: a conversational model for Dutch. arXiv preprint arXiv:2412.04092. ↗
  39. Varghese, C., Harrison, E. M., O’Grady, G., et al. (2024). Artificial intelligence in surgery. Nature Medicine, 30:1257–1268. ↗
  40. Zhang, W., Deng, Y., Liu, B., Pan, S., and Bing, L. (2024). Sentiment analysis in the era of large language models: a reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3881–3906.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kandala-2025-llms-vs-traditional-sentiment-tools/