Elamur Research⌕
☰
home / papers / 2025 / Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?

Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?

Paper · 2025 · Ratna Kandala and Katie Hoemann

Summary

The paper asks whether Dutch-adapted large language models can now replace classic dictionary-based tools for estimating emotional valence in everyday Flemish (Belgian Dutch), which is under-resourced in NLP. The authors used about 25,000 open-ended texts from 102 Dutch speakers in Belgium, collected through ambulatory assessment. In this method, participants repeatedly answered, in daily life, a prompt about what was happening and how they felt. The writer also rated the valence of each answer, choosing any value between −50 and +50. Three Dutch-tuned LLMs (ChocoLlama-8B-Instruct, Reynaerde-7B-chat, GEITje-7B-ultra) were asked to predict these self-ratings. Their predictions were compared with scores from LIWC (Dutch dictionary) and Pattern. The LLMs aligned worse with human ratings than the lexicon tools did, and Pattern.nl gave the best results. The authors attribute the LLMs' weakness to training corpora dominated by English and formal registers. They argue that lexicons remain the practical choice for Flemish until narrative-style benchmarks and fine-tuning data are available.

Key points

  • The dataset has roughly 25,000 spontaneous Flemish narratives from 102 participants, each labelled with a self-reported valence from −50 to +50 rather than by external annotators.
  • The LLMs compared were ChocoLlama-8B-Instruct, Reynaerde-7B-chat and GEITje-7B-ultra. The lexicon baselines were LIWC (Dutch adaptation, 90+ categories, PosEmo/NegEmo) and Pattern.nl.
  • Pattern.nl did better than LIWC. The authors credit its composite polarity scoring, as opposed to raw emotion-word counts. Both lexicon tools beat the Dutch LLMs in coverage and fairness.
  • The paper cites the skew in pretraining data: Llama-2 is about 89.7% English and GPT-3 about 92.65% English. ChocoLlama added 32 billion Dutch tokens, but this still did not cover colloquial narrative language.
  • Stated limitations:
    • only one elicitation prompt was used;
    • larger models (e.g., LLaMA-2 70B) and hybrid systems were not tested;
    • lexicons still cannot handle sarcasm or idioms.

Interesting ideas

  • I1. Using the text author's own rating as ground truth avoids the bias of third-party annotators interpreting someone else's feelings, which makes the benchmark a cleaner test of what the writer actually meant.
  • I2. The paper's sarcasm examples show positive words placed inside a clearly bad situation, such as an "amazing" Monday stuck in traffic. Word-level polarity alone gets these wrong.
  • I3. A score built from words plus their modifiers (Pattern's composite polarity) beat simple counting of emotion words. This suggests that how valence is combined matters as much as the lexicon itself.
  • I4. Reporting coverage (how often a tool produces a usable output) alongside alignment exposed a weakness of generative models that a single accuracy figure could hide.
  • I5. The authors blame part of the gap on register mismatch: training data are formal, while the target texts are first-person and colloquial.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Where attitude or sentiment labels are needed, prefer author-provided or source-intrinsic labels over LLM-teacher judgments. Use them as an independent check on the teacher tiers.
  • I2 (frames): Flag likely irony when evaluative words in the tree clash with the valence of the event frame they describe (being stuck, crashing). This gives an explainable irony cue.
  • I3 (world): Attach curated valence values to lemmas in the compiled knowledge. Compose them along UD dependencies (negation, intensifiers, scare quotes) to set states in the text's world, instead of tallying words.
  • I4 (learn): In every evaluation, report the abstention or failure rate separately from correctness, so a component cannot look good by silently skipping hard inputs.
  • I5 (tree): Check parser and tagger quality specifically on informal first-person narratives (fables, diaries), and add such texts to the silver-data pools if accuracy drops.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of KansasLawrence · United States8 works · 32 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 267 works

Cite

Cite as Ratna Kandala and Katie Hoemann. 2025. Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?. arXiv:2506.04139 · Presented at LLM Evaluation Workshop, NeurIPS 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.04139 · arXiv 2506.04139

@misc{kandala2025are,
  title = {{Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?}},
  author = {Ratna Kandala and Katie Hoemann},
  year = {2025},
  doi = {10.48550/arxiv.2506.04139},
  eprint = {2506.04139},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.04139}
}

References

40 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandridis, G., Varlamis, I., Korovesis, K., Caridakis, G., & Tsantilas, P. (2021). A survey on sentiment analysis and opinion mining in Greek social media. Information, 12, Article 331. https://doi.org/10.3390/info12080331 ↗
  2. Ali, H., Hashmi, E., Yildirim, S. Y., & Shaikh, S. (2024). Analyzing Amazon products sentiment: A comparative study of machine and deep learning, and transformer-based techniques. Electronics, 13(7), Article 1305. ↗
  3. Augustyniak, Ł., Woźniak, S., Gruza, M., Gramacki, P., Rajda, K., Morzy, M., & Kajdanowicz, T. (2023). Massively multilingual corpus of sentiment datasets and multi-faceted sentiment classification benchmark. arXiv Preprint arXiv:2306.07902. https://arxiv.org/abs/2306.07902 ↗
  4. Blitzer, J., Dredze, M., & Pereira, F. (2007). Biographies, Bollywood, boom-boxes: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (pp. 440–447). Association for Computational Linguistics. find
  5. Boot, P., Zijlstra, H., & Geenen, R. (2017). The Dutch translation of the Linguistic Inquiry and Word Count (LIWC) 2007 dictionary. Dutch Journal of Applied Linguistics, 6(1), 65–76. ↗
  6. Bordia, S., & Bowman, S. R. (2019). Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (pp. 7–15). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-3002 ↗
  7. Chun, J. (2021). SentimentArcs: A novel method for self-supervised sentiment analysis of time series shows SOTA transformers can struggle finding narrative arcs. arXiv Preprint arXiv:2110.09454. https://arxiv.org/abs/2110.09454 ↗
  8. Dashtipour, K., Gogate, M., Adeel, A., Larijani, H., & Hussain, A. (2021). Sentiment analysis of Persian movie reviews using deep learning. Entropy, 23, Article 596. ↗
  9. De Smedt, T., & Daelemans, W. (2012). Pattern for Python. Journal of Machine Learning Research, 13, 2063–2067. ↗
  10. Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., Jones, M., Krettek-Cobb, D., Lai, L., Mitchell, N. J., Ong, D. C., Dweck, C. S., Gross, J. J., & Pennebaker, J. W. (2023). Using large language models in psychology. Nature Reviews Psychology, 2(11), 688–701. ↗
  11. Fiok, K., Karwowski, W., Gutierrez, E., & Wilamowski, M. (2021). Analysis of sentiment in tweets addressed to a single domain-specific Twitter account: Comparison of model performance and explainability of predictions. Expert Systems with Applications, 186, 115771. ↗
  12. Frisina, P. G., Borod, J. C., & Lepore, S. J. (2004). A meta-analysis of the effects of written emotional disclosure on the health outcomes of clinical populations. The Journal of Nervous and Mental Disease, 192(9), 629–634. ↗
  13. Gatti, L., & van Stegeren, J. (2020). Improving Dutch sentiment analysis in Pattern. Computational Linguistics in the Netherlands Journal, 10, 73–89. find
  14. Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 8, pp. 216–225). AAAI Press. find
  15. Li, Z., Shi, Y., Liu, Z., Yang, F., Payani, A., Liu, N., & Du, M. (2024). Quantifying multilingual performance of large language models across languages. arXiv Preprint arXiv:2404.11553. https://arxiv.org/abs/2404.11553 ↗
  16. Ligthart, A., Catal, C., & Tekinerdogan, B. (2021). Systematic reviews in sentiment analysis: A tertiary study. Artificial Intelligence Review, 54, 4997–5053. ↗
  17. Liu, Z., Zhang, T., Yang, K., Thompson, P., Yu, Z., & Ananiadou, S. (2023). Emotion detection for misinformation: A review. arXiv Preprint arXiv:2311.00671. https://arxiv.org/abs/2311.00671 ↗
  18. Liu, Z., Yang, K., Xie, Q., Zhang, T., & Ananiadou, S. (2024). EmoLLMs: A series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 5487– 5496). ACM. find
  19. Martínez, G., Molero, J. D., González, S., Conde, J., Brysbaert, M., & Reviriego, P. (2025). Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57, Article 5. https://doi.org/10.3758/s13428-024-02515-z ↗
  20. Meeus, M., Rathé, A., Rémy, F., Delobelle, P., Decorte, J.-J., & Demeester, T. (2024). ChocoLlama: Lessons learned from teaching Llamas Dutch. arXiv Preprint arXiv:2412.07633. https://arxiv.org/abs/2412.07633 ↗
  21. Mishra, K., Priya, P., Burja, M., & Ekbal, A. (2023). e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 13952–13967). Association for Computational Linguistics. find
  22. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3), 436–465. ↗
  23. Muhammad, S., Abdulmumin, I., Ayele, A., Ousidhoum, N., Adelani, D., Yimam, S., Ahmad, I., Beloucif, M., Mohammad, S., Ruder, S., Hourrane, O., Jorge, A., Brazdil, P., Ali, F., David, D., Osei, S., Shehu-Bello, B., Lawan, F., Gwadabe, T., Rutunda, S., … Opoku, B. (2023). AfriSenti: A Twitter sentiment analysis benchmark for African languages. In Proceedings of EMNLP 2023. Association for Computational Linguistics. ↗
  24. Obradovich, N., Khalsa, S. S., Khan, W. U., … (2024). Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry Neuroscience, 2, Article 8. ↗
  25. Pataranutaporn, P., Liu, R., Finn, E., … (2023). Influencing human-AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5, 1076–1085. ↗
  26. Pennebaker, J. W. (1997). Writing about emotional experiences as a therapeutic process. Psychological Science, 8(3), 162–166. ↗
  27. Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic Inquiry and Word Count: LIWC 2001. Lawrence Erlbaum Associates. find
  28. Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). Linguistic Inquiry and Word Count (LIWC2007): Operator’s manual. LIWC.net. find
  29. Pennebaker, J. W., Booth, R. J., Boyd, R. L., & Francis, M. E. (2015). Linguistic Inquiry and Word Count: LIWC2015. Pennebaker Conglomerates. find
  30. Qiu, J., Lam, K., Li, G., … (2024). LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6, 1418–1420. ↗
  31. Radaideh, M. I., Hwang, O. H., & Radaideh, M. I. (2024). Fairness and social bias quantification in large language models for sentiment analysis. SSRN Preprint. https://doi.org/10.2139/ssrn.XXXXXX ↗
  32. ReBatch. (2024). Reynaerde-7B-Chat [Large language model]. Hugging Face. https://huggingface.co/ReBatch/Reynaerde-7B-Chat ↗
  33. Reusens, M., Reusens, M., Callens, M., Vanden Broucke, S., & Baesens, B. (2022). Benchmark study for Flemish Twitter sentiment analysis. SSRN Preprint. https://ssrn.com/abstract=4096559 ↗
  34. Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4), 1064–1082. https://doi.org/10.1111/ajps.12103 ↗
  35. Shteynberg, G., Halpern, J., Sadovnik, A., … (2024). Does it matter if empathic AI has no empathy? Nature Machine Intelligence, 6, 496–497. ↗
  36. Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton Ferrer, C., Chen, M., Cucurull, G., et al. ↗
  37. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv Preprint arXiv:2307.09288. https://arxiv.org/abs/2307.09288 ↗
  38. Vanroy, B. (2024). GEITje 7B Ultra: A conversational model for Dutch. arXiv Preprint arXiv:2412.04092. https://arxiv.org/abs/2412.04092 ↗
  39. Varghese, C., Harrison, E. M., O’Grady, G., … (2024). Artificial intelligence in surgery. Nature Medicine, 30, 1257–1268. ↗
  40. Zhang, W., Deng, Y., Liu, B., Pan, S., & Bing, L. (2024). Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3881–3906). Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kandala-2025-lexicon-based-tools-still-gold/