Elamur Research⌕
☰
home / papers / 2026 / Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark

Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark

Paper · 2026 · Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska

Summary

Wikipedia articles must follow the Neutral Point of View (NPOV) policy, so any sentiment they carry is rare and implicit. This makes them a hard test for sentiment tools, especially when comparing language editions. The authors build a multilingual, multi-domain benchmark from the lead sections of important articles in English, German, Spanish, Polish and Russian. The articles come from two curated lists of about 1,000 entries each, which were merged and deduplicated. Sentences, segmented with Stanza, were labeled by GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Each model ran three times, and only sentences on which all nine runs agreed entered the benchmark. Two multilingual BERT-style classifiers were then scored against these consensus labels. The scoring covered single sentences and whole documents, using two proposed aggregation strategies. The results show clear differences in label distributions, agreement and run-to-run stability across models and languages. The resource offers a dependable reference for low-subjectivity text. However, the authors caution that consensus labels are not true ground truth and may favour easier cases.

Key points

  • Corpus size: sentences per language after preprocessing were 21,991 (en), 12,715 (de), 17,021 (es), 10,707 (pl) and 12,739 (ru). Each sentence ID combines the page ID and the sentence index.
  • Annotation scheme: the LLMs used four labels (positive, neutral, negative, n/a) with one shared prompt. Batches were sent as JSON lines of up to 25k tokens. The repeated runs measure intra-model stability as a stand-in for confidence.
  • Classifiers tested:
    • modBERT (clapAI/modernBERT-base-multilingual-sentiment): about 150M parameters, trained on roughly 3.93M samples, reported F1 of 80.16. Polish is not among its training languages.
    • mul-BERT (tabularisai/multilingual-sentiment-analysis): DistilBERT-based, trained only on LLM-generated synthetic data, covers 23 languages including Polish.
  • Evaluation metric: agreement is measured mainly with Cohen's κ. The authors warn that κ is unreliable on neutral-dominated data, where κ = 1.0 can simply mean every label was "neutral".
  • Stated limitations: unanimous-consensus filtering trades coverage for reliability and drops borderline sentences. Other limits are the focus on lead sections only, five languages, a small set of models, and sensitivity to the prompt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Poznań University of Economics and BusinessPoznan · Poland3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska. 2026. Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.63

@inproceedings{strozyna2026evaluating,
  title = {{Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark}},
  author = {Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.63},
  url = {https://doi.org/10.18653/v1/2026.gem-main.63}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emmy Danny Ajik, Aminu Bashir Suleiman, and Muhammad Ibrahim. 2023. Enhancing user experience through sentiment analysis for katsina state transport agency: A textblob approach. FUDMA Journal of Sciences, 7(6):117–122. find
  2. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resour…
  3. Hadis Bashiri and Hassan Naderi. 2024. Comprehensive review and comparative analysis of transformer models in sentiment analysis. Knowledge and Information Systems, 66(12):7305–7361. find
  4. Anton Borg and Martin Boldt. 2020. Using vader sentiment and svm for predicting customer response sentiment. Expert Systems with Applications, 162:113746. find
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  6. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  7. Himanshu Gautam, Abhishek Gaur, and Dharmendra Kumar Yadav. 2025. A survey on the impact of pre-trained language models in sentiment classification task. International Journal of Data Science and Analytics, 20(6):5197–5235. find
  8. K Puttaswamy Gowda, R Porwal, C Ramesh, SS Tiwari, K Srivastava, R Rambabu, and SG Rao. 2025. Transformers in sentiment analysis: A paradigm shift in deep learning research. J. Inf. Syst. Eng. Manag., 10(5):2468–4376. find
  9. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konr… — ChatGPT: Jack of all trades, master of none
  10. Włodzimierz Lewoniewski, Milena Stróżyna, Izabela Czumałowska, and Elżbieta Lewańska. 2026a. Science fiction and fantasy in wikipedia: Exploring structural and semantic cues. arXiv preprint arXiv:2602.24229. — [arXiv:2602.24229](https://arxiv.org/abs/2602.24229) find
  11. Włodzimierz Lewoniewski, Milena Stróżyna, Izabela Czumałowska, Aleksandra Wojewoda, and Krzysztof W˛ecel. 2026b. Cross-Topic Sentiment Analysis of Wikipedia Articles: A Comparative Study of AI Models. In Artificial Intelligence for Knowledge Acquisition and Management. AI4KAM 2025. IFIP Advances in… find
  12. Yanying Mao, Qun Liu, and Yu Zhang. 2024. Sentiment analysis methods, applications, and challenges: A systematic literature review. Journal of King Saud University - Computer and Information Sciences, 36. find
  13. Sorin Adam Matei and Caius Dobrescu. 2011. Wikipedia’s "neutral point of view": Settling conflict through ambiguity. The Information Society, 27(1):40–51. find
  14. Milena Stróżyna, Włodzimierz Lewoniewski, Izabela Czumałowska, and Aleksandra Wojewoda. 2025. Sentiment analysis of wikipedia articles about companies: A comparison of different models. In International Conference on Business Information Systems, pages 101–115. Springer. find
  15. Madhumita Sushil, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Dmytro S. Lituiev, and Atul J. Butte. 2024. A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/strozyna-2026-evaluating-multilingual-sentiment-classifiers-llm/