Elamur Research⌕
☰
home / papers / 2026 / Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals

Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals

Paper · 2026 · Tina Munda, Filip Dobranić, Uroš Šmajdek et al.

Summary

The paper asks whether instruction-tuned LLMs, used without supervised fine-tuning, can judge sentiment toward specific targets in noisy OCR text from historical Slovene newspapers. Labelled data for this setting is scarce, and the language is both morphologically rich and historically variable. The authors draw texts from three sPeriodika newspapers. Using a hand-built lexicon, they extract nominal and adjectival mentions of collective identities (ethno-national, regional and geographic groups). They then manually annotate 371 mentions for sentiment toward each marked target.

Several models are compared in a few-shot setup using standard classification metrics:

  • the Slovene-adapted GaMS3-12B-Instruct;
  • general-purpose models of similar size;
  • two smaller variants.

The analysis also breaks down performance by grammatical form and by referential type. This reveals systematic asymmetries that skew aggregated results. A dataset-scale aggregation with the best model is offered only as a demonstration of scalability. The authors conclude that aggregated polarity should be read as a directional tendency rather than a precise measurement. They also argue that interpretative validity matters more to digital humanities work than raw F1 scores.

Key points

  • Data: Slovenski narod (~183M tokens), Slovenec (~137M) and Slovenka (~1.6M), over 320M tokens in total. Only issues up to 31 December 1914 are included, for copyright reasons.
  • Benchmark: 371 manually annotated collective-identity mentions, framed as aspect-level (targeted) sentiment classification.
  • Models:
    • GaMS3-12B-Instruct (Gemma 3-based, continually pretrained on Slovene);
    • Gemma-3-12B-IT, LLaMA 3.1 and DeepSeek-R1-Distill-Qwen-14B;
    • the smaller Gemma-3-4B-IT and DeepSeek-R1-Distill-Qwen-7B, included to test size effects.
  • Lexicon construction:
    • Noun lemmas were compiled manually from Slovenka.
    • Adjective candidates were found via suffixes such as -ski/-ški, including the historical spellings -zki/-žki. In the two larger papers, they were filtered at a minimum of 90 occurrences and then reviewed.
    • Adjectives were mapped to nominal identity labels (e.g., nemški → Nemci).
  • Diagnostics: Accuracy differs by realisation (noun vs. adjective) and by referential type (collective actor vs. non-agentive entity). These differences must be accounted for when aggregating sentiment across a corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LjubljanaLjubljana · Slovenia13 works · 38,693 citations🔬Institute of Contemporary HistoryLjubljana · Slovenia4 works · 32 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 worksLiterature and Literary Theory · 541 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Tina Munda, Filip Dobranić, Uroš Šmajdek, Oliver Pejić, Ciril Bohak, Vojko Gorjanc and Darja Fišer. 2026. Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/22hvcbc23rts

@inproceedings{munda2026benchmarking,
  title = {{Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals}},
  author = {Tina Munda, Filip Dobranić, Uroš Šmajdek, Oliver Pejić, Ciril Bohak, Vojko Gorjanc and Darja Fišer},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  year = {2026},
  doi = {10.63317/22hvcbc23rts},
  url = {https://doi.org/10.63317/22hvcbc23rts}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. find
  2. Swoichha Adhikari, Manan Gangwani, and Adithi Varadarajan. 2024. Aspect-based sentiment analysis for slovene texts: Models, lexicons, and embeddings. In 2024 IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI), volume 2, pages 1–6… find
  3. Erik Cambria, Björn Schuller, Yunqing Xia, and Catherine Havasi. 2017. New avenues in opinion mining and sentiment analysis. IEEE Intelligent Systems, 28(2):15–21. find
  4. William B Cavnar, John M Trenkle, et al. 1994. Ngram-based text categorization. In Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval, volume 161175, page 14. Las Vegas, NV. find
  5. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273–297.
  6. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Filip Dobranić, Bojan Evkoski, and Nikola Ljubešić. 2024. A lightweight approach to a giga-corpus of historical periodicals: The story of a slovenian historical newspaper collection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalua… find
  9. Darja Fišer, Jasmina Smailovic, Tomaž Erjavec, Igor Mozetic, and Miha Grcar. 2016. Sentiment annotation of slovene user-generated content. In Proceedings of the 2016 conference language technologies and digital humanities (JTDH 2016), pages 65–70. find
  10. Aaron Grattafiori and Abhimanyu Dubey et al. 2024. The Llama 3 Herd of Models.
  11. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of C.J. find
  12. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth International Conference on Weblogs and Social Media (ICWSM-14), pages 216–225. find
  13. Marjeta Keršič-Svetel. 1996. Češko-slovenski stiki med svetovnima vojnama. Zbirka Zgodovinskega časopisa. Zveza zgodovinskih društev Slovenije, Ljubljana. find
  14. Oto Luthar, Igor Grdina, Marjeta Šašel Kos, Petra Svoljšak, Peter Kos, Dušan Kos, Peter Štih, Alja Brglez, and Martin Pogačar. 2008. The land between: a history of slovenia. find
  15. Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330.
  16. Iftikhar Muhammad, Marco Rospocher, Timotej Knez, and Slavko Žitnik. 2025. Benchmarking large language models for target-based financial sentiment analysis. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 785–795, Cagliari, Italy. CEUR Workshop Pr… find
  17. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics.
  18. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 380–385.
  19. Domen Vreš, Martin Božič, Aljaž Potočnik, Tomaž Martinčič, and Marko Robnik-Šikonja. 2024. Generative model for less-resourced language with 1 billion parameters. arXiv preprint arXiv:2410.06898. — [arXiv:2410.06898](https://arxiv.org/abs/2410.06898) find
  20. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881– 3906. Association for Computational Linguistics.
  21. Filip Čuček. 2016. Svoji k svojim: na poti k dokončni nacionalni razmejitvi na spodnjem Štajerskem v 19. stoletju. find
  22. Slavko Žitnik, Neli Blagus, and Marko Bajec. 2022. Target-level sentiment analysis for news articles. Knowledge-Based Systems, 249:108939. find
  23. Language Resource References Filip Dobranić, Bojan Evkoski, and Nikola Ljubešić. 2023. Corpus of slovenian periodicals (1771-1914) sPeriodika 1.0. Slovenian language resource repository CLARIN.SI. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/munda-2026-benchmarking-llms-aspect-based-sentiment/