Elamur Research⌕
☰
home / papers / 2026 / Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian

Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian

Paper · 2026 · Kiril Simov, Nikolay Paev, Petya Osenova et al.

Summary

The authors argue that core Bulgarian NLP tasks, which Social Sciences and Humanities research relies on, do not require multi-billion-parameter systems. They pre-train compact transformer models (BERT, ModernBERT and T5 variants, all under 2B parameters) only on Bulgarian text. They then fine-tune them for tokenization, POS tagging, lemmatization, dependency parsing, NER, named entity linking, word sense disambiguation and event extraction.

Fine-tuning uses freely available resources, some of which the authors extended with new annotation. Parsing follows a two-stage graph-based scheme: first arc selection, then relation labeling. WSD classifies context–gloss pairs. Event extraction uses a joint generative Text2Event-style T5 model.

The authors report that the fine-tuned models outperform the Bulgarian systems they previously knew of. Prompted ChatGPT 4.0 did worse on POS, lemmatization and UD parsing and was roughly on par for WSD. The paper also offers a scheme that merges all annotations into HTML documents for viewing and editing.

The work matters because small models can be cheaply retrained and run on modest hardware, which makes them practical for SSH researchers.

Key points

  • Pre-training used about 29B Bulgarian tokens. Models are BERT, ModernBERT and T5 at several sizes, each below 2B parameters.
  • BulTreeBank was expanded from its original 256,000-token constituent version to 2,445,407 tokens with morphological annotation.
  • The Bulgarian portion of the Balto-Slavic (SlavNER) corpus has about 400,000 tokens. The authors linked its NEs to Wikipedia URLs and lemmatized every token.
  • The Bulgarian Event Corpus (BEC) has 227 documents and 291,196 tokens from history, biographies and ethnography. It is annotated for NEs, event triggers and roles, and co-reference chains.
  • Coreference resolution, textual entailment and sentiment analysis are still in progress. The supplied excerpt omits the per-task scores (e.g., LAS), so no figures are given here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Bulgarian Academy of SciencesSofia · Bulgaria7 works · 554 citations🔬Institute of Information and Communication TechnologiesSofia · Bulgaria3 works · 10 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMorphology · 405 works

Cite

Cite as Kiril Simov, Nikolay Paev, Petya Osenova, Teodor Valchev and Stefan Marinov. 2026. Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5n9bq7g3it2s

@inproceedings{simov2026small,
  title = {{Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian}},
  author = {Kiril Simov, Nikolay Paev, Petya Osenova, Teodor Valchev and Stefan Marinov},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  year = {2026},
  doi = {10.63317/5n9bq7g3it2s},
  url = {https://doi.org/10.63317/5n9bq7g3it2s}
}

References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kalina Bontcheva, Hamish Cunningham, Ian Roberts, Angus Roberts, Valentin Tablan, Niraj Aswani, and Genevieve Gorrell. 2013. Gate teamware: a web-based, collaborative text annotation framework. Language Resources and Evaluation, 47(4):1007–1029. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  3. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I…
  4. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pa…
  5. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66– 71, Brussels, Belgium. Assoc…
  6. Viet Dac Lai. 2022. Event extraction: A survey. find
  7. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the…
  8. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G.C. de Souza, Alexandra Birch, and André F.T. Martins. 2025. Eurollm: Multilingual language mod…
  9. Ryan McDonald. 2006. Discriminative Training and Spanning Tree Algorithms for Dependency Parsing. Ph.D. thesis. find
  10. Ryan McDonald, Kevin Lerman, and Fernando Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proceedings of the Tenth Conference on Computational Natural Language Learning, CoNLL-X ’06, page 216–220, USA. Association for Computational Linguistics.
  11. Mariana Neves and Jurica Ševa. 2019. An extensive review of tools for manual annotation of documents. Briefings in Bioinformatics, 22(1):146– 163. find
  12. Saziye Betül Özates, Arzucan Özgür, Tunga Güngör, and Balkiz Öztürk. 2020. A hybrid approach to dependency parsing: Combining rules and morphology with deep learning. CoRR, abs/2002.10116. — [arXiv:2002.10116](https://arxiv.org/abs/2002.10116) find
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019a. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
  14. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019b. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  16. Yikang Shen, Shawn Tan, Alessandro Sordoni, Siva Reddy, and Aaron Courville. 2021. Explicitly modeling syntax in language models with incremental parsing and a dynamic oracle. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  17. Étienne Simon, Helene Olsen, Huiling You, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2024. Generative approaches to event extraction: Survey and outlook. In Proceedings of the Workshop on the Future of Event Detection (FuturED), pages 73–86, Miami, Florida, USA. Association for Computational Li… find
  18. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021a. Fast Word-Piece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  19. Yang Song, Xin Cai Ong, Hwee Tou Ng, and Qian Lin. 2021b. Improved Word Sense Disambiguation with Enhanced Sense Representations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4311–4320. ACL. find
  20. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  21. Houquan Zhou, Yang Hou, Zhenghua Li, Xuebin Wang, Zhefeng Wang, Xinyu Duan, and Min Zhang. 2023. How well do large language models understand syntax? an evaluation by asking natural language questions. 8. find
  22. Language Resource References Marta Bañón, Mălina Chichirău, Miquel Esplà-Gomis, Mikel Forcada, Aarón Galiano-Jiménez, Taja Kuzman, Nikola Ljubešić, Rik van Noord, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Peter Rupnik, Vit Suchomel, Antonio Toral, and Jaume Zaragoza-Bernabeu. 2023. MaCoCu: Massive… find
  23. Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technolo…
  24. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Com…
  25. Jakub Piskorski, Bogdan Babych, Zara Kancheva, Olga Kanishcheva, Maria Lebedeva, Michał Marcińczuk, Preslav Nakov, Petya Osenova, Lidia Pivovarova, Senja Pollak, Pavel Přibáň, Ivaylo Radev, Marko Robnik-Sikonja, Vasyl Starko, Josef Steinberger, and Roman Yangarber. 2021. Slav-NER: the 3rd cross-ling… find
  26. Kiril Simov and Petya Osenova. 2023. Recent Developments in BTB-WordNet. In Proceedings of the 12th Global Wordnet Conference, pages 220– 227, University of the Basque Country, Donostia - San Sebastian, Basque Country. Global Wordnet Association. find
  27. Kiril Simov, Petya Osenova, Milena Slavcheva, Sia Kolkovska, Elisaveta Balabanova, Dimitar Doikoff, Krassimira Ivanova, Alexander Simov, and Milen Kouylekov. 2002. Building a linguistically interpreted corpus of Bulgarian: the Bul-TreeBank. In Proceedings of the Third International Conference on Lan… find
  28. Kiril Simov, Nikolay Paev, Petya Osenova, and Stefan Marinov. 2025. Bulgarian event extraction with llms. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI era, pages 1163– 1171, Varna, Bulgaria. I… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/simov-2026-small-can-be-beautiful-llms/