Elamur Research⌕
☰
home / papers / 2026 / SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction

SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction

Summary

Keyphrase extraction for heavily inflected, low-resource languages such as Slovak has received little attention, mainly because no sizeable evaluation resource existed. The authors collected nearly all records from the Slovak Central Register of Theses and Dissertations and cleaned them in several stages. They removed duplicates, stripped metadata prepended to abstracts, recovered keyphrases that students had appended to abstract text, filtered out non-Slovak content with lingua, and normalized keyphrase lists using Stanza. The resulting dataset, SlovKE, contains 227,432 abstracts with author-assigned keyphrases. That is roughly 25 times larger than the previous biggest Slovak collection and close in scale to KP20K.

On this data they benchmark YAKE, TextRank and KeyBERT (with SlovakBERT embeddings) and test KeyLLM with GPT-3.5-turbo. The extractive baselines score poorly under exact matching but much better under partial matching. The authors attribute this gap to inflected surface forms in the text failing to match the nominative keyphrases authors assign. KeyLLM generates phrases rather than copying them, which narrows the gap. Human judgment also shows it captures relevant concepts that strict metrics miss. These findings matter for other morphologically rich languages.

Key points

  • Data source and scale: In 2024, the register listed 794,527 records, of which 793,722 were retrieved. About 20% of abstracts labeled as Slovak turned out to be in English.
  • Final filters: Abstracts were kept at 500–2000 characters, with 4–15 keyphrases per document and at most four words per keyphrase.
  • Baseline results: The best unsupervised baseline reaches 11.6% exact-match F1@6, compared with up to 51.5% under partial matching, a gap of about 40 points.
  • KeyLLM setup: KeyLLM was tested with and without embedding clustering (kinit/slovakbert-sts-stsb, thresholds 0.75/0.85/0.90). This is its first application to Slovak.
  • Manual evaluation: Human evaluation of 100 documents (κ = 0.61) found morphological mismatch to be the main error for extractive methods. KeyLLM's main weakness was unmotivated adjective extraction.
  • Availability: The dataset is published on Hugging Face (NaiveNeuron/SlovKE), and the evaluation code is on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/322dk2ztk5dj

@inproceedings{stevanak2026slovke,
  title = {{SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction}},
  author = {Števaňák, Dávid and Suppa, Marek},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/322dk2ztk5dj},
  url = {https://doi.org/10.63317/322dk2ztk5dj}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Rabah Alzaidy, Cornelia Caragea, and C Lee Giles. 2019. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The world wide web conference, pages 2551– 2557. find
  3. Marco Basaldella, Elisa Antolli, Giuseppe Serra, and Carlo Tasso. 2018. Bidirectional lstm recurrent neural network for keyphrase extraction. In Italian Research Conference on Digital Libraries, pages 180–187. Springer. find
  4. Slobodan Beliga and Sanda Martincic-Ipsic. 2016. Network-enabled keyword extraction for under-resourced languages. In Semantic Keyword-Based Search on Structured Data Sources - COST Action IC1302 Second International KEY-STONE Conference, IKC 2016, Cluj-Napoca, Romania, September 8-9, 2016, Revised… find
  5. Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509:257–289. find
  6. Martin Docekal and Pavel Smrz. 2022. Query-based keyphrase extraction from long documents. arXiv preprint arXiv:2205.05391. — [arXiv:2205.05391](https://arxiv.org/abs/2205.05391) find
  7. Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019. Kptimes: A large-scale dataset for keyphrase generation on news documents. arXiv preprint arXiv:1911.12559. find
  8. Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022. Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1233–1246, Seattle, Un… find
  9. Nikolaos Giarelis, Nikos Kanakaris, and Nikos Karacapilidis. 2021. A comparative assessment of state-of-the-art methods for multilingual unsupervised keyphrase extraction. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 635–645. Springer. find
  10. Maarten Grootendorst. 2020. Keybert: Minimal keyword extraction with bert. find
  11. Maarten Grootendorst. 2021. Keyllm: Keyphrase extraction using large language models. https://maartengr.github.io/KeyBERT/guides/keyllm.html. Accessed: 2024-03-04. ↗
  12. Kazi Saidul Hasan and Vincent Ng. 2014. Automatic keyphrase extraction: A survey of the state of the art. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1262–1273, Baltimore, Maryland. Association for Computational Linguistic… find our card
  13. Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 216–223. find our card
  14. Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010. SemEval-2010 task 5 : Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21– 26, Uppsala, Sweden. Association for Computational Linguistics. find our card
  15. Pruthwik Kulkarni, Krishnandu Singh, Manish Kulkarni, and Manish Shrivastava. 2022. MAKED: A multi-lingual automatic keyword extraction dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), pages 3614–3623, Marseille, France. European Language Resources… find
  16. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  17. Lingua. 2021. Lingua: Language detection library. https://github.com/lingua-project/lingua. Accessed: 2024-03-04. ↗
  18. Ling Liu. 2009. Encyclopedia of database systems. find
  19. Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. arXiv preprint arXiv:1704.06879. find our card
  20. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. find
  21. Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In International conference on Asian digital libraries, pages 317–326. Springer. find
  22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. find our card
  23. Eirini Papagiannopoulou and Grigorios Tsoumakas. 2020. A review of keyphrase extraction. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(2):e1339. find
  24. Seoyeon Park and Cornelia Caragea. 2020. Scientific keyphrase identification and classification by pre-trained language models intermediate task transfer learning. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 5409–5419, Barcelona, Spain (Online). I… find
  25. Piotr P ezik, Agnieszka Mikołajczyk, Adam Wawrzyński, Bartłomiej Nitoń, and Maciej Ogrodniczuk. 2022. Keyword extraction from short texts with a text-to-text transfer transformer. In Asian Conference on Intelligent Information and Database Systems, pages 530–542. Springer. find
  26. Matúš Pikuliak, Štefan Grivalský, Martin Konôpka, Miroslav Blšták, Martin Tamajka, Viktor Bachratý, Marian Simko, Pavol Balážik, Michal Trnka, and Filip Uhlárik. 2022. SlovakBERT: Slovak masked language model. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7156–7168,… find
  27. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082. find our card
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  29. Ian Rogers. 2002. The Google PageRank algorithm and how it works. https://www.cs.princeton.edu/~chazelle/courses/BIB/pagerank.htm. ↗
  30. Dhruva Sahrawat, Debanjan Mahata, Haimin Zhang, Mayank Kulkarni, Agniv Sharma, Rakesh Gosangi, Amanda Stent, Yaman Kumar, Rajiv Shah, and Roger Zimmermann. 2020. Keyphrase extraction as sequence labeling using contextualized embeddings. In Advances in Information Retrieval: 42nd European Conference… find
  31. Olivier Salaün, Frédéric Piedboeuf, Guillaume Le Berre, David Alfonso-Hermelo, and Philippe Langlais. 2024. EUROPA: A legal multilingual keyphrase generation dataset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12718–12736… find
  32. Mingyang Song, Yi Feng, and Liping Jing. 2023. A survey on recent advances in keyphrase extraction from pre-trained language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2108–2119, Dubrovnik, Croatia. Association for Computational Linguistics. find
  33. Ayla Rigouts Terryn, Véronique Hoste, and Els Lefever. 2020. In no uncertain terms: A dataset for monolingual and multilingual automatic term extraction from comparable corpora. Language Resources and Evaluation, 54(2):385–418. find
  34. Reut Tsarfaty, Djamé Seddah, Sandra Kübler, and Joakim Nivre. 2013. Parsing morphologically rich languages: Introduction to the special issue. Computational Linguistics, 39(1):15–22. find
  35. Dávid Varga, Simon Horvát, Zoltán Szoplák, L’ubomír Antoni, Stanislav Krajci, Peter Gurský, and Laura Bachnáková Rózenfeldová. 2022. Keyphrase extraction from slovak court decisions. In Proceedings of the 22nd Conference Information Technologies - Applications and Theory (ITAT 2022), volume 3226 of… find
  36. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC‘00), Athens, Greece. European Language Resources Association (ELRA). find our card
  37. Di Wu, Wasi Uddin Ahmad, and Kai-Wei Chang. 2022. Pre-trained language models for keyphrase generation: A thorough empirical study. arXiv preprint arXiv:2212.10233. — [arXiv:2212.10233](https://arxiv.org/abs/2212.10233) find
  38. Andrej Zelinka. 2023. Keyphrase extraction from slovak scientific documents. Bachelor’s thesis, Comenius University, Bratislava, Bratislava, Slovakia. Available at https://opac.crzp.sk/?fn=detailBiblioForm&sid= C03CB0FBA6588971DD3A2BD1B6BF. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stevanak-2026-slovke-large-scale-dataset-llm/