Elamur Research⌕
☰
home / papers / 2026 / Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature

Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature

Paper · 2026 · Maximilian Kahler, Katja Konermann, Lisa Kluge et al.

Summary

The German National Library (DNB) assigns subject terms from the Integrated Authority File (GND) to incoming publications. Because this label set is huge, sparse and heavily long-tailed, the authors treat automated indexing as Extreme Multi-Label Classification (XMLC). They compare three families of systems on German scientific textbooks. The first is six supervised XMLC algorithms (DiSMEC++, Omikuji, ZestXML, AttentionXML, XR-Transformer, NGAME). The second is a lexical matching baseline. The third is three in-house prototypes built on embeddings or LLMs (EBM, LLM-Ensemble, KI-FSPrompt). Systems receive either the title alone or the first 30,000 characters of the full text. Scoring combines binary matching against earlier manual indexing with graded relevance judgements from subject librarians on unseen documents. XR-Transformer, which uses transformer-based dense features, wins on the overall binary metrics. The LLM-based generative methods do better on graded relevance and on rare labels. The authors present this as the first XMLC benchmark on purely German data. Its practical lesson is that generative approaches complement supervised XMLC rather than replace it, which suggests ensembles.

Key points

  • The label space, GND-204K, holds 204,056 entities drawn from six categories: topical subject headings, place names, personal names, titles of works, conferences and corporate bodies. It contains every entity used at least once in the manually indexed records, out of roughly 1.4 million GND concepts in total.
  • There are two tasks:
    • Book-Titles: about 951K training documents covering 193,921 distinct labels.
    • Fulltext-30k: about 167K training documents covering 67,757 distinct labels.
  • Both tasks share a test set of 4,651 documents, split evenly across 20 subject groups. Development used separate dev-train and dev-test splits in a four-way hold-out design.
  • Each method in the expert evaluation was rated on its own disjoint subsample of 500–600 documents. KI-FSPrompt was left out of this rating because of resource limits.
  • The authors call KI-FSPrompt the most promising prototype. They advise practitioners to build production systems around the Annif framework and combine backends into ensembles.

Interesting ideas

  • I1. The label vocabulary was frozen before the data were split, so every split naturally contains rare labels missing from training; these "zero-shot labels" can then be measured as a separate group.
  • I2. The test set is stratified to give each of 20 subject groups equal weight, so domains with plenty of training data cannot dominate the score.
  • I3. Exact match against an existing gold standard is complemented by graded expert ratings on unseen data, which reveals suggestions that are useful but absent from the gold labels.
  • I4. Each annotator sees every record only once, because each method is judged on a non-overlapping subsample, which avoids carry-over between systems.
  • I5. KI-FSPrompt retrieves similar training examples to use as few-shot material in a prompt, so an untuned LLM still draws on supervised data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building evaluation sets for frames or WordNet senses, fix the inventory first and report test items with no training occurrence on their own, to expose long-tail failure.
  • I2 (answer): School-course and fable QA tests could be balanced by subject or genre, so per-domain weaknesses are visible rather than averaged away.
  • I3 (learn): Alongside LAS-style exact scores, a graded judgement tier (human or LLM teacher) could reward frame or role outputs that are plausible but differ from the silver data.
  • I4 (learn): When comparing parser or prag variants with human or LLM judges, giving each system its own disjoint items prevents the judge from anchoring on an earlier system's output.
  • I5 (frames): For rare VerbNet classes, retrieving treebank sentences with the same verb as prompt examples could help an LLM teacher label semantic roles more consistently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Deutsche NationalbibliothekFrankfurt am Main · Germany2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Maximilian Kahler, Katja Konermann, Lisa Kluge and Markus Schumacher. 2026. Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature. arXiv:2607.14882.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.14882 · arXiv 2607.14882

@misc{kahler2026does,
  title = {{Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature}},
  author = {Maximilian Kahler, Katja Konermann, Lisa Kluge and Markus Schumacher},
  year = {2026},
  doi = {10.48550/arxiv.2607.14882},
  eprint = {2607.14882},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.14882}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohit Babbar and Bernhard Schölkopf. 2017. DiSMEC: Distributed sparse machines for extreme multi-label classification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 721–729. Association for Computing Machinery. ↗
  2. K Bhatia, K Dahiya, H Jain, P Kar, A Mittal, Y Prabhu, and M Varma. 2016. The extreme classification repository: Multi-label datasets and code. Dataset and benchmark repository. find
  3. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2019a. Extreme multi-label legal text classification: A case study in EU legislation. In Proceedings of the Natural Legal Language Processing Workshop 2019, pages 78–87. Association for Computational Linguistics. ↗
  4. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019b. Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322. Association for Computational Linguistics.
  5. Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. 2020. Taming pre-trained transformers for extreme multi-label text classification. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3163–3171. ACM. find
  6. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-embedding: https://www.dnb.de/ki-projekt https://www.nhr-verein.de/unsere-partner Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint. ↗
  7. Kunal Dahiya, Nilesh Gupta, Deepak Saini, Akshay Soni, Yajun Wang, Kushal Dave, Jian Jiao, K. Gururaj, Prasenjit Dey, Amit Singh, Deepesh Hada, Vidit Jain, Bhawna Paliwal, Anshul Mittal, Sonu Mehta, Ramachandran Ramjee, Sumeet Agarwal, Purushottam Kar, and Manik Varma. 2023. NGAME: Negative mining-aware mini-batching for extreme classification. WSDM 2023 - Proceedings of the 16th ACM International Conference on Web Search and Data Mining, pages 258–266. find
  8. Arpan Dasgupta, Siddhant Katyan, Shrutimoy Das, and Pawan Kumar. 2023. Review of extreme multilabel classification. arXiv preprint. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint.
  10. Jennifer D’Souza, Sameer Sadruddin, Maximilian Kähler, Andrea Salfinger, Luca Zaccagna, Francesca Incitti, Lauro Snidaro, and Osma Suominen. 2026. An extreme multi-label text classification (XMTC) library dataset: What if we took "use of practical AI in digital libraries" seriously? arXiv preprint.
  11. Nicolai Erbs, Iryna Gurevych, and Marc Rittberger. 2013. Bringing order to digital libraries: From keyphrase extraction to index term assignment. D-Lib Magazine, 19:1–16. ↗
  12. Eibe Frank, Gordon W Paynter, Ian H Witten, Carl Gutwin, and Craig G Nevill-Manning. 1999. Domain-specific keyphrase extraction. In Proceedings of the 16th International Joint Conference on Artificial Intelligence (IJCAI99), pages 668–673. Morgan Kaufmann. find
  13. Koraljka Golub. 2021. Automated subject indexing: An overview. Cataloging & Classification Quarterly, 59:702–719. ↗
  14. Nilesh Gupta, Sakina Bohra, Yashoteja Prabhu, Saurabh Purohit, and Manik Varma. 2021. Generalized zero-shot extreme multi-label learning. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 527–535. find
  15. Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016. Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 13-17-Augu, pages 935–944. ACM. find
  16. Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, and Manik Varma. 2023. Renee: End-to-end training of extreme classification models. In Proceedings of Machine Learning and Systems, volume 5, pages 646–665. Curan. find
  17. Jaana Kekäläinen and Kalervo Järvelin. 2002. Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53:1120–1129. ↗
  18. Sujay Khandagale, Han Xiao, and Rohit Babbar. 2020. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning, 109:2099– 2119. ↗
  19. Lisa Kluge and Maximilian Kähler. 2025. DNB-AI-Project at SemEval-2025 task 5: An LLM-Ensemble approach for automated subject indexing. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1118–1128. Association for Computational Linguistics. ↗
  20. Maximilian Kähler, Lisa Kluge, and Katja Konermann. 2025. DNB-AI-Project at the GermEval-2025 LLMs4Subjects task: Kifsprompt - knowledge-injected few-shot prompting. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 455–464. HsH Applied Academics. find
  21. Maximilian Kähler and Clemens Rietdorf. 2025. Embedding based matching for automated subject indexing. Code repository. find
  22. Olena Medelyan, Eibe Frank, and Ian H Witten. 2009. Human-competitive tagging using automatic keyphrase extraction. ACL and AFNLP, pages 6–7. find
  23. Olena Medelyan and Ian H. Witten. 2008. Domain-independent automatic keyphrase indexing with small training sets. Journal of the American Society for Information Science and Technology, 59:1026– 1040. ↗
  24. Maximillian Merrillees and Lan Du. 2021. Stratified sampling for extreme multi-label data. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12713 LNAI:334–345. find
  25. Diego Ortego, Marlon Rodríguez, Mario Almagro, Kunal Dahiya, David Jimenez, and Juan C SanMiguel. 2026. Large language models meet extreme multi-label classification: Scaling and multi-modal framework. Proceedings of the AAAI Conference on Artificial Intelligence, 40:24630–24638. ↗
  26. Christoph Poley, Sandro Uhlmann, Frank Busse, Jan-Helge Jacobs, Maximilian Kähler, Matthias Nagelschmidt, and Markus Schumacher. 2025. Automatic subject cataloguing at the german national library. LIBER Quarterly: The Journal of the Association of European Research Libraries, 35:1–29. ↗
  27. Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. find
  28. The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018, pages 993–1002. find
  29. Erik Schultheis and Rohit Babbar. 2021. Speeding-up one-vs-all training for extreme classification via smart initialization. arXiv preprint. ↗
  30. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. jina-embeddings-v3: Multilingual embeddings with task LoRA. arXiv preprint. ↗
  31. Osma Suominen. 2019. Annif: DIY automated subject indexing using multiple algorithms. LIBER Quarterly, 29. ↗
  32. Osma Suominen. 2021. Maui like lexical matching. Code repository and documentation. find
  33. Osma Suominen, Juho Inkinen, and Mona Lehtinen. 2025a. Annif at SemEval-2025 task 5: Traditional XMTC augmented by LLMs. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 2424–2431. Association for Computational Linguistics. ↗
  34. Osma Suominen, Juho Inkinen, and Mona Lehtinen. 2025b. Annif at the GermEval-2025 LLMs4Subjects task: Traditional XMTC augmented by efficient LLMs. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 447–454. HsH Applied Academics. ↗
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 2017-December:5999–6009.
  36. Tong Wei, Zhen Mao, Jiang-Xin Shi, Yu-Feng Li, and Min-Ling Zhang. 2022. A survey on extreme multi-label learning. arXiv preprint. ↗
  37. S N Wood. 2011. Fast stable restricted maximum likelihood and marginal likelihood estimation of semi-parametric generalized linear models. Journal of the Royal Statistical Society (B), 73:3–36. find
  38. Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, Shanfeng Zhu, and Shanghai Key. 2019. AttentionXML: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. NIPS’19: Proceedings of the 33rd International Conference on Neural Information Processing Systems. ↗
  39. Jiong Zhang, Wei cheng Chang, Hsiang fu Yu, and Inderjit S. Dhillon. 2021. Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. arXiv preprint. find
  40. Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42:89. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kahler-2026-generative-ai-supersede-supervised-xmlc/