Elamur Research⌕
☰
home / papers / 2026 / Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers

Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers

Paper · 2026 · Sergei Bagdasarov, Diego Alves, Stefan Fischer et al.

Summary

The Royal Society Corpus (RSC) holds more than 40,000 contributions to the Royal Society's journals, from 1665 to 1996. It has rich metadata but no discipline label for each paper, which makes field-specific historical studies hard. The authors test whether open generative LLMs can fill this gap. Two annotators labelled 763 RSC texts, sampled evenly across all periods, with an 11-class scheme derived from the Royal Society's own taxonomy. Eight models from four families were compared on this set under zero-shot prompts of increasing detail and under few-shot variants. The three strongest models were then checked on the contemporary SciTex corpora, which use a different taxonomy, and applied to the whole RSC. At least two of the three models chose the same label for 98.3% of documents. Most disagreements fall in twentieth-century, interdisciplinary papers. A diachronic study shows disciplinary diversity slowly narrowing, with shifts around the Chemical Revolution and modern physics. The resulting labels are released publicly on GitHub.

Key points

  • The gold set has 763 RSC texts. Annotators agreed on 556 of them (72.87%), with overall Cohen's κ = 0.7. Their typical confusions were biology vs. earth sciences (fossil anatomy), chemistry/physics/engineering, and miscellaneous vs. humanities.
  • The scheme contains 11 classes: physics, chemistry, biology, medicine, mathematics, astronomy, earth sciences, computer science, engineering and technology, humanities, and a miscellaneous category. Miscellaneous covers non-scientific items such as obituaries and indexes. The cross-disciplinary class was dropped so that each paper gets a single label.
  • The eight models come from the Gemma, GPT-OSS, Llama and Qwen families. They were tested with three zero-shot prompts and with few-shot prompts giving one, two or three examples per discipline (11 to 33 examples).
    • The minimal prompt gives a 200-word snippet.
    • The extended prompt adds the title.
    • The detailed prompt adds a journal-editor role and a description of the data.
  • The top three models were validated on SciTex, which covers the 1970s to early 2000s and has interdisciplinary computer-science compositions. Papers on which all three models disagreed were examined with topic modeling.
  • Labels and paper metadata are published in the repository s-bagdasarov/RSC-Disciplines.

Interesting ideas

  • I1. Running several different LLMs over the same item and keeping a label when at least two agree yields large-scale annotation with a built-in confidence signal, without fine-tuning.
  • I2. Disagreement between models was treated as information rather than noise: the split cases turned out to mark genuinely mixed, interdisciplinary texts.
  • I3. Measuring human inter-annotator agreement on the same sample, and studying where the annotators clash, gives a realistic ceiling and a map of inherently fuzzy class boundaries.
  • I4. Prompts were ablated step by step (snippet only, then plus title, then plus role and data description), so the gain from each piece of context can be measured.
  • I5. A catch-all class for non-content documents such as obituaries and indexes stops the model from forcing junk into real categories.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Multi-teacher LLM tiers could label frames or speech acts this way, so that majority-agreed items become silver data and split votes are flagged for review.
  • I2 (learn): When witness parsers disagree on a sentence, we could log it as a possibly ambiguous construction and mine these cases for repair rules or targeted evaluation, instead of simply discarding them.
  • I3 (learn): Before scoring a parser or role labeller against gold data, we could estimate human agreement on a small subset so that we do not chase errors that humans also make.
  • I4 (learn): LLM teacher prompts for VerbNet role or frame annotation could be built in such controlled increments, to show which context actually helps.
  • I5 (world): Ontology or entity-typing steps could include an explicit "not applicable" bucket, so that boilerplate or meta-text does not pollute the world of the text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksLiterature and Literary Theory · 519 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksHistorical linguistics & language change · 249 works

Cite

Cite as Sergei Bagdasarov, Diego Alves, Stefan Fischer and Elke Teich. 2026. Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3j9wvu86v48t

@inproceedings{bagdasarov2026using,
  title = {{Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers}},
  author = {Sergei Bagdasarov, Diego Alves, Stefan Fischer and Elke Teich},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3j9wvu86v48t},
  url = {https://doi.org/10.63317/3j9wvu86v48t}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hassan H. M. Altarturi, Mohammed Saadoon, and Nor Badrul Anuar. 2023. Web Content Topic Modeling Using LDA and HTML Tags. PeerJ Computer Science, 9:e1459. ↗
  2. Won Choi, Hye-Min Yoon, Min-Ho Hyun, Hye-Jin Lee, Ju-Won Seol, Kyung-Doo Lee, et al. 2023. Building an Annotated Corpus for Automatic Metadata Extraction from Multilingual Journal Article References. PLoS ONE, 18(1):e0280637. ↗
  3. Stefania Degaetano-Ortlieb, Hans Kermes, Ekaterina Lapshinova-Koltunski, and Elke Teich. 2013. SciTex – A Diachronic Corpus for Analyzing the Development of Scientific Registers. In Paul Bennett, Martin Durrell, Sandra Scheible, and Richard J. Whitt, editors, New Methods in Historical Corpus Linguistics, volume 3, pages 93– 104. Narr. find
  4. Jennifer D’souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, and Diana Slawig. 2025. find
  5. The Germeval 2025 2nd LLMs4Subjects Shared Task Dataset. Online dataset. Data managers and curators: Jennifer D’Souza (Data manager), Sameer Sadruddin (Data curator). find
  6. Peter Fankhauser, Jörg Knappen, and Elke Teich. 2016. Topical Diversification over Time in the Royal Society Corpus. In Proceedings of Digital Humanities (DH’16), Krakow, Poland. find
  7. Stefan Fischer, Jörg Knappen, Katrin Menzel, and Elke Teich. 2020. The Royal Society Corpus 6.0: Providing 300+ Years of Scientific Writing for Humanistic Study. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 794–802, Marseille, France. European Language Resources Association. find
  8. GemmaTeam. 2025. Gemma 3 Technical Report. find
  9. Lijimol George and P. Sumathy. 2023. An Integrated Clustering and BERT Framework for Improved Topic Modeling. International Journal of Information Technology, 15:2187 – 2195. ↗
  10. Monika Glowacka-Musial. 2022. Applying Topic Modeling for Automated Creation of Descriptive Metadata for Digital Collections. Information Technology and Libraries. ↗
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan et al. 2024. The Llama 3 Herd of Models.
  12. Maarten Grootendorst. 2022. BERTopic: Neural Topic Modeling with a Class-based TF-IDF Procedure. arXiv preprint arXiv:2203.05794. ↗
  13. Clara Wan Ching Ho. 2025. UBFFM at the GermEval-2025 LLMs4Subjects Task: What if we take “You are an expert in subject indexing” In Proceedings of the 21st Con-seriously? ference on Natural Language Processing (KON-VENS 2025): Workshops, pages 471–478, Hannover, Germany. HsH Applied Academics. find
  14. Maximilian Kähler, Lisa Kluge, and Katja Konermann. 2025. DNB-AI-project at the GermEval-2025 LLMs4Subjects Task: KIFSPrompt - Knowledge-Injected Few-Shot Prompting. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 455–464, Hannover, Germany. HsH Applied Academics. find
  15. Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. 2023. ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification.
  16. Verena Lyding, Egon Stemle, Claudia Borghetti, Marco Brunello, Sara Castagnoli, Felice Dell’Orletta, Henrik Dittmann, Alessandro Lenci, and Vito Pirrelli. 2014. The PAISÀ Corpus of Italian Web Texts. In Proceedings of the 9th Web as Corpus Workshop (WaC-9), pages 36–43, Gothenburg, Sweden. Association for Computational Linguistics. find
  17. Katrin Menzel, Jörg Knappen, and Elke Teich. 2021. Generating Linguistically Relevant Metadata for the Royal Society Corpus. Research in Corpus Linguistics, Challenges in combining structured and unstructured data in corpus development (special issue), 9(1):1–18. ↗
  18. Ines Montani, Matthew Honnibal, Adriane Boyd, Sofie Van Landeghem, and Henning Peters. 2023. spaCy: Industrial-strength Natural Language Processing in Python. find
  19. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card.
  20. QwenTeam. 2025. Qwen3 Technical Report. find
  21. Lari Repo, Bryan Hashimoto, Antti Liimatta, Lotta Saario, Tanja Säily, Iiro L. I. Tiihonen, Mikko Tolonen, and Veronika Laippala. 2024. Towards Automatic Register Classification in Unrestricted Databases of Historical English. In Simon Coats and Veronika Laippala, editors, Linguistics across Disciplinary Borders: The March of Data, pages 97–126. Bloomsbury Publishing, London. ↗
  22. Aditi Roy and Saptarshi Ghosh. 2023. Automated Subject Identification using the Universal Decimal Classification: The ANN Approach. SRELS Journal of Information Management. ↗
  23. Parisa Shirali, Zahra Sarlak, and Ebrahim Ansari. 2025. Last Minute at the GermEval-2025 LLMs4Subjects Task: Few-Shot Contrastive Learning for Multilingual Multi-Label Classification. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 465–470, Hannover, Germany. HsH Applied Academics. find
  24. Mayank Singh, Abhijeet Kumar, Sasidhar Dona-parthi, and Gayatri Karambelkar. 2025. Leveraging Retrieval Augmented Generative LLMs For Automated Metadata Description Generation to Enhance Data Catalogs. find
  25. Tanja Säily, Jukka Suomela, Florent Perek, Jimena Jiménez Real, and Turo Vartiainen. 2025. Using Large Language Models to Enrich Corpus Metadata: The Case of Novels in COHA. In 46th Annual Conference of the International Computer Archive of Modern and Medieval English (ICAME 46), Vilnius, Lithuania. find
  26. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. A. Prompts find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bagdasarov-2026-llms-automatic-discipline-annotation-diachronic/