Elamur Research⌕
☰
home / papers / 2026 / Large Language Models Are Good Term Extractors: A Systematic Evaluation

Large Language Models Are Good Term Extractors: A Systematic Evaluation

Summary

This study asks how well current large language models handle automatic term extraction (ATE), the task of finding domain-specific vocabulary in specialised texts. Earlier LLM work was narrow: mostly English, sentence-by-sentence input, few prompts, and no comparison of model size or reasoning. The author tests GPT-5 and Mistral-family models on the full multilingual ACTER dataset, which covers four domains in English, French and Dutch. Whole documents are fed to the models where the context window allows. The prompts are built on ACTER's own human annotation guidelines. Besides listing terms, the models assign a category label and a confidence score, and can add remarks useful for terminology management.

F1 ranges from .36 to .72. That is competitive with supervised systems and close to human inter-annotator agreement (.59). Larger models beat smaller ones, while reasoning variants help only slightly. Error analysis shows that many counted "errors" are defensible boundary choices or valid normalisations. The author therefore argues that, for high-resource languages, research attention should shift to subsequent applications like producing definitions and building ontologies.

Key points

  • Models: six configurations in total.
    • GPT-5-mini and GPT-5-large, each run with minimal and high reasoning effort.
    • Mistral-large-2411 and Magistral-medium-2507.
    • mistral-small-2506 and magistral-small-2507 were dropped because they followed instructions poorly.
  • Data: the full ACTER dataset, 12 corpora covering Corruption, Equitation, Heart Failure and Wind Energy in EN, FR and NL. Documents were processed whole, falling back to the original annotation splits or character-based splits only when necessary.
  • Output and pilot: models produce TSV with term, label (Specific / Common / Out-of-Domain Term, optionally Named Entity), confidence (50–100) and remarks. Pilot tests on Heart Failure EN/NL covered 144 settings per corpus, totalling 52,574 runs.
  • Results: F1 of .36–.72, against a human agreement ceiling of about .59 F1. Most apparent hallucinations turn out to be normalisations. Weak spots remain in fine-grained labelling and in overly general expressions.
  • Resources: code and prompts are published at github.com/AylaRT/llmate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/49gta5itne3q

@inproceedings{ayla2026large,
  title = {{Large Language Models Are Good Term Extractors: A Systematic Evaluation}},
  author = {Ayla, Rigouts Terryn,},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/49gta5itne3q},
  url = {https://doi.org/10.63317/49gta5itne3q}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikita Astrakhantsev, D. Fedorenko, and D. Yu. Turdakov. 2015. Methods for Automatic Term Recognition in Domain-specific Text Collections: A Survey. Programming and Computer Software, 41(6):336–349. find
  2. Hamed Babaei Giglou, Jennifer D’Souza, and Sören Auer. 2023. LLMs4OL: Large Language Models for Ontology Learning. In The Semantic Web – ISWC 2023, pages 408–427, Cham. Springer Nature Switzerland. find
  3. Shubhanker Banerjee, Bharathi Raja Chakravarthi, and John Philip McCrae. 2024. Large Language Models for Few-Shot Automatic Term Extraction. In Amon Rapp, Luigi Di Caro, Farid Meziane, and Vijayan Sugumaran, editors, Natural Language Processing and Information Systems. find
  4. NLDB 2024, volume 14762 of Lecture Notes in Computer Science, pages 137–150. Springer Nature Switzerland, Cham. find
  5. Julien Breton, Mokhtar Mokhtar Billami, Max Chevalier, Ha Thanh Nguyen, Ken Satoh, Cassia Trojahn, and May Myo Zin. 2025. Leveraging LLMs for legal terms extraction with limited annotated data. Artificial Intelligence and Law. find
  6. Yongchan Chun, Minhyuk Kim, Dongjun Kim, Chanjun Park, and Heuiseok Lim. 2025. Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9916–9926, Vienna, Austria. Association for Computation… find
  7. Orphée De Clercq, Marjan Van de Kauter, Els Lefever, and Veronique Hoste. 2015. LT3: Applying Hybrid Terminology Extraction to Aspect-Based Sentiment Analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 719–724, Denver, Colorado. Association for Com… find
  8. Patrick Drouin. 2003. Term Extraction Using Non-Technical Corpora as a Point of Leverage. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 9(1):99–115. find
  9. Katerina T. Frantzi and Sophia Ananiadou. 1999. The C-value/NC-value Domain-independent Method for Multi-word Term Extraction. Journal of Natural Language Processing, 6(3):145–179. find
  10. Amir Hazem, Merieme Bouhandi, Florian Boudin, and Beatrice Daille. 2022. Cross-lingual and Cross-domain Transfer Learning for Automatic Term Extraction from Low Resource Data. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 648–662, Marseille, France. Eu… find
  11. Barbara Heinisch. 2025. Large language models for terminology work: A question of the right prompt? Journal for Language Technology and Computational Linguistics, 38(2):13–30. find
  12. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst., 43(2):42:1–42:55. find our card
  13. Els Lefever. 2016. A Hybrid Approach to Domain-Independent Taxonomy Learning. Applied Ontology, 11(3):255–278. find
  14. Lieve Macken, Els Lefever, and Véronique Hoste. 2013. TExSIS: Bilingual Terminology Extraction from Parallel Corpora Using Chunk-based Alignment. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 19(1):1–30. find
  15. Małgorzata Marciniak, Piotr Rychlik, and Agnieszka Mykowiecka. 2023. TermoUD — a Language-independent Terminology Extraction Tool. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 178–186, Dubrovnik, Croatia. Association for Comput… find
  16. Małgorzata Marciniak, Piotr Rychlik, and Agnieszka Mykowiecka. 2025. Do transformer-based token classification methods solve the problem of terminology extraction? Natural Language Processing, pages 1–26. find
  17. Ayla Rigouts Terryn, Véronique Hoste, Patrick Drouin, and Els Lefever. 2020a. find
  18. TermEval 2020: Shared Task on Automatic Term Extraction Using the Annotated Corpora for Term Extraction Research (ACTER) Dataset. In Proceedings of the 6th International Workshop on Computational Terminology (COMPUTERM 2020), pages 85– 94, Marseille, France. European Language Resources Association. find
  19. Ayla Rigouts Terryn, Véronique Hoste, and Els Lefever. 2020b. In No Uncertain Terms: A Dataset for Monolingual and Multilingual Automatic Term Extraction from Comparable Corpora. Language Resources and Evaluation, 54(2):385– 418. find
  20. Ayla Rigouts Terryn, Véronique Hoste, and Els Lefever. 2022. Tagging Terms in Text: A Supervised Sequential Labelling Approach to Automatic Term Extraction. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 28(1). find
  21. Ayla Rigouts Terryn, Véronique Hoste, and Els Lefever. 2021. HAMLET: Hybrid Adaptable Machine Learning approach to Extract Terminology. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 27(2):40. find
  22. Antonio San Martín. 2024. What Generative Artificial Intelligence Means for Terminological Definitions. find
  23. Hanh Thi-Hong Tran, Carlos-Emiliano González-Gallardo, Antoine Doucet, and Senja Pollak. 2025. LlamATE: Automated terminology extraction using large-scale generative language models. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 31(1):5–36. find
  24. Hanh Thi Hong Tran, Carlos-Emiliano González-Gallardo, Julien Delaunay, Antoine Doucet, and Senja Pollak. 2024. Is Prompting What Term Extraction Needs? In Elmar Nöth, Aleš Horák, and Petr Sojka, editors, Text, Speech, and Dialogue, volume 15048, pages 17–29. Springer Nature Switzerland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/terryn-2026-large-language-models-are-good/