Elamur Research⌕
☰
home / papers / 2026 / Reference-Free Evaluation of Taxonomies

Reference-Free Evaluation of Taxonomies

Paper · 2026 · Pascal Wullschleger, Majid Zarharan, Donnacha Daly et al.

Summary

When taxonomies are generated or extended automatically, there is usually no gold taxonomy to compare them against. The standard triplet-based precision/recall/F1 evaluation therefore cannot be used in practice. The authors propose two metrics that need no reference.

  • Concept Similarity Correlation (CSC) measures robustness. It computes Kendall's τ between two quantities over all concept pairs: the cosine similarity of embedded concept descriptions, and the Wu & Palmer taxonomic similarity.
  • NLI Verification (NLIV) measures logical adequacy, meaning whether each child really is a subtype of its parent. A pre-trained NLI model scores each parent-child edge, using the child's description as premise and several Hearst-pattern "is-a" sentences as hypotheses. Edge scores are combined with a geometric mean along root-to-node walks, so errors high in the hierarchy are penalized more heavily.

Both metrics were tested on five taxonomies from different domains, using deliberately corrupted gold taxonomies. Their scores tracked the degree of damage and correlated well with reference-based F1, outperforming earlier metrics such as Semantic Proximity (SP) and RaTE. They also predicted downstream hierarchical classification performance. This makes it possible to compare candidate taxonomies when no gold standard exists, without any fine-tuning.

Key points

  • CSC targets a gap in SP. SP essentially treats groups of leaf siblings as clusters, so it is expected to miss misplaced non-leaf subtrees. CSC is designed to catch both leaf and non-leaf misclassifications.
  • NLIV comes in two variants. NLIV-S (strong) requires entailment, while NLIV-W (weak) only requires that the hypothesis is not contradicted. On SemEval-Food and MeSH, NLIV-S showed higher precision and NLIV-W higher recall. ROC-AUC was around 0.8 when distinguishing real edges from random pairs.
  • Robustness to synonyms. When concept names were swapped for WordNet synonyms, CSC rankings stayed stable. NLIV was more sensitive, with τ ranging roughly from 0.48 to 0.93 depending on the dataset, but still more stable than RaTE.
  • Backbone swaps preserve rankings. The default models were BART-L and MiniLM-L6. Replacing them with other NLI models (RoBERTa-L, DeBERTa-L, DeBERTa-ML) or embedding models (BGE-M3, Para-MPNet, ML-E5-L) kept taxonomy rankings largely consistent. Absolute scores, however, are not comparable across models or concept spaces.
  • Availability. Code is published at github.com/wullli/reference-free-taxonomy-eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations🎓National University of Ireland, MaynoothMaynooth · Ireland5 works · 10 citations

Also on the byline: University of Lucerne.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Pascal Wullschleger, Majid Zarharan, Donnacha Daly, Marc Pouly and Jennifer Foster. 2026. Reference-Free Evaluation of Taxonomies. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1273

@inproceedings{wullschleger2026reference,
  title = {{Reference-Free Evaluation of Taxonomies}},
  author = {Pascal Wullschleger, Majid Zarharan, Donnacha Daly, Marc Pouly and Jennifer Foster},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1273},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1273}
}

References

44 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024. Semantic sensitivities and inconsistent predictions: Measuring the fragility of NLI models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–444… find
  2. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The Semantic Web, pages 722–735, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  3. Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 task 13: Taxonomy extraction evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1081–1091, San Diego, California. Association for Computational Linguistics. find
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  5. Jiaoyan Chen, Freddy Lécué, Yuxia Geng, Jeff Z. Pan, and Huajun Chen. 2020. Ontology-guided Semantic Composition for Zero-shot Learning. In Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, pages 850–854. find
  6. Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. 2017. Gram: Graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, page 787–795, New York, NY,… find
  7. Christiane Fellbaum, editor. 2019. Wordnet. Language, Speech and Communication. Bradford Books, Cambridge, MA. find
  8. Yuxia Geng, Jiaoyan Chen, Zhuo Chen, Jeff Z. Pan, Zhiquan Ye, Zonggang Yuan, Yantao Jia, and Huajun Chen. 2021. Ontozsl: Ontology-enhanced zero-shot learning. In Proceedings of the Web Conference 2021, WWW ’21, page 3325–3336, New York, NY, USA. Association for Computing Machinery. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  10. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  11. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
  12. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 2005. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63–S63. find
  13. David Jurgens and Mohammad Taher Pilehvar. 2016. SemEval-2016 task 14: Semantic taxonomy enrichment. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1092–1102, San Diego, California. Association for Computational Linguistics. find
  14. Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura E. Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 364–371.
  15. Maxat Kulmanov, Wang Liu-Wei, Yuan Yan, and Robert Hoehndorf. 2019. El embeddings: Geometric construction of models for the description logic el++. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6103–6109. International Joint Conference… find
  16. Phillippe Langlais and Tianjian Lucas Gao. 2023. RaTE: a reproducible automatic taxonomy evaluation by filling the gap. In Proceedings of the 15th International Conference on Computational Semantics, pages 173–182, Nancy, France. Association for Computational Linguistics. find
  17. Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. 2024. Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert-nli. Political Analysis, 32(1):84–100. find
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  19. Carolyn E. Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265–266. Historical Article. find
  20. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. CoRR, abs/1908.08345.
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  22. Zichen Liu, Hongyuan Xu, Yanlong Wen, Ning Jiang, HaiYing Wu, and Xiaojie Yuan. 2021. TEMP: Taxonomy expansion with dynamic margin loss through taxonomy-paths. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3854–3863, Online and Punta Cana, Dominican… find
  23. Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018. Kame: Knowledge-based attention model for diagnosis prediction in healthcare. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, page 743–752, New York, NY, U… find
  24. Emaad Manzoor, Rui Li, Dhananjay Shrouty, and Jure Leskovec. 2020. Expanding taxonomies with implicit edge semantics. In Proceedings of The Web Conference 2020, WWW ’20, page 2044–2054, New York, NY, USA. Association for Computing Machinery. find
  25. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  26. Alina Petukhova and Nuno Fachada. 2023. Mn-ds: A multilabeled news dataset for news articles hierarchical classification. Data, 8(5). find
  27. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  28. Jiaming Shen, Zhihong Shen, Chenyan Xiong, Chi Wang, Kuansan Wang, and Jiawei Han. 2020. Taxoexpan: Self-supervised taxonomy expansion with position-enhanced graph neural network. In Proceedings of The Web Conference 2020, WWW ’20, page 486–497, New York, NY, USA. Association for Computing Machinery… find
  29. Jingchuan Shi, Hang Dong, Jiaoyan Chen, Zhe Wu, and Ian Horrocks. 2024. Taxonomy completion via implicit concept insertion. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 2159–2169, New York, NY, USA. Association for Computing Machinery. find
  30. Peter W Shor. 1995. A new proof of cayley’s formula for counting labeled trees. Journal of Combinatorial Theory, Series A, 71(1):154–158. find
  31. Michael Unterkalmsteiner and Waleed Abdeen. 2023. A compendium and evaluation of taxonomy quality attributes. Expert Systems, 40(1):e13098. find
  32. Paola Velardi, Stefano Faralli, and Roberto Navigli. 2013. OntoLearn reloaded: A graph-based algorithm for taxonomy induction. Computational Linguistics, 39(3):665–707. find
  33. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.
  34. Suyuchen Wang, Ruihui Zhao, Yefeng Zheng, and Bang Liu. 2022. Qen: Applicable taxonomy completion via evaluating full taxonomic relations. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 1008–1017, New York, NY, USA. Association for Computing Machinery. find
  35. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  36. Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd Annual Meeting on Association for Computational Linguistics, ACL ’94, page 133–138, USA. Association for Computational Linguistics. find
  37. Pascal Wullschleger, Simone Lionetti, Donnacha Daly, Francesca Volpe, and Grégoire Caro. 2022. Autoregressive self-attention models for diagnosis prediction on electronic health records. In 2022 IEEE International Conference on Big Data (Big Data), pages 1950–1956. find
  38. Hongyuan Xu, Ciyi Liu, Yuhang Niu, Yunong Chen, Xiangrui Cai, Yanlong Wen, and Xiaojie Yuan. 2023. TacoPrompt: A collaborative multi-task prompt learning method for self-supervised taxonomy completion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1… find
  39. Alessandro Zangari, Matteo Marcuzzo, Matteo Rizzo, Lorenzo Giudice, Andrea Albarelli, and Andrea Gasparetto. 2024. Hierarchical text classification and its foundations: A review of current research. Electronics, 13(7). find
  40. Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Zhenyu Wu, Shangbin Feng, and Meng Jiang. 2025. Code-Taxo: Enhancing taxonomy expansion with limited examples via code language prompts. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4131–4144, Vienna, Austria. Association for C… find
  41. Qingkai Zeng, Jinfeng Lin, Wenhao Yu, Jane Cleland-Huang, and Meng Jiang. 2021. Enhancing taxonomy completion with concept generation via fusing relational representations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 2104–2113, New York, NY, U… find
  42. Jieyu Zhang, Xiangchen Song, Ying Zeng, Jiaze Chen, Jiaming Shen, Yuning Mao, and Lei Li. 2021. Taxonomy completion via triplet matching network. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4662–4670. find
  43. Yazhe Zhang and Y. Z. Zhang. 2015. On the number of leaves in a random recursive tree. Brazilian Journal of Probability and Statistics, 29(4):897–908. find
  44. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wullschleger-2026-reference-free-evaluation-taxonomies/