Elamur Research⌕
☰
home / papers / 2026 / Large Language Models as Oracles for Ontology Alignment

Large Language Models as Oracles for Ontology Alignment

Summary

Ontology alignment systems still struggle to produce highly reliable correspondences. Asking human experts to check doubtful mappings helps, but it becomes too costly for large ontologies. The authors test whether an LLM can take over that verification role. They extend the matcher LogMap so that only the mappings it flags as uncertain (M_ask) go to an LLM-based Oracle. The Oracle answers a True/False question built from ontology-driven prompts. These prompts vary in phrasing style, amount of surrounding context such as parent classes, and whether synonyms are included.

The evaluation covers nine tasks from the OAEI anatomy, largebio and bio-ml datasets. It compares 30 LLM–prompt combinations, measuring diagnostic quality with Sensitivity, Specificity and Youden's index. The full pipeline is compared with automatic LogMap and with simulated Oracles at varying error rates. LLM Oracles improve LogMap's alignments and placed second overall in the OAEI 2025 bio-ml track, though they remain clearly below a perfect Oracle. Because the LLM is queried only for hard cases, the approach keeps computational and financial costs modest.

Key points

  • The LLM is invoked only for uncertain LogMap mappings. Prompt templates are built from three features: natural-language-friendly phrasing (NLF), extended context (EC) and synonyms (S).
  • The experiments combine five LLMs (GPT-4o Mini and several Google Gemini Flash models) with six prompt templates. Open-weight Mistral, Llama and Qwen were additionally tested on anatomy.
  • Benchmarks are nine OAEI matching tasks from the anatomy, largebio and bio-ml datasets.
  • LogMap combined with an LLM Oracle (LogMapLLM) ranked top-2 overall in the OAEI 2025 bio-ml track.
  • The code is released on GitHub (city-artificial-intelligence/logmap-llm, rai-ukraine-kga-llm) and Zenodo. The authors acknowledge several limitations:
    • no study with human domain experts;
    • possible training-data leakage;
    • a focus on equivalence mappings only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Language models & pretraining Knowledge graphs & ontologies

 Institutions

🎓Ukrainian Catholic UniversityLviv · Ukraine2 works · 10 citations

Also on the byline: City St George's, University of London.

Cite

Cite as Sviatoslav Lushnei, Dmytro Shumskyi, Severyn Shykula, Ernesto Jiménez-Ruiz and Artur d'Avila Garcez. 2026. Large Language Models as Oracles for Ontology Alignment. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.110

@inproceedings{lushnei2026large,
  title = {{Large Language Models as Oracles for Ontology Alignment}},
  author = {Sviatoslav Lushnei, Dmytro Shumskyi, Severyn Shykula, Ernesto Jiménez-Ruiz and Artur d'Avila Garcez},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.110},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.110}
}

References

49 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reham Alharbi, Jacopo de Berardinis, Paul Groth, Albert Meroño-Peñuela, Elena Simperl, and Valentina Tamma. 2024. find
  2. Harmonising Generative AI and Semantic Web Technologies (HGAIS 2024). In Special Session co-located with the 23rd International Semantic Web Conference (ISWC 2024), volume 3953 of CEUR Workshop Proceedings. find
  3. Reihaneh Amini, Sanaz Saki Norouzi, Pascal Hitzler, and Reza Amini. 2024. Towards Complex Ontology Alignment Using Large Language Models. In 6th Conf. on Knowledge Graphs and Semantic Web (KGSWC), volume 15459 of LNCS, pages 17–31. find
  4. Amazon Bedrock. 2025a. Meta’s Llama in Amazon Bedrock. https://aws.amazon.com/ bedrock/meta/ (accessed Sept. 2025). ↗
  5. Amazon Bedrock. 2025b. Mistral AI in Amazon Bedrock. https://aws.amazon.com/ bedrock/mistral/ (accessed Sept. 2025). ↗
  6. Olivier Bodenreider. 2004. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic acids research. find
  7. Jiaoyan Chen, Pan Hu, Ernesto Jiménez-Ruiz, Ole Magnus Holter, Denvar Antonyrajah, and Ian Horrocks. 2021a. OWL2Vec*: Embedding of OWL ontologies. Machine Learning, 110(7):1813–1845. find
  8. Jiaoyan Chen, Ernesto Jiménez-Ruiz, Ian Horrocks, Denvar Antonyrajah, Ali Hadian, and Jaehun Lee. 2021b. Augmenting Ontology Alignment by Semantic Embedding and Distant Supervision. In 18th Extended Semantic Web Conference (ESWC), volume 12731 of LNCS, pages 392–408. find
  9. Xi Chen, Weiguo Xia, Ernesto Jiménez-Ruiz, and Valerie V. Cross. 2014. Extending an ontology alignment system with BioPortal: a preliminary analysis. In Posters & Demos Track within the 13th International Semantic Web Conference, volume 1272 of CEUR Workshop Proceedings, pages 313–316. find
  10. Stefan Dernbach, Khushbu Agarwal, Alejandro Zuniga, Michael Henry, and Sutanay Choudhury. 2024. GLaM: Fine-Tuning Large Language Models for Domain Knowledge Graph Alignment via Neighborhood Partitioning and Generative Subgraph Encoding. In Proceedings of the AAAI 2024 Spring Symposium Series, pages… find
  11. Zlatan Dragisic, Valentina Ivanova, Huanyu Li, and Patrick Lambrix. 2017. Experiences from the anatomy track in the ontology alignment evaluation initiative. J. Biomed. Semant., 8(1):56:1–56:28. find
  12. Jérôme Euzenat and Pavel Shvaiko. 2013. Ontology Matching, Second Edition. Springer. find
  13. Daniel Faria, Emanuel Santos, Booma Sowkarthiga Balasubramani, Marta C Silva, Francisco M Couto, and Catia Pesquita. 2025. AgreementMakerLight. Semantic Web, 16(2):SW–233304. find
  14. Daniel Faria, Marta Contreiras Silva, Pedro Cotovio, Lucas Ferraz, Laura Balbi, and Catia Pesquita. 2023. Results for Matcha and Matcha-DL in OAEI 2023. In Proceedings of the 18th International Workshop on Ontology Matching co-located with the 22nd International Semantic Web Conference, volume 3591… find
  15. Gemini. 2025a. Gemini API rate limits. https://ai.google.dev/gemini-api/ docs/rate-limits (accessed Aug. 2025). ↗
  16. Gemini. 2025b. Gemini APIs pricing. https://ai.google.dev/gemini-api/docs/ pricing (accessed Aug. 2025). ↗
  17. Gemini. 2025c. OpenAI’s SDK endpoint. https://ai.google.dev/gemini-api/ docs/openai (accessed Aug. 2025). ↗
  18. Yuan He, Jiaoyan Chen, Denvar Antonyrajah, and Ian Horrocks. 2022a. BERTMap: A BERT-Based Ontology Alignment System. In 36th AAAI Conference on Artificial Intelligence, pages 5684–5691. find
  19. Yuan He, Jiaoyan Chen, Hang Dong, and Ian Horrocks. 2023. Exploring large language models for ontology alignment. In Proceedings of the Posters, Demos and Industry Tracks co-located with 22nd International Semantic Web Conference (ISWC), volume 3632 of CEUR Workshop Proceedings. find
  20. Yuan He, Jiaoyan Chen, Hang Dong, Ernesto Jiménez-Ruiz, Ali Hadian, and Ian Horrocks. 2022b. Machine Learning-Friendly Biomedical Datasets for Equivalence and Subsumption Ontology Matching. In 21st International Semantic Web Conference (ISWC), volume 13489 of LNCS, pages 575–591. find
  21. Sven Hertling and Heiko Paulheim. 2023. OLaLa: Ontology Matching with Large Language Models. In Proceedings of the 12th Knowledge Capture Conference 2023 (K-CAP), pages 131–139. ACM. find
  22. Vivek Iyer, Arvind Agarwal, and Harshit Kumar. 2021. VeeAlign: Multifaceted Context Representation Using Dual Attention for Ontology Alignment. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10780–10792. find
  23. Ernesto Jiménez-Ruiz, Bernardo Cuenca Grau, Yujiao Zhou, and Ian Horrocks. 2012a. Large-scale interactive ontology matching: Algorithms and implementation. In 20th European Conference on Artificial Intelligence (ECAI), volume 242 of Front. Artif. Intell. Appl., pages 444–449. IOS Press. find
  24. Ernesto Jiménez-Ruiz and Bernardo Cuenca Grau. 2011. Logmap: Logic-based and scalable ontology matching. In 10th Int’l Semantic Web Conference (ISWC), volume 7031 of LNCS, pages 273–288. find
  25. Ernesto Jiménez-Ruiz, Bernardo Cuenca Grau, and Ian Horrocks. 2012b. Exploiting the UMLS metathesaurus in the ontology alignment evaluation initiative. In Proceedings of the 2nd International Workshop on Exploiting Large Knowledge Repositories, volume 882 of CEUR Workshop Proceedings. find
  26. Ernesto Jiménez-Ruiz, Bernardo Cuenca Grau, Ian Horrocks, and Rafael Berlanga Llavori. 2011. Logic-based assessment of the compatibility of UMLS ontology sources. J. Biomed. Semant., 2(S-1):S2. find
  27. Hamed Babaei Giglou, Jennifer D’Souza, Felix Engel, and Sören Auer. 2024. LLMs4OM: Matching Ontologies with Large Language Models. CoRR, abs/2404.10317. — [arXiv:2404.10317](https://arxiv.org/abs/2404.10317) find
  28. Ernesto Jiménez-Ruiz, Sviatoslav Lushnei, Dmytro Shumskyi, Severyn Shykula, and Artur d’Avila Garcez. 2025. LogMap Family welcomes LogMapLLM in the OAEI 2025. In Proceedings of the 20th International Workshop on Ontology Matching (OM 2025), pages 178–181. find
  29. Zhigang Hao, Wolfgang Mayer, Jingbo Xia, Guoliang Li, Li Qin, and Zaiwen Feng. 2023. Ontology alignment with semantic and structural embeddings. J. Web Semant., 78:100798. find
  30. Prodromos Kolyvakis, Alexandros Kalousis, and Dimitris Kiritsis. 2018. DeepAlignment: Unsupervised ontology matching with refined word vectors. In Proceedings of NAACL, pages 787–798. find
  31. Huanyu Li, Zlatan Dragisic, Daniel Faria, Valentina Ivanova, Ernesto Jiménez-Ruiz, Patrick Lambrix, and Catia Pesquita. 2019. User validation in ontology alignment: functional assessment and impact. Knowl. Eng. Rev., 34:e15. find
  32. Siyan Liu, Qinglong Tian, Yukun Liu, and Pengfei Li. 2024. Joint statistical inference for the area under the roc curve and youden index under a density ratio model. Mathematics, 12(13):2118. find
  33. Lam Nguyen, Erika I. Barcelos, Roger H. French, and Yinghui Wu. 2025. KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models. In 24th Int’l Semantic Web Conference (ISWC), volume 16140 of LNCS, pages 629–649. find
  34. Ikechukwu Nkisi-Orji, Nirmalie Wiratunga, Stewart Massie, Kit-Ying Hui, and Rachel Heaven. 2018. Ontology alignment based on word embedding and random forest classification. In ECML-PKDD, pages 557–572. Springer. find
  35. Sanaz Saki Norouzi, Mohammad Saeid Mahdavinejad, and Pascal Hitzler. 2023. Conversational ontology alignment with ChatGPT. In 18th Int’l Workshop on Ontology Matching co-located with the 22nd International Semantic Web Conference, volume 3591 of CEUR Workshop Proceedings, pages 61–66. find
  36. Natalya Fridman Noy, Nigam H. Shah, Patricia L. Whetzel, Benjamin Dai, Michael Dorf, Nicholas Griffith, Clément Jonquet, Daniel L. Rubin, Margaret-Anne D. Storey, Christopher G. Chute, and Mark A. Musen. 2009. BioPortal: ontologies and integrated data resources at the click of a mouse. Nucleic Acid… find
  37. OpenAI. 2025a. Chat Completions API. https://platform.openai.com/docs/ guides/text?api-mode=chat (accessed Aug. 2025). ↗
  38. OpenAI. 2025b. OpenAI’s GPT-4o mini API limits and pricing. https://platform.openai.com/docs/models/gpt-4o-mini (accessed Aug. 2025). ↗
  39. Samira Oulefki, Lamia Berkani, Ladjel Bellatreche, Nassim Boudjenah, and Aı̈cha Mokhtari. 2024. Results for BioGITOM in OAEI 2024. In 19th International Workshop on Ontology Matching co-located with the 23rd International Semantic Web Conference (ISWC), volume 3897 of CEUR Workshop Proceedings, page… find
  40. Mina Abd Nikooie Pour, Alsayed Algergawy, Florence Amardeilh, Reihaneh Amini, Omaima Fal-latah, Daniel Faria, Irini Fundulaki, Ian Harrow, Sven Hertling, Pascal Hitzler, Martin Huschka, Liliana Ibanescu, Ernesto Jiménez-Ruiz, Naouel Karam, Amir Laadhar, Patrick Lambrix, Huanyu Li, Ying Li, Franck M… find
  41. Mina Abd Nikooie Pour, Alsayed Algergawy, Eva Blomqvist, Patrice Buche, Jiaoyan Chen, Adrien Coulet, Julien Cufi, Hang Dong, Daniel Faria, Lucas Ferraz, Pedro Giesteira Cotovio, Yuan He, Sven Hertling, Ian Horrocks, Liliana Ibanescu, Sarika Jain, Ernesto Jiménez-Ruiz, Naouel Karam, Felix Kraus, and… find
  42. Mina Abd Nikooie Pour, Eva Blomqvist, Pedro Giesteira Cotovio, Adrien Coulet, Lucas Ferraz, Sven Hertling, Sarika Jain, Ernesto Jiménez-Ruiz, Felix Kraus, Patrick Lambrix, Huanyu Li, Ying Li, Xianhao Liu, Pierre Monnin, Heiko Paulheim, Catia Pesquita, Abhisek Sharma, Pavel Shvaiko, Marta Silva, and… find
  43. Zhangcheng Qiang, Weiqing Wang, and Kerry Taylor. 2024. Agent-OM: Leveraging Large Language Models for Ontology Matching. VLDB Endow-mentVol, V. 18, No. 3. find
  44. Kent A. Shefchek and 1 others. 2020. The Monarch Initiative in 2019: an integrative data and analytic platform connecting phenotypes to genotypes across species. Nucleic Acids Research. find
  45. Maria Taboada, Diego Martı́nez, Mohammed Arideh, and Rosa Mosquera. 2025. Ontology matching with Large Language Models and prioritized depth-first search. Inf. Fusion, 123:103254. find
  46. Marius-Horatiu Totoian, Anca Marginean, Philipp Blohm, and Mir Nawab Hussain. 2024. HybridOM: Ontology Matching using Hybrid Search. In 19th International Workshop on Ontology Matching co-located with the 23rd International Semantic Web Conference (ISWC), volume 3897 of CEUR Workshop Proceedings, pa… find
  47. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. our card
  48. W. J. Youden. 1950. Index for rating diagnostic tests. Cancer, 3(1):32–35. find
  49. Rui Zhang, Yixin Su, Bayu Distiawan Trisedya, Xiaoyan Zhao, Min Yang, Hong Cheng, and Jianzhong Qi. 2024. AutoAlign: Fully Automatic and Effective Knowledge Graph Alignment Enabled by Large Language Models. IEEE Trans. Knowl. Data Eng., 36(6):2357–2371. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lushnei-2026-large-language-models-as-oracles/