Elamur Research⌕
☰
home / papers / 2026 / A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction

A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction

Paper · 2026 · Marco Martinelli, Stefano Marchesin, Vanessa Bonato et al.

Summary

Biomedical information extraction (IE) resources tend to cover few entity or relation types, work at sentence level, and depend on noisy distant supervision. This is a particular problem for the fast-growing gut-brain axis literature, which links gut microbiota to neurological and psychiatric disorders. The authors present GutBrainIE, a benchmark of 1,647 PubMed titles and abstracts. Experts, trained lay annotators and automatic methods annotated the documents, and the annotations are grouped into quality tiers (Platinum, Gold, Silver, Bronze). The schema is unusually detailed. It has 13 entity types, 17 overloaded relation predicates and concept links to standard vocabularies. Annotators used GLiNER pre-annotations in the MetaTron platform. The benchmark defines four tasks: NER, NEL, mention-level RE (M-RE) and concept-level RE (C-RE). It also ships baselines, evaluation scripts and Codabench leaderboards with a hidden test set. In a public evaluation campaign, NER proved largely solvable even in this specialised domain, while RE remained clearly harder. GutBrainIE matters as a reproducible, demanding testbed for domains with specialised terminology and relations that span sentences.

Key points

  • Corpus construction: two PubMed query rounds in 2024 returned 1,662 documents. After removing duplicates and low-relevance items, 1,647 remained.
  • Schema: 13 entity types (including microbiome, dietary supplement and statistical technique) and 17 relation predicates, which combine into 55 distinct relation triples. Mentions are linked to 6 biomedical vocabularies, plus a custom ontology for terms that have no match.
  • Annotation team and support: 7 experts and 40 terminography master's students did the work. GLiNER pre-annotations (NuNER Zero checkpoint, temperature 0.8) reduced annotation time from about 30 minutes to 10–15 minutes per document.
  • Inter-annotator agreement: Fleiss' κ was 0.89 for NER and Cohen's κ was 0.88. RE agreement was much lower, at 0.43. An expert revision pass then corrected or removed 204 entities and 135 relations.
  • Evaluation campaign: the NER and M-RE tasks featured in an international campaign with 17 teams and nearly 200 submissions. Data and code are on Zenodo (record 16,845,409). Planned extensions include Alzheimer's and Multiple Sclerosis documents and manual revision of the Silver and Bronze tiers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PaduaPadua · Italy18 works · 4,260 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation extraction · 167 works

Cite

Cite as Marco Martinelli, Stefano Marchesin, Vanessa Bonato, Giorgio Di Nunzio, Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani and Gianmaria Silvello. 2026. A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.301

@inproceedings{martinelli2026domain,
  title = {{A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction}},
  author = {Marco Martinelli, Stefano Marchesin, Vanessa Bonato, Giorgio {Di Nunzio}, Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani and Gianmaria Silvello},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.301},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.301}
}

References

55 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khaled Alomar, Halil Ibrahim Aysel, and Xiaohao Cai. 2023. Data augmentation in classification and segmentation: A survey and new strategies. Journal of Imaging, 9(2):46. find
  2. Omar Alonso and Gary Marchionini. 2019. The Practice of Crowdsourcing. Morgan & Claypool Publishers. find
  3. Sultan Alrowili and Vijay Shanker. 2021. BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 221–227, Online. Association for Computational Linguistics. find
  4. Saadullah Amin, Katherine Ann Dunfield, Anna Vechkaeva, and Guenter Neumann. 2020. A data-driven approach for noise reduction in distantly supervised biomedical relation extraction. arXiv preprint arXiv:2005.12565. find
  5. Saadullah Amin, Pasquale Minervini, David Chang, Pontus Stenetorp, and Günter Neumann. 2022. Meddistant19: towards an accurate benchmark for broad-coverage biomedical relation extraction. arXiv preprint arXiv:2204.04779. find
  6. Jeremy Appleton. 2018. The gut-brain axis: influence of microbiota on mood and mental health. Integrative Medicine: A Clinician’s Journal, 17(4):28. find
  7. Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267– D270. find
  8. Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne Bernard. 2024. NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data. find
  9. Marilia Carabotti, Annunziata Scirocco, Maria Antonietta Maselli, and Carola Severi. 2015. The gut-brain axis: interactions between enteric microbiota, central and enteric nervous systems. Annals of gastroenterology: quarterly publication of the Hellenic Society of Gastroenterology, 28(2):203. find
  10. Yukun Chen, Thomas A Lasko, Qiaozhu Mei, Joshua C Denny, and Hua Xu. 2015. A study of active learning methods for named entity recognition in clinical text. Journal of biomedical informatics, 58:11–18. find
  11. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP… find
  12. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  13. Nigel Collier, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73–78.
  14. Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017. A neural network multi-task learning approach to biomedical named entity recognition. BMC bioinformatics, 18:1–14. find
  15. John F Cryan, Kenneth J O’Riordan, Kiran Sandhu, Veronica Peterson, and Timothy G Dinan. 2020. The gut microbiome in neurological disorders. The Lancet Neurology, 19(2):179–194. find
  16. Kartik Detroja, C.K. Bhensdadia, and Brijesh S. Bhatt. 2023. A survey on Relation Extraction. Intelligent Systems with Applications, 19:200244. find
  17. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  18. Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. 2019. Hypergraph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3558–3565. find
  19. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  20. Kuzman Ganchev, Fernando Pereira, Mark Mandel, Steven Carroll, and Peter White. 2007. Semi-automated named entity annotation. In Proceedings of the linguistic annotation workshop, pages 53–56. find
  21. Shivani Ghaisas, Joshua Maher, and Anumantha Kan-thasamy. 2016. Gut microbiome in health and disease: Linking the microbiome–gut–brain axis and environmental factors in the pathogenesis of systemic and neurodegenerative diseases. Pharmacology & therapeutics, 158:52–62. find
  22. Sreyan Ghosh, Utkarsh Tyagi, Manan Suri, Sonal Kumar, S Ramaneswaran, and Dinesh Manocha. 2023. Aclm: A selective-denoising based generative data augmentation approach for low-resource complex ner. arXiv preprint arXiv:2306.00928. find
  23. Robert Greinacher and Franziska Horn. 2018. The DAL-PHI annotation framework & how its pre-annotations can improve annotator efficiency. arXiv preprint arXiv:1808.05558. — [arXiv:1808.05558](https://arxiv.org/abs/1808.05558) find
  24. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23.
  25. Yue Guo and Yi Yang. 2024. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032. — [arXiv:2406.19032](https://arxiv.org/abs/2406.19032) find
  26. Gibong Hong, Veronica Hindle, Nadine M Veasley, Hannah D Holscher, and Halil Kilicoglu. 2025. DiMB-RE: mining the scientific literature for diet-microbiome associations. Journal of the American Medical Informatics Association, 32(6):998–1006. find
  27. Karim M Ibrahim, Elena V Epure, Geoffroy Peeters, and Gael Richard. 2020. Confidence-based weighted loss for multi-label classification with missing labels. In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 291–295. find
  28. Ornella Irrera, Stefano Marchesin, and Gianmaria Silvello. 2024. MetaTron: advancing biomedical annotation empowering relation annotation and collaboration. BMC bioinformatics, 25(1):112. find
  29. Rezarta Islamaj, Chih-Hsuan Wei, David Cissel, Nicholas Miliaras, Olga Printseva, Oleg Rodionov, Keiko Sekiya, Janice Ward, and Zhiyong Lu. 2021. NLM-Gene, a richly annotated gold standard dataset for gene entities that addresses ambiguity and multi-species gene recognition. Journal of biomedical in… find
  30. Peter D Karp. 2016. How much does curation cost? Database, 2016:baw110. find
  31. Jung-Jae Kim, Xu Han, Vivian K Lee, and Dietrich Rebholz Schuhmann. 2013. GRO Task: Populating the Gene Regulation Ontology with events and relations. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 50–57. find
  32. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015. The CHEMDNER corpus of chemicals and drugs and its annotation principles. Journal of cheminformatics, 7(Suppl 1):S2. find
  33. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. BioASQ-QA: A manually curated corpus for Biomedical Question Answering. Scientific Data, 10(1):170. find
  34. Sang Gyu Kwak and Jong Hae Kim. 2017. Central limit theorem: the cornerstone of modern statistics. Korean journal of anesthesiology, 70(2):144. find
  35. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746.
  36. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. find
  37. Database, 2016. knowledge graphs. Journal of Artificial Intelligence for Medical Sciences, 1(3):30–42. find
  38. Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. BioRED: a rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23(5):bbac282. find
  39. M Martinelli, G Silvello, V Bonato, GM Di Nunzio, N Ferro, O Irrera, S Marchesin, L Menotti, F Vezzani, et al. 2025. Overview of GutBrainIE@ CLEF 2025: Gut-Brain Interplay Information Extraction. In CLEF 2025 Working Notes, volume 4038, pages 65–98. find
  40. Marco Martinelli, Stefano Marchesin, and Gianmaria Silvello. 2026. Efficient and Reliable Estimation of Named Entity Linking Quality: A Case Study on GutBrainIE. find
  41. Marie Mikulová, Milan Straka, Jan Štěpánek, Barbora Štěpánková, and Jan Hajič. 2023. Quality and efficiency of manual annotation: Pre-annotation bias. arXiv preprint arXiv:2306.09307. find
  42. Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Martin Krallinger, Miguel Rodríguez-Ortega, Eduard Rodriguez-López, Natalia Loukachevitch, Andrey Sakhovskiy, Elena Tutubalina, Dimitris Dimitriadis, et al. 2025. Overview of bioasq 2025: The thirteenth bioasq challenge on large-scale bio… find
  43. Yesol Park, Gyujin Son, and Mina Rho. 2024. Biomedical flat and nested named entity recognition: Methods, challenges, and advances. APPLIED SCIENCES-BASEL, 14(20):1–23. find
  44. Sampo Pyysalo, Tomoko Ohta, and Sophia Ananiadou. 2013. Overview of the cancer genetics (cg) task of bionlp shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 58–66. find
  45. Peng Su, Gang Li, Cathy Wu, and K Vijay-Shanker. 2019. Using distant supervision to augment manually annotated data for relation extraction. PloS one, 14(7):e0216913. find
  46. Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022. Revisiting DocRED–Addressing the False Negative Problem in Relation Extraction. arXiv preprint arXiv:2205.12696.
  47. Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 165–174. find
  48. Bui Duc Tho, Minh-Tien Nguyen, Dung Tien Le, Lin-Lung Ying, Shumpei Inoue, and Tri-Thanh Nguyen. 2024. Improving biomedical Named Entity Recognition with additional external contexts. Journal of Biomedical Informatics, 156:104674. find
  49. Ting Liu, Xueli Pan, Xu Wang, K Anton Feenstra, Jaap Heringa, and Zhisheng Huang. 2021. Exploring the microbiota-gut-brain axis for mental disorders with Erik M Van Mulligen, Annie Fourrier-Reglat, David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A Kors, and Laura I Furlong. 2012.… find
  50. Ruili Wang, Feng Hou, Steven F Cahan, Li Chen, Xiaoyun Jia, and Wanting Ji. 2022. Fine-grained entity typing with a type taxonomy: a systematic review. IEEE Transactions on Knowledge and Data Engineering, 35(5):4794–4812. find
  51. Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35(10):1745–1752. find
  52. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777, Flor… find
  53. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining Language Models with Document Links. In Association for Computational Linguistics (ACL).
  54. Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. GLiNER: Generalist model for named entity recognition using bidirectional transformer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  55. Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14612–14620. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/martinelli-2026-domain-specific-curated-benchmark-entity/