Elamur Research⌕
☰
home / papers / 2026 / From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs

From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs

Paper · 2026 · Jessica M. Lundin, Usman Nasir Nakakana and Guillaume Chabot-Couture

Summary

Hand-built medical multiple-choice benchmarks are fixed sets of questions. They can leak into training corpora, they report only overall scores, and they cannot show which parts of a given guideline were actually tested. The authors address this using the WHO IMCI handbook, an 80-page collection of flowcharts and checklists for managing childhood illness. PDF parsers and LLMs could not recover the handbook's colour-coded decision logic, so a pediatric clinician manually encoded it as a directed knowledge graph. The system walks every condition's links to produce MCQA items across five relationship types. Patient ages, templates and age-matched distractors are randomized, so new variants can be drawn at evaluation time and regenerated whenever guidelines change. Five language models were tested. They handled symptom recognition well but did worse on treatment protocols and clinical management. The approach matters because it offers guaranteed coverage of the guideline, resistance to verbatim contamination and fine-grained diagnostics. Its schema could also be reused for other guidelines built on explicit decision logic.

Key points

  • The graph has 200+ nodes and 300+ edges. Node types are Condition (31), Symptom (79), Treatment (84), FollowUp (15) and Severity (4). Edge types are INDICATES, TREAT, FOLLOW and TRIAGE.
  • Each of the five question types has its own set of four templates, which are used to generate the questions. Ages are drawn within each condition's valid range. Each item has three distractors sampled uniformly from candidates of the same type that suit the patient's age group.
  • 432 questions were generated automatically. Their validity rests on the expert-authored graph rather than on reviewing the outputs afterwards.
  • The best frontier models scored about 71–72%. Without a human baseline, it is hard to say whether this is a strong or weak result.
  • The main limitations are:
    • the graph was built by a single curator, so inter-rater reliability could not be measured;
    • only text-based MCQA is covered;
    • building the graph is still a manual step.
  • Code and data are available at github.com/jessicalundin/graph_testing_harness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jessica M. Lundin, Usman Nasir Nakakana and Guillaume Chabot-Couture. 2026. From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.evaleval-1.34

@inproceedings{lundin2026guidelines,
  title = {{From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs}},
  author = {Jessica M. Lundin, Usman Nasir Nakakana and Guillaume Chabot-Couture},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.34},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.34}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 155:102938. find
  2. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. Healthbench: Evaluating large language models towards improved human health. Preprint…
  3. Hang Dong, Jiaoyan Chen, Yuan He, and Ian Horrocks. 2023. Ontology enrichment from texts: A biomedical dataset for concept discovery and placement. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 5316–5320, New York, NY, USA. Associatio… find
  4. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of ICLR.
  5. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  6. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of EMNLP-IJCNLP, pages 2567–2577.
  7. Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations. Preprint, arXiv:2303.18027.
  8. Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2022. Frenchmedmcqa: A french multiple-choice question answering dataset for medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Informa… find
  9. Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Viknesh Sounderajah, Jonathan Carlson, Matthew P Lungren, Bay Gross, Peter Hames, Mustafa Suleyman, Dominic King, and Eric Horvitz. 2025. Sequential diagnosis with language models. Preprin… — [arXiv:2506.22405](https://arxiv.org/abs/2506.22405) find
  10. Tobi Olatunji, Abraham Owodunni, Tassallah Abdullahi, Ayokunmi Ilesanmi, Olalekan Obadun, Aimérou Ndiaye Etori, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael Best, and 1 others. 2024. Afrimed-qa: A pan-african, multi-specialty, medical question-answering benchmark dataset. arXiv preprint arX…
  11. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248–260. PMLR.
  12. Anka Reuel, Avijit Ghosh, Jenny Chim, Andrew Tran, Yanan Long, Jennifer Mickel, Usman Gohar, Srishti Yadav, Pawan Sasanka Ammanamanchi, Mowafak Allaham, Hossein A. Rahmani, Mubashara Akhtar, Felix Friedrich, Robert Scholz, Michael Alexander Riegler, Jan Batzner, and 1 others. 2025. Who evaluates AI’… — [arXiv:2511.05613](https://arxiv.org/abs/2511.05613) find
  13. Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts. 2022. Radqa: A question answering dataset to improve comprehension of radiology reports. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6250–6259, Marseille, France. European Language Resources Assoc… find
  14. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, and 6 others. 2024. Towards con… find
  15. David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, Florence, Italy. Association for Computational Linguistics.
  16. WHO. 2014. Integrated management of childhood illness - chart booklet. Technical report, World Health Organization. Technical document. find
  17. Sheng Zhang, Xin Zhang, Hui Wang, Jiajun Cheng, Pei Li, and Zhaoyun Ding. 2017. Chinese medical question answer matching using end-to-end character-level multi-scale cnns. Applied Sciences, 7(8). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lundin-2026-guidelines-guarantees-graph-based-evaluation/