OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
Paper · 2026 · Rian Touchent and Éric de la Clergerie
Summary
Medical coding means mapping clinical documents to codes such as CIM-10 diagnoses, CCAM procedures and ATC drugs. It depends on how codes relate to each other: hierarchy, causation, differential diagnosis and exclusions. Ordinary clinical text rarely states these links, so encoders pretrained on it never learn them. OntoBook generates training text from the ontologies instead. It samples biased random walks over the ontology graphs so that rare semantic edges are actually traversed. Qwen3-235B then rewrites each walk as French textbook-style prose without adding or dropping facts. Finally, the 149M-parameter ModernCamemBERT is trained on this text with masked language modeling plus a second objective that predicts the relation between code pairs, with both objectives using the same documents. On French coding benchmarks the approach beats MLM-only pretraining, and the gain is largest on Distemist-FR. Using shared data for both objectives turns out to be essential. The work shows that structured knowledge alone, without aligned real-world text, can yield useful pretraining data for a lower-resource medical language.
Key points
- Gains over MLM-only pretraining: +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist-FR. Cantemist-FR is the third benchmark.
- Feeding the two objectives different data instead of the same textbooks lowers scores by about 30 points.
- The released corpus has about 1.3M synthetic textbooks (~1.3B tokens): 402,328 from CIM-10, 762,958 from CCAM and 138,980 from ATC. Pretrained checkpoints are also released.
- The ontologies come from the ANS terminology server (SMT) in RDF/OWL format. Only CIM-10 has non-hierarchical edges: 1,317 causal, 5,739 exclusion and 341 manifestation edges.
- Walks are 8–20 steps long. CIM-10 uses five walk types: Etiologie, Diagnostic Différentiel, Codage Double, Syndrome and Cross-Chapter.
Institutions
France84 works · 25,414 citations🎓Sorbonne UniversitéParis · France32 works · 7,270 citationsFields, topics and concepts
📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works
🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 192 works
Cite
Cite as Rian Touchent and Éric de la Clergerie. 2026. OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining. Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26.
@inproceedings{touchent2026ontobook,
title = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
author = {Rian Touchent and Éric {de la Clergerie}},
booktitle = {Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26},
year = {2026},
doi = {10.63317/37ik5npnrvv6},
url = {https://doi.org/10.63317/37ik5npnrvv6}
}References
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/touchent-2026-ontobook-ontology-grounded-synthetic-textbooks/