Elamur Research⌕
☰
home / papers / 2026 / Datasets for a Chatbot for Clinical Trial Search

Datasets for a Chatbot for Clinical Trial Search

Paper · 2026 · Yumeng Yang, Ethan Ludmir and Kirk Roberts

Summary

Finding suitable clinical trials is hard for patients because eligibility criteria are long, technical, and inconsistent across trials. The authors argue that a conversational pre-screening assistant can only be trusted if its components are built and tested on solid, domain-specific data. They therefore release three benchmark resources for cancer trials, though the methods are not cancer-specific.

The first resource is a gold standard of manually labelled criterion pairs. It was used to pick the best way to cluster 12,596 criteria into shared concepts. The second is a set of human-written questions, used to iteratively tune the prompts of the medical LLM MedGemma so its questions are clinically faithful and readable for patients. The third is a human-validated collection of criterion–question–answer tuples, used to test whether a model correctly judges eligibility from a patient's reply.

The main outcome is that SBERT embeddings with K-Means gave the best criterion grouping. More broadly, the work supplies reusable building blocks and baseline prompts for future trial-matching chatbots.

Key points

  • Source corpus: 500 trials were sampled, 100 each for breast, lung, prostate, colorectal and pancreatic cancer, giving 12,596 inclusion/exclusion criteria.
  • Clustering setup: Four embeddings (SBERT, USE, ClinicalTrialBERT, TF-IDF+SVD) were combined with K-Means and HDBSCAN at several settings, for 20 configurations. The text also mentions 30 variants at one point, which is inconsistent.
  • Pair annotation: Two annotators labelled 500 disputed criterion pairs, chosen because they were co-clustered in 5–15 of the runs. Agreement was Cohen's Kappa 0.742, with raw agreement of 0.915.
  • Clustering results: SBERT had the best average F1 (0.71). HDBSCAN favoured precision over recall. The top setup, SBERT + K-Means with k=200, reached F1 0.84 (P 0.94, R 0.76), although the setup description lists only k=50 and k=300.
  • Question generation and released prompts: MedGemma's questions were refined against the human benchmark using BERTScore and ROUGE-L. The prompts for question generation and answer assessment are released as baselines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas Health Science Center at HoustonHouston · United States7 works · 195 citations🏥The University of Texas MD Anderson Cancer CenterHouston · United States2 works

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Yumeng Yang, Ethan Ludmir and Kirk Roberts. 2026. Datasets for a Chatbot for Clinical Trial Search. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3pk9cracmxy7

@inproceedings{yang2026datasets,
  title = {{Datasets for a Chatbot for Clinical Trial Search}},
  author = {Yumeng Yang, Ethan Ludmir and Kirk Roberts},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3pk9cracmxy7},
  url = {https://doi.org/10.63317/3pk9cracmxy7}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the the 2nd Clinical Natural Language Processing Workshop. Association for Computational Linguistics.
  2. Corey W Arnold, Andrea Oh, Shawn Chen, and William Speier. 2016. Evaluating topic model interpretability from a primary care physician perspective. Computer methods and programs in biomedicine, 124:67–75. find
  3. Hala T. Borno, Li Zhang, Sylvia Zhang, Celia Kaplan, Alexander Bell, Brian Bakke, Amy Lin, Rahul Aggarwal, and Eric J. Small. 2022. Mobile Clinical Trial Matching Technology in Medical Oncology Clinic: A Pilot Feasibility Study. JCO Clinical Cancer Informatics, 6:e2100182. the Conference on Empirica… find
  4. Edward S. Kim, Suanna S. Bruinooge, Samantha Roberts, Gwynn Ison, Nancy U. Lin, Lia Gore, Thomas S. Uldrick, Stuart M. Lichtman, Nancy Roach, and Julia A. Beaver, et al. 2017. Broadening Eligibility Criteria to Make Clinical Trials More Representative: American Society of Clinical Oncology and Frien… find
  5. Felix Köpcke and Hans-Ulrich Prokosch. 2014. Employing Computers for the Recruitment into Clinical Trials: A Comprehensive Systematic Review. Journal of Medical Internet Research, 16(7):e161. find
  6. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4).
  7. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text summarization branches out, pages 74–81.
  8. Vivek H Murthy, Harlan M Krumholz, and Cary P Gross. 2004. Participation in Cancer Clinical Trials: Race-, Sex-, and Age-Based Disparities. JAMA, 291(22):2720–2726. find
  9. Ricardo J.G.B. Campello, Davoud Moulavi, and Joerg Sander. 2013. Density-Based Clustering Based on Hierarchical Density Estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 160–172. Springer. find
  10. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicoló Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, and Weishung Liu, et al. 2023. Can generalist foundation models outcompete special-purpose tuning? Case study in medicine. arXiv. find
  11. Martin R. Cowie, Juuso I. Blomster, Lesley H. Curtis, Sylvie Duclaux, Ian Ford, Fleur Fritz, Samantha Goldman, Salim Janmohamed, Jörg Kreuzer, and Mark Leenay, et al. 2017. Electronic health records to facilitate clinical research. Clinical Research in Cardiology, 106(1):1–9. find
  12. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, and Elahe Vedadi, et al. 2024. Capabilities of Gemini Models in Medicine. arXiv preprint arXiv:2404.18416.
  13. Betina Idnay, Caitlin Dreisbach, Chunhua Weng, and Rebecca Schnall. 2021. A systematic review on natural language processing systems for eligibility prescreening in clinical research. Journal of the American Medical Informatics Association, 29(1):197–206. find
  14. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, and Charles Lau, et al. 2025. MedGemma Technical Report. arXiv, 2507.05201.
  15. Tian Kang, Shaodian Zhang, Youlan Tang, Gregory W Hruby, Alexander Rusanov, Noémie Elhadad, and Chunhua Weng. 2017. EliIE: An open-source information extraction system for clinical trial eligibility criteria. In Proceedings of Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung… find
  16. Joseph M Unger, Elise Cook, Eric Tai, and Archie Bleyer. 2016. American Society of Clinical Oncology Educational Book, volume 36, chapter The Role of Clinical Trial Participation in Cancer Research: Barriers, Evidence, and Strategies. ASCO. find
  17. Chunhua Weng, Xiaoying Wu, Zhihui Luo, Mary Regina Boland, Dimitri Theodoratos, and Stephen B Johnson. 2011. EliXR: a system for extracting and encoding eligibility criteria in clinical trials. Journal of the American Medical Informatics Association, 18(Supplement 1). find
  18. Yumeng Yang, Ashley Gilliam, Ethan B Ludmir, and Kirk Roberts. 2024a. Exploring the Generalization of Cancer Clinical Trial Eligibility Classifiers across Diseases. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING… find
  19. Yumeng Yang, Soumya Jayaraj, Ethan Ludmir, and Kirk Roberts. 2024b. Text Classification of Cancer Clinical Trial Eligibility Criteria. In Proceedings of the AMIA Annual Symposium. find
  20. Chi Yuan, Patrick B Ryan, Casey Ta, Yixuan Guo, Ziran Li, Jill Hardin, Rupa Makadia, Peng Jin, Ning Shang, Tian Kang, and Chunhua Weng. Criteria2Query: a natural language interface to clinical databases for cohort definition. find
  21. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-datasets-chatbot-clinical-trial-search/