Elamur Research⌕
☰
home / papers / 2026 / Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

Paper · 2026 · Alberto Testoni and Iacer Calixto

Summary

This paper asks whether clinically irrelevant statements about a patient's sexual orientation or religion change how accurate and how well calibrated LLMs are on medical questions. The question matters because clinical systems may use model confidence to decide when to defer to a clinician.

The authors took 2,364 adult-patient vignettes from MedQA-USMLE-4-options. They built counterfactual versions by adding one templated sentence stating sexual orientation, religion, or both. Nine general-purpose and biomedical LLMs were tested. Uncertainty was estimated with semantic entropy over repeated sampled answers, and calibration was measured with the Brier score.

The main findings:

  • A "heterosexual" cue behaves almost like the unmodified question.
  • A "homosexual" cue consistently lowers accuracy and worsens calibration across models.
  • Combining orientation with religion often causes harms that are not simply the sum of the two separate effects.
  • A clinician-validated open-ended generation study showed the same degradation, so the effect is not limited to the multiple-choice format.

The authors argue that counterfactual calibration testing should become a routine robustness check before clinical deployment.

Key points

  • Data and identities tested: 2,364 questions were sampled from MedQA-USMLE-4-options. Identity sentences covered heterosexual/homosexual orientation, Catholic/Muslim/atheist affiliation, and orientation-plus-religion pairs.
  • Models: Llama-3.2-3B, Qwen3-4B, Bio-Medical-Llama-3-8B, Llama-3.1-8B, Qwen3-30B, Llama-3.1-70B, OpenBioLLM-70B, GPT-4.1-mini and GPT-5.1.
  • Uncertainty method: semantic entropy computed from K = 10 samples (top-p 0.9, T 0.7, answer options reshuffled each time). The resulting entropy was normalized by log 4, and confidence was defined as 1 minus that value. ECE and AUROC are reported in the appendix.
  • Size of the effects: in some identity conditions, the two 70B models showed relative Brier score increases of more than 25%. The limitations section mentions an accuracy drop of up to 7.2 percentage points.
  • Robustness checks and open-ended setup: results held when the identity was embedded in the vignette's opening sentence and when "gay" replaced "homosexual." The open-ended case study used GPT-5-mini to convert questions, cluster answers and judge correctness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏥Amsterdam University Medical CentersAmsterdam · Netherlands3 works🔬Amsterdam Public HealthAmsterdam · Netherlands2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Alberto Testoni and Iacer Calixto. 2026. Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.36

@inproceedings{testoni2026calibrated,
  title = {{Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA}},
  author = {Alberto Testoni and Iacer Calixto},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.36},
  url = {https://doi.org/10.18653/v1/2026.acl-short.36}
}

References

43 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306.
  2. Yaara Artsi, Vera Sorin, Benjamin S Glicksberg, Panagiotis Korfiatis, Girish N Nadkarni, and Eyal Klang. 2025. Large language models in real-world clinical workflows: a systematic review of applications and implementation. Frontiers in Digital Health, 7:1659134. find
  3. George Baker. 2023. Medqa-usmle-4-options. https://huggingface.co/datasets/GBaker/ MedQA-USMLE-4-options. ↗
  4. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  5. Nicola Luigi Bragazzi, Rola Khamisy-Farah, Manlio Converti, and Italian Working-Group on LGB-TIQ Mental Health. 2022. Ensuring equitable, inclusive and meaningful gender identity-and sexual orientation-related data collection in the healthcare sector: insights from a critical, pragmatic systematic r… find
  6. Glenn W Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1. find
  7. Riccardo Cantini, Nicola Gabriele, Alessio Orsino, and Domenico Talia. 2025. Is reasoning all you need? find
  8. Crystal T Chang, Neha Srivathsa, Charbel Bou-Khalil, Akshay Swaminathan, Mitchell R Lunn, Kavita Mishra, Sanmi Koyejo, and Roxana Daneshjou. 2025. Evaluating anti-lgbtqia+ medical bias in large language models. PLOS Digital Health, 4(9):e0001001. find
  9. Susan D Cochran, J Greer Sullivan, and Vickie M Mays. 2003. Prevalence of mental disorders, psychological distress, and mental health services use among lesbian, gay, and bisexual adults in the united states. Journal of consulting and clinical psychology, 71(1):53. find
  10. Patricia Hill Collins. 2019. Intersectionality as Critical Social Theory. Duke University Press. find
  11. ContactDoctor. 2024. Contactdoctor-bio-medical: A high-performance biomedical language model. https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B. ↗
  12. James M Dahlhamer, Adena M Galinsky, Sarah S Joestl, and Brian W Ward. 2016. Barriers to health care among adults identifying as sexual minorities: A us national study. American journal of public health, 106(6):1116–1122. find
  13. Krishnamurthy Dvijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, Laura Culp, Mayank Daswani, Jan Freyberg, Christopher Kelly, Atilla Kiraly, Timo Kohlberger, Scott McKinney, Basil Mustafa, Vivek Na… find
  14. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  15. Chris Grasso, Hilary Goldhammer, Russell J Brown, and BW Furness. 2020. Using sexual orientation and gender identity data in electronic health records to assess for disparities in preventive health screening services. International journal of medical informatics, 142:104245. find
  16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  17. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. find
  18. James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29– 36. find
  19. Micaela Hirsch, Marina Elichiry, Blas Radi, Tamara Quiroga, David Restrepo, Luciana Benotti, Veronica Xhardez, Jocelyn Dunstan, and Enzo Ferrante. 2026. Implicit bias in LLMs for transgender populations. arXiv preprint arXiv:2602.13253. — [arXiv:2602.13253](https://arxiv.org/abs/2602.13253) find
  20. Guido Ivetta, Marcos J Gomez, Sofía Martinelli, Pietro Palombini, M Emilia Echeveste, Nair Carolina Mazzeo, Beatriz Busaniche, and Luciana Benotti. 2025. HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America. In… find
  21. Yuelyu Ji, Wenhe Ma, Sonish Sivarajkumar, Hang Zhang, Eugene M Sadhu, Zhuochun Li, Xizhi Wu, Shyam Visweswaran, and Yanshan Wang. 2025. Mitigating the risk of health inequity exacerbated by large language models. npj Digital Medicine, 8(1):246. find
  22. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  23. Michael King, Joanna Semlyen, Sharon See Tai, Helen Killaspy, David Osborn, Dmitri Popelyuk, and Irwin Nazareth. 2008. A systematic review of mental disorder, suicide, and deliberate self harm in lesbian, gay and bisexual people. BMC psychiatry, 8(1):70. find
  24. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations. find
  25. Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. 2024. Uncertainty as a fairness measure. Journal of Artificial Intelligence Research, 81:307– 335. find
  26. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2024. find
  27. Kristine E Lynch, Patrick R Alba, Olga V Patterson, Benjamin Viernes, Gregorio Coronado, and Scott L DuVall. 2020. The utility of clinical notes for sexual minority health research. American Journal of Preventive Medicine, 59(5):755–763. find
  28. AI Meta. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI Blog. Retrieved December, 20:2024. find
  29. Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2025. Reading between the prompts: How stereotypes shape LLM’s implicit personalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20367–20400, Suzhou, China. Association for Computational…
  30. Ankit Pal and Malaikannan Sankarasubbu. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/ OpenBioLLM-Llama3-70B. ↗
  31. Jahan C. Penny-Dimri, Magdalena Bachmann, William R. Cooke, Sam Mathewlynn, Samuel Dock-ree, John Tolladay, Jannik Kossen, Lin Li, Yarin Gal, and Gabriel Davis Jones. 2025. Measuring large language model uncertainty in women’s health using semantic entropy and perplexity: a comparative study. The La… find
  32. Rezwana Rahman, Jennifer Lapum, and Nadia Prendergast. 2024. “treat me like a person”: Unveiling healthcare narratives of muslim women who wear islamic head coverings through a poststructural narrative study. Canadian Journal of Nursing Research, 56(4):377–387. find
  33. Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric J Topol. 2022. Ai in health and medicine. Nature medicine, 28(1):31–38. find
  34. Rajat Rawat, Hudson McBride, Rajarshi Ghosh, Dhiyaan Nirmal, Jong Moon, Dhruv Alamuri, Sean O’Brien, and Kevin Zhu. 2024. DiversityMedQA: A benchmark for assessing demographic biases in medical diagnosis using large language models. In Proceedings of the Third Workshop on NLP for Positive Impact, pa… find
  35. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguisti… find
  36. Atiquer Rahman Sarkar, Yao-Shun Chuang, Noman Mohammed, and Xiaoqian Jiang. 2024. De-identification is not enough: a comparison between de-identified and synthetic clinical notes. Scientific reports, 14(1):29669. find
  37. Christopher P Scheitle, Jacqui Frost, and Elaine Howard Ecklund. 2023. The association between religious discrimination and health: disaggregating by types of discrimination experiences, religious tradition, and forms of health. Journal for the scientific study of religion, 62(4):845–868. find
  38. Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. 2024. Evaluation and mitigation of cognitive biases in medical language models. npj Digital Medicine, 7(1):295. find
  39. Christian T Sinclair and Drew A Rosielle. 2020. Avoid stigmatizing language about atheist patients. Journal of Pain and Symptom Management, 60(6):e30. find
  40. Carl G Streed Jr, Chris Grasso, Sari L Reisner, and Kenneth H Mayer. 2020. Sexual orientation and gender identity data collection: clinical and public health importance. American Journal of Public Health, 110(7):991–993. find
  41. Alberto Testoni and Iacer Calixto. 2026. Mind the gap: Benchmarking LLM uncertainty and calibration with specialty-aware clinical QA and reasoning-based behavioural features. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Lon…
  42. Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems, 37:15356–15385. find
  43. Travis Zack, Eric Lehman, Mirac Suzgun, Jorge A. Rodriguez, Leo Anthony Celi, Judy Gichoya, Dan Jurafsky, Peter Szolovits, David W. Bates, Raja-Elie E. Abdulnour, Atul J. Butte, and Emily Alsentzer. 2024. Assessing the potential of gpt-4 to perpetuate racial and gender biases in health care: a model… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/testoni-2026-calibrated-not-everyone-how-sexual/