Elamur Research⌕
☰
home / papers / 2026 / Policy-Sensitive Fairness Evaluation in Automated Scoring of Clinical Communication

Policy-Sensitive Fairness Evaluation in Automated Scoring of Clinical Communication

Paper · 2026 · Saed Rezayi, Le An Ha, Victoria Yaneva et al.

Summary

The paper asks whether an automated scorer of medical students' patient-communication skills treats demographic groups equally, and whether that answer changes depending on how much each error type is penalized. The setting is formative: the Communication Learning Assessment (CLA) tool, where learners record spoken replies to video patient cases. Four trained NBME annotators marked case-specific patient-centered behaviors (PCBs) in these replies. ACTA, a DeBERTa-based encoder model, detects the same behaviors.

The authors define an encounter-level loss that weights false negatives and false positives with adjustable coefficients. They then compare groups by gender and race using a disparity measure that is adjusted for case. FN:FP ratios range from 1:5 to 5:1.

With equal weights, no group differences reach statistical significance. Across the other ratios the picture also stays largely stable, apart from a few isolated exceptions. The authors argue that error weighting is a value judgment and should be stated explicitly. In formative tools this matters because missed credit can discourage learners, while unearned credit can hide gaps in their skills.

Key points

  • The data cover 1,383 records from 223 US 3rd- and 4th-year medical students across 8 clinical cases. A rubric defines 25 PCBs, with 1–4 per case. Annotators were blind to learner demographics.
  • ACTA reached an average binary F1 of 0.92 per PCB (range 0.86–0.97). Fairness was assessed on out-of-fold predictions from 5-fold cross-validation.
  • Human-annotated PCB prevalence was similar across the large groups, at roughly 59–63%. Small gender groups were reported only descriptively.
  • Uncertainty was estimated with 1,000 bootstrap resamples stratified by case, and Bonferroni correction was applied.
  • The only significant gaps appeared in the White/Caucasian vs Asian comparison at 2× and 3× FN weight. These gaps vanished at higher weights, and the authors treat them as a possible Type I error or a local effect.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Ho Chi Minh City University of Foreign Languages - Information Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksSocial Sciences · 1,772 worksSafety Research · 543 worksEducation · 203 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Saed Rezayi, Le An Ha, Victoria Yaneva, Polina Harik, Janet Mee and Jason Snyder. 2026. Policy-Sensitive Fairness Evaluation in Automated Scoring of Clinical Communication. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.40

@inproceedings{rezayi2026policy,
  title = {{Policy-Sensitive Fairness Evaluation in Automated Scoring of Clinical Communication}},
  author = {Saed Rezayi, Le An Ha, Victoria Yaneva, Polina Harik, Janet Mee and Jason Snyder},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.40},
  url = {https://doi.org/10.18653/v1/2026.bea-1.40}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nico Andersen, Julia Mang, Frank Goldhammer, and Fabian Zehner. 2025. Algorithmic fairness in automatic short answer scoring. International Journal of Artificial Intelligence in Education, pages 1–38. find
  2. Indrani Dey, Dana Gnesdilow, Rebecca Passonneau, and Sadhana Puntambekar. 2024. Potential pitfalls of false positives. In International Conference on Artificial Intelligence in Education, pages 469–476. Springer. find
  3. A Helen M Ernst, Anja Prinz-Weiß, Jörg Wittwer, and Thamar Voss. 2025. Discrepancy between performance and feedback affects mathematics student teachers’ self-efficacy but not their self-assessment accuracy. Frontiers in Psychology, 15:1391093. find
  4. Gerardo Flores, Alyssa Hasegawa Smith, Julia Fukuyama, and Ashia C. Wilson. 2025. Aligning evaluation with clinical priorities: Calibration, label shift, and error costs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  5. Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021. The (im) possibility of fairness: Different value systems require different mechanisms for fair decision making. Communications of the ACM, 64(4):136–143. find
  6. Matthew S Johnson, Xiang Liu, and Daniel F McCaffrey. 2022. Psychometric methods to evaluate measurement and algorithmic bias in automated scoring. Journal of Educational Measurement, 59(3):338– 361. find
  7. Matthew S Johnson and Daniel F McCaffrey. 2023. Evaluating fairness of automated scoring in educational measurement. Advancing natural language processing in educational assessment, 142. find
  8. Tiffany Wenting Li, Silas Hsu, Max Fowler, Zhilin Zhang, Craig Zilles, and Karrie Karahalios. 2023. Am I wrong, or is the Autograder wrong? Effects of AI grading mistakes on learning. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, pages 159–176. find
  9. Diane Litman, Haoran Zhang, Richard Correnti, Lindsay Clare Matsumura, and Elaine Wang. 2021. A fairness evaluation of automated methods for scoring text evidence usage in writing. In International Conference on Artificial Intelligence in Education, pages 255–267. Springer. find
  10. Anastassia Loukina, Nitin Madnani, and Klaus Zechner. 2019. The many dimensions of algorithmic fairness in educational applications. In Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications, pages 1–10. find
  11. Annotation Guideline 1. Annotate Complete Components. Ensure each annotation captures the complete text relevant to the label, even if the sentence is grammatically incomplete. 2. Annotate all instances of a concept or label. More variations of a label or concept will better train the automated scor… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rezayi-2026-policy-sensitive-fairness-evaluation-automated/