📄 Otter at MedExAct2026: Diverse Encoder Ensemble for Medical Decision Span Detection

Summary

The paper describes the Otter team's system for the MedExACT 2026 shared task. The task is to find spans in clinical discharge summaries that express medical decisions, using 9 categories from the DICTUM taxonomy. Results are scored with a fairness-aware metric that combines overall performance with performance on the worst demographic subgroup. Discharge summaries are often longer than 512 tokens, and the labeled training set amounts to only several hundred documents, so the authors framed the problem as BIO tagging with sliding windows. Instead of tuning one model, they trained a pool of encoders that differ in three ways: backbone initialization, loss function, and training data. Greedy forward selection chose 10 of these models, and their outputs were combined by span-level majority voting. The ensemble reached a final test score of 58.09, compared with 52.7 for the best single model on validation. The main message is that model diversity, not the strength of each member, drives the gains: voting filters out errors made by individual models. The error analysis also shows that span boundaries and confusable categories remain open problems.

Key points

• Candidate pool. The backbones are BiomedBERT-base and GatorTron-base, plus DAPT variants of each. The DAPT variants were further pre-trained with entity-centric masking on 7,814 MIMIC-III discharge summaries. Each backbone was trained with cross-entropy and with focal loss.
• Data augmentation. Some models saw extra training data: synthetic notes from MedGemma-27B (two rounds, 104 and 152 documents) or silver-labeled MIMIC-III documents. Six of the 10 selected models use augmented data.
• Results. On test, the system scored 58.09 final, with Span F1 51.81 and Token F1 66.66. On validation, the ensemble scored 56.9. DAPT-GatorTron was the weakest individual model (48.6) but was still selected, which supports the diversity argument.
• Error analysis.
  • Two-thirds of false negatives (1,936 of 2,905) were missed by every model in the ensemble.
  • Among boundary errors, 62% involve only the start boundary, and 61% are spans predicted too short.
  • Defining Problem and Evaluating Test Result are frequently confused with each other.
  • Rare categories score poorly; Gathering Information reaches only 12.2 Span F1.
• Caveats.
  • The authors found 28 mislabeled Treatment Goal spans in one Hispanic-subgroup validation document. Because that subgroup has a single document, this distorts the worst-group score.
  • A BiomedBERT boundary refiner was used in the test submission, but its validation gain (73.79) was measured on the same data it was trained on, so it does not show how well it generalizes.
  • Code is available on GitHub.