Elamur Research⌕
☰
home / papers / 2026 / GoBlueInformatics at #SMM4H-HeaRD 2026: Long-Context Encoders and Generative Biomedical LLMs for Pathological TNM Stage Prediction

GoBlueInformatics at #SMM4H-HeaRD 2026: Long-Context Encoders and Generative Biomedical LLMs for Pathological TNM Stage Prediction

Paper · 2026 · Shangqing Wei

Summary

This shared-task paper addresses SMM4H-HeaRD 2026 Task 6. The task is to assign the T (T1–T4), N (N0–N3) and M (M0–M1) components of pathological TNM stage to de-identified TCGA pathology reports. Two things make it hard: the documents are lengthy and varied and often only imply the stage, and the M label is heavily skewed.

The author compared two families of systems. The discriminative ones were a chunked BioClinicalBERT ensemble, a single-pass BioClinical-ModernBERT-large ensemble with an 8,192-token window, and the BigBird-based BB-TEN classifiers. The generative one was OpenBioLLM-8B, tuned with LoRA to return a single JSON object of integer T, N and M values, with a regex fallback for parsing.

The ModernBERT ensemble was submitted to the first test set and scored 0.993 micro-F1 and 0.915 macro-F1, ahead of the BB-TEN baseline. On the harder second test set, the LLM extractor raised mean macro-F1 from 0.533 to 0.795. Overall, long-context encoders work well when evidence is explicit, while constrained generation helps on difficult reports. Rare T4 cases remain largely unsolved.

Key points

  • The training data has 6,774 TCGA reports with partial labels: 5,853 have T, 4,826 have N and 3,916 have M. Only the 2,418 fully labeled reports were used for LLM tuning, and these split into 2,257 M0 and 161 M1.
  • The LoRA setup used rank 16, alpha 32, a 4,096-token limit and 3 epochs on one RTX 5090 GPU. Decoding was greedy, with JSON parsing, a regex fallback and mode-based filling of missing fields.
  • On the hard Test set 2 (499 reports), macro-F1 improved over the organizer baseline:
    • T rose from 0.454 to 0.626.
    • N rose from 0.591 to 0.758.
    • M rose from 0.554 to 1.000.
  • On a 363-report internal held-out split, fine-tuned BB-TEN had the best mean F1 (0.864). OpenBioLLM-8B LoRA scored 0.849 but led on exact match (0.799). R1-Distill-8B LoRA improved only on N.
  • The main weakness is the T4 class, with a per-class F1 of just 0.091. Code and data are released on GitHub (Will-Wei7/GoBlueInformatics-at-SMM4H-HeaRD-2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Shangqing Wei. 2026. GoBlueInformatics at #SMM4H-HeaRD 2026: Long-Context Encoders and Generative Biomedical LLMs for Pathological TNM Stage Prediction. Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.smm4h-1.18

@inproceedings{wei2026goblueinformatics,
  title = {{GoBlueInformatics at \#SMM4H-HeaRD 2026: Long-Context Encoders and Generative Biomedical LLMs for Pathological TNM Stage Prediction}},
  author = {Shangqing Wei},
  booktitle = {Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks},
  year = {2026},
  doi = {10.18653/v1/2026.smm4h-1.18},
  url = {https://doi.org/10.18653/v1/2026.smm4h-1.18}
}

References

9 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78. Association for Computational Linguistics.
  2. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Preprint, arXiv:2501.12948.
  3. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  4. Jenna Kefeli, Jacob Berkowitz, Jose M. Acitores Cortina, Kevin K. Tsang, and Nicholas P. Tatonetti. 2024. Generalizable and automated classification of TNM stage from pathology reports with external validation. Nature Communications, 15:8916. find
  5. Jenna Kefeli and Nicholas P. Tatonetti. 2024. TCGA-reports: A machine-readable pathology report resource for benchmarking text-based AI models. Patterns, 5(3):100933. find
  6. Guillermo Lopez-Garcia, Jose Miguel Acitores Cortina, Jacob Berkowitz, Joey Chan, Ganesh Chandrasekar, Sumon Kanti Dey, Ivan Flores Amaro, Fernando Gallego, Lauren Gryboski, Ari Z. Klein, Martin Krallinger, Salvador Lima-López, Tomohiro Nishiyama, Lisa Raithel, Ahmad Rezaie Mianroodi, Roland Roller,… find
  7. Ankit Pal and Malaikannan Sankarasubbu. 2024. OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/ OpenBioLLM-Llama3-70B. ↗
  8. Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J. Pollard, Eric Lehman, Alistair E. W. Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. 2025. BioClinical ModernBERT: A state-of-the-art long-context encoder for biomedical and clinical NLP. Preprint, arXiv:250… — [arXiv:2506.10896](https://arxiv.org/abs/2506.10896) find
  9. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. In Advances in Neural Information Processing Systems.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2026-goblueinformatics-smm4h-heard-2026-long/