Elamur Research⌕
☰
home / papers / 2026 / PEI at #SMM4H-HeaRD 2026: Enhancing Patient Metadata Detection via Hypothesis-Conditioned Classification and Paraphrase-Based Data Augmentation

PEI at #SMM4H-HeaRD 2026: Enhancing Patient Metadata Detection via Hypothesis-Conditioned Classification and Paraphrase-Based Data Augmentation

Summary

The Paul-Ehrlich-Institute team built a system for Task 5 of #SMM4H-HeaRD 2026. The task asks whether a sentence from a SARS-CoV-2 sequencing article contains patient metadata, such as demographics, symptoms, treatments, outcomes or location. The data are 22,147 English sentences drawn from 150 Lit-Covid papers, and only 13.3% of them are positive. The authors compare an encoder, BioM-BERT-PubMed-PMC-Large, with a generative LLM, Mistral-Nemo-Instruct-2407. They expanded the training data by having Qwen3-8B rewrite each deduplicated sentence once. Mistral was improved through prompt engineering, fine-tuning and prompt constraints derived from error analysis. BioM-BERT was recast as an NLI-style task that pairs each sentence with a hypothesis about patient information. Augmentation helped both models. The most restrictive hypothesis gave BioM-BERT its best validation F1 of 0.821, compared with 0.750 for the best Mistral run. On the hidden test set, the submitted model reached 0.786 F1, above both the mean and the median of participants. The results suggest that encoders still beat LLMs on imbalanced biomedical sentence classification, even when both get the same augmentation.

Key points

  • Augmentation: Qwen3-8B paraphrased 14,419 training and 2,134 validation sentences on one A100, which took about 4.9 and 0.7 days. The paraphrases kept their meaning closely (BioM-BERT cosine similarity 0.985) while changing the wording moderately (ROUGE-L 0.637).
  • Mistral-Nemo: F1 rose from 0.423 zero-shot to 0.713 after fine-tuning and 0.742 with paraphrased data. Prompts refined to curb false positives on section titles and location cues brought it to 0.750. A hypothesis-conditioned prompt scored slightly lower (0.710).
  • BioM-BERT: F1 went from 0.801 to 0.813 with augmentation. Hypotheses H1 and H2 scored 0.811 and 0.812, while H3, the most explicit one, reached 0.821 and produced fewer false negatives.
  • Test set: The submitted model scored P 0.797, R 0.774 and F1 0.786. This beats the mean (0.729) and median (0.754) of participating systems and the earlier 0.776 baseline of Klein et al. (2025).
  • Release: The best model is published on Hugging Face as pei-germany/biom-bert-smm4h2026-task5-patient-metadata.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing · Life sciences & medicine

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.smm4h-1.24

@inproceedings{zeidi2026pei,
  title = {{PEI at \#SMM4H-HeaRD 2026: Enhancing Patient Metadata Detection via Hypothesis-Conditioned Classification and Paraphrase-Based Data Augmentation}},
  author = {Zeidi, Farnaz and Christof, Roman and Zeidi, Farnoush and König, Renate and Childs, Liam},
  booktitle = {Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks},
  year = {2026},
  doi = {10.18653/v1/2026.smm4h-1.24},
  url = {https://doi.org/10.18653/v1/2026.smm4h-1.24}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sultan Alrowili and Vijay Shanker. 2021. BioM-Transformers: Building large biomedical language models with BERT, ALBERT and ELECTRA. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 221–227. Association for Computational Linguistics. ↗
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. ↗
  3. Zhiyuan Chen, Andrew S. Azman, Xinhua Chen, Junyi Zou, Yuyang Tian, Ruijia Sun, and Xiangyanyu Xu. 2022. Global landscape of SARS-CoV-2 genomic surveillance and data sharing. Nature Genetics, 54(4):499–507. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  5. Nathan D. Grubaugh, Jason T. Ladner, Philippe Lemey, Andrew Rambaut, Oliver G. Pybus, Edward C. Holmes, and Kristian G. Andersen. 2019. Tracking virus outbreaks in the twenty-first century. Nature Microbiology, 4(1):10–19. ↗
  6. Ari Z. Klein, Davy Weissenbacher, Karen O’Connor, Amir Elyaderani, Ivan Flores Amaro, Takeshi Onishi, Su Golder, Kaelen Spiegel, Matthew Scotch, and Graciela Gonzalez-Hernandez. 2025. Detection of patient metadata in published articles for genomic epidemiology using machine learning and large language models. medRxiv. ↗
  7. Guillermo Lopez-Garcia, Jose Miguel Acitores Cortina, Jacob Berkowitz, Joey Chan, Sumon Kanti Dey, Ivan Flores Amaro, Fernando Gallego, Lauren Gry-boski, Ari Z. Klein, Farnoush Zeidi Kolehparcheh, Martin Krallinger, Salvador Lima-López, Tomohiro Nishiyama, Lisa Raithel, Ahmad Rezaie Mianroodi, Amirali Rezaie Mianroodi, Roland Roller, Judith Rosell, Frank Rudzicz, and 9 others. 2026. Overview of the 11th social media mining for health (#smm4h) and health real-world data (heard) shared tasks at acl 2026. In Proceedings of the 11th Social Media Mining for Health (#SMM4H) and Health Real-World Data (HeaRD) Workshop and Shared Tasks. Association for Computational Linguistics. find
  8. Karen O’Connor, Davy Weissenbacher, Amir Elyaderani, Ebbing Lautenbach, Matthew Scotch, and Graciela Gonzalez-Hernandez. 2025. Patient-related metadata reported in sequencing studies of SARS-CoV-2: Protocol for a scoping review and bibliometric analysis. JMIR Research Protocols, 14(1):e58567. A.2 ↗
  9. Qwen Team. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. ↗ our card
  10. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005. The paraphrasing process aims to preserve the original meaning while varying wording and sentence structure. Examples are shown below: Example 1: The virus was originally introduced into Houston many times independently. Paraphrased: The virus was first introduced into Houston on multiple separate occasions. find
  11. Thang Ta, Abu Rahman, Lotfollah Najjar, and Alexander Gelbukh. 2024. ThangDLU at #SMM4H 2024: Encoder-decoder models for classifying text data on social disorders in children and adolescents. In Proceedings of the 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks, pages 1–4. ↗
  12. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275. Example 2: Study sample descriptions and sequencing results. Paraphrased: Examine the sample data and sequencing results. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zeidi-2026-pei-smm4h-heard-2026-enhancing/