Elamur Research⌕
☰
home / papers / 2026 / MedMind AI at #SMM4H-HeaRD 2026: Data Extraction and Generation Using Prompt Engineering and Structured Outputs (Tasks 1–6)

MedMind AI at #SMM4H-HeaRD 2026: Data Extraction and Generation Using Prompt Engineering and Structured Outputs (Tasks 1–6)

Paper · 2026 · Aatish Pradhan and Brian M. Habersberger

Summary

The MedMind AI team entered all six SMM4H-HeaRD 2026 shared tasks. Their aim was to test whether general-purpose LLMs, used without fine-tuning, can rival domain-tuned models on health text extraction and generation. The tasks were:

  • multilingual adverse drug event detection
  • insomnia evidence in MIMIC-III notes
  • flu vaccination status on Twitter
  • Dial2Note SOAP note generation
  • patient metadata detection
  • TNM staging from pathology reports

Each pipeline combined a system prompt, the input document, and a strictly enforced JSON schema, followed by rule-based mapping to final labels. Task-specific fixes included:

  • language-specific prompts for ADE annotation quirks
  • a regex pre-annotator whose draft the LLM corrects
  • injected tweet timestamps for temporal context
  • explicit metastasis-search directives to counter majority-class bias

The hybrid regex-LLM system ranked first overall on Task 2 (insomnia). Results beat participant means on most other tasks, but the system lagged on metadata detection. The authors also observe that BLEU-style metrics can penalize faithful notes when reference notes contain hallucinated details. The work shows that cheap, schema-constrained prompting is a viable alternative to fine-tuning.

Key points

  • Models and cost: gpt-5.4-mini ran at temperature 0.0 for most tasks, and gpt-5.4 was used for multilingual Task 1. Processing took about 1.2 s per document. Across 132,177 API requests the cost averaged roughly $0.0024 per sample.
  • Task 2 (best among all participants):
    • Subtask 1 F1: 0.8649 (participant mean 0.6805).
    • Subtask 2 label F1: 0.8764.
    • Subtask 2 exact-match span score: 0.7170 (participant mean 0.3129).
  • Other test results:
    • Task 1 macro F1: 0.6518 (participant mean 0.5465).
    • Task 3 subtasks: 0.8932 and 0.9326.
    • Task 6 Test Set 1: macro F1 around 0.996–0.998 for T, N and M.
    • Task 5: 0.5860, below the participant mean of 0.7290.
  • "Trap commitment bias": the authors define this as a schema forcing an early negative categorization, after which the model ignores later positive cues. Switching to a simple boolean output reduced the problem.
  • Prompt design lessons:
    • Removing few-shot examples cut hallucinations in Task 4 SOAP generation.
    • Many annotation conventions are dataset-specific, for example German phytomedicines being labeled as drugs.
    • Pipelines do not transfer to new tasks without manual redesign, and the approach depends on closed-weight models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

Cite

Cite as Aatish Pradhan and Brian M. Habersberger. 2026. MedMind AI at #SMM4H-HeaRD 2026: Data Extraction and Generation Using Prompt Engineering and Structured Outputs (Tasks 1–6). Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.smm4h-1.31

@inproceedings{pradhan2026medmind,
  title = {{MedMind AI at \#SMM4H-HeaRD 2026: Data Extraction and Generation Using Prompt Engineering and Structured Outputs (Tasks 1–6)}},
  author = {Aatish Pradhan and Brian M. Habersberger},
  booktitle = {Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks},
  year = {2026},
  doi = {10.18653/v1/2026.smm4h-1.31},
  url = {https://doi.org/10.18653/v1/2026.smm4h-1.31}
}

References

7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alistair Johnson, Tom Pollard, and Roger Mark. 2016. MIMIC-III Clinical Database. PhysioNet. Version 1.4. find
  2. J. Kefeli, J. Berkowitz, J. M. Acitores Cortina, and 1 others. 2024. Generalizable and automated classification of TNM stage from pathology reports with external validation. Nature Communications, 15:8916. find
  3. Ari Z. Klein, Davy Weissenbacher, Karen O’Connor, Amir Elyaderani, Ivan Flores Amaro, Takeshi Onishi, Su Golder, Kaelen Spiegel, Matthew Scotch, and Graciela Gonzalez-Hernandez. 2025. Detection of patient metadata in published articles for genomic epidemiology using machine learning and large langua… find
  4. Ahmad Rezaie Mianroodi, Amirali Rezaie, Niko Grisel Todorov, Cyril Rakovski, and Frank Rudzicz. 2025. Medsynth: Realistic, synthetic medical dialogue-note pairs. Preprint, arXiv:2508.01401. — [arXiv:2508.01401](https://arxiv.org/abs/2508.01401) find
  5. Aatish Pradhan, Brian M. Habersberger, James H. Wade, Denver Dsouza, and Nihal Paul. 2025. LLM pros at SMM4H-HeaRD 2025 data extraction using prompt engineering and structured outputs (task 1, 2, 3, 4, 5, 6). In Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media.… find
  6. D. Xu, G. L. García, K. O’Connor, H. Holston, A. Z. Klein, I. F. Amaro, M. Scotch, and G. Gonzalez-Hernandez. 2024. Mining social media data for influenza vaccine effectiveness using a large language model and chain-of-thought prompting. In AMIA Annual Symposium Proceedings, pages 1404–1413. America… find
  7. Guillermo Lopez-Garcia, Jose Miguel Acitores Cortina, Jacob Berkowitz, Joey Chan, Ganesh Chandrasekar, Sumon Kanti Dey, Ivan Flores Amaro, Fernando Gallego, Lauren Gryboski, Ari Z Klein, Martin Krallinger, Salvador Lima-López, Tomohiro Nishiyama, Lisa Raithel, Ahmad Rezaie Mianroodi, Roland Roller,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pradhan-2026-medmind-ai-smm4h-heard-2026/