Elamur Research⌕
☰
home / papers / 2026 / Bhramastra at #SMM4H-HeaRD 2026: A Multi-Stage Hunter-Judge Pipeline using DSPy-Optimized LLMs for Multilingual ADE Detection

Bhramastra at #SMM4H-HeaRD 2026: A Multi-Stage Hunter-Judge Pipeline using DSPy-Optimized LLMs for Multilingual ADE Detection

Paper · 2026 · Bhaarat Pachori

Summary

This paper presents Team Bhramastra's system for #SMM4H-HeaRD 2026 Shared Task 1. The task is detecting personal Adverse Drug Events (ADEs) in social media posts across seven languages, one of which is a Farsi test set with no training data. The core difficulties are severe class imbalance and telling apart a symptom the drug was taken for (indication) from one the drug caused (reaction).

The system, called Hunter-Judge, has two stages. First, a fine-tuned mDeBERTa-v3-base model screens posts with recall as the priority. It is trained on the official data plus translated CADEC examples, using a weighted loss and per-language probability thresholds. Second, posts it flags go to a Gemini-2.5-Flash "Judge" that is optimized with DSPy and run through LangGraph. The Judge maps brand names to INN substances and retrieves evidence from FDA drug labels to decide whether the case is an indication or a reaction.

The pipeline had the best average F1 of all participating teams (0.6653) and transferred well to Farsi. This suggests that LLM adjudication grounded in medical evidence can help pharmacovigilance in low-resource languages.

Key points

  • Final test scores were an average F1 of 0.6653, compared with a task mean of 0.5465 and median of 0.5798. Validation F1 was 0.6772. Farsi reached 0.5863 zero-shot, about 21 points above the median.
  • Filtering funnel: of 42,736 test records, the Hunter flagged 11,753 (27.5%) and the Judge confirmed 1,555, removing roughly 96% of the input.
  • Used alone, the Hunter had recall of about 0.93 but precision of only 0.32. The Judge corrected 822 Hunter errors and introduced 87 new ones, a difference that is significant under McNemar's test (p < 0.001).
  • Thresholds were calibrated from percentiles of ADE probability scores on the training data. For example, the P5 threshold for Russian was 0.0089. Training used 47,547 training rows and 8,136 validation rows, with a class imbalance of about 1:12.67.
  • Japanese was the weakest language (F1 0.5631). Its errors came from the screening stage: posts the Hunter drops are never seen by the Judge. The authors also note added latency from relying on an LLM API.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksPsychology · 1,649 worksSocial Psychology · 330 worksPharmacology, Toxicology and Pharmaceutics · 7 worksToxicology · 6 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works

Cite

Cite as Bhaarat Pachori. 2026. Bhramastra at #SMM4H-HeaRD 2026: A Multi-Stage Hunter-Judge Pipeline using DSPy-Optimized LLMs for Multilingual ADE Detection. Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.smm4h-1.9

@inproceedings{pachori2026bhramastra,
  title = {{Bhramastra at \#SMM4H-HeaRD 2026: A Multi-Stage Hunter-Judge Pipeline using DSPy-Optimized LLMs for Multilingual ADE Detection}},
  author = {Bhaarat Pachori},
  booktitle = {Proceedings of the 11th Social Media Mining for Health Research and Applications (SMM4H-HeaRD 2026) Workshop and Shared Tasks},
  year = {2026},
  doi = {10.18653/v1/2026.smm4h-1.9},
  url = {https://doi.org/10.18653/v1/2026.smm4h-1.9}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Conference on Empirical Methods in Natural Language Processing.
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Thomas Henighan, Rewon Child, Aditya Quinn McNemar. 1947. Note on the sampling error of t… find
  3. Harsha Nori, Nicholas King, Scott Mayer McKinney, Robert Flavell, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. ArXiv, abs/2303.13375.
  4. Distribution Audit: Linguistic Uncertainty Signals Tokens identified during the linguistic gap analysis that correlated with lower model confidence across diverse scripts: 1. English (en): conversational noise and causal markers: ’your’, ’fuck’, ’thats’, ’can’, ’better’, ’user_____’, ’an’, ’humira’,… find
  5. Bhaarat Pachori. 2026. Agentic ai in pharmacovigilance: A position paper on opportunities, challenges, and implementation. Proceedings of the 19th International Joint Conference on Biomedical Engineering Systems and Technologies. find
  6. Abeed Sarker, Karen O’Connor, Rachel Ginn, Matthew Scotch, Karen Glen Smith, Daniel Malone, and Graciela Gonzalez. 2016. Social media mining for toxicovigilance: Automatic monitoring of prescription medication abuse from twitter. Drug Safety, 39:231 – 240. 2. Russian (ru): pharmaceutical terms and c… find
  7. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems. 3. German (de): fillers and treatment-specific tokens: ’ja’, ’hab’, ’pi’, ’grüße’, ’u’, ’eigentlich…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pachori-2026-bhramastra-smm4h-heard-2026-multi/