Elamur Research⌕
☰
home / papers / 2026 / Innov8rs at CRF Filling 2026: An Iterative Multi-LLM Ensemble Pipeline with Dynamic Few-Shot Retrieval and Data-Driven Precision Filtering

Innov8rs at CRF Filling 2026: An Iterative Multi-LLM Ensemble Pipeline with Dynamic Few-Shot Retrieval and Data-Driven Precision Filtering

Paper · 2026 · Samminga Sainath Rao, Sumit Mishra and Chanchal Suman

Summary

In this system paper, team Innov8rs tackles Case Report Form (CRF) filling, the shared task organized at CL4Health 2026. The task requires predicting 134 structured items from English and Italian emergency-department notes about patients with dyspnea. Most items are never mentioned in a given note, so the gold answer is usually "unknown". LLMs tend to guess definite values, often "no", for absent information, which the authors call completion bias.

The authors built their pipeline step by step through error analysis. It has four parts:

  • a three-stage prompt redesign;
  • TF-IDF retrieval of three similar silver-annotated notes as few-shot examples;
  • schema-constrained normalization of the outputs;
  • an ensemble of Gemini 2.5 Flash and Llama 3.3 70B.

Their central addition is a post-hoc filter. It blocks predictions for items whose precision on the development data falls below a cutoff. On the English development set, this filter raised macro-F1 from 0.541 to 0.703.

On the test set, however, the system scored only 0.47, and the filter slightly hurt results. This suggests that item statistics calibrated on 80 patients do not transfer reliably.

Key points

  • The baseline prompt produced over 4,800 false positives on 80 dev patients. Explicitly making "unknown" the default answer cut false positives by about 80%.
  • The precision filter used a threshold of τ = 0.33, keeping 34 of 134 items. English dev false positives fell from 816 to 171, with true positives dropping from 352 to 305. On Italian dev, F1 rose from 0.500 to 0.683.
  • Dynamic few-shot retrieval worked over a silver corpus of 2,667 notes and was used for English only. For Italian, the silver IDs did not match the notes, so static examples were used instead.
  • Rate limits were handled by rotating across 11 Google and 2 Groq API keys. Even so, Gemini covered only 49 of the 200 test notes.
  • The best test macro-F1 was 0.47, from the "Conservative" ensemble, where Gemini acts mainly as a veto. Applying the dev-calibrated filter lowered the test score to 0.46.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Institute of Technology WarangalWarangal · India2 works · 1,029 citations🎓Rajiv Gandhi Institute of Petroleum TechnologyRaebareli · India3 works · 3 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation retrieval & search · 473 works

Cite

Cite as Samminga Sainath Rao, Sumit Mishra and Chanchal Suman. 2026. Innov8rs at CRF Filling 2026: An Iterative Multi-LLM Ensemble Pipeline with Dynamic Few-Shot Retrieval and Data-Driven Precision Filtering. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2eg7dfwaf4o5

@inproceedings{rao2026innov8rs,
  title = {{Innov8rs at CRF Filling 2026: An Iterative Multi-LLM Ensemble Pipeline with Dynamic Few-Shot Retrieval and Data-Driven Precision Filtering}},
  author = {Samminga Sainath Rao, Sumit Mishra and Chanchal Suman},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2eg7dfwaf4o5},
  url = {https://doi.org/10.63317/2eg7dfwaf4o5}
}

References

10 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Pietro Ferrazzi, Mattia Franzin, Alberto Lavelli, and Bernardo Magnini. 2026a. Small LLMs for Medical NLP: A Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian. find
  3. Pietro Ferrazzi, Soumitra Ghosh, Alberto Lavelli, and Bernardo Magnini. 2026b. Overview of the crf 2026 shared task on clinical case report forms filling. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health), Palma, Mallorca (Spain). ELRA. find
  4. Pietro Ferrazzi, Alberto Lavelli, and Bernardo Magnini. 2025. Converting Annotated Clinical Cases into Structured Case Report Forms. In Proceedings of the 24th Workshop on Biomedical Language Processing, pages 307–318, Viena, Austria. Association for Computational Linguistics. find
  5. Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta, Vicky Rubini, and Bernardo Magnini. 2026. Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department. find
  6. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
  7. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1906– 1919.
  8. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 2655–2671.
  9. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  10. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rao-2026-innov8rs-crf-filling-2026-iterative/