Elamur Research⌕
☰
home / papers / 2026 / PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain

PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain

Paper · 2026 · Kamyar Zeinalipour, Andrea Zugarini, Asya Zanollo et al.

Summary

LLM-based medical question answering needs test sets grounded in authoritative documents, but almost all such benchmarks are in English. In Italy, the official product summaries (RCP) published by AIFA are the reference source on medicines, yet no QA resource had been built on them. The authors present PharmaQA.IT, an extractive QA benchmark derived from RCPs in the Silver portion of PharmaER.IT. Their semi-automatic pipeline turns leaflet pages into images and selects representative pages by content type. The multimodal model Qwen3-VL-235B-A22B then proposes question–answer pairs in simple or complex variants, each time adopting a randomly drawn pharmaceutical professional persona. Native-Italian linguists then accepted or rejected each pair against comprehensibility, completeness, relevance and grounding criteria. The resulting pairs cover dosage, contraindications, interactions, warnings and pharmacological properties, with structured JSON answers and evidence. Under a standard RAG baseline with varied chunking, strong open models get close to proprietary systems. Data, code and prompts are released on Hugging Face, filling a gap in Italian biomedical NLP.

Key points

  • Source material: 1,077 RCP leaflets of at least 10 pages from the PharmaER.IT Silver corpus. The final dataset holds 861 validated QA pairs and is intended mainly for evaluation, not training.
  • Generation used Qwen3-VL-235B-A22B on page images. Complex prompt variants were chosen with 25% probability, and validation was a binary accept/discard decision by independent annotators.
  • The retrieval baseline uses multilingual-e5-base with top-k = 10. Retrieval hit@10 rises with chunk size: 0.436 at 64–16, 0.511 at 128–32, 0.580 at 256–64 and 0.615 at 512–128.
  • Among closed-source models at 512–128 chunking, DeepSeek-Chat scored best (ROUGE-L F1 0.679, EM 0.432), followed by GPT-4.1 (0.636 / 0.367). Gemini-2.5-Flash was lowest (0.329 / 0.005).
  • Stated limitations:
    • The benchmark is text-only, since figures and tables were ignored.
    • Answers are single-document factoids, with no multi-hop or unanswerable questions.
    • The annotator pool was small.
    • Questions may carry stylistic bias from the generating model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SienaSiena · Italy10 works · 198 citations

Also on the byline: Istituto Universitario di Studi Superiori di Pavia.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 works

Cite

Cite as Kamyar Zeinalipour, Andrea Zugarini, Asya Zanollo and Leonardo Rigutini. 2026. PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.70

@inproceedings{zeinalipour2026pharmaqa,
  title = {{PharmaQA.IT: an Italian dataset for Q\&A in the pharmaceutical domain}},
  author = {Kamyar Zeinalipour, Andrea Zugarini, Asya Zanollo and Leonardo Rigutini},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.70},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.70}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Alves and Luisa Coheur. 2022. Bootstrapped distant supervision for named entity recognition. In Proceedings of LREC. find
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th 5 Villanova: https://www.opencup.gov.it/portale/ web/opencup/home/progetto/-/cup/B29J24000850005 6 RESPIRA: https://www.opencup.gov.it/portale/ we… ↗
  3. Annual Meeting of the Association for Computational Linguistics, pages 4623–4637. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Holger Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020b. Retrieval-augmented generation for knowledge… — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  4. J. Li and 1 others. 2016. Biocreative v cdr task corpus. In Proceedings of BioCreative. find
  5. Bernardo Magnini and 1 others. 2006. I-cab: the italian content annotation bank. In Proceedings of CLiC-it. find
  6. Telmo Menezes and Benjamin Roth. 2019. Distant supervision for ner: A systematic study. arXiv. find
  7. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368.
  8. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024b. Clue-instruct: Text-based clue generation for educational crossword puzzles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and… find
  9. Chris Quirk and 1 others. 2016. Distant supervision for clinical ner. Journal of Biomedical Informatics. find
  10. N. Ringland and 1 others. 2019. Nne: A distantly supervised named entity dataset. In Proceedings of ACL. find
  11. Simone Tedeschi and Roberto Navigli. 2022. MultiN-ERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In Findings of the Association for Computational Linguistics: NAACL 2022, pages 801–812, Seattle, United States. Association for Computationa… find
  12. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, and 1 others. 2015a. An overview of the bioasq large-scale biomedical semantic indexing and question… find
  13. George Tsatsaronis and 1 others. 2015b. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics. find
  14. Jiawei Zhou, Yu Su, Yijia Wang, Junghyun Chung, Chen Li, Huan Chen, and Xiang Ren. 2023. Universalner: A universal toolkit for cross-domain and multilingual named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). find
  15. Andrea Zugarini and Leonardo Rigutini. 2025a. Pharmaer.it: An italian dataset for entity recognition in the pharmaceutical domain. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 1171–1180. CEUR Workshop Proceedings. find
  16. Andrea Zugarini and Leonardo Rigutini. 2025b. Pharmaer.it: an italian dataset for named entity recognition in the pharmaceutical domain. In Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)., Cagliari, Italy. CEUR Workshop Proceedings. find
  17. Andrea Zugarini, Kamyar Zeinalipour, Achille Fusco, and Asya Zanollo. 2024a. ECWCA - educational CrossWord clues answering: A CALAMITA challenge. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1239–1244, Pisa, Italy. CEUR Workshop Proceedings. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeinalipour-2026-pharmaqa-it-italian-dataset-q/