Elamur Research⌕
☰
home / papers / 2026 / CoVaPh: A Vision-Language Multi-Agent Dialogue System for Tool-Augmented Pharmacogenetic Reasoning and Personalized Guidance

CoVaPh: A Vision-Language Multi-Agent Dialogue System for Tool-Augmented Pharmacogenetic Reasoning and Personalized Guidance

Paper · 2026 · Shang-Chun Luke Lu, Hsin Yang, Hui-Hsin Xue et al.

Summary

Clinicians who turn a patient's genotype into a dosing decision must search CPIC guidelines, where much of the key content sits in tables and figures. Generic LLMs may also invent advice for drug–gene pairs that have no guideline at all. CoVaPh tackles both problems with a multi-agent pipeline built around the open OpenReasoning-Nemotron-32B reasoner. A rewriting agent, LoRA-tuned on synthetic valid and invalid queries (partly generated with Gemini 2.5 Pro), turns free-form questions into structured requests. Each request names the drug, the gene and the canonical CPIC guideline title, or the agent refuses when no guideline exists. These requests drive hybrid retrieval: a multimodal vector index over guideline PDF pages, using ColQwen embeddings and a Nemotron Nano VLM for visual content, plus live CPIC API calls for recommendations and phenotype frequencies. Answers were compared with pharmacist-curated references and scored by LLM judges. The full system matches or beats Grok-4 and Gemini 2.5 Pro, with its clearest advantage in accuracy. The work suggests that mid-sized open models with suitable tools can support high-stakes clinical QA in resource-limited settings.

Key points

  • Query rewriter model: the paper describes it as a fine-tuned 32B model, but the setup section names Llama Nemotron 3.3 Super 49B.
    • Training used PEFT LoRA with r=8, α=16 and dropout 0.05.
    • It ran for 3 epochs on two A100 80 GB GPUs, taking about 12 hours.
    • The training data covered six CPIC question templates, such as dose adjustment, risk stratification and genotype–phenotype mapping.
  • Evaluation set: 12 difficult questions kept out of training, along with 819 simulated questions from clinical scenarios. These were generated by frontier models and filtered by pharmaceutical professionals.
  • Scoring: answers were rated 1–5 on Accuracy, Completeness, Clarity and Relevance. GPT-5 (medium) and Grok-4-fast served as judges.
  • Ablation on the 32B answer generator:
    • Accuracy rose from 3.58 (no retrieval) to 4.13 (rewriter + guideline RAG) and 4.29 (+ CPIC API).
    • Completeness rose from 4.17 to 4.42 and then 4.63.
    • Clarity rose from 4.42 to 4.67 and then 4.83.
  • Comparison with baselines: against Grok-4, Gemini 2.5 Pro and MedGemma 27B Multimodal, CoVaPh reports +2–3% overall and +6–9% accuracy. The abstract states a roughly 10% gain. Completeness, clarity and relevance showed no material differences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksDialogue systems · 624 worksVision & language · 657 works

Cite

Cite as Shang-Chun Luke Lu, Hsin Yang, Hui-Hsin Xue, Ping Lin Tsai, Yu Jing Weng, Shiou-Chi Li, Jen-Wei Huang and Hui Hua Chang. 2026. CoVaPh: A Vision-Language Multi-Agent Dialogue System for Tool-Augmented Pharmacogenetic Reasoning and Personalized Guidance. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lu2026covaph,
  title = {{CoVaPh: A Vision-Language Multi-Agent Dialogue System for Tool-Augmented Pharmacogenetic Reasoning and Personalized Guidance}},
  author = {Shang-Chun Luke Lu, Hsin Yang, Hui-Hsin Xue, Ping Lin Tsai, Yu Jing Weng, Shiou-Chi Li, Jen-Wei Huang and Hui Hua Chang},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.42/}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Asai, Z. Wu, Y. Wang, and et al. 2023. SELF-RAG: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.
  2. G. Xiong, Q. Jin, X. Wang, and et al. 2024. Improving retrieval-augmented generation in medicine with iterative follow-up questions. arXiv preprint arXiv:2408.00727. — [arXiv:2408.00727](https://arxiv.org/abs/2408.00727) find
  3. L. Gao, X. Ma, J. Lin, and J. Callan. 2022. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496.
  4. S. Yao, J. Zhao, D. Yu, and et al. 2022. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  5. S. Gehrmann, S. Douglas, S. Hyland, and et al. 2024. PGxQA: A benchmark for question answering on pharmacogenomic guidelines. arXiv preprint arXiv:2404.14498. — [arXiv:2404.14498](https://arxiv.org/abs/2404.14498) find
  6. S. Yin, C. Fu, S. Zhao, and et al. 2023. A survey on vision-language-action models. arXiv preprint arXiv:2311.17143. — [arXiv:2311.17143](https://arxiv.org/abs/2311.17143) find
  7. P. Lewis, E. Perez, A. Piktus, and et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS).
  8. Z. Li, J. Wang, Z. Jiang, and et al. 2024. DMQR-RAG: Diverse multi-query rewriting for retrieval-augmented generation. arXiv preprint arXiv:2411.13154. — [arXiv:2411.13154](https://arxiv.org/abs/2411.13154) find
  9. H. Liu, C. Li, Q. Wu, and Y. J. Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  10. J. Mao, Y. Liu, J. Ma, and et al. 2023. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283. — [arXiv:2305.14283](https://arxiv.org/abs/2305.14283) find
  11. Mercer. 2025. 2025 global talent trends study. Technical report, Mercer LLC. find
  12. NVIDIA. 2024. NVIDIA nemotron nano VL. https://huggingface.co/nvidia/ Llama-Nemotron-Nano-VL-8B. ↗
  13. Y. Qin, S. Liang, Y. Wen, and et al. 2023. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv preprint arXiv:2307.16789.
  14. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2411.16223. find
  15. X. Ren, L. Wang, and Y. Yang. 2024. A surprisingly simple yet effective multi-query rewriting method for dense retrieval. arXiv preprint arXiv:2406.18960. — [arXiv:2406.18960](https://arxiv.org/abs/2406.18960) find
  16. T. Schick, J. Dwivedi-Yu, R. Dessì, and et al. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
  17. L. Zhang, B. Li, S. Li, and et al. 2023. A survey on vision-language models: Towards comprehensive representation. arXiv preprint arXiv:2306.09243. — [arXiv:2306.09243](https://arxiv.org/abs/2306.09243) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2026-covaph-vision-language-multi-agent/