Elamur Research⌕
☰
home / papers / 2026 / Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge

Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge

Paper · 2026 · Arpan Phukan, Manish Gupta and Asif Ekbal

Summary

The paper presents multi-hop visual question generation. Given two images and facts from a knowledge graph such as Wikipedia, the model must write one question that combines facts about both pictured entities with a visual link between the images. Standard supervised training compares output against one fixed reference, so it penalizes other valid phrasings.

The authors build on the multi-image portion of WebQA. For each sample they add two extra "silver" questions, one from GPT-4V and one from Gemini, plus RAG-retrieved Wikipedia documents. Their framework, M3RQG, is model-agnostic. It feeds images, captions and retrieved facts to a multimodal encoder. Several weight-sharing decoders then let training use all reference questions at once. Training also uses PPO with ROUGE-based rewards and a named-entity overlap term that ties questions to the retrieved facts and reduces hallucination.

Applied to BART, Phi-3.5 and LLaVA, M3RQG produced more fluent, relevant and reasoning-heavy questions than versions trained without PPO. This matters for educational and assistive question-generation tools.

Key points

  • Task definition: a valid question must use at least one KG attribute of each entity and at least one visual relation between the two images. It should not name the entities directly.
  • Data: the authors extend WebQA with one GPT-4V and one Gemini question per sample (alongside the original gold question) and with RAG documents. Code and data are at github.com/thePhukan/M3RQG, but dataset access is limited to non-commercial academic use.
  • Training: the method combines a multi-decoder setup with shared weights, PPO with ROUGE rewards, and a named-entity overlap loss.
  • Results: Phi 3.5-V + M3RQG was the strongest model according to both GPT-4o judging and human evaluation. In the human evaluation, annotators rated 100 random outputs, and agreement was reported with Cohen's kappa.
  • Limitations:
    • Experiments cover English only.
    • Evaluation uses a single dataset (WebQA).
    • The method generates questions without answers.
    • Output is sometimes unstable because all backbones share the same prompt format.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft (India)Hyderabad · India10 works · 2,129 citations🎓Indian Institute of Technology PatnaPatna · India21 works · 374 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksQuestion answering · 441 works

Cite

Cite as Arpan Phukan, Manish Gupta and Asif Ekbal. 2026. Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.202

@inproceedings{phukan2026learning,
  title = {{Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge}},
  author = {Arpan Phukan, Manish Gupta and Asif Ekbal},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.202},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.202}
}

References

66 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. find
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, and 1 others. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. \#\# Ethical Consideration…
  5. This research upholds the highest standards of ethical practice in data handling and dissemination. All Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. 2017. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer… find
  6. Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495–16504. find
  7. Guanliang Chen, Jie Yang, Claudia Hauff, and Geert-Jan Houben. 2018. Learningq: a large-scale dataset for educational question generation. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12. find
  8. Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. 2022. MuRAG: Multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558–5570, Abu Dhabi, United Arab Em…
  9. Association for Computational Linguistics (TACL), volume 9, pages 346–361. find
  10. Pranay Gupta and Manish Gupta. 2022. Newskvqa: Knowledge-aware news video question answering. In Pacific-asia conference on knowledge discovery and data mining, pages 3–15. Springer. find
  11. Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. Manymodalqa: Modality disambiguation and qa over diverse inputs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7879–7886. find
  12. Xuehai He and Xin Eric Wang. 2023. Multimodal graph transformer for multimodal question answering. arXiv preprint arXiv:2305.00581. find
  13. Yi-Ting Huang, Ya-Min Tseng, Yeali S. Sun, and Meng Chang Chen. 2014. Tedquiz: Automatic quiz generation for ted talks video clips to assess listening comprehension. In 2014 IEEE 14th International Conference on Advanced Learning Technologies, pages 350–354. find
  14. Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2023. Toward subgraph-guided knowledge graph question generation with graph neural networks. IEEE Transactions on Neural Networks and Learning Systems. find
  15. Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2024. Explainable multi-hop question generation: An end-to-end approach without intermediate question labeling. arXiv preprint arXiv:2404.00571. find
  16. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  17. Amrith Krishna, Plaban Bhowmick, Krishnendu Ghosh, Archana Sahu, and Subhayan Roy. 2015. Automatic generation and insertion of assessment items in online video courses. In Companion Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI ’15 Companion, page 1–4, New York… find
  18. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32.
  19. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
  20. Ritu Gala, Revathi Vijayaraghavan, Valmik Nikam, and Arvind Kiwelekar. 2021. Real-Time Cognitive Evaluation of Online Learners through Automatically Generated Questions . In 2021 IEEE 20th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), pages 53–58, Los Alamitos, C… find
  21. Vaibhav Kumar and Alan W Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296–7301, Online. Association for Computational Linguistics. find
  22. Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrishnan, Guilin Qi, Lianli Gao, and Yuan-Fang Li. 2019. Difficulty-controllable multi-hop question generation from knowledge graphs. In The Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proc… find
  23. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of th…
  25. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. In Transactions of the Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal… find
  26. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  27. Yongqi Li, Wenjie Li, and Liqiang Nie. 2022. Mmcoqa: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220–4231. find
  28. TongXin Liao, Bin Xu, YiKe Han, Shuai Li, and Shuo Zhang. 2023. Brqg: A bart-based retouching framework for multi-hop question generation. In International Conference on Advanced Data Mining and Applications, pages 165–179. Springer. find
  29. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. Workshop on Text Summariation Branches Out, Post-Conference Workshop of ACL 2004, pages 74–81.
  30. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.
  31. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, and 1 others. 2025. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299. find
  32. Karen Mazidi and Rodney Nielsen. 2014. Linguistic considerations in automatic question generation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 321–326. find
  33. Rahul Mehta, Bhavyajeet Singh, Vasudeva Varma, and Manish Gupta. 2024. Circuitvqa: A visual question answering dataset for electrical circuit images. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 440– 460. Springer. find
  34. Li Mi, Syrielle Montariol, Javiera Castillo Navarro, Xianjie Dai, Antoine Bosselut, and Devis Tuia. 2024. Convqg: Contrastive visual question generation with multimodal guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4207–4215. find
  35. Rajarshee Mitra, Rhea Jain, Aditya Srikanth Veerubhotla, and Manish Gupta. 2021. Zero-shot multilingual interrogative question generation for" people also ask" at bing. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3414–3422. find
  36. Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, and Marzieh Saeidi. 2023. Rquge: Reference-free metric for evaluating question generation by answering the question. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6845– 686… find
  37. Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1802–1813.
  38. Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. arXiv preprint arXiv:1905.08949. — [arXiv:1905.08949](https://arxiv.org/abs/1905.08949) find
  39. Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan. 2020a. Semantic graphs for generating deep questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1463–1475.
  40. Youcheng Pan, Baotian Hu, Qingcai Chen, Yang Xiang, and Xiaolong Wang. 2020b. Learning to generate diverse questions from keywords. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8224– 8228. IEEE. find
  41. Anupam Pandey, Deepjyoti Bodo, Arpan Phukan, and Asif Ekbal. 2025. The quest for visual understanding: A journey through the evolution of visual question answering. arXiv preprint arXiv:2501.07109. — [arXiv:2501.07109](https://arxiv.org/abs/2501.07109) find
  42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  43. Charulata Patil and Manasi Patwardhan. 2020. Visual question generation: The state of the art. ACM Computing Surveys (CSUR), 53(3):1–22. find
  44. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
  45. Arpan Phukan, Manish Gupta, and Asif Ekbal. 2024. ECIS-VQG: Generation of entity-centric information-seeking questions from videos. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14411–14436, Miami, Florida, USA. Association for Computational Linguis… find
  46. Arpan Phukan, Anupam Pandey, Deepjyoti Bodo, and Asif Ekbal. 2025. Videochain: A transformer-based framework for multi-hop video question generation. arXiv preprint arXiv:2511.08348. find
  47. Peng Qi, Xiao Yang, Yiming Zhang, Danqi Chen, and Christopher D Manning. 2020. Answering complex open-domain questions through multi-hop dense retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3071–3080. find
  48. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  49. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  50. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017a. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  51. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017b. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  52. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8876–8884. find
  53. Abigale Stangl, Nitin Verma, Kenneth R. Fleischmann, Meredith Ringel Morris, and Danna Gurari. 2021. Going beyond one-size-fits-all image descriptions to satisfy the information wants of people who are blind or have low vision. In Proceedings of the 23rd International ACM SIGACCESS Conference on Com… find
  54. Dan Su, Yan Xu, Wenliang Dai, Ziwei Ji, Tiezheng Yu, and Pascale Fung. 2020. Multi-hop question generation with graph convolutional network. Proceedings of Findings of the Association for Computational Linguistics (EMNLP). find
  55. Yuxuan Sun, Kai Zhang, and Yu Su. 2023. Multimodal question answering for unified information extraction. arXiv preprint arXiv:2310.03017. — [arXiv:2310.03017](https://arxiv.org/abs/2310.03017) find
  56. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.06039. — [arXiv:2104.06039](https://arxiv.org/abs/2104.06039) find
  57. Harsh Trivedi, Vidhisha Balachandran, Daniel Khashabi, and Dan Roth. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-hop question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  58. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  59. Bingning Wang, Xiaochuan Wang, Ting Tao, Qi Zhang, and Jingfang Xu. 2020. Neural question generation with answer pivot. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9138–9145. find
  60. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the…
  61. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2413– 2427. find
  62. Weichao Wang, Shi Feng, Daling Wang, and Yifei Zhang. 2019. Answer-guided and semantic coherent question generation in open-domain conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… find
  63. Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V. Le, Thang Luong, and Golnaz Ghiasi. 2024b. Haloquest: A visual hallucination dataset for advancing multimodal reasoning. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXVII,… find
  64. Zichao Wang, Andrew S. Lan, Weili Nie, Andrew E. Waters, Phillip J. Grimaldi, and Richard G. Baraniuk. 2018. Qg-net: a data-driven question generation model for educational content. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, L@S ’18, New York, NY, USA. Association for Co… find
  65. Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-gen: an enhanced multi-flow pre-training and fine-tuning framework for natural language generation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial…
  66. Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018a. Visual curiosity: Learning to ask questions to learn visual recognition. In Conference on Robot Learning, pages 63–80. PMLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/phukan-2026-learning-ask-multi-decoder-fine/