Elamur Research⌕
☰
home / papers / 2025 / Multi-Agent Interactive Question Generation Framework for Long Document Understanding

Multi-Agent Interactive Question Generation Framework for Long Document Understanding

Summary

Long multi-page documents with complex layouts remain hard for Large Vision-Language Models (LVLMs). The authors attribute much of this to a lack of fine-grained multi-page QA training data, especially for Arabic, and note that existing long-document benchmarks depend on expensive human annotation. They propose a fully automated pipeline with two parts:

  • Collection. An agent-driven web scraper gathers long English and Arabic documents, filtering out short, restrictively licensed or unusable files.
  • Generation. A chain of five LLM agents creates and refines questions over chunked pages. Inputs are page images, LVLM-based OCR and YOLO layout analysis.

Inside the chain, separate agents write questions, filter them, answer them, assess them and validate evidence. Questions are made harder whenever the answering agent succeeds too often.

The result is AraEngLongBench, a set of 6,732 questions over documents that are longer and more text-dense than those of earlier benchmarks. Tested LVLMs score below 50% on average. All models perform worse on Arabic and rarely abstain correctly on unanswerable questions. This shows that automatically generated questions can still be demanding for current models.

Key points

  • Pipeline steps:

    • PDF pages are rendered to images with pdf2image.
    • Text is read by LVLM-based OCR, and YOLO detects layout elements.
    • Documents are split into chunks that overlap by 10 pages.

  • Five agents:

    • Agent 1 generates questions and later refines them.
    • Agent 2 filters them.
    • Agent 3 answers them from the images.
    • Agent 4 writes reference answers, rates difficulty and returns feedback.
    • Agent 5 performs a final check of evidence consistency.
    • If Agent 4 measures more than 40% accuracy, Agent 1 is pushed to make the questions harder.

  • Dataset (AraEngLongBench): 6,732 questions in 12 types.

    • Reasoning: 4,160 (61.8%).
    • Hypothetical Reasoning: 798 (11.9%).
    • Unanswerable: 454 (6.7%).
    • Every entry stores the question with its answer, the pages and sources that support it, a written rationale, and a validation result.

  • Evaluation splits: results are broken down by context length (SC < 100, MC 100–200, LC > 200), single-page vs cross-page, and unanswerable questions, for both English and Arabic.

  • Average accuracy:

    ModelAverage
    Gemini-1.5 Pro49.8%
    Gemini-2.0 Flash49.6%
    Qwen 2.5 VL44.5%
    Qwen 2 VL38.4%
    GPT-4o29.8%
    Llama 3.2 VL32.4% (most categories NA)

    Accuracy on unanswerable questions mostly falls between about 3% and 18%.

Interesting ideas

  • I1. Difficulty is adjusted by feedback: whenever the solver agent passes a set accuracy threshold, the generator is told to write harder questions, so the benchmark stays near the edge of what models can do. — For Mova (learn): In self-play, a question generator over a fable's world could raise difficulty (more hops, more referents) whenever Mova's reader answers too large a share correctly.
  • I2. Answering, judging and evidence checking are handled by separate agents, and a final validator confirms each answer against its source pages instead of trusting the generator. — For Mova (learn): LLM teacher tiers could keep a separate "evidence witness" that must point to the supporting sentence before a silver QA item or parse is accepted.
  • I3. Unanswerable questions are included deliberately as a scored category, which exposes how badly models fail to abstain. — For Mova (answer): School tests and fable QA should contain questions whose answer is absent from the text, as negative controls checking that the world model says "not stated" rather than guessing.
  • I4. Results are split into single-page and cross-page questions, which separates local lookup from reasoning that must link distant parts of a text. — For Mova (answer): Evaluation could report accuracy separately for answers grounded in one sentence and for those needing facts joined across sentences, showing where the world-building step breaks.
  • I5. Long inputs are cut into overlapping chunks so that context carries across boundaries for downstream processing. — For Mova (world): When building entity states and event order for long texts, overlapping windows could keep referents and holdings continuous across segment edges before they are merged into one world.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Question answering · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.20145 · arXiv 2507.20145

@misc{wang2025multi,
  title = {{Multi-Agent Interactive Question Generation Framework for Long Document Understanding}},
  author = {Wang, Kesen and Toibazar, Daulet and Alfulayt, Abdulrahman and Albadawi, Abdulaziz S. and Alkahtani, Ranya A. and Ibrahim, Asma A. and Alhomoud, Haneen A. and Mohamed, Sherif and Moreno, Pedro J.},
  year = {2025},
  doi = {10.48550/arxiv.2507.20145},
  eprint = {2507.20145},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.20145}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [8] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” arXiv preprint arXiv:2203.10244, 2022. ↗ our card
  2. [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. ↗ our card
  3. [9] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar, “Infographicvqa,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1697–1706. find
  4. [2] Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024, Accessed: 2025-02-11. find our card
  5. [3] Anthropic, “Claude 3 haiku: Our fastest model yet,” 2024, Accessed: 2025-02-11. find
  6. [10] Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua, “Towards complex document understanding by discrete reasoning,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4857–4866. find
  7. [11] Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al., “Document understanding dataset and evaluation (dude),” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 19528–19540. find
  8. [12] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al., “Mmlongbench-doc: Benchmarking long-context document understanding with visualizations,” arXiv preprint arXiv:2407.01523, 2024. ↗
  9. [13] Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al., “Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating,” arXiv preprint arXiv:2412.18424, 2024. ↗
  10. [14] Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, and Lidong Bing, “M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework,” arXiv preprint arXiv:2411.06176, 2024. ↗
  11. [15] Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S Khan, Salman Khan, and Rao M Anwer, “Camel-bench: A comprehensive arabic lmm benchmark,” arXiv preprint arXiv:2410.18976, 2024. ↗
  12. [16] Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar, “Doclaynet: A large human-annotated dataset for document-layout segmentation,” in Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp. 3743–3751. find
  13. [17] Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim, “A large-scale multi-document summarization dataset from the wikipedia current events portal,” arXiv preprint arXiv:2005.10070, 2020. ↗
  14. [18] Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalickỳ, Jiřı́ Matas, Antoine Doucet, Mickaël Coustaty, et al., “Docile benchmark for document information localization and extraction,” in International Conference on Document Analysis and Recognition. Springer, 2023, pp. 147–166. find
  15. [19] Minesh Mathew, Ruben Tito, Dimosthenis Karatzas, R Manmatha, and CV Jawahar, “Document visual question answering challenge 2020,” arXiv:2008.08899, 2020. arXiv preprint [20] Linmei Hu, Duokang Wang, Yiming Pan, Jifan Yu, Yingxia Shao, Chong Feng, and Liqiang Nie, “Novachart: A large-scale dataset towards chart understanding and generation of multimodal large language models,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 3917–3925. ↗
  16. [21] Ray Smith, “An overview of the tesseract ocr engine,” in Ninth international conference on document analysis and recognition (ICDAR 2007). find
  17. [22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al., “Learning transferable visual models from natural language supervision,” arXiv preprint arXiv:2103.00020, 2021. ↗ our card
  18. [23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023. ↗ our card
  19. [24] Omid Faraji Kar, Alessio Tonioni, Petra Poklukar, Anshul Kulshrestha, Amir Zamir, and Federico Tombari, “Brave: Broadening the visual encoding of vision-language models,” arXiv preprint arXiv:2404.07204, 2024. ↗
  20. [25] N. Pangakis, S. Wolken, and N. Fasching, “Automated annotation with generative ai requires validation,” arXiv preprint, 2023. find
  21. [26] Xinyi Yang, Lei Zhang, Saurabh Trivedi, Qian Liao, Byron C. Wallace, and Matthew Lease, “Labelvizier: Error profiling and interactive data annotation for long-document understanding,” arXiv preprint, 2023. find
  22. [27] H. Kim, K. Mitra, R. L. Chen, S. Rahman, and D. Zhang, “Meganno+: A human-llm collaborative annotation system,” arXiv preprint, 2024. find
  23. [28] Jeremy Belval, “pdf2image: A python library to convert pdf pages to images using poppler,” GitHub repository, 2018. find
  24. [29] Xinyu Zhang, Lin Zhang, Jiaxin Wang, et al., “Texthawk2: A large vision-language model excels in bilingual ocr and grounding with 16x fewer tokens,” arXiv preprint arXiv:2410.05261, 2024. ↗
  25. [30] Zhaoyi Liu, Yifan Li, Qi Wang, et al., “Towards general ocr application via a vision-language model,” arXiv preprint arXiv:2501.15558, 2024. ↗
  26. [31] Author(s), “The yolo framework: A comprehensive review of evolution and applications,” Computers, vol. 13, no. 12, pp. 336, 2024. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-multi-agent-interactive-question-generation/