Elamur Research⌕
☰
home / papers / 2026 / Switching Heads and Softening Tokens: Turnkey Solutions to Visually Grounded Document QA

Switching Heads and Softening Tokens: Turnkey Solutions to Visually Grounded Document QA

Paper · 2026 · Ximing Wen, Wenbo Li, Sudipta Paul et al.

Summary

The paper tackles visually grounded document question answering (VGDQA), where a model must answer a question about a document and also mark where each answer sits on the page. Existing systems often assume one contiguous answer per question or rely on external pre- and post-processing, so they cannot be trained end to end. The authors adapt OCR-based LLMs into self-contained VGDQA systems in two ways. The first is a single-head model that adds coordinate tokens to the vocabulary. It is trained with a "softening token" objective: an expected coordinate is computed from the token probabilities, which makes an MSE loss differentiable. The second is a dual-head model in which a special token hands generation over to a regression head that outputs continuous boxes. This model is trained with MSE plus an IoU loss. An ensemble then takes the answer set and initial boxes from the single-head model and refines the coordinates with the dual-head output. Evaluated on BoundingDocs, the ensemble beats both of its components and prior baselines. This points to a practical, multi-answer path to interpretable document QA.

Key points

  • Single-head model: It adds 1,001 tokens (<B0>–<B1000>) for normalized coordinates, and each answer is followed by four box tokens. Removing the SoftMSE loss lowers grounding accuracy from 0.796 to 0.762. On queries with five or more answers, accuracy falls from 0.618 to 0.549.
  • Dual-head model: A <SWITCH_HEAD> token alternates between the text head and the box regression head. Without the IoU loss, accuracy collapses from 0.854 to 0.042, Acc-Small falls to 0.011, and training diverges (NaN loss) after the second epoch.
  • Ensemble design: The single-head model fixes the answer structure, which avoids the dual-head model's spurious switching. The dual-head model supplies the finer box coordinates.
  • Language results: On BoundingDocs v2.0, which covers eight languages, the ensemble scores 0.879 on English. It scores much lower on Chinese (0.338) and Japanese (0.333), which the authors attribute to English-heavy training data.
  • Limitations: Performance depends on OCR quality and on the pretrained instruction-tuned LLM. It also degrades on documents longer than 4K tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Samsung (South Korea)Seoul · South Korea9 works · 1,214 citations🏢Samsung ElectronicsSeoul · South Korea2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksQuestion answering · 487 works

Cite

Cite as Ximing Wen, Wenbo Li, Sudipta Paul, Yashas Malur Saidutta, Kalpa Gunaratna and Srinivas Chappidi. 2026. Switching Heads and Softening Tokens: Turnkey Solutions to Visually Grounded Document QA. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1818

@inproceedings{wen2026switching,
  title = {{Switching Heads and Softening Tokens: Turnkey Solutions to Visually Grounded Document QA}},
  author = {Ximing Wen, Wenbo Li, Sudipta Paul, Yashas Malur Saidutta, Kalpa Gunaratna and Srinivas Chappidi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1818},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1818}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, and R. Manmatha. 2024. Docformerv2: Local features for document understanding. In AAAI. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. CoRR.
  3. Alessio Chen, Simone Giovannini, Andrea Gemelli, Fabio Coppini, and Simone Marinai. 2025. Towards reliable and interpretable document question answering via vlms. CoRR. find
  4. Simone Giovannini, Fabio Coppini, Andrea Gemelli, and Simone Marinai. 2025. Boundingdocs: a unified dataset for document question answering with spatial annotations. CoRR. find
  5. Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. 2025. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. In ACL.
  6. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR.
  7. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document AI with unified text and image masking. In ACM MM.
  8. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In ECCV.
  9. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. LISA: reasoning segmentation via large language model. In CVPR. find
  10. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICML. 2025. A bounding box is worth one to…
  11. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. 2024. Layoutllm: Layout instruction tuning with large language models for document understanding. In CVPR. find
  12. Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Ser-Nam Lim, and Rajiv Ramnath. 2025. Dlava: Document language and vision assistant for answer localization with enhanced interpretability and trustworthiness. In ICML Workshops. find
  13. Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. 2024. Docllm: A layout-aware generative language model for multimodal document understanding. In ACL. find
  14. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, and 56 others. 2025. Internvl3.5: Advancing open-source multimodal…
  15. Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. Deepseek-ocr: Contexts optical compression. CoRR.
  16. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In ACL.
  17. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR.
  18. Yinan Zhou, Yuxin Chen, Haokun Lin, Shuyu Yang, Li Zhu, Zhongang Qi, Chen Ma, and Ying Shan. 2025. DOGR: towards versatile visual document grounding and referring. In CVPR. find
  19. Zhaoqing Zhu, Chuwei Luo, Zirui Shao, Feiyu Gao, Hangdi Xing, Qi Zheng, and Ji Zhang. 2025. A simple yet effective layout token in large language models for document understanding. In CVPR. find
  20. Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. 2025. Do-clayllm: An efficient multi-modal extension of large language models for text-rich document understanding. In CVPR. find
  21. Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, Hao Liu, and Can Huang. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2026-switching-heads-softening-tokens-turnkey/