Elamur Research⌕
☰
home / papers / 2026 / FormGym: Doing Paperwork with Agents

FormGym: Doing Paperwork with Agents

Paper · 2026 · Matthew Toles, Isaac Song, Rattandeep Singh et al.

Summary

Completing printable, PDF-style forms from a user's personal information is common and tedious, yet no benchmark existed to measure how accurately automated systems do it end to end. The authors define the task as follows: given an empty form image and a source persona (free text or another document image), the agent must write the right string into each field's bounding box using an editor API. They build FormGym by converting FUNSD, XFUND and Form-NLU into empty forms, removing the filled-in values through content-aware fill. They also add a manually annotated Auto Loans dataset with long forms, several personas, values composed from multiple facts, and image-based source documents.

Baseline vision language agents (VLAs) score near zero, mainly because they cannot locate where fields are. To address this, the authors train FieldFinder, an open-vocabulary detector that returns a field's bounding box from its name. With FieldFinder, frontier VLAs rise from 5.6% to 27.0% average accuracy. They also beat Claude Computer Use at a small fraction of its cost and time. The benchmark exposes spatial reasoning as a key weakness of current agents on document-style paperwork.

Key points

  • FormGym has 25,466 train and 3,889 test field examples across 1,709 train and 215 test forms, averaging 18.1 fields per form. 46% of test documents are in seven non-English languages, and 65% are scanned.
  • Auto Loans comes from American auto loan application PDFs. It has four personas and per-field correctness functions, so it can accept variant formats such as different phone number styles.
  • Claude Computer Use is less accurate than Claude 4 + FieldFinder. It costs $5.40 versus $0.043 per 1000 fields (over 100x more) and is about 84x slower.
  • Claude 4 given ground-truth field coordinates reaches 74–83% accuracy. This falls to 49% when the persona is supplied as an image.
  • Even with FieldFinder, localization remains the largest error source on every dataset except Form-NLU. Code and data are released at github.com/mtoles/formgym.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 worksEngineering · 133 worksIndustrial and Manufacturing Engineering · 10 works

Cite

Cite as Matthew Toles, Isaac Song, Rattandeep Singh and Zhou Yu. 2026. FormGym: Doing Paperwork with Agents. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.175

@inproceedings{toles2026formgym,
  title = {{FormGym: Doing Paperwork with Agents}},
  author = {Matthew Toles, Isaac Song, Rattandeep Singh and Zhou Yu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.175},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.175}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https://www.anthropic.com/news/ 3-5-models-and-computer-use. Accessed: 2015-10-05. ↗
  2. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003. find
  3. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and 1 others. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In International conference on machine learning, pages 642–652. PMLR.
  4. Common Crawl. 2023. Common crawl corpus. https://commoncrawl.org/. A freely available web crawl dataset. Accessed October 6, 2025. ↗
  5. Yihao Ding, Siqu Long, Jiabin Huang, Kaixuan Ren, Xingxiang Luo, Hyunsuk Chung, and Soyeon Caren Han. 2023. Form-nlu: Dataset for the form natural language understanding. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pa… find
  6. Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, and Goran Glavaš. 2025. Centurio: On drivers of multilingual ability of large vision-language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  7. Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214. — [arXiv:2404.07214](https://arxiv.org/abs/2404.07214) find
  8. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for gui agents. Preprint, arXiv:2410.05243. find
  9. Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR). find
  10. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.
  11. Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. 2019. Tablebank: A benchmark dataset for table detection and recognition. arXiv preprint arXiv:1903.01949. — [arXiv:1903.01949](https://arxiv.org/abs/1903.01949) find
  12. Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. 2020. Docbank: A benchmark dataset for document layout analysis. arXiv preprint arXiv:2006.01038. find
  13. Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021. Selfdoc: Self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5660. find
  14. Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, and 11 others. 2024. Autoglm: Autonomous foundation agents fo… — [arXiv:2411.00820](https://arxiv.org/abs/2411.00820) find
  15. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  16. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203. — [arXiv:2408.00203](https://arxiv.org/abs/2408.00203) find
  17. Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. Preprint, arXiv:2309.08172. — [arXiv:2309.08172](https://arxiv.org/abs/2309.08172) find
  18. OpenAI. 2025. Introducing operator. https://openai.com/index/introducing-operator/. Accessed: 2015-10-05. ↗
  19. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281–14290…
  20. Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. 2022. Doclaynet: A large human-annotated dataset for document-layout segmentation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 3743–3751. find
  21. Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE. find
  22. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. Preprint, arXiv:2408.07199. find
  23. Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022. Formnet: Structural encoding beyond sequential modeling in form document information extraction. arXiv preprint arXiv:2203.08411. find
  24. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In Proceedings of the 13th Interna… find
  25. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025a. Ui-tars: Pioneering automated gui interaction with…
  26. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025b. Ui-tars: Pioneering automated gui interaction with…
  27. Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang Ji. 2024a. Falcon-ui: Understanding gui before following user instructions. Preprint, arXiv:2412.09362. — [arXiv:2412.09362](https://arxiv.org/abs/2412.09362) find
  28. Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Am-lekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. 2024b. Scribeagent: Towards specialized web agents using production-scale workflow data. Preprint, arXiv:2411.15004. — [arXiv:2411.15004](https://arxiv.org/abs/2411.15004) find
  29. University of California, San Francisco. Ucsf industry documents library: Tobacco collection. https://www.industrydocuments.ucsf.edu/tobacco/about/. A digital corpus of internal tobacco industry documents curated by UCSF. Accessed October 6, 2025. ↗
  30. USOMB. 2016. Information collection budget of the united states government, 2016. Technical report, Executive Office of the President, Washington, D.C. Office of Information and Regulatory Affairs. find
  31. Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. 2025. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. Preprint, arXiv:2505.16421.
  32. Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. 2024. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818–4829. find
  33. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200.
  34. Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2022. Xfund: a benchmark dataset for multilingual visually rich form understanding. In Findings of the association for computational linguistics: ACL 2022, pages 3214–3224. find
  35. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. Preprint, arXiv:2310.11441. find
  36. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045.
  37. Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. 2022. Vlmbench: A compositional benchmark for vision-and-language manipulation. Advances in Neural Information Processing Systems, 35:665–678. find
  38. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE. find
  39. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toles-2026-formgym-doing-paperwork-agents/