Elamur Research⌕
☰
home / papers / 2026 / DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures

DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures

Paper · 2026 · Benno Uthayasooriyar, Antoine Ly, Franck Vermet et al.

Summary

Layout-aware encoders for document understanding (forms, receipts, invoices) usually add absolute 2D bounding-box embeddings to a BERT-style model, sometimes with Cartesian relative biases in attention. The authors argue that this setup generalizes poorly. Absolute positions change when content shifts on the page, and horizontal/vertical offsets hide relations such as "above" or "nearby". DocPolarBERT removes absolute 2D embeddings entirely. Instead, its self-attention uses two learned relative biases: the distance and the angle between bounding-box centers, measured from the query token. Both values are discretized into a small number of buckets. The model is pre-trained only on openly available data, a mix of OCR-IDL and DOCILE (about 1.8M documents, over six times fewer than IIT-CDIP). It is then fine-tuned for named-entity recognition on FUNSD, SROIE, CORD, PAYSLIPS and DOCILE. It reaches competitive or better results than comparable baselines, with its strongest gains on table-heavy documents. The work suggests that a well-designed attention bias can partly offset smaller pre-training corpora. This makes it relevant for lightweight, vision-free, privacy-friendly document processing.

Key points

  • Encoding scheme: Relative polar encoding replaces 2D position embeddings. Distances are capped and split into 4 buckets (a quantile-based variant works best), and angles are split into 8 bins. Both are added as attention biases in the style of Shaw et al. (2018).
  • Ablation on absolute embeddings: Adding absolute 2D embeddings back hurts performance. Average NER F1 drops from 81.34 without them to 80.23 with them.
  • Bucketing strategy: Results vary by less than 2 F1 points across schemes. The 4-quantile scheme scores 81.34 on average, just ahead of the best LayoutLMv2/v3 (T5-style) variant at 81.22.
  • Sequence-length generalization: Models are trained on short documents and tested on long ones. DocPolarBERT averages 74.86, against 74.58 for a vision-free LayoutLMv3* and 71.02 for LayoutLM. It leads on PAYSLIPS and DOCILE* but trails on FUNSD, SROIE and CORD.
  • Efficiency and limitations: Per-document inference is slower because of the relative bias matrices, but the model scales well across 4 GPUs. Evaluation covers NER only. Code, data and models are released at github.com/buthaya/docpolarbert.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France84 works · 25,414 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🔬Institut de Recherche en Informatique et Systèmes AléatoiresRennes · France7 works · 1,944 citations🎓Institut National des Sciences Appliquées de RennesRennes · France2 works · 34 citations🎓Université de RennesRennes · France3 works🔬Laboratoire de Mathématiques de Bretagne AtlantiqueBrest · France2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Benno Uthayasooriyar, Antoine Ly, Franck Vermet and Caio Corro. 2026. DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.181

@inproceedings{uthayasooriyar2026docpolarbert,
  title = {{DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures}},
  author = {Benno Uthayasooriyar, Antoine Ly, Franck Vermet and Caio Corro},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.181},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.181}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Conclusion We propose a novel relative positional encoding of layout structure based on the polar coordinate system. This approach allows the model to directly encode semantically crucial information (e.g. table headers are at the top row, no matter how far it is). Experimentally, our approach ou… find
  2. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 38546–38556. Cur…
  3. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003. find
  4. Haoli Bai, Zhiguang Liu, Xiaojun Meng, Li Wentao, Shuang Liu, Yifeng Luo, Nian Xie, Rongfu Zheng, Liangwei Wang, Lu Hou, Jiansheng Wei, Xin Jiang, and Qun Liu. 2023. Wukong-reader: Multi-modal pre-training for fine-grained visual document understanding. In Proceedings of the 61st Annual Meeting of t… find
  5. Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7096–7116, Online. Association for Computational Linguistics…
  6. Ali Furkan Biten, Rubèn Tito, Lluis Gomez, Ernest Valveny, and Dimosthenis Karatzas. 2023. OCR-IDL: OCR Annotations for Industry Document Library Dataset, page 241–252. Springer Nature Switzerland. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. 2021. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39–50. find
  9. Jean-Philippe Thiran Guillaume Jaume, Hazim Kemal Ekenel. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In Accepted to ICDAR-OST. find
  10. Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR). find
  11. Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page… find
  12. Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. 2022. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10767– 107… find
  13. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 4083–4091, New York, NY, USA. Association for Computing Machinery.
  14. Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. find
  15. Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo. 2021. Spatial dependency parsing for semi-structured document information extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 330–343, Online. Association for Computational Linguist… find
  16. Zhouqiang Jiang, Bowen Wang, Junhao Chen, and Yuta Nakashima. 2025. ReLayout: Towards real-world document understanding via layout-enhanced pre-training. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3778–3793, Abu Dhabi, UAE. Association for Computational L… find
  17. D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. 2006. Building a test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR06, page 665–666. ACM. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  19. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations. find
  20. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. 2024. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15630–15640. find
  21. Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: a consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019. find
  22. Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. 2021. Going full-tilt boogie on document understanding with text-image-layout transformer. In Document Analysis and Recognition – ICDAR 2021, pages 732–747, Cham. Springer International Publishi… find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  24. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New…
  25. Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. 2024. Instructdoc: A dataset for zero-shot generalization of visual document understanding with instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19071–19079. find
  26. Yi Tu, Ya Guo, Huan Chen, and Jinyang Tang. 2023. Layoutmask: Enhance text-layout interaction in multi-modal pre-training for document understanding. arXiv preprint arXiv:2305.18721. find
  27. Benno Uthayasooriyar, Antoine Ly, Franck Vermet, and Caio Corro. 2025. Training LayoutLM from scratch for efficient named-entity recognition in the insurance domain. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narra… find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  29. Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. 2024b. Length generalization of causal transformers without position encoding. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14024–14040, Bangkok, Thailand. Association for C… find
  30. Zhengxuan Wu, Christopher D. Manning, and Christopher Potts. 2023. ReCOGS: How incidental details of a logical form overshadow an evaluation of semantic interpretation. Transactions of the Association for Computational Linguistics, 11:1719–1733. find
  31. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Comp…
  32. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2019. Layoutlm: Pre-training of text and layout for document image understanding.
  33. Yuekun Yao and Alexander Koller. 2022. Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  34. Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalický, Jiří Matas, Antoine Doucet, Mickaël Coustaty, and Dimosthenis Karatzas. 2023. DocILE Benchmark for Document Information Localization and Extraction. In Document Analysis and Recognition - IC-DAR 20… find
  35. Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. 2024a. DocLLM: A layout-aware generative language model for multimodal document understanding. In Proceedings of the 62nd Annual Meeting of the Association for Comput… find
  36. Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. LiLT: A simple yet effective language-independent layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7747–7757, Dublin, Irel… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uthayasooriyar-2026-docpolarbert-pre-trained-model-document/