Elamur Research
☰
home / papers / 2019 / LayoutLM: Pre-training of Text and Layout for Document Image Understanding

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1912.13318

Summary

Most pre-trained language models, such as BERT, treat a document as a flat sequence of tokens. They ignore where words appear on the page and what they look like, and this information is crucial for scanned business documents like forms, receipts and invoices. LayoutLM extends the BERT architecture with 2-D position embeddings derived from each token's OCR bounding box. During fine-tuning, it can also add visual features of token regions extracted with Faster R-CNN. The model is pre-trained without labels on the IIT-CDIP Test Collection 1.0, which has over 6 million documents and about 11 million page images. Its main objective is a Masked Visual-Language Model (MVLM), which predicts hidden words from their textual and spatial context. This can optionally be combined with Multi-label Document Classification (MDC). On FUNSD (form understanding), SROIE (receipt information extraction) and RVL-CDIP (document image classification), LayoutLM sets new state-of-the-art results and clearly beats text-only BERT and RoBERTa. The work showed that jointly pre-training text and layout pays off for Document AI, and the code and checkpoints were released publicly.

Key points

  • Reported gains over the previous best: FUNSD F1 rose from 70.72 to 79.27, SROIE F1 from 94.02 to 95.24 (LayoutLM LARGE, 11M images, 1 epoch), and RVL-CDIP accuracy from 93.07% to 94.42% (LayoutLM BASE with image features).
  • The input combines token embeddings, four 2-D coordinate embeddings (x0, y0, x1, y1), and optional image embeddings. Model sizes are about 113M parameters for BASE, 160M for BASE with image features, and 343M for LARGE.
  • More pre-training data helps. On SROIE, LayoutLM BASE (MVLM) improves from 0.9388 F1 with 500K images to 0.9438 with 11M images. The strongest text-only model, RoBERTa LARGE, reaches 0.9280.
  • Visual features matter most for classification. On RVL-CDIP, text+layout LayoutLM BASE scores 91.78%, rising to 94.42% with image embeddings. Text-only BERT and RoBERTa reach about 89.8–90.1%.
  • The scope is limited to scanned documents; digital-born files are excluded. The paper appeared at KDD 2020, and the code and models are available at aka.ms/layoutlm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

180 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 177 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3394486.3403172 · arXiv 1912.13318

@misc{xu2019layoutlm,
  title = {{LayoutLM: Pre-training of Text and Layout for Document Image Understanding}},
  author = {Xu, Yiheng and Li, Minghao and Cui, Lei and Huang, Shaohan and Wei, Furu and Zhou, Ming},
  year = {2019},
  doi = {10.1145/3394486.3403172},
  eprint = {1912.13318},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3394486.3403172}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Zeshan Afzal, Andreas Kölsch, Sheraz Ahmed, and Marcus Liwicki. 2017. Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) 01 (2017),… find
  2. Arindam Das, Saikat Roy, and Ujjwal Bhattacharya. 2018. Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networks. 2018 24th International Conference on Pattern Recognition (ICPR) (2018), 3180–3185. find
  3. Tyler Dauphinee, Nikunj Patel, and Mohammad Mehdi Rashidi. 2019. Modular Multimodal Architecture for Document Classification. ArXiv abs/1912.04376 (2019). — [arXiv:1912.04376](https://arxiv.org/abs/1912.04376) find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Jaekyu Ha, Robert M Haralick, and Ihsin T Phillips. 1995. Document page decomposition by the bounding-box project. In Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 2. IEEE, 1119–1122. find
  6. Jaekyu Ha, Robert M Haralick, and Ihsin T Phillips. 1995. Recursive XY cut using bounding boxes of connected components. In Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 2. IEEE, 952–955. find
  7. Leipeng Hao, Liangcai Gao, Xiaohan Yi, and Zhi Tang. 2016. A Table Detection Method for PDF Documents Based on Convolutional Neural Networks. 2016 12th IAPR Workshop on Document Analysis Systems (DAS) (2016), 287–292. find
  8. Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. 2015 13th International Conference on Document Analysis and Recognition (ICDAR) (2015), 991–995. find
  9. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017. Mask R-CNN. CoRR abs/1703.06870 (2017). arXiv:1703.06870 http://arxiv.org/abs/1703.06870 — [arXiv:1703.06870](https://arxiv.org/abs/1703.06870) ↗
  10. Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW) 2 (2019), 1–6. find
  11. Anoop R Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018. Chargrid: Towards Understanding 2D Documents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational… find
  12. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. https://arxiv.org/abs/1… ↗ our card
  13. Frank Lebourgeois, Z Bublinski, and H Emptoz. 1992. A fast and efficient method for extracting text paragraphs and graphics from unconstrained documents. In Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol. II. Conference B: Pattern Recognition Methodology and Systems. IE… find
  14. D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. 2006. Building a Test Collection for Complex Document Information Processing. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, Washington, USA) (SI… — [doi:10.1145/1148170.1148307](https://doi.org/10.1145/1148170.1148307) find
  15. Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph Convolution for Multimodal Information Extraction from Visually Rich Documents. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv abs/1907.11692 (2019). find
  17. S. Marinai, M. Gori, and G. Soda. 2005. Artificial neural networks for document analysis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 1 (Jan 2005), 23–35. https://doi.org/10.1109/TPAMI.2005.4 — [doi:10.1109/tpami.2005.4](https://doi.org/10.1109/tpami.2005.4) ↗
  18. L. O’Gorman. 1993. The document spectrum for page layout analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 11 (Nov 1993), 1162– 1173. https://doi.org/10.1109/34.244677 — [doi:10.1109/34.244677](https://doi.org/10.1109/34.244677) ↗
  19. Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (2015), 1137–1149. find our card
  20. Ritesh Sarkhel and Arnab Nandi. 2019. Deterministic Routing between Layout Abstractions for Multi-Scale Classification of Visually Rich Documents. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial In… — [doi:10.24963/ijcai.2019/466](https://doi.org/10.24963/ijcai.2019/466) find
  21. Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. 2017. DeepDeSRT: Deep Learning for Detection and Structure Recognition of Tables in Document Images. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) 01 (2017), 1162–1167. find
  22. Michael Shilman, Percy Liang, and Paul Viola. 2005. Learning nongenerative grammatical models for document analysis. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Vol. 2. IEEE, 962–969. find
  23. Anikó Simon, J-C Pret, and A Peter Johnson. 1997. A fast algorithm for bottom-up document layout analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 3 (1997), 273–277. find
  24. Carlos Soto and Shinjae Yoo. 2019. Visual Detection with Context for Document Layout Analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computati… find
  25. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. 2016. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In AAAI. find
  26. Matheus Palhares Viana and Dário Augusto Borges Oliveira. 2017. Fast CNN-Based Document Layout Analysis. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) (2017), 1173–1180. find
  27. H. Wei, M. Baechler, F. Slimane, and R. Ingold. 2013. Evaluation of SVM, MLP and GMM Classifiers for Layout Analysis of Historical Documents. In 2013 12th International Conference on Document Analysis and Recognition. 1220–1224. https: //doi.org/10.1109/ICDAR.2013.247 — [doi:10.1109/icdar.2013.247](https://doi.org/10.1109/icdar.2013.247) find
  28. Xiaowei Yang, Ersin Yumer, Paul Asente, Mike Kraley, Daniel Kifer, and C. Lee Giles. 2017. Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 4342–4351. find
  29. Xu Zhong, Jianbin Tang, and Antonio Jimeno-Yepes. 2019. PubLayNet: largest dataset ever for document layout analysis. ArXiv abs/1908.07836 (2019). — [arXiv:1908.07836](https://arxiv.org/abs/1908.07836) find