Elamur Research⌕
☰
home / papers / 2026 / eSciBench: An Extensible Scientific PDF Extraction Benchmark

eSciBench: An Extensible Scientific PDF Extraction Benchmark

Summary

Pulling structured information such as authors, affiliations, references, tables and equations out of scientific PDFs could save Digital Humanities projects a great deal of manual metadata entry. Existing evaluation resources fall short. They either contain few scientific pages or, like the benchmark of Meuschke et al. (2023), inherit flaws from DocBank: labels derived from Pdfplumber parsing, no categories for several metadata types, merged reference blocks, partial page coverage and tangled evaluation code. The authors therefore built eSciBench from scratch. It contains 100 arXiv articles (1 949 pages) whose PDFs were compiled from LaTeX sources. Gold annotations for 15 labels were derived automatically with pylatexenc and then manually checked. Testing 13 open-source extractors gives a mixed picture. Specialised tools such as GROBID and Science Parse handle titles, authors and references well. Equations, lists, footers and especially tables remain difficult for many systems. The data, annotations and a cleanly designed evaluation toolkit are public. The benchmark can be extended with any paper that has LaTeX source available.

Key points

  • Corpus: 100 arXiv papers from 2009–2025, spanning eight arXiv categories (Physics, Mathematics, Computer Science, EESS, etc.), totalling 1 949 pages.
  • Annotation: 15 labels (e.g., Abstract, Author, Affiliation, Email, Equation, Table, Reference, Keyword, Pub Date) are produced in JSON from LaTeX via pylatexenc's LatexNodes2Text. Manual curation took about 15 minutes per article, mostly due to varied equation markup.
  • Critique of prior work: DocBank-based ground truth is built with Pdfplumber, which may favour Tabula. It also lumps keywords, emails and affiliations into paragraph text and annotates only some pages per document.
  • Results: GROBID reaches F1 0.955 on references and 0.975 on titles, and Science Parse reaches 0.985 on titles. Table extraction is poor for Camelot (0.146), PyMuPDF (0.191), Pdfplumber (0.237) and Tabula (0.314), and better for Unstructured (0.785) and Docling (0.741).
  • Availability: All materials are released at github.com/TNoahT/eSciBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Evaluation & benchmarks

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4sxku4i2piqq

@inproceedings{taillon2026escibench,
  title = {{eSciBench: An Extensible Scientific PDF Extraction Benchmark}},
  author = {Taillon, Noah Tremblay and Langlais, Phillippe},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4sxku4i2piqq},
  url = {https://doi.org/10.63317/4sxku4i2piqq}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Narayan S. Adhikari and Shradha Agarwal. 2025. A Comparative Study of PDF Parsing Tools Across Diverse Document Categories. find
  2. Manuel Aristarán and Mike Tigas. 2013. Introducing Tabula. https://source.opennews.org/articles/introducing-tabula/. ↗
  3. Artifex Software, Inc. 2025. MuPDF. arXiv. 1991. arxiv: Open access to scholarly articles. Accessed: 2025-10-20. find
  4. Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, Lokesh Mishra, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and… find
  5. Hannah Bast and Claudius Korzen. 2017. A Benchmark and Evaluation for Text Extraction from PDF. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–10, Toronto, ON, Canada. IEEE. find
  6. Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. Nougat: Neural Optical Understanding for Academic Documents. find
  7. Andy Byers and Ivonne Lujano. 2023. Article and journal metadata - OA Journals Toolkit. find
  8. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Trans. Intell. Syst. Technol., 2(3). find
  9. Glyph & Cog, LLC. 2004. Pdftotext(1). https://linux.die.net/man/1/pdftotext. ↗
  10. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV). find our card
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. find our card
  12. Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. 2020a. Table-Bank: Table benchmark for image-based table detection and recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1918–1925, Marseille, France. European Language Resources Ass… find
  13. Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. 2020b. DocBank: A Benchmark Dataset for Document Layout Analysis. find
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  15. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. find our card
  16. Patrice Lopez. 2009. Grobid: Combining automatic bibliographic data recognition and term extraction for scholarly publications. In Proceedings of the 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), volume 5714 of Lecture Notes in Computer Science, page… find
  17. Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. find
  18. Norman Meuschke, Apurva Jagdale, Timo Spinde, Jelena Mitrović, and Bela Gipp. 2023. A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents. volume 13972, pages 383–405. find
  19. Shivam Naik, Khurram Azeem Hashmi, Alain Pagani, Marcus Liwicki, Didier Stricker, and Muhammad Zeshan Afzal. 2022. Investigating attention mechanism for page object detection in document images. 12:7486. find
  20. Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. 2022. TableFormer: Table Structure Understanding with Transformers. find
  21. Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. 2025. OmniDocBench: Benchmarking Diverse PDF Document Parsing wi… find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  23. Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter W. J. Staar. 2022. DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3743– 3751. find
  24. Python Software Foundation. 2025. The Element-Tree XML API. find
  25. Štěpán Šimsa, Milan Šulc, Michal Uřičář, Yash Patel, Ahmed Hamdi, Matěj Kocián, Matyáš Skalický, Jiří Matas, Antoine Doucet, Mickaël Coustaty, and Dimosthenis Karatzas. 2023. DocILE Benchmark for Document Information Localization and Extraction. find
  26. The Apache Software Foundation. 2025. Apache PDFBox \| A Java PDF Library. https://pdfbox.apache.org/. ↗
  27. Dominika Tkaczyk, Pawel Szostek, Mateusz Fedoryszak, Piotr Jan Dendek, and Lukasz Bolikowski. 2015. CERMINE. International Journal on Document Analysis and Recognition (IJDAR), 18(4):317–335. find
  28. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern,… find our card
  29. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. CoRR, abs/1804.07036. — [arXiv:1804.07036](https://arxiv.org/abs/1804.07036) find
  30. Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. 2024. DETRs Beat YOLOs on Real-time Object Detection. find
  31. Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: Data, model, and evaluation. In European Conference on Computer Vision, pages 564–580, Glaskow, UK. find
  32. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. PubLayNet: Largest dataset ever for document layout analysis. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taillon-2026-escibench-extensible-scientific-pdf-extraction/