Elamur Research⌕
☰
home / papers / 2025 / Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents

Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents

Paper · 2025 · Samuel Backer and Louis Hyman

Summary

Backer and Hyman ask how to judge LLM-assisted OCR for historians, and whether cheaper correction pipelines change research outcomes. Their test bed is the Samuel Gompers correspondence at the Library of Congress: clean scans of faded letter-press copies, about 20,000 of which volunteers have transcribed. They compare several systems against these human transcriptions:

  • Tesseract as the baseline;
  • vision-to-text LLaMA 3.2 and ChatGPT-4o;
  • text-only ChatGPT-4o used to correct Tesseract output;
  • local BART and ByT5 correctors, fine-tuned on either human ("gold") or LLM-produced ("silver") targets.

Vision models win on every measure. By CER/WER, the fine-tuned correctors look worse than raw Tesseract. By precision, NER results and BERT embedding similarity, however, they clearly beat Tesseract and come close to the flagship models. Silver training data works almost as well as gold, and a large silver set outperforms a smaller gold one.

The authors draw two conclusions. Edit-distance metrics can hide gains that matter for humanities work, so evaluation should follow the task. And machine-made transcriptions could replace much crowdsourced labor for institutions with limited budgets.

Key points

  • Data: the Gompers collection holds about 500,000 letters. The authors used 10,000 crowd-transcribed letters (from the "We The People" project) for training and 1,000 for testing; letters average 134.4 words.
  • Text-only ChatGPT-4o correction:
    • WER fell 42% on average and 68% at the median versus Tesseract.
    • CER fell 15% on average and 41% at the median.
    • This contradicts earlier pessimistic findings that used ChatGPT-4.
  • BART and ByT5 correctors by CER/WER:
    • Trained on 100, 1,000 and 10,000 pairs, they mostly scored below Tesseract.
    • Only the median WER of the 10K models improved.
    • The best ByT5 models still had error rates above 80%.
    • At 10K examples, gold- and silver-trained models were within 3% on median CER and WER.
  • Precision:
    • The best corrector, BART trained on 10K LLaMA transcriptions, reached 92.6, against 95.6 for ChatGPT-4o and 96.5 for LLaMA.
    • The best fine-tuned models improved precision by more than 30% over Tesseract.
  • Downstream tasks:
    • With spaCy NER, the vision models beat Tesseract by 157%, with a 14% gap to the best fine-tuned model.
    • ByT5 slightly beat BART on NER, possibly because BART tends to hallucinate.
    • On BERT cosine similarity, the top trained models were 5% above Tesseract and within 0.5% of a flagship multimodal model.

Interesting ideas

  • I1. Strong but costly models can label a large training set so that a small local model inherits much of their skill; more silver data outweighed less gold data.
  • I2. Pick the metric by which error is costlier: historians accept gaps but not invented content, so precision fits them better than edit distance.
  • I3. An upstream change can look harmful on surface metrics yet help downstream, so it should also be judged by its effect on later stages.
  • I4. A model that rewrites fluently can score well overall while hallucinating, so a fabrication-sensitive check should sit beside aggregate scores.
  • I5. Cosine similarity of sentence embeddings between gold and system output was used as a cheap proxy for whether meaning survived the edits.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers or an LLM teacher produce silver trees, quantity can partly compensate for imperfect labels, which justifies scaling silver sets before buying more gold annotation.
  • I2 (world): score extracted entities, states and holdings mainly on whether asserted facts are true, and penalize fabricated facts more heavily than missed ones.
  • I3 (tokens): evaluate normalization or tokenizer changes on noisy input by whether the resulting UD trees and lemmas match gold, not only by character-level differences.
  • I4 (back): test the text → graph → text compressor by checking that each regenerated statement is supported by the source graph, not just by how similar the surface text looks.
  • I5 (learn): add a meaning-preservation check as an extra signal in evaluation runs, alongside negative controls, to see when surface errors do not change content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of MaineOrono · United States2 works · 11 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Samuel Backer and Louis Hyman. 2025. Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents. Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.nlp4dh-1.21

@inproceedings{backer2025bootstrapping,
  title = {{Bootstrapping AI: Interdisciplinary Approaches to Assessing OCR Quality in English-Language Historical Documents}},
  author = {Samuel Backer and Louis Hyman},
  booktitle = {Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities},
  year = {2025},
  doi = {10.18653/v1/2025.nlp4dh-1.21},
  url = {https://doi.org/10.18653/v1/2025.nlp4dh-1.21}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Kunga Bang. 2024. Exploring generative large language models for post-ocr enhancement of historical texts. find
  3. Angel Beshirov, Milena Dobreva, Dimitar Dimitrov, Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2024. Post-ocr text correction for bulgarian historical documents. arXiv preprint arXiv:2409.00527. ↗
  4. Callum William Booth, Alan Thomas, and Robert Gaizauskas. 2024. Bln600: A parallel corpus of machine/human transcribed nineteenth century newspaper texts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2440–2446. find
  5. Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Frédéric Kaplan. 2024. Post-correction of historical text transcripts with large language models: An exploratory study. LaTeCH-CLfL 2024, pages 133–159. find
  6. Jonathan Bourne. 2024. Clocr-c: Context leveraging ocr correction with pre-trained language models. arXiv preprint arXiv:2408.17428. ↗
  7. Ryan Cordell. 2017. " q i-jtb the raven": Taking dirty ocr seriously. Book History, 20(1):188–225. find
  8. Florian Debaene, Aaron Maladry, Els Lefever, and Veronique Hoste. 2025. Evaluating transformers for ocr post-correction in early modern dutch theatre. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10367–10374. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  11. Alex Ghiriti, Wolfgang Göderle, and Roman Kern. 2024. Exploring the capabilities of gpt4-vision as ocr engine. In International Conference on Theory and Practice of Digital Libraries, pages 3–12. Springer. find
  12. Jo Guldi. 2023. The dangerous art of text mining: A methodology for digital history. Cambridge University Press. ↗
  13. Arthur Hemmer, Mickaël Coustaty, Nicola Bartolo, and Jean-Marc Ogier. 2024. Confidence-aware document ocr error detection. In International Workshop on Document Analysis Systems, pages 213–228. Springer. find
  14. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  15. Mark Humphries. 2023. History and generative ai. Teaching History, 57(3):4–9. find
  16. Mark Humphries, Lianne C Leddy, Quinn Downton, Meredith Legace, John McConnell, Isabella Murray, and Elizabeth Spence. 2024. Unlocking the archives: Using large language models to transcribe handwritten historical documents. arXiv preprint arXiv:2411.03340. ↗
  17. Jenna Kanerva, Cassandra Ledins, Siiri Käpyaho, and Filip Ginter. 2025. Ocr error post-correction with llms in historical documents: No free lunches. arXiv preprint arXiv:2502.01205. ↗
  18. Seorin Kim, Julien Baudru, Wouter Ryckbosch, Hugues Bersini, and Vincent Ginis. 2025. Early evidence of how llms outperform traditional systems on ocr/htr tasks for historical records. arXiv preprint arXiv:2501.11623. ↗
  19. Mike Lewis. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  20. Lucian Li. 2024. Handwriting recognition in historical documents with multimodal llm. arXiv preprint arXiv:2410.24034. Library of Congress, Manuscript Division. n.d. American Federation of Labor Records. https://crowd.loc.gov/campaigns/afl/. ↗
  21. Laura Manrique-Gómez, Tony Montes, Arturo Rodríguez-Herrera, and Rubén Manrique. 2024. Historical ink: 19th century latin american spanish newspaper corpus with llm ocr correction. arXiv preprint arXiv:2407.12838. ↗
  22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.
  23. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. ↗
  24. James Zhang, Wouter Haverals, Mary Naydan, and Brian W Kernighan. 2024. Post-ocr correction with openai’s gpt models on challenging english prosody texts. In Proceedings of the ACM Symposium on Document Engineering 2024, pages 1–4. find
  25. Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of ocr evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing, pages 13–18. find
  26. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830. find
  27. Christophe Rigaud, Antoine Doucet, Mickaël Coustaty, and Jean-Philippe Moreux. 2019. Icdar 2019 competition on post-ocr text correction. In 2019 international conference on document analysis and recognition (ICDAR), pages 1588–1593. IEEE. find
  28. Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE. ↗
  29. Myriam C Traub, Jacco Van Ossenbruggen, and Lynda Hardman. 2015. Impact analysis of ocr quality on research tasks in digital archives. In Research and Advanced Technology for Digital Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Poznań, Poland, September 14-18, 2015, Proceedings 19, pages 252– 263. Springer. find
  30. Daniel van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020. Assessing the impact of ocr quality on downstream nlp tasks. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 1: ARTIDIGH, pages 484–496. INSTICC, SciTePress. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/backer-2025-bootstrapping-ai-interdisciplinary-approaches-assessing/