Elamur Research⌕
☰
home / papers / 2026 / Reference-free Evaluation at Inference for NER/NEL over OCRed Historical Texts

Reference-free Evaluation at Inference for NER/NEL over OCRed Historical Texts

Summary

Extracting entities from digitized historical texts is hampered by OCR noise. When systems run on new collections, the gold annotations needed to measure NER and NEL quality are usually absent. OCR metrics such as CER and WER also say little about how much entities themselves are damaged. The authors therefore frame quality estimation as regression: from an input text and a system's entity output, a model predicts the F1 score a reference-based evaluation would assign. LLMs and encoder-based transformers are fine-tuned where gold labels exist and then applied without references at inference. Experiments use HIPE-2020 (English, French, German). The evaluated outputs come from fine-tuned XLM-RoBERTa models run on texts with controlled synthetic noise. Prompted Gemma 27B and standard confidence scores serve as comparisons. The authors report that fine-tuned LLMs give reliable estimates and model contextual and historical uncertainty better than confidence-based measures. The supplied extract omits the numerical results. The method offers a low-cost way to flag archive documents that need re-OCR or correction.

Key points

  • The estimator learns p_θ(x, e) ≈ F1 at sentence or document level rather than per entity, which gives a steadier training signal and reflects overall usability.
  • Noise follows an adapted model from Jaud et al. (2025). Two parameters set the share of corrupted tokens and the number of characters replaced per token (1–4), producing 20 configurations. These were reduced to four levels (L1–L4) for each target: entity tokens, surrounding context, or all tokens.
  • When all tokens are corrupted, WER spans 0.15–0.45 for both FR and DE, with CER reaching 0.37 (FR) and 0.27 (DE). Entity-only corruption keeps WER below about 0.05.
  • In the in-context learning baseline, Gemma 27B returns an F1 value in JSON. For NER it scores BIO tags (pers, loc, org, prod); for NEL it scores Wikidata QIDs.
  • Code, datasets and the joint NER/NEL predictions on corrupted data are publicly released on GitLab.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/24wx62fjkprg

@inproceedings{nguyen2026reference,
  title = {{Reference-free Evaluation at Inference for NER/NEL over OCRed Historical Texts}},
  author = {Nguyen, Tien-Nam and Jatowt, Adam and Hamdi, Ahmed and Coustaty, Mickael and Tran, Thi Hong Hanh and Doucet, Antoine},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/24wx62fjkprg},
  url = {https://doi.org/10.63317/24wx62fjkprg}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Emanuela Boroş, Ahmed Hamdi, Elvys Linhares Pontes, Luis-Adrián Cabrera-Diego, Jose G Moreno, Nicolas Sidere, and Antoine Doucet. 2020. Alleviating digitization errors in named entity recognition for historical documents. In Proceedings of the 24th conference on computational natural language learni… find
  3. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th international conference on computational linguistics, pages 6788–6796. find our card
  4. Guillaume Chiron, Antoine Doucet, Mickaël Coustaty, Muriel Visani, and Jean-Philippe Moreux. 2017. Impact of ocr errors on the use of digital libraries: towards a better access to information. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–4. IEEE. find
  5. Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive entity linking. Transactions of the Association for Computational Linguistics, 10:274–290. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. find
  7. Maud Ehrmann, Ahmed Hamdi, Elvys Linhares Pontes, Matteo Romanello, and Antoine Doucet. 2023. Named entity recognition and classification in historical documents: A survey. ACM Computing Surveys, 56(2):1–47. find
  8. Maud Ehrmann, Matteo Romanello, Simon Clematide, and Antoine Doucet. 2022a. Introducing the HIPE 2022 Shared Task:Named Entity Recognition and Linking in Multilingual Historical Documents. In Proceedings of the 44 d European Conference on IR Research (ECIR 2022), Stavanger, Norway. Lecture Notes in… find
  9. Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, Antoine Doucet, Simon Clematide, Gulielmo Faggioli, Nicola Ferro, Alan Hanbury, and Martin Potthast. 2022b. Extended overview of hipe-2022: Named entity recognition and linking in multilingual historical documents. In CEUR Workshop Proceedings, 3180,… find
  10. Alexandre Gefen. 2014. Les enjeux épistémologiques des humanités numériques. Socio-La nouvelle revue des sciences sociales, pages 61–74. find
  11. Carlos-Emiliano González-Gallardo, Emanuela Boros, Edward Giamphy, Ahmed Hamdi, José G Moreno, and Antoine Doucet. 2023a. Injecting temporal-aware knowledge in historical named entity recognition. In European Conference on Information Retrieval, pages 377–393. Springer. find
  12. Carlos-Emiliano González-Gallardo, Emanuela Boros, Nancy Girdhar, Ahmed Hamdi, Jose G Moreno, and Antoine Doucet. 2023b. Yes but.. can chatgpt identify entities in historical documents? In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 184–189. IEEE. find
  13. Carlos-Emiliano González-Gallardo, Hanh Thi Hong Tran, Ahmed Hamdi, and Antoine Doucet. 2024. Leveraging open large language models for historical named entity recognition. In International Conference on Theory and Practice of Digital Libraries, pages 379–395. Springer. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  15. Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named entity recognition in query. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 267–274, New York, NY, USA. Association for Computing Machinery. find
  16. Ahmed Hamdi, Elvys Linhares Pontes, Nicolas Sidere, Mickaël Coustaty, and Antoine Doucet. 2023. In-depth analysis of the impact of ocr errors on named entity recognition and linking. Natural Language Engineering, 29(2):425–448. find
  17. Tran Thi Hong Hanh, Antoine Doucet, Nicolas Sidere, Jose G Moreno, and Senja Pollak. 2021. Named entity recognition architecture combining contextual and global features. In International Conference on Asian Digital Libraries, pages 264–276. Springer. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  19. Alexandre Jaud, Ahmed Hamdi, Antoine Doucet, Adam Jatowt, and Mickaël Coustaty. 2025. Beyond cer and wer: How does ocr really impact information retrieval? In 2025 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 30–39. find
  20. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. find
  21. Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. 2025. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv:2503.05061. — [arXiv:2503.05061](https://arxiv.org/abs/2503.05061) find
  22. Bruce W Lee and Jason Hyung-Jong Lee. 2023. Prompt-based learning for text readability assessment. arXiv preprint arXiv:2302.13139. find
  23. Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation. arXiv preprint arXiv:2410.20774. find
  24. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579. find our card
  25. Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024. Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993. — [arXiv:2404.15993](https://arxiv.org/abs/2404.15993) find
  26. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13452–13460. find
  27. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. find our card
  28. Ludovic Moncla and Hédi Zeghidi. 2025. Token and span classification for entity recognition in french historical encyclopedias. arXiv preprint arXiv:2506.02872. find
  29. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. find
  30. Thi-Tuyet-Hai Nguyen, Adam Jatowt, Mickaël Coustaty, Nhu-Van Nguyen, and Antoine Doucet. 2019. Deep statistical analysis of ocr errors for effective post-ocr processing. In 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 29–38. find
  31. Hassan S Shavarani and Anoop Sarkar. 2023. Spel: Structured prediction for entity linking. arXiv preprint arXiv:2310.14684. find
  32. Wenjun Sun, Hanh Thi Hong Tran, Carlos-Emiliano González-Gallardo, Mickaël Coustaty, and Antoine Doucet. 2024. Lit: Label-informed transformers on token-based classification. In International Conference on Theory and Practice of Digital Libraries, pages 144–158. Springer. find
  33. Hanh Thi Hong Tran, Nishan Chatterjee, Senja Pollak, and Antoine Doucet. 2024a. Deberta beats behemoths: A comparative analysis of fine-tuning, prompting, and peft approaches on legallensner. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 371–380. find
  34. Hanh Thi Hong Tran, Tien Nam Nguyen, Antoine Doucet, and Senja Pollak. 2024b. L3i++ at semeval-2024 task 8: Can fine-tuned large language model detect multigenerator, multidomain, and multilingual black-box machine-generated text? In Proceedings of the 18th International Workshop on Semantic Evaluat… find
  35. Crina Tudor, Beata Megyesi, and Robert Östling. 2025. Prompting the past: Exploring zero-shot learning for named entity recognition in historical texts using prompt-answering LLMs. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Hu… find
  36. Daniel Vollmers, Hamada Zahera, Diego Moussallem, and Axel-Cyrille Ngonga Ngomo. 2025. Contextual augmentation for entity linking using large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8535–8545. find
  37. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428. find our card
  38. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2019. Scalable zero-shot entity linking with dense entity retrieval. arXiv preprint arXiv:1911.03814. find our card
  39. Amy Xin, Yunjia Qi, Zijun Yao, Fangwei Zhu, Kaisheng Zeng, Xu Bin, Lei Hou, and Juanzi Li. 2024. Llmael: Large language models are good context augmenters for entity linking. arXiv preprint arXiv:2407.04020. — [arXiv:2407.04020](https://arxiv.org/abs/2407.04020) find
  40. Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2022. Global entity disambiguation with bert. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3264–3271. find
  41. Zhen Zhang, Yuhua Zhao, Hang Gao, and Mengting Hu. 2024. Linkner: linking local named entity recognition models to large language models using uncertainty. In Proceedings of the ACM Web Conference 2024, pages 4047–4058. 10. Appendices 10.1. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-reference-free-evaluation-inference-ner/