Elamur Research⌕
☰
home / papers / 2026 / OCRTurk: A Comprehensive OCR Benchmark for Turkish

OCRTurk: A Comprehensive OCR Benchmark for Turkish

Paper · 2026 · Deniz Yılmaz, Evren Ayberk Munis, Cagri Toraman et al.

Summary

Most document-parsing benchmarks focus on English and other high-resource languages. For Turkish, the only earlier OCR benchmark used synthetic text drawn from a single book and scored raw text alone. OCRTurk fills this gap with a real-world evaluation set of 180 Turkish PDF pages. The pages are balanced across four document types (academic papers, theses, non-academic documents, slideshows) and three difficulty tiers, which are defined by how complex the layout is.

Ground truth is stored as Markdown, with tables in HTML, equations in LaTeX and figures as PNG. Annotators drafted it with help from LLMs and then corrected it by hand, character by character. Seven OCR systems were tested using element-wise metrics, mainly Normalized Edit Distance (NED).

PaddleOCR performed best overall and led at every difficulty level. Other systems were strong on specific elements. Non-academic documents were the easiest type, and slideshows were the hardest because their layouts are irregular. The benchmark and evaluation scripts are public, so Turkish-specific parsing can be checked before deploying a model.

Key points

  • The dataset has 180 pages: 45 per document category and 60 per difficulty level (15 per cell). Sources include arXiv, DergiPark, YÖK TEZ, MEB OGM Materyal and Ankara University open course material.
  • Difficulty tiers:
    • Easy: pages containing only text.
    • Medium: text plus a single simple element (a one-line equation, a plain table, or a figure without sub-figures).
    • Hard: anything more complex.
  • The evaluation covered seven systems: PaddleOCR, OlmOCR2, Docling, Nvidia Nemotron v1.1, DeepSeekOCR, NanonetsOCR2 and HuanyanOCR.
  • Results by model:
    • PaddleOCR led most element metrics except figures, and it was the only model above average in all four categories.
    • DeepSeekOCR was best at figure extraction.
    • HuanyanOCR was best at Turkish character sensitivity.
    • NanonetsOCR2 was best on equations but worst on tables.
  • Common errors included mishandled Turkish special characters, equations written in text mode or with wrong variables, and tables that were skipped or read as plain text. The resource is available at github.com/metunlp/ocrturk.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Middle East Technical UniversityAnkara · Türkiye9 works · 1,541 citations🎓Politecnico di TorinoTurin · Italy11 works · 18 citations

Also on the byline: Roketsan.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Deniz Yılmaz, Evren Ayberk Munis, Cagri Toraman, Süha Kağan Köse, Burak Aktaş, Mehmet Can Baytekin and Bilge Kaan Görür. 2026. OCRTurk: A Comprehensive OCR Benchmark for Turkish. Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigturk-1.16

@inproceedings{yilmaz2026ocrturk,
  title = {{OCRTurk: A Comprehensive OCR Benchmark for Turkish}},
  author = {Deniz Yılmaz, Evren Ayberk Munis, Cagri Toraman, Süha Kağan Köse, Burak Aktaş, Mehmet Can Baytekin and Bilge Kaan Görür},
  booktitle = {Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.sigturk-1.16},
  url = {https://doi.org/10.18653/v1/2026.sigturk-1.16}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. A. Al-Zubaidi, M. M. Mijwil, and A. Sh. Alsaadi. 2019. Two-dimensional optical character recognition of mouse drawn in Turkish capital letters using multi-layer perceptron classification. Journal of Southwest Jiaotong University, 54(4). \#\# Ethical Considerations https://chatgpt.com/ https://gemin… ↗
  2. OCRTurk is constructed using publicly accessible documents from platforms, such as ArXiv, Dergi-Avinash Anand, Raj Jaiswal, Pijush Bhuyan, Mohit Gupta, Siddhesh Bangar, Md. Modassir Imam, Rajiv Ratn Shah, and Shin’ichi Satoh. 2023. TC-OCR: TableCraft OCR for efficient detection & recognition of tabl… find
  3. Harsh Lunia, Ajoy Mondal, and C. V. Jawahar. 2023. IndicSTR12: A Dataset for Indic Scene Text Recognition, page 233–250. Springer Nature Switzerland. find
  4. G. E. Bartos, Y. Hoşcan, A. Kauer, and É. Hajnal. 2020. A multilingual handwritten character dataset: T-h-e dataset. Acta Polytechnica Hungarica, 17(9):141– 160. find
  5. Rachit S. Munjal, Arun D. Prabhu, Nikhil Arora, Sukumar Moharana, and Gopi Ramena. 2021. Stride: Scene text recognition in-device. In 2021 International Joint Conference on Neural Networks (IJCNN), page 1–8. IEEE. find
  6. Y. Du, P. Chen, X. Ying, and Z. Chen. 2025. Docptbench: Benchmarking end-to-end photographed document parsing and translation. find
  7. L. Fu, B. Yang, Z. Kuang, J. Song, Y. Li, L. Zhu, Q. Luo, X. Wang, H. Lu, M. Huang, Z. Li, G. Tang, B. Shan, C. Lin, Q. Liu, B. Wu, H. Feng, H. Liu, C. Huang, and J. Tang. 2024. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. find
  8. S. Fu, N. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola. 2023. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. Preprint, arXiv:2306.09344. — [arXiv:2306.09344](https://arxiv.org/abs/2306.09344) find
  9. M. G. Öztürk, Durmus Ö. Sahin, and Erdal Kiliç. 2025. Turkish optical character recognition under the lens: A systematic review of language-specific challenges, dataset scarcity, and open-source limitations. IEEE Access, 13:168977–168997. find
  10. Dilek Z. Hakkani-Tür, Kemal Oflazer, and Gökhan Tür. 2002. Statistical morphological disambiguation for agglutinative languages. Computers and the Humanities, 36(4):381–410. find
  11. A. Heakl, A. Sohail, M. Ranjan, R. Hossam, G. Shazan Ahmad, Mohamed El-Geish, Omar Maher, Zhiqiang Shen, Fahad Khan, and Salman Khan. 2025. Kitab-bench: A comprehensive multi-domain benchmark for Arabic ocr and document understanding. Preprint, arXiv:2502.14949.
  12. Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE. find
  13. F. Kizilirmak. 2022. Offline handwriting recognition using deep learning with emphasis on data augmentation effects. Master’s thesis, Sabancı University. find
  14. M. Kuncan, E. Vardar, K. Kaplan, and H. M. Ertunç. 2020. Turkish handwriting recognition system using multi-layer perceptron. Journal of Mechatronics and Artificial Intelligence in Engineering. find
  15. T.-Y. Lin, M. Maire, S. Belongie, and 1 others. 2014. Microsoft COCO: Common Objects in Context. Springer International Publishing.
  16. J. Memon, M. Sami, and R. A. Khan. 2020. Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr). Preprint, arXiv:2001.00139. — [arXiv:2001.00139](https://arxiv.org/abs/2001.00139) find
  17. Kemal Oflazer. 2014. Turkish and its challenges for language processing. Language resources and evaluation, 48(4):639–653. find
  18. Berke Oral, Erdem Emekligil, Seçil Arslan, and Gülşen Eryiǧit. 2020. Information extraction from text intensive and visually rich banking documents. Information Processing and Management, 57(6). find
  19. L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Zhao, M. Jiang, X. Zhao, J. Shi, F. Wu, P. Chu, M. Liu, Z. Li, C. Xu, B. Zhang, B. Shi, Z. Tu, and C. He. 2024. Omnidocbench: Benchmarking diverse PDF document parsing with comprehensive annotations. find
  20. Smita Pallavi, Raj Ratn Pranesh, and Sumit Kumar. 2020. A conglomerate of multiple ocr table detection and extraction. Preprint, arXiv:2010.08591. — [arXiv:2010.08591](https://arxiv.org/abs/2010.08591) find
  21. Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019. Cord: A consolidated receipt dataset for post-ocr parsing. find
  22. Parshva Dhilankumar Patel. 2025. Design and implementation of an ocr-powered pipeline for table extraction from invoices. Preprint, arXiv:2507.07029. — [arXiv:2507.07029](https://arxiv.org/abs/2507.07029) find
  23. J. Poznanski, A. Rangapur, J. Borchardt, J. Dunkelberger, R. Huff, D. Lin, C. Wilhelm, K. Lo, and L. Soldaini. 2025. olmocr: Unlocking trillions of tokens in PDFs with vision-language models.
  24. Cagri Sayallar, Ahmet Sayar, and Nurcan Babalik. 2023. An ocr engine for printed receipt images using deep learning techniques. International Journal of Advanced Computer Science and Applications, 14(2). find
  25. A. Şevik. 2019. Derin öğrenme ile Türkçe font ve karakter tanıma. Master’s thesis, Düzce University. find
  26. Çağrı Toraman, Ahmet Kaan Sever, Ayşe Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Osman Gürlek, Sarp Kantar, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Ol… — TurkBench: A Benchmark for Evaluating Turkish Large Language Models
  27. Elif Ecem Umutlu, Ayse Aysu Cengiz, Ahmet Kaan Sever, Seyma Erdem, Burak Aytan, Busra Tufan, Abdullah Topraksoy, Esra Darıcı, and Cagri Toraman. 2025. Evaluating the quality of benchmark datasets for low-resource languages: A case study on Turkish. In Proceedings of the Fourth Workshop on Generation… find
  28. C. Wickrema, S. Leary, S. Sarkar, M. Giglio, E. Bianchi, E. Mace, and M. Twardowski. 2025. Benchmarking image similarity metrics for novel view synthesis applications. Preprint, arXiv:2506.12563. — [arXiv:2506.12563](https://arxiv.org/abs/2506.12563) find
  29. Equations We observe incorrect variables and missing conditions for the partial expressions in the models. For example, The expression  b P h −1   a \< b ise,  Z  k= a f (kh)h, b h f (t)∆t = a 0,  P a −1    − h f (kh)h, k= h b Z. Yang, J. Tang, Z. Li, P. Wang, J. Wan, H. Zhong, X. Liu, M. Ya… find
  30. Y. Yılmaz, E. G. Hanoğlu, A. G. Özkan, and K. Özto-prak. 2025. Benchmarking OCR and vision-language models for Turkish text recognition: A comprehensive evaluation using synthetic data. find
  31. Ahmed Zeer, Eren Dogan, Yusuf Erdem, Elif İnce, Osama Shbib, M. Egemen Uzun, Atahan Uz, M. Kaan Yuce, H. Toprak Kesgin, and M. Fatih Amasyali. 2024. Cosmos-llava: Chatting with the visual. In 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), pages 1–7. IEEE. find
  32. Yufeng Zhong, Zhixiong Zeng, Lei Chen, Longrong Yang, Liming Zheng, Jing Huang, Siqi Yang, and Lin Ma. 2025. Doctron-formula: Generalized formula recognition in complex and structured scenarios. Preprint, arXiv:2508.00311. — [arXiv:2508.00311](https://arxiv.org/abs/2508.00311) find
  33. A to ignore whitespaces and inline math equations as well. For example, "m ve n’nin reel sayı olması" is interpreted as "mven’nin reel sayı olması". is interpreted as Z b a f (t)∆t = a = b ise, a \> b ise  P b −1 h    k= a f (kh)h, h 0,  a   − P h −1 a f (kh)h, k= h find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yilmaz-2026-ocrturk-comprehensive-ocr-benchmark-turkish/