Elamur Research⌕
☰
home / papers / 2022 / An End-to-End OCR Framework for Robust Arabic-Handwriting Recognition using a Novel Transformers-based Model and an Innovative 270 Million-Words Multi-Font Corpus of Classical Arabic with Diacritics

An End-to-End OCR Framework for Robust Arabic-Handwriting Recognition using a Novel Transformers-based Model and an Innovative 270 Million-Words Multi-Font Corpus of Classical Arabic with Diacritics

Summary

Recognizing historical Arabic text is still unreliable. Old printings show inconsistent spelling, cursive and shifted glyphs, ink bleed, stains and decorative layouts with marginal notes, and large labeled datasets are scarce. This second-phase study from Helwan University builds a complete OCR pipeline for Arabic text lines. The stages are pre-processing, page and text-line segmentation, sequential image enhancement, transcription, and post-correction during decoding. The authors also build a very large synthetic resource: about 30.5 million text-line images paired with ground truth, covering 270 million words of Classical Arabic with diacritics in 12 fonts. The recognizer drops the usual CNN feature extractor. It uses a BEiT vision transformer as the encoder and a standard Transformer as the decoder. The model was trained on 100,000 line images, far more than the 15,000 used in the first phase, and reaches a character error rate of 4.46%, outperforming convolutional backbones. The work matters because it shows that large rendered corpora plus pure-transformer recognition can make digitizing Arabic heritage texts more practical.

Key points

  • The corpus has roughly 30.5M text-line images and 270M words, with diacritics and 12 fonts. The authors compare it with APTI, the previous largest line-level Arabic set at about 45M words.
  • The architecture is a BEiT encoder with a vanilla Transformer decoder and no CNN backbone. It reaches CER 4.46% after training on 100,000 images.
  • For segmentation, Detectron-2 handled most cases well. YOLO-V5 often missed marginal text and other difficult layout elements.
  • PyTorch Profiler showed that cuDNN convolution was the largest cost (about 29% of CUDA time). Linear-attention variants such as Performer (FAVOR+) therefore gave no real speedup. The 1-cycle learning-rate schedule sped up convergence.
  • Among the decoding strategies tried (greedy, beam search, diverse beam search), Word Beam Search worked best. BibAlex provided compute: 64 nodes, each with two K80 GPUs.

Interesting ideas

  • I1. Generating labeled training data by rendering a large text corpus in many fonts gives unlimited exact ground truth where human annotation is scarce. — For Mova (learn): Mova could build synthetic, perfectly labeled sentences (for example, templated constructions with known frames) as negative and positive controls alongside silver data.
  • I2. The authors profiled before optimizing and found that the attention speedups targeted a part that was not the bottleneck. This shows that measurement should drive optimization choices. — For Mova (tree): Profile gparse layer by layer before adopting efficient-attention tricks, since tokenization, feature building or data loading may dominate instead.
  • I3. Augmentations were designed to imitate real defects in the target domain, such as rotated lines and marginal-note placement, rather than generic noise. — For Mova (learn): Perturb training sentences in realistic ways, such as typos, missing punctuation, or casual-register rewrites, so the parser stays robust on texts like fables and school tests.
  • I4. Post-correction happens at decoding time through Word Beam Search, which constrains outputs to plausible words instead of fixing them in a separate pass. — For Mova (tokens): Lemma and morphology decoding could be restricted to forms licensed by the lexicon, with a traceable reason whenever a candidate is rejected.
  • I5. The system is a staged pipeline (segment, enhance, recognize, correct) in which each module can be swapped and compared separately, as with YOLO versus Detectron-2. — For Mova (answer): Evaluate each reader stage in isolation with stage-specific metrics, so a failed answer can be traced to the step that caused it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2208.11484 · arXiv 2208.11484

@misc{mostafa2022end,
  title = {{An End-to-End OCR Framework for Robust Arabic-Handwriting Recognition using a Novel Transformers-based Model and an Innovative 270 Million-Words Multi-Font Corpus of Classical Arabic with Diacritics}},
  author = {Mostafa, Aly and Mohamed, Omar and Ashraf, Ali and Elbehery, Ahmed and Jamal, Salma and Salah, Anas and Ghoneim, Amr S.},
  year = {2022},
  doi = {10.48550/arxiv.2208.11484},
  eprint = {2208.11484},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.11484}
}

References

52 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Addakiri, K., Bahaj, M., 2012. On-line handwritten arabic character recognition using artificial neural network. International Journal of Computer Applications 55. find
  2. Ahmad, R., Naz, S., Afzal, M.Z., Rashid, S.F., Liwicki, M., Dengel, A., 2020. A deep learning based arabic script recognition system: benchmark on khat. Int. Arab J. Inf. Technol. 17, 299–305. find
  3. Alginahi, Y., 2010. Preprocessing techniques in character recognition. Character recognition 1, 1–19. find
  4. Alrabiah, M., Salman, A., Atwell, E., 2014. King saud university corpus of classical arabic (ksucca). Department of Computer Science, King Saud University . find
  5. Althobaiti, H., Lu, C., 2017. A survey on arabic optical character recognition and an isolated handwritten arabic character recognition algorithm using encoded freeman chain code, in: 2017 51st Annual conference on information sciences and systems (CISS), IEEE. pp. 1–6. find
  6. Altwaijry, N., Al-Turaiki, I., 2021. Arabic handwriting recognition system using convolutional neural network. Neural Computing and Applications 33, 2249–2261. find
  7. Ayesh, M., Mohammad, K., Qaroush, A., Agaian, S., Washha, M., 2017. A robust line segmentation algorithm for arabic printed text with diacritics. find
  8. Electronic Imaging 2017, 42–47. find
  9. Baevski, A., Zhou, H., Mohamed, A., Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477 . ↗ our card
  10. Bahdanau, D., Cho, K., Bengio, Y., 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . ↗ our card
  11. Barakat, B., Droby, A., Kassis, M., El-Sana, J., 2018. Text line segmentation for challenging handwritten document images using fully convolutional network, in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE. pp. 374–379. find
  12. Bassil, Y., Alwani, M., 2012. Ocr post-processing error correction algorithm using google online spelling suggestion. arXiv preprint arXiv:1204.0191 . ↗
  13. Bieniecki, W., Grabowski, S., Rozenberg, W., 2007. Image preprocessing for improving ocr accuracy, in: 2007 international conference on perspective technologies and methods in MEMS design, IEEE. pp. 75–80. find
  14. Boiangiu, C.A., Cananau, D.C., Petrescu, S., Moldoveanu, A., 2009. Ocr post processing based on character pattern matching. Annals of DAAAM & Proceedings . find
  15. Bui, Q.A., Mollard, D., Tabbone, S., 2017. Selecting automatically pre-processing methods to improve ocr performances, in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), IEEE. pp. 169–174. find
  16. Chen, K.N., Chen, C.H., Chang, C.C., 2012. Efficient illumination compensation techniques for text images. Digital Signal Processing 22, 726–733. find
  17. Chickering, D.M., 2002. Optimal structure identification with greedy search. Journal of machine learning research 3, 507–554. find
  18. Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al., 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 . ↗
  19. Clausner, C., Antonacopoulos, A., Mcgregor, N., Wilson-Nunn, D., 2018. Icfhr 2018 competition on recognition of historical arabic scientific manuscripts– rasm2018, in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE. pp. 471–476. find
  20. Darwish, S.M., Elzoghaly, K.O., 2020. An enhanced offline printed arabic ocr model based on bio-inspired fuzzy classifier. IEEE Access 8, 117770–117781. find
  21. Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . ↗
  22. Dreuw, P., Rybach, D., Gollan, C., Ney, H., 2009. Writer adaptive training and writing variant model refinement for offline arabic handwriting recognition, in: 2009 10th International Conference on Document Analysis and Recognition, IEEE. pp. 21–25. of Encyclopaedia, T.E., . Arabic language. URL: https://www.britannica.com/topic/Arabic-language. ↗
  23. Fasha, M., Hammo, B., Obeid, N., Widian, J., 2020. A hybrid deep learning model for arabic text recognition. arXiv preprint arXiv:2009.01987 . ↗ our card
  24. Graves, A., 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 . ↗
  25. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J., 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, in: Proceedings of the 23rd international conference on Machine learning, pp. 369–376. find
  26. Gruuening, T., Leifert, G., Strauss, T., Labahn, R., 2017. A robust and binarization-free approach for text line detection in historical documents, in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), IEEE. pp. 236–241. find
  27. Haraty, R., 2004. Arabic text recognition. The International Arab Journal of Information Technology . find
  28. Herbert, H., 1982. The history of ocr, optical character recognition. Manchester Center, VT: Recognition Technologies Users Association . find
  29. Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural computation 9, 1735–1780. Jocher, G., Stoken, A., Borovec, J., NanoCode012, Chaurasia, A., TaoXie, Changyu, L., V, A., Laughing, tkianai, yxNONG, Hogan, A., lorenzomammana, AlexWang1900, Hajek, J., Diaconu, L., Marc, Kwon, Y., oleg, wanghaoyang0106, Defretin, Y., Lohia, A., ml5ah, Milanko, B., Fineran, B., Khromov, D., Yiwei, D., Doug, Durgesh, Ingham, F., 2021. alytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations. URL: https://doi.org/10.5281/zenodo.4679653, doi:10.5281/zenodo.4679653. ↗
  30. Kahle, P., Colutto, S., Hackl, G., Mühlberger, G., 2017. Transkribus-a service platform for transcription, recognition and retrieval of historical documents, in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), IEEE. pp. 19–24. find
  31. Khirbat, G., 2017. Ocr post-processing text correction using simulated annealing (opteca), in: Proceedings of the Australasian Language Technology Association Workshop 2017, pp. 119–123. find
  32. Kise, K., 2014. Page segmentation techniques in document analysis. find
  33. Lawgali, A., 2015. A survey on arabic character recognition. International Journal of Signal Processing, Image Processing and Pattern Recognition . find
  34. Lawgali, A., Angelova, M., Bouridane, A., 2013. Hacdb: Handwritten arabic characters database for automatic character recognition, in: European Workshop on Visual Information Processing (EUVIP), IEEE. pp. 255–259. find
  35. LeCun, Y., et al., 1989. Generalization and network design strategies. Connectionism in perspective 19, 18. find
  36. Lee, J., Hayashi, H., Ohyama, W., Uchida, S., 2019. Page segmentation using a convolutional neural network with trainable co-occurrence features, in: 2019 ultr-International Conference on Document Analysis and Recognition (ICDAR), IEEE. pp. 1023–1028. LLC, M., . Caere corporation history URL: http://www.fundinguniverse.com/company-histories/caere-corporation-history/. ↗
  37. Mahmoud, S.A., Ahmad, I., Al-Khatib, W.G., Alshayeb, M., Parvez, M.T., Märgner, V., Fink, G.A., 2014. Khatt: An open arabic offline handwritten text database. Pattern Recognition 47, 1096–1112. find
  38. Mostafa, A., Mohamed, O., Ashraf, A., Elbehery, A., Jamal, S., Khoriba, G., Ghoneim, A.S., 2021. Ocformer: A transformer-based model for arabic handwritten text recognition, in: 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), IEEE. pp. 182–186. find
  39. Neudecker, C., Baierer, K., Federbusch, M., Boenig, M., Würzner, K.M., Hartmann, V., Herrmann, E., 2019. Ocr-d: An end-to-end open source ocr framework for historical printed documents, in: Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage, pp. 53–58. find
  40. Osman, H., Zaghw, K., Hazem, M., Elsehely, S., 2020. An efficient language-independent multi-font ocr for arabic script. arXiv preprint arXiv:2009.09115 . Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al., 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32, 8026–8037. ↗ our card
  41. Pechwitz, M., Maddouri, S.S., Märgner, V., Ellouze, N., Amiri, H., et al., 2002. Ifn/enit-database of handwritten arabic words, in: Proc. of CIFED, Citeseer. pp. 127–136. find
  42. Scheidl, H., Fiel, S., Sablatnig, R., 2018. Word beam search: A connectionist temporal classification decoding algorithm, in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE. pp. 253–258. find
  43. Slimane, F., Ingold, R., Kanoun, S., Alimi, A.M., Hennebert, J., 2009. Database and evaluation protocols for arabic printed text recognition. DIUF-University of Fribourg-Switzerland . find
  44. Smith, L.N., 2017. Cyclical learning rates for training neural networks, in: 2017 IEEE winter conference on applications of computer vision (WACV), IEEE. pp. 464–472. find
  45. Stahlberg, F., Vogel, S., 2016. Qatip–an optical character recognition system for arabic heritage collections in libraries, in: 2016 12th IAPR Workshop on Document Analysis Systems (DAS), IEEE. pp. 168–173. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, ., Polosukhin, I., 2017. Attention is all you need, in: Advances in neural information processing systems, pp. 5998–6008. find
  46. Vijayakumar, A.K., Cogswell, M., Selvaraju, R.R., Sun, Q., Lee, S., Crandall, D., Batra, D., 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424 . ↗ our card
  47. Wang, C., Wu, A., Pino, J., Baevski, A., Auli, M., Conneau, A., 2021. Large-scale self-and semi-supervised learning for speech translation. arXiv preprint arXiv:2104.06678 . ↗
  48. Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H., 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 . ↗
  49. Wiseman, S., Rush, A.M., 2016. Sequence-to-sequence learning as beam-search optimization. arXiv preprint arXiv:1606.02960 . ↗
  50. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R., 2019. Detectron2. https://github.com/facebookresearch/detectron2. ↗
  51. Younes, M., Abdellah, Y., 2015. Segmentation of arabic handwritten text to lines. Procedia Computer Science 73, 115–121. find
  52. Yousef, W.A., Ibrahime, O.M., Madbouly, T.M., Mahmoud, M.A., 2019. Learning meters of arabic and english poems with recurrent neural networks: a step forward for language understanding and synthesis. arXiv preprint arXiv:1905.05700 . ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mostafa-2022-end-end-ocr-framework-robust/