Elamur Research⌕
☰
home / papers / 2024 / HATFormer: Historic Handwritten Arabic Text Recognition with Transformers

HATFormer: Historic Handwritten Arabic Text Recognition with Transformers

Summary

Archives in the Middle East and North Africa hold many millions of handwritten Arabic pages, mostly from the 19th and early 20th centuries. Current Arabic handwriting recognizers do not make these pages searchable. Arabic is hard for HTR for three reasons: the script is cursive, a letter's shape depends on its position in the word, and small diacritics can change a word's meaning. Labelled historical data is also scarce. The authors build HATFormer on the TrOCR design, with a BEiT vision encoder and a RoBERTa text decoder, and add three Arabic-specific parts:

  • a BlockProcessor that turns long text-line images into encoder input without squashing them horizontally;
  • an Arabic byte-level BPE tokenizer for compact text;
  • a training pipeline that starts with realistic synthetic line images and then moves to real data.

On Muharaf, the largest public historical Arabic handwriting dataset, HATFormer reaches 8.6% CER. This is about 51% better than the strongest earlier baseline. The work shows that an English-centred HTR recipe can be adapted to a low-resource script, which helps digitisation and search in the digital humanities.

Key points

  • The architecture is a transformer-only encoder–decoder: BEiT encodes 16×16 px patches and RoBERTa decodes over an Arabic BBPE vocabulary, with beam search at the end.
  • CER is 8.6% on Muharaf (public) and 4.2% on a larger private dataset. For scale, Muharaf has just over 1,600 pages, KHATT about 1,000 pages, and MADCAT over 40,000 pages, but MADCAT is not historical.
  • The BlockProcessor beats two alternatives:
    • TrOCR's fixed 384×384 resize, which loses horizontal detail;
    • Pix2Struct's aspect-preserving scaling, which makes a patch cover a character fragment in one image but a whole character in another.
  • Ablations:
    • removing the synthetic Stage-1 fine-tuning raises CER by 4.2 points;
    • training from random weights gives 86.0% CER, while starting from TrOCR's English checkpoint works far better.
  • Known limits: the model depends on good line segmentation, has trouble with strongly slanted lines, and is expensive to train. LoRA is suggested as future work. The authors also release an OCR Error Diagnostic App and a synthetic data generator.

Interesting ideas

  • I1. Keeping each input unit at the same scale matters more than keeping the whole input's shape: a patch should mean roughly the same thing in every image, so the model does not have to relearn what a unit is from one example to the next. — For Mova (tokens): Make the tokenizer produce units of stable granularity whatever the sentence length or source, so downstream features keep one meaning everywhere.
  • I2. Even a pretrained model for a very different script (English OCR into Arabic HTR) is a far better start than random initialisation, which collapsed to 86% CER here. — For Mova (learn): Initialise gparse from a pretrained checkpoint even when its domain or dialect does not match, and keep a random-init run as a negative control to show what pretraining contributes.
  • I3. Training happens in stages: first on plentiful realistic synthetic data, then on the small real set, with a measured drop in error from the synthetic stage. — For Mova (learn): Train first on large silver data agreed by witness parsers, then fine-tune on gold treebanks, and ablate the silver stage so its value is measured rather than assumed.
  • I4. Errors are sorted into linguistically motivated groups (context-free character substitutions, context-dependent ones, diacritics), and the score is reported under progressively looser normalisation to separate harmless mistakes from serious ones. — For Mova (tree): Report LAS under graded relaxations (for example ignoring label subtypes or UD dialect differences) to show which parser errors actually matter for frames.
  • I5. A dedicated diagnostic app lets people browse and debug common recognition errors, which turns raw metrics into something a human can explain. — For Mova (learn): A similar viewer that lines up disagreements between parsers and frame errors would help find candidates for self-induced repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

4 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.02179 · arXiv 2410.02179

@misc{chan2024hatformer,
  title = {{HATFormer: Historic Handwritten Arabic Text Recognition with Transformers}},
  author = {Chan, Adrian and Mijar, Anupam and Saeed, Mehreen and Wong, Chau-Wai and Khater, Akram},
  year = {2024},
  doi = {10.48550/arxiv.2410.02179},
  eprint = {2410.02179},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.02179}
}

References

60 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abandah, G. and Al-Hourani, A. Challenges and preprocessing recommendations for MADCAT dataset of handwritten Arabic documents. In International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, pp. 1–9. IEEE, 2018. find
  2. Abbas, M. and Smaili, K. Comparison of topic identification methods for Arabic language. In International Conference on Recent Advances in Natural Language Processing, pp. 14–17, 2005. find
  3. Abbas, M., Smaili, K., Berkani, D., et al. Evaluation of topic identification methods on Arabic corpora. Journal of Digital Information Management, 9(5):185–192, 2011. find
  4. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2015. find our card
  5. Balm, G. An introduction to optical character reader considerations. Pattern Recognition, 2(3):151–166, 1970. find
  6. Bao, H., Dong, L., Piao, S., and Wei, F. BEiT: BERT pre-training of image transformers. International Conference on Learning Representations, 2022. find
  7. Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S. Introducing our multimodal models. https://www.adept.ai/blog/fuyu-8b, 2023. ↗
  8. Coquenet, D., Chatelain, C., and Paquet, T. DAN: A segmentation-free document attention network for handwritten document recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8227– 8243, 2023. find
  9. Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I. M., et al. Patch n’Pack: NaViT, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024. find
  10. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021. find
  11. Fadeeva, A., Schlattner, P., Maksai, A., Collier, M., Kokiopoulou, E., Berent, J., and Musat, C. Representing online handwriting for recognition in large vision-language models. arXiv preprint arXiv:2402.15307, 2024. ↗
  12. Faizullah, S., Ayub, M. S., Hussain, S., and Khan, M. A. A survey of OCR in Arabic language: Applications, techniques, and challenges. Applied Sciences, 13(7):4584, 2023. find
  13. Fujitake, M. DTrOCR: Decoder-only transformer for optical character recognition. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8025–8035, 2024. find
  14. Graves, A. Sequence transduction with recurrent neural networks. International Conference of Machine Learning Workshop, 2012. find our card
  15. Graves, A. and Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM networks. In IEEE International Joint Conference on Neural Networks, volume 4, pp. 2047–2052, 2005. find
  16. Graves, A. and Schmidhuber, J. Offline handwriting recognition with multidimensional recurrent neural networks. Advances in Neural Information Processing Systems, 21, 2008. find
  17. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. International Conference on Machine Learning, pp. 369– 376, 2006. find
  18. Graves, A., Fernández, S., and Schmidhuber, J. Multidimensional recurrent neural networks. In International Conference on Artificial Neural Networks, pp. 549–558, 2007. find
  19. Graves, A., Mohamed, A.-r., and Hinton, G. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 6645–6649, 2013. find
  20. Grosicki, E., Carré, M., Geoffrois, E., Augustin, E., Preteux, F., and Messina, R. RIMES. https://doi.org/10.5281/zenodo.10812725, 2024. ↗
  21. Hao, Y., Dong, L., Wei, F., and Xu, K. Visualizing and understanding the effectiveness of BERT. In Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pp. 4143–4152. Association for Computational Linguistics, 2019. find
  22. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016. find
  23. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation MIT-Press, 1997. find
  24. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022. find
  25. Jo, T. Machine learning foundations. Springer, 2021. find
  26. Lamtougui, H., Moubtahij, H., Fouadi, H., and Satori, K. An efficient hybrid model for arabic text recognition. Computers, Materials & Continua, 74(2):2871–2888, 2023. find
  27. LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, 2, 1989. find
  28. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., and Song, Z. MADCAT Phase 1 Training Set LDC2012T15. https://catalog.ldc.upenn.edu/LDC2012T15, 2012. ↗
  29. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., and Song, Z. MADCAT Phase 2 Training Set LDC2013T09. https://catalog.ldc.upenn.edu/LDC2013T09, 2013a. ↗
  30. Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., and Song, Z. MADCAT Phase 3 Training Set LDC2013T15. https://catalog.ldc.upenn.edu/LDC2013T15, 2013b. ↗
  31. Lee, K., Joshi, M., Turc, I. R., Hu, H., Liu, F., Eisenschlos, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893– 18912, 2023. find
  32. Levenshtein, V. Binary codes capable of correcting deletions, insertions, and reversals. Proceedings of the Soviet Physics Doklady, 1966. find
  33. Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., and Wei, F. TrOCR: Transformer-based optical character recognition with pre-trained models. In AAAI Conference on Artificial Intelligence, volume 37, pp. 13094–13102, 2023. find
  34. Liang, M. and Hu, X. Recurrent convolutional neural network for object recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3367–3375, 2015. find
  35. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. ↗
  36. Mahmoud, S. A., Ahmad, I., Alshayeb, M., Al-Khatib, W. G., Parvez, M. T., Fink, G. A., Märgner, V., and El Abed, H. KHATT: Arabic offline handwritten text database. In International Conference on Frontiers in Handwriting Recognition, pp. 449–454. IEEE, 2012. find
  37. Marti, U.-V. and Bunke, H. The IAM-database: An english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition, 5:39–46, 2002. find
  38. Michael, J., Labahn, R., Grüning, T., and Zöllner, J. Evaluating sequence-to-sequence models for handwritten text recognition. In IEEE International Conference on Document Analysis and Recognition, pp. 1286–1293, 2019. find
  39. Mohri, M., Rostamizadeh, A., and Talwalkar, A. Foundations of machine learning. MIT press, 2018. find
  40. Momeni, S. and BabaAli, B. A transformer-based approach for Arabic offline handwritten text recognition. Signal, Image and Video Processing, 18(4):3053–3062, 2024. find
  41. Mosbach, M., Andriushchenko, M., and Klakow, D. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. International Conference on Learning Representations, 2021. find
  42. Mostafa, A., Mohamed, O., Ashraf, A., Elbehery, A., Jamal, S., Khoriba, G., and Ghoneim, A. S. OCFormer: A transformer-based model for Arabic handwritten text recognition. In IEEE International Mobile, Intelligent, and Ubiquitous Computing Conference, pp. 182–186, 2021. find
  43. Najam, R. and Faizullah, S. Analysis of recent deep learning techniques for Arabic handwritten-text OCR and post-OCR correction. Applied Sciences, 13(13):7568, 2023. find
  44. Neudecker, C., Baierer, K., Gerber, M., Clausner, C., Antonacopoulos, A., and Pletschacher, S. A survey of ocr evaluation tools and metrics. In International Workshop on Historical Document Imaging and Processing, pp. 13– 18, 2021. find
  45. Parres, D. and Paredes, R. Fine-tuning vision encoder– decoder transformers for handwriting text recognition on historical documents. In International Conference on Document Analysis and Recognition, pp. 253–268. Springer, 2023. find
  46. Puigcerver, J. Are multidimensional recurrent layers really necessary for handwritten text recognition? In IEEE International Conference on Document Analysis and Recognition, volume 1, pp. 67–72, 2017. find
  47. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. OpenAI blog, 2018. find
  48. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  49. Rawls, S., Cao, H., Mathai, J., and Natarajan, P. How to efficiently increase resolution in neural ocr models. In IEEE International Workshop on Arabic and Derived Script Analysis and Recognition, pp. 140–144, 2018. find
  50. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 1, pp. 318–362, 1986. find
  51. Saad, M. and Alijla, B. WikiDocsAligner: An off-the-shelf Wikipedia documents alignment tool. In Palestinian International Conference on Information and Communication Technology, 2017. find
  52. Saeed, M., Chan, A., Mijar, A., Moukarzel, J., Habchi, G., Younes, C., Elias, A., Wong, C.-W., and Khater, A. Muharaf: Manuscripts of handwritten arabic dataset for cursive text recognition. arXiv preprint arXiv:2406.09630, 2024. ↗
  53. Shi, B., Bai, X., and Yao, C. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11): 2298–2304, 2016. find
  54. Shtaiwi, R. E., Abandah, G. A., and Sawalhah, S. A. End-to-end machine learning solution for recognizing handwritten Arabic documents. In International Conference on Information and Communication Systems, pp. 180–185, 2022. find
  55. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pp. 5998–6008, 2017. find
  56. Wang, J. and Hu, X. Gated recurrent convolution neural network for ocr. In Advances in Neural Information Processing Systems, volume 30, 2017. find
  57. Wang, T., Zhu, Y., Jin, L., Luo, C., Chen, X., Wu, Y., Wang, Q., and Cai, M. Decoupled attention network for text recognition. In AAAI Conference on Artificial Intelligence, volume 34, pp. 12216–12224, 2020. find
  58. Wick, C., Zöllner, J., and Grüning, T. Transformer for handwritten text recognition using bidirectional post-decoding. In International Conference on Document Analysis and Recognition, pp. 112–126. Springer, 2021. find
  59. Wigington, C., Tensmeyer, C., Davis, B., Barrett, W., Price, B., and Cohen, S. Start, follow, read: End-to-end full-page handwriting recognition. In European Conference on Computer Vision, pp. 367–383, 2018. find
  60. Zhang, Y., Nie, S., Liu, W., Xu, X., Zhang, D., and Shen, H. T. Sequence-to-sequence domain adaptation network for robust text image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2740–2749, 2019. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chan-2024-hatformer-historic-handwritten-arabic-text/