Elamur Research⌕
☰
home / papers / 2024 / AraSpell: A Deep Learning Approach for Arabic Spelling Correction

AraSpell: A Deep Learning Approach for Arabic Spelling Correction

Summary

Arabic spelling correction has mostly relied on rule-based or statistical pipelines that first detect an error and then pick a correction. Labelled pairs of erroneous and corrected text are scarce. AraSpell avoids this shortage by building its own training data. The authors took the Arabic Wikipedia 2021 dump and cleaned and filtered it into more than 6.9 million short Modern Standard Arabic sentences, which they treat as trusted targets. A random error-injection procedure then corrupts each sentence. Three character-level seq2seq architectures (a vanilla RNN, an RNNB variant and a Transformer) learn to map the corrupted text back to the original. They were trained at 5% and 10% corruption, and on mixed and varied noise levels.

On a 100K-sentence test set, the Transformer trained on mixed data performs best. At 5% noise it brings errors down from 5.03% CER / 29.72% WER to 1.11% / 4.8%. At 10% noise it brings them down from 10.02% / 50.94% to roughly 2.8–2.9% / 10.65%. The work shows that end-to-end correction trained only on synthetic noise can work well for a lower-resourced language.

Key points

  • Data source and filtering:
    • The source is the Arabic Wikipedia 2021 dump, with 711,230 articles.
    • Cleaning removed diacritics, truncated characters repeated more than twice to two, and split lam-alef ligature codepoints into separate characters.
    • Lines were dropped if they contained digits, single-character words, citations, or more than two corpus-unique words.
    • Lines were kept only if they had 3–20 words and 15–128 characters.
  • Error injection:
    • Each sentence receives ⌊length × ψ⌋ operations, each drawn uniformly from a fixed set.
    • The set covers insertion, deletion, substitution (with a random character or a keyboard neighbour), transposition of adjacent characters, and mapping of predefined patterns.
  • Transformer setup:
    • 4 layers, 512 dimensions and 8 heads.
    • Adam with 4000 warm-up steps and KL-divergence loss.
    • Dropout of 0.1 and label smoothing of 0.1.
    • Trained on two RTX 3080 Ti GPUs.
  • Best results (Transformer_mixed):
    • At ψ=5%, character error reduction is 77.93% and word error reduction is 83.84%.
    • At ψ=10%, the reductions are 72.05% and 79.09%.
    • The Transformer beat both RNN variants in every condition.
  • Effect of mixing noise levels: training on combined 5% and 10% data helped all architectures. Code and data are released on GitHub (msalhab96/AraSpell).

Interesting ideas

  • I1. Supervised pairs can be made for free by trusting a carefully filtered clean corpus and corrupting it programmatically, with the number of edits proportional to sentence length; this turns any clean text into labelled training material. — For Mova (learn): Inject within-word typos (no space insertion or deletion, so token boundaries and gold trees stay valid) into treebank sentences. This gives robustness test sets and negative controls for the parsers.
  • I2. Scores are reported as error reduction relative to the uncorrected corrupted input, not only as absolute CER/WER; this shows how much the system actually fixes compared with doing nothing. — For Mova (learn): Evaluate any repair or normalization rule against a "do nothing" baseline. A rule then has to prove that it lowers errors.
  • I3. A simple curriculum helps when training stalls: the vanilla RNN was first trained on 5% noise until it learned alignment, then moved to 10%, which avoided a roughly 20× slowdown. — For Mova (tree): When training gparse on hard or noisy silver data, warm it up on clean, easy sentences first. This can stabilize early learning.
  • I4. Training on a mixture of noise levels gave better results across test conditions than matching the training noise to the test noise. — For Mova (tree): Mix clean and lightly corrupted sentences when training the perceptron parser. This should make it degrade gracefully on messy real-world English.
  • I5. Unicode normalization (splitting ligature codepoints, collapsing over-repeated characters) is done before anything else, so the model sees a compact, consistent alphabet. — For Mova (tokens): Normalize English input the same way before tokenization, for example the fi/fl ligatures, curly quotes and elongations like "sooooo". This keeps lemmas and tags consistent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.06981 · arXiv 2405.06981

@misc{salhab2024araspell,
  title = {{AraSpell: A Deep Learning Approach for Arabic Spelling Correction}},
  author = {Salhab, Mahmoud and Abu-Khzam, Faisal},
  year = {2024},
  doi = {10.48550/arxiv.2405.06981},
  eprint = {2405.06981},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.06981}
}

References

30 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Olaronke, I., Olaleke, J.: A systematic review of natural language processing in healthcare. International Journal of Information Technology and Computer Science 08, 44–50 (2015). https://doi.org/10.5815/ijitcs.2015. 08.07 ↗
  2. Prakash, K.: Advances in natural language processing - a survey of current research trends, development tools and industry applications. International Journal of Recent Technology and Engineering (2019) find
  3. Boudad, N., Faizi, R., Rachid, O.h.t., Chiheb, R.: Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal 9 (2017). https://doi.org/10.1016/j.asej.2017.04.007 ↗
  4. Alwajeeh, A., Al-Ayyoub, M., Hmeidi, I.: On authorship authentication of arabic articles. In: 2014 5th International Conference on Information and Communication Systems (ICICS), pp. 1–6 (2014). https://doi.org/ 10.1109/IACS.2014.6841973 ↗
  5. Al-Ayyoub, M., Nuseir, A., Alsmearat, K., Jararweh, Y., Gupta, B.: Deep learning for arabic nlp: A survey. Journal of Computational Science 26, 522–531 (2018). https://doi.org/10.1016/j.jocs.2017.11.011 ↗
  6. Napoles, C., Cahill, A., Madnani, N.: The effect of multiple grammatical errors on processing non-native writing, pp. 1–11 (2016). https://doi.org/ 10.18653/v1/W16-0501 ↗
  7. Bassil, Y., Alwani, M.: OCR post-processing error correction algorithm using google online spelling suggestion. CoRR abs/1204.0191 (2012) https://arxiv.org/abs/1204.0191 ↗
  8. Moslem, Y., Haque, R., Way, A.: Arabisc: Context-sensitive neural spelling checker. In: NLPTEA (2020) find
  9. Kukich, K.: Techniques for automatically correcting words in text. ACM Comput. Surv. 24, 377–439 (1992) find
  10. Gao, M., Xu, C., Shi, P.: Hierarchical character tagger for short text spelling error correction. In: WNUT (2021) find
  11. Wang, Y.-R., Liao, Y.-F.: Word vector/conditional random field-based chinese spelling error detection for sighan-2015 evaluation. In: SIGHAN@IJCNLP (2015) find
  12. Zaky, D., Romadhony, A.: An lstm-based spell checker for indonesian text. In: 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pp. 1–6 (2019). https://doi.org/10.1109/ICAICTA.2019.8904218 ↗
  13. Kinaci, A.C.: Spelling correction using recurrent neural networks and character level n-gram. 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), 1–4 (2018) find
  14. Nawar, M.N., Ragheb, M.M.: Fast and robust arabic error correction system. In: ANLP@EMNLP (2014) find
  15. Shaalan, Khaled and Khaled, and Allam, and Amin, and Gomah, and Abdallah: Towards automatic spell checking for arabic. (2003) find
  16. Alkanhal, M.I., Al-Badrashiny, M.A., Alghamdi, M.M., Al-Qabbany, A.O.: Automatic stochastic arabic spelling correction with emphasis on space insertions and deletions. IEEE Transactions on Audio, Speech, and Language Processing 20(7), 2111–2122 (2012). https://doi.org/10.1109/ TASL.2012.2197612 ↗
  17. Hassan, Y., Aly, M., Atiya, A.: Arabic spelling correction using supervised learning. (2014). https://doi.org/10.3115/v1/W14-3615 ↗
  18. Attia, M., Pecina, P., Samih, Y., Shaalan, K., Van Genabith, J.: Arabic spelling error detection and correction. Nat. Lang. Eng. 22(5), 751–773 (2016) find
  19. Shaalan, K., Samih, Y., Attia, M., Pecina, P., Genabith, J.: Arabic word generation and modelling for spell checking. (2012) find
  20. Alkhatib, M., Monem, A.A., Shaalan, K.: Deep learning for arabic error detection and correction. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 19(5) (2020). https://doi.org/10.1145/3373266 ↗
  21. Abandah, G., Suyyagh, A., Khedher, M.: Correcting arabic soft spelling mistakes using bilstm-based machine learning. International Journal of Advanced Computer Science and Applications 13, 815–829 (2022). https://doi.org/10.14569/IJACSA.2022.0130594 ↗ our card
  22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. CoRR abs/1706.03762 (2017) https://arxiv.org/abs/1706.03762 ↗
  23. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. CoRR abs/2005.12872 (2020) https://arxiv.org/abs/2005.12872 ↗ our card
  24. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. CoRR abs/2010.11929 (2020) https://arxiv.org/abs/2010.11929 ↗
  25. Chen, X., Wu, Y., Wang, Z., Liu, S., Li, J.: Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. CoRR abs/2010.11395 (2020) https://arxiv.org/abs/2010.11395 ↗
  26. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv (2020). https://doi.org/10.48550/ARXIV.2005.08100. https://arxiv.org/abs/2005.08100 ↗ our card
  27. Li, N., Liu, S., Liu, Y., Zhao, S., Liu, M., Zhou, M.: Neural Speech Synthesis with Transformer Network. arXiv (2018). https://doi.org/10.48550/ ARXIV.1809.08895. https://arxiv.org/abs/1809.08895 ↗
  28. Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., Qin, T., Liu, T.-Y.: MultiSpeech: Multi-Speaker Text to Speech with Transformer. arXiv (2020). https://doi.org/10.48550/ARXIV.2006.04664. https://arxiv.org/ abs/2006.04664 ↗
  29. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014) find
  30. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR abs/1512.00567 (2015) https://arxiv.org/abs/1512.00567 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salhab-2024-araspell-deep-learning-approach-arabic/