Elamur Research⌕
☰
home / papers / 2020 / Arabisc: Context-Sensitive Neural Spelling Checker

Arabisc: Context-Sensitive Neural Spelling Checker

Paper · 2020 · Yasmin Moslem, Rejwanul Haque and Andy Way

Summary

Spell checkers face two kinds of mistakes. Some misspellings are not words at all (NWE). Others are valid dictionary words used in the wrong place (RWE). Neural systems that rewrite a whole sentence tend to "fix" words that were already correct. Arabisc avoids this by splitting the job into detection and then correction, both driven by a word-level LSTM language model.

For each word, the model predicts a ranked list of likely candidates from its context. If the actual word is missing from that list, it is flagged. Replacement suggestions are then ranked by Levenshtein distance to the flagged word. Two variants were compared: one that sees only the preceding words, and a dual-input version that also reads the following words in reverse order.

The model was trained on cleaned Arabic text from News Commentary and MultiUN. It was tested on 20 UN sentences with injected errors of both types. Using right-hand context, matching lemmas, and fixing earlier errors first sharply reduced overcorrection. The resulting system beat LanguageTool and Sakhr Tadqeek on suggestion quality, which is useful for languages with few resources, such as Arabic.

Key points

  • Training data: News Commentary v11 from OPUS gave 213,036 Arabic sentences after cleaning. Adding part of MultiUN brought the total to 554,622. Preprocessing stripped diacritics (Tashkil), punctuation, digits and Latin characters.

  • Architecture:

    • embedding of dimension 256;
    • bidirectional LSTM with 512 units, then an LSTM with 128 units;
    • softmax output over the full vocabulary;
    • trained with Adam at a learning rate of 0.001.

  • Test data: 20 unseen UN corpus sentences. RWEs came from the confusion lists of Al-Jefri and Mahmoud (2013). NWEs were created by deleting, inserting, substituting or swapping adjacent letters, following Damerau.

  • RWE results: every variant flagged all 20 errors. Overcorrections fell from 9 (single-input) to 5 (dual-input), then 3 with lemma matching or previous-word correction, and 1 with both together.

  • NWE results:

    SystemDetectedExact suggestionsOvercorrections
    Dual-Input+Prev20190
    LanguageTool1911not reported here
    Sakhr Tadqeek1814not reported here

    The off-the-shelf tools also accepted rare, outdated word forms that were actually errors in context.

Interesting ideas

  • I1. Gate correction behind detection. Only words that fall outside the model's top candidates are eligible for change, which keeps correct material safe from unnecessary rewriting.
  • I2. Reverse the right context. The words after the target are fed backwards as a second branch, so both branches approach the target word from its neighbours. This is a cheap way to give a many-to-one model access to both sides.
  • I3. Check membership at lemma level. A word counts as acceptable if its lemma appears among the predicted candidates, so legitimate inflected forms are not punished. This cut overcorrection noticeably.
  • I4. Repair left to right before moving on. Fixing an earlier error before scoring later words stops one mistake from corrupting the context for the next.
  • I5. Score overcorrection as its own metric. Synthetic errors from confusion lists and Damerau-style edits give a controlled test set. Counting false flags on clean words separately acts as a negative control.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a normalization step before the tagger could alter only tokens the language model flags, leaving an explainable "flagged because not predicted" trace.
  • I2 (tokens): ambiguous lemmas or POS tags could be resolved by checking predictions from both the left and right branches.
  • I3 (learn): when witness parsers or the LLM teacher disagree only on inflection, comparing lemmas could prevent false conflicts in silver data.
  • I4 (learn): self-induced repair rules could be applied incrementally, each fix updating the context used to evaluate the following decision.
  • I5 (learn): evaluations could inject known perturbations and count unwarranted changes on untouched words, not only successful fixes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yasmin Moslem, Rejwanul Haque and Andy Way. 2020. Arabisc: Context-Sensitive Neural Spelling Checker. Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.nlptea-1.2

@inproceedings{moslem2020arabisc,
  title = {{Arabisc: Context-Sensitive Neural Spelling Checker}},
  author = {Yasmin Moslem, Rejwanul Haque and Andy Way},
  booktitle = {Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications},
  year = {2020},
  doi = {10.18653/v1/2020.nlptea-1.2},
  url = {https://doi.org/10.18653/v1/2020.nlptea-1.2}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Majed Al-Jefri and Sabri Mahmoud. 2013. Context-Sensitive Arabic Spell Checker Using Context Words and N-Gram Language Models. In 2013 Taibah University International Conference on Advances in Information Technology for the Holy Quran and Its Sciences, Madinah, Saudi Arabia, pages 258–263. find
  2. Monojit Choudhury, Markose Thomas, Animesh Mukherjee, Anupam Basu, and Niloy Ganguly. 2007. How Difficult is it to Develop a Perfect Spellchecker? A Cross-Linguistic Analysis through Com-https://github.com/ymoslem/Arabisc plex Network Approach. In Proceedings of the Second Workshop on TextGraphs: Graph-Based Algorithms for Natural Language Processing, pages 81– 88, Rochester, NY, USA. Association for Computational Linguistics. ↗
  3. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010.
  4. Fred J. Damerau. 1964. A Technique for Computer Detection and Correction of Spelling Errors. Commun. ACM, 7(3):171–176. ↗
  5. Suzan Verberne. 2002. Context-sensitive Spell Checking Based on Word Trigram Probabilities. Master thesis Taal, Spraak & Informatica, University of Nijmegen, Nijmegen, the Netherlands. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. ↗
  7. Tal Weiss. 2016. Deep Spelling, Rethinking Spelling Correction in the 21st Century. machinelearn-ings.co, accessed September 10, 2020. find
  8. Matthias Hertel. 2019. Neural Language Models for Spelling Correction. Master’s thesis, Albert-Ludwigs-Universität Freiburg im Breisgau Technische Fakultät, Freiburg, Germany. find
  9. Paul J Werbos. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560. ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. ↗
  11. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  12. Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. CoRR, abs/1804.10959. ↗
  13. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics-Doklady, Vol. 10, pages 707–710. find
  14. James L. Peterson. 1986. A Note on Undetected Typing Errors. Commun. ACM, 29(7):633–637. ↗
  15. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by backpropagating errors. Nature, 323(6088):533. find
  16. Sivasurya Santhanam. 2020. Context based Text-generation using LSTM networks. ↗
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. ↗
  18. Qing Sun, Stefan Lee, and Dhruv Batra. 2017. Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, USA, pages 1339–1348. find
  19. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC’2012), pages 2214–2218, Istanbul, Turkey. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moslem-2020-arabisc-context-sensitive-neural-spelling/