Elamur Research⌕
☰
home / papers / 2018 / A Systematic Review of Automated Grammar Checking in English Language

A Systematic Review of Automated Grammar Checking in English Language

Paper · 2018 · Madhvi Soni and Jitendra Singh Thakur

Summary

The survey asks how far automatic detection and correction of English grammar errors had progressed, mainly to serve non-native writers, and where the gaps remained. The authors ran a formal systematic review. They wrote ten research questions, built a Google Scholar query from population, intervention and outcome keyword groups, and filtered the hits down to 12 primary studies. For each study they recorded the technique, workflow, target errors, data, results, tool availability, strengths and weaknesses in a shared table. They sorted the approaches into rule-based, machine-learning and hybrid families. They also proposed their own error taxonomy, because earlier taxonomies from Naber, Wagner et al., Lee et al. and Yuan disagree with one another.

The main finding is that the field is fragmented. Every system targets a different subset of errors and reports results on its own test data, so the numbers cannot be compared. Most tools are not released. Errors at the sentence level and the semantic level are rarely tackled. The authors argue for shared test sets and evaluation broken down by error type.

Key points

  • Paper selection: 113 candidate papers were found. After removing 35 duplicates, 36 papers rejected on abstract or introduction, 29 rejected on full text, and 1 more rejected for missing technical detail, 12 studies remained. For the CoNLL-2013 and CoNLL-2014 shared tasks, only the top system was kept.
  • Error taxonomy: five top-level categories, each with subtypes:
    • sentence structure,
    • punctuation,
    • spelling,
    • syntax,
    • semantic.
  • Criteria behind the taxonomy: how frequent the error is, what makes the text invalid, the span needed to detect it (word window vs. whole sentence), how hard it is to detect, and keeping overlap between categories small.
  • Technique trade-offs:
    • Rule-based systems suit language learning, but writing the rules is laborious.
    • Statistical systems save that labour, but how well they work hinges on how large the training corpus is and what kind of text it contains.
    • Hybrids need each component placed where it fits best.
  • Data and results: the reviewed work draws on NUCLE, Gigaword, Google Web 1T 5-gram, CLC, FCE, EVP and the CoNLL-2014 data. The hybrid system of Felice et al. handles 28 error types. Reported precision is typically well under 70% and recall around 20–35%. Run-on sentences remain essentially unaddressed.

Interesting ideas

  • I1. Tie each error type to the smallest span that reveals it: a preposition slip shows up in a few neighbouring words, while a fragment needs the shape of the full parse tree. This lets cheap local checks run before expensive global ones.
  • I2. Label an error by what it violates (spelling conventions, grammar rules, sentence-structure rules, or meaning) rather than by surface symptom. The label then points toward the right fix.
  • I3. Instead of a single overall score, evaluate on a shared test set where every faulty sentence carries an error-type label, so the best method for each type becomes visible.
  • I4. Rule-based checking is valued for teaching because every flag can be explained, so statistical parts should be kept only where rules are impractical.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): sort tree sanity checks into local head/dependent window rules and whole-tree pattern rules, and run the cheap ones first so each flagged problem says which scope caught it.
  • I2 (learn): tag every self-induced repair rule and every silver-data disagreement with a violation cause, so repairs can be grouped and audited by the kind of rule broken.
  • I3 (learn): add type-labelled corrupted sentences as negative controls and report accuracy per corruption type, showing which pipeline stage fails on which kind of damage.
  • I4 (answer): when a student-facing reader marks an answer wrong, prefer an explicit rule trace and fall back to learned scores only with a stated reason.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksGrammatical error correction & learner language · 80 works

Cite

Cite as Madhvi Soni and Jitendra Singh Thakur. 2018. A Systematic Review of Automated Grammar Checking in English Language. arXiv:1804.00540.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.00540 · arXiv 1804.00540

@misc{soni2018systematic,
  title = {{A Systematic Review of Automated Grammar Checking in English Language}},
  author = {Madhvi Soni and Jitendra Singh Thakur},
  year = {2018},
  doi = {10.48550/arxiv.1804.00540},
  eprint = {1804.00540},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.00540}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M Bender, Dan Flickinger, Stephan Oepen, Annemarie Walsh, and Timothy Baldwin. 2004. Arboretum: Using a precision grammar for grammar checking in CALL. In Instil/icall symposium 2004. find
  2. Johnny Bigert. 2004. Probabilistic Detection of Context-Sensitive Spelling Errors.. In LREC. find
  3. Chris Brockett, William B Dolan, and Michael Gamon. 2006. Correcting ESL errors using phrasal SMT techniques. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 249–256.
  4. Martin Chodorow, Joel R Tetreault, and Na-Rae Han. 2007. Detection of grammatical errors involving prepositions. In Proceedings of the fourth ACL-SIGSEM workshop on prepositions. Association for Computational Linguistics, 25–30. find
  5. Daniel Dahlmeier and Hwee Tou Ng. 2011. Grammatical error correction with alternating structure optimization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 915–923. find
  6. Mariano Felice, Zheng Yuan, Øistein E Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering.. In CoNLL Shared Task. 15–24. find
  7. AS Hornby. 1995. Guide to Patterns and Usage in English. (1995). find
  8. An-Ta Huang, Tsung-Ting Kuo, Ying-Chun Lai, and Shou-De Lin. 2010. Discovering Correction Rules for Auto Editing. Computational Linguistics and Chinese Language Processing 15, 3-4 (2010), 219–236. ↗
  9. Staffs Keele et al. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn. find
  10. Akshat Kumar and Shivashankar Nair. 2007. An artificial immune system based approach for English grammar checking. Artificial immune systems (2007), 348–357. find
  11. John Lee and Stephanie Seneff. 2008. Correcting Misuse of Verb Forms.. In ACL. 174–182. find
  12. Nina H Macdonald. 1983. Human factors and behavioral science: The UNIXâĎć Writer’s Workbench software: Rationale and design. Bell Labs Technical Journal 62, 6 (1983), 1891–1908. find
  13. Maxim Mozgovoy. 2011. Dependency-based rules for grammar checking with LanguageTool. In Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on. IEEE, 209–212. find
  14. Daniel Naber. 2003. A rule-based style and grammar checker. (2003). find
  15. Michael Neuman. 1991. RightWriter 3.1. (1991). find
  16. Jong C Park, Martha Stone Palmer, and Clay Washburn. 1997. An English Grammar Checker as a Writing Aid for Students of English as a Second Language.. In ANLP. 24. find
  17. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, and Dan Roth. 2013. The University of Illinois system in the CoNLL-2013 shared task. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task. 13–19. find
  18. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. 2014. The Illinois-Columbia System in the CoNLL-2014 Shared Task.. In CoNLL Shared Task. 34–42. find
  19. Grigori Sidorov, Anubhav Gupta, Martin Tozer, Dolors Catala, Angels Catena, and Sandrine Fuentes. 2013. Rule-based System for Automatic Grammar Correction Using Syntactic N-grams for English Language Learning (L2).. In CoNLL Shared Task. 96–101. find
  20. Guihua Sun, Xiaohua Liu, Gao Cong, Ming Zhou, Zhongyang Xiong, John Lee, and Chin-Yew Lin. 2007. Detecting erroneous sentences using automatically mined sequential patterns. In ACL. 23–30. find
  21. Cornelia Tschichold, Franck Bodmer, Etienne Cornu, Franqois Grosjean, Lysiane Grosjean, N Kübler, N Léwy, and Corinne Tschumi. 1997. Developing a new grammar checker for English as a second language. Proc. From Research to Commercial Applications: Making NLP Work in Practice (1997), 7–12. find
  22. Joachim Wagner, Jennifer Foster, and Josef van Genabith. 2007. A comparative evaluation of deep and shallow approaches to the automatic detection of common grammatical errors. Association for Computational Linguistics. find
  23. PC Wren and H Martin. 2000. English Grammar & Composition. S. Chand & Company Ltd (2000). find
  24. Chak Yan Yeung and John Lee. 2015. Automatic Detection of Sentence Fragments.. In ACL (2). 599–603. find
  25. Zheng Yuan. 2017. Grammatical error correction in non-native English. Technical Report. University of Cambridge, Computer Laboratory. find
  26. Tao Yue, Lionel C Briand, and Yvan Labiche. 2011. A systematic review of transformation approaches between user requirements and analysis models. Requirements Engineering 16, 2 (2011), 75–99. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soni-2018-systematic-review-automated-grammar-checking/