Elamur Research⌕
☰
home / papers / 2010 / Using an Error-Annotated Learner Corpus to Develop an ESL/EFL Error Correction System

Using an Error-Annotated Learner Corpus to Develop an ESL/EFL Error Correction System

Summary

Most systems that flag preposition mistakes in learner English learn only from correct native-speaker text, so they never see the kinds of confusions learners actually make. The authors ask whether training directly on learner essays with tutor corrections works better. They use the Chungdahm English Learner Corpus, a large collection of essays by Korean schoolchildren with corrections from their tutors. From it they extracted a sub-corpus of essays in which preposition feedback was clearly given. Each preposition occurrence is encoded as a pair: the student's choice and the correct form. A maximum entropy (ME) classifier then predicts the correct form from surrounding words, n-grams and parser-derived head relations. Even though the tutors did not mark every error, the model reaches very high precision on replacement errors. It also beats models trained on edited native text, despite using far less training data. The result suggests that existing, imperfectly annotated learner data can support practical error correction.

Key points

  • Chungdahm Corpus:
    • About 131M words in 861,481 essays on 1,545 prompts.
    • Written by Korean-L1 students aged 10–16.
    • Contains roughly 6.6M tutor annotations across 45 feedback classes.
  • Scope of the model:
    • Restricted to a set of 10 prepositions (with, to, on, of, in, from, for, by, at, about) together with a NULL option, a set that accounts for about 99% of student error tokens.
    • After parser-based filtering, 117,665 genuine preposition errors remained.
  • Experimental subset:
    • 111,060 essays (~20.5M words) containing ~1.2M preposition tokens, with a 9.6% preposition error rate.
    • About 978K tokens were used for ME training and ~200K were held out for development.
  • Results on replacement errors:
    • Detection: 93.3% precision, 14.8% recall.
    • Detection plus correction: 81.7% precision, 13.2% recall.
    • Outperformed native-trained models.
  • Limitation: the annotations are not exhaustive, so many real errors in the data went unmarked. The authors argue that full annotation is nonetheless unnecessary to surpass native-trained baselines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Education

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{han2010using,
  title = {{Using an Error-Annotated Learner Corpus to Develop an ESL/EFL Error Correction System}},
  author = {Han, Na-Rae and Tetreault, Joel and Lee, Soo-Hwa and Ha, Jin-Young},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/24p3dsfhqi8f},
  url = {https://doi.org/10.63317/24p3dsfhqi8f}
}

References

22 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Chodorow, Joel R. Tetreault, and Na-Rae Han. 2007. Detection of grammatical errors involving prepositions. In Proceedings of the 4th ACL-SIGSEM Workshop on Prepositions, pages 25–30. find
  2. Rachele De Felice and Stephen G Pulman. 2007. Automatically acquiring models of preposition use. In Proceedings of the ACL-07 Workshop on Prepositions. find
  3. Rachele De Felice and Stephen G. Pulman. 2008. A classifier-based approach to preposition and determiner error correction in L2 english. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), Manchester, UK. find our card
  4. Michael Gamon, Jianfeng Gao, Chris Brockett, Alexander Klementiev, William Dolan, Dmitriy Belenko, and Lucy Vanderwende. 2008. Using contextual speller techniques and language modeling for ESL error correction. In Proceedings of The Third International Joint Conference on Natural Language Processing… find
  5. Na-Rae Han, Martin Chodorow, and Claudia Leacock. 2004. Detecting errors in English article usage with a maximum entropy classifier trained on a large, diverse corpus. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC 2004), Lisbon, Portugal. find
  6. Na-Rae Han, Martin Chodorow, and Claudia Leacock. 2006. Detecting errors in English article usage by non-native speakers. Natural Language Engineering, 12(2):115–129. find
  7. George Heidorn. 2000. Intelligent writing assistance. In Robert Dale, Herman Moisl, and Harold Somers, editors, Handbook of Natural Language Processing, pages 181– 207. Marcel Dekker, Inc. find
  8. Emi Izumi, Kiyotaka Uchimoto, Toyomi Saiga, Thepchai Supnithi, and Hitoshi Isahara. 2003. Automatic error detection in the Japanese learners’ English spoken data. In ACL ’03: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 145– 148, Morristown, NJ, USA. Ass… find
  9. Emi Izumi, Kiyotaka Uchimoto, and Hitoshi Isahara. 2004. SST speech corpus of Japanese learners’ English and automatic detection of learners’ errors. ICAME Journal, 28:31–48. http://icame.uib.no/ij28/. ↗
  10. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association for Computational Linguistics, pages 423–230. find our card
  11. Kevin Knight and Ishwar Chander. 1994. Automated postediting of documents. In AAAI ’94: Proceedings of the Twelfth National Conference on Artificial Intelligence (vol. 1), pages 779–784. find
  12. John Lee and Stephanie Seneff. 2006. Automatic grammar correction for second-language learners. In Proceedings of Interspeech 2006, Pittsburgh, PA. find
  13. John Lee and Stephanie Seneff. 2008. Correcting misuse of verb forms. In Proceedings of ACL 2008, Columbus, Ohio. find
  14. Ting Liu, Mingh Zhou, Jianfeng Gao, Endong Xun, and Changning Huan. 2000. PENS: A machine-aided English writing system for Chinese users. In Proceedings of ACL 2000, pages 529–536. find
  15. Adwait Ratnaparkhi. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, University of Pennsylvania. find
  16. Hisami Suzuki and Kristina Toutanova. 2006. Learning to predict case markers in Japanese. In Proceedings of COLING-ACL, pages 1049–1056. find
  17. Joel Tetreault and Martin Chodorow. 2008a. The ups and downs of preposition error detection. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), Manchester, UK. find
  18. Joel R. Tetreault and Martin Chodorow. 2008b. Native judgments of non-native usage: Experiments in preposition error detection. In COLING Workshop on Human Judgments in Computational Linguistics. find
  19. Joel Tetreault and Martin Chodorow. 2009. Examining the use of region web counts for ESL error detection. In Proceedings of Web as Corpus Workshop (WAC-5), San Sebastian, Spain. find
  20. Jenine Turner and Eugene Charniak. 2007. Language modeling for determiner selection. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Lin-guisticsLinguistics; Companion Volume, Short Papers, pages 177–180. find
  21. Richard Xiao. 2008. Well-known and influential corpora. In Anke Lüdeling and Merja Kytö, editors, Corpus Linguistics: An International Handbook, Handbooks of Linguistics and Communication Science. Mouton de Gruyter, Berlin. find
  22. Xing Yi, Jianfeng Gao, and William B. Dolan. 2008. Web-based english proofing system for English as a second language users. In Proceedings of The Third International Joint Conference on Natural Language Processing (IJCNLP 2008). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2010-error-annotated-learner-corpus-develop/