Elamur Research⌕
☰
home / papers / 2013 / Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English

Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English

Paper · 2013 · Daniel Dahlmeier, Hwee Tou Ng and Siew Mei Wu

Summary

The paper describes NUCLE, a fully error-annotated English corpus of texts written by learners of English. It is freely available for research. The corpus was created because training and evaluating automatic grammatical error correction systems lacked a large open resource: existing annotated corpora were either small or closed. NUCLE consists of essays written by students at the National University of Singapore. They were annotated by professional English instructors from CELC using WAMP, a web-based tool developed specifically for this corpus. Each error is assigned a category and a proposed correction, and the marked span is kept as short as possible. The authors also publish, for the first time, a study of inter-annotator agreement. To compare different ways of marking errors fairly, they reduce the annotations to the level of individual tokens. The study showed that grammatical error correction is difficult even for humans. According to the corpus statistics, errors are rare and their distribution has a long tail. The most frequent errors involve prepositions, collocations, and articles. The paper has become the reference description of the corpus and makes it easier for other researchers to work with it.

Key points

  • Size: about 1,400 essays, more than one million words. Errors are annotated and corrected throughout the entire corpus.
  • Annotation scheme: 27 error categories in 13 groups, including Vt, Vform, SVA, ArtOrDet, Nn, Wcip, Rloc, Mec. The scheme is based on a tag set previously developed at CELC. For comparison, the Cambridge Learner Corpus (CLC) has 88 categories.
  • Agreement study: 100 essays were taken from the pilot data, 4 of which were discarded because of very low quality. Each of the remaining 96 was annotated independently by two annotators, and these essays were not included in the official NUCLE. Before comparison, the marked spans were normalized: they were extended to word boundaries, identical tokens at the edges were trimmed, and the spans were projected onto individual tokens. Tokens were aligned by minimum edit distance.
  • Most frequent errors: Wcip 15.69 %, Rloc 13.71 %, ArtOrDet 12.88 %, Nn 8.49 %, Mec 7.06 %. These five categories account for 57.83 % of all annotations, and the ten most frequent for 79.66 %.
  • Comparison with other resources: the annotated portion of ICLE and the Chinese Learner English Corpus in Rozovskaya & Roth has 63 thousand words. The HOO 2011 corpus has roughly 20 thousand words each for training and testing and covers only NLP papers. FCE essays are shorter than NUCLE essays and contain more errors per unit of text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Linguistic annotation · 26 works

Cite

Cite as Daniel Dahlmeier, Hwee Tou Ng and Siew Mei Wu. 2013. Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{dahlmeier2013building,
  title = {{Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English}},
  author = {Daniel Dahlmeier, Hwee Tou Ng and Siew Mei Wu},
  booktitle = {Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2013},
  url = {https://aclanthology.org/W13-1703/}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  2. D. Dahlmeier and H.T. Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of HLT-NAACL, pages 568–572. find
  3. R. Dale and A. Kilgarriff. 2011. Helping Our Own: The HOO 2011 pilot shared task. In Proceedings of the Generation Challenges Session at the 13th European Workshop on Natural Language Generation, pages 242–249.
  4. R. Dale, I. Anisimoff, and G. Narroway. 2012. HOO 2012: A report on the preposition and determiner error correction shared task. In Proceedings of the Seventh Workshop on Innovative Use of NLP for Building Educational Applications, pages 54–62.
  5. S. Granger, F. Dagneaux, E. Meunier, and M. Paquot. 2002. The International Corpus of Learner English. Presses Universitaires de Louvain, Louvain-la-Neuve, Belgium. find
  6. S. Gui and H. Yang. 2003. Zhongguo Xuexizhe Yingyu Yuliaohu (Chinese Learner English Corpus). Shanghai Waiyu Jiaoyu Chubanshe. In Chinese. find
  7. J.R. Landis and G.G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  8. C. Leacock, M. Chodorow, M. Gamon, and J. Tetreault. 2010. Automated Grammatical Error Detection for Language Learners. Morgan & Claypool Publishers. ↗
  9. J. Lee, J. Tetreault, and M. Chodorow. 2009. Human evaluation of article and noun number usage: Influences of context and construction variability. In Proceedings of the Linguistic Annotation Workshop III (LAW3), pages 60–63. find
  10. N. Madnani, J. Tetreault, M. Chodorow, and R. Rozovskaya. 2011. They can help: using crowdsourcing to improve the evaluation of grammatical error detection systems. In Proceedings of ACL:HLT, pages 508–513. find
  11. D. Nicholls. 2003. The Cambridge learner corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 Conference, pages 572–581. find
  12. A. Rozovskaya and D. Roth. 2010. Annotating ESL errors: Challenges and rewards. In Proceedings of the Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 28–36. find
  13. J. Tetreault and M. Chodorow. 2008. Native judgments of non-native usage: Experiments in preposition error detection. In Proceedings of the Workshop on Human Judgements in Computational Linguistics, pages 24– 32. find
  14. J. Tetreault, E. Filatova, and M. Chodorow. 2010. Rethinking grammatical error annotation and evaluation with the Amazon Mechanical Turk. In Proceedings of the Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 45–48. find
  15. H. Yannakoudakis, T. Briscoe, and B. Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of ACL:HLT, pages 180– 189.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dahlmeier-2013-building-large-annotated-corpus-learner/