Elamur Research⌕
☰
home / papers / 2014 / The Illinois-Columbia System in the CoNLL-2014 Shared Task

The Illinois-Columbia System in the CoNLL-2014 Shared Task

Paper · 2014 · Alla Rozovskaya, Kai-Wei Chang, Mark Sammons et al.

Summary

This paper presents the Illinois-Columbia entry to the CoNLL-2014 shared task on correcting grammatical mistakes in essays by non-native English writers. Unlike the 2013 edition, the 2014 task evaluated every error type in the data. Errors are rare even in learner text, so systems struggle to find them without many false alarms. The authors started from the Illinois system that won CoNLL-2013. That system uses separate confusion-set classifiers for articles, prepositions, noun number, verb agreement and verb form. Some classifiers are trained on learner text with the Averaged Perceptron, and others on native n-gram counts with Naïve Bayes. The team made three additions: new modules for word form, punctuation/capitalization and tone errors; combined models that merge learner-trained and native-trained classifiers; and joint inference to handle errors that interact. Training used the NUCLE corpus, and the CoNLL-2013 test set served as development data. The system placed second on the original test annotations and first on the revised ones. This shows that the error-specific classifier design can be extended to broad error coverage.

Key points

  • Data: about 1.2M words of teacher-corrected NUCLE essays from the National University of Singapore, using 28 error categories. Roughly 30K words from the CoNLL-2013 test set served as development data.
  • Error sparsity: only about 2% of prepositions in the training data are wrong. Error inflation (adding artificial errors to training data) was used to raise recall.
  • Development F0.5 gains:
    • The baseline scored 33.17.
    • Model combination raised this to 34.92.
    • New error modules raised it to 37.09.
    • Joint inference gave a final 37.13.
  • Test results:
    • Original annotations: P 41.78, R 24.88, F0.5 36.79.
    • Revised annotations: P 52.44, R 29.89, F0.5 45.57.
  • Coverage gaps: the system does not handle wrong collocation (Wci) or local redundancy (Rloc-). The error analysis points to cascading POS-tagging mistakes and sentences with multiple errors as frequent sources of failure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States254 works · 56,747 citations🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth and Nizar Habash. 2014. The Illinois-Columbia System in the CoNLL-2014 Shared Task. Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1704

@inproceedings{rozovskaya2014illinois,
  title = {{The Illinois-Columbia System in the CoNLL-2014 Shared Task}},
  author = {Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth and Nizar Habash},
  booktitle = {Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task},
  year = {2014},
  doi = {10.3115/v1/w14-1704},
  url = {https://doi.org/10.3115/v1/w14-1704}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brants and A. Franz. 2006. Web 1T 5-gram Version 1. Linguistic Data Consortium, Philadelphia, PA. find
  2. A. J. Carlson, J. Rosen, and D. Roth. 2001. Scaling up context sensitive text correction. In IAAI. find
  3. D. Dahlmeier and H.T. Ng. 2012. Better evaluation for grammatical error correction. In NAACL, pages 568–572, Montréal, Canada, June. Association for Computational Linguistics. find
  4. D. Dahlmeier, H.T. Ng, and S.M. Wu. 2013. Building a large annotated corpus of learner english: The nus corpus of learner english. In Proc. of the NAACL HLT 2013 Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, Georgia, June. Association for Computational Ling…
  5. R. Dale and A. Kilgarriff. 2011. find
  6. Helping Our Own: The HOO 2011 pilot shared task. In Proceedings of the 13th European Workshop on Natural Language Generation.
  7. R. Dale, I. Anisimoff, and G. Narroway. 2012. A report on the preposition and determiner error correction shared task. In Proc. of the NAACL HLT 2012 Seventh Workshop on Innovative Use of NLP for Building Educational Applications, Montreal, Canada, June. Association for Computational Linguistics. find
  8. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In ACL.
  9. Yoav Freund and Robert E. Schapire. 1996. Experiments with a new boosting algorithm. In Proc. 13th International Conference on Machine Learning. find
  10. A. R. Golding and D. Roth. 1999. A Winnow based approach to context-sensitive spelling correction. Machine Learning. find
  11. H.T. Ng, S.M. Wu, Y. Wu, C. Hadiwinoto, and J. Tetreault. 2013. The conll-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12, Sofia, Bulgaria, August. Association for Computational Linguis… find
  12. H. T. Ng, S. M. Wu, T. Briscoe, C. Hadiwinoto, R. H. Susanto, and C. Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, Baltimore, Maryland, USA, June. Association for Computat… find
  13. V. Punyakanok and D. Roth. 2001. The use of classifiers in sequential inference. In NIPS. find
  14. A. Rozovskaya and D. Roth. 2011. Algorithm selection and model adaptation for esl correction tasks. In ACL. find
  15. A. Rozovskaya and D. Roth. 2013. Joint learning and inference for grammatical error correction. In EMNLP, 10. find
  16. A. Rozovskaya, M. Sammons, and D. Roth. 2012. The UI system in the HOO 2012 shared task on error correction. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Workshop on Innovative Use of NLP for Building Educational Application… find
  17. A. Rozovskaya, K.-W. Chang, M. Sammons, and D. Roth. 2013. The University of Illinois system in the CoNLL-2013 shared task. In CoNLL Shared Task. find
  18. A. Rozovskaya, D. Roth, and V. Srikumar. 2014. Correcting grammatical verb errors. In EACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rozovskaya-2014-illinois-columbia-system-conll-2014/