Elamur Research⌕
☰
home / papers / 2018 / Generating Feedback for English Foreign Language Exercises

Generating Feedback for English Foreign Language Exercises

Paper · 2018 · Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy et al.

Summary

The paper tackles a practical gap: few tutoring tools used in real classrooms give learners immediate, metalinguistic feedback on grammar. Instead of trying to parse ill-formed learner English directly, the authors work in the opposite direction. For each exercise, they start from the correct target answers and, before any student interacts with the system, generate the expected range of correct and incorrect variants. Each variant is paired with a diagnosis and a scaffolding message. At answer time, an online component flexibly matches what the student typed against these precomputed hypotheses. The method is built into FeedBook, a web workbook for 7th-grade English used in German schools. It was evaluated on about 33 thousand logged answers. Frequent error patterns, especially tense errors, are covered well. A long tail of rare deviations, paraphrases and meaning errors still receives only generic feedback. The work matters because task knowledge narrows the interpretation space, which makes feedback both reliable and fast. Wrong feedback is particularly harmful to language learners.

Key points

  • Data come from a March 2018 snapshot of FeedBook: 6341 submissions by 538 students. Of 234 tasks, 111 have immediate feedback (64 short-answer and 47 fill-in-the-blanks).
  • The analysis pipeline is built on UIMA. On top of standard NLP tools, 40 custom annotators mark properties such as complex tenses or irregular comparatives, aimed at the 7th-grade grammar curriculum.
  • Specific metalinguistic feedback was given for 25.28% of answers that differed from the target. Its 696 answer types covered 3,483 instances, roughly a 1:5 type-to-token ratio. The 10,297 default-feedback cases were spread over 5,717 types.
  • Specific feedback was more frequent for answers close to the target. It reached 30.8% in the lowest edit-distance tertile and only 16.6% in the highest.
  • Tasks with feedback attracted more work: 36 of the 50 most-submitted tasks (72%) offer immediate feedback.

Interesting ideas

  • I1. Rather than modelling broken learner language with mal-rules, generate likely wrong forms offline from the well-formed key answer and simply match at runtime; this keeps analysis on standard language and moves search cost out of runtime.
  • I2. Recognising that a learner's word has the same lemma as the expected one but the wrong inflection (e.g. -ing instead of simple present, or a missing -e drop) turns a "wrong word" verdict into a precise form message.
  • I3. Comparing how many distinct types an error pattern has with how many instances it covers shows where hand-built rules pay off and where the long tail is not worth modelling.
  • I4. Slicing the evaluation by edit distance to the reference (tertiles) reveals that coverage falls as answers drift further from the target.
  • I5. Teacher-written corrections are stored in a feedback memory and proposed again when the identical form recurs for the same exercise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A school-test checker could expand each key answer into tagged wrong variants, so that a mismatch returns an explained diagnosis instead of a bare "wrong".
  • I2 (tokens): The morphology module could generate systematic misinflections and misspellings per lemma, letting the tagger map faulty forms back to lemma and feature with an explicit error label.
  • I3 (learn): Repair-rule induction could be prioritised by token coverage per type, with rare singletons left to a fallback.
  • I4 (learn): Evaluation reports could be stratified by distance from the gold output to expose where the pipeline only handles near-misses.
  • I5 (learn): Corrections from LLM teacher tiers could be cached per task and input form, so repeated cases are resolved cheaply and stay consistent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany78 works · 41,280 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena Möller, Florian Nuxoll and Detmar Meurers. 2018. Generating Feedback for English Foreign Language Exercises. Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-0513

@inproceedings{rudzewitz2018generating,
  title = {{Generating Feedback for English Foreign Language Exercises}},
  author = {Björn Rudzewitz, Ramon Ziai, Kordula {De Kuthy}, Verena Möller, Florian Nuxoll and Detmar Meurers},
  booktitle = {Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2018},
  doi = {10.18653/v1/w18-0513},
  url = {https://doi.org/10.18653/v1/w18-0513}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luis von Ahn. 2013. Duolingo: Learn a language for free while helping to translate the web. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, pages 1–2. find
  2. Detmar Meurers and Markus Dickinson. 2017. Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67(2). http://dx.doi.org/10.1111/lang.12233. ↗
  3. Luiz Amaral and Detmar Meurers. 2011. On using intelligent computer-assisted language learning in real-life foreign language teaching and learning. Re-CALL, 23(1):4–24. ↗
  4. Detmar Meurers, Kordula De Kuthy, Verena Möller, Florian Nuxoll, Björn Rudzewitz, and Ramon Ziai. 2018. Digitale Differenzierung benötigt Informationen zu Sprache, Aufgabe und Lerner. Zur Gener-ierung von individuellem Feedback in einem interaktiven Arbeitsheft. FLuL – Fremdsprachen Lehren und Lernen, 47(2). In press. find
  5. Richard Eckart de Castilho and Iryna Gurevych. 2014. A broad-coverage collection of portable NLP components for building shareable analysis pipelines. In Proceedings of the Workshop on Open Infrastructures and Analysis Frameworks for HLT (OIAF4HLT), pages 1–11, Dublin, Ireland.
  6. Detmar Meurers, Ramon Ziai, Niels Ott, and Stacey Bailey. 2011. Integrating parallel analysis modules to evaluate the meaning of answers to reading comprehension questions. IJCEELL. Special Issue on Automatic Free-text Evaluation, 21(4):355–369. http://purl.org/dm/papers/ meurers-ziai-ott-bailey-11.html. ↗
  7. Jinho D Choi and Martha Palmer. 2012. Fast and robust part-of-speech tagging using dynamic model selection. In Proceedings of the 50th Annual Meeting of the ACL, pages 363–367. find
  8. William DeSmedt. 1995. Herr Kommissar: An ICALL conversation simulator for intermediate German. In V. Melissa Holland, Jonathan Kaplan, and Michelle Sams, editors, Intelligent Language Tutors: Theory Shaping Technology, pages 153–174. Lawrence Erlbaum Associates Inc., New Jersey. find
  9. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of English. Natural Language Engineering, 7(3):207–233. ↗
  10. Edward F. Moore. 1959. The shortest path through a maze. In Proceedings of the International Symposium on the Theory of Switching, pages 285–292. Harvard University Press. find
  11. Thilo Götz and Oliver Suhre. 2004. Design and implementation of the uima common analysis system. IBM Systems Journal, 43(3):476–489. ↗
  12. Noriko Nagata. 2002. BANZAI: An application of natural language processing to web-based language learning. CALICO Journal, 19(3):583–599. find
  13. Trude Heift. 2003. Multiple learner errors and meaningful feedback: A challenge for ICALL systems. CALICO Journal, 20(3):533–548. ↗
  14. Noriko Nagata. 2009. Robo-Sensei’s NLP-based error detection and feedback generation. CALICO Journal, 26(3):562–579. ↗
  15. Trude Heift and Mathias Schulze. 2007. Errors and Intelligence in Computer-Assisted Language Learning: Parsers and Pedagogues. Routledge. find
  16. Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, and Detmar Meurers. 2017. Developing a web-based workbook for english supporting the interaction of students and teachers. In Proceedings of the Joint 6th Workshop on NLP for Computer Assisted Language Learning and 2nd Workshop on NLP for Research on Language Acquisition, pages 36–46. http://aclweb.org/anthology/W17-0305.pdf. ↗
  17. Ronja Laarmann-Quante. 2016. Automating multi-level annotations of orthographic properties of German words and children’s spelling errors. In Language Teaching, Learning and Technology, pages 14–22. http://dx.doi.org/10.21437/LTLT.2016-3. ↗
  18. Sébastien L’Haire and Anne Vandeventer Faltin. 2003. Error diagnosis in the FreeText project. CALICO Journal, 20(3):481–495. ↗
  19. Helmut Schmid. 2005. A programming language for finite state transducers. In Proceedings of the 5th International Workshop on Finite State Methods in Natural Language Processing, pages 308–309. find
  20. Alison Mackey. 2006. Feedback, noticing and instructed second language learning. Applied Linguistics, 27(3):405–430. ↗
  21. David A. Schneider and Kathleen F. McCoy. 1998. Recognizing syntactic errors in the writing of second language learners. In Proceedings of the 17th COLING and the 36th Annual meeting of the ACL, pages 1198–1204, Montreal. ↗
  22. Detmar Meurers. 2012. Natural language processing and language learning. In Carol A. Chapelle, editor, Encyclopedia of Applied Linguistics, pages 4193– 4205. Wiley, Oxford. http://purl.org/dm/papers/ meurers-12.html. ↗
  23. Burr Settles and Brendan Meeder. 2016. A trainable spaced repetition model for language learning. In Proceedings of the 54th Annual Meeting of the ACL, volume 1, pages 1848–1858. ↗
  24. Detmar Meurers. 2015. Learner corpora and natural language processing. In Sylviane Granger, Gaëtanelle Gilquin, and Fanny Meunier, editors, The ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rudzewitz-2018-generating-feedback-english-foreign-language/