Elamur Research⌕
☰
home / papers / 2023 / Sentence-level Feedback Generation for English Language Learners: Does Data Augmentation Help?

Sentence-level Feedback Generation for English Language Learners: Does Data Augmentation Help?

Paper · 2023 · Shabnam Behzad, Amir Zeldes and Nathan Schneider

Summary

The paper addresses feedback comment generation (GenChal 2022: FCG). Given a learner's English sentence and the span containing a preposition error, the system must write a short explanation of what is wrong and how to fix it, not just correct it. The authors fine-tune T5-Large, marking the error span with delimiter characters, on about 4.9k annotated sentences from ICNALE essays. They then test whether pseudo data helps. They extract extra sentences with preposition errors from corrected learner corpora using ERRANT, label them with comments from their own fine-tuned model, and compare multi-stage and combined fine-tuning with balanced and unbalanced amounts of pseudo data. T5 beats the official pointer-generator baseline by roughly 14 test BLEU points and ranks third on the leaderboard. Augmentation gives only marginal gains: a tiny in-domain set helps slightly, and larger out-of-domain sets help less or hurt. The authors also compare BLEU against human judgments and classify the system's mistakes, which gives useful guidance for evaluating explanation generators.

Key points

  • Data: the FCG corpus has 4,868/170/215 train/dev/test sentences from ICNALE, whose essays cover only two topics (part-time jobs, smoking bans). Comments were written in Japanese and then translated to English.
  • The best system, T5-Large with combined fine-tuning on 544 in-domain ICNALE pseudo sentences, reaches 47.25 test BLEU. The official baseline scores 33.40, and plain T5 scores 47.11. Human-evaluation F1 is about 61–62 versus 31.16 for the baseline. The leaderboard entry is 0.472 BLEU, third behind 0.486.
  • Out-of-domain pseudo data came from W&I+LOCNESS (6,973 sentences, or 5,000 in the balanced setting). Combined fine-tuning always edged out multi-stage fine-tuning by up to about 1 BLEU, and the larger unbalanced set lowered scores in both setups.
  • Per-sample comparison showed that BLEU above 60 mostly agrees with human labels. However, about 49 test items (23%) were judged correct by humans despite BLEU below 50.
  • Among outputs judged wrong, about 54% were entirely incorrect, about 22% mixed a right correction with a faulty explanation, and about 14% fell into a third mixed category. The model sometimes repeated the same wrong advice across sentences with the same error type, such as "on" for surfaces.

Interesting ideas

  • I1. Marking the target span by wrapping it in plain delimiter tokens lets a standard seq2seq model condition on a location without changing the architecture. It is a cheap way to focus generation on one region of a sentence.
  • I2. Pseudo data can be built in two steps: first an automatic error-type annotator (ERRANT) filters an unlabelled corrected corpus down to the relevant phenomenon, then the current model labels only those filtered examples. This keeps augmentation targeted rather than generic.
  • I3. A few hundred in-domain pseudo examples beat thousands of out-of-domain ones, and adding more noisy pseudo data actively hurt. Volume of silver data is no substitute for domain match and cleanliness.
  • I4. Checking the automatic metric item by item against human labels showed where it is trustworthy (high scores) and where it misses correct paraphrases. This shows how to calibrate an overlap metric before relying on it.
  • I5. The error taxonomy separates the correctness of the explanation from the correctness of the suggestion. It also notes that a vague but correct comment beats a detailed wrong one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): when generating an explanation about a specific node or phrase, the surface span could be delimited in the input so the text realizer knows exactly which piece the comment concerns.
  • I2 (learn): silver data could be selected by phenomenon (for example, only sentences with preposition-attachment ambiguity) before witness parsers label them, so training effort goes where the model is weak.
  • I3 (learn): silver-data growth should be gated by held-out gains with a negative control, and in-genre text (fables, school tasks) should be preferred over bulk out-of-genre corpora.
  • I4 (back): compressor round-trip outputs scored by n-gram overlap should be spot-checked against meaning-graph equivalence to find the score band where overlap stops reflecting correctness.
  • I5 (answer): explanations that student-readers give for test answers could be graded separately for justification and for final choice, and the system could fall back to a generic hint when its specific reasoning is low-confidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Shabnam Behzad, Amir Zeldes and Nathan Schneider. 2023. Sentence-level Feedback Generation for English Language Learners: Does Data Augmentation Help?. Proceedings of the 16th International Natural Language Generation Conference: Generation Challenges.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{behzad2023sentence,
  title = {{Sentence-level Feedback Generation for English Language Learners: Does Data Augmentation Help?}},
  author = {Shabnam Behzad, Amir Zeldes and Nathan Schneider},
  booktitle = {Proceedings of the 16th International Natural Language Generation Conference: Generation Challenges},
  year = {2023},
  doi = {10.18653/v1/2023.inlg-genchal.8},
  url = {https://doi.org/10.18653/v1/2023.inlg-genchal.8}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.
  2. Kristen Lorincz and Rebekah Gordon. 2012. Difficulties in learning prepositions and possible solutions. Linguistic Portfolios, 1(1):14. find
  3. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  4. Shun Morimoto and Shawn Loewen. 2007. A comparison of the effects of image-schema-based instruction and translation-based instruction on the acquisition of L2 polysemous words. Language Teaching Research, 11(3):347–372. ↗
  5. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215, Hong Kong, China. Association for Computational Linguistics. find
  6. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 825–835, Osaka, Japan. ↗
  7. The COLING 2016 Organizing Committee. find
  8. Ryo Nagata, Masato Hagiwara, Kazuaki Hanawa, Masato Mita, Artem Chernodub, and Olena Nahorna. 2021. Shared task on feedback comment generation for language learners. In Proceedings of the 14th Sylviane Granger. 2014. The computer learner corpus: a versatile new source of data for SLA research. International Conference on Natural Language Generation, pages 320–324, Aberdeen, Scotland, UK. Association for Computational Linguistics. Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 198–208, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  9. Ryo Nagata, Kentaro Inui, and Shin’ichiro Ishikawa. 2020. Creating corpora for research in feedback comment generation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 340–345, Marseille, France. European Language Resources Association. find
  10. Preksha Nema and Mitesh M. Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950–3959, Brussels, Belgium. Association for Computational Linguistics. ↗
  11. Ildiko Pilan, John Lee, Chak Yan Yeung, and Jonathan Webster. 2020. A dataset for investigating the impact of feedback on student revision outcome. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 332–339, Marseille, France. European Language Resources Association. find
  12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  13. Elior Sulem, Omri Abend, and Ari Rappoport. 2018. BLEU is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744, Brussels, Belgium. Association for Computational Linguistics. ↗
  14. Yujin Takahashi, Satoru Katsumata, and Mamoru Komachi. 2020. Grammatical error correction using pseudo learner corpus considering learner’s error tendency. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 27–32, Online. Association for Computational Linguistics. find
  15. Andrea Tyler and Vyvyan Evans. 2003. The semantics of English prepositions: Spatial scenes, Embodied meaning, and Cognition. Cambridge University Press. find
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Neural Information Processing Systems (NIPS).
  17. Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A comprehensive survey of grammatical error correction. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1–51. ↗
  18. Max White and Alla Rozovskaya. 2020. A comparative study of synthetic data generation methods for grammatical error correction. In Proceedings of the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/behzad-2023-sentence-level-feedback-generation-english/