Elamur Research⌕
☰
home / papers / 2024 / EDEN: Empathetic Dialogues for English Learning

EDEN: Empathetic Dialogues for English Learning

Paper · 2024 · Li Siyan, Teresa Shao, Zhou Yu et al.

Summary

Chatbots are often used as English conversation partners, but whether they actually help learners is rarely tested. Earlier classroom research linked how supportive students find their teacher (perceived affective support, PAS) to their L2 grit, meaning persistence in language learning. This paper asks whether the same link holds for a chatbot. The authors build EDEN, an open-source spoken chit-chat system for English practice. It combines several parts: a grammar-correction model trained for transcribed speech rather than written text, an open-domain conversation model, personalization options, and empathetic feedback that is triggered when the learner shows negative sentiment or pauses for a long time. In a preliminary user study they compare three conditions: no empathetic feedback, fixed empathetic feedback and adaptive empathetic feedback. The adaptive version produced the highest PAS. One PAS component, perceived appreciation, predicted higher grit after the interaction, but overall PAS did not. The work is a first step toward dialogue interventions aimed at learner persistence, and it releases its data, code and checkpoints.

Key points

  • The grammar-correction training data has 7,894 sentence–correction pairs.
    • The inputs are Whisper-Medium transcripts of native Mandarin speakers.
    • GPT-4-Turbo produced the corrections, because no ground truth existed.
  • Two local models were fine-tuned with PEFT on one GPU: Llama-2 7B and Flan-T5-XL.
    • GPT-4 judged both corrections valid on 61.6% of the validation utterances where the models disagreed.
    • Llama-2 came out slightly ahead overall.
  • Prolific raters (six native speakers) compared the two models in two studies:
    • In a precision-oriented test on 40 sentences, Llama-2 won 46.3% of comparisons and lost 35.8%.
    • In a recall-oriented test on 31 sentences that an expert had marked as erroneous, the two models were roughly even (39.3% vs 40.3%).
  • The pipeline routes each turn in a fixed order:
    • Empathy triggers (negative sentiment, long pauses) are checked first.
    • Template-based grammar feedback with a correction hierarchy comes next.
    • Otherwise the turn goes to the chit-chat model.
    • Learners' follow-up questions about feedback go to ChatGPT, and a transition module then resumes the conversation.
  • Further findings:
    • PAS correlated positively with conversation-quality ratings.
    • English proficiency did not correlate with grit change.
    • Design choices were informed by a Twitter/X survey with over 450 responses.

Interesting ideas

  • I1. The models were compared only on inputs where their outputs differ, with one test set built for precision and another for recall, which makes a limited human-rating budget go much further.
  • I2. A strong but costly LLM labels in-domain data once, and a small local model is then trained on those labels for the per-utterance task, avoiding an API call on every input.
  • I3. Corrections are ranked in a hierarchy so the user is not flooded with every error at once, and only the most relevant issue is surfaced.
  • I4. Requests the core model cannot handle are routed to a separate module, and an explicit transition step then hands control back to the main flow.
  • I5. Non-lexical signals such as hesitation and negative sentiment act as explicit, inspectable triggers that select a different response strategy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When witness parsers disagree, the disputed arcs could be sampled into separate precision-style and recall-style sets for LLM or human adjudication, instead of rating whole trees.
  • I2 (learn): This matches the LLM-teacher-tier idea; expensive teacher annotations could be distilled into the perceptron or gparse parser for spoken or learner-style English.
  • I3 (back): Explanations generated from the meaning graph could follow a priority order and report the most important finding first, keeping the output readable.
  • I4 (answer): Readers could detect out-of-scope questions, send them to a dedicated handler, and record that handoff in the explanation trace instead of forcing an answer.
  • I5 (frames): Pragmatic analysis could expose affect and hesitation cues as named features that feed speech-act and implicature decisions transparently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 works

Cite

Cite as Li Siyan, Teresa Shao, Zhou Yu and Julia Hirschberg. 2024. EDEN: Empathetic Dialogues for English Learning. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.200

@inproceedings{siyan2024eden,
  title = {{EDEN: Empathetic Dialogues for English Learning}},
  author = {Li Siyan, Teresa Shao, Zhou Yu and Julia Hirschberg},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.200},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.200}
}

References

55 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emmanuel Ayedoun, Yuki Hayashi, and Kazuhisa Seta. 2015. A conversational agent to encourage willingness to communicate in the context of english as a foreign language. Procedia Computer Science, 60:1433–1442. find
  2. Emmanuel Ayedoun, Yuki Hayashi, and Kazuhisa Seta. 2020. Toward personalized scaffolding and fading of motivational support in l2 learner–dialogue agent interactions: an exploratory study. IEEE Transactions on Learning Technologies, 13(3):604–616. ↗
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  4. Christopher Bryant and Ted Briscoe. 2018. Language model based grammatical error correction without annotated training data. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 247–253, New Orleans, Louisiana. Association for Computational Linguistics. find
  5. Leshem Choshen, Matanel Oren, Dmitry Nikolaev, and Omri Abend. 2021. Serrant: a syntactic classifier for english grammatical error types. arXiv preprint arXiv:2104.02310. ↗
  6. Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023. Analyzing the performance of gpt-3.5 and gpt-4 in grammatical error correction. arXiv preprint arXiv:2303.14342. ↗
  7. Karl Daher, Jacky Casas, Omar Abou Khaled, and Elena Mugellini. 2020. Empathic chatbot response for medical assistance. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pages 1–3. find
  8. Mauro De Gennaro, Eva G Krumhuber, and Gale Lucas. 2020. Effectiveness of an empathic chatbot in combating adverse effects of social exclusion on mood. Frontiers in psychology, 10:495952. ↗
  9. David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. Simsensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 1061–1068. ↗
  10. Sidney D’mello and Art Graesser. 2013. Autotutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4):1–39. find
  11. Jinming Du and Ben Kei Daniel. 2024. Transforming language education: A systematic review of ai-powered chatbots for english as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6:100230. ↗
  12. Majid Elahi Shirvan, Tahereh Taherian, Mojdeh Shah-nama, and Elham Yazdanmehr. 2021. A longitudinal study of foreign language enjoyment and l2 grit: A latent growth curve modeling. Frontiers in Psychology, 12:720326. ↗
  13. Jalil Fathi, Masoud Rahimi, and Ali Derakhshan. 2024. Improving efl learners’ speaking skills and willingness to communicate via artificial intelligence-mediated interactions. System, 121:103254. ↗
  14. Hua Leong Fwa. 2018. An architectural design and evaluation of an affective tutoring system for novice programmers. International Journal of Educational Technology in Higher Education, 15(1):1–19. ↗
  15. Daeun Han. 2021. An analysis of korean efl learners’ experience on english classes using ai chatbot. Robotics & AI Ethics, 6(3):1–9. ↗
  16. Muhammad Asif Hasan, Nurul Fazmidar Mohd Noor, Siti Soraya Binti Abdul Rahman, and Mohammad Mustaneer Rahman. 2020. The transition from intelligent to affective tutoring system: a review and open issues. IEEE Access, 8:204612–204638. ↗
  17. Mae-Hyang Hwang and JeeEun Karin Nam. 2021. Enhancing grit: Possibility and intervention strategies. Multidisciplinary perspectives on grit: Contemporary theories, assessments, applications and critiques, pages 77–93. ↗
  18. Qiaolei Jiang, Yadi Zhang, and Wenjing Pian. 2022. Chatbot as an emergency exist: Mediated empathy for resilience via human-ai interaction during the covid-19 pandemic. Information processing & management, 59(6):103074. ↗
  19. Anisia Katinskaia and Roman Yangarber. 2023. Grammatical error correction for sentence-level assessment in language learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 488–502, Toronto, Canada. Association for Computational Linguistics. ↗
  20. Gholam Hassan Khajavy and Elham Aghaee. 2022. The contribution of grit, emotions and personal bests to foreign language learning. Journal of Multilingual and Multicultural Development, pages 1–15. ↗
  21. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714. ↗
  22. Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. Large language models are state-of-the-art evaluator for grammatical error correction. arXiv preprint arXiv:2403.17540. ↗
  23. Lucas Kohnke. 2023. L2 learners’ perceptions of a chatbot as a potential independent language learning tool. International Journal of Mobile Learning and Organisation, 17(1-2):214–226. ↗
  24. Seungjun Lee, Yoonna Jang, Chanjun Park, Jungseob Lee, Jaehyung Seo, Hyeonseok Moon, Sugyeong Eo, Seounghoon Lee, Bernardo Yahya, and Heuiseok Lim. 2023. PEEP-talk: A situational dialogue-based chatbot for English education. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 190–207, Toronto, Canada. Association for Computational Linguistics.
  25. Chengchen Li and Jean-Marc Dewaele. 2021. How classroom environment and general grit predict foreign language classroom anxiety of chinese efl students. Journal for the Psychology of Language Learning, 3(2):86–98. ↗
  26. Yu Li, Chun-Yen Chen, Dian Yu, Sam Davidson, Ryan Hou, Xun Yuan, Yinghua Tan, Derek Pham, and Zhou Yu. 2022. Using chatbots to teach languages. In Proceedings of the Ninth ACM Conference on Learning@ Scale, pages 451–455. find
  27. Yu Li, Shang Qu, Jili Shen, Shangchao Min, and Zhou Yu. 2023. Curriculum-driven edubot: A framework for developing language learning chatbots through synthesizing conversational data. arXiv preprint arXiv:2309.16804. ↗
  28. Kai-Hui Liang, Sam Davidson, Xun Yuan, Shehan Pan-ditharatne, Chun-Yen Chen, Ryan Shea, Derek Pham, Yinghua Tan, Erik Voss, and Luke Fryer. 2023. Chat-Back: Investigating methods of providing grammatical error feedback in a GUI-based language learning chatbot. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 83–99, Toronto, Canada. Association for Computational Linguistics. find
  29. Hao-Chiang Koong Lin, Ching-Ju Chao, and Tsu-Ching Huang. 2015. From a perspective on foreign language learning anxiety to develop an affective tutoring system. Educational Technology Research and Development, 63:727–747. ↗
  30. Hao-Chiang Koong Lin, Chih-Hung Wu, and Ya-Ping Hsueh. 2014. The influence of using affective tutoring system in accounting remedial instruction on learning performance and usability. Computers in Human Behavior, 41:514–522. ↗
  31. Diane Litman and Kate Forbes-Riley. 2014. Evaluating a spoken dialogue system that detects and adapts to user affective states. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 181–185. find
  32. Xin Tang, Ming-Te Wang, Jiesi Guo, and Katariina Salmela-Aro. 2019. Building grit: The longitudinal pathways between mindset, commitment, grit, and academic outcomes. Journal of youth and adolescence, 48:850–863. ↗
  33. Diane Litman and Scott Silliman. 2004. Itspoke: An intelligent tutoring spoken dialogue system. In Demonstration papers at HLT-NAACL 2004, pages 5–8. find
  34. Yasser Teimouri, Luke Plonsky, and Farhad Tabandeh. 2022. L2 grit: Passion and perseverance for second-language learning. Language Teaching Research, 26(5):893–918. find
  35. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft. ↗
  36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  37. Chanjun Park, Yoonna Jang, Seolhwa Lee, Sungjin Park, and Heuiseok Lim. 2022. FreeTalky: Don’t be afraid! conversations made easier by a humanoid robot using persona-based dialogue. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1242–1248, Marseille, France. European Language Resources Association. find
  38. Andrew Pueschel and Mary L Tucker. 2018. Achieving grit through the growth mindset. Journal of Instructional Pedagogies, 20. find
  39. Annisa Ristya Rahmanti, Hsuan-Chia Yang, Bagas Suryo Bintoro, Aldilas Achmad Nurse-tyo, Muhammad Solihuddin Muhtar, Shabbir Syed-Abdul, and Yu-Chuan Jack Li. 2022. Slimme, a chatbot with artificial empathy for personal weight management: system design and finding. Frontiers in Nutrition, 9:870775. ↗
  40. Gonul Sakiz. 2007. Does teacher affective support matter? An investigation of the relationship among perceived teacher affective support, sense of belonging, academic emotions, academic self-efficacy beliefs, and academic effort in middle school mathematics classrooms. Ph.D. thesis, The Ohio State University. find
  41. Olga C Santos, Mar Saneiro, Jesus G Boticario, and María Cristina Rodriguez-Sanchez. 2016. Toward interactive context-aware affective educational recommendations in computer-assisted language learning. New Review of Hypermedia and Multimedia, 22(1-2):27–57. ↗
  42. Nuobei Shi, Qin Zeng, and Raymond Lee. 2020. The design and implementation of language learning chatbot with xai using ontology and transfer learning. arXiv preprint arXiv:2009.13984. ↗
  43. Silero. 2021. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/ silero-vad. ↗
  44. Li Siyan, Teresa Shao, Zhou Yu, and Julia Hirschberg. 2024. Using adaptive empathetic responses for teaching english. arXiv preprint arXiv:2404.13764. ↗
  45. Amy JC Trappey, Aislyn PC Lin, Kevin YK Hsu, Charles V Trappey, and Kevin LK Tu. 2022. Development of an empathy-centric counseling chatbot system capable of sentimental dialogue analysis. Processes, 10(5):930. ↗
  46. Renee Von Worde. 2003. Students’ perspectives on foreign language anxiety. Inquiry, 8(1):n1. find
  47. Chih Hung Wu, Hao-Chiang Koong Lin, Tao-Hua Wang, Tzu-Hsuan Huang, and Yueh-Min Huang. 2022. Affective mobile language tutoring system for supporting language learning. Frontiers in Psychology, 13:833327. ↗
  48. Wangjiao Wu, Yabing Wang, and Ruifang Huang. 2023. Teachers matter: exploring the impact of perceived teacher affective support and teacher enjoyment on l2 learner grit and burnout. System, 117:103096. ↗
  49. Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 3506–3510. find
  50. Hyejin Yang, Heyoung Kim, Jang Ho Lee, and Dongkwang Shin. 2022. Implementation of an ai chatbot as an english conversation partner in efl speaking classes. ReCALL, 34(3):327–343. ↗
  51. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. Lm-critic: Language models for unsupervised grammatical error correction. arXiv preprint arXiv:2109.06822. ↗
  52. Xun Yuan, Derek Pham, Sam Davidson, and Zhou Yu. 2022. ErAConD: Error annotated conversational dialog dataset for grammatical error correction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 76–84, Seattle, United States. Association for Computational Linguistics. find
  53. Chunpeng Zhai and Santoso Wibowo. 2022. A systematic review on cross-culture, humor and empathy dimensions in conversational chatbots: The case of second language acquisition. Heliyon. A Chatbot Design from Siyan et al. (2024): More Details the survey below. ↗
  54. Siyan et al. (2024) proposes a novel adaptive and empathetic English-teaching chatbot. The chatbot detects heightened negative emotions and prolonged pauses in student speech using an existing wav2vec 2.0 speech emotion detection model (Baevski et al., 2020) and a voice activity detection toolkit (Silero, 2021). When these signals are captured, ChatGPT is used to generate empathetic feedback using the past three student utterances. The authors use DSPy (Khattab et al., 2023) to optimize their ChatGPT prompt such that the resulting colloquial feedback sounds empathetic and contains specific examples. In addition to providing empathetic feedback, the chatbot offers grammatical feedback on student utterances using a fine-tuned Llama-2 (Touvron et al., 2023) model for grammar correction. The grammar correction model is trained on the ErA-ConD dataset (Yuan et al., 2022), which contains written dialogue utterances and their expert grammar corrections. find
  55. The SERRANT (Choshen et al., 2021) package is then used to locate the specific grammatical errors by comparing the generated correction and the original sentence. The grammatical feedback is a combination of a rephrase (e.g. Maybe you meant "had" rather than "has") and a template-based explanation for each error type (Liang et al., 2023). A transition module is designed to aid smoother transitions between different chatbot system components. It connects empathetic or grammatical feedback to the original conversation. The module classifies whether student utterances constitute an English-learning-related query using pre-written rules. If a student utterance is classified as a relevant query, the system prompts ChatGPT to generate an answer for the query, otherwise, a randomly selected pre-defined connector phrase is used to transition back to the original conversation directly. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siyan-2024-eden-empathetic-dialogues-english-learning/