Elamur Research⌕
☰
home / papers / 2023 / PEEP-Talk: A Situational Dialogue-based Chatbot for English Education

PEEP-Talk: A Situational Dialogue-based Chatbot for English Education

Paper · 2023 · Seungjun Lee, Yoonna Jang, Chanjun Park et al.

Summary

Learners of English as a foreign language rarely get to practise realistic conversation. Existing tutoring chatbots tend to be rule-based, cope poorly with off-topic input, and give little feedback. PEEP-Talk addresses this with three parts:

  • Conversation module: DialoGPT fine-tuned on a new corpus, SITUATION-CHAT, which was built from AI Hub's Korean-English dialog corpus. Human annotators enriched its predefined situations with descriptions.
  • Context detector (CD): built on XLNet, it scores how well a learner's turn fits the current situation and how linguistically acceptable it is. It switches to a new situation when the talk drifts.
  • Grammar error correction (GEC) module: gives feedback on the learner's errors.

DialoGPT LARGE gave the best perplexity and response-ranking accuracy. In a two-week controlled study, users improved in speaking and grammar and reported less learning anxiety than a control group. Users liked the conversation and correction features, but rated the situation-switching detector lower. The work shows that an education chatbot can combine open generation with situation grounding and feedback, and be checked with real learners.

Key points

  • SITUATION-CHAT:
    • Train split: 16,298 dialogues and 65,192 utterances.
    • Validation and test splits: 1,000 dialogues and 4,000 utterances each.
    • Average turn length is 14 words, across 303 situations (shopping, traffic, travel, etc.).
  • Fine-tuning: follows Wolf et al. (2019b). The model input is the situation, the dialogue history and candidate answers.
  • Test results:
    • DialoGPT LARGE reached PPL 12.04 and Hits@1 0.877, with the gold reply ranked among 19 random candidates.
    • The Profile Memory baseline scored PPL 45.02 and Hits@1 0.069.
    • GPT, GPT-2 and smaller DialoGPT variants scored about 0.84–0.86 Hits@1.
  • User study (ANCOVA, pre/post tests):
    • Groups: 18 experimental and 17 control participants; the experimental group used the system 30 minutes a day for two weeks.
    • Speaking improved in all dimensions except accuracy.
    • Grammar adjusted mean: 64.51 vs 63.80.
    • Anxiety adjusted mean: 3.04 vs 3.14.
  • Satisfaction survey:
    • Over 70% found the replies natural and contextual.
    • 70–80% found situational dialogue and grammar correction useful.
    • The CD module got only 46%, apparently because it changed the situation when users wanted to stay on it.

Interesting ideas

  • I1. Each dialogue is tied to an explicit, human-written description of the scene, so generation is anchored to a named situation rather than only to history.
  • I2. The CD scores how well each new turn fits the active situation and triggers a switch when the fit falls too low.
  • I3. Feedback is split into separate, inspectable signals (situation fit, acceptability, concrete correction) instead of one opaque grade.
  • I4. Hits@1 against 19 randomly drawn distractor replies is a cheap ranking test that complements perplexity.
  • I5. Users disliked the detector because it switched topic too eagerly, which shows that a confident shift detector needs restraint and user override.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): a text's world could start from a declared scenario frame, such as "at a shop" or "job interview", that sets expected entities and events and makes later inferences traceable to that prior.
  • I2 (frames): a similar per-sentence fit score against the current frame set could flag topic shifts in tales or dialogues and open a new discourse segment, with the score kept as an explanation.
  • I3 (answer): when grading or answering school tests, report relevance, well-formedness and the specific fix as distinct outputs, so every verdict can be explained.
  • I4 (learn): candidate parses or answers could be scored against sampled negative alternatives, giving a built-in negative control alongside raw likelihood.
  • I5 (frames): segment or speech-act switches should need sustained evidence across several sentences before changing context, to avoid breaking coherent discourse on one noisy sentence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institute for Infocomm ResearchSingapore · Singapore43 works · 14,598 citations🎓Korea UniversitySeoul · South Korea26 works · 9,490 citations🏢UpStageYongin-si · South Korea6 works · 728 citations🎓Hankuk University of Foreign StudiesSeoul · South Korea2 works · 36 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 works

Cite

Cite as Seungjun Lee, Yoonna Jang, Chanjun Park, Jungseob Lee, Jaehyung Seo, Hyeonseok Moon, Sugyeong Eo, Seounghoon Lee, Bernardo Yahya and Heuiseok Lim. 2023. PEEP-Talk: A Situational Dialogue-based Chatbot for English Education. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-demo.18

@inproceedings{lee2023peep,
  title = {{PEEP-Talk: A Situational Dialogue-based Chatbot for English Education}},
  author = {Seungjun Lee, Yoonna Jang, Chanjun Park, Jungseob Lee, Jaehyung Seo, Hyeonseok Moon, Sugyeong Eo, Seounghoon Lee, Bernardo Yahya and Heuiseok Lim},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-demo.18},
  url = {https://doi.org/10.18653/v1/2023.acl-demo.18}
}

References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Al Hosni. 2014. Speaking difficulties encountered by young efl learners. International Journal on Studies in English Language and Literature (IJSELL), 2(6):22–30. find
  2. Reem Alsadoon. 2021. Chatting with ai bot: Vocabulary learning assistant for saudi efl learners. English Language Teaching, 14(6):135–157. ↗
  3. Ali A Altalbe and Brett Wilkinson. 2013. Designing games for learning english as a second language. In 2013 International Conference on Computer Applications Technology (ICCAT), pages 1–7. IEEE. find
  4. Carol A Chapelle. 2005. Computer-assisted language learning. In Handbook of research in second language teaching and learning, pages 767–780. Routledge. find
  5. Shu-Yun Chien, Gwo-Jen Hwang, and Morris Siu-Yung Jong. 2020. Effects of peer assessment within the context of spherical video-based virtual reality on efl students’ english-speaking performance and learning perceptions. Computers & Education, 146:103751. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  7. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. ↗
  8. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  9. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology. find
  10. L Fryer and K Nakao. 2009. Assessing chatbots for efl learner use. In JALT2008 conference proceedings, pages 849–857. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  12. Luke Fryer and Rollo Carpenter. 2006. Bots as language learning tools. Language Learning & Technology, 10(3):8–14. ↗
  13. Cynthia Luna Scott. 2015. The futures of learning 2: What kind of learning for the 21st century? find
  14. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393. ↗
  15. Nuria Haristiani. 2019. Artificial intelligence (ai) chatbot as language learning medium: An inquiry. In Journal of Physics: Conference Series, volume 1387, page 012020. IOP Publishing. find
  16. Elaine K Horwitz, Michael B Horwitz, and Joann Cope. 1986. Foreign language classroom anxiety. The Modern language journal, 70(2):125–132. ↗
  17. Loae Fakhri Ahmad Jdetawy. 2011. Problems encountered by arab efl learners. Language in India, 11(3). find
  18. Hyangeun Ji, Insook Han, and Yujung Ko. 2022. A systematic review of conversational ai in language education: focusing on the collaboration with human teachers. Journal of Research on Technology in Education, pages 1–16. ↗
  19. Jiyou Jia and Meixian Ruan. 2008. Use chatbot csiec to facilitate the individual learning in english instruction: A case study. In International conference on intelligent tutoring systems, pages 706–708. Springer. find
  20. Na-Young Kim, Yoonjung Cha, and Hea-Suk Kim. 2019. Future english learning: Chatbots and artificial intelligence. Multimedia-Assisted Language Learning, 22(3):32–53. find
  21. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  22. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 588–593. find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. ↗
  24. Chanjun Park, Yeongwook Yang, Chanhee Lee, and Heuiseok Lim. 2020. Comparison of the evaluation metrics for neural grammatical error correction with overcorrection. IEEE Access, 8:106264–106272. ↗
  25. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  27. Curtis L Klinghoffer. 2008. Situational dialogues in a community college: English as a second language curriculum. Online Submission. find
  28. Claire Kramsch. 2014. Teaching foreign languages in an era of globalization: Introduction. The modern language journal, 98(1):296–311. ↗
  29. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  30. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957. ↗
  31. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126.
  32. Kanyakorn Sermsook, Jiraporn Liamnimitr, and Ratta-neekorn Pochakorn. 2017. The impact of teacher corrective feedback on efl student writers’ grammatical improvement. English Language Teaching, 10(10):43–49. ↗
  33. Tzu-Yu Tai and Howard Hao-Jan Chen. 2020. The impact of google assistant on adolescent efl learners’ willingness to communicate. Interactive Learning Environments, pages 1–18. ↗
  34. George Terzopoulos and Maya Satratzemi. 2020. Voice assistants and smart speakers in everyday life and in education. Informatics in Education, 19(3):473–490. ↗
  35. Gladys Tyen, Mark Brenchley, Andrew Caines, and Paula Buttery. 2022. Towards an open-domain chatbot for language practice. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 234–249. find
  36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR. ↗
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019a. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  38. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019b. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149. ↗
  39. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  40. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2023-peep-talk-situational-dialogue-based/