Elamur Research⌕
☰
home / papers / 2025 / Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations

Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations

Paper · 2025 · Rena Gao, Carsten Roever and Jey Han Lau

Summary

Most dialogue datasets and quality metrics come from native-speaker conversations and focus on fluency or coherence. They overlook how English as a Second Language (ESL) speakers handle interaction, for example how they develop topics, adjust register, or open and close a conversation. The authors propose a two-tier annotation framework for ESL dialogues. At the dialogue level, each conversation receives four interactivity scores on a 1–5 scale. Below that, 17 fine-grained linguistic features are marked as spans, split into token-level and utterance-level phenomena. Using this scheme they build and release SLEDE, an annotated ESL dialogue corpus. They then train simple classifiers that predict the interactivity scores from the micro-level features. These feature-based models outperform a BERT baseline that reads the raw dialogue. Some features, such as reference words, turn out to be strongly linked to interactivity quality. The work therefore connects low-level linguistic signals to higher-level communicative skill and points toward tools for assessing spoken ESL interaction.

Key points

  • Interactivity labels: the four dialogue-level labels are topic management, tone appropriateness, conversation opening and conversation closing.
  • Micro-level features: the 17 features include 7 token-level ones (e.g., reference word, noun & verb collocation, code-switching, routinized resources) and utterance-level ones (e.g., backchannels, feedback in next turn, collaborative finishes, epistemic copulas).
  • SLEDE corpus: SLEDE (Second Language English Dialogue Evaluation) contains 120 dialogues. The authors report strong inter-annotator agreement and very long dialogues. Data and code are on GitHub (RenaGao/2024InteractiveMetrics).
  • Models and ablation: the classifiers are Logistic Regression, Random Forest and Naive Bayes, compared with a BERT baseline. In the ablation, token-level features slightly beat utterance-level ones. Combining both sets works best in most settings; the exceptions are RF for topic and tone.
  • Main limitation: the models are not end-to-end, because they need manually annotated micro-level features as input. Automating the extraction of these features is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Rena Gao, Carsten Roever and Jey Han Lau. 2025. Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gao2025interaction,
  title = {{Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations}},
  author = {Rena Gao, Carsten Roever and Jey Han Lau},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.729/}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Makoto Abe and Carsten Roever. 2019. Interactional competence in l2 text-chat interactions: First-idea proffering in task openings. Journal of Pragmatics, 144:1–14. ↗
  2. Evelina D Galaczi. 2014. Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests? Applied linguistics, 35(5):553–574. ↗
  3. Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022. Deam: Dialogue coherence evaluation using amr-based semantic manipulations. arXiv preprint arXiv:2203.09711. ↗
  4. Yugo Hayashi. 2020. Gaze awareness and metacognitive suggestions by a pedagogical conversational agent: an experimental investigation on interventions to support collaborative learning process and performance. International Journal of Computer-Supported Collaborative Learning, 15(4):469–498. ↗
  5. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. ↗
  6. Daniel MK Lam. 2018. What counts as “responding”? contingency on previous speaker contribution as a feature of interactional competence. Language Testing, 35(3):377–401. find
  7. Holy Lovenia, Bryan Wilie, Willy Chung, Min Zeng, Samuel Cahyawijaya, Su Dan, and Pascale Fung. 2022. Clozer: Adaptable data augmentation for cloze-style reading comprehension. arXiv preprint arXiv:2203.16027. ↗
  8. Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David Traum, and Maxine Eskenazi. 2022. Interactive evaluation of dialog track at dstc9. arXiv preprint arXiv:2207.14403. ↗
  9. John Pill. 2016. Drawing on indigenous criteria for more authentic assessment in a specific-purpose language test: Health professionals interacting with patients. Language Testing, 33(2):175–193. ↗
  10. Thomas Warren. 2017. Cross-cultural Communication: Perspectives in theory and practice. Routledge. find
  11. Steve Whittaker, Yvonne Rogers, Elena Petrovskaya, and Hongbin Zhuang. 2021. Designing personas for expressive robots: Personality in the new breed of moving, speaking, and colorful social home robots. J. Hum.-Robot Interact., 10(1). find
  12. Jingxuan Wu and Carsten Roever. 2021. Proficiency and preference organization in second language mandarin chinese refusals. The Modern Language Journal, 105(4):897–918. ↗
  13. Foong Ha Yap and Anindita Sahoo. 2024. Versatile copulas and their stance-marking uses in conversational odia, an indo-aryan language. Lingua, 297:103641. ↗
  14. Juan-Pedro Rica-Peromingo. 2009. The Status of English in Spain, pages 168–174. find
  15. Carsten Roever and David W Dai. 2021. Reconceptualizing interactional competence for language testing. Assessing speaking in context: Expanding the construct and its applications, pages 23–49. find
  16. Carsten Roever and Naoki Ikeda. 2023. The relationship between l2 interactional competence and proficiency. Applied Linguistics, page amad053. ↗
  17. Emanuel A Schegloff. 1968. Sequencing in conversational openings 1. American anthropologist, 70(6):1075–1095. find
  18. Emanuel A Schegloff and Harvey Sacks. 1973. Opening up closings. ↗
  19. Isis H Settles, Martinque K Jones, NiCole T Buchanan, and Kristie Dotson. 2021. Epistemic exclusion: Scholar (ly) devaluation that marginalizes faculty of color. Journal of Diversity in Higher Education, 14(4):493. ↗
  20. Abdullayev Shaxobiddin. 2024. A discourse analysis of modal verbs in modern english: Patterns and functions. Journal of new century innovations, 50(2):145– 147. find
  21. Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. arXiv preprint arXiv:2005.00583. ↗
  22. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗
  23. Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2025-interaction-matters-evaluation-framework-interactive/