Elamur Research⌕
☰
home / papers / 2016 / Towards Using Conversations with Spoken Dialogue Systems in the Automated Assessment of Non-Native Speakers of English

Towards Using Conversations with Spoken Dialogue Systems in the Automated Assessment of Non-Native Speakers of English

Paper · 2016 · Diane Litman, Steve Young, Mark Gales et al.

Summary

Human-run English speaking exams usually include a live conversation with an examiner, but automated scoring has so far handled only monologue tasks, such as answering a prompt. Litman and colleagues ran a pilot to see whether ready-made components could close this gap. They used existing spoken dialogue systems to gather conversations with non-native speakers of English, and they scored those conversations two ways: by a human CEFR expert and by an existing grader built for prompted, non-interactive speech. There were three corpora, each from a different system and a different recruitment method: Laptop, Restaurant, and Bus. The Combined set had 67 dialogues. The expert could assess most of them, but rating was hard, because a well-working system draws out little speech and a failing one makes the dialogue unnatural. The automatic grader, GP-BULATS, gave consistently lower scores than the human. Even so, its scores correlated moderately with the human ones on the AMT-collected data and on the combined set. The authors present the results as a proof of concept: dialogue-based assessment looks feasible, but data and systems need adaptation.

Key points

  • Three corpora, three recruitment methods:

    • Laptop: Cambridge SDS, accessed by phone, recruited via AMT with a location restriction.
    • Restaurant: VocalIQ SDS, accessed through a browser microphone, recruited via AMT with the task title written only in Hindi.
    • Bus: non-native control-condition callers from the first Spoken Dialogue Challenge; 22 of the 75 control callers were non-native.

  • Human assessment: a single CEFR expert rated 55 of the 67 dialogues; poor audio was a common reason for skipping the rest. When the expert gave two adjacent levels, the pair was recoded as an in-between level, such as "B1B2".

  • Recruitment effects: AMT recruitment produced more mid-level speakers than the SDC callers did, but all three pools leaned toward high proficiency. Restaurant scores were significantly lower than the others (ANOVA with Bonferroni correction).

  • The grader: GP-BULATS is a Gaussian process over audio features and ASR-based fluency features. It outputs a 0–30 score together with an uncertainty estimate. It was trained on 994 BULATS learners and reached Pearson r = 0.83 on 226 test speakers.

  • Results on dialogue speech (Pearson r between human and automatic grades):

    CorpusrpNote
    Restaurant.69.01significant
    Laptop.41.07trend
    Bus−.11n.s.fewest words per turn (2.6)
    Combined (n=50).59.01significant

    Five Bus dialogues could not be graded at all.

Interesting ideas

  • I1. When a rater hesitates between two neighbouring ordinal labels, the authors keep the hesitation as a new half-step label instead of forcing a choice; this preserves uncertainty while still giving a usable ordered scale.
  • I2. The strong combined correlation (.59) hides a negative correlation on one subset (Bus), so the paper reports per-corpus figures next to the pooled one.
  • I3. Recruitment channel alone (location filter, Hindi task title, research callers) changed the skill profile of the collected population, so the way data is gathered shapes what an evaluation actually measures.
  • I4. The grader returns an uncertainty estimate and simply cannot score inputs where its features are undefined (no recognised words), rather than inventing a grade.
  • I5. An off-the-shelf model can be systematically biased in absolute terms (scoring everyone too low) yet still rank speakers roughly correctly, so agreement and calibration are worth checking separately.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers or LLM teacher tiers split between two adjacent judgements, store a "between" silver label rather than discarding the item or picking one arbitrarily.
  • I2 (learn): always break evaluation scores down by text source or genre so that a failing slice is not masked by the overall average.
  • I3 (learn): tag every test text with its collection route and check that negative controls and test sets do not all come from one easy channel.
  • I4 (answer): readers should abstain explicitly with a reason when the tree or world lacks the material a question needs, instead of guessing.
  • I5 (learn): when reusing an external parser or scorer outside its domain, measure rank agreement and offset separately, and recalibrate before trusting raw outputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 works

Cite

Cite as Diane Litman, Steve Young, Mark Gales, Kate Knill, Karen Ottewell, Rogier van Dalen and David Vandyke. 2016. Towards Using Conversations with Spoken Dialogue Systems in the Automated Assessment of Non-Native Speakers of English. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-3635

@inproceedings{litman2016using,
  title = {{Towards Using Conversations with Spoken Dialogue Systems in the Automated Assessment of Non-Native Speakers of English}},
  author = {Diane Litman, Steve Young, Mark Gales, Kate Knill, Karen Ottewell, Rogier {van Dalen} and David Vandyke},
  booktitle = {Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2016},
  doi = {10.18653/v1/w16-3635},
  url = {https://doi.org/10.18653/v1/w16-3635}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Øistein E. Andersen, Helen Yannakoudakis, Fiona Barker, and Tim Parish. 2013. Developing and testing a self-assessment and tutoring system. In Proceedings 8th Workshop on Innovative Use of NLP for Building Educational Applications, pages 32–41. find
  2. Christopher M Mitchell, Keelan Evanini, and Klaus Zechner. 2014. A trialogue-based spoken dialogue system for assessment of english language learners. In Proceedings International Workshop on Spoken Dialogue Systems. find
  3. Suma Bhat, Huichao Xue, and Su-Youn Yoon. 2014. Shallow analysis based assessment of syntactic complexity for automated speech scoring. In Proceedings 52nd Annual Meeting of the Association for Computational Linguistics, pages 1305–1315. find
  4. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialog state tracking using recurrent neural networks. In Proceedings 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing, pages 794–799. find
  5. Alan W. Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, Jason D. Williams, Kai Yu, Steve Young, and Maxine Eskenazi. 2011. Spoken dialog challenge 2010: Comparison of live and control test results. In Proceedings of SIGDIAL, pages 2–7. find
  6. Rod D Roscoe, Danica Kugler, Scott A Crossley, Jennifer L Weston, and Danielle S McNamara. 2012. Developing pedagogically-guided threshold algorithms for intelligent automated essay feedback. In FLAIRS Conference. find
  7. Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004. Automated essay evaluation: The criterion online writing service. Ai Magazine, 25(3):27. ↗
  8. Paul Seedhouse, Andrew Harris, Rola Naeb, Eda Üstünel, et al. 2014. Relationship between speaking features and band descriptors: A mixed methods study, the. IELTS Research Reports Online Series, page 30. find
  9. CEFR. 2001. Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press. find
  10. Vinay Shashidhar, Nishant Pandey, and Varun Aggarwal. 2015. Automatic spontaneous speech grading: A novel feature derivation technique using the crowd. In Proceedings 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing, pages 1085–1094. find
  11. Pei-Hao Su, Chuan-Hsun Wu, and Lin-Shan Lee. 2015. A recursive dialogue game for personalized computer-aided pronunciation training. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 23(1):127– 141. ↗
  12. Jesse Thomason and Diane Litman. 2013. Differences in user responses to a wizard-of-oz versus automated system. In Proceedings of NAACL-HLT, pages 796– 801. find
  13. Rogier C. van Dalen, Kate M. Knill, and Mark J. F. Gales. 2015. Automatically grading learners’ English using a Gaussian process. In Proceedings Sixth Workshop on Speech and Language Technology in Education (SLaTE), pages 7–12. find
  14. David Vandyke, Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialogue success classifiers for policy training. In ASRU. find
  15. Xinhao Wang, Keelan Evanini, and Klaus Zechner. 2013. Coherence modeling for the automated assessment of spontaneous spoken responses. In Proceedings Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 814–819. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/litman-2016-conversations-spoken-dialogue-systems-automated/