Elamur Research⌕
☰
home / papers / 2016 / Off-topic Response Detection for Spontaneous Spoken English Assessment

Off-topic Response Detection for Spontaneous Spoken English Assessment

Paper · 2016 · Andrey Malinin, Rogier Van Dalen, Kate Knill et al.

Summary

Automated graders of spoken English mostly score pronunciation and fluency. Judging content requires first checking whether a candidate actually answered the question, rather than misunderstanding it or reciting memorised text. Earlier systems measured bag-of-words similarity, such as cosine distance over tf-idf or LSA vectors, between a response and example answers. That discards word order, and when each example answer is kept as a separate point (KNN), classification slows down as the data grows.

The authors propose a topic-adapted RNNLM. A recurrent language model receives an extra context vector describing the question topic, built with LDA or LSA from pooled example answers. A response is classified by ranking its likelihood under each topic. Experiments used ASR and reference transcripts from the Cambridge BULATS business English exam. Off-topic answers were simulated by borrowing valid answers to other questions.

The RNNLM systems beat the KNN baseline on both topic classification and off-topic detection, and they improved more with additional training data. This is a step toward content-aware automatic speaking assessment.

Key points

  • The baseline is a KNN classifier using cosine distance in an LSA topic space, with one point per example response. The proposal feeds a topic vector f_q into both the hidden and output layers of an RNNLM.

  • Topic vectors are computed from merged example responses per question rather than from question text. Related parts of multi-part questions share one topic label through a question→topic mapping M.

  • Systems were trained on 490 and 10,004 candidates. Using N-best topic hypotheses lowered false rejection for all systems, and the RNNs benefited from larger N more than KNN did.

  • Off-topic responses were sampled in two ways:

    • naive: taken from any section.
    • directed: taken from the same section, which is harder.

  • Equal Error Rates on EVAL (directed/naive) were:

    SystemDirected EER (%)Naive EER (%)
    KNN12.59.0
    RNN18.06.0
    RNN25.04.5

    RNN1 outperformed KNN at every ROC operating point.

  • One limitation is that a new question needs example responses before deployment. These can come from ASR output rather than hand transcription.

Interesting ideas

  • I1. Instead of measuring vector distance to a topic, the method scores how probable the whole word sequence is under each topic-conditioned model, so word order counts toward relevance.
  • I2. Negatives are built by reusing correct answers to other questions, in an easy "any section" variant and a hard "same section" variant. The score is the expected error over all such selections, not one random draw.
  • I3. Topics are defined by pooling many real answers rather than the prompt text, and sub-questions sharing a scenario are merged under one label. This makes the topic representation robust to answer variety.
  • I4. An ROC curve is traced by varying how many top-ranked topics count as acceptable, giving a single knob for trading false accepts against false rejects.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A reader could check whether a generated or student answer fits the question by comparing its plausibility under the asked question's frame against alternative questions, rather than by word overlap.
  • I2 (learn): Evaluation with negative controls could use the same two tiers, reporting both random and same-domain distractor answers to show whether the system really discriminates close cases.
  • I3 (world): Knowledge about a tale or school topic could be anchored in aggregated accepted answers, with related sub-questions grouped so the world model is not split into confusingly similar fragments.
  • I4 (answer): A QA reader that accepts an answer if it matches any of its top-N interpretations could expose N as a calibrated, explainable strictness setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Latent semantic indexing · 29 works

Cite

Cite as Andrey Malinin, Rogier Van Dalen, Kate Knill, Yu Wang and Mark Gales. 2016. Off-topic Response Detection for Spontaneous Spoken English Assessment. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1102

@inproceedings{malinin2016off,
  title = {{Off-topic Response Detection for Spontaneous Spoken English Assessment}},
  author = {Andrey Malinin, Rogier {Van Dalen}, Kate Knill, Yu Wang and Mark Gales},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1102},
  url = {https://doi.org/10.18653/v1/p16-1102}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, March. ↗
  2. Lucy Chambers and Kate Ingham. 2011. The BU-LATS online speaking test. Research Notes, 43:21– 25. find
  3. Xie Chen, Yongqiang Wang, Xunying Liu, Mark J.F. Gales, and P.C. Woodland. 2014. Efficient GPU-based Training of Recurrent Neural Network Language Models Using Spliced Sentence Bunch. In Proc. INTERSPEECH. find
  4. Xie Chen, Tian Tan, Xunying Liu, Pierre Lanchantin, Moquan Wan, Mark J.F. Gales, and Philip C. Woodland. 2015. Recurrent Neural Network Language Model Adaptation for Multi-Genre Broadcast Speech Recognition. In Proc. INTERSPEECH. find
  5. X. Chen, X. Liu, Y. Qian, M.J.F. Gales, and P.C. Woodland. 2016. CUED-RNNLM – an open-source toolkit for efficient training and evaluation of recurrent neural network language models. In Proc. ICASSP. find
  6. Keelan Evanini and Xinhao Wang. 2014. Automatic detection of plagiarized spoken responses. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications. find
  7. Keelan Evanini, Shasha Xie, and Klaus Zechner. 2013. Prompt-based Content Scoring for Automated Spoken Language Assessment. In Proc. NAACL-HLT. find
  8. Thomas L. Griffiths and Mark Steyvers. 2004. Finding Scientific Topics. Proceedings of the National Academy of Sciences, 101:5228–5235. ↗
  9. Thomas K Landauer, Peter W. Foltz, and Darrell Laham. 1998. Introduction to Latent Semantic Analysis. Discourse Processes, 25:259–284. find
  10. Xunying Liu, Y. Wang, Xie Chen, Mark J.F. Gales, and Philip C. Woodland. 2014. Efficient Lattice Rescoring using Recurrent Neural Network Language Models. In Proc. INTERSPEECH. find
  11. Angeliki Metallinou and Jian Cheng. 2014. Using Deep Neural Networks to Improve Proficiency Assessment for Children English Language Learners. In Proc. INTERSPEECH. find
  12. Tomas Mikolov and Geoffrey Zweig. 2012. Context Dependent Recurrent Neural Network Language Model. In Proc. IEEE Spoken Language Technology Workshop (SLT). find
  13. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent Neural Network Based Language Model. In Proc. INTERSPEECH. find
  14. Tomas Mikolov. 2012. Statistical Language Models Based on Neural Networks. Ph.D. thesis, Brno University of Technology. find
  15. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  16. Xuan-Hieu Phan and Cam-Tu Nguyen. 2007. GibbsLDA++: A C/C++ implementation of latent Dirichlet allocation (LDA). http://gibbslda.sourceforge.net/. ↗
  17. Barbara Seidlhofer. 2005. English as a lingua franca. ELT journal, 59(4):339. ↗
  18. A Stolcke. 2002. SRILM – an extensible language modelling toolkit. In Proc. ICSLP. find
  19. Haipeng Wang, Anton Ragni, Mark J. F. Gales, Kate M. Knill, Philip C. Woodland, and Chao Zhang. 2015. Joint Decoding of Tandem and Hybrid Systems for Improved Keyword Spotting on Low Resource Languages. In Proc. INTERSPEECH. find
  20. Shasha Xie, Keelan Evanini, and Klaus Zechner. 2012. Exploring Content Features for Automated Speech Scoring. In Proc. NAACL-HLT. find
  21. Helen Yannakoudakis. 2013. Automated assessment of English-learner writing. Technical Report UCAM-CL-TR-842, University of Cambridge Computer Laboratory. find
  22. Su-Youn Yoon and Shasha Xie. 2014. Similarity-Based Non-Scorable Response Detection for Automated Speech Scoring. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications. find
  23. Steve Young, Gunnar Evermann, Mark J. F. Gales, Thomas Hain, Dan Kershaw, Xunying (Andrew) Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, Valtcho Valtchev, and Phil Woodland. 2009. The HTK book (for HTK Version 3.4.1). University of Cambridge. find
  24. Klaus Zechner, Derrick Higgins, Xiaoming Xi, and David M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10):883–895. Spoken Language Technology for Education Spoken Language. ↗
  25. Chau Zhang and Philip C. Woodland. 2015. A General Artificial Neural Network Extension for HTK. In Proc. INTERSPEECH. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malinin-2016-off-topic-response-detection-spontaneous/