Elamur Research⌕
☰
home / papers / 2019 / Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Paper · 2019 · Xinhao Wang, Keelan Evanini, Matthew Mulholland et al.

Summary

Test takers in high-stakes English speaking exams sometimes recite memorized material from online sources instead of speaking spontaneously. This distorts scores from both human raters and automated scorers. The authors, from ETS, built a classifier that separates such canned spoken responses from genuine ones. Responses are first transcribed with a Kaldi-based ASR system whose language model is adapted to new prompts without supervision. Two feature families are then compared. The first measures text similarity between a transcript and a pool of known source passages, using n-gram overlap and Word Mover's Distance. The second contrasts proficiency measures, such as fluency, across the responses of a single speaker, and needs no source passages at all. The data set is deliberately skewed, with about 43 control responses for each plagiarized one. On it, n-gram overlap features performed best, with an F1 of 0.761. The authors also ran the system on a full live administration, where it caught nearly 90% of confirmed cases. Because experts then only had to review a small flagged subset, the system greatly reduces manual effort.

Key points

  • Data:
    • 1,557 confirmed plagiarized responses to independent speaking tasks, gathered over several years.
    • 224 source passages identified by human raters.
    • A control set of 66,400 responses from 33,200 speakers: 166 test forms with 200 speakers each, giving a 43:1 imbalance.
  • Feature comparison:
    • n-gram overlap features alone reached F1 = 0.761.
    • WMD features alone reached only 0.649.
    • Combining the two gave 0.76, so adding WMD brought no gain.
  • Proficiency features: the source-free speaking-proficiency features were very precise (0.8) but recalled almost nothing (0.009). Canned delivery proved hard to tell apart from spontaneous speech.
  • Operational check:
    • The system was run on one administration of 13,516 responses from 6,758 speakers, of which only 39 (0.29%) were confirmed plagiarized.
    • A union ensemble of random forest and AdaBoost models flagged 850 responses and found 35 of the 39 cases (recall 0.897).
  • Earlier baseline: the authors' previous system reached F1 = 70.6% on simulated data. This work adds the improved ASR, the refined similarity features and the live validation.

Interesting ideas

  • I1. Exact shared word sequences beat semantic-similarity measures here, because copied text keeps idiosyncratic phrasings that paraphrase-tolerant metrics tend to blur.
  • I2. The authors evaluated with a very large pool of negative controls at a realistic base rate, then confirmed the result on a complete live batch rather than trusting balanced test sets.
  • I3. For triage, a response was flagged if any model in the ensemble flagged it, which deliberately maximizes recall because a human reviews everything flagged anyway.
  • I4. Each flagged response came with a label naming the source it probably drew on, which made expert review faster and the decision explainable.
  • I5. The source-free features compared a speaker against their own other responses instead of against external references; this weak but precise signal could still complement content features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): when grading student answers on fables, an n-gram overlap check against the source text could separate copied sentences from answers that were actually inferred from the world model.
  • I2 (learn): negative-control evaluations should be run at realistic error base rates, so that precision claims hold up when the rare target class is drowned in normal cases.
  • I3 (learn): disagreement or any-witness flags from the several parsers could route sentences to the LLM teacher tier, trading cheap false positives for fewer silent errors in silver data.
  • I4 (answer): every answer could carry a pointer to the text span or world fact it relied on, so reviewers can verify it quickly.
  • I5 (frames): register or style shifts within one author's text, measured against that same text's baseline, could serve as a high-precision cue for quoted or borrowed material when attributing speech acts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝Educational Testing ServicePrinceton · United States27 works · 2,212 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian and James V. Bruno. 2019. Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4445

@inproceedings{wang2019application,
  title = {{Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency}},
  author = {Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian and James V. Bruno},
  booktitle = {Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2019},
  doi = {10.18653/v1/w19-4445},
  url = {https://doi.org/10.18653/v1/w19-4445}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This study proposed a system which can benefit a high-stakes assessment of English speaking proficiency by automatically detecting potentially plagiarized spoken responses, and investigated the empirical effectiveness of two different types of features. One is based on automatic plagiarism S. M. Alzahrani, N. Salim, and A. Abraham. 2012. Understanding plagiarism linguistic patterns, textual features, and detection methods. IEEE Transaction on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 42(2):133–149. find
  2. S. Brin, J. Davis, and H. Garcia-Molina. 1995. find
  3. Copy Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. detection mechanisms for digital documents. In Proceedings of the ACM SIGMOD Annual Conference, pages 398–409. ↗
  4. C. Chen, J. Yeh, and H. Ke. 2010. Plagiarism detection using ROUGE and WordNet. Journal of Computing, 2(3):34–44. ↗
  5. M. Potthast, M. Hagen, T. Gollub, M. Tippmann, J. Kiesel, P. Rosso, E. Stamatatos, and B. Stein. 2013. Overview of the 5th International Competition on Plagiarism Detection. In CLEF 2013 Evaluation Labs and Workshop – Working Notes Papers. find
  6. P. Cullen, A. French, and V. Jakeman. 2014. The Official Cambridge Guide to IELTS. Cambridge University Press. find
  7. ETS. 2012. The Official Guide to the TOEFL R Test, Fourth Edition. McGraw-Hill. find
  8. M. Potthast, B. Stein, A. Barrón-Cedeño, and P. Rosso. 2010. An evaluation framework for plagiarism detection. In Proceedings of the 23rd International Conference on Computational Linguistics. find
  9. Keelan Evanini and Xinhao Wang. 2014. Automatic detection of plagiarized spoken responses. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22–27. find
  10. Y. Rubner, C. Tomasi, and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In Proceedings of the Sixth International Conference on Computer Vision. find
  11. A. Finch, Y. Hwang, and E. Sumita. 2005. Using machine translation evaluation techniques to determine sentence-level semantic equivalence. In Proceedings of the Third International Workshop on Paraphrasing, pages 17–24. find
  12. N. Shivakumar and H. Garcia-Molina. 1995. SCAM: A copy detection mechanism for digital documents. In Proceedings of the Second Annual Conference on the Theory and Practice of Digital Libraries. find
  13. A. Hauptmann. 2006. Automatic spoken document retrieval. In Ketih Brown, editor, Encylclopedia of Language and Linguistics (Second Edition), pages 95–103. Elsevier Science. find
  14. E. Stamatatos. 2011. Plagiarism detection using stopword n-grams. American Society for Information Science and Technology, 62(12):2512–2527. ↗
  15. T. C. Hoad and J. Zobel. 2003. Methods for identifying versioned and plagiarised documents. Journal of the American Society for Information Science and Technology, 54:203–215. find
  16. B. Stein, N. Lipka, and P. Prettenhofer. 2011. Intrinsic plagiarism analysis. Language Resources and Evaluation, 45(1):63–82. ↗
  17. Ö. Uzuner, B. Katz, and T. Nahnsen. 2005. Using syntactic information to identify plagiarism. In Proceedings of the 2nd Workshop on Building Educational Applications using NLP. Ann Arbor. find
  18. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957–966, Lille, France. find
  19. X. Wang, K. Evanini, J. Bruno, and M. Mulholland. 2016. Automatic plagiarism detection for spoken responses in an assessment of english language proficiency. In Proceedings of the IEEE Spoken Language Technology Workshop, pages 121–128. find
  20. P. Longman. 2010. The Official Guide to Pearson Test of English Academic. Pearson Education ESL. find
  21. C. Lyon, R. Barrett, and J. Malcolm. 2006. Plagiarism is easy, but also easy to detect. Plagiary, 1:57–65. find
  22. K. Zechner, D. Higgins, and X. Xi. 2007. Speechrater SM : A construct-driven approach to scoring spontaneous non-native speech. In Proceedings of the International Speech Communication Association Special Interest Group on Speech and Language Technology in Education, pages 128–131. find
  23. N. Madnani, J. Tetreault, and M. Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–190, Montréal, Canada. Association for Computational Linguistics.
  24. K. Zechner, D. Higgins, X. Xi, and D. M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10):883–895. ↗
  25. M. Mozgovoy, T. Kakkonen, and E. Sutinen. 2007. Using natural language parsers in plagiarism detection. In Proceedings of the ISCA Workshop on Speech and Language Technology in Education (SLaTE). find
  26. K. Zechner and X. Wang. 2013. Automated content scoring of spoken responses in an assessment for teachers of english. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, page 73–81.
  27. T. Nahnsen, Ö. Uzuner, and B. Katz. 2005. Lexical chains and sliding locality windows in content-based text similarity detection. CSAIL Technical Report, MIT-CSAIL-TR-2005-034. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2019-application-automatic-plagiarism-detection-system/