Elamur Research⌕
☰
home / papers / 2024 / Logging Keystrokes in Writing by English Learners

Logging Keystrokes in Writing by English Learners

Paper · 2024 · Georgios Velentzas, Andrew Caines, Rita Borgo et al.

Summary

Most NLP work on learner essays, such as automated scoring or grammatical error correction, looks only at the finished text. This paper captures the whole writing process instead. The authors built a browser-based authoring page with a JavaScript keylogger and recruited 1,006 crowdworkers on Prolific, both native and non-native English speakers. Each person did two tasks: they retyped a fixed passage, then wrote an original essay. Every key press, key release, mouse click, content change and periodic text snapshot was stored with a millisecond timestamp. An automarker and three human assessors graded the essays on a 0–12 scale and mapped them to CEFR levels. The result is KUPA-KEYS, a public non-commercial dataset that pairs keystroke logs with proficiency labels and demographic metadata. The authors suggest several uses:

  • finding typing patterns that predict proficiency for automated assessment;
  • detecting moments when a writer struggles, so a tool can offer help;
  • spotting pasted or AI-generated text;
  • cognitive and linguistic study of how sentences take shape.

Key points

  • Participants: 1,006 people. About a fifth are native speakers, and 42 first languages are represented, most often Polish, Portuguese and Spanish. The median age is 26, with a range of 18–75.
  • Copy task: a passage of 300 words taken from a speech by Steve Jobs, chosen because it contains 197 distinct English digraphs. The same passage was used in the Buffalo dataset (Sun et al., 2016).
  • Essay task: each writer was randomly given one of ten prompts. The minimum length was 250 words for cohort 1 and 150 words for cohorts 2–3.
  • Grading: each essay has a mark from the W&I automarker and from three human markers, all on a 0–12 scale, each with a CEFR level. Unlike the Kaggle "Linking Writing Processes to Writing Quality" data, the characters are not anonymised.
  • Event log: events are recorded in JSON with types down/up/click/input/capture, plus modifier flags, cursor ranges and text. Timing accuracy is about ±5 ms. The data is released on Hugging Face as ALTACambridge/KUPA-KEYS under CC BY-NC-SA 4.0.

Interesting ideas

  • I1. Each writer also completes a fixed transcription task, which gives a personal baseline for typing speed so that essay-writing behaviour can be normalised per writer.
  • I2. An Anderson-Darling test compares each writer's key-latency distribution between the copy task and the essay; a high p-value hints that the "essay" was actually transcribed from another source.
  • I3. The log keeps raw user actions, the resulting content edits and periodic full-text snapshots side by side, so the document state can be rebuilt and checked from several redundant views.
  • I4. The copy passage was selected to maximise coverage of distinct digraphs, a deliberate coverage-driven choice of test material.
  • I5. Each essay gets one automatic grade and three independent human grades, which keeps disagreement visible instead of collapsing to a single label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run every witness parser on a fixed reference text as well, so that per-witness quirks can be calibrated out before their votes on silver data are combined.
  • I2 (learn): Compare score distributions of a model on genuine items against a known-easy control set; if they look the same, the model may be echoing rather than reasoning, which makes a useful negative control.
  • I3 (world): Store the world of a text as an ordered event log plus periodic state snapshots, which makes event ordering and state changes both replayable and self-checking.
  • I4 (tokens): Pick tokenizer and morphology regression texts that maximise coverage of distinct affixes, clitics and character patterns rather than using random sentences.
  • I5 (learn): Keep all LLM-teacher and parser-witness verdicts per item, and use their spread as an uncertainty signal when deciding which silver examples to trust.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 worksEducation · 212 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Georgios Velentzas, Andrew Caines, Rita Borgo, Erin Pacquetet, Clive Hamilton, Taylor Arnold, Diane Nicholls, Paula Buttery, Thomas Gaillat, Nicolas Ballier and Helen Yannakoudakis. 2024. Logging Keystrokes in Writing by English Learners. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{velentzas2024logging,
  title = {{Logging Keystrokes in Writing by English Learners}},
  author = {Georgios Velentzas, Andrew Caines, Rita Borgo, Erin Pacquetet, Clive Hamilton, Taylor Arnold, Diane Nicholls, Paula Buttery, Thomas Gaillat, Nicolas Ballier and Helen Yannakoudakis},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/46u76s9rbrvw},
  url = {https://doi.org/10.63317/46u76s9rbrvw}
}

References

62 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alejandro Acien, Aythami Morales, Ruben Vera-Rodriguez, Julian Fierrez, and John V. Monaco. 2020. TypeNet: Scaling up keystroke biometrics. In 2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1–7. find
  2. Svenja Adolphs. 2005. “I don’t think I should learn all this” – a longitudinal view of attitudes towards ‘native speaker’ English. In Claus Gnutzmann and Frauke Intemann, editors, The globalisation of English and the English language classroom, pages 119–131. Tübingen: Narr Verlag. find
  3. Jeffrey D Allen. 2010. An analysis of pressure-based keystroke dynamics algorithms. Ph.D. thesis, Southern Methodist University. find
  4. Cem Alptekin. 2002. Towards intercultural communicative competence in ELT. ELT Journal, 56:57–64. ↗
  5. Veerle M Baaijen and David Galbraith. 2018. Discovery through writing: Relationships with writing processes and text quality. Cognition and Instruction, 36(3):199–223. ↗
  6. Yukino Baba and Hisami Suzuki. 2012. How are spelling errors generated and corrected? a study of corrected and uncorrected spelling errors using keystroke logs. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 373–377, Jeju Island, Korea. Association for Computational Linguistics. find
  7. Nicolas Ballier, Erin Pacquetet, and Taylor Arnold. 2019. Investigating Keylogs as Time-Stamped Graphemics. In Proceedings of Graphemics in the 21st Century, Brest 2018, pages 353–365. find
  8. Ritwik Banerjee, Song Feng, Jun Seok Kang, and Yejin Choi. 2014. Keystroke patterns as prosody in digital writings: A case study with deceptive reviews and essays. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1469–1473. find
  9. Evgeny Chukharev-Hudilainen. 2019. Empowering automated writing evaluation with keystroke logging. In Observing writing, pages 125–142. Brill. ↗
  10. Evgeny Chukharev-Hudilainen, Aysel Saricaoglu, Mark Torrance, and Hui-Hsien Feng. 2019. Combined deployable keystroke logging and eyetracking for investigating L2 writing fluency. Studies in Second Language Acquisition, 41(3):583–604. ↗
  11. Rianne Conijn, Emily Dux Speltz, and Evgeny Chukharev-Hudilainen. 2021. Automated extraction of revision events from keystroke data. Reading and Writing, pages 1–26. ↗
  12. Vivian Cook. 1999. Going beyond the native speaker in language teaching. TESOL Quarterly, 33:185–209. ↗
  13. Scott A. Crossley, Jennifer L. Weston, Susan T. McLain Sullivan, and Danielle S. McNamara. 2011. The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3):282–311. ↗
  14. Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018. Observations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1– 12. find
  15. Anna Maria Feit, Daryl Weir, and Antti Oulasvirta. 2016. How we type: Movement strategies and performance in everyday typing. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 4262–4273. find
  16. Julian Fierrez, Javier Galbally, Javier Ortega-Garcia, Manuel R Freire, Fernando Alonso-Fernandez, Daniel Ramos, Doroteo Torre Toledano, Joaquin Gonzalez-Rodriguez, Juan A Siguenza, Javier Garrido-Salas, et al. 2010. Biosecurid: a multimodal biometric database. Pattern Analysis and Applications, 13(2):235– 246. ↗
  17. David Galbraith and Veerle Baaijen. 2019. Aligning keystrokes with cognitive processes in writing. In E. Lindgren and K. Sullivan, editors, Observing writing: Insights from keystroke logging and handwriting, pages 306–325. Leiden: Brill. ↗
  18. Romain Giot, Mohamad El-Abed, and Christophe Rosenberger. 2009. GREYC keystroke: a benchmark for keystroke dynamics biometric systems. In Biometrics: Theory, Applications, and Systems, 2009. BTAS’09. IEEE 3rd International Conference on Biometrics: Theory, Applications and Systems, pages 1–6. find
  19. Romain Giot, Mohamad El-Abed, and Christophe Rosenberger. 2012. Web-based benchmark for keystroke dynamics biometric systems: A statistical analysis. In 2012 Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pages 11–15. IEEE. ↗
  20. Nahuel González, Enrique P. Calot, Jorge S. Ier-ache, and Waldo Hasperué. 2021. On the shape of timings distributions in free-text keystroke dynamics profiles. Heliyon, 7(11):e08413. ↗
  21. Adam Goodkind and Andrew Rosenberg. 2015. Muddying the multiword expression waters: How cognitive demand affects multiword expression production. In Proceedings of the 11th Workshop on Multiword Expressions, pages 87– 95, Denver, Colorado. Association for Computational Linguistics. find
  22. Daniele Gunetti and Claudia Picardi. 2005. Keystroke analysis of free text. ACM Transactions on Information and System Security (TISSEC), 8(3):312–347. ↗
  23. Kilem Gwet. 2002. Inter-rater reliability: dependency on trait prevalence and marginal homogeneity. Statistical Methods for Inter-Rater Reliability Assessment Series, 2:1–9. find
  24. Syed Zulkarnain Syed Idrus, Estelle Cherrier, Christophe Rosenberger, and Patrick Bours. 2013. Soft biometrics database: A benchmark for keystroke dynamics biometric systems. In 2013 International Conference of the BIOSIG Special Interest Group (BIOSIG), pages 1–8. IEEE. find
  25. Pilsung Kang and Sungzoon Cho. 2015. Keystroke dynamics-based user authentication using long and free text strings from various input devices. Information Sciences, 308:72–93. ↗
  26. M. Karnan, M. Akila, and N. Krishnaraj. 2011. Biometric personal authentication using keystroke dynamics: A review. Applied Soft Computing, 11(2):1565–1573. The Impact of Soft Computing for the Progress of Artificial Intelligence. ↗
  27. Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In IJCAI, volume 19, pages 6300–6308. find
  28. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2024. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36.
  29. John Leggett, Glen Williams, Mark Usnick, and Mike Longnecker. 1991. Dynamic identity verification via keystroke characteristics. International Journal of Man-Machine Studies, 35(6):859–870. ↗
  30. Mariëlle Leijten, Eric Van Horenbeeck, and Luuk Van Waes. 2019. Analysing keystroke logging data from a linguistic perspective. In E. Lindgren and K. Sullivan, editors, Observing writing: Insights from keystroke logging and handwriting, page 71–95. Leiden: Brill. ↗
  31. Hilbert Locklear, Sathya Govindarajan, Zdeňka Sitová, Adam Goodkind, David Guy Brizan, Andrew Rosenberg, Vir V. Phoha, Paolo Gasti, and Kiran S. Balagani. 2014. Continuous authentication with cognition-centric text production and revision features. In IEEE International Joint Conference on Biometrics. find
  32. Cerstin Mahlow. 2015. Learning from Errors: Systematic Analysis of Complex Writing Errors for Improving Writing Technology. In Language Production, Cognition, and the Lexicon, pages 419– 438. ↗
  33. Cerstin Mahlow, Malgorzata Anna Ulasik, and Don Tuggener. 2022. Extraction of transforming sequences and sentence histories from writing process data: a first step towards linguistic modeling of writing. Reading and Writing, pages 1–40. ↗
  34. Emanuele Maiorana, Himanka Kalita, and Patrizio Campisi. 2021. Mobile keystroke dynamics for biometric recognition: An overview. IET Biometrics, 10(1):1–23. ↗
  35. Danielle S. McNamara, Scott A. Crossley, Rod D. Roscoe, Laura K. Allen, and Jianmin Dai. 2015. A hierarchical classification approach to automated essay scoring. Assessing Writing, 23:35– 59. ↗
  36. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. ↗
  37. John V Monaco, Ned Bakelman, Sung-Hyuk Cha, and Charles C Tappert. 2012. Developing a keystroke biometric system for continual authentication of computer users. In 2012 European Intelligence and Security Informatics Conference, pages 210–216. find
  38. John V Monaco, Gonzalo Perez, Charles C Tappert, Patrick Bours, Soumik Mondal, Su-dalai Rajkumar, Aythami Morales, Julian Fierrez, and Javier Ortega-Garcia. 2015. One-handed keystroke biometric identification competition. In 2015 International Conference on Biometrics (ICB), pages 58–64. IEEE. find
  39. John V Monaco, John C Stewart, Sung-Hyuk Cha, and Charles C Tappert. 2013. Behavioral biometric verification of student identity in online course assessment and authentication of authors in literary works. In 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pages 1–8. find
  40. John V. Monaco and Charles C. Tappert. 2016. Obfuscating keystroke time intervals to avoid identification and impersonation. abs/1609.07612. ↗
  41. CoRR, Jugurta R Montalvão Filho and Eduardo O Freire. 2006. On the equalization of keystroke timing histograms. Pattern Recognition Letters, 27(13):1440–1446. ↗
  42. Mozilla Foundation. 2023. Mozilla Web-Docs: High precision timing. https://developer.mozilla.org/en-US/ docs/Web/API/Performance_API/High_ precision_timing. Accessed: 2023-10-17. ↗
  43. Christopher Murphy, Jiaju Huang, Daqing Hou, and Stephanie Schuckers. 2017. Shared dataset on natural human-computer interaction to support continuous authentication research. In 2017 IEEE International Joint Conference on Biometrics (IJCB), pages 525–530. IEEE. find
  44. Erin Pacquetet. 2024. The effect of linguistic properties on typing behaviors and production processes. Ph.D. thesis, University at Buffalo. find
  45. Alen Peacock, Xian Ke, and Matthew Wilkerson. 2004. Typing patterns: A key to user identification. IEEE Security and Privacy, 2(5):40–47. ↗
  46. Robert Phillipson. 1992. Linguistic Imperialism. Oxford University Press, Oxford, UK. find
  47. Barbara Plank. 2016. Keystroke dynamics as signal for shallow syntactic parsing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 609–619. ↗
  48. Barbara Plank. 2018. Predicting authorship and author traits from keystroke dynamics. In Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, pages 98– 104, New Orleans, Louisiana, USA. Association for Computational Linguistics. find
  49. Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3):2495–2527. ↗
  50. Giorgio Roffo, Cinzia Giorgetta, Roberta Ferrario, Walter Riviera, and Marco Cristani. 2014. Statistical analysis of personality and identity in chats using a keylogging platform. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 224–231. find
  51. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-generated text be reliably detected? arXiv:2303.11156. ↗
  52. Moritz Jonas Schaeffer, Michael Carl, Isabel Lacruz, and Akiko Aizawa. 2016. Measuring cognitive translation effort with activity units. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 331–345. find
  53. John C Stewart, John V Monaco, Sung-Hyuk Cha, and Charles C Tappert. 2011. An investigation of keystroke and stylometry traits for authenticating online test takers. In 2011 International Joint Conference on Biometrics (IJCB), pages 1–7. find
  54. Giuseppe Stragapede, Paula Delgado-Santos, Ruben Tolosana, Ruben Vera-Rodriguez, Richard Guest, and Aythami Morales. 2023. Mobile keystroke biometrics using transformers. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–6. find
  55. Yan Sun, Hayreddin Ceker, and Shambhu Upadhyaya. 2016. Shared keystroke dataset for continuous authentication. In 2016 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE. find
  56. Charles C Tappert, Mary Villani, and Sung-Hyuk Cha. 2010. Keystroke biometric identification and authentication on long-text input. In Behavioral biometrics for human identification: Intelligent applications, pages 342–367. ↗
  57. Pin Shen Teh, Andrew Beng Jin Teoh, Connie Tee, and Thian Song Ong. 2011. A multiple layer fusion approach on keystroke dynamics. Pattern Analysis and Applications, 14(1):23–36. find
  58. Ivor Timmis. 2002. Native-speaker norms and International English: a classroom view. ELT Journal, 56:240–249. ↗
  59. Vishaal Udandarao, Mohit Agrawal, Rajesh Kumar, and Rajiv Ratn Shah. 2020. On the inference of soft biometrics from typing patterns collected in a multi-device environment. In 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), pages 76–85. find
  60. Esra Vural, Jiaju Huang, Daqing Hou, and Stephanie Schuckers. 2014. Shared research dataset to support development of keystroke authentication. In IEEE International joint conference on biometrics, pages 1–8. IEEE. find
  61. Helen Yannakoudakis and Ronan Cummins. 2015. Evaluating the performance of automated text scoring systems. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 213–223. find
  62. Mo Zhang, Jiangang Hao, Chen Li, and Paul Deane. 2016. Classification of writing patterns using keystroke logs. In Quantitative psychology research, pages 299–314. Springer. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/velentzas-2024-logging-keystrokes-writing-english-learners/