Elamur Research⌕
☰
home / papers / 2020 / A Process-oriented Dataset of Revisions during Writing

A Process-oriented Dataset of Revisions during Writing

Paper · 2020 · Rianne Conijn, Emily Dux Speltz, Menno van Zaanen et al.

Summary

Research on how people revise text has mostly compared finished drafts. Tools for studying revision as it happens are scarce, and keystroke logs are usually reduced to crude counts such as backspace totals. The authors address this gap with an openly available, richly annotated dataset of revisions made inside a single draft. They took writing sessions recorded in CyWrite, a browser-based editor that logs timestamped keystrokes and eye fixations. The 65 sessions cover English L1 graduate students summarising an academic article, L1 undergraduates writing TOEFL-style argumentative essays, and L2 writers responding to similar prompts. Revision events were segmented automatically from the logs. Each event was then described by 49 features: some coded by trained annotators, others derived by rules. The features span eight of the ten properties in the authors' revision tagset. The resulting resource holds 7,120 revisions, and the paper illustrates it with an example experiment and a visualisation of revisions. Its value is that fine-grained revision behaviour across writer groups and tasks can now be studied on shared data.

Key points

  • The data come from CyWrite sessions of 65 writers: 20 L1 graduate students (abstracts), 20 L1 undergraduates (argumentative essays) and 25 L2 writers (similar prompts). It is available at hdl.handle.net/10411/VBDYGX.
  • There are 7,120 revision events, about 110 per writer on average (SD 53).
  • Each event has 18 manually annotated and 31 automatically extracted features. Together they describe a revision's orientation and how it is evaluated, the kind of action performed, which linguistic domain it affects, where in the text and when in the session it takes place, how long it lasts, and how it is sequenced. Two tagset properties, processing and trigger, were left out because the logs cannot capture them.
  • Five trained annotators worked from a detailed guide and a visual replay that included the eye-fixation marker. Fifteen documents (23%) were double-coded by different annotator pairs, and agreement was measured with Krippendorff's alpha.
  • A revision event begins at a deletion, or when the cursor is moved elsewhere and typing starts. Its end, where fluent writing resumes, was judged by humans. Fluent production was defined relative to the leading edge (Lindgren et al., 2019), ignoring trailing spaces and leftover planning notes.

Interesting ideas

  • I1. The pipeline proposes event boundaries automatically with simple rules and leaves to humans only the hard call of where an event ends, which keeps manual effort focused.
  • I2. The dataset separates rule-derived features from human-judged ones and reports inter-rater reliability only for the judged ones, so users know which labels carry interpretive uncertainty.
  • I3. Orientation is coded hierarchically: first surface versus meaning-changing, then a subtype. Because the subtypes are mutually exclusive, agreement is measured once per category rather than per label.
  • I4. Edits are classified by whether they only touch form (spelling, grammar, wording) or actually alter content, which gives a principled way to compare two versions of a text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data building could let rules propose spans cheaply and send only ambiguous boundary decisions to LLM teacher tiers or human review.
  • I2 (learn): Every silver label could record its provenance (rule, witness parser, LLM, human), and agreement statistics would then be computed only where judgement was involved.
  • I3 (frames): Speech-act or implicature labels could get the same coarse-then-fine scheme, with agreement measured at the category level so that rare fine labels do not distort reliability.
  • I4 (back): Diffs between the input and the text → graph → text output could be sorted into surface-only versus meaning-changing, with the second kind flagged as compressor errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tilburg UniversityTilburg · Netherlands50 works · 6,429 citations🎓University of AntwerpAntwerp · Belgium17 works · 3,490 citations🎓Iowa State UniversityAmes · United States8 works · 400 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 worksComputer Science Applications · 102 works

Cite

Cite as Rianne Conijn, Emily Dux Speltz, Menno van Zaanen, Luuk Van Waes and Evgeny Chukharev-Hudilainen. 2020. A Process-oriented Dataset of Revisions during Writing. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{conijn2020process,
  title = {{A Process-oriented Dataset of Revisions during Writing}},
  author = {Rianne Conijn, Emily Dux Speltz, Menno {van Zaanen}, Luuk {Van Waes} and Evgeny Chukharev-Hudilainen},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  doi = {10.34894/ny0dss},
  url = {https://doi.org/10.34894/ny0dss}
}

References

34 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barkaoui, K. (2016). What and when second-language learners revise when responding to timed writing tasks on the computer: The roles of task type, second language proficiency, and keyboarding skills. The Modern Language Journal, 100(1):320–340. ↗
  2. Boytsov, L. (2011). Indexing methods for approximate dictionary searching: Comparative analysis. Journal of Experimental Algorithmics (JEA), 16:1–91. find
  3. Chukharev-Hudilainen, E., Saricaoglu, A., Torrance, M., and Feng, H.-H. (2019). Combined deployable keystroke logging and eyetracking for investigating L2 writing fluency. Studies in Second Language Acquisition, 41(3):583–604. ↗
  4. Chukharev-Hudilainen, E. (2019). Empowering automated writing evaluation with keystroke logging. In E. Lindgren et al., editors, Observing Writing, pages 125–142. Brill. ↗
  5. Conijn, R., Dux Speltz, E., Zaanen, M. v., Van Waes, L., and Chukharev-Hudilainen, E. (2020). A product and process oriented tagset for revisions in writing. PsyArXiv. psyarxiv.com/h25ak. ↗
  6. Daxenberger, J. and Gurevych, I. (2012). A corpus-based study of edit categories in featured and non-featured Wikipedia articles. Proceedings of COLING 2012, pages 711–726. find
  7. Faigley, L. and Witte, S. (1981). Analyzing revision. College composition and communication, 32(4):400–414. find
  8. Fitzgerald, J. (1987). Research on revision in writing. Review of educational research, 57(4):481–506. ↗
  9. Flower, L. and Hayes, J. R. (1980). The cognition of discovery: Defining a rhetorical problem. College composition and communication, 31(1):21–32. ↗
  10. Hayes, A. F. and Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77–89. ↗
  11. Kollberg, P. (1996). S-notation as a tool for analysing the episodic structure of revisions. In European writing conferences. find
  12. Krippendorff, K. (2011). Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC), 43:1–10. find
  13. Leijten, M. and Van Waes, L. (2013). Keystroke logging in writing research: Using inputlog to analyze and visualize writing processes. Written Communication, 30(3):358– 392. find
  14. Levy, C. M. and Ransdell, S. (1996). Writing signatures. In The science of writing: Theories, methods, individual differences and applications. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, US. find
  15. Lindgren, E. and Sullivan, K. P., (2006a). Analysing online revision, pages 157–188. Elsevier. find
  16. Lindgren, E. and Sullivan, K. P., (2006b). Writing and the analysis of revision: An overview, pages 31–40. Elsevier. find
  17. Lindgren, E., Westum, A., Outakoski, H., and Sullivan, K. P. H. (2019). Revising at the leading edge: Shaping ideas or clearing up noise. In Observing Writing, pages 346–365. Brill, Leiden, The Netherlands. ↗
  18. Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., and Clayphan, A. (2015). LATUX: An iterative workflow for designing, validating and deploying learning analytics visualisations. Journal of Learning Analytics, 2(3):9–39. ↗
  19. Min, H.-T. (2006). The effects of trained peer review on EFL students’ revision types and writing quality. Journal of second language writing, 15(2):118–141. ↗
  20. Monahan, B. D. (1984). Revision strategies of basic and competent writers as they write for different audiences. Research in the Teaching of English, pages 288–304. ↗
  21. Ranalli, J. R., Feng, H.-H., and Chukharev-Hudilainen, E. (2018). The affordances of process-tracing technologies for supporting L2 writing instruction. Language Learning & Technology, 23(2):1–11. ↗
  22. Scardamalia, M. and Bereiter, C. (1983). The development of evaluative, diagnostic and remedial capabilities in children’s composing. In M. Martlew, editor, The psychology of written language. Developmental and educational perspectives, pages 67–95. New York: McMillan. find
  23. Sommers, N. (1980). Revision strategies of student writers and experienced adult writers. College composition and communication, 31(4):378–388. ↗
  24. Stevenson, M., Schoonen, R., and de Glopper, K. (2006). Revising in two languages: A multi-dimensional comparison of online writing revisions in L1 and FL. Journal of Second Language Writing, 15(3):201–233. ↗
  25. Thorson, H. (2000). Using the computer to compare foreign and native language writing processes: A statistical and case study approach. The Modern Language Journal, 84(2):155–170. ↗
  26. Van Horenbeeck, E., Pauwaert, T., Van Waes, L., and Leijten, M. (2015). A generic XML-structure for logging human computer interaction [white paper]. Retrieved from http://www.inputlog.net/wp-content/uploads/Generic XML structure version-1 3.pdf. ↗
  27. Van Waes, L. and Schellens, P. J. (2003). Writing profiles: The effect of the writing mode on pausing and revision patterns of experienced writers. Journal of pragmatics, 35(6):829–853. ↗
  28. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., and Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10):1500–1509. ↗
  29. Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Assche, F., Parra, G., and Klerkx, J. (2014). Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing, 18(6):1499–1514. ↗
  30. Wobbrock, J. O. and Myers, B. A. (2006). Analyzing the input stream for character-level errors in unconstrained text entry evaluations. ACM Transactions on Computer-Human Interaction (TOCHI), 13(4):458–489. ↗
  31. Xu, C. (2018). Understanding online revisions in L2 writing: A computer keystroke-log perspective. System, 78:104–114. ↗
  32. Zhang, F. and Litman, D. (2015). Annotation and classification of argumentative writing revisions. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 133–143. find
  33. Zhang, M., Hao, J., Li, C., and Deane, P. (2016). Classification of writing patterns using keystroke logs. In L. Andries van der Ark, et al., editors, Quantitative Psychology Research: The 80th Annual Meeting of the Psychometric Society, Beijing, 2015, pages 299–314. Springer International Publishing. ↗
  34. Zhu, M., Zhang, M., and Deane, P. (2019). Analysis of keystroke sequences in writing logs. ETS Research Report Series. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/conijn-2020-process-oriented-dataset-revisions-during/