Elamur Research⌕
☰
home / papers / 2025 / A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction

A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction

Paper · 2025 · Cameron Morin and Matti Marttinen Larsson

Summary

Corpus linguists can now search enormous text collections, but turning raw hits into analysable data still requires hand classification, and that limits sample sizes. Morin and Marttinen Larsson expand their earlier small, supervised proof of concept into an automated four-stage workflow:

  1. They write a detailed prompt containing the criteria, examples and tricky cases.
  2. They test that prompt on several independent random samples before launch.
  3. They run unattended batches through the OpenAI API with GPT-5.
  4. They check the output afterwards with stratified samples.

Applied to all hits for consider in COHA, the system first separated evaluative uses (judging something as having a property) from other senses. It then labelled each evaluative use by complement type: zero, to be, or as. Both tasks reached at least 98% accuracy. The whole job took under 60 hours and cost about $104. A Bayesian multinomial GAM fitted to the resulting data revealed genre-dependent paths of change that had not been reported before. The authors link these paths to formality and to opposing pulls toward shorter and fuller forms. The study shows that LLM annotation can open corpus questions that were previously impractical to study.

Key points

  • GPT-5 annotated 143,933 concordance lines in total, all taken from COHA (Corpus of Historical American English, 1820s–2010s, five genres). Manual work at 10 seconds per line would take at least 400 hours.
  • The first task (evaluative vs. non-evaluative) reduced 99,406 tokens to 44,527 evaluative ones. The second task assigned these to zero, to be, or as complements. Accuracy was 98%+ on both.
  • API fees for both tasks totalled about $104. The authors argue that this makes large-scale annotation affordable even for modestly funded studies.
  • The earlier pipeline (Claude 3.5 Sonnet, supervised step by step) had reached just over 90% accuracy on a held-out set of 102 lines.
  • The authors acknowledge several limits:
    • Results depend on prompt quality.
    • Subjective or impressionistic coding schemes fit the approach poorly.
    • Even a small error rate can bias downstream statistics on huge datasets.
    • Expert linguists are still needed to design the task and judge data quality.

Interesting ideas

  • I1. The prompt is validated on several separate random samples before the full run, so the prompt is effectively frozen and tested like a model before deployment, rather than corrected during labelling.
  • I2. Post-hoc quality checks are stratified by decade and genre, which exposes errors that concentrate in older or unusual registers instead of hiding them in one global accuracy figure.
  • I3. A cheap filtering pass first isolates the relevant sense of a polysemous verb, and a second pass then does the fine-grained labelling only on those survivors.
  • I4. The complement contrast (zero vs. to be vs. as) is a well-defined syntactic variable that an LLM labels near-perfectly, making it a clean probe for small-clause analysis.
  • I5. The authors found that embedding instructions inside a structured prompt (XML tags, explicit reasoning steps) worked better than handing the model an external guideline document, consistent with RAG losing to Chain of Thought in related work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Freeze each LLM teacher-tier prompt and score it on multiple fresh samples before it produces silver data, logging per-sample variance as a stability signal.
  • I2 (learn): Report silver-data and parser accuracy per genre/period stratum (fables, school tests, older prose), not just overall, so weak slices trigger targeted repair rules.
  • I3 (frames): Separate the evaluative "consider X (as/to be) Y" frame from the deliberative "consider options" sense before assigning VerbNet roles, so attribute-ascription frames are not confused with mental-activity frames.
  • I4 (tree): Use LLM-labelled consider/regard/deem lines as a targeted test set for how the parsers attach small-clause predicates (xcomp vs. obl with "as"), checking dialect-conversion consistency.
  • I5 (learn): Inline the UD annotation guideline excerpts and edge cases directly into teacher prompts in tagged sections instead of attaching retrieved docs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GothenburgGothenburg · Sweden34 works · 1,625 citations🎓Université Paris CitéParis · France19 works · 1,426 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksLiterature and Literary Theory · 519 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Cameron Morin and Matti Marttinen Larsson. 2025. A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction. arXiv:2510.12306.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.12306 · arXiv 2510.12306

@misc{morin2025large,
  title = {{A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction}},
  author = {Cameron Morin and Matti Marttinen Larsson},
  year = {2025},
  doi = {10.48550/arxiv.2510.12306},
  eprint = {2510.12306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.12306}
}

References

45 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. (2025, September 29). Introducing Claude Sonnet 4.5. Anthropic.com. https://www.anthropic.com/news/claude-sonnet-4-5 Atıl, B., Aykent, S., Chittams, A., Fu, L., Passonneau, R. J., Radcliffe, E., Rajagopal, G. R., Sloan, A., Tudrej, T., Ture, F., Wu, Z., Xu, L., & Baldwin, B. (2025). Non-determinism of “deterministic” LLM system settings in hosted environments. In M. Akter, T. Chowdhury, S. Eger, C. Leiter, J. Opitz, & E. Çano (Eds.), Proceedings of the 5th workshop on evaluation and comparison of NLP systems (pp. 135–148). Association for Computational Linguistics. ↗
  2. Belal, M., She, J., & Wong, S. (2023). Leveraging ChatGPT as text annotation tool for sentiment analysis. arXiv:2306.17177v1. ↗
  3. Biber, D. (1988). Variation across speech and writing. Cambridge University Press. ↗
  4. Bresnan, J., Cueni, A., Nikitina, T., & Baayen, R. H. (2007). Predicting the dative alternation. In G. Bouma, I. Kraemer, & J. Zwarts (Eds.), Cognitive foundations of interpretation (pp. 69–94). KNAW. find
  5. Bürkner, P. C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80(1), 1–28. ↗
  6. Bybee, J. (2002). Word frequency and context of use in the lexical diffusion of phonetically conditioned sound change. Language Variation and Change, 14(3), 261–290. ↗
  7. Bybee, J. (2006). From usage to grammar: The mind’s response to repetition. Language, 82(4), 711–733. ↗
  8. Bybee, J. (2010). Language, usage and cognition. Cambridge University Press. ↗
  9. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5), 423–429. ↗
  10. Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1). 1–32. ↗
  11. Coates, J. (1995). The expression of root and epistemic possibility in English. In J. L. Bybee & S. Fleischman (Eds.), Modality in grammar and discourse (pp. 55–66). John Benjamins. ↗
  12. Cuskley, C., Woods, R., & Flaherty, M. (2024). The limitations of large language models for understanding human language and cognition. Open Mind: Discoveries in Cognitive Science, 8, 1058–1083. ↗
  13. Davies, M. (2002). The Corpus of Historical American English (COHA). English-corpora.org. https://www.english-corpora.org/coha/ ↗
  14. Davies, M. (2016). The News on the Web (NOW) corpus. English-corpora.org. https://www.english-corpora.org/now/ Fonteyn, L., Manjavacas, E., & De Regt, J. (2025). Using machine learning to automate data annotation in corpus linguistics: A case study with MacBERTh. International Journal of Corpus Linguistics, 30(3), 296–315. ↗
  15. Fuoli, M., Huang, W., Littlemore, J., Turner, S., & Wilding, E. (2026). Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning. Applied Corpus Linguistics, 6(2), 100204. ↗
  16. Geeraerts, D., Speelman, D., Heylen, K., Montes, M., De Pascale, S., Franco, K., & Lang, M. (2023). Lexical variation and change: A distributional semantic approach. Oxford University Press. find
  17. Goldberg, A. E. (1995). Constructions: A construction grammar approach to argument structure. University of Chicago Press. find
  18. Gries, S. T., & Berez, A. L. (2017). Linguistic annotation in/for corpus linguistics. In N. Ide & J. Pustejovsky (Eds.), Handbook of linguistic annotation (pp. 379–409). Springer. ↗
  19. Grieve, J., Nini, A., & Guo, D. (2018). Mapping lexical innovation on American social media. Journal of English Linguistics, 46(4), 293–319. ↗
  20. Hawkins, J. A. (2014). Cross-linguistic variation and efficiency. Oxford University Press. Heylighen, F., & Dewaele, J. -M. (1999). Formality of language: Definition, measurement and behavioral determinants (Internal Report). Center “Leo Apostel”, Free University of Brussels. http://pespmc1.vub.ac.be/Papers/Formality.pdf ↗
  21. Hoffmann, T., & Trousdale, G. (2011). Variation, change and constructions in English. Cognitive Linguistics, 22(1), 1–23. ↗
  22. Hovy, E., & Lavid, J. (2010). Towards a ‘science’ of corpus annotation: A new methodological challenge for corpus linguistics. International Journal of Translation, 22(1), 13–36. find
  23. Jacques, G. (2023). Estimative constructions in cross-linguistic perspective. Linguistic Typology, 27(1), 157–194. ↗
  24. Jaeger, T. F. (2010). Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61(1), 23–62. ↗
  25. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv:2001.08361v1. ↗
  26. Kilgarriff, A., Baisa, V., Bušta, J., Jakubíček, M., Kovář, V., Michelfeit, J., Rychlý, P., & Suchomel, V. (2014). The Sketch Engine: Ten years on. Lexicography, 1(1), 7–36. find
  27. Leclercq, B., & Morin, C. (2023). No equivalence: A new principle of no synonymy. Constructions, 15(1). ↗
  28. Leclercq, B., & Morin, C. (2025). The meaning of constructions. Cambridge University Press. ↗
  29. Leivada, E., Dentella, V., & Günther, F. (2024). Evaluating the language abilities of humans vs large language models: Three caveats. Biolinguistics, 18, 1–16. find
  30. Levshina, N. (2022). Communicative efficiency: Language structure and use. Cambridge University Press. find
  31. Lorenz, D. (2013). Contractions of English semi-modals: The emancipating effect of frequency [Doctoral dissertation, Albert-Ludwigs-Universität Freiburg]. Freiburg University Library. Luccioni, A. S., Viguier, S., & Ligozat, A. -L. (2023). Estimating the carbon footprint of BLOOM, a 176B parameter language model. The Journal of Machine Learning Research, 24(1), 11990–12004. find
  32. Marttinen Larsson, M. (2023). Modelling incipient probabilistic grammar change in real time: The grammaticalisation of possessive pronouns in European Spanish locative adverbial constructions. Corpus Linguistics and Linguistic Theory, 19(2), 177–206. find
  33. Marttinen Larsson, M. (2024). Probabilistic reduction and constructionalization: A usage-based diachronic account of the diffusion and conventionalization of the Spanish la de <noun> que construction. Cognitive Linguistics, 35(4), 579–602. find
  34. Marttinen Larsson, M. (2025). Pathways of actualization across regional varieties and the real-time dynamics of syntactic change. Language Variation and Change, 37(1), 31–57. ↗
  35. Marttinen Larsson, M., Morin, C., Coussé, E., & Álvarez López, L. (forthcoming). USER-GRAM. Morin, C., & Marttinen Larsson, M. (2025). Large corpora and large language models: A replicable method for automating grammatical annotation. Linguistics Vanguard, 11(1), 501–510. ↗
  36. Morin, C., & Marttinen Larsson, M. (2026). Four-phase pipeline for LLM-assisted corpus annotation [Figure]. Zenodo. ↗
  37. Nevalainen, T., & Raumolin-Brunberg, H. (2016). Historical sociolinguistics: Language change in Tudor and Stuart England (2nd ed.). Routledge. find
  38. OpenAI. (2025, August 7). Introducing GPT-5. OpenAI.com. https://openai.com/index/introducing-gpt-5/ Ostyakova, L., Mikhailova, A., Molchanova, M., & Konovalov, V. (2025). Redefining annotation practices: Leveraging large language models for discourse annotation. In A. Panchenko, D. Gubanov, M. Khachay, A. Kutuzov, N. Loukachevitch, A. Kuznetsov, I. Nikishina, M. Panov, P. M. Pardalos, A. V. Savchenko, E. Tsymbalov, E. Tutubalina, A. Kasieva, & D. I. Ignatov (Eds.), Analysis of images, social networks and texts: 12th International Conference, AIST 2024, Bishkek, Kyrgyzstan, October 17–19, 2024, revised selected papers (pp. 131–147). Springer. ↗
  39. Ren, S., Tomlinson, B., Black, R. W., & Torrance, A. W. (2024). Reconciling the contrasting narratives on the environmental impact of large language models. Scientific Reports, 14, 26310. ↗
  40. Rickford, J. R., Wasow, T. A., Mendoza-Denton, N., & Espinoza, J. (1995). Syntactic variation and change in progress: Loss of the verbal Coda in topic-restricting as far as constructions. Language, 71(1), 102–131. ↗
  41. Rudnicka, K. (2021). In order that — A data-driven study of symptoms and causes of obsolescence. Linguistics Vanguard, 7(1), 20200092. find
  42. Smith, N., Hoffmann, S., & Rayson, P. (2008). Corpus tools and methods, today and tomorrow: Incorporating linguists’ manual annotations. Literary and Linguistic Computing, 23(2), 163–180. ↗
  43. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In A. Korhonen, D. Traum, & L. Màrquez (Eds.), Proceedings of the 57th annual meeting of the Association for Computational Linguistics (pp. 3645–3650). Association for Computational Linguistics.
  44. The Britannica Dictionary (n.d.) Ask the Editor: “Consider” and “consider as”. Britannica.com. https://www.britannica.com/dictionary/eb/qa/consider-and-consider-as Torrent, T. T., Hoffmann, T., Almeida, A. L., & Turner, M. (2024). Copilots for linguists: AI, constructions, and frames. Cambridge University Press. ↗
  45. Yu, D., Li, L., Su, H., & Fuoli, M. (2024). Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology. International Journal of Corpus Linguistics, 29(4), 534–561. Address for correspondence Matti Marttinen Larsson Department of Romance Studies and Classics Stockholm University Sweden matti.marttinen@su.se https://orcid.org/0000-0002-6224-7872 Co-author information Cameron Morin Approches Linguistiques Théoriques, Appliquées et Expérimentales: Langues et cultures connectées Paris Cité University cameron.morin@u-paris.fr Publication history Date received: 14 October 2025 Date accepted: 12 June 2026 Published online: 9 October 2026 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/morin-2025-large-scale-pipeline-llm-assisted/