Elamur Research⌕
☰
home / papers / 2024 / Automatic Annotation of Grammaticality in Child-Caregiver Conversations

Automatic Annotation of Grammaticality in Child-Caregiver Conversations

Summary

The authors developed a tool that automatically assesses the grammaticality of children's utterances in naturalistic conversations between children and adults. Until now, such research has relied on labor-intensive manual coding of small samples. As a result, conclusions about grammar acquisition, in particular about the benefit of parental corrections, have often contradicted one another. The team designed an annotation scheme that takes the dialogue context into account: elliptical answers, repetitions and self-corrections are assessed with reference to the preceding turns. Following this scheme, more than 4,000 utterances from the English portion of CHILDES were manually annotated. For ungrammatical utterances, the error type was additionally specified. On these data, an SVM on n-grams, an LSTM and fine-tuned Transformer models were compared. The Transformer models performed best: their results nearly reached the level of inter-annotator agreement among humans. The trained models were then run on a corpus roughly a hundred times the size of the annotated set. This confirmed the expected gradual increase in grammaticality with age. The work shows that modern NLP methods make it possible to study children's language development at scale and reproducibly.

Key points

  • Data: more than 4,000 child utterances (at least two words long) from English CHILDES carry one of three labels: grammatical, ambiguous or ungrammatical. Ungrammatical utterances are further divided into more than a dozen error categories, among them subject omission.

  • Models: several approaches were compared:

    • a baseline SVM on n-gram features (BPE, without context);
    • a single-layer LSTM with 512 hidden units, pre-trained as a language model on CHILDES;
    • fine-tuned BERT (bert-base-uncased), GPT2, RoBERTa (roberta-large) and DeBERTa (deberta-v3-large).

    To let the models distinguish between speakers, the special tokens [CHI] and [CAR] were prepended to each turn.

  • Result: the fine-tuned Transformer models outperformed the baselines and nearly matched human annotators in quality.

  • Application: the models automatically annotated all English CHILDES transcripts for children aged 2–5 years. The proportion of grammatical utterances increases steadily with age.

  • Resources: the annotations, models and code are openly released on GitHub (mitjanikolaus/childes-grammaticality). The paper is distributed under the CC BY-NC 4.0 license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

Cite as Mitja Nikolaus, Abhishek Agrawal, Petros Kaklamanis, Alex Warstadt and Abdellah Fourtassi. 2024. Automatic Annotation of Grammaticality in Child-Caregiver Conversations. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2403.14208

@inproceedings{nikolaus2024automatic,
  title = {{Automatic Annotation of Grammaticality in Child-Caregiver Conversations}},
  author = {Mitja Nikolaus, Abhishek Agrawal, Petros Kaklamanis, Alex Warstadt and Abdellah Fourtassi},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  eprint = {2403.14208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.14208}
}

References

60 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adger. 2003. Core syntax: A minimalist approach, volume 20. Oxford University Press. find
  2. Roger Brown. 1973. A First Language: The Early Stages. Harvard University Press. find
  3. Roger Brown and Camille Hanlon. 1970. Derivational complexity and order of acquisition in child speech. Cognition and the development of language. find
  4. Krista Byers-Heinlein, Christina Bergmann, Catherine Davies, Michael C. Frank, J. Kiley Hamlin, Melissa Kline, Jonathan F. Kominsky, Jessica E. Kosie, Casey Lew-Williams, Liquan Liu, Meghan Mastroberardino, Leher Singh, Connor P. G. Waddell, Martin Zettersten, and Melanie Soderstrom. 2020. Building… find
  5. Anny Castilla-Earls, David J. Francis, and Aquiles Iglesias. 2022. The Complex Role of Utterance Length on Grammaticality: Multivariate Multilevel Analysis of English and Spanish Utterances of First-Grade English Learners. Journal of Speech, Language, and Hearing Research : JSLHR, 65(1):238–252. find
  6. Anny P. Castilla-Earls, Brittany Harvey, Katrina Fulcher-Rood, and Christopher D. Barr. 2020. The Impact of Clinical Review Bias on Child Language Grammaticality. Communication Disorders Quarterly, 41(4):214–221. find
  7. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. find
  8. Noam Chomsky. 1957. Syntactic Structures. Mouton. find
  9. Michelle M. Chouinard and Eve V. Clark. 2003. Adult reformulations of child errors as negative evidence. Journal of Child Language, 30(3):637–669. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  11. Sarita L. Eisenberg and Ling-Yu Guo. 2013. Differentiating Children With and Without Language Impairment Based on Grammaticality. Language, Speech, and Hearing Services in Schools, 44(1):20–31. find
  12. Raquel Fernández, Jonathan Ginzburg, and Shalom Lappin. 2007. Classifying Non-Sentential Utterances in Dialogue: A Machine Learning Approach. Computational Linguistics, 33(3):397–427. find
  13. Michael C. Frank, Elika Bergelson, Christina Bergmann, Alejandrina Cristia, Caroline Floc-cia, Judit Gervain, J. Kiley Hamlin, Erin E. Hannon, Melissa Kline, Claartje Levelt, Casey Lew-Williams, Thierry Nazzi, Robin Panneton, Hugh Rabagliati, Melanie Soderstrom, Jessica Sullivan, Sandra Waxman, and… find
  14. E Mark Gold. 1967. Language identification in the limit. Information and Control, 10(5):447–474. find
  15. Ling-Yu Guo and Phyllis Schneider. 2016. Differentiating School-Aged Children With and Without Language Impairment Using Tense and Grammaticality Measures From a Narrative Task. Journal of Speech, Language, and Hearing Research, 59(2):317–329. find
  16. Randy Allen Harris. 1993. The linguistics wars. Oxford University Press. find
  17. Eve V Clark. 2016. First Language Acquisition. Cambridge University Press. find
  18. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2022. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. In Proceedings of the International Conference on Learning Representations. our card
  19. M. J. Demetras, Kathryn Nolan Post, and Catherine E. Snow. 1986. Feedback to first language learners: the role of repetitions and clarification questions*. Journal of Child Language, 13(2):275–292. find
  20. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED AT-TENTION. In Proceedings of the International Conference on Learning Representations. find
  21. Sarah Hiller and Raquel Fernandez. 2016. A Data-driven Investigation of Corrective Feedback on Subject Omission Errors in First Language Acquisition. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 105–114, Berlin, Germany. Association for Computational… find
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  23. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics. our card
  24. Jong-Bok Kim and Peter Sells. 2008. English syntax: An introduction. CSLI publications Stanford, CA. find
  25. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, San Diego, CA, USA. find
  26. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. find
  27. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive Science, 41(5):1202–1241. find
  28. S. Lawrence, C.L. Giles, and S. Fong. 2000. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 12(1):126– 140. Conference Name: IEEE Transactions on Knowledge and Data Engineering. find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  30. Ilya Loshchilov and Frank Hutter. 2018. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Representations. find
  31. Virginia A. Marchman and Elizabeth Bates. 1994. Continuity in lexical and morphological development: a test of the critical mass hypothesis. Journal of Child Language, 21(2):339–366. Publisher: Cambridge University Press. find
  32. Gary F. Marcus. 1993. Negative evidence in language acquisition. Cognition, 46(1):53–85. find
  33. B. W. Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451. find
  34. James L. McClelland and Karalyn Patterson. 2002. Rules or connections in past-tense inflections: What does the evidence rule out? Trends in cognitive sciences, 6(11):465–472. find
  35. Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and Ekaterina Artemova. 2022. RuCoLA: Russian Corpus of Linguistic Acceptability. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5207– 5227, Abu Dhabi, United Arab Emi… find
  36. James L. Morgan, Katherine M. Bonamo, and Lisa L. Travis. 1995. Negative evidence on negative evidence. Developmental Psychology, 31:180–197. find
  37. Keith E Nelson, Gaye Carskaddon, and John D Bonvillian. 1973. Syntax Acquisition: Impact of Experimental Variation in Adult Verbal Interaction with the Child. Child Development, 44(3):497–504. find
  38. Mitja Nikolaus and Abdellah Fourtassi. 2023. Communicative Feedback in Language Acquisition. New Ideas in Psychology. find
  39. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12:2825– 2830. find
  40. Steven Piantadosi. 2023. Modern language models refute Chomsky’s approach to language. Lingbuzz Preprint, lingbuzz, 7180. find
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  42. Mabel L. Rice, Filip Smolik, Denise Perpich, Travis Thompson, Nathan Rytting, and Megan Blossom. 2010. Mean Length of Utterance Levels in 6-month Intervals for Children 3 to 9 Years with and without Language Impairments. Journal of speech, language, and hearing research : JSLHR, 53(2):333. find
  43. David E. Rumelhart and James L. McClelland. 1986. On learning the past tenses of English verbs. In Psycholinguistics: Critical Concepts in Psychology, Volume 4, pages 216–271. Routledge. find
  44. Kenji Sagae, Alon Lavie, and Brian MacWhinney. 2005. Automatic measurement of syntactic development in child language. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL ’05, pages 197–204, Ann Arbor, Michigan. Association for Computational Linguistics. find
  45. Joachim Wagner, Jennifer Foster, and Josef van Genabith. 2009. Judging Grammaticality: Experiments in Sentence Classification. CALICO Journal, 26(3):474–490. find
  46. Anne S. Warlaumont, Jeffrey A. Richards, Jill Gilk-erson, and D. Kimbrough Oller. 2014. A Social Feedback Loop for Speech Development and Its Reduction in Autism. Psychological Science, 25(7):1314–1324. find
  47. Matthew Saxton. 1997. The Contrast Theory of negative input. Journal of Child Language, 24(1):139–161. find
  48. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. find
  49. Matthew Saxton. 2000. Negative evidence and negative feedback: immediate effects on the grammaticality of child speech. First Language, 20(60):221–252. find
  50. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  51. Matthew Saxton, Phillip Backley, and Clare Gall-away. 2005a. Negative input for grammatical errors: effects after a lag of 12 weeks. Journal of Child Language, 32(3):643–672. find
  52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, and Morgan Funtowicz. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language proce… our card
  53. Matthew Saxton, Carmel Houston–Price, and Natasha Dawson. 2005b. The prompt hypothesis: Clarification requests as corrective input for grammatical errors. Applied Psycholinguistics, 26(3):393–414. find
  54. Hollis S. Scarborough. 1990. Index of Productive Syntax. Applied Psycholinguistics, 11(1):1–22. find
  55. Taiga Someya, Yushi Sugimoto, and Yohei Oseki. 2023. JCoLA: Japanese Corpus of Linguistic Acceptability. ArXiv:2309.12676 [cs]. find
  56. Sofía M. Souto, Laurence B. Leonard, and Patricia Deevy. 2014. Identifying Risk for Specific Language Impairment with Narrow and Global Measures of Grammar. Clinical linguistics & phonetics, 28(10):741–756. \#\# 10. find
  57. Language Resource References MacWhinney, Brian. 2000. The CHILDES Project: Tools for analyzing talk. Transcription format and programs. Psychology Press. find
  58. Dominique Sportiche, Hilda Koopman, and Edward Stabler. 2013. An Introduction to Syntactic Analysis and Theory. John Wiley & Sons. find
  59. Michael Tomasello. 2003. Constructing a Language: A Usage-Based Theory of Language Acquisition. Harvard University Press. find
  60. Daniela Trotta, Raffaele Guarasci, Elisa Leonardelli, and Sara Tonelli. 2021. Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2929–2940, Punta Cana, Dominican Republic. Association for… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nikolaus-2024-automatic-annotation-grammaticality-child-caregiver/