Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts
Summary
Common measures of preschoolers' language development, such as IPSyn, take into account only correctly used grammatical constructions. They do not analyze the child's errors. The authors examine how well existing NLP methods can automatically detect grammatical errors in transcripts of child speech. To this end, they used the Paradise corpus, which consists of conversations between six-year-old English-speaking children and adults. Some of the children were assigned to a group with language impairment (LI), and the rest to a typically developing (TD) group. Each child utterance was manually annotated for ten error types. Most of these types involve verbs, since verbs cause the greatest difficulty for children with LI. For six categories, two approaches were compared: rules based on matching parse tree patterns, and statistical classifiers. These categories include missing verb or copula, subject–auxiliary or subject–main verb agreement, and missing infinitive marker "to". The statistical models generally performed better, and recall exceeded 84% in all categories. Automatic error analysis can complement clinical language assessment, and error statistics may serve as a feature for identifying LI.
Key points
- The Paradise corpus contains 677 transcripts (623 TD and 54 LI), i.e. 108,711 child utterances and 394,290 words. The mean length of utterance (MLU) is 3.64.
- The most frequent error is a missing copula: 36.67% of all errors. In second place is incorrect verb use (14.59%). Children with LI made errors more often in almost all categories. The exceptions are missing article and missing subject.
- Inter-annotator agreement was checked on 386 utterances. The mean Cohen's kappa was 77.7%, with 43 disagreements.
- At the sentence level, binary classifiers were applied sequentially, one after another. The best result, an F1 of 0.967, was achieved by classifiers with features that do not rely on rules. The multinomial naive Bayes classifier obtained the same F1 for missing "to".
- Limitations of the work: the system does not account for incomplete utterances, disfluencies, or dialectal constructions (e.g., in African American English). Missing articles and prepositions are not yet detected.
Topics Psycholinguistics & language acquisition Grammatical error correction & learner language
Cite
Cite as Khairun-nisa Hassanali and Yang Liu. 2011. Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts. Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications.
@inproceedings{hassanali2011measuring,
title = {{Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts}},
author = {Khairun-nisa Hassanali and Yang Liu},
booktitle = {Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications},
year = {2011},
url = {https://aclanthology.org/W11-1411/}
}References
- Mari I. Bowden and Richard K. Fox. 2002. A Diagnostic Approach to the Detection of Syntactic Errors in English for Non-Native Speakers. Technical report, The University of Texas-Pan American. find
- Andrew Caines and Paula Buttery. 2010. You talking to me?: A predictive model for zero auxiliary constructions. In Proceedings of the 2010 Workshop on NLP and Linguistics: Finding the Common Ground, pages 43–51. find
- Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pages 132–139. our card
- Lloyd M. Dunn. 1965. Peabody picture vocabulary test. American Guidance Service Circle Pines, MN. find
- Jens Eeg-Olofsson and Ola Knuttson. 2003. Automatic grammar checking for second language learners-the use of prepositions. In Proceedings of NoDaLiDa. find
- Keyur Gabani. 2009. Automatic identification of language impairment in monolingual English-speaking children. Master’s thesis, The University Of Texas At Dallas. find
- Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1):10–18. find
- Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. Automated Grammatical Error Detection for Language Learners. Synthesis Lectures on Human Language Technologies, 3(1):1–134. find
- John Lee and Stephanie Seneff. 2006. Automatic grammar correction for second-language learners. In Proceedings of INTERSPEECH-2006, pages 1978–1981. find
- John Lee and Stephanie Seneff. 2008. Correcting misuse of verb forms. In Proceedings of ACL-08:HLT, pages 174–182. find
- Jack L. Paradise, Thomas F. Campbell, Christine A. Dollaghan, Heidi M. Feldman, Bernard S. Bernard, D. Kathleen Colborn, Howard E. Rockette, Janine E. Janosky, Dayna L. Pitcairn, Marcia Kurs-Lasky, et al. 2005. Developmental outcomes after early or delayed insertion of tympanostomy tubes. New Englan… find
- Mabel L. Rice, Kenneth Wexler, and Patricia L. Cleave. 1995. Specific language impairment as a period of extended optional infinitive. Journal of Speech and Hearing Research, 38(4):850. find
- Kenji Sagae, Eric Davis, Alon Lavie, Brian MacWhinney, and Shuly Wintner. 2007. High-accuracy annotation and parsing of CHILDES transcripts. In Proceedings of the Workshop on Cognitive Aspects of Computational Language Acquisition, pages 25–32. our card
- Hollis S. Scarborough. 1990. Index of productive syntax. Applied Psycholinguistics, 11(01):1–22. find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/hassanali-2011-measuring-language-development-early-childhood/