📄 Morphosyntactic Analysis of the CHILDES and TalkBank Corpora

Summary

The paper describes the toolchain that adds morphological and syntactic annotation to spoken-language transcripts in CHILDES and the wider TalkBank collection. CHILDES holds about 60 million words of child–adult talk in 26 languages, and TalkBank about 63 million words of adult conversation. Off-the-shelf taggers fit poorly. Many were closed-source and built for written text, and they ignored conversational material such as interjections, onomatopoeia, babbling and code mixing. Finite-state tools also required more programming skill than many linguists have.

The author presents MOR, a tagger that works by generation. Its lexicons are plain text files that non-programmers can extend. Allomorphy rules (arules) compile a runtime lexicon stored in a trie, and concatenation rules (crules) assemble morphemes into analyses of whole words. A companion program, GRASP, adds dependency relations.

MOR currently covers 11 languages, while GRASP dependency analysis has been built for three. For English, tagging accuracy is 98% on adult speech and 97% on child speech. The annotated corpora already support computational models of language acquisition and automatic clinical profiles such as DSS and IPSyn.

Key points

• Data: TalkBank comes from 128 projects collected over 40 years and is transcribed in CHAT, which converts to TalkBank XML. Only about a quarter of CHILDES transcripts are linked to media.
• MOR grammars cover English, Spanish, Japanese, Mandarin, Cantonese, French, Italian, German, Dutch, Danish and Hebrew; Cantonese-English bilingual corpora serve as the bilingual case study. They are distributed through CLAN.
• The English lexicon is split into 87 files, one per POS type, and 27 of these are compound types (e.g., noun+noun vs noun+participle).
• The %mor notation uses specific marks:
  • # after prefixes and | between category and stem;
  • - for transparent affixes and & for fusional ones;
  • ~ for clitics, as in mod|do~neg|not;
  • + for compounds.
• GRASP for English uses 42 grammatical relations plus 13 ellipsis relations. Chinese and Spanish need only small changes, while Japanese adds eight relations. Output is compared against gold-standard files with TRNFIX.

Interesting ideas

• I1. MOR generates all allomorphs ahead of time into a trie instead of recognising word forms, which makes overgeneration easier to control and debugging output more readable.
• I2. Rule systems are split by interaction type: arules are strictly ordered with specific rules first, and one firing blocks later ones without chaining. Crules, by contrast, all fire in parallel, and a thread survives only if it covers the whole word.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en morphology crate could precompile every allomorph of each lemma into a trie, so each analysis traces back to a specific lexicon entry plus a specific rule.
• A lock-and-key feature check blocks overgeneration: a rule requires a feature to be absent (e.g., ![gem no]), and exceptional entries like "put" carry that feature, so no "putting"-style error appears. — (tokens): Irregular lemmas could carry explicit blocking features rather than exception lists hidden in code, which keeps every refusal explainable.
• I2 (tokens): Mova could adopt this split, with deterministic ordered rules for stem variants and exhaustive parallel analysis that keeps only complete-word parses.
• Fixed multiword function words such as "so that" and "in order to" are treated as single tokens (so_that, in_order_to), which the author reports improved syntactic analysis. — (tree): Merging such connectives before parsing could reduce spurious attachments, as long as a conversion step maps them back to UD's fixed relation.
• A fast whole-corpus run (mor +xl) lists every word the grammar fails to recognise, which turns lexicon maintenance into a routine coverage check. — (learn): Mova could run an unknown-word report after each lexicon or corpus change, as a cheap regression signal before training silver data.