📄 Constructivist Tokenization for English

Summary

The paper argues that English tokenization is not a solved problem. For semantic parsing and for modeling child language acquisition, the right units are morphemes rather than words. This follows constructivist theories such as Distributed Morphology, where meaning composition starts below the word, so a form like "missed" splits into the root miss plus a past-tense morpheme.

The authors take two steps:

• Rule-based converter. It uses gold PTB part-of-speech tags and the WordNet Lemmatizer to split words into lexical roots and functional morphemes (TAM, PLR, and similar). They apply it to the Penn Treebank, the CHILDES Treebank and raw CHILDES transcripts.
• Neural tokenizer. A character-level LSTM is trained on gold plus silver data to segment raw text directly.

Results:

• With gold tags, the converter reaches about 98–99% accuracy, and most remaining errors come from the source annotation.
• On raw child speech, accuracy falls to around 88–90%.
• The LSTM averages above 99%.

The output is a set of gold, silver and bronze morpheme-tokenized corpora. These can support acquisition modeling and morpheme-level semantics that align with DeepBank/ERS.

Key points

• Rule-based system accuracy (Table 2):
  • PTB: 97.81% on 6,069 tokens.
  • CTB: 99.01% on 5,161 tokens.
  • Child-directed speech (CDS): 89.84%.
  • Child-produced speech (CPS): 87.96%.
• Errors added by the algorithm itself (beyond errors already in the input annotations) stayed under 1% on every source. Most of these came from the lemmatizer:
  • plurals that were not singularized;
  • -ing verbs whose stem ends in e;
  • ambiguous irregular forms such as "saw".
• Tagging scheme: each morpheme gets a simplified root POS tag (VB, N, …) plus a function tag for suffixes, e.g. TAM for verbal tense/aspect/mood and PLR for the plural.
• Released corpora:
  • roughly 18,340 hand-checked gold tokens from CTB (brown-adam, brown-sarah, hslld) and PTB wsj 00;
  • silver corpora from PTB/CTB trees;
  • bronze corpora from the CHILDES Bloom corpus (176,700 CDS tokens and 126,286 CPS tokens).
• Neural tokenizer: an LSTM that assigns B/I/O labels to characters, trained on gold plus large silver data, with over 99% average accuracy.

Interesting ideas

• I1. Split each word into a lemma plus functional morphemes with their own tags, so tense and plurality become separate units that carry meaning instead of hiding in the word's tag.
• I2. Handle irregular forms by lemmatizing first and then reattaching the canonical suffix chosen from the POS tag, rather than writing string-stripping rules that break on "went" or "children".
• I3. Attribute every error to its source (original annotation, lemmatizer, or conversion algorithm) so you can measure how much noise the converter adds on top of its input.
• I4. Use a rule-based converter over gold annotations to mass-produce silver data, then train a character-level BIO tagger on it so the scheme also works on raw text without annotations.
• I5. Treat tense and plural morphemes as the elementary units of composition, so that time and number attach to the event and the entity directly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): crate en could expose an optional morpheme layer (root + TAM/PLR tokens) derived from the tagger, giving downstream steps explicit tense and number units.
• I2 (tokens): our lemmatizer plus tag could generate normalized morpheme sequences, with a targeted exception list for known failures like -e stems and "saw".
• I3 (learn): when converting treebanks or building silver data, report each tool's own added error rate separately from upstream noise to decide where repair rules pay off.
• I4 (learn): rule-converted silver segmentations could train a small learned segmenter that generalizes beyond the rules, checked against a hand-verified gold slice.
• I5 (world): explicit TAM and PLR units could feed event ordering and entity cardinality in the text's world model without re-deriving them from word tags.