‘Indicatements' that character language models learn English morpho-syntactic units and regularities

Summary

The paper asks whether a character-level language model that reads raw, unsegmented English actually picks up morphology, or only seems to because it shares parameters across similar strings. The authors train a 'wordless' character LSTM LM (spaces are ordinary symbols) on 7M characters of English Wikipedia and examine it in several ways. They inspect sampled nonce words, look for interpretable hidden units, attach a frozen-encoder segmentation probe (C2M) and a POS probe (C2T), and measure how likely derivational suffixes are after bases of different word classes.

The LM produces plausible over-regularized forms such as breaked and indicatement. It also develops a single "word unit" that fires at word ends and at word-like bases inside longer words. Morpheme boundaries appear to be learned mainly by extending knowledge of word boundaries. The LM respects the selectional restrictions of most suffixes tested. This matters because it gives direct evidence for, and a limit on, claims that character models make morphological analysis unnecessary. The mechanism should work well for English but less well for languages whose morphemes rarely coincide with words.

Key points

• Model and training: a one-layer LSTM with 256 hidden units, learning rate 0.003, minibatch 50 and dropout 0.2, trained on the first 7M characters of English Wikipedia (90/10 train/dev split, not lowercased).
• Sampled text: roughly 1 in 20 sampled tokens is a nonce word, checked against the training data and the pyenchant dictionaries.
• Hidden units: about 40 units looked interpretable. Examples include a punctuation unit, a Latinate -ion suffix unit and a word-boundary unit. The word unit's activation correlates only 0.08 with the predicted probability of a following space, so it is not simply a space predictor.
• Segmentation probe: C2M was trained on gold segmentations from MorphoChallenge 2010 and Hutmegs 1.0 (2275 word forms), using the LM's frozen encoder. It reached an overall F1 of 53.3. Scores per boundary type ranged widely, from 27.6 to 90.9.
• Suffix selection: 11 of 19 derivational suffixes were more probable after bases of the correct class, among them -ment, -able, -ness and -ity. The -ion errors trace to frequency: the suffix appears in 18,945 words, while its bases appear alone only about 9,755 times, so the LM treats bases like populat as nominal.

Interesting ideas

• I1. Frozen-encoder probes: attach small task heads (segmentation, tagging) to a frozen pretrained encoder, so the head's score measures what the encoder already knows rather than what fine-tuning adds.
• I2. Over-regularization as a diagnostic: nonce forms like breaked or indicatement show that a model applies a regular rule which a lexical exception should have blocked. Collecting such errors shows which rules were learned and which exceptions are missing.
• I3. Ruling out a trivial explanation: the authors checked that the word unit is not just a "space comes next" detector by correlating its activation with the model's space probability.
• I4. Selectional restrictions via conditional probability: they tested whether a suffix is more probable after bases of its required word class, which turns a morpho-syntactic constraint into a measurable score.
• I5. Frequency explains category drift: a base seen far more often with a suffix than without it gets absorbed into the suffix's word class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Use frozen-encoder probes on gparse's transformer layers to check which layers carry morphology versus syntax before deciding where to tap features.
• I2 (tokens): Feed generated or misspelled forms to the lemmatizer and flag cases where a regular analysis overrides a known irregular, as tests for the exception tables.
• I3 (learn): When a feature or repair rule seems to capture a linguistic notion, add a negative control that correlates it with the obvious surface cue it might be proxying.
• I4 (world): Apply the same contrast to compiled knowledge, such as which derivational suffixes attach to which verb classes, to validate WordNet/VerbNet-derived derivation links.
• I5 (tokens): When the tagger mislabels a bound base, check base-alone versus base+suffix counts in the corpus before blaming the model.