📄 What Kind of Language is Easy to Language-Model Under Curriculum Learning?

Summary

Prior work showed that some language models (LMs) learn typologically common word orders more easily than rare ones. This study asks whether that finding holds when the input is ordered from simple to complex, as in "starting small" accounts of child language acquisition. The authors reuse the 96 GCG-based artificial languages of El-Naggar et al. (2025a,b). These languages vary along seven binary word-order parameters and range from Japanese-like (head-final) to English-like orders. RNN, LSTM and Transformer LMs were trained in Fairseq with a length-based curriculum, and the results were compared with earlier runs that used randomly shuffled data. The curriculum made perplexity on short in-distribution sentences worse but helped generalization to longer sentences. It also generally weakened the correlation between learnability and real-world word-order frequency, although the Long test set still showed significant alignment. The authors conclude that the training scenario interacts strongly with apparent LM inductive biases. Non-curriculum studies may therefore overstate how closely LM biases match typological patterns, or simple length ordering may be too crude a model of acquisition.

Key points

• Data and evaluation sets: Each artificial language has 80K training sentences of 3–8 words. Evaluation uses three test sets: Short (same lengths as training), Medium (9–10 words) and Long (11–20 words).
• Curriculum design: Training runs in six stages, one per sentence length from 3 to 8. Each stage replays 5% of the data from every earlier stage to limit forgetting. Every stage except the last lasts 2 epochs; the last uses early stopping with 5-epoch patience. All models are trained with three seeds.
• Typological alignment (TA) metric: TA is the Pearson correlation between a model's perplexity and how frequent each word order is among the world's languages. Negative values mean better alignment. The curriculum shifted TA most for RNN and LSTM on the Short and Medium sets, usually for the worse.
• Grammaticality judgments: The authors tested case-marker swaps and transitive-to-intransitive verb swaps on 500 items per language. The curriculum lowered accuracy for every model. For example, RNN verb-judgment accuracy dropped from 77.4 to 63.0, and Transformer accuracy dropped from 81.0 to 74.4.
• Framing: The authors present the study as exploratory and draw no firm claim that curriculum learning helps or hurts alignment. They call for testing other curriculum strategies, such as ordering by construction type.