📄 How much capacity does Turkish inflection require? An empirical study of GRU encoder–decoder bottlenecks.

Summary

The paper asks how much capacity neural sequence-to-sequence models need for morphological inflection, since standard models with 300–500 dimensions may be oversized. Using the Turkish data of the CoNLL–SIGMORPHON 2018 Shared Task (10k training and 1k development examples), the author trains tiny GRU encoder–decoders without attention. The stem and the feature bundle are compressed into a bottleneck vector of size h, and h is varied systematically. Accuracy follows an S-shaped curve. It is almost zero at the smallest size, climbs between h=16 and h=48, and nears its ceiling at h=64, which is about 70k parameters. Linear probes show that the bottleneck encodes vowel backness, the basis of Turkish harmony, even when the model cannot inflect at all. Rounding and height emerge only with more capacity. Adding attention does not remove the need to pass morphosyntactic features through the bottleneck. These results give a reproducible lower bound on capacity for this architecture and data, and they bear on interpretability and cognitive-learnability arguments.

Key points

• Experimental sweep: bottleneck sizes from h=8 to h=128 (10 values × 5 seeds = 50 runs; ~2.6k–95k parameters).
  • Character embedding size was fixed at d=8 after a separate sweep.
  • Training used AdamW on a single T4 GPU.
• Sequence accuracy: 0.6% at h=8, 53.6% at h=24 and 92.0% at h=64.
  • Variance across seeds is highest in the transition zone.
  • A harmony-aware edit distance falls steadily, from 6.5 to 0.15 by h=96.
• Backness probes: logistic-regression probes recover [±back] of the stem-final vowel at 96.1–99.9% for every size, against a 52.9% majority baseline.
• Rounding and height probes: [±round] and [±high] stay near their baselines (84.5% and 66.1%) at low h.
  • Turkish orthography makes all three features equally transparent, so this pattern argues against a purely orthographic explanation of the backness result.
• Bahdanau attention ablation:
  • Attention without the context vector collapses to about 1% accuracy, producing outputs with correct harmony but the wrong morphology.
  • Using the context only to initialize the decoder (INIT_CTX) gives 31.6% at h=16.
  • Feeding the context at every step (CONCAT_CTX) gives 84.7% at h=16 and 92.5% at h=24.