📄 Efficient Architectures For Low-Resource Machine Translation

Summary

The paper asks how the shape of a Transformer affects translation quality when parallel data is very scarce, and whether large models are worth their cost there. The authors trained several hundred Fairseq Transformer configurations on four extremely low-resource pairs, each with 21k–50k sentence pairs: English-Akkadian, German-Lower Sorbian, English-Manipuri, and a simulated English-Italian set. They varied four architectural knobs (encoder depth, decoder depth, embedding size, feed-forward width) and held training settings fixed. First they changed one knob at a time, then they explored combinations. To judge cost against benefit, they introduced PIES, which measures how many million parameters a model spends per point of a translation metric. Large models gave the top ChrF, but small configurations reached similar quality: about 95% fewer parameters for a ChrF loss of 5.3–14.8%. Depth mattered most, and deeper stacks hurt both quality and efficiency. The work matters for anyone building MT where hardware and data are limited, and the paper ends with recommended hyperparameter ranges.

Key points

• Datasets: EvaCun 2023 (eng-akk, 45,269 pairs), WMT22 low-resource task (deu-dsb, 40,194), WikiMatrix random sample (eng-ita, 50,000, tested on Flores-200 splits), and WMT23 Indic task (eng-mni, 21,287).
• Training setup:
  • Baseline 4_4_256_1024_2 (enc_dec_embs_ffw_heads), with 2 attention heads and shared embeddings.
  • Separate 4k SentencePiece BPE vocabularies for source and target.
  • Early stopping after 20 epochs without BLEU gain, or a cap of 50k updates.
• PIES is model size divided by metric score, divided by 10^6; lower is better. Parameter counts come from a closed-form formula over E, V, F and the layer counts.
• Scores were bootstrapped over 200 batches of 400 test sentences. Reported metrics are BLEU, ChrF, ChrF++ and COMET, with ChrF as the main one.
• The strongest ChrF models were large and sometimes very lopsided:
  • 12_2_1024_4096_2 was best for both deu-dsb and eng-ita.
  • 2_16_1024_256_2 was best for eng-mni.
• Small models of about 9–10M parameters averaged roughly 42–43 ChrF with PIES around 0.21. Base-sized models of about 40–48M averaged roughly 45 ChrF with PIES near 0.9–1.5.

Interesting ideas

• I1. Express cost as parameters spent per metric point, so a slightly weaker but far smaller model can be ranked fairly against a bulky winner.
• I2. Count parameters with a closed-form formula over layer sizes, which lets you estimate model cost for a whole grid before training anything.
• I3. Run a two-stage sweep: first change one knob with the rest fixed, then search combinations. Afterwards, correlate each knob and their interactions with both quality and efficiency to see which knobs actually matter.
• I4. Report bootstrapped scores over many resampled test batches rather than one number, which steadies comparisons between close configurations on small test sets.
• I5. Encoder and decoder depth need not match, and lopsided stacks sometimes won. Asymmetry is therefore a design option worth testing, not a default to avoid.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Compute "million parameters per LAS point" for gparse variants, so model-size decisions rest on an explicit efficiency number rather than raw accuracy alone.
• I2 (tree): Write a similar size formula for gparse and use it to prune architecture sweeps to configurations that fit our GPU budget.
• I3 (tree): Apply this to the perceptron and transformer parsers, for example to learn whether depth or hidden width drives LAS on small treebanks.
• I4 (learn): Bootstrap LAS and answer accuracy over resampled test subsets, so a reported gain can be checked against noise alongside our negative controls.
• I5 (back): When sizing a neural component for graph-to-text generation or translation, try uneven encoder/decoder depths instead of assuming a symmetric layout.