📄 Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models

Summary

The paper asks whether the six classic obstacles for neural MT identified by Koehn and Knowles (2017) still apply when translation is done by decoder-only LLMs. The six are domain mismatch, parallel data volume, rare words, long sentences, attention-based word alignment, and beam search. The authors fine-tune Llama2-7B and Llama2-13B for German-to-English in two ways. LLM-SFT uses bilingual pairs mixed with Alpaca. LLM-CPT-SFT adds continued pretraining on concatenated pairs before Alpaca tuning. Both are compared with Fairseq Transformer (Enc2Dec) baselines on WMT23 and OPUS multi-domain data, scored with BLEU and COMET-DA.

LLMs largely remove the need for large parallel corpora in well-resourced languages and handle long inputs well. Domain shift and rare words remain unsolved, and alignment and beam search matter less in the LLM setting. The study also identifies three LLM-era problems:

• slow inference,
• weak translation into languages underrepresented in pretraining, such as Ukrainian and Hebrew,
• automatic metrics that disagree with human judgments.

The work provides an updated research agenda for LLM-based MT, which is relevant to low-resource directions such as English-to-Ukrainian.

Key points

• Setup: Training draws up to 20M pairs from roughly 300M WMT23 De-En pairs. Testing uses generaltest2023 to avoid data leakage. Domain experiments use OPUS multi-domain sets.
• Domain results: In the law domain, LLM-SFT reaches 62.0 BLEU and 88.0 COMET-DA, about 3.0 BLEU and 4.9 COMET-DA above Enc2Dec. Moving from Koran to Law, however, costs up to 40.0 BLEU and 14.4 COMET-DA.
• Long inputs and speed: LLMs translate sentences of about 80 words well and manage documents of up to 512 words. Their inference was about 100 times slower than Enc2Dec in the authors' experiments.
• Ukrainian: Ukrainian makes up about 0.07% of Llama2 pretraining data, and En→Uk quality jumps between 100k and 500k parallel pairs. ALMA-7B and TowerInstruct-7B reach 88.3 and 87.0 COMET-DA on En→Uk without Ukrainian pretraining, which the authors attribute to Russian data.
• Evaluation gap: On the WMT23 literary Zh-En test, human scores (MQM) and automatic scores (d-BLEU) for GPT-4, a document-tuned Llama2-7B, and Google Translate correlate at Pearson −0.53.