预训练神经机器翻译研究进展分析

Summary

Neural machine translation (NMT) usually relies on large parallel corpora, which exist for only a few language pairs such as Chinese–English and English–French and are concentrated in domains like news and patents. Monolingual text is far easier to collect. This survey, from the NLP Lab at Northeastern University (Shenyang), reviews how pre-trained models (PTM) trained on such data can strengthen NMT. The authors call this approach pre-trained NMT (PTNMT) and present it as the first standalone, systematic review of the topic.

They identify three sources of mismatch between PTMs and NMT:

• Training data: monolingual versus bilingual, which can cause copied or hallucinated output.
• Architecture: encoder-only or decoder-only models versus encoder–decoder translation systems.
• Training objective: unsupervised pre-training versus supervised translation, which can lead to catastrophic forgetting.

The work is organized into four groups: pre-training strategies, strategies for integrating PTMs into NMT, model and data analysis, and applications in specific settings. A key observation is that PTNMT is well validated on low-resource data, but using PTMs efficiently in high-resource NMT remains open.

Key points

• Pre-training families covered:
  • masked language modeling variants: BERT, XLM with CLM/TLM, CMLM, MASS, T5, VECO;
  • denoising autoencoding: BART, mBART, IndicBART;
  • code-switching: ALM, mRASP, mRASP2, CeMAT.
• BART and mBART: BART uses five noising functions, from token masking to document rotation. mBART is pre-trained on monolingual corpora in 25 languages.
• Limits at high resource: mBART helps low- and medium-resource translation. With more than 25 million parallel sentence pairs, its performance drops slightly, possibly because supervised training overwrites the pre-trained weights.
• mRASP and mRASP2: mRASP improves high-resource translation as well and supports zero-shot translation. mRASP2 adds monolingual data and contrastive learning in the encoder.
• Integration and publication: integration strategies include fine-tuning, prompting and information fusion. The paper appeared in 中文信息学报 vol. 38, no. 6 (2024), pp. 1–23.