Elamur Research⌕
☰
home / papers / 2020 / Norm-Based Curriculum Learning for Neural Machine Translation

Norm-Based Curriculum Learning for Neural Machine Translation

Paper · 2020 · Xuebo Liu, Houtim Lai, Derek F. Wong et al.

Summary

Training neural machine translation (NMT) systems is slow and expensive, especially deep or large-batch Transformer models on high-resource data. Curriculum learning (CL) can help by presenting easier examples before harder ones. However, existing difficulty measures are either cheap but shallow (sentence length, word rarity) or informative but costly (uncertainty from auxiliary models). Competence-based CL also requires a task-specific curriculum length, which is tuned using a full baseline run.

The authors use vector norms in three ways:

  • Sentence difficulty: the sum of word-vector norms from a simple word embedding model trained on the source corpus. This captures both rarity and context-specific "significant" words.
  • Model competence: the growing norm of the NMT source embedding, which lets the curriculum set its own pace.
  • Sentence weights: norm-based weights that rescale the loss to reduce over-training on easy data.

On WMT'14 English–German and WMT'17 Chinese–English, the approach beats strong Transformer baselines in both BLEU and training speed. It changes only the sampling pipeline and objective, adding no parameters. This makes it practical for cutting the cost of large NMT training.

Key points

  • Main results: +1.17 BLEU on WMT'14 English–German and +1.56 BLEU on WMT'17 Chinese–English, with training speedups of 2.22x and 3.33x respectively.
  • Difficulty measure: sentence difficulty is d(x) = Σ‖xᵢ‖ over embeddings from a lightweight word embedding model (E_w2v) that trains quickly on CPU. On WMT'14 En–De source data, rare and context-insensitive words receive higher norms.
  • Competence schedule: the method builds on the competence-based CL of Platanios et al. (2019), using the square-root competence function with c₀ = 0.01. It replaces the task-dependent length hyperparameter λₜ, normally set from the step where a baseline reaches 90% of its final BLEU, with the norm of the NMT source embedding E_nmt. The motivation is that this norm and dev-set BLEU both rise sharply until about 30K steps.
  • Effect of sentence weights: the En–De test set (3,003 sentences) was split into three 1,001-sentence subsets by length-, frequency- and norm-based difficulty. Plain competence-based CL did better on easy sentences but worse on medium ones. Norm-based sentence weights removed this bias and also improved competence-based CL when combined with it.
  • Case study: on a hard multi-clause sentence, the vanilla Transformer dropped the final clause, while the norm-based CL model translated the whole sentence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China10 works · 1,905 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Xuebo Liu, Houtim Lai, Derek F. Wong and Lidia S. Chao. 2020. Norm-Based Curriculum Learning for Neural Machine Translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.41

@inproceedings{liu2020norm,
  title = {{Norm-Based Curriculum Learning for Neural Machine Translation}},
  author = {Xuebo Liu, Houtim Lai, Derek F. Wong and Lidia S. Chao},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.41},
  url = {https://doi.org/10.18653/v1/2020.acl-main.41}
}

References

41 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.
  2. Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training Deeper Neural Machine Translation Models with Transparent Attention. In EMNLP 2018. find
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ICML 2009.
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. In TACL 2017.
  5. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP 2018. find
  6. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In ICLR 2019.
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In ICML 2017.
  8. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. In EMNLP 2019.
  9. Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. find
  10. FRAGE: Frequency-Agnostic Word Representation. In NIPS 2018. find
  11. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In ACL 2018.
  12. Adriaan M J Schakel and Benjamin J Wilson. 2015. Measuring Word Significance using Distributed Representations of Words. arXiv. find
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP 2013.
  14. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015. find
  15. Tom Kocmi and Ondrej Bojar. 2017. Curriculum Learning and Minibatch Bucketing in Neural Machine Translation. In RANLP 2017. find
  16. Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019. Reinforcement learning based curriculum optimization for neural machine translation. In NAACL 2019. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL 2016.
  18. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS 2014. find
  19. Joseph P Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL 2010.
  20. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR 2017.
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017.
  22. Xuebo Liu, Derek F. Wong, Lidia S. Chao, and Yang Liu. 2019a. Latent attribute based hierarchical decoder for neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2103–2112. find
  23. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In ACL 2019.
  24. Xuebo Liu, Derek F. Wong, Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu. 2019b. Shared-private bilingual word embeddings for neural machine translation. In ACL 2019. find
  25. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159–165. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR 2013.
  27. Toan Nguyen and David Chiang. 2019. Improving lexical choice in neural machine translation. In NAACL-HLT 2019. find
  28. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT@EMNLP 2018.
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL 2002.
  30. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP 2016.
  31. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In NAACL 2019. find
  32. Ofir Press and Lior Wolf. 2017. Using the Output Embedding to Improve Language Models. In EACL 2017.
  33. Benjamin J Wilson and Adriaan M J Schakel. 2015. Controlled Experiments for Word Embeddings. arXiv. find
  34. Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Depth growing for neural machine translation. In ACL 2019. find
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv. find
  36. Biao Zhang, Ivan Titov, and Rico Sennrich. 2019a. Improving deep transformer with depth-scaled initialization and merged attention. In EMNLP 2019. find
  37. Dakun Zhang, Jungi Kim, Josep Crego, and Jean Senellart. 2017. find
  38. Boosting Neural Machine Translation. In IJCNLP 2017. find
  39. Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An Empirical Exploration of Curriculum Learning for Neural Machine Translation. arXiv. find
  40. Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019b. Curriculum Learning for Domain Adaptation in Neural Machine Translation. In NAACL 2019.
  41. Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020. Uncertainty-aware curriculum learning for neural machine translation. In ACL 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2020-norm-based-curriculum-learning-neural/