Elamur Research
☰
home / papers / 2019 / fairseq: A Fast, Extensible Toolkit for Sequence Modeling

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-4009.pdf

Summary

The authors present FAIRSEQ, an open-source PyTorch toolkit for sequence modeling. It aims to serve both research and production for tasks such as machine translation, summarization and language modeling. The design is modular. Users can plug in their own models, criterions, tasks, optimizers and learning rate schedulers while reusing the rest of the framework. Checkpoints store the full training state, and older checkpoints are upgraded automatically.

On the engineering side, the toolkit supports:

  • length-based batching to minimize padding
  • multi-GPU and multi-machine training via NCCL2, which overlaps gradient synchronization with the backward pass and accumulates gradients to reduce straggler idle time
  • FP16 training with dynamic loss scaling
  • fast incremental decoding with several search strategies

Reference models reach strong results. These include a big Transformer on WMT En–De and En–Fr, adaptive-input Transformer language models on WikiText-103 and One Billion Word, and a Transformer summarizer on CNN-DailyMail. The toolkit matters because it pairs fast, scalable training with a flexible codebase and released pretrained models. It is distributed under a BSD license.

Key points

  • Translation: the big Transformer, trained on WMT'16 for En–De and WMT'14 for En–Fr, scores 29.3 BLEU on En–De and 43.2 on En–Fr (newstest14). Training takes about 1.3 h (En–De) and about 7.2 h (En–Fr) on 128 V100 GPUs.
  • FP16 inference: half precision raises decoding throughput from 88.1 to 136.0 sentences/sec (+54%) on a V100 with no accuracy loss. The benchmark was the WMT'14 En–De test set.
  • Language modeling: adaptive-input Transformer LMs reach a test perplexity of 18.7 on WikiText-103 (previous best cited: 29.2) and 23.0 on One Billion Word (previous best cited: 28.0).
  • Summarization: on full-text CNN-DailyMail, the model scores ROUGE-1/2/L of 40.1/17.6/36.8. Adding pretrained LM representations (trained on 193M sentences) raises this to 41.6/18.9/38.5.
  • Extensibility and availability: there are five plug-in types (models, criterions, tasks, optimizers, LR schedulers), including an Adafactor implementation. The code is at github.com/pytorch/fairseq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

1595 papers of our library cite it (of the 140673 whose bibliographies we read); 60 of them have a page here, 1535 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-4009 · arXiv 1904.01038

@inproceedings{ott2019fairseq,
  title = {{fairseq: A Fast, Extensible Toolkit for Sequence Modeling}},
  author = {Ott, Myle and Edunov, Sergey and Baevski, Alexei and Fan, Angela and Gross, Sam and Ng, Nathan and Grangier, David and Auli, Michael},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)},
  year = {2019},
  doi = {10.18653/v1/n19-4009},
  eprint = {1904.01038},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-4009}
}

References

51 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv, abs/1612.04426. find
  2. Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted transformer network for machine translation. arxiv, 1711.02132. find
  3. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  4. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv, abs/1812.10464. find our card
  5. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2018. Sockeye: A Toolkit for Neural Machine Translation. arXiv, abs/1712.05690. — [arXiv:1712.05690](https://arxiv.org/abs/1712.05690) find
  6. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In Proc. of ICLR. find
  7. Yun Chen, Victor OK Li, Kyunghyun Cho, and Samuel R Bowman. 2018. A stable and effective learning strategy for trainable greedy decoding. arXiv, abs/1804.07915. find
  8. Hieu Hoang, Philipp Koehn, Ulrich Germann, Kenneth Heafield, and Barry Haddow. 2006. multi-bleu.perl. https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/ generic/multi-bleu.perl. ↗
  9. Shamil Chollampatt and Hwee Tou Ng. 2018. A multilayer convolutional encoder-decoder neural network for grammatical error correction. arXiv, abs/1801.08831. find
  10. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv, abs/1602.02410. find
  11. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proc. of ICML. find our card
  12. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proc. of ACL 2018, System Demo… find our card
  13. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proc. of ICLR. find
  14. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018a. Understanding back-translation at scale. In Conference of the Association for Computational Linguistics (ACL). find
  15. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Proc. of AAAI. find our card
  16. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In Proc. ACL. find
  17. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, et al. 2018b. Classical structured prediction losses for sequence to sequence learning. In Proc. of NAACL. find
  18. Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Carl Case, and Paulius Micikevicius. 2018. OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models. In Proc. of Workshop for NLP Open Source Software. find
  19. Angela Fan, David Grangier, and Michael Auli. 2018a. Controllable abstractive summarization. In ACL Workshop on Neural Machine Translation and Generation. find our card
  20. Angela Fan, Mike Lewis, and Yann Dauphin. 2018b. Hierarchical neural story generation. In Proc. of ACL. find
  21. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proc. of EMNLP. find
  22. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv, abs/1902.01109. find
  23. Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In ACL Workshop on Text Summarization Branches Out. find
  24. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML. find our card
  25. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv, abs/1808.10792. find our card
  26. Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018. Controlling length in abstractive summarization using a convolutional neural network. In Proc. of EMNLP. find
  27. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for gpus. In Proc. of ICML. find
  28. Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. In Proc. of ICLR. find
  29. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv, abs/1701.06538. find
  30. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP. find
  31. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. arXiv, abs/1803.08240. find
  32. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235. find
  33. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. In Proc. of ICLR. find
  34. Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. arXiv, abs/1902.07816. — [arXiv:1902.07816](https://arxiv.org/abs/1902.07816) find
  35. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. arXiv, abs/1705.06476. find our card
  36. Kaitao Song, Xu Tan, Di He, Jianfeng Lu, Tao Qin, and Tie-Yan Liu. 2018. Double path networks for sequence to sequence learning. arXiv, abs/1806.04856. find
  37. Makoto Morishita, Yusuke Oda, Graham Neubig, Koichiro Yoshino, Katsuhito Sudoh, and Satoshi Nakamura. 2017. An empirical study of mini-batch creation strategies for neural machine translation. In Proc. of WMT. find
  38. A. Vaswani, S. Bengio, E. Brevdo, F. Chollet, A. N. Gomez, S. Gouws, L. Jones, Ł. Kaiser, N. Kalchbrenner, N. Parmar, R. Sepassi, N. Shazeer, and J. Uszkoreit. 2018. Tensor2Tensor for Neural Machine Translation. arXiv, abs/1803.07416. find our card
  39. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In SIGNLL Conference on Computational Natural Language Learning. find
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS. find
  41. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv, abs/1808.08745. find
  42. Ashwin K Vijayakumar, Michael Cogswell, Ram-prasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424. find our card
  43. Myle Ott, Michael Auli, David Grangier, and MarcAu-relio Ranzato. 2018a. Analyzing uncertainty in neural machine translation. In Proc. of ICML. find our card
  44. Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convlutions. In Proc. of ICLR. find
  45. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018b. Scaling neural machine translation. In Proc. of WMT. find our card
  46. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  47. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv, abs/1804.08771. find our card
  48. Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. 2018. Fast parametric learning with activation memorization. arXiv, abs/1803.10049. — [arXiv:1803.10049](https://arxiv.org/abs/1803.10049) find
  49. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  50. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL. find
  51. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proc. of NAACL. find our card