Elamur Research
☰
home / papers / 2017 / DyNet: The Dynamic Neural Network Toolkit

DyNet: The Dynamic Neural Network Toolkit

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1701.03980

Summary

DyNet is an open-source toolkit for building neural networks whose structure can vary from one input to the next. The authors argue that frameworks built on static declaration (Theano, TensorFlow, CNTK) fix a computation graph before any data arrives. This makes several cases awkward: variable-length sequences, tree- or graph-shaped inputs, elaborate inference procedures, and outputs whose shape depends on intermediate results. DyNet instead builds the graph implicitly while ordinary C++ or Python code runs. Each training example can therefore have its own architecture, and developers write control flow in the host language. Because a fresh graph is created for every example, construction cost must stay low, so the design relies on a heavily optimized C++ core and a compact graph representation. In single-machine benchmarks, DyNet matched or beat static toolkits in speed and clearly outpaced Chainer, another dynamic framework. Its implementations were also typically shorter than Theano or TensorFlow equivalents. This is especially valuable for NLP, where parsers, translators and taggers often have input-dependent structure.

Key points

  • DyNet is released under the Apache 2.0 license at github.com/clab/dynet, with documentation at dynet.readthedocs.io. It offers C++ and Python interfaces and was originally called cnn.
  • Implementation size was measured in non-comment characters (Table 6):
    • Language model: DyNet Python 3766, Chainer 3553, Theano 4257, TF 4922.
    • BiLSTM tagger: DyNet Python 4532, Theano 5581, TF 5137.
  • Sparse parameter updates pay off for CPU training with large embedding matrices but give little or no gain on GPU.
  • The toolkit underlies many NLP systems, including stack LSTM parsers (lstm-parser), bist-parser, RNNG, stack-lstm-ner, morphological inflection models, and the MT toolkits Lamtram and nmtkit.
  • Current limitations and plans:
    • DyNet runs on a single CPU or GPU without model parallelism.
    • Planned work includes multi-device execution and automatic on-the-fly merging of similarly shaped operations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

183 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 166 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1701.03980 · arXiv 1701.03980

@misc{neubig2017dynet,
  title = {{DyNet: The Dynamic Neural Network Toolkit}},
  author = {Neubig, Graham and Dyer, Chris and Goldberg, Yoav and Matthews, Austin and Ammar, Waleed and Anastasopoulos, Antonios and Ballesteros, Miguel and Chiang, David and Clothiaux, Daniel and Cohn, Trevor and Duh, Kevin and Faruqui, Manaal and Gan, Cynthia and Garrette, Dan and Ji, Yangfeng and Kong, Lingpeng and Kuncoro, Adhiguna and Kumar, Gaurav and Malaviya, Chaitanya and Michel, Paul and Oda, Yusuke and Richardson, Matthew and Saphra, Naomi and Swayamdipta, Swabha and Yin, Pengcheng},
  year = {2017},
  doi = {10.48550/arxiv.1701.03980},
  eprint = {1701.03980},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1701.03980}
}

References

65 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016. find our card
  2. Roee Aharoni, Yoav Goldberg, and Yonatan Belinkov. Improving sequence to sequence learning for morphological inflection generation: The BIU-MIT systems for the SIGMORPHON 2016 shared task for morphological reinflection. pages 41–48, 2016. find
  3. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1545–1554, 2016. find our card
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015. find our card
  5. Miguel Ballesteros and Leo Wanner. A neural network architecture for multilingual punctuation generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1048–1053, 2016. find
  6. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3(Feb):1137–1155, 2003. find
  7. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: A CPU and GPU math compiler in Python. In Proc. 9th Python in Science Conf, pages 1–7, 2010. find
  8. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. In Annual Conference of the Association for Computational Linguistics (ACL), pages 1466–1477, 2016. find
  9. Jacob Buckman, Miguel Ballesteros, and Chris Dyer. Transition-based dependency parsing with heuristic backtracking. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2313–2318, 2016. find
  10. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015. find our card
  11. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  12. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. Incorporating structural alignment biases into an attentional neural translation model. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 87… find
  13. Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. Technical report, Idiap, 2002. find
  14. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537, 2011. find our card
  15. Trinh-Minh-Tri Do and Thierry Artières. Neural conditional random fields. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010. find
  16. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011. find
  17. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. Transition-based dependency parsing with stack long short-term memory. In Annual Conference of the Association for Computational Linguistics (ACL), pages 334–343, 2015. find our card
  18. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 199–209, 2016. find
  19. Jason Eisner, Eric Goldlust, and Noah A. Smith. Compiling comp ling: Practical weighted dynamic programming and the Dyna language. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2005. find
  20. Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990. find
  21. Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. Morphological inflection generation using character sequence to sequence learning. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 634–643, 2016. find
  22. Jessica Ficler and Yoav Goldberg. A neural network for coordination boundary prediction. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 23–32, 2016. find
  23. Hila Gonen and Yoav Goldberg. Semi supervised preposition-sense disambiguation using multilingual data. In International Conference on Computational Linguistics (COLING), pages 2718– 2729, 2016. find
  24. Joshua Goodman. Classes for fast maximum entropy training. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), volume 1, pages 561–564. IEEE, 2001. find
  25. Matthew R. Gormley, Mark Dredze, and Jason Eisner. Approximation-aware dependency parsing by belief propagation. Transactions of the Association for Computational Linguistics, 3, 2015. find
  26. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In International Conference on Machine Learning (ICML), 2006. find
  27. Andreas Griewank. Automatic differentiation of algorithms: theory, implementation, and application. In proceedings of the first SIAM Workshop on Automatic Differentiation, 1991. find
  28. Gaël Guennebaud, Benoı̂t Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. ↗
  29. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine… find
  30. Sepp Hochreiter and Jürgen Schmidhuber. 9(8):1735–1780, 1997. find
  31. Robin J. Hogan. Fast reverse-mode automatic differentiation using expression templates in C++. ACM Transactions on Mathematical Software (TOMS), 40(4):26, 2014. find
  32. Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015. find our card
  33. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. A neural network for factoid question answering over paragraphs. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 633–644, 2014. find
  34. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  35. Eliyahu Kiperwasser and Yoav Goldberg. Easy-first dependency parsing with hierarchical tree LSTMs. Transactions of the Association for Computational Linguistics, 4:445–461, 2016. find
  36. Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327, 2016. find our card
  37. Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. Improving sentence compression by learning to predict gaze. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1528–1533, 2016. find
  38. Lingpeng Kong, Chris Dyer, and Noah A. Smith. Segmental recurrent neural networks. In International Conference on Learning Representations (ICLR), 2016. find
  39. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Neural Information Processing Systems (NIPS), pages 1097–1105, 2012. find
  40. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 260–270, 2016. find
  41. Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, and Shuicheng Yan. Semantic object parsing with graph lstm. 2016. find
  42. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv preprint arXiv:1511.04586, 2015. find our card
  43. Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. Deep learning with dynamic computation graphs. In Submitted to International Conference on Learning Representations (ICLR), 2017. find
  44. Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 5528–5531. IEEE, 2011. find
  45. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. find
  46. Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pages 372–376, 1983. Long short-term memory. Neural computation, find
  47. Graham Neubig and Chris Dyer. Generalizing and hybridizing count-based and neural language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1163–1172, 2016. find
  48. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. Learning multilingual named entity recognition from Wikipedia. Artificial Intelligence, 194:151–175, 2012. find
  49. Barbara Plank, Anders Søgaard, and Yoav Goldberg. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Annual Conference of the Association for Computational Linguistics (ACL), pages 412–418, 2016. find our card
  50. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Neural Information Processing Systems (NIPS), pages 693–701, 2011. find
  51. Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal estimated sub-gradient solver for SVM. Mathematical programming, 127(1):3–30, 2011. find
  52. Vered Shwartz and Ido Dagan. Cogalex-v shared task: Lexnet - integrated path-based and distributional method for the identification of semantic relations. In Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex - V), pages 80–85, 2016. find
  53. Vered Shwartz, Yoav Goldberg, and Ido Dagan. Improving hypernymy detection with an integrated path-based and distributional method. In Annual Conference of the Association for Computational Linguistics (ACL), pages 2389–2398, 2016. find our card
  54. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. find
  55. Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural language with recursive neural networks. In International Conference on Machine Learning (ICML), pages 129–136, 2011. find
  56. Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, 2016. find our card
  57. Xu Sun. Asynchronous parallel learning for neural networks and structured models with dense features. In International Conference on Computational Linguistics (COLING), pages 192–202, 2016. find
  58. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In Annual Conference of the International Speech Communication Association (Inter-Speech), 2012. find
  59. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In Neural Information Processing Systems (NIPS), pages 3104–3112, 2014. find
  60. Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. Greedy, joint syntactic-semantic parsing with stack lstms. In Conference on Natural Language Learning (CoNLL), pages 187–197, 2016. find
  61. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In Annual Conference of the Association for Computational Linguistics (ACL), 2015. find
  62. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. find
  63. R.E. Wengert. A simple automatic derivative evaluation program. Communications of the ACM, 7(8):463–464, 1964. find
  64. Dong Yu, Adam Eversole, Mike Seltzer, Kaisheng Yao, Oleksii Kuchaiev, Yu Zhang, Frank Seide, Zhiheng Huang, Brian Guenter, Huaming Wang, Jasha Droppo, Geoffrey Zweig, Chris Rossbach, Jie Gao, Andreas Stolcke, Jon Currey, Malcolm Slaney, Guoguo Chen, Amit Agarwal, Chris Basoglu, Marko Padmilac, Alexe… find
  65. Heiga Zen, Andrew Senior, and Mike Schuster. Statistical parametric speech synthesis using deep neural networks. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 7962–7966. IEEE, 2013. find