Elamur Research⌕
☰
home / papers / 2015 / A Primer on Neural Network Models for Natural Language Processing

A Primer on Neural Network Models for Natural Language Processing

Paper · 2015 · Yoav Goldberg

Summary

For years, NLP relied on linear classifiers such as SVMs and logistic regression applied to huge, sparse feature vectors. Many researchers lacked the background to adopt the non-linear neural models that were starting to outperform these classifiers. Goldberg's primer addresses this gap. It is a self-contained tutorial rather than an empirical study, so it introduces no new dataset, corpus or benchmark. Using one consistent notation, it first explains how to replace one-hot feature dimensions with trainable dense embeddings. It then covers fully connected feed-forward networks, convolution and pooling layers, recurrent networks, and recursive (tree-structured) networks. It also presents the computation-graph abstraction that makes gradient computation automatic. Each architecture is tied to NLP tasks where it has reportedly helped, including parsing, tagging, sentiment analysis and question answering. The main contribution is teaching rather than results. It gives practitioners a practical map for applying existing neural tools to their own problems and is intended as an entry point to neural NLP.

Key points

  • Covered: input encoding, MLPs, CNNs, RNNs (including a stack extension), RecNNs and computation graphs.

  • Deliberately excluded: in-depth treatment of language and acoustic modeling, machine translation, multimodal work, attention, large-vocabulary training tricks and autoencoders.

  • Core pipeline:

    • extract core linguistic features;
    • look up an embedding vector for each;
    • concatenate and/or sum the vectors;
    • feed the result to a non-linear classifier, training the embeddings jointly with the network.

  • Cited evidence:

    • Swapping a parser's linear model for a feed-forward net improved syntactic parsing (Chen & Manning, 2014; Weiss et al., 2015; Pei et al., 2015; Durrett & Klein, 2015).
    • CNNs worked well for document classification (Johnson & Zhang, 2015) and sentiment (Kim, 2014).

  • Recursive network variants discussed:

    • label embeddings in the composition function (Qian et al., 2015);
    • per-symbol-pair matrices (Socher et al., 2013);
    • the matrix-vector model and the recursive neural tensor network;
    • Tree-LSTMs (Tai, Socher & Manning, 2015).

    All are trained with backpropagation (historically called BPTS) and SGD.

  • Terminology and further reading: "feature" means a linguistic input such as a word or POS tag, as opposed to an input-vector entry. The Bengio et al. (2015) book is recommended for deeper theory.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bar-Ilan UniversityRamat Gan · Israel146 works · 24,845 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yoav Goldberg. 2015. A Primer on Neural Network Models for Natural Language Processing. arXiv:1510.00726.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1510.00726 · arXiv 1510.00726

@misc{goldberg2015primer,
  title = {{A Primer on Neural Network Models for Natural Language Processing}},
  author = {Yoav Goldberg},
  year = {2015},
  doi = {10.48550/arxiv.1510.00726},
  eprint = {1510.00726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1510.00726}
}

References

150 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adel, H., Vu, N. T., & Schultz, T. (2013). Combination of Recurrent Neural Networks and Factored Language Models for Code-Switching Language Modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 206–211, Sofia, Bulgaria. As… find
  2. Ando, R., & Zhang, T. (2005a). A High-Performance Semi-Supervised Learning Method for Text Chunking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pp. 1–9, Ann Arbor, Michigan. Association for Computational Linguistics. find
  3. Ando, R. K., & Zhang, T. (2005b). A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6, 1817–1853. find
  4. Auli, M., Galley, M., Quirk, C., & Zweig, G. (2013). Joint Language and Translation Modeling with Recurrent Neural Networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1044–1054, Seattle, Washington, USA. Association for Computational Linguistics. find
  5. Auli, M., & Gao, J. (2014). Decoder Integration and Expected BLEU Training for Recurrent Neural Network Language Models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 136–142, Baltimore, Maryland. Association for Computationa… find
  6. Ballesteros, M., Dyer, C., & Smith, N. A. (2015). Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 349–359, Lisbon, Portugal. Association for Computational Linguistics.
  7. Bansal, M., Gimpel, K., & Livescu, K. (2014). Tailoring Continuous Word Representations for Dependency Parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 809–815, Baltimore, Maryland. Association for Computational Linguis…
  8. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv:1502.05767 [cs]. — [arXiv:1502.05767](https://arxiv.org/abs/1502.05767) find
  9. Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. arXiv:1206.5533 [cs]. find
  10. Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003). A Neural Probabilistic Language Model. J. Mach. Learn. Res., 3, 1137–1155.
  11. Bengio, Y., Goodfellow, I. J., & Courville, A. (2015). Deep Learning. Book in preparation for MIT Press. find
  12. Bitvai, Z., & Cohn, T. (2015). Non-Linear Text Regression with a Deep Convolutional Neural Network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 180–1… find
  13. Botha, J. A., & Blunsom, P. (2014). Compositional Morphology for Word Representations and Language Modelling. In Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China. *Award for best application paper*. find
  14. Bottou, L. (2012). Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade, pp. 421–436. Springer. find
  15. Charniak, E., & Johnson, M. (2005). Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pp. 173–180, Ann Arbor, Michigan. Association for Computational Linguistics.
  16. Chen, D., & Manning, C. (2014). A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 740–750, Doha, Qatar. Association for Computational Linguistics.
  17. Chen, Y., Xu, L., Liu, K., Zeng, D., & Zhao, J. (2015). Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volu…
  18. Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014a). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111, Doha, Qatar. Association for Computatio…
  19. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014b). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),…
  20. Chrupala, G. (2014). Normalizing tweets with edit scripts and recurrent neural embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 680–686, Baltimore, Maryland. Association for Computational Linguistics. find
  21. Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs]. find
  22. Collins, M. (2002). Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, pp. 1–8. Association for Computational Linguistics.
  23. Collins, M., & Koo, T. (2005). Discriminative Reranking for Natural Language Parsing. Computational Linguistics, 31 (1), 25–70.
  24. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160–167. ACM. find
  25. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12, 2493–2537.
  26. Crammer, K., & Singer, Y. (2002). On the algorithmic implementation of multiclass kernel-based vector machines. The Journal of Machine Learning Research, 2, 265–292. find
  27. Creutz, M., & Lagus, K. (2007). Unsupervised Models for Morpheme Segmentation and Morphology Learning. ACM Trans. Speech Lang. Process., 4 (1), 3:1–3:34. find
  28. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2 (4), 303–314. find
  29. Dahl, G., Sainath, T., & Hinton, G. (2013). Improving deep neural networks for LVCSR using rectified linear units and dropout. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8609–8613. de Gispert, A., Iglesias, G., & Byrne, B. (2015). Fast and Accurate… find
  30. Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., & Xu, K. (2014). Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 49–54, Baltimore, Marylan…
  31. Dong, L., Wei, F., Zhou, M., & Xu, K. (2015). Question Answering over Freebase with Multi-Column Convolutional Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume… find
  32. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12, 2121– 2159. find
  33. Duh, K., Neubig, G., Sudoh, K., & Tsukada, H. (2013). Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 678–683, Sofia, Bulgaria. Associa… find
  34. Durrett, G., & Klein, D. (2015). Neural CRF Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 302– 312, Beijing, China. Association for Computation… find
  35. Dyer, C., Ballesteros, M., Ling, W., Matthews, A., & Smith, N. A. (2015). Transition-Based Dependency Parsing with Stack Long Short-Term Memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P…
  36. Elman, J. L. (1990). Finding Structure in Time. Cognitive Science, 14 (2), 179–211. find
  37. Faruqui, M., & Dyer, C. (2014). Improving Vector Space Word Representations Using Multilingual Correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 462–471, Gothenburg, Sweden. Association for Computational Linguistics.
  38. Filippova, K., Alfonseca, E., Colmenares, C. A., Kaiser, L., & Vinyals, O. (2015). Sentence Compression by Deletion with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 360–368, Lisbon, Portugal. Association for Computational Linguistics.
  39. Gal, Y., & Ghahramani, Z. (2015). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv:1506.02142 [cs, stat]. find
  40. Gao, J., Pantel, P., Gamon, M., He, X., & Deng, L. (2014). Modeling Interestingness with Deep Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2–13, Doha, Qatar. Association for Computational Linguistics. find
  41. Giménez, J., & Màrquez, L. (2004). SVMTool: A general POS tagger generator based on Support Vector Machines. In Proceedings of the 4th LREC, Lisbon, Portugal.
  42. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, pp. 249–256. find
  43. Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 315–323. find
  44. Goldberg, Y., & Elhadad, M. (2010). An Efficient Algorithm for Easy-First Non-Directional Dependency Parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 742–750, Los Angeles, California. Association f… find
  45. Goldberg, Y., & Levy, O. (2014). word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv:1402.3722 [cs, stat]. find
  46. Goldberg, Y., & Nivre, J. (2013). Training Deterministic Parsers with Non-Deterministic Oracles. Transactions of the Association for Computational Linguistics, 1 (0), 403– 414. find
  47. Goldberg, Y., Zhao, K., & Huang, L. (2013). Efficient Implementation of Beam-Search Incremental Parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 628–633, Sofia, Bulgaria. Association for Computational Linguistics. find
  48. Goller, C., & Küchler, A. (1996). Learning Task-Dependent Distributed Representations by Backpropagation Through Structure. In In Proc. of the ICNN-96, pp. 347–352. IEEE. find
  49. Graves, A. (2008). Supervised sequence labelling with recurrent neural networks. Ph.D. thesis, Technische Universität München. find
  50. Greff, K., Srivastava, R. K., Koutnı́k, J., Steunebrink, B. R., & Schmidhuber, J. (2015). LSTM: A Search Space Odyssey. arXiv:1503.04069 [cs].
  51. Hal Daumé III, Langford, J., & Marcu, D. (2009). Search-based Structured Prediction. Machine Learning Journal (MLJ). find
  52. Harris, Z. (1954). Distributional Structure. Word, 10 (23), 146–162. find
  53. Hashimoto, K., Miwa, M., Tsuruoka, Y., & Chikayama, T. (2013). Simple Customization of Recursive Neural Networks for Semantic Relation Classification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1372–1376, Seattle, Washington, USA. Association for C… find
  54. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:1502.01852 [cs]. find
  55. Henderson, M., Thomson, B., & Young, S. (2013). Deep Neural Network Approach for the Dialog State Tracking Challenge. In Proceedings of the SIGDIAL 2013 Conference, pp. 467–471, Metz, France. Association for Computational Linguistics. find
  56. Hermann, K. M., & Blunsom, P. (2013). The Role of Syntax in Vector Space Models of Compositional Semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 894–904, Sofia, Bulgaria. Association for Computational Linguistics. find
  57. Hermann, K. M., & Blunsom, P. (2014). Multilingual Models for Compositional Distributed Semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 58–68, Baltimore, Maryland. Association for Computational Linguistics.
  58. Hihi, S. E., & Bengio, Y. (1996). Hierarchical Recurrent Neural Networks for Long-Term Dependencies. In Touretzky, D. S., Mozer, M. C., & Hasselmo, M. E. (Eds.), Advances in Neural Information Processing Systems 8, pp. 493–499. MIT Press. find
  59. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580 [cs]. find
  60. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9 (8), 1735–1780. find
  61. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2 (5), 359–366. find
  62. Irsoy, O., & Cardie, C. (2014). Opinion Mining with Deep Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 720–728, Doha, Qatar. Association for Computational Linguistics. find
  63. Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., & Daumé III, H. (2014a). A Neural Network for Factoid Question Answering over Paragraphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 633–644, Doha, Qatar. Association for Computational… find
  64. Iyyer, M., Enns, P., Boyd-Graber, J., & Resnik, P. (2014b). Political Ideology Detection Using Recursive Neural Networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1113–1122, Baltimore, Maryland. Association for Computatio…
  65. Iyyer, M., Manjunatha, V., Boyd-Graber, J., & Daumé III, H. (2015). Deep Unordered Composition Rivals Syntactic Methods for Text Classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language…
  66. Johnson, R., & Zhang, T. (2014). Effective Use of Word Order for Text Categorization with Convolutional Neural Networks. arXiv:1412.1058 [cs, stat].
  67. Johnson, R., & Zhang, T. (2015). Effective Use of Word Order for Text Categorization with Convolutional Neural Networks. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 103–112, Denver, Colorado. A…
  68. Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 2342–2350. find
  69. Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014). A Convolutional Neural Network for Modelling Sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 655–665, Baltimore, Maryland. Association for Computational Lingui…
  70. Karpathy, A., Johnson, J., & Li, F.-F. (2015). Visualizing and Understanding Recurrent Networks. arXiv:1506.02078 [cs].
  71. Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  72. Kingma, D., & Ba, J. (2014). arXiv:1412.6980 [cs].
  73. Adam: A Method for Stochastic Optimization.
  74. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Pereira, F., Burges, C. J. C., Bottou, L., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc.
  75. Kudo, T., & Matsumoto, Y. (2003). Fast Methods for Kernel-based Text Analysis. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pp. 24–31, Stroudsburg, PA, USA. Association for Computational Linguistics.
  76. Le, P., & Zuidema, W. (2014). The Inside-Outside Recursive Neural Network model for Dependency Parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 729–739, Doha, Qatar. Association for Computational Linguistics. find
  77. Le, P., & Zuidema, W. (2015). The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1155–1164, Lisbon, Portugal. Association for Computation… find
  78. Le, Q. V., Jaitly, N., & Hinton, G. E. (2015). A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. arXiv:1504.00941 [cs]. find
  79. LeCun, Y., & Bengio, Y. (1995). Convolutional Networks for Images, Speech, and Time-Series. In Arbib, M. A. (Ed.), The Handbook of Brain Theory and Neural Networks. MIT Press. find
  80. LeCun, Y., Bottou, L., Orr, G., & Muller, K. (1998a). Efficient BackProp. In Orr, G., & K, M. (Eds.), Neural Networks: Tricks of the trade. Springer. find
  81. Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998b). Gradient Based Learning Applied to Pattern Recognition.. find
  82. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A tutorial on energy-based learning. Predicting structured data, 1, 0. find
  83. LeCun, Y., & Huang, F. (2005). Loss functions for discriminative training of energybased models.. AIStats. find
  84. Levy, O., & Goldberg, Y. (2014a). Dependency-Based Word Embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 302–308, Baltimore, Maryland. Association for Computational Linguistics. find
  85. Levy, O., & Goldberg, Y. (2014b). Neural Word Embedding as Implicit Matrix Factorization. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 27, pp. 2177–2185. Curran Associates, Inc. find
  86. Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics, 3 (0), 211–225.
  87. Lewis, M., & Steedman, M. (2014). Improved CCG Parsing with Semi-supervised Supertagging. Transactions of the Association for Computational Linguistics, 2 (0), 327–338.
  88. Li, J., Li, R., & Hovy, E. (2014). Recursive Deep Models for Discourse Parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2061–2069, Doha, Qatar. Association for Computational Linguistics.
  89. Ling, W., Dyer, C., Black, A. W., & Trancoso, I. (2015a). Two/Too Simple Adaptations of Word2Vec for Syntax Problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1299–1304, Denver, Colorado. As…
  90. Ling, W., Dyer, C., Black, A. W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., & Luis, T. (2015b). Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 15…
  91. Liu, Y., Wei, F., Li, S., Ji, H., Zhou, M., & WANG, H. (2015). A Dependency-Based Neural Network for Relation Classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2… find
  92. Ma, J., Zhang, Y., & Zhu, J. (2014). Tagging The Web: Building A Robust Web Tagger with Neural Network. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 144–154, Baltimore, Maryland. Association for Computational Linguistics. find
  93. Ma, M., Huang, L., Zhou, B., & Xiang, B. (2015). Dependency-based Convolutional Neural Networks for Sentence Embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Shor… find
  94. McCallum, A., Freitag, D., & Pereira, F. C. (2000). Maximum Entropy Markov Models for Information Extraction and Segmentation.. In ICML, Vol. 17, pp. 591–598. find
  95. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs].
  96. Mikolov, T., Joulin, A., Chopra, S., Mathieu, M., & Ranzato, M. (2014). Learning Longer Memory in Recurrent Neural Networks. arXiv:1412.7753 [cs]. — [arXiv:1412.7753](https://arxiv.org/abs/1412.7753) find
  97. Mikolov, T., Karafiát, M., Burget, L., Cernocky, J., & Khudanpur, S. (2010). Recurrent neural network based language model.. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pp. 1045–1048. find
  98. Mikolov, T., Kombrink, S., Lukáš Burget, Černocky, J. H., & Khudanpur, S. (2011). Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pp. 5528–5531. IEEE. find
  99. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 26, pp. 31…
  100. Mikolov, T. (2012). Statistical language models based on neural networks. Ph.D. thesis, Ph. D. thesis, Brno University of Technology. find
  101. Mnih, A., & Kavukcuoglu, K. (2013). Learning word embeddings efficiently with noise-contrastive estimation. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 26, pp. 2265–2273. Curran Associates, Inc. find
  102. Mrkšić, N., Ó Séaghdha, D., Thomson, B., Gasic, M., Su, P.-H., Vandyke, D., Wen, T.-H., & Young, S. (2015). Multi-domain Dialog State Tracking using Recurrent Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joi… find
  103. Neidinger, R. (2010). Introduction to Automatic Differentiation and MATLAB Object-Oriented Programming. SIAM Review, 52 (3), 545–563. find
  104. Nguyen, T. H., & Grishman, R. (2015). Event Detection and Domain Adaptation with Convolutional Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers…
  105. Nivre, J. (2008). Algorithms for Deterministic Incremental Dependency Parsing. Computational Linguistics, 34 (4), 513–553.
  106. Okasaki, C. (1999). Purely Functional Data Structures. Cambridge University Press, Cambridge, U.K.; New York. find
  107. Pascanu, R., Mikolov, T., & Bengio, Y. (2012). On the difficulty of training Recurrent Neural Networks. arXiv:1211.5063 [cs]. find
  108. Pei, W., Ge, T., & Chang, B. (2015). An Effective Neural Network Model for Graph-based Dependency Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.… find
  109. Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  110. Pollack, J. B. (1990). Recursive Distributed Representations. Artificial Intelligence, 46, 77–105. find
  111. Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4 (5), 1 – 17. find
  112. Qian, Q., Tian, B., Huang, M., Liu, Y., Zhu, X., & Zhu, X. (2015). Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natu… find
  113. Rong, X. (2014). word2vec Parameter Learning Explained. arXiv:1411.2738 [cs]. find
  114. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323 (6088), 533–536. find
  115. Santos, C. D., & Zadrozny, B. (2014). Learning Character-level Representations for Part-of-Speech Tagging.. pp. 1818–1826. find
  116. Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45 (11), 2673–2681. find
  117. Shawe-Taylor, J., & Cristianini, N. (2004). Kernel Methods for Pattern Analysis. Cambridge University Press. find
  118. Smith, N. A. (2011). Linguistic Structure Prediction. Synthesis Lectures on Human Language Technologies. Morgan and Claypool. find
  119. Socher, R. (2014). Recursive Deep Learning For Natural Language Processing and Computer Vision. Ph.D. thesis, Stanford University. find
  120. Socher, R., Bauer, J., Manning, C. D., & Andrew Y., N. (2013). Parsing with Compositional Vector Grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 455–465, Sofia, Bulgaria. Association for Computational Linguistics. find
  121. Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012). Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1201–1211, Jeju Island, Korea.…
  122. Socher, R., Lin, C. C.-Y., Ng, A. Y., & Manning, C. D. (2011). Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Getoor, L., & Scheffer, T. (Eds.), Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July… find
  123. Socher, R., Manning, C., & Ng, A. (2010). Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks. In Proceedings of the Deep Learning and Unsupervised Feature Learning Workshop of {NIPS} 2010, pp. 1–9. find
  124. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA…
  125. Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.-Y., Gao, J., & Dolan, B. (2015). A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computatio…
  126. Sundermeyer, M., Alkhouli, T., Wuebker, J., & Ney, H. (2014). Translation Modeling with Bidirectional Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 14–25, Doha, Qatar. Association for Computational Linguistics. find
  127. Sundermeyer, M., Schlüter, R., & Ney, H. (2012). LSTM Neural Networks for Language Modeling.. In INTERSPEECH. find
  128. Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In Proceedings of the 30th international conference on machine learning (ICML-13), pp. 1139–1147. find
  129. Sutskever, I., Martens, J., & Hinton, G. E. (2011). Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024. find
  130. Sutskever, I., Vinyals, O., & Le, Q. V. V. (2014). Sequence to Sequence Learning with Neural Networks. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 27, pp. 3104–3112. Curran Associates, Inc.
  131. Tai, K. S., Socher, R., & Manning, C. D. (2015). Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processin… find
  132. Tamura, A., Watanabe, T., & Sumita, E. (2014). Recurrent Neural Networks for Word Alignment Model. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1470–1480, Baltimore, Maryland. Association for Computational Linguistics. find
  133. Tieleman, T., & Hinton, G. (2012). Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. find
  134. Van de Cruys, T. (2014). A Neural Network Approach to Selectional Preference Acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 26–35, Doha, Qatar. Association for Computational Linguistics.
  135. Vaswani, A., Zhao, Y., Fossum, V., & Chiang, D. (2013). Decoding with Large-Scale Neural Language Models Improves Translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1387–1392, Seattle, Washington, USA. Association for Computational Linguistics…
  136. Wager, S., Wang, S., & Liang, P. S. (2013). Dropout Training as Adaptive Regularization. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., & Weinberger, K. Q. (Eds.), Advances in Neural Information Processing Systems 26, pp. 351–359. Curran Associates, Inc. find
  137. Wang, P., Xu, J., Xu, B., Liu, C., Zhang, H., Wang, F., & Hao, H. (2015a). Semantic Clustering and Convolutional Neural Network for Short Text Categorization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natu… find
  138. Wang, X., Liu, Y., SUN, C., Wang, B., & Wang, X. (2015b). Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Languag… find
  139. Watanabe, T., & Sumita, E. (2015). Transition-based Neural Constituent Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1169–1179, Beijing, China.… find
  140. Weiss, D., Alberti, C., Collins, M., & Petrov, S. (2015). Structured Training for Neural Network Transition-Based Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: L… find
  141. Werbos, P. J. (1990). Backpropagation through time: What it does and how to do it.. Proceedings of the IEEE, 78 (10), 1550 – 1560. find
  142. Weston, J., Bordes, A., Yakhnenko, O., & Usunier, N. (2013). Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1366–1371, Seattle, Washington, USA. Association for Comp… find
  143. Xu, W., Auli, M., & Clark, S. (2015). CCG Supertagging with a Recurrent Neural Network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 250–255, Beijing,… find
  144. Yin, W., & Schütze, H. (2015). Convolutional Neural Network for Paraphrase Identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 901–911, Denver, Colorado. Association for Computational… find
  145. Zaremba, W., Sutskever, I., & Vinyals, O. (2014). Recurrent Neural Network Regularization. arXiv:1409.2329 [cs].
  146. Zeiler, M. D. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv:1212.5701 [cs].
  147. Zeng, D., Liu, K., Lai, S., Zhou, G., & Zhao, J. (2014). Relation Classification via Convolutional Deep Neural Network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 2335–2344, Dublin, Ireland. Dublin City University and Associat…
  148. Zhou, H., Zhang, Y., Huang, S., & Chen, J. (2015). A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pro… find
  149. Zhu, C., Qiu, X., Chen, X., & Huang, X. (2015a). A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing… find
  150. Zhu, X., Sobhani, P., & Guo, H. (2015b). Long Short-Term Memory Over Tree Structures. arXiv:1503.04881 [cs]. — [arXiv:1503.04881](https://arxiv.org/abs/1503.04881) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goldberg-2015-primer-neural-network-models-natural/