Elamur Research
☰
home / papers / 2011 / Natural Language Processing (almost) from Scratch

Natural Language Processing (almost) from Scratch

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1103.0398

Summary

In the late 2000s, most strong NLP systems paired linear models with hand-designed, task-specific features, often built from the outputs of other tools. Collobert and colleagues test whether a single neural architecture can handle four classic tagging tasks with little linguistic engineering: POS tagging, chunking, NER and SRL. Their network maps words to vectors through a lookup table. It then applies a window-based or convolutional design, using max pooling and HardTanh layers. Training uses backpropagation with either a word-level or a sentence-level log-likelihood. To make up for ignoring prior knowledge, the authors first train a language model on roughly 852 million mostly unlabeled words using a ranking objective. They then transfer the learned word representations into the supervised taggers. They also study multitask training and the effect of adding a few standard engineered features. The approach performs well on most benchmarks, and it forms the basis of a freely released tagger that is accurate, fast and light on resources. The work matters because it showed that representations learned from raw text can replace much manual feature design, a central idea in later embedding-based NLP.

Key points

  • Benchmarks: POS on WSJ following the Toutanova et al. (2003) setup (sections 0–18 for training, 22–24 for testing, 45 tags); chunking on CoNLL 2000; NER on CoNLL 2003 (Reuters); SRL on CoNLL 2005 (WSJ section 23 plus 3 Brown corpus sections for testing).
  • Reference state-of-the-art results: POS 97.33% accuracy (Shen et al., 2007), CHUNK 95.23% F1 (Shen and Sarkar, 2005), NER 89.31% F1 (Ando and Zhang, 2005) and SRL 77.92% F1 (Koomen et al., 2005).
  • Unsupervised pretraining uses about 852M words. The language model is trained with a hinge-style ranking loss, max(0, 1 − f(x) + f(x^(w))), which scores a real text window above a corrupted version.
  • The sentence-level log-likelihood adds learned tag-transition scores. Its normalization term is computed with a forward recursion that uses logadd.
  • The model avoids heavy dependencies on external pipelines, such as parse trees for SRL or gazetteers for NER, which competing systems typically rely on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition · Semantic role labeling · Part-of-speech tagging

Cited by

1137 papers of our library cite it (of the 140673 whose bibliographies we read); 69 of them have a page here, 1068 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1103.0398 · arXiv 1103.0398

@misc{collobert2011natural,
  title = {{Natural Language Processing (almost) from Scratch}},
  author = {Collobert, Ronan and Weston, Jason and Bottou, Leon and Karlen, Michael and Kavukcuoglu, Koray and Kuksa, Pavel},
  year = {2011},
  doi = {10.48550/arxiv.1103.0398},
  eprint = {1103.0398},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1103.0398}
}

References

85 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6:1817–1953, 11 2005. find
  2. R. M. Bell, Y. Koren, and C. Volinsky. The BellKor solution to the Netflix Prize. Technical report, AT&T Labs, 2007. http://www.research.att.com/~volinsky/netflix . ↗
  3. Y. Bengio and R. Ducharme. A neural probabilistic language model. In NIPS 13, 2001. find
  4. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, NIPS 19, 2007. find
  5. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In International Conference on Machine Learning, ICML, 2009. find our card
  6. L. Bottou. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nı̂mes 91, Nimes, France, 1991. EC2. find
  7. L. Bottou. Online algorithms and stochastic approximations. In David Saad, editor, Online Learning and Neural Networks. Cambridge University Press, Cambridge, UK, 1998. find
  8. L. Bottou and P. Gallinari. A framework for the cooperation of learning algorithms. In D. Touretzky and R. Lippmann, editors, Advances in Neural Information Processing Systems, volume 3. Morgan Kaufmann, Denver, 1991. find
  9. L. Bottou, Y. LeCun, and Yoshua Bengio. Global training of document processing systems using graph transformer networks. In Proc. of Computer Vision and Pattern Recognition, pages 489–493, Puerto-Rico, 1997. IEEE. find
  10. ∂C logadd ∂δ t (i) = ∂C logadd e δ t−1 (i)+[A] i, j P δ t−1 (k)+[A] . k, j ∂δ t (j) k e = J. S. Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In F. Fogelman Soulié and J. Hérault, editors, Neurocomputing:… find
  11. P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. D. Pietra, and J C. Lai. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479, 1992a. find
  12. P. F. Brown, V. J. Della Pietra, R. L. Mercer, S. A. Della Pietra, and J. C. Lai. An estimate of an upper bound for the entropy of english. Computational Linguistics, 18(1):31–41, 1992b. find our card
  13. C. J. C. Burges, R. Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 193–200. MIT Press, Cambridge, MA, 2007. ar Xi v R. Caruana. Multitask Learning. Machine Learn… find
  14. O. Chapelle, B. Schlkopf, and A. Zien. Semi-Supervised Learning. Adaptive computation and machine learning. MIT Press, Cambridge, Mass., USA, 09 2006. find
  15. E. Charniak. A maximum-entropy-inspired parser. Proceedings of the first conference on North American chapter of the Association for Computational Linguistics, pages 132–139, 2000. find
  16. H. L. Chieu. Named entity recognition with a maximum entropy approach. In In Proceedings of the Seventh Conference on Natural Language Learning (CoNLL-2003, pages 160–163, 2003. find
  17. N. Chomsky. Three models for the description of language. Information Theory, 2(3):113–124, September 1956. find
  18. S. Clémençon and N. Vayatis. Ranking the best instances. Journal of Machine Learning Research, 8:2671–2699, December 2007. find
  19. W. W. Cohen, R. E. Schapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243–270, 1998. find
  20. T. Cohn and P. Blunsom. Semantic role labelling with tree conditional random fields. In Ninth Conference on Computational Natural Language (CoNLL), 2005. find
  21. M. Collins. Head-Driven Statistical Models for Natural Language Parsing. PhD thesis, University of Pennsylvania, 1999. find
  22. R. Collobert. Large Scale Machine Learning. PhD thesis, Université Paris VI, 2004. find
  23. T. Cover and R. King. A convergent gambling estimate of the entropy of english. IEEE Transactions on Information Theory, 24(4):413–421, July 1978. find
  24. IRE Transactions on R. Florian, A. Ittycheriah, H. Jing, and T. Zhang. Named entity recognition through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, pages 168–171. Association for Computational Linguistics, 2003. find
  25. D. Gildea and D. Jurafsky. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288, 2002. find our card
  26. D. Gildea and M. Palmer. The necessity of parsing for predicate argument recognition. Proceedings of the 40th Annual Meeting of the ACL, pages 239–246, 2002. find
  27. J. Giménez and L. Màrquez. SVMTool: A general POS tagger generator based on support vector machines. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC’04), 2004. find
  28. A. Haghighi, K. Toutanova, and C. D. Manning. A joint model for semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005). Association for Computational Linguistics, June 2005. ar Xi v Z. S. Harris. Mathematical Structures of Language. find
  29. John Wiley & Sons Inc., 1968. find
  30. D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie. Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research, 1:49–75, 2001. ISSN 1532-4435. find
  31. G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural Comp., 18(7):1527–1554, July 2006. find
  32. K. Hollingshead, S. Fisher, and B. Roark. Comparing and combining finite-state and context-free parsers. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 787–794. Association for Computational Linguistics, 2005. find
  33. F. Huang and A. Yates. Distributional representations for handling sparsity in supervised sequence-labeling. In Proceedings of the Association for Computational Linguistics (ACL), pages 495–503. Association for Computational Linguistics, 2009. find
  34. F. Jelinek. Continuous speech recognition by statistical methods. Proceedings of the IEEE, 64(4):532–556, 1976. find
  35. T. Joachims. Transductive inference for text classification using support vector machines. In ICML, 1999. find
  36. D. Klein and C. D. Manning. Natural language grammar induction using a constituent-context model. In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 35–42. MIT Press, Cambridge, MA, 2002. find
  37. T. Koo, X. Carreras, and M. Collins. Simple semi-supervised dependency parsing. In Proceedings of the Association for Computational Linguistics (ACL), pages 595–603, 2008. find our card
  38. P. Koomen, V. Punyakanok, D. Roth, and W. Yih. Generalized inference with multiple semantic role labeling systems (shared task paper). In Ido Dagan and Dan Gildea, editors, Proc. of the Annual Conference on Computational Natural Language Learning (CoNLL), pages 181–184, 2005. find
  39. T. Kudo and Y. Matsumoto. Chunking with support vector machines. In In Proceedings of the 2nd Meeting of the North American Association for Computational Linguistics: NAACL 2001, pages 1–8. Association for Computational Linguistics, 2001. find
  40. T. Kudoh and Y. Matsumoto. Use of support vector learning for chunk identification. In Proceedings of CoNLL-2000 and LLL-2000, pages 142–144, 2000. find
  41. J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Eighteenth International Conference on Machine Learning, ICML, 2001. ar Xi v Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner. Gradient based learning applied to doc… find
  42. Y. LeCun. A learning scheme for asymmetric threshold networks. Cognitiva 85, pages 599–604, Paris, France, 1985. find
  43. Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In G.B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998. find
  44. D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361–397, 2004. find our card
  45. P. Liang. Semi-supervised learning for natural language. Master’s thesis, Massachusetts Institute of Technology, 2005. find
  46. P. Liang, H. Daumé, III, and D. Klein. Structure compilation: trading structure for features. In International conference on Machine learning (ICML), pages 592–599. ACM, 2008. find
  47. D. Lin and X. Wu. Phrase clustering for discriminative learning. In Proceedings of the Association for Computational Linguistics (ACL), pages 1030–1038. Association for Computational Linguistics, 2009. find
  48. N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. In Machine Learning, pages 285–318, 1988. find
  49. A. McCallum and Wei Li. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, pages 188–191. Association for Computational Linguistics, 2003. find
  50. D. McClosky, E. Charniak, and M. Johnson. Effective self-training for parsing. Proceedings of HLT-NAACL 2006, 2006. find our card
  51. In Proceedings of R. McDonald, K. Crammer, and F. Pereira. Flexible text segmentation with structured multilabel classification. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 987–994. Association for Computational L… find
  52. S. Miller, H. Fox, L. Ramshaw, and R. Weischedel. A novel use of statistical parsing to extract information from text. 6th Applied Natural Language Processing Conference, 2000. find
  53. S. Miller, J. Guinness, and A. Zamanian. Name tagging with word clusters and discriminative training. In Proceedings of HLT-NAACL, pages 337–342, 2004. find
  54. A Mnih and G. E. Hinton. Three new graphical models for statistical language modelling. In International Conference on Machine Learning, ICML, pages 641–648, 2007. find
  55. G. Musillo and P. Merlo. Robust Parsing of the Proposition Bank. ROMAND 2006: Robust Methods in Analysis of Natural language Data, 2006. ar Xi v R. M. Neal. Bayesian Learning for Neural Networks. Number 118 in Lecture Notes in Statistics. find
  56. Springer-Verlag, New York, 1996. find
  57. D. Okanohara and J. Tsujii. A discriminative language model with pseudo-negative samples. Proceedings of the 45th Annual Meeting of the ACL, pages 73–80, 2007. find
  58. M. Palmer, D. Gildea, and P. Kingsbury. The proposition bank: An annotated corpus of semantic roles. Comput. Linguist., 31(1):71–106, 2005. ISSN 0891-2017. find
  59. J. Pearl. Probabilistic Reasoning in Intelligent Systems. Morgan Kaufman, San Mateo, 1988. find
  60. D. C. Plaut and G. E. Hinton. Learning sets of filters using back-propagation. Computer Speech and Language, 2:35–61, 1987. find
  61. M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130–137, 1980. find
  62. S. Pradhan, W. Ward, K. Hacioglu, J. Martin, and D. Jurafsky. Shallow semantic parsing using support vector machines. Proceedings of HLT/NAACL-2004, 2004. find
  63. S. Pradhan, K. Hacioglu, W. Ward, J. H. Martin, and D. Jurafsky. Semantic role chunking combining complementary syntactic views. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 217–220. Association for Computational Linguistics, June 2005. find
  64. V. Punyakanok, D. Roth, and W. Yih. The necessity of syntactic parsing for semantic role labeling. In IJCAI, pages 1117–1123, 2005. find
  65. L. Ratinov and D. Roth. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL), pages 147–155. Association for Computational Linguistics, 2009. find our card
  66. A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Eric Brill and Kenneth Church, editors, Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 133–142. Association for Computational Linguistics, 1996. find
  67. B. Rosenfeld and R. Feldman. Using Corpus Statistics on Entities to Improve Semi-supervised Relation Extraction from the Web. Proceedings of the 45th Annual Meeting of the ACL, pages 600–607, 2007. find
  68. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by back-propagating errors. In D.E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 1, pages 318–362. MIT Press, 1986. find
  69. H. Schütze. Distributional part-of-speech tagging. In Proceedings of the Association for Computational Linguistics (ACL), pages 141–148. Morgan Kaufmann Publishers Inc., 1995. ar Xi v H. Schwenk and J. L. Gauvain. Connectionist language modeling for large vocabulary continuous speech recognition. I… find
  70. F. Sha and F. Pereira. Shallow parsing with conditional random fields. In NAACL ’03: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 134–141. Association for Computational Linguistics, 2003. find
  71. C. E. Shannon. Prediction and entropy of printed english. Bell Systems Technical Journal, 30:50–64, 1951. find
  72. H. Shen and A. Sarkar. Voting between multiple data representations for text chunking. Advances in Artificial Intelligence, pages 389–400, 2005. find
  73. L. Shen, G. Satta, and A. K. Joshi. Guided learning for bidirectional sequence classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), 2007. find
  74. N. A. Smith and J. Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 354–362. Association for Computational Linguistics, 2005. find
  75. S. C. Suddarth and A. D. C. Holden. Symbolic-neural systems and the use of hints for developing complex systems. International Journal of Man-Machine Studies, 35(3):291– 311, 1991. find
  76. X. Sun, L.-P. Morency, D. Okanohara, and J. Tsujii. Modeling latent-dynamic in shallow parsing: a latent conditional model with improved inference. In COLING ’08: Proceedings of the 22nd International Conference on Computational Linguistics, pages 841–848. Association for Computational Linguistics,… find
  77. C. Sutton and A. McCallum. Joint parsing and semantic role labeling. In Proceedings of CoNLL-2005, pages 225–228, 2005a. find
  78. C. Sutton and A. McCallum. Composition of conditional random fields for transfer learning. Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 748–754, 2005b. find
  79. C. Sutton, A. McCallum, and K. Rohanimanesh. Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence Data. JMLR, 8: 693–723, 2007. find
  80. J. Suzuki and H. Isozaki. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. In Proceedings of ACL-08: HLT, pages 665–673, Columbus, Ohio, June 2008. Association for Computational Linguistics. find
  81. W. J. Teahan and J. G. Cleary. The entropy of english using ppm-based models. In In Data Compression Conference (DCC’96), pages 53–62. IEEE Computer Society Press, 1996. ar Xi v K. Toutanova, D. Klein, C. D. Manning, and Y. Singer. Feature-rich part-of-speech tagging with a cyclic dependency network… find
  82. J. Turian, L. Ratinov, and Y. Bengio. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the Association for Computational Linguistics (ACL), pages 384–392. Association for Computational Linguistics, 2010. find our card
  83. N. Ueffing, G. Haffari, and A. Sarkar. Transductive learning for statistical machine translation. Proceedings of the 45th Annual Meeting of the ACL, pages 25–32, 2007. find
  84. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K.J. Lang. Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(3):328–339, 1989. find
  85. J. Weston, F. Ratle, and R. Collobert. Deep learning via semi-supervised embedding. In Proceedings of the 25th international conference on Machine learning, pages 1168–1175. ACM, 2008. find