Elamur Research⌕
☰
home / papers / 2018 / Design Challenges and Misconceptions in Neural Sequence Labeling

Design Challenges and Misconceptions in Neural Sequence Labeling

Paper · 2018 · Jie Yang, Shuailong Liang and Yue Zhang

Summary

Published neural sequence labeling results are often hard to reproduce and compare. Studies differ in data splits, preprocessing, handcrafted features, hyperparameters, hardware, and whether they report a single best run or a mean over seeds. Some even reach opposite conclusions, for example on which optimizer works best. Yang, Liang and Zhang address this by re-implementing twelve architectures in one PyTorch framework, released as NCRF++. Each model is built from three choices: a character encoder (none, CNN, or LSTM), a word encoder (CNN or LSTM), and an output layer (softmax or CRF). They evaluate all twelve on CoNLL 2003 NER, CoNLL 2000 chunking, and PTB POS tagging, using repeated runs and significance testing. Their reimplementations match or exceed the originally reported scores. The main findings are:

  • Character information clearly helps, mostly on out-of-vocabulary words.
  • Word-level LSTMs usually beat word-level CNNs, but CNNs decode faster.
  • CRF helps NER and chunking but not POS tagging.

This gives practitioners a trustworthy, controlled basis for choosing architectures and settles several disputed claims in earlier work.

Key points

  • Twelve models are compared, covering every combination of {no char, char CNN, char LSTM} × {word CNN, word LSTM} × {softmax, CRF}. This includes combinations that no earlier paper had tried.
  • The authors list six sources of inconsistency in prior work: dataset, preprocessing, features, hyperparameters, evaluation protocol, and hardware.
  • In controlled NER experiments:
    • BIOES tagging beats BIO (p < 0.05).
    • SGD outperforms Adagrad, Adadelta, RMSProp and Adam (p < 0.01), though it converges more slowly.
    • GloVe 100d embeddings beat SENNA.
    • The GPU vs CPU difference is not significant (p > 0.2).
  • In the OOV analysis on NER, adding a character CNN to WLSTM+CRF raises F1 on out-of-both-vocabulary entities from 69.68 to 81.58.
  • Decoding speed was measured on 10,000 NER sentences with a GTX 1080. CRF and character LSTM slow decoding most. A character CNN adds little cost.

Interesting ideas

  • I1. Modular grid of components. Splitting a model into interchangeable layers and testing the full combination grid exposes untried configurations. It also separates the effect of each component, which is a reusable design for any architecture comparison, including Ukrainian NLP pipelines.
  • I2. Statistics over single scores. Reporting means over several random seeds and running significance tests makes claims about optimizers or tag schemes much more robust. The authors also analyze errors using the median-scoring run rather than the best one, which avoids showcasing a lucky seed.
  • I3. Four-way OOV breakdown. Test items are split by whether they appear in the training vocabulary, the embedding vocabulary, both, or neither. This shows exactly where a component like character encoding adds value, and it is especially relevant for morphologically rich languages.
  • I4. Speed as an evaluation axis. Measuring decoding time alongside accuracy showed that a character CNN gives most of the gain of a character LSTM at little cost. It also showed that word CNNs are a strong option when throughput matters.
  • I5. Testing folklore claims. The paper reran contested claims, such as GPU vs CPU accuracy and SGD vs Adam, with repeated experiments instead of accepting them from earlier papers. This habit is cheap and prevents inherited misconceptions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Jie Yang, Shuailong Liang and Yue Zhang. 2018. Design Challenges and Misconceptions in Neural Sequence Labeling. Proceedings of the 27th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yang2018design,
  title = {{Design Challenges and Misconceptions in Neural Sequence Labeling}},
  author = {Jie Yang, Shuailong Liang and Yue Zhang},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1327/}
}

References

38 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. TACL, 4:357–370.
  2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493– 2537.
  3. Hong-Jie Dai, Po-Ting Lai, Yung-Chun Chang, and Richard Tzong-Han Tsai. 2015. Enhancing of chemical compound and drug name recognition using representative tag scheme and fine-grained tokenization. Journal of cheminformatics, 7(S1):S14. find
  4. Cıcero dos Santos, Victor Guimaraes, RJ Niterói, and Rio de Janeiro. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of NEWS 2015 The Fifth Named Entities Workshop, page 25.
  5. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  6. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 315–323. find
  7. James Hammerton. 2003. Named entity recognition with long short-term memory. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 172–175. Association for Computational Linguistics. find
  8. Kazuma Hashimoto, Yoshimasa Tsuruoka, Richard Socher, et al. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923–1933.
  9. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991.
  10. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. find
  11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  12. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL-HLT, pages 260–270.
  13. Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551. find
  14. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proc…
  15. Liyuan Liu, Jingbo Shang, Frank Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. AAAI.
  16. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In EMNLP, pages 879–888. find
  17. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF. In ACL, volume 1, pages 1064–1074.
  18. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330.
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
  20. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In CoNLL, pages 78–86.
  21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  22. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL, volume 1, pages 1756–1765.
  23. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In CoNLL, pages 147–155.
  24. Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2121– 2130. find
  25. Nils Reimers and Iryna Gurevych. 2017a. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799. find
  26. Nils Reimers and Iryna Gurevych. 2017b. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348.
  27. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1818–1826. find
  28. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929– 1958. find
  29. Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2670–2680.
  30. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31. find
  31. Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7, pages 127–132. Association for Computational Linguistics…
  32. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In HLT-NAACL, pages 142–147.
  33. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume…
  34. Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270. find
  35. Jie Yang, Yue Zhang, and Fei Dong. 2017a. Neural reranking for named entity recognition. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 784–792. find
  36. Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017b. Transfer learning for sequence tagging with hierarchical recurrent networks. In ICLR. find
  37. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
  38. Feifei Zhai, Saloni Potdar, Bing Xiang, and Bowen Zhou. 2017. Neural models for sequence chunking. In AAAI, pages 3365–3371. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-design-challenges-misconceptions-neural-sequence/