Elamur Research
☰
home / papers / 2017 / Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-1035.pdf

Summary

Neural sequence taggers train non-deterministically, so the random seed alone can push a model into local minima that generalize differently. Reimers and Gurevych argue that comparing systems by one test score is therefore unreliable. They repeatedly retrained two recent NER systems, varying nothing but the seed: the one by Lample et al. (2016) and the one by Ma and Hovy (2016). They also trained about 50,000 BiLSTM taggers on five tasks: POS (Penn Treebank), Chunking (CoNLL 2000), NER (CoNLL 2003), Entities (ACE 2005) and Events (TempEval 3).

Seed choice alone produced statistically significant gaps of up to a full F1 point. Picking the best run on the development set did not fix this. The comparison of score distributions also reversed the published ranking of the two NER systems.

The authors propose reporting distributions over multiple runs instead of single scores. Using distributions, they identify design choices that give both higher and more stable performance. Their implementation is released on GitHub (UKPLab/emnlp2017-bilstm-cnn-crf). The work matters for fair benchmarking and reproducibility in NLP.

Key points

  • Ma and Hovy system (86 seeds): F1 ranged from 89.99% to 91.00%, against a reported 91.21%.
  • Lample et al. system (41 seeds): F1 ranged from 90.19% to 91.14%, with a higher median and smaller variance. A Kolmogorov-Smirnov test showed the two distributions differ (p < 0.01), so Lample et al. comes out ahead, reversing the published ranking.
  • Development scores are a weak guide: dev and test scores correlated only weakly (Spearman ρ = 0.229). For Lample et al., the run with the best dev score reached only 90.31% on test.
  • Seed-only differences across tasks: on CoNLL 2003 NER, two runs with identical hyperparameters differed by 0.38% F1 at the median and 1.08% at the 95th percentile. On ACE 2005 Entities, the gap reached 8.23 points (82.5% vs 74.3%).
  • Robust design choices:
    • Variational dropout was the best option in 83.5% of 479 configurations.
    • Gradient normalization helped, while gradient clipping did not.
    • Adam, Nadam and RMSProp varied less than SGD, Adagrad and Adadelta.
    • Two stacked LSTM layers generally performed best.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Named entity recognition

Cited by

193 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1035 · arXiv 1707.09861

@inproceedings{reimers2017reporting,
  title = {{Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging}},
  author = {Reimers, Nils and Gurevych, Iryna},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1035},
  eprint = {1707.09861},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1035}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. CoRR, abs/1603.06042. find our card
  2. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching Word Vectors with Subword Information. arXiv preprint arXiv:1607.04606. find
  3. Timothy Dozat. 2015. Incorporating Nesterov Momentum into Adam. find
  4. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res., 12:2121–2159. find
  5. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research, 11:625–660. find
  6. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from Reproduction Problems: What Replication Failure Teaches Us. In ACL (1), pages 1691–1701. The Association for Computer Linguistics. find
  7. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019–1027. find our card
  8. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. CoRR, abs/1609.04836. find
  9. Diederik P. Kingma and Jimmy Ba. 2014. A Method for Stochastic Optimization. abs/1412.6980. find
  10. Alexandros Komninos and Suresh Manandhar. 2016. Dependency based embeddings for sentence classification tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1490–1500, San Diego, California. As… find
  11. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. CoRR, abs/1603.01360. find
  12. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541–551. find
  13. Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. 1998. Efficient BackProp. In Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, pages 9–50, London, UK, UK. Springer-Verlag. find
  14. Omer Levy and Yoav Goldberg. 2014. Dependency-Based Word Embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 302–308. find
  15. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AIS-TATS10). Society for Artificial Intelligence and Statistics. find
  16. Geoffrey Hinton. 2012. Neural Networks for Machine Learning - Lecture 6a - Overview of mini-batch gradient descent. find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997a. Flat Minima. Neural Computation, 9(1):1–42. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997b. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  19. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. CoRR, abs/1508.01991. find our card
  20. Adam: CoRR, Qi Li, Heng Ji, and Liang Huang. 2013. Joint Event Extraction via Structured Prediction with Global Features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 73–82, Sofia, Bulgaria. Association for Computational Li… find
  21. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. CoRR, abs/1603.01354. find our card
  22. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Comput. Linguist., 19(2):313–330. find our card
  23. Frank J. Massey. 1951. The kolmogorov-smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253):68–78. find
  24. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. 2014. An Efficient Approach for Assessing Hyperparameter Importance. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, pages I– 754–I–762. JMLR.org. find
  25. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. CoRR, abs/1301.3781. find
  27. Yurii Nesterov. 1983. A method of solving a convex programming problem with convergence rate O(1/sqr(k)). Soviet Mathematics Doklady, 27:372– 376. find
  28. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the Difficulty of Training Recurrent Neural Networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13, pages III–1310–III– 1318. JMLR.org. find
  29. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  30. Nils Reimers and Iryna Gurevych. 2017. Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks. arXiv preprint arXiv:1707.06799. — [arXiv:1707.06799](https://arxiv.org/abs/1707.06799) find
  31. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, Berlin, Germany. Association for Computational Linguis… find our card
  32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res., 15(1):1929– 1958. find
  33. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich Part-of-speech Tagging with a Cyclic Dependency Network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol… find
  34. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  35. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701. find our card