Elamur Research⌕
☰
home / papers / 2017 / Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

Paper · 2017 · Rotem Dror, Gili Baumer, Marina Bogomolov et al.

Summary

NLP systems are now routinely evaluated on many datasets spanning languages, domains and genres. A common practice is to run a significance test on each dataset and count how many fall below a threshold such as 0.05. The authors show that this practice does not control the chance of making at least one false claim, and the problem grows as more datasets are added. They adapt replicability analysis from statistics to answer two questions: on how many datasets one algorithm truly beats another (counting), and which datasets those are (identification). For counting, they use the partial conjunction approach of Benjamini and Heller, with Fisher-based p-values for independent datasets and Bonferroni-based p-values for dependent ones. For identification, they use Holm's procedure, which bounds the family-wise error. They test the framework on synthetic examples and four tasks: dependency parsing across several domains, part-of-speech tagging in many languages, sentiment classification transferred between domains, and predicting word similarity. The sound estimates often differ substantially from naive counts. They can be higher when datasets are independent and lower when datasets overlap. This gives NLP researchers a principled and compact way to report results across many evaluation sets.

Key points

  • The paper appeared in TACL vol. 5 (2017), and the code is released at github.com/rtmdrr/replicability-analysis-NLP.
  • In the Mate vs. Redshift parsing comparison, the analysis finds evidence of an effect on 5 of 7 evaluation sets, whereas naive counting gives only 2.
  • In multilingual POS tagging (MIMICK vs. Char→Tag), the analysis supports 16 of 23 sets with an effect, compared with 11 by naive counting.
  • For dependent setups, naive counting overstates the gains of new algorithms. In sentiment classification at α = 0.05, it reports 10 of 12 sets, while the Bonferroni-based estimate is 6.
  • Suggested extensions include Simes-based tests, Hochberg's or Hommel's procedures under positive dependence, and FDR control (Benjamini–Hochberg) when some errors are acceptable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technion – Israel Institute of TechnologyHaifa · Israel59 works · 7,590 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Rotem Dror, Gili Baumer, Marina Bogomolov and Roi Reichart. 2017. Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00074 · arXiv 1709.09500

@article{dror2017replicability,
  title = {{Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets}},
  author = {Rotem Dror, Gili Baumer, Marina Bogomolov and Roi Reichart},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00074},
  eprint = {1709.09500},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00074}
}

References

71 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL.
  2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL.
  3. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of EMNLP. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting se… find
  4. C. Glenn Begley and Lee M. Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature, 483(7391):531–533. find
  5. Yoav Benjamini and Ruth Heller. 2008. Screening for partial conjunction hypotheses. Biometrics, 64(4):1215–1222. find
  6. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), pages 289– 300. find
  7. Yoav Benjamini, Abba M. Krieger, and Daniel Yekutieli. 2006. Adaptive linear step-up procedures that control the false discovery rate. Biometrika, pages 491–507. find
  8. Yoav Benjamini, Ruth Heller, and Daniel Yekutieli. 2009. Selective inference in complex research. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 367(1906):4255–4271. find
  9. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of EMNLP-CoNLL. find
  10. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of EMNLP.
  11. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL.
  12. Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of COLING.
  13. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of ACL.
  14. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research (JAIR), 49:1–47. find
  15. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-x shared task on multilingual dependency parsing. In Proceedings of CoNLL.
  16. Yee Seng Chan and Hwee Tou Ng. 2007. Domain adaptation with active learning for word sense disambiguation. In Proceedings of ACL. find
  17. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of HLT-NAACL.
  18. Minmin Chen, Yixin Chen, and Kilian Q. Weinberger. 2011. Automatic feature decomposition for single view co-training. In Proceedings of ICML. find
  19. Jinho D. Choi, Joel Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In Proceedings of ACL.
  20. Open Science Collaboration. 2012. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6):657–660. find
  21. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational linguistics, 29(4):589–637. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of ACL.
  22. Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Stanford dependencies: A cross-linguistic typology. In Proceedings of LREC. Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets… find
  23. Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895–1923. find
  24. Bradley Efron and Robert J. Tibshirani. 1994. An introduction to the bootstrap. CRC press. find
  25. Alessio Farcomeni. 2007. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical Methods in Medical Research. find
  26. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at wordvec-tors.org. In Proceedings of the ACL: System Demonstrations. find
  27. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001a. Placing search in context: The concept revisited. In Proceedings of WWW. find
  28. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001b. Placing search in context: The concept revisited. In Proceedings of WWW. find
  29. Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang, Xian Wu, and Zhong Su. 2009. Domain adaptation with latent semantic association for named entity recognition. In Proceedings of HLT-NAACL. find
  30. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of ACM SIGKDD. find
  31. Ruth Heller, Marina Bogomolov, and Yoav Benjamini. 2014. Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study. Proceedings of the National Academy of Sciences, 111(46):16262–16267. find
  32. Thomas Herndon, Michael Ash, and Robert Pollin. 2014. Does high public debt consistently stifle economic growth? a critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2):257–279. find
  33. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  34. Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Simlex-999: Evaluating semantic models with (genuine) similarity estimation.
  35. Computational Linguistics. Yosef Hochberg. 1988. A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4):800–802. find
  36. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70. find
  37. Gerhard Hommel. 1988. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2):383–386. find
  38. Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of EMNLP.
  39. Matthew Honnibal, Yoav Goldberg, and Mark Johnson. 2013. A non-monotonic arc-eager transition system for dependency parsing. In Proceedings of CoNLL.
  40. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation.
  41. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the tenth Machine Translation Summit. find
  42. Jeffrey T. Leek and Roger D Peng. 2015. Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences, 112(6):1645–1646. find
  43. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL. find
  44. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of EMNLP.
  45. Thomas M. Loughin. 2004. A systematic comparison of methods for combining p-values from independent tests. Computational Statistics & Data Analysis, 47(3):467–485. find
  46. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of CoNLL.
  47. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330.
  48. Edison Marrese-Taylor and Yutaka Matsuo. 2017. Replication issues in syntax-based aspect extraction for opinion mining. In Proceedings of the Student Research Workshop at EACL. find
  49. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  50. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS.
  51. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1–28. find
  52. Ramal Moonesinghe, Muin J. Khoury, and A. Cecile J. W. Janssens. 2007. Most published research findings are false but a little replication goes a long way. PLoS Med, 4(2):e28. Aurélie Névéol, Cyril Grouin, Kevin Bretonnel Cohen, and Aude Robert. 2016. Replicability of research in biomedical natur… find
  53. Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  54. The CoNLL 2007 shared task on dependency parsing. In Proceedings of CoNLL. Diarmuid Ó Séaghdha and Anna Korhonen. 2014. Probabilistic distributional semantics. Computational Linguistics, 40(3):587–631.
  55. Prasad Patil, Roger D. Peng, and Jeffrey Leek. 2016. A statistical definition for reproducibility and replicability. bioRxiv. find
  56. Roger D. Peng. 2011. Reproducible research in computational science. Science, 334(6060):1226–1227. find
  57. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP.
  58. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  59. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of EMNLP. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis us… find
  60. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Computing word relatedness using temporal semantic analysis. In Proceedings of WWW. find
  61. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  62. Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of CoNLL. find
  63. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL.
  64. R. John Simes. 1986. An improved Bonferroni procedure for multiple tests of significance. Biometrika, pages 751–754. find
  65. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP. Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Héctor Martı́nez Alonso. 2014. What’s in a p-v…
  66. Anders Søgaard. 2013. Estimating effect size across datasets. In Proceedings of HLT-NAACL. find
  67. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245–251. find
  68. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. find
  69. Springer. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83. find
  70. Dongqiang Yang and David M.W. Powers. Verb similarity on the taxonomy of WordNet. In Proceedings of the 3rd International WordNet Conference. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of CoNLL. find
  71. Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of CoNLL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dror-2017-replicability-analysis-natural-language-processing/