Elamur Research⌕
☰
home / papers / 2018 / Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results

Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results

Paper · 2018 · Matt Crane

Summary

The paper examines how reproducible neural results in NLP really are. It argues that releasing source code does not guarantee that others can reproduce a reported number. Crane uses answer selection for open-domain question answering as a test case. The exemplar model is the simple Siamese CNN by Severyn and Moschitti (2015), in the PyTorch reimplementation by Sequiera et al. (2017). It is evaluated on TrecQA (RAW variant) and WikiQA with AP and RR.

Hyper-parameters are held fixed throughout. Only one environmental factor is changed at a time, for example the code revision used. Differences within each condition are checked with a paired Wilcoxon signed rank test.

The main finding is that these normally unreported factors can move scores by more than most published state-of-the-art gains. Code without a reproducible environment is therefore of limited value. The paper also shows that a learning-free baseline beats several older published systems, exposing the use of weak baselines. For each factor, it recommends ways to make reported results reproducible.

Key points

  • A non-learning baseline sums IDF weights of the terms shared by question and candidate, after removing 127 nltk stopwords. On TrecQA RAW it reaches 0.701 AP and 0.769 RR. This beats a number of earlier state-of-the-art systems, while Yih et al. (2013) had reported only 0.6531 AP and 0.7071 RR for it.
  • Successive state-of-the-art gains on TrecQA and WikiQA, compiled from the ACL Wiki and from papers, are mostly small. Many are around 0.01–0.02 AP or less.
  • Running different commits of the same model repository changed TrecQA AP from 0.7495 (cf0e269) to about 0.67 in later versions. These drops were significant at p < 0.01.
  • Some revisions would not run inside the Docker environment. Others gave different results from one run to the next.
  • The TrecQA RAW and CLEAN variants are not comparable with each other. CLEAN removes questions that have no positive answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksInformation Systems and Management · 41 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Matt Crane. 2018. Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00018

@article{crane2018questionable,
  title = {{Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results}},
  author = {Matt Crane},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00018},
  url = {https://doi.org/10.1162/tacl_a_00018}
}

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/crane-2018-questionable-answers-question-answering-research/