Elamur Research
☰
home / papers / 2015 / WikiQA: A Challenge Dataset for Open-Domain Question Answering

WikiQA: A Challenge Dataset for Open-Domain Question Answering

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D15-1237.pdf

Summary

The paper addresses a weakness in how answer sentence selection has been benchmarked. The standard QASent dataset, derived from TREC-QA, only kept candidate sentences that overlap lexically with the question and dropped questions lacking a correct answer. This favors simple word-matching systems and hides a realistic difficulty. The authors release WikiQA as an alternative. Its questions come from real Bing search queries that users clicked through to Wikipedia. Every sentence in the summary section of the linked page serves as a candidate, and crowdworkers labeled which sentences answer the question.

Questions with no correct candidate are retained. This supports a new task, answer triggering: deciding whether any candidate answers the question and, if so, choosing one.

Experiments show a reversal relative to QASent. Lexical models such as LCLR do well on QASent, but sentence-level semantic models (CNN) do better on WikiQA. Answer triggering is still hard, with F1 around 30%. The resource gives a more realistic and more demanding testbed for open-domain QA.

Key points

  • WikiQA has 3,047 questions sampled from Bing logs (May 2010–July 2011) and 29,258 candidate sentences, of which 1,473 are answers. It is split 70/10/20 into train, dev and test.
  • Roughly two-thirds of the questions have no correct candidate sentence. About 20.3% of answer sentences share no content words with their question.
  • Measured by MAP/MRR on WikiQA, CNN-Cnt (CNN plus word-count features) scores best at 0.6520/0.6652, ahead of LCLR at 0.5993/0.6086. On QASent, LCLR at 0.6954/0.7617 is roughly matched by CNN-Cnt.
  • For answer triggering, the authors propose question-level precision, recall and F1. CNN-Cnt reaches an F1 of 30.61, which rises to 32.17 when a question-length feature (QLen) is added.
  • Answer-phrase annotations are also included, and the data and evaluation script are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

306 papers of our library cite it (of the 140673 whose bibliographies we read); 27 of them have a page here, 279 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1237

@inproceedings{yang2015wikiqa,
  title = {{WikiQA: A Challenge Dataset for Open-Domain Question Answering}},
  author = {Yang, Yi and Yih, Wen-tau and Meek, Christopher},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1237},
  url = {https://doi.org/10.18653/v1/d15-1237}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1011–1019. find our card
  2. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML), pages 1188–1196. find our card
  3. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings the International Conference on Computational Linguistics (COLING), pages 556–562. find
  4. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pages 3111–3119. find
  5. Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP), pages 458–467. find
  6. Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 question answering track evaluation. In Proceedings of the Text Retrieval Conference TREC-8, page 82. find
  7. Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP), pages 22–32. find
  8. Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In Proceedings of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 858–867. find
  9. Wen-tau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  10. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. In Proceedings of the Deep Learning and Representation Learning Workshop: NIPS-2014. find