Elamur Research
☰
home / papers / 2017 / SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine

SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1704.05179

Summary

Popular reading-comprehension benchmarks of the time, such as SQuAD and CNN/DailyMail, build each question from a pre-selected passage, so the supplied context is always clean and relevant. A real open-domain QA system, by contrast, first retrieves documents through a search engine and must cope with off-topic or fragmentary text.

SearchQA reverses the usual construction order. The authors crawled Jeopardy! clue–answer pairs from J! Archive, sent each clue to Google as a query, and attached the returned snippets as context. They then removed snippets that would make the task trivial, such as those containing episode air dates, the verbatim clue, or words like "trivia". They also dropped tuples whose answer was missing from the context or longer than three words. Metadata from both Jeopardy! and Google was kept.

The dataset was evaluated with human volunteers, a TF-IDF baseline, and an Attention Sum Reader with an n-gram extension. Machines trail people noticeably, which makes SearchQA a realistic, noisy-context benchmark for retrieval-based question answering.

Key points

  • The dataset has 140,461 question–answer pairs and about 6.9M snippets in total, averaging 49.6 snippets per question. Answers average 1.47 tokens. The release is at github.com/nyu-dl/SearchQA, together with PyTorch code for ASR and n-gram ASR.
  • The train/validation/test split (99,820 / 13,393 / 27,248) uses non-overlapping years. Validation and test questions come from later years than training questions, to test generalization to future questions.
  • On the Jeopardy! side, the metadata records each clue's category, its monetary value, the episode number and the date it aired; on the Google side, it records each snippet's URL, title and related links.
  • Thirteen NYU volunteers scored 66.97% accuracy on unigram answers and an F1 of 57.62 on n-gram answers.
  • On the test set, TF-IDF Max reached only 12.7% unigram accuracy. ASR reached 41.3% (65.1% Acc@5), and n-gram ASR reached an F1 of 22.8.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

197 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1704.05179 · arXiv 1704.05179

@misc{dunn2017searchqa,
  title = {{SearchQA: A New Q\&A Dataset Augmented with Context from a Search Engine}},
  author = {Dunn, Matthew and Sagun, Levent and Higgins, Mike and Guney, V. Ugur and Cirik, Volkan and Cho, Kyunghyun},
  year = {2017},
  doi = {10.48550/arxiv.1704.05179},
  eprint = {1704.05179},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1704.05179}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  2. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075 . find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014). find
  4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701. find our card
  5. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301 . find our card
  6. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547 . find
  7. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  8. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. pages 1097–1105. find
  9. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 . http://pytorch.org/ ↗ our card
  10. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457 . find
  11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 . find our card
  12. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1):1929–1958. find
  13. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. NewsQA: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 . find our card
  14. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems. pages 2692–2700. find
  15. Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256 . — [arXiv:1610.05256](https://arxiv.org/abs/1610.05256) find