Elamur Research
☰
home / papers / 2016 / MS MARCO: A Human Generated MAchine Reading COmprehension Dataset

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1611.09268

Summary

Before 2016, most machine reading comprehension (MRC) and QA benchmarks were small or synthetic. Many were built from crowd-written questions over clean text such as Wikipedia, and their answers were limited to spans. MS MARCO, from Microsoft, starts from real information needs instead. Its questions are anonymized Bing (and Cortana) search queries, filtered to keep only question-like ones. Each is paired with passages extracted automatically from web pages that Bing retrieved. Crowd editors read the passages and marked those that support an answer. They then either wrote a free-form answer or labelled the question unanswerable, and for a subset they rewrote answers as standalone, well-formed sentences. The authors define three tasks: answerability detection with answer synthesis, well-formed answer generation, and passage ranking. Baseline models fall far short of human ensembles. Version 2.1 is harder than v1.1 because extractive models cannot detect unanswerable questions or use words absent from the passages. MS MARCO's scale, noisy web text, and evidence spread across passages made it a standard benchmark for MRC and neural IR.

Key points

  • Scale: 1,010,916 questions, 8,841,823 passages drawn from 3,563,535 web documents, and 182,669 human-rewritten well-formed answers. It is over ten times larger than SQuAD.
  • MS MARCO is the only dataset in the paper's comparison with real user-log questions and query segment labels. Some questions have several answers and some have none.
  • Early v1.1 baselines, measured by ROUGE-L on a subset: best single passage 0.351, a DSSM-like ranker 0.177, a memory-network seq2seq model 0.119, and plain seq2seq 0.089. On the numeric subset, ReasoNet reached 58.9% accuracy and AS Reader 55.0%.
  • On v2.1, a BiDaF-style span model scores 0.150 ROUGE-L on the novice task and 0.170 on the intermediate task, compared with 0.268 on v1.1. Human ensembles reach 0.737 and 0.630.
  • Machine models beat the v1.1 human baseline within about 15 months. The authors responded by building a stronger human baseline from the five best editors on 1,427 sampled questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

1102 papers of our library cite it (of the 140673 whose bibliographies we read); 79 of them have a page here, 1023 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1611.09268

@misc{bajaj2016ms,
  title = {{MS MARCO: A Human Generated MAchine Reading COmprehension Dataset}},
  author = {Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and Rosenberg, Mir and Song, Xia and Stoica, Alina and Tiwary, Saurabh and Wang, Tong},
  year = {2016},
  eprint = {1611.09268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.09268}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amazon Alexa. Amazon alexa. http://alexa.amazon.com/, 2018. ↗
  2. Amazon Echo. Amazon echo. https://en.wikipedia.org/wiki/Amazon\_Echo, 2018. ↗
  3. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  4. S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, pages 65–72, 2005. find our card
  5. J. Cheng, L. Dong, and M. Lapata. Long short-term memory-networks for machine reading. abs/1601.06733, 2016. URL http://arxiv.org/abs/1601.06733. ↗ our card
  6. C. Clark and M. Gardner. Simple and effective multi-paragraph reading comprehension. CoRR, abs/1710.10723, 2017. URL http://arxiv.org/abs/1710.10723. ↗ our card
  7. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018. find
  8. Cortana. Cortana personal assistant. cortana/, 2018. http://www.microsoft.com/en-us/mobile/experiences/ ↗
  9. G. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30–42, 2012. find
  10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fe. Imagenet: Alarge-scalehierarchicalimagedatabas. CVPR, 2009. URL http://www.image-net.org/papers/imagenet\_cvpr09.pdf. ↗
  11. L. Deng and X. Huang. Challenges in adopting speech recognition. Communications of the ACM, 47(1):69–75, 2004. find
  12. M. Dunn, L. Sagun, M. Higgins, V. U. Güney, V. Cirik, and K. Cho. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/1704.05179, 2017. find our card
  13. B. H. Frank. Google brain chief: Deep learning takes at least 100,000 examples. https://venturebeat.com/2017/10/23/google-brain-chief-says-100000-examples-is-enough-data-for-deep-learning/, 2017. ↗
  14. K. Ganesan. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks. 2018. find
  15. J. Gao, M. Galley, and L. Li. Neural approaches to conversational ai. arXiv preprint arXiv:1809.08267, 2018. find
  16. T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. III, and K. Crawford. Datasheets for datasets. 2018. find
  17. Google Assistant. Google assistant. https://assistant.google.com/, 2018. ↗
  18. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2015. URL https: //arxiv.org/abs/1512.03385. find
  19. W. He, K. Liu, Y. Lyu, S. Zhao, X. Xiao, Y. Liu, Y. Wang, H. Wu, Q. She, X. Liu, T. Wu, and H. Wang. Dureader: a chinese machine reading comprehension dataset from real-world applications. CoRR, abs/1711.05073, 2017. find our card
  20. K. M. Hermann, T. Kociský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. 2015a. URL https://arxiv.org/abs/1506.03340. ↗ our card
  21. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701, 2015b. find our card
  22. G. Hinton, L. Deng, D. Yu, G. Dalh, and A. Mohamed. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, 2012. find
  23. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  24. CoRR, P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2333–2338. ACM, 2013. find
  25. R. Kadlec, M. Schmid, O. Bajgar, and J. Kleindienst. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547, 2016. find
  26. T. Kociský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. CoRR, abs/1712.07040, 2017. find our card
  27. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, 2017. find our card
  28. C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain, 2004. find
  29. S. Maples. The rouge-ar: A proposed extension to the rouge evaluation metric for abstractive text summarization. 2017. find
  30. B. Mitra and N. Craswell. An introduction to neural information retrieval. Foundations and Trends R in Information Retrieval (to appear), 2018. find
  31. B. Mitra, G. Simon, J. Gao, N. Craswell, and L. J. Deng. A proposal for evaluating answer distillation from web data. 2016. find
  32. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002. find our card
  33. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. 2016. URL https://arxiv.org/abs/1606.05250. ↗ our card
  34. P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. find our card
  35. S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends R in Information Retrieval, 3(4):333–389, 2009. find our card
  36. M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional attention flow for machine comprehension. CoRR, abs/1611.01603, 2016. find our card
  37. Y. Shen, P.-S. Huang, J. Gao, and W. Chen. Reasonet: Learning to stop reading in machine comprehension. arXiv preprint arXiv:1609.05284, 2016. — [arXiv:1609.05284](https://arxiv.org/abs/1609.05284) find
  38. Siri. Siri personal assistant. http://www.apple.com/ios/siri/, 2018. ↗
  39. S. Sukhbaatar, J. Weston, R. Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448, 2015. find
  40. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215, 2014. URL http://arxiv.org/abs/1409.3215. ↗
  41. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. Newsqa: A machine comprehension dataset. In Rep4NLP@ACL, 2017. find our card
  42. J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merrienboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. 2015. URL https://arxiv.org/abs/1502.05698. ↗ our card
  43. A. Wissner-Gross. Datasets over algorithms. Edge. com. Retrieved, 8, 2016. find
  44. S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. Van Durme†. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. find our card