Natural Questions: A Benchmark for Question Answering Research

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

1981 papers of our library cite it (of the 140673 whose bibliographies we read); 136 of them have a page here, the first 100 below, 1845 not yet.

• Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
• Muppet: Massive Multi-task Representations with Pre-Finetuning
• Synthetic QA Corpora Generation with Roundtrip Consistency
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models
• PaLM 2 Technical Report
• AraBERT: Transformer-based Model for Arabic Language Understanding
• Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Qwen Technical Report
• STARC: Structured Annotations for Reading Comprehension
• Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
• InternLM2 Technical Report
• A Survey on Evaluation of Large Language Models
• HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Pre-Training with Whole Word Masking for Chinese BERT
• Revisiting Pre-Trained Models for Chinese Natural Language Processing
• Promptagator: Few-shot Dense Retrieval From 8 Examples
• Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• Moshi: a speech-text foundation model for real-time dialogue
• Event Extraction by Answering (Almost) Natural Questions
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• English Machine Reading Comprehension Datasets: A Survey
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
• HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark
• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
• RARR: Researching and Revising What Language Models Say, Using Language Models
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Gemini: A Family of Highly Capable Multimodal Models
• Improving alignment of dialogue agents via targeted human judgements
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
• The False Promise of Imitating Proprietary LLMs
• Evaluating Large Language Models: A Comprehensive Survey
• TrustLLM: Trustworthiness in Large Language Models
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• Survey of Hallucination in Natural Language Generation
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• SpanBERT: Improving Pre-training by Representing and Predicting Spans
• Large Language Models Struggle to Learn Long-Tail Knowledge
• RealTime QA: What's the Answer Right Now?
• Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• UNIFIEDQA: Crossing Format Boundaries with a Single QA System
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
• Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
• Inquisitive Question Generation for High Level Text Comprehension
• Hurdles to Progress in Long-form Question Answering
• Internet-augmented language models through few-shot prompting for open-domain question answering
• Latent Retrieval for Weakly Supervised Open Domain Question Answering
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• MLQA: Evaluating Cross-lingual Extractive Question Answering
• PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
• CMMLU: Measuring massive multitask language understanding in Chinese
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Search-o1: Agentic Search-Enhanced Large Reasoning Models
• Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning
• WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
• What Makes Good In-Context Examples for GPT-3?
• Evaluating Verifiability in Generative Search Engines
• Lost in the Middle: How Language Models Use Long Contexts
• Entity-Based Knowledge Conflicts in Question Answering
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Query Rewriting in Retrieval-Augmented Large Language Models
• Generation-Augmented Retrieval for Open-domain Question Answering
• DocVQA: A Dataset for VQA on Document Images
• Teaching language models to support answers with verified quotes
• SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
• Generative Representational Instruction Tuning
• A Comprehensive Overview of Large Language Models
• Text and Code Embeddings by Contrastive Pre-Training
• Large Dual Encoders Are Generalizable Retrievers
• TALM: Tool Augmented Language Models
• Language Models as Knowledge Bases?
• KILT: a Benchmark for Knowledge Intensive Language Tasks
• PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• A guide to the dataset explosion in QA, NLI, and commonsense reasoning
• Improving Passage Retrieval with Zero-Shot Question Generation
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
• SCROLLS: Standardized CompaRison Over Long Language Sequences
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
• REPLUG: Retrieval-Augmented Black-Box Language Models
• Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
• Retrieval Augmentation Reduces Hallucination in Conversation
• BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00276

@misc{kwiatkowski2019natural,
  title = {{Natural Questions: A Benchmark for Question Answering Research}},
  author = {Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav},
  year = {2019},
  doi = {10.1162/tacl\_a\_00276},
  url = {https://doi.org/10.1162/tacl_a_00276}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A BERT Baseline for the Natural Questions. arXiv preprint:1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) find
2. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 845–855, Melbourne. find our card
3. Melis, and Edward Grefenstette. 2018. The narrative qa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6317–328. find
4. Luc Devroye, László Györfi, and Gábor Lugosi. 1997. A Probabilistic Theory of Pattern Recognition, corrected 2nd edition, volume 31 of Applications of Mathematics. Springer. find
5. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794. Copenhagen. find our card
6. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. Dureader: A Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Quest… find our card
7. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can A suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels. find
8. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics. find
9. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches. find our card
10. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS’15. Cambridge, MA. find our card
11. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. Austin, TX. find
12. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the International Conference on Learning Representations. find our card
13. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association f… find our card
14. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen. find our card
15. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. find our card
16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLUE: A method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia. find
17. Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Languag… find
18. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
19. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. find our card
20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, TX. find our card
21. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon. find our card
22. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. find our card
23. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… find our card
24. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the find our card