WebGPT: Browser-assisted question-answering with human feedback
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2112.09332
Summary
Long-form question answering (LFQA) asks a system to write a paragraph-length reply to an open-ended question, and such systems still fall short of human quality. The authors let GPT-3 operate a simplified, text-only web browser. The model issues commands such as searching through the Microsoft Bing Web Search API, clicking links, scrolling and quoting passages, and the quoted passages become references for its final answer. Humans can do the same task through a graphical interface. The team therefore collected human demonstrations for behavior cloning and pairwise answer comparisons for a reward model. They then optimized further with reinforcement learning and rejection sampling. Most questions came from ELI5. The strongest system was a 175B behavior-cloned model that picks the best of 64 samples by reward score. Judges preferred its answers over the human demonstrators' answers 56% of the time and over the top-voted Reddit answer 69% of the time. The work shows that retrieval and synthesis can be trained jointly from human feedback, and that cited sources make fact-checking much easier. Out-of-distribution questions, however, remain difficult.
Key points
- Models were fine-tuned from GPT-3 at 760M, 13B and 175B parameters. They act in a text-based browser with commands such as Search, Find in page, Quote, Scroll and End: Answer.
- The team gathered about 6,000 demonstrations (92% ELI5) and about 21,500 comparisons (98% ELI5), with a few extra questions from sources such as TriviaQA. The comparison dataset is publicly released.
- The four training methods were behavior cloning, reward modeling, reinforcement learning and rejection sampling. The best results came from behavior cloning plus rejection sampling. RL helped mainly when inference-time compute was limited.
- On ELI5, the best model's answers were preferred over the human demonstrators' answers 56% of the time and over the top-voted Reddit answer 69% of the time (with references removed for the Reddit comparison).
- On the adversarial TruthfulQA benchmark, 75% of answers were true and 54% were both true and informative. This beats base GPT-3 but remains below human performance.
Cited by
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Qwen Technical Report
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- Enabling Large Language Models to Generate Text with Citations
- RARR: Researching and Revising What Language Models Say, Using Language Models
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Improving alignment of dialogue agents via targeted human judgements
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
- Evaluating Large Language Models: A Comprehensive Survey
- ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
- CogAgent: A Visual Language Model for GUI Agents
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
- TrustLLM: Trustworthiness in Large Language Models
- Survey of Hallucination in Natural Language Generation
- Challenges and Applications of Large Language Models
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Language Models can Solve Computer Tasks
- Understanding the Effects of RLHF on LLM Generalisation and Diversity
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Internet-augmented language models through few-shot prompting for open-domain question answering
- API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Generative Judge for Evaluating Alignment
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- Let's Verify Step by Step
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Evaluating Verifiability in Generative Search Engines
- LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
- Quark: Controllable Text Generation with Reinforced Unlearning
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
- Teaching language models to support answers with verified quotes
- Augmented Language Models: a Survey
- GAIA: a benchmark for General AI Assistants
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- A Comprehensive Overview of Large Language Models
- Capabilities of GPT-4 on Medical Challenge Problems
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Gorilla: Large Language Model Connected with Massive APIs
- Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
- Tool Learning with Foundation Models
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- Self-critiquing models for assisting human evaluators
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- Towards Understanding Sycophancy in Language Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- Reflexion: Language Agents with Verbal Reinforcement Learning
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
- Preference Ranking Optimization for Human Alignment
- ASQA: Factoid Questions Meet Long-Form Answers
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
- LaMDA: Language Models for Dialog Applications
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Solving math word problems with process- and outcome-based feedback
- FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- Executable Code Actions Elicit Better LLM Agents
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
- The Rise and Potential of Large Language Model Based Agents: A Survey
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- Retrieval meets Long Context Large Language Models
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- ReAct: Synergizing Reasoning and Acting in Language Models
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- CREPE: Open-Domain Question Answering with False Presuppositions
- RRHF: Rank Responses to Align Language Models with Human Feedback without tears
- Active Example Selection for In-Context Learning
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- Generative Verifiers: Reward Modeling as Next-Token Prediction
- A Survey of Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- Large Language Models for Information Retrieval: A Survey
- ToolQA: A Dataset for LLM Question Answering with External Tools
Cite
@misc{nakano2021webgpt,
title = {{WebGPT: Browser-assisted question-answering with human feedback}},
author = {Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and Jiang, Xu and Cobbe, Karl and Eloundou, Tyna and Krueger, Gretchen and Button, Kevin and Knight, Matthew and Chess, Benjamin and Schulman, John},
year = {2021},
doi = {10.48550/arxiv.2112.09332},
eprint = {2112.09332},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2112.09332}
}References
- L. Adolphs, B. Boerschinger, C. Buck, M. C. Huebscher, M. Ciaramita, L. Espeholt, T. Hofmann, and Y. Kilcher. Boosting search engines with interactive agents. arXiv preprint arXiv:2109.00527, 2021. — [arXiv:2109.00527](https://arxiv.org/abs/2109.00527) find
- S. Bhakthavatsalam, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, and P. Clark. Think you have solved direct-answer question answering? Try ARC-DA, the direct-answer AI2 reasoning challenge. arXiv preprint arXiv:2102.03315, 2021. — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) find
- N. Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. find
- T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. find
- H. Cheng, Y. Shen, X. Liu, P. He, W. Chen, and J. Gao. UnitedQA: A hybrid approach for open domain question answering. arXiv preprint arXiv:2101.00178, 2021. find
- D. Chong and J. N. Druckman. Framing theory. Annu. Rev. Polit. Sci., 10:103–126, 2007. find
- P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. find
- O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders. Truthful AI: Developing and governing AI that does not lie. arXiv preprint arXiv:2110.06674, 2021. — [arXiv:2110.06674](https://arxiv.org/abs/2110.06674) find
- A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. ELI5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019. find our card
- D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, et al. Building watson: An overview of the deepqa project. AI magazine, 31 (3):59–79, 2010. find
- K. Goddard, A. Roudsari, and J. C. Wyatt. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1): 121–127, 2012. find
- I. Gur, U. Rueckert, A. Faust, and D. Hakkani-Tur. Learning to navigate the web. arXiv preprint arXiv:1812.09195, 2018. — [arXiv:1812.09195](https://arxiv.org/abs/1812.09195) find
- K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. find
- M. Harms. Crystal Society. Crystal Trilogy. CreateSpace Independent Publishing Platform, 2016. ISBN 9781530773718. find
- G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. find
- M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. find our card
- V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020. find our card
- K. Krishna, A. Roy, and M. Iyyer. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332, 2021. find our card
- J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. find
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv preprint arXiv:2005.11401, 2020a. find
- P. Lewis, P. Stenetorp, and S. Riedel. Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint arXiv:2008.02637, 2020b. find
- S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. find our card
- J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. find
- D. Metzler, Y. Tay, D. Bahri, and M. Najork. Rethinking search: Making experts out of dilettantes. arXiv preprint arXiv:2105.02274, 2021. — [arXiv:2105.02274](https://arxiv.org/abs/2105.02274) find
- B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. find
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
- T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017. find
- K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021. find our card
- N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325, 2020. find
- X. Yuan, J. Fu, M.-A. Cote, Y. Tay, C. Pal, and A. Trischler. Interactive machine comprehension with information seeking agents. arXiv preprint arXiv:1908.10449, 2019. find