Elamur Research
☰
home / papers / 2022 / Teaching language models to support answers with verified quotes

Teaching language models to support answers with verified quotes

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2203.11147

Summary

Large language models frequently get fact-based queries right, but users cannot tell which claims to trust without checking them, because models can invent convincing falsehoods. This paper presents GopherCite, a system built by finetuning the 280B-parameter Gopher model. It pairs each answer with a verbatim excerpt from a source so a reader can verify the claim directly. The authors call this task "self-supported question-answering" (SQA). Their "Inline Evidence" approach generates the answer and its quote as one string, using a dedicated markup and constrained sampling so that every quote exactly matches the retrieved text. Sources come from Google Search results or a user-supplied document. Training combines supervised finetuning with reinforcement learning from human preferences (RLHP). A reward model learned from rater judgments also reranks candidate answers and decides when the system should abstain. In human evaluation on filtered NaturalQuestions and ELI5 subsets, most answers were judged both plausible and supported, and abstention increased reliability further. Tests on TruthfulQA show that well-cited claims can still be false, so citation is only one part of building trustworthy models.

Key points

  • Output format: %<Claim>%(Document title)%[Quote from document]%. Constrained decoding guarantees that quotes are copied exactly from the context documents.
  • Human raters judged answers on two criteria: "plausible" (a sensible, on-topic reply) and "supported" (the quote is enough to verify the answer).
  • At inference, the system retrieves the top-K Google Search results and draws N>K samples, each conditioned on one truncated document. The reward model then picks the best sample. This reward model predicts pairwise human preferences and also classifies each answer as plausible or supported.
  • Answers were high-quality 80% of the time on the NaturalQuestions subset and 67% on the ELI5 subset. Declining the roughly one-third of questions where the model was least confident raised these to 90% and 80%. The NaturalQuestions figure exceeds the human baseline when humans answer every question.
  • Compared with related systems, LaMDA shows only URLs, while WebGPT issues its own search queries. GopherCite instead reads thousands of tokens of uncurated multi-page context. TruthfulQA results reveal it can select misleading evidence, so source trustworthiness and cherry-picking remain open problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

159 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.11147 · arXiv 2203.11147

@misc{menick2022teaching,
  title = {{Teaching language models to support answers with verified quotes}},
  author = {Menick, Jacob and Trebacz, Maja and Mikulik, Vladimir and Aslanides, John and Song, Francis and Chadwick, Martin and Glaese, Mia and Young, Susannah and Campbell-Gillingham, Lucy and Irving, Geoffrey and McAleese, Nat},
  year = {2022},
  doi = {10.48550/arxiv.2203.11147},
  eprint = {2203.11147},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.11147}
}

References

65 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Akrour, M. Schoenauer, and M. Sebag. Preference-based policy learning. In D. Gunopulos, T. Hofmann, D. Malerba, and M. Vazirgiannis, editors, Machine Learning and Knowledge Discovery in Databases, pages 12–27, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-23780-5. find
  2. L. Aroyo and C. Welty. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24, Mar. 2015. doi: 10.1609/aimag.v36i1.2564. URL https://ojs.aaai. org/index.php/aimagazine/article/view/2564 . ↗ our card
  3. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. Das-Sarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A general language assistant as a laborat… find our card
  4. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonya… find
  5. G. Brockman, M. Murati, P. Welinder, and OpenAI. Openai api. https://openai.com/blog/openai-api/ , 2020. Accessed: 2022-02-19. ↗
  6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  7. H. Cheng, Y. Shen, X. Liu, P. He, W. Chen, and J. Gao. Unitedqa: A hybrid approach for open domain question answering. CoRR, abs/2101.00178, 2021. URL https://arxiv.org/abs/2101.00178 . ↗
  8. P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences, 2017. URL https://arxiv.org/abs/1706.03741 . ↗
  9. P. F. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. CoRR, abs/1810.08575, 2018. URL http://arxiv.org/abs/1810.08575 . ↗
  10. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 . ↗ our card
  11. Cohere. Cohere api \| cohere. https://cohere.ai/api , 2021. Accessed: 2022-02-19. ↗
  12. R. El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010. find
  13. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. Eli5: Long form question answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558– 3567, 2019. find our card
  14. Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017. find
  15. Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, pages 2151–2159. PMLR, 2019. find
  16. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737–1746. PMLR, 2015. find
  17. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. REALM: retrieval-augmented language model pre-training. CoRR, abs/2002.08909, 2020. URL https://arxiv.org/abs/2002.08909 . ↗
  18. C. Hube. Bias in wikipedia. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW ’17 Companion, page 717–721, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 9781450349147. doi: 10.1145/3041021.3053375. URL h… — [doi:10.1145/3041021.3053375](https://doi.org/10.1145/3041021.3053375) find
  19. G. Irving, P. Christiano, and D. Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899 . ↗
  20. G. Izacard and E. Grave. Leveraging passage retrieval with generative models for open domain question answering. CoRR, abs/2007.01282, 2020. URL https://arxiv.org/abs/2007.01282 . ↗ our card
  21. A. Jacovi and Y. Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online, July 2020. Association for Computational Linguistics. doi: 10.… find our card
  22. N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645–1654. PMLR, 2017. find
  23. N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, À. Lapedriza, N. Jones, S. Gu, and R. W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456, 2019. URL http://arxiv.org/abs/1907.00456 . — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) ↗
  24. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. CoRR, abs/1705.03551, 2017. URL http://arxiv.org/abs/1705.03551 . ↗ our card
  25. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8: 64–77, 2020. find our card
  26. A. Kamath, R. Jia, and P. Liang. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462, 2020. URL https://arxiv.org/abs/2006.09462 . ↗
  27. V. Karpukhin, B. Oguz, S. Min, L. Wu, S. Edunov, D. Chen, and W. Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, abs/2004.04906, 2020. URL https://arxiv.org/abs/2004.04906 . ↗ our card
  28. N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858, 2019. URL http://arxiv. org/abs/1909.05858 . find our card
  29. M. Komeili, K. Shuster, and J. Weston. Internet-augmented dialogue generation. abs/2107.07566, 2021. URL https://arxiv.org/abs/2107.07566 . ↗ our card
  30. T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018. find our card
  31. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019. find our card
  32. M. Lamm, J. Palomaki, C. Alberti, D. Andor, E. Choi, L. B. Soares, and M. Collins. QED: A framework and dataset for explanations in question answering. CoRR, abs/2009.06354, 2020. URL https: //arxiv.org/abs/2009.06354 . find
  33. V. Latcinnik and J. Berant. Explaining question answering models through text generation. CoRR, abs/2004.05569, 2020. URL https://arxiv.org/abs/2004.05569 . — [arXiv:2004.05569](https://arxiv.org/abs/2004.05569) ↗
  34. A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. URL https://arxiv.org/abs/2203.05115 . ↗ our card
  35. J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871, 2018. URL http://arxiv.org/abs/1811.07871 . ↗
  36. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021a. find
  37. P. Lewis, P. Stenetorp, and S. Riedel. Question and answer test-train overlap in open-domain question answering datasets. In EACL, 2021b. find
  38. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs, Aug. 2021. find
  39. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. CoRR, abs/2109.07958, 2021. URL https://arxiv.org/abs/2109.07958 . ↗ our card
  40. A. Liska, T. Kocisky, E. Gribovskaya, T. Terzi, E. Sezener, D. Agrawal, C. de Masson d’Autume, T. Scholtes, M. Zaheer, S. Young, E. Gilsenan-McMahon, S. Austin, and A. Lazaridou. Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. Forthcoming, 2022. find
  41. CoRR, A. Liu, S. Swayamdipta, N. A. Smith, and Y. Choi. WANLI: worker and AI collaboration for natural language inference dataset creation. CoRR, abs/2201.05955, 2022. URL https://arxiv.org/abs/2201.05955 . ↗ our card
  42. B. Martin. Persistent bias on wikipedia: Methods and responses. Social Science Computer Review, 36(3):379–388, 2018. doi: 10.1177/0894439317715434. URL https://doi.org/10.1177/0894439317715434 . — [doi:10.1177/0894439317715434](https://doi.org/10.1177/0894439317715434) ↗
  43. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. find
  44. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332, 2021. URL https… find our card
  45. S. Narang, C. Raffel, K. Lee, A. Roberts, N. Fiedel, and K. Malkan. Wt5?! training text-to-text models to explain their predictions. CoRR, abs/2004.14546, 2020. URL https://arxiv.org/abs/2004.14546 . ↗
  46. R. Y. Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. R. Bowman. Quality: Question answering with long input texts, yes! CoRR, abs/2112.08608, 2021. URL https://arxiv.org/abs/2112.08608 . ↗ our card
  47. E. Perez, S. Karamcheti, R. Fergus, J. Weston, D. Kiela, and K. Cho. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  48. E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. CoRR, abs/2202.03286, 2022. URL https: //arxiv.org/abs/2202.03286 . find our card
  49. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mel… find
  50. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.1865… find our card
  51. G. Ras, N. Xie, M. van Gerven, and D. Doran. Explainable deep learning: A field guide for the uninitiated. CoRR, abs/2004.14545, 2020. URL https://arxiv.org/abs/2004.14545 . — [arXiv:2004.14545](https://arxiv.org/abs/2004.14545) ↗
  52. A. Roberts, C. Raffel, and N. Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18… find
  53. S. Schmitt, J. J. Hudson, A. Zídek, S. Osindero, C. Doersch, W. M. Czarnecki, J. Z. Leibo, H. Küttler, A. Zisserman, K. Simonyan, and S. M. A. Eslami. Kickstarting deep reinforcement learning. CoRR, abs/1803.03835, 2018. URL http://arxiv.org/abs/1803.03835 . — [arXiv:1803.03835](https://arxiv.org/abs/1803.03835) ↗
  54. M. Schoenauer, R. Akrour, M. Sebag, and J.-C. Souplet. Programming by feedback. In E. P. Xing and T. Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1503–1511, Bejing, China, 22–24 Jun 2014. PMLR. URL… find
  55. N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018. find
  56. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053 . ↗
  57. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zheng, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro. Using deepspeed and megatron to train megatron-turing NL… find our card
  58. N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https://arxiv.org/abs/2009.01325 . ↗
  59. R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon,… find our card
  60. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p… find our card
  61. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. URL https://arxiv.org/abs/2112.04359 . ↗ our card
  62. C. Wirth, J. Fürnkranz, and G. Neumann. Model-free preference-based reinforcement learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 2222–2228. AAAI Press, 2016. find
  63. C. Wirth, R. Akrour, G. Neumann, and J. Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46, 2017. URL http://jmlr. org/papers/v18/16-634.html . find
  64. W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang, X. Zhang, C. Li, Z. Gong, Y. Yao, X. Huang, J. Wang, J. Yu, Q. Guo, Y. Yu, Y. Zhang, J. Wang, H. Tao, D. Yan, Z. Yi, F. Peng, F. Jiang, H. Zhang, L. Deng, Y. Zhang, Z. Lin, C. Zhang, S. Zhang, M. Guo, S. Gu, G. Fan, Y. Wa… find
  65. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. F. Christiano, and G. Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL http: //arxiv.org/abs/1909.08593 . find