TruthfulQA: Measuring How Models Mimic Human Falsehoods
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.acl-long.229.pdf
Summary
Language models often produce fluent but false statements. Some of these errors come from imperfect learning, but others appear because human-written training text itself repeats misconceptions. The authors call the second kind imitative falsehoods. To measure them, they built TruthfulQA, a zero-shot benchmark of 817 author-written questions in 38 categories, including health, law, finance, politics, conspiracies and fiction. Each question has true and false reference answers and a supporting source. Part of the set was adversarially filtered against GPT-3-175B. Humans rated answers for truthfulness and informativeness. An answer counts as truthful if it makes no false claim, so declining to answer is acceptable. A multiple-choice variant and a fine-tuned GPT-3 evaluator allow cheap automatic scoring. The authors tested GPT-3, GPT-Neo/J, GPT-2 and the T5-based UnifiedQA. The best model was truthful on only 58% of questions, compared with 94% for humans. Within model families, larger models were generally less truthful. This suggests that scaling alone will not fix truthfulness and that training objectives other than imitating web text are needed.
Key points
- Dataset: TruthfulQA has 817 questions in 38 categories, with a median length of 9 words. Of these, 437 are "filtered" questions, kept because GPT-3-175B tended to answer them wrongly. The other 380 were written later without that test. The data and code are at github.com/sylinrl/TruthfulQA.
- Main result: The best model was GPT-3-175B with a "helpful" prompt, truthful on 58% of questions against a 94% human baseline. It also gave false but informative answers 42% of the time, compared with 6% for humans.
- Inverse scaling: Across GPT-3, GPT-Neo/J, GPT-2 and UnifiedQA, the largest models tended to score worst. For example, the 175B GPT-3 repeats the superstition about seven years of bad luck after breaking a mirror, while smaller models do not.
- Automated judge: A GPT-3 model fine-tuned on human truth labels predicted human judgments with 90–96% accuracy on held-out models. A similar informativeness judge reached 86.3% on UnifiedQA.
- Strict standard of truth: Claims count as true only if they are literally true about the real world, not true within a belief system such as astrology. The benchmark does not cover long-form or interactive generation.
Cited by
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Refusal in Language Models Is Mediated by a Single Direction
- A General Language Assistant as a Laboratory for Alignment
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- A Survey on Evaluation of Large Language Models
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
- Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Improving alignment of dialogue agents via targeted human judgements
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
- Evaluating Large Language Models: A Comprehensive Survey
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- TrustLLM: Trustworthiness in Large Language Models
- Qwen2.5-Coder Technical Report
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
- Survey of Hallucination in Natural Language Generation
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
- Challenges and Applications of Large Language Models
- BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Measuring Faithfulness in Chain-of-Thought Reasoning
- CMMLU: Measuring massive multitask language understanding in Chinese
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- Teaching language models to support answers with verified quotes
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- Large Language Models: A Survey
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- Orca 2: Teaching Small Language Models How to Reason
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- WebGPT: Browser-assisted question-answering with human feedback
- A Comprehensive Overview of Large Language Models
- Large Language Diffusion Models
- GPT-4 Technical Report
- GPT-4o System Card
- Discovering Language Model Behaviors with Model-Written Evaluations
- Red Teaming Language Models with Language Models
- Tool Learning with Foundation Models
- Leveraging Large Language Models for Multiple Choice Question Answering
- Code Llama: Open Foundation Models for Code
- Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
- Personality Traits in Large Language Models
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
- Towards Understanding Sycophancy in Language Models
- Prompting GPT-3 To Be Reliable
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Aligning Large Multimodal Models with Factually Augmented RLHF
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- Galactica: A Large Language Model for Science
- How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- RedPajama: an Open Dataset for Training Large Language Models
- Emergent Abilities of Large Language Models
- Long-form factuality in large language models
- Measuring short-form factuality in large language models
- Ethical and social risks of harm from Language Models
- LESS: Selecting Influential Data for Targeted Instruction Tuning
- Effective Long-Context Scaling of Foundation Models
- Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
- WizardLM: Empowering large pre-trained language models to follow complex instructions
- Hallucination is Inevitable: An Innate Limitation of Large Language Models
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
- Large Language Model Unlearning
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- GLM-130B: An Open Bilingual Pre-trained Model
- Instruction Tuning for Large Language Models: A Survey
- How Language Model Hallucinations Can Snowball
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- A Survey of Large Language Models
- Large Language Models Are Human-Level Prompt Engineers
- Representation Engineering: A Top-Down Approach to AI Transparency
Cite
@inproceedings{lin2022truthfulqa,
title = {{TruthfulQA: Measuring How Models Mimic Human Falsehoods}},
author = {Lin, Stephanie and Hilton, Jacob and Evans, Owain},
booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2022},
doi = {10.18653/v1/2022.acl-long.229},
eprint = {2109.07958},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/2022.acl-long.229}
}References
- Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: fact extraction and verification over unstructured and structured information. CoRR, abs/2106.05707. — [arXiv:2106.05707](https://arxiv.org/abs/2106.05707) find
- Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam Mc-Candlish, Chri… find our card
- Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3528–3539, Brussels, Belgium. Association for Com… find
- Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association… find
- Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. 2021. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.033… — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) find
- Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
- Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Doro… find our card
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. find our card
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. find
- CohereAI. 2021. co:here api. https://cohere. ai/api. Accessed: 2021-08-19. find
- Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online. Association… find our card
- Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2021. Scarecrow: A framework for scrutinizing machine text. CoRR, abs/2107.01294. — [arXiv:2107.01294](https://arxiv.org/abs/2107.01294) find
- Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: developing and governing AI that does not lie. CoRR, abs/2110.06674. — [arXiv:2110.06674](https://arxiv.org/abs/2110.06674) find
- William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961. find
- Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 478–487, Online. Association for Computational Linguistics. find
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. CoRR, abs/2009.03300. find our card
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. find our card
- Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… find our card
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361. find
- Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. CoRR, abs/2103.14659. find
- Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Com… find our card
- Sarah Kreps, R. Miles McCain, and Miles Brundage. 2020. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation. Journal of Experimental Political Science, page 1–14. find
- Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940–4957, Online. Association for Computat… find our card
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
- Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. CoRR, abs/2005.11401. find
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
- Robert L Logan IV, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife hillary: Using knowledge-graphs for fact-aware language modeling. arXiv preprint arXiv:1906.07241. find
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. find
- Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of GPT-3 and advanced neural language models. CoRR, abs/2009.06807. — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) find
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assis… find our card
- OpenAI. 2020. Openai api. https://openai. com/blog/openai-api/. Accessed: 2021-08-19. find
- OpenAI. 2021. Gpt-3 powers the next generation of apps. https://openai.com/blog/ gpt-3-apps/. Accessed: 2021-09-06. ↗
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almieda, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2021. Train… find
- Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. CoRR, abs/2105.11447. find our card
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… find
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. find
- Arpit Rajauria. 2020. tuner007/pegasus\_paraphrase. https://huggingface.co/tuner007/ pegasus\_paraphrase. Accessed: 2021-08-16. ↗
- Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational… find our card
- Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. 2019. Are we safe yet? the limitations of distributional features for fake news detection. CoRR, abs/1908.09805. — [arXiv:1908.09805](https://arxiv.org/abs/1908.09805) find
- Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
- Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. find
- Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567. find our card
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
- Irene Solaiman and Christy Dennison. 2021. Process for adapting language models to society (palms) with values-targeted datasets. https://cdn. openai.com/palms.pdf. find
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize from human feedback. CoRR, abs/2009.01325. find
- Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. CoRR, abs/2010.07079. find
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616. find our card
- Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021. Understanding the capabilities, limitations, and societal impact of large language models. CoRR, abs/2102.02503. find
- Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. As… find
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
- Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. CoRR, abs/1912.08777. find our card
- Eric Wallace and Jordan Boyd-Graber. 2018. Trick me if you can: Adversarial writing of trivia challenge questions. In Proceedings of ACL 2018, Student Research Workshop, pages 127–133, Melbourne, Australia. Association for Computational Linguistics. find
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. find
- Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/ kingoflolz/mesh-transformer-jax. ↗