Emergent Abilities of Large Language Models

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

1227 papers of our library cite it (of the 140673 whose bibliographies we read); 88 of them have a page here, 1139 not yet.

• Quiet-STaR: Language models can teach themselves to think before speaking
• The Falcon Series of Open Language Models
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
• Qwen Technical Report
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• A Survey on Evaluation of Large Language Models
• TheoremQA: A Theorem-driven Question Answering Dataset
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
• Promptagator: Few-shot Dense Retrieval From 8 Examples
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• A Survey on In-context Learning
• Faith and Fate: Limits of Transformers on Compositionality
• Complexity-Based Prompting for Multi-Step Reasoning
• Specializing Smaller Language Models towards Multi-Step Reasoning
• MiniLLM: On-Policy Distillation of Large Language Models
• FOLIO: Natural Language Reasoning with First-Order Logic
• How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
• Large Language Models Can Self-Improve
• Towards Reasoning in Large Language Models: A Survey
• TrustLLM: Trustworthiness in Large Language Models
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• Survey of Hallucination in Natural Language Generation
• A Survey on Large Language Models for Code Generation
• Challenges and Applications of Large Language Models
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker
• Deductive Verification of Chain-of-Thought Reasoning
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision
• LLM-Pruner: On the Structural Pruning of Large Language Models
• Augmented Language Models: a Survey
• Orca 2: Teaching Small Language Models How to Reason
• Progress measures for grokking via mechanistic interpretability
• A Comprehensive Overview of Large Language Models
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages
• GPT-4 Technical Report
• Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Towards Making the Most of ChatGPT for Machine Translation
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
• ChatDev: Communicative Agents for Software Development
• Reasoning with Language Model Prompting: A Survey
• Tool Learning with Foundation Models
• Leveraging Large Language Models for Multiple Choice Question Answering
• Are Emergent Abilities of Large Language Models a Mirage?
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Personality Traits in Large Language Models
• Role-Play with Large Language Models
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• Language Models are Multilingual Chain-of-Thought Reasoners
• Distilling Reasoning Capabilities into Smaller Language Models
• Prompting GPT-3 To Be Reliable
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• UL2: Unifying Language Learning Paradigms
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• SCOTT: Self-Consistent Chain-of-Thought Distillation
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Jailbroken: How Does LLM Safety Training Fail?
• Larger language models do in-context learning differently
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• BloombergGPT: A Large Language Model for Finance
• The Rise and Potential of Large Language Model Based Agents: A Survey
• Self-Evaluation Guided Beam Search for Reasoning
• Retrieval meets Long Context Large Language Models
• Hallucination is Inevitable: An Innate Limitation of Large Language Models
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
• Generate rather than Retrieve: Large Language Models are Strong Context Generators
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• GLM-130B: An Open Bilingual Pre-trained Model
• Active Example Selection for In-Context Learning
• Multimodal Chain-of-Thought Reasoning in Language Models
• Prompting Large Language Model for Machine Translation: A Case Study
• A Survey of Large Language Models
• Large Language Models Are Human-Level Prompt Engineers
• Large Language Models for Information Retrieval: A Survey
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.07682 · arXiv 2206.07682

@misc{wei2022emergent,
  title = {{Emergent Abilities of Large Language Models}},
  author = {Wei, Jason and Tay, Yi and Bommasani, Rishi and Raffel, Colin and Zoph, Barret and Borgeaud, Sebastian and Yogatama, Dani and Bosma, Maarten and Zhou, Denny and Metzler, Donald and Chi, Ed H. and Hashimoto, Tatsunori and Vinyals, Oriol and Liang, Percy and Dean, Jeff and Fedus, William},
  year = {2022},
  doi = {10.48550/arxiv.2206.07682},
  eprint = {2206.07682},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.07682}
}

References

103 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend, Tom Kwiatkowski, Nathaniel J Smith, Sharon Goldwater, and Mark Steedman. Bootstrapping language acquisition. Cognition, 164:116–143, 2017. URL https://homepages.inf.ed.ac.uk/ sgwater/papers/cognition17-bootstrapping.pdf. ↗
2. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. URL https://arxiv.org/ abs/2204.… ↗ our card
3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS, 2022. URL https://arxiv.org/abs/2204.14198. ↗ our card
4. Philip W. Anderson. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science, 177(4047):393–396, 1972. URL http://www.lanais.famaf.unc.edu.ar/cursos/ em/Anderson-MoreDifferent-1972.pdf. ↗
5. Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441, 2022. URL https://arxiv.org/abs/2210.02441. ↗
6. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684, 2021. URL https://arxiv.org/abs/2112.10684. ↗
7. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. URL https://arxiv.org/abs/2112.00861. ↗ our card
8. Fabrice Bellard. gpt2tc: Text completion and compression using GPT-2, 2021. URL https://bellard. org/libnc/gpt2tc.html. Accessed Apr. 26, 2022. find
9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021. URL https://dl.acm.org/doi/pdf/10.1145/3442188. 3445922… — [doi:10.1145/3442188](https://doi.org/10.1145/3442188) ↗
10. BIG-Bench. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. URL https://arxiv.org/abs/2206.04615. ↗ our card
11. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. URL https://arxiv.org/abs/2108. 07258. ↗ our card
12. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426, 2021. URL https: //a… find
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. URL https://papers.nips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstrac… ↗
14. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. USENIX Security, 2021. URL https://www.usenix.org/conference/usenixsecurity21/ prese… ↗ our card
15. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. URL https://arxiv.org/abs/2202.07646. ↗
16. Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055, 2022. URL https://arxiv.org/abs/2205.05055. ↗
17. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗
18. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL https://arxiv.org/abs/2210.11416. ↗
19. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
20. Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. Wordcraft: A human-AI collaborative editor for story writing. arXiv preprint arXiv:2107.07430, 2021. URL https://arxiv.org/abs/2107. 07430. — [arXiv:2107.07430](https://arxiv.org/abs/2107.07430) ↗
21. Antonella Corradini and Timothy O’Connor. Emergence in science and philosophy, volume 6. Routledge, 2010. URL https://books.google.com/books?hl=en&lr=&id=55RaBwAAQBAJ&oi= fnd&pg=PP1&dq=Emergence+in+science+and+philosophy&ots=2\_8VNDXLfv&sig=1aisq\_ WouF95Cx58WWMZ0Gq3RNk. ↗
22. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. URL https://arxiv.org/abs/1807.03819. ↗
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. URL https://aclanthology.org/ N19-1423. ↗
24. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mixture-of-experts. ICML, 2021. URL https://arxiv.org/abs/2112.06905. ↗ our card
25. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961. ↗
26. Stephanie Forrest. Emergent computation: Self-organizing, collective, and cooperative phenomena in natural and artificial computing networks. Physica D: Nonlinear Phenomena, 42(1-3):1–11, 1990. URL https://www.sciencedirect.com/science/article/abs/pii/016727899090063U. — [arXiv:pii/0167278](https://arxiv.org/abs/pii/0167278) ↗
27. Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, et al. Predictability and surprise in large generative models. arXiv preprint arXiv:2202.07785, 2022. URL https://arxiv.org/abs/2202.07785. ↗
28. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. ACL, 2021. doi: 10.18653/v1/2021.acl-long.295. URL https://aclanthology.org/2021.acl-long.295. ↗ our card
29. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP, 2020. doi: 10.18653/v1/ 2020.findings-emnlp.301. URL https://aclanthology.org/2020.findings-emnlp.301. ↗ our card
30. Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016. URL https://arxiv.org/abs/1603.08983. ↗
31. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ICML, 2020. URL https://arxiv.org/abs/2002.08909. ↗
32. David A. Harper and Paul A. Lewis. New perspectives on emergence in economics. New Perspectives on Emergence in Economics, pp. 2–3, 2012. URL https://www.sciencedirect. com/science/article/pii/S0167268112000200?casa\_token=fLs2nCYo\_64AAAAA: H2sSpSygJmEqXgmpM4jLyeppph3C4TgEsaSXm5RkOpT0r4q2A1x9Su3u4uyc… ↗
33. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2021a. URL https://openreview.net/ forum?id=d7KBjmI3GmQ. ↗ our card
34. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916, 2021b. URL https://arxiv.org/abs/2109.13916. ↗
35. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. NeurIPS, 2022. URL https://arxiv.org/abs/2203.15556. ↗
36. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. EMNLP, 2021. URL https://aclanthology. org/2021.emnlp-main.564. find our card
37. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022. URL https://arxiv.org/pdf/2201.07207. ↗ our card
38. Bernardo A. Huberman and Tad Hogg. Phase transitions in artificial intelligence systems. Artificial Intelligence, 33(2):155–171, 1987. URL https://www.sciencedirect.com/science/article/ abs/pii/0004370287900336. — [arXiv:pii/0004370](https://arxiv.org/abs/pii/0004370) ↗
39. Harold Y. Hwang, Yoh Iwasa, Masashi Kawasaki, Bernhard Keimer, Naoto Nagaosa, and Yoshinori Tokura. Emergent phenomena at oxide interfaces. Nature Materials, 11(2):103–113, 2012. URL https://www. nature.com/articles/nmat3223. find
40. Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. ICML, 2017. URL https://arxiv.org/abs/1608.05343. — [arXiv:1608.05343](https://arxiv.org/abs/1608.05343) ↗
41. Dan Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall series in Artificial Intelligence. Pearson Prentice Hall, 2009. ISBN 9780131873216. URL https://books.google.com/books?id… ↗
42. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. URL https://arxiv.org/abs/2207.05221. ↗
43. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. ICML, 2022. URL https://arxiv.org/abs/2202.06539. ↗ our card
44. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. ↗
45. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS, 2022. URL https://arxiv.org/abs/2205.11916. ↗ our card
46. Andrew K. Lampinen, Ishita Dasgupta, Stephanie C.Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. Can language models learn from explanations in context? Findings of EMNLP, 2022. URL https://arxiv.org/abs/2204.02329. ↗ our card
47. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. ACL, 2022a. URL https://arxiv.org/abs/2107.06499. ↗
48. Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. CHI, 2022b. URL https://arxiv.org/abs/2201.06796. ↗
49. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ICLR, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb. ↗
50. Percy Liang. Semi-supervised learning for natural language. PhD thesis, Massachusetts Institute of Technology, 2005. URL https://www-cs.stanford.edu/~pliang/papers/meng-thesis.pdf. ↗
51. Percy Liang, Rishi Bommasani, Kathleen A. Creel, and Rob Reich. The time is now to develop community norms for the release of foundation models, 2022. URL https://crfm.stanford.edu/2022/05/17/ community-norms.html. ↗
52. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. URL https://arxiv.org/abs/2109.07958. ↗ our card
53. Christopher D. Manning. Human language understanding & reasoning. Daedalus, 151(2):127–138, 2022. URL https://www.amacad.org/publication/human-language-understanding-reasoning. ↗
54. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020. URL https://www.pnas.org/doi/10.1073/pnas. 1907367… — [doi:10.1073/pnas](https://doi.org/10.1073/pnas) ↗
55. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. URL https://arxiv. org/abs/1806.08730. find our card
56. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://huggingface.co/datasets/wikitext. ↗ our card
57. Scott Miller, Jethran Guinness, and Alex Zamanian. Name tagging with word clusters and discriminative training. In NAACL, 2004. URL https://aclanthology.org/N04-1043. ↗
58. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. ACL, 2022a. URL https://arxiv.org/abs/2108.04106. ↗ our card
59. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022b. URL https://arxiv.org/abs/2202.12837. ↗ our card
60. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. URL https://openrev… find our card
61. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, et al. In-context learning and induction heads. Transformer Circuits, 2022. URL https: //transformer-circuits.pub/2022/in-context-learning-and-induction-heads/ index.html. find
62. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv.org/abs/2203.02155. ↗
63. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL, 2002. URL https://aclanthology.org/P02-1040.pdf. ↗ our card
64. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of ACL, 2022. URL https://arxiv.org/abs/2110.08193. ↗ our card
65. Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. ICLR, 2022. URL https://openreview.net/forum?id=gJcEM8sxHK. ↗
66. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. URL https://arxiv.org/abs/2202.03286. ↗ our card
67. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. NAACL, 2019. URL https://aclanthology.org/ N19-1128. ↗ our card
68. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8), 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language\_models\_ are\_unsupervised\_multitask\_learners.pdf. ↗
69. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. URL https://arxiv.org/abs… ↗
70. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL https://jmlr.org/papers/v21/20-074.html. ↗
71. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. URL https://arxiv.org/ abs/2204.06125. ↗ our card
72. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206, 2022. URL https://arxiv.org/ abs/2202.07206. — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) ↗
73. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. URL https://arxiv.org/abs/2102.07350. ↗ our card
74. Rachel Rudinger, Chandler May, and Benjamin Van Durme. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, 2017. URL https://aclanthology.org/W17-1609. ↗ our card
75. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4. ↗ our card
76. Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. ICLR, 2021. URL https://arxiv.org/abs/2010.03648. — [arXiv:2010.03648](https://arxiv.org/abs/2010.03648) ↗
77. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. NAACL, June 2021. URL https://aclanthology.org/2021.naacl-main.185. ↗ our card
78. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022. URL https://arxiv.org/abs/ 221… ↗ our card
79. Jacob Steinhardt. On the risks of emergent behavior in foundation models, October 2021. URL https://bounded-regret.ghost.io/ on-the-risks-of-emergent-behavior-in-foundation-models/. Accessed Apr 13, 2022. ↗
80. Jacob Steinhardt. Future ml systems will be qualitatively different, 2022. URL https://bounded-regret. Accessed May 20, ghost.io/future-ml-systems-will-be-qualitatively-different/. 2022. find
81. Mirac Suzgun, Nathan Scales, Nathaneal Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny ZHou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. URL https: //arxiv.org/a… find our card
82. Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022a. URL https://arxiv.org/abs/2205.05131. ↗ our card
83. Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06991, 2022b. URL https://arxiv.org/abs/2202.06991. ↗ our card
84. Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399, 2022c. URL https://arxiv.org/abs/2210.11399. ↗
85. Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. In ACL Big Science Workshop, 2022. URL https://aclanthology.org/2022.bigscience-1.11/. ↗
86. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239. ↗ our card
87. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. URL https://arxiv.org/abs/1806.02847. ↗
88. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? ICML, 2022a. URL https://arxiv.org/abs/2204.05832. ↗
89. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b. URL https: //arxiv.org/abs/2203.11171. find our card
90. Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning. NeurIPS, 2021a. URL https://openreview.net/ forum?id=MDMV2SxCboX. ↗
91. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. Frequency effects on syntactic rule learning in transformers. EMNLP, 2021b. doi: 10.18653/v1/2021.emnlp-main.72. URL https://aclanthology. org/2021.emnlp-main.72. find
92. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ICLR, 2022a. URL https: //openreview.net/forum?id=gEZrGCozdqR. find
93. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. NeurIPS, 2022b. URL https://arxiv. org/abs/2201.11903. find our card
94. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. URL https://arxiv.org/abs/2112.04359. ↗ our card
95. Tongshuang Wu, Michael Terry, and Carrie J. Cai. AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. arXiv preprint arXiv:2110.01691, 2021. URL https://arxiv.org/abs/2110.01691. — [arXiv:2110.01691](https://arxiv.org/abs/2110.01691) ↗
96. Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J. Cai. PromptChainer: Chaining large language model prompts through visual programming. arXiv preprint arXiv:2203.06566, 2022a. URL https://arxiv.org/abs/2203.06566. — [arXiv:2203.06566](https://arxiv.org/abs/2203.06566) ↗
97. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022b. find
98. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ICLR, 2022. URL https://arxiv.org/abs/2111.02080. ↗ our card
99. Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. URL https://arxiv. org/a… find our card
100. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. When do you need billions of words of pretraining data? In ACL, 2021. doi: 10.18653/v1/2021.acl-long.90. URL https://aclanthology. org/2021.acl-long.90. find
101. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. ICML, 2021. URL https://arxiv.org/abs/2102.09690. ↗ our card
102. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. URL https://arxiv.org/abs/2205.10625. ↗ our card
103. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906, 2022. URL https: //arxiv.org/abs/2202.08906. find