Multitask Prompted Training Enables Zero-Shot Task Generalization

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1028 papers of our library cite it (of the 140673 whose bibliographies we read); 104 of them have a page here, the first 100 below, 924 not yet.

• SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
• Flamingo: a Visual Language Model for Few-Shot Learning
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• The Falcon Series of Open Language Models
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• A General Language Assistant as a Laboratory for Alignment
• Llemma: An Open Language Model For Mathematics
• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• Qwen Technical Report
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
• Promptagator: Few-shot Dense Retrieval From 8 Examples
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• Precise Zero-Shot Dense Retrieval without Relevance Labels
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models
• MiniLLM: On-Policy Distillation of Large Language Models
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
• Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
• Large Language Models Can Self-Improve
• Editing Models with Task Arithmetic
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• A Survey on Large Language Models for Code Generation
• Challenges and Applications of Large Language Models
• Language Models can Solve Computer Tasks
• SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
• Large language models are zero-shot reasoners
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Can language models learn from explanations in context?
• What matters when building vision-language models?
• Multi-step Jailbreaking Privacy Attacks on ChatGPT
• Self-Alignment with Instruction Backtranslation
• Few-shot Learning with Multilingual Language Models
• Deductive Verification of Chain-of-Thought Reasoning
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
• AgentBench: Evaluating LLMs as Agents
• DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• LILA: A Unified Benchmark for Mathematical Reasoning
• Reframing Instructional Prompts to GPTk's Language
• Crosslingual Generalization through Multitask Finetuning
• Generative Representational Instruction Tuning
• Scalable Extraction of Training Data from (Production) Language Models
• A Comprehensive Overview of Large Language Models
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Gorilla: Large Language Model Connected with Massive APIs
• Instruction Tuning with GPT-4
• Discovering Language Model Behaviors with Model-Written Evaluations
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
• Tool Learning with Foundation Models
• Improving Passage Retrieval with Zero-Shot Question Generation
• Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• mGPT: Few-Shot Learners Go Multilingual
• Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• One Embedder, Any Task: Instruction-Finetuned Text Embeddings
• Aligning Large Multimodal Models with Factually Augmented RLHF
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• UL2: Unifying Language Learning Paradigms
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
• MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• RedPajama: an Open Dataset for Training Large Language Models
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Emergent Abilities of Large Language Models
• BloombergGPT: A Large Language Model for Finance
• The Rise and Potential of Large Language Model Based Agents: A Survey
• LESS: Selecting Influential Data for Targeted Instruction Tuning
• C-Pack: Packed Resources For General Chinese Embeddings
• An Explanation of In-context Learning as Implicit Bayesian Inference
• UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
• Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
• WizardLM: Empowering large pre-trained language models to follow complex instructions
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• TIES-Merging: Resolving Interference When Merging Models
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
• MAmmoTH2: Scaling Instructions from the Web
• GLM-130B: An Open Bilingual Pre-trained Model
• AgentTuning: Enabling Generalized Agent Abilities for LLMs
• AlignScore: Evaluating Factual Consistency with A Unified Alignment Function
• Instruction Tuning for Large Language Models: A Survey
• How Language Model Hallucinations Can Snowball

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2110.08207 · arXiv 2110.08207

@misc{sanh2021multitask,
  title = {{Multitask Prompted Training Enables Zero-Shot Task Generalization}},
  author = {Sanh, Victor and Webson, Albert and Raffel, Colin and Bach, Stephen H. and Sutawika, Lintang and Alyafeai, Zaid and Chaffin, Antoine and Stiegler, Arnaud and Scao, Teven Le and Raja, Arun and Dey, Manan and Bari, M Saiful and Xu, Canwen and Thakker, Urmish and Sharma, Shanya Sharma and Szczechla, Eliza and Kim, Taewoon and Chhablani, Gunjan and Nayak, Nihal and Datta, Debajyoti and Chang, Jonathan and Jiang, Mike Tian-Jian and Wang, Han and Manica, Matteo and Shen, Sheng and Yong, Zheng Xin and Pandey, Harshit and Bawden, Rachel and Wang, Thomas and Neeraj, Trishala and Rozen, Jos and Sharma, Abheesht and Santilli, Andrea and Fevry, Thibault and Fries, Jason Alan and Teehan, Ryan and Bers, Tali and Biderman, Stella and Gao, Leo and Wolf, Thomas and Rush, Alexander M.},
  year = {2021},
  doi = {10.48550/arxiv.2110.08207},
  eprint = {2110.08207},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.08207}
}

References

112 references, 52 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaib… find our card
2. Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785, 2019. find
3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pages 6–4. Venice, 2006. find
4. Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678, 2020. doi: 10.1162/tacl a 00338. URL https://doi.org/10.… find
5. Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In EMNLP, 2019. find
6. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
7. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009. find
8. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Lingu… find
9. BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation, 2021. URL https://github.com/google/BIG-bench/. ↗
10. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020. find our card
11. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Doro… find our card
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
13. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
14. Rich Caruana. Multitask learning. Mach. Learn., 28(1):41–75, 1997. 1007379606734. URL https://doi.org/10.1023/A:1007379606734. doi: 10.1023/A: — [doi:10.1023/a:1007379606734](https://doi.org/10.1023/a:1007379606734) ↗
15. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium, October-November 2018. Associ… find our card
16. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. CoRR, abs/1905.10044, 2019. URL http://arxiv.org/abs/1905.10044. ↗ our card
17. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. find
18. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki,… find
19. ACM, 2008. doi: 10.1145/1390156.1390177. URL https: //doi.org/10.1145/1390156.1390177. — [doi:10.1145/1390156.1390177](https://doi.org/10.1145/1390156.1390177) find
20. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer, 2005. find
21. Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. arXiv:1908.05803v2, 2019. find our card
22. Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11–20, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-… find our card
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
24. William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. find our card
25. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019. find our card
26. Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model, 2019. find our card
27. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) find
28. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics, 2007. find
29. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237, 2019. find
30. Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009. find
31. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34, 2003. find
32. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/1611.015collin87, 2016. URL http://arxiv.org/abs/1611.01587. ↗ our card
33. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701, 2015. find our card
34. Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL https://aclanthology.org/H01-1069. ↗
35. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In arXiv:1909.00277v2, 2019. find our card
36. Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. arXiv:1707.06209v1, 2017. find our card
37. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017. find our card
38. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. find our card
39. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single QA system. CoRR, abs/2005.00700, 2020a. URL https://arxiv.org/abs/2005.00700. ↗ our card
40. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online, November 2020b. Associati… find our card
41. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2, 2020. find
42. Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. arXiv prepr… find
43. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019. find our card
44. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. find our card
45. Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with application to the biography domain. CoRR, abs/1603.07771, 2016. URL http://arxiv.org/abs/1603. 07771. ↗ our card
46. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. find
47. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, S”oren Auer, et al. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195, 2015. find
48. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691, 2021. URL https://arxiv.org/abs/2104.08691. ↗
49. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. find
50. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patr… find our card
51. Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https://aclanthology.org/C02-1150. ↗
52. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, Online, November… find
53. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. In MRQA@EMNLP, 2019. find
54. Robert L Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353, 2021. find
55. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142– 150, Portland, Oregon, USA… find our card
56. Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172, 2013. find
57. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730, 2018. URL http: //arxiv.org/abs/1806.08730. find our card
58. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. CoRR, abs/1902.01007, 2019. URL http://arxiv.org/abs/ 1902.01007. ↗ our card
59. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. find
60. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. CoRR, abs/2104.08773, 2021. URL https://arxiv.org/abs/2104.08773. ↗
61. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, November 2020. Association for Computational L… find our card
62. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745, 2018. find
63. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. find
64. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005. find
65. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Co… find our card
66. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. find our card
67. Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694, March 2019. doi: 10.1162/ tacl a 00293. URL https://aclanthology.org/Q19-1043. ↗ our card
68. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. CoRR, abs/2105.11447, 2021. URL https://arxiv.org/abs/2105.11447. ↗ our card
69. Mohammad Taher Pilehvar and os’e Camacho-Collados. Wic: 10, 000 example pairs for evaluating context-sensitive representations. CoRR, abs/1808.09121, 2018. URL http://arxiv.org/abs/1808. 09121. ↗ our card
70. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce… find our card
71. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Associati… find our card
72. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
73. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67, 2020. find
74. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016. find our card
75. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. CoRR, abs/2102.07350, 2021. URL https://arxiv.org/abs/2102.07350. ↗ our card
76. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online, November 2020. Association for Computational Linguistics.… find
77. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. find
78. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confere… find
79. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6398. ↗
80. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, June 2018. Assoc… find our card
81. Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015. doi: 10.18653/v1/d15-1044. URL http://dx.doi.org/10.18653/v1/ D15-1044. find
82. Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. In Meeting of the Association for Computational Linguistics (ACL), 2018. find our card
83. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. URL http://arxiv. org/abs/1907.10641. find
84. Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online, April 2021. Association fo… find our card
85. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54–63, 2020. find our card
86. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. CoRR, abs/1704.04368, 2017. URL http://arxiv.org/abs/1704.04368. ↗
87. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018. find
88. Reddy Siva, Chen Danqi, and Manning Christopher D. Wikiqa: A challenge dataset for open-domain question answering. arXiv, 2018. find
89. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. Release strategies and the social impacts of language models. CoRR, abs/1908.09203, 2019. URL http://arxiv.org/abs/1908.09203. ↗ our card
90. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, 2019. find our card
91. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 2019. URL https://arxiv.org/abs/1902.00164v1. — [arXiv:1902.00164](https://arxiv.org/abs/1902.00164) ↗
92. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. ”quartz: An open-domain dataset of qualitative relationship questions”. EMNLP, ”2019”. find
93. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: A dataset and models for answering questions about qualitative relationships. CoRR, abs/1811.08048, 2018. URL http://arxiv.org/abs/1811.08048. — [arXiv:1811.08048](https://arxiv.org/abs/1811.08048) ↗
94. Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7… find
95. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537, 2019a. URL http://arxiv.org/abs/1905.00537. ↗ our card
96. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2019b. In the Proceedings of ICLR. find our card
97. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner. CoRR, abs/2104.14690, 2021. URL https://arxiv.org/abs/2104.14690. — [arXiv:2104.14690](https://arxiv.org/abs/2104.14690) ↗
98. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. find
99. Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts?, 2021. URL https://arxiv.org/abs/2109.01247. ↗ our card
100. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2021. find
101. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents, 2018. find our card
102. Adina Williams, Tristan Thrush, and Douwe Kiela. Anlizing the adversarial natural language inference dataset. arXiv preprint arXiv:2010.12729, 2020. find
103. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835, 2021. URL https://arxiv.org/abs/2104. 08835. ↗
104. Yang Yi, Yih Wen-tau, and Christopher Meek. WikiQA: A Challenge Dataset for Open-Domain Question Answering. Association for Computational Linguistics, page 2013–2018, 2015. doi: 10.18653/v1/D15-1237. find our card
105. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. find
106. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. find our card
107. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657, 2015a. find our card
108. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015b. find our card
109. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL, 2019. find our card
110. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find our card
111. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021. find our card
112. Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. CoRR, abs/2104.04670, 2021. URL https://arxiv.org/abs/2104.04670. ↗