Elamur Research⌕
☰
home / papers / 2023 / When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks

When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks

Paper · 2023 · Hao Peng, Xiaozhi Wang, Jianhui Chen et al.

Summary

Large language models are mostly used through in-context learning (ICL), but its limits are poorly mapped. The authors study "specification-heavy" tasks, where annotation guidelines are long and intricate enough that a person needs hours of training. Classic information extraction is a typical case: the ACE 2005 event guideline runs to 77 pages. They assemble 18 such tasks across six categories and test six LLMs, ranging from FLAN-UL2 to GPT-4. ICL scores usually stay below half of fine-tuned state-of-the-art systems.

Error analysis and targeted experiments point to three causes:

  • Models read context superficially, relying on prior knowledge or missing key words.
  • Their interpretation of label schemas diverges from human definitions.
  • They handle long inputs poorly, including long task specifications.

Fine-tuning FLAN-UL2 brings results close to SoTA, and gains grow with model size. The weakness therefore lies in current alignment methods rather than in the models themselves. A small instruction-tuning set built for these tasks lifts FLAN-UL2's ICL to the level of text-davinci-003. This matters for anyone extracting structured data through prompting.

Key points

  • Benchmark: 18 tasks in six groups.
    • NER: CoNLL 2003, ACE 2005, FewNERD
    • RE: TACRED, SemEval, FewRel 2.0, DocRED
    • ED: ACE 2005, MAVEN, RichERE
    • EAE: ACE 2005, RichERE
    • Event relations: MATRES, MAVEN-Causal/Subevent/Temporal
    • Sentiment: SST-5, GoEmotions
  • Models evaluated: FLAN-UL2, Alpaca, Vicuna, ChatGPT, text-davinci-003, GPT-4.
  • Gap to SoTA: GPT-4, the strongest model, still trails badly. It gets 33.7 F1 on ACE 2005 event detection versus a SoTA of 73.5, and 25.2 on TACRED versus 76.8. Open models often score near zero on relation and event tasks.
  • Typical error examples: event types are confused, for instance BUSINESS predicted as TRANSACTION. Models also keep their original answer after small edits that change the gold label.
  • Fine-tuning and alignment: fine-tuning FLAN-UL2 (20B) reaches results mostly comparable to SoTA, and fine-tuning shows positive scaling with model size. Instruction tuning used 10 hand-written instructions per task, augmented by shuffling the label schema.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓Xi’an Jiaotong-Liverpool UniversitySuzhou · China10 works · 319 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou and Juanzi Li. 2023. When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks. arXiv:2311.08993.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.08993 · arXiv 2311.08993

@misc{peng2023when,
  title = {{When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks}},
  author = {Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou and Juanzi Li},
  year = {2023},
  doi = {10.48550/arxiv.2311.08993},
  eprint = {2311.08993},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.08993}
}

References

121 references, 63 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023. URL https://arxiv.org/pdf/2305.10403. pdf. ↗
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. URL https://arxiv.org/pdf/… ↗
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022b. URL https://arxiv.org/pdf/2212.08073.pdf. ↗
  4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of — A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity https://www.ldc.upenn.edu/ https://github.com/facebookresearch/llama/blob/main/LICENSE 4 https://github.com/face… ↗
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  6. Walker Christopher, Strassel Stephanie, Medero Julie, and Maeda Kazuaki. ACE 2005 multilingual training corpus, 2005. URL https://catalog.ldc.upenn.edu/LDC2006T06. ↗
  7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL https://arxiv.org/pdf/2210. 11416.pdf. ↗
  8. Amir DN Cohen, Shachar Rosenman, and Yoav Goldberg. Relation classification as two-way span-prediction. arXiv preprint arXiv:2010.04829, 2020. URL https://arxiv.org/pdf/ 2010.04829.pdf. — [arXiv:2010.04829](https://arxiv.org/abs/2010.04829) ↗
  9. Linguistic Data Consortium. ACE (Automatic Content Extraction) English annotation guidelines for events. Version, 5(4), 2005. URL https://www.ldc.upenn.edu/sites/www.ldc. upenn.edu/files/english-events-guidelines-v5.4.3.pdf. ↗
  10. Xuan-Quy Dao and Ngoc-Bich Le. Investigating the effectiveness of chatgpt in mathematical reasoning and problem solving: Evidence from the vietnamese national high school graduation examination. arXiv preprint arXiv:2306.06331, 2023. URL https://arxiv.org/pdf/ 2306.06331.pdf. ↗
  11. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of ACL, pp. 4040– 4054, 2020. URL https://aclanthology.org/2020.acl-main.372.pdf. ↗
  12. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of ACL-IJCNLP, pp. 3198–3213, 2021. URL https://aclanthology.org/2021.acl-long. 248.pdf. ↗
  13. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904, 2022. URL https://arxiv.org/pdf/2203… — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) ↗
  14. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. URL https://arxiv.org/pdf/2304.06767. pdf. ↗
  15. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022. URL https://arxiv.org/pdf/2301.00234.pdf. ↗
  16. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of ChatGPT. arXiv preprint arXiv:2301.13867, 2023. URL https://arxiv.org/pdf/2301. 13867.pdf. ↗
  17. Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. URL https://arxiv.org/pdf/2001.09768.pdf. — [arXiv:2001.09768](https://arxiv.org/abs/2001.09768) ↗
  18. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. URL https://ar…
  19. Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. Exploring the feasibility of ChatGPT for event extraction. arXiv preprint arXiv:2303.03836, 2023a. URL https://arxiv.org/pdf/ 2303.03836.pdf. ↗
  20. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of EMNLP-IJCNLP, pp. 6250–6255, 2019. URL https://aclanthology.org/D19-1649.pdf. ↗
  21. Yuan Gao, Ruili Wang, and Feng Hou. How to design translation prompts for ChatGPT: An empirical study. arXiv preprint arXiv:2304.02182, 2023b. URL https://arxiv.org/pdf/ 2304.02182.pdf. — [arXiv:2304.02182](https://arxiv.org/abs/2304.02182) ↗
  22. Siddhant Garg, Thuy Vu, and Alessandro Moschitti. TANDA: Transfer and adapt pre-trained transformer models for answer sentence selection. In Proceedings of AAAI, pp. 7780–7788, 2020. URL https://arxiv.org/pdf/1911.04118.pdf. — [arXiv:1911.04118](https://arxiv.org/abs/1911.04118) ↗
  23. Lise Getoor and Ben Taskar. Global inference for entity and relation identification via linear programming formulation, 2007. URL https://cogcomp.seas.upenn.edu/papers/ RothYi07.pdf. ↗
  24. Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. Is information extraction solved by ChatGPT? An analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450, 2023. URL https://arxiv.org/pdf/2305. 14450.pdf. — [arXiv:2305.14450](https://arxiv.org/abs/2305.14450) ↗
  25. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In Proceedings of ICLR, 2021. URL https://openreview.net/pdf?id=0RDcd5Axok. ↗
  26. Franz A Heinsen. An algorithm for routing vectors in sequences. arXiv preprint arXiv:2211.11754, 2022. URL https://arxiv.org/pdf/2211.11754.pdf. — [arXiv:2211.11754](https://arxiv.org/abs/2211.11754) ↗
  27. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.
  28. ACL 2010, pp. 33, 2010. URL https://aclanthology.org/S10-1006.pdf. — SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals ↗
  29. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In Proceedings of ICLR, 2020. URL https://openreview.net/pdf?id=dNy_RKzJacY. ↗
  30. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of NeurIPS Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/pdf?id=7Bywt2mQsCe. ↗
  31. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of ICML, pp. 2790–2799, 2019. URL http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf. ↗
  32. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675, 2023. URL https://arxiv.org/pdf/ 2308.00675.pdf. ↗
  33. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022. URL https:…
  34. Wenxiang Jiao, Wenxuan Wang, JT Huang, Xing Wang, and ZP Tu. Is ChatGPT a good translator? Yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745, 2023. URL https://arxiv.org/pdf/2301.08745.pdf. ↗
  35. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of EMNLP, pp. 4497–4512, 2022. doi: 10.18653/v1/2022.findings-emnlp.329. URL https://aclanthology.org/2022.findi… ↗
  36. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023. URL https://arxiv.org/pdf/2307.10169.pdf. ↗
  37. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021. URL https://arxiv.org/pdf/2103.14659.pdf. ↗
  38. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. ChatGPT: Jack of all trades, master of none. Information Fusion, pp. 101861, 2023. URL https://arxiv.org/ pdf/2302.10724.pdf. ↗
  39. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.07871, 2018. URL https://arxiv.org/pdf/1811.07871.pdf. ↗
  40. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. Evaluating ChatGPT’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633, 2023a. URL https://arxiv.org/pdf/2304.11633.pdf. ↗
  41. Jing Li, Shanshan Feng, and Billy Chiu. Few-shot relation extraction with dual graph neural network interaction. IEEE Transactions on Neural Networks and Learning Systems, 2023b. URL https://ieeexplore.ieee.org/document/10143375. ↗
  42. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124, 2023c. URL https://arxiv.org/pdf/2309.07124.pdf. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) ↗
  43. Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable BERT. In Proceedings of IJCAI, pp. 1800–1806, 2019. URL https://www.ijcai.org/Proceedings/ 2019/0249.pdf. ↗
  44. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arxiv:2211.09110, 2022. URL https://arxiv.org/pdf/2211. 09110.pdf. ↗
  45. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL — Let's Verify Step by Step https://arxiv.org/pdf/2305.20050.pdf. ↗
  46. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of ChatGPT and GPT-4. arXiv preprint arXiv:2304.03439, 2023a. URL https://arxiv.org/pdf/2304.03439.pdf. ↗
  47. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022. URL https://proceedings.neurips.cc/pape… ↗
  48. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023b. URL https://arxiv.org/pdf/2307.03172.pdf. ↗
  49. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of EMNLP, pp. 7052– 7063, 2021. URL https://aclanthology.org/2021.emnlp-main.565.pdf. ↗
  50. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. URL https://arxiv.org/ pdf/2301.13688.pdf. ↗
  51. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2018. URL https://openreview.net/pdf?id=Bkg6RiCqY7. ↗
  52. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023. URL https://arxiv.org/pdf/2304. 09842.pdf. ↗
  53. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of ACL, pp. 8086–8098, 2022. URL https://aclanthology.org/2022.acl-long. 556.pdf. ↗
  54. Youmi Ma, An Wang, and Naoaki Okazaki. DREEAM: Guiding attention with evidence for improving document-level relation extraction. In Proceedings of EACL, pp. 1963–1975, 2023. URL https://aclanthology.org/2023.eacl-main.145.pdf. ↗
  55. Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, and Jing Shao. Prompt for Extraction? PAIE: Prompting argument interaction for event argument extraction. In Proceedings of ACL, pp. 6759–6774, 2022. URL https://aclanthology.org/2022.acl-long. 466.pdf. ↗
  56. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of ACL, pp. 9802–9822, 2023. URL https://aclanthology.org/2023.acl-long.546.pdf. ↗
  57. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of EMNLP, pp. 11048–11064, 2022. URL https://aclanthology.org/ 2022.emnlp-main.759.pdf. ↗
  58. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to GPTk’s language. In Findings of ACL, pp. 589–612, 2022a. URL https://aclanthology.org/2022.findings-acl.50.pdf. ↗
  59. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of ACL, pp. 3470–3487, 2022b. URL https://aclanthology.org/2022.acl-long.244.pdf. ↗
  60. Qiang Ning, Hao Wu, and Dan Roth. A multi-axis annotation scheme for event temporal relations. In Proceedings of ACL, pp. 1318–1328, 2018. URL https://aclanthology.org/ P18-1122.pdf. ↗
  61. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. ↗
  62. OpenAI. Improving mathematical reasoning with process supervision, 2023a. URL https://openai.com/research/ improving-mathematical-reasoning-with-process-supervision. ↗
  63. OpenAI. Introducing superalignment, 2023b. introducing-superalignment. find
  64. URL — GPT-4 Technical Report https://openai.com/blog/R OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023c. URL https://arxiv.org/pdf/2303.08774.pdf. ↗
  65. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. URL http…
  66. Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning” learns” in-context: Disentangling task recognition and task learning. arXiv preprint arXiv:2305.09731, 2023. URL https://arxiv.org/pdf/2305.09731.pdf. ↗
  67. Sachin Pawar, Girish K Palshikar, and Pushpak Bhattacharyya. Relation extraction: A survey. arXiv preprint arXiv:1712.05191, 2017. URL https://arxiv.org/pdf/1712.05191.pdf. — [arXiv:1712.05191](https://arxiv.org/abs/1712.05191) ↗
  68. Hao Peng, Xiaozhi Wang, Feng Yao, Zimu Wang, Chuzhao Zhu, Kaisheng Zeng, Lei Hou, and Juanzi Li. OmniEvent: A comprehensive, fair, and easy-to-use toolkit for event understanding. arXiv preprint arXiv:2309.14258, 2023. URL https://arxiv.org/pdf/2309.14258. pdf. ↗
  69. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of EMNLP, pp. 3419–3448, 2022. URL https://aclanthology.org/ 2022.emnlp-main.225.pdf. ↗
  70. Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. URL https://arxiv.org/pdf/1811.01088.pdf. ↗
  71. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. URL https://arxiv.org/pdf/2108.12409.pdf. ↗
  72. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023. URL https://arxiv.org/pdf/2302.06476.pdf. ↗
  73. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10): 1872–1897, 2020. URL https://arxiv.org/abs/2003.08271. ↗
  74. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. URL https://jmlr.org/papers… ↗
  75. Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019. URL https://arxiv.org/pdf/ 1910.01708.pdf. — [arXiv:1910.01708](https://arxiv.org/abs/1910.01708) ↗
  76. Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL, pp. 142–147, 2003. URL https://aclanthology.org/W03-0419.pdf. ↗
  77. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of EMNLP, pp. 12007–12021, 2022. doi: 10.18653/v1/2022. emnlp-main.823. URL https://…
  78. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. arXiv preprint arXiv:2303.17580, 2023. URL https://arxiv.org/pdf/2303.17580.pdf. ↗
  79. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of ACL, pp. 11289–11310, 2023. doi: 10.18653/v1/2023.acl-long.632. URL https://aclanthology.org/2023.acl-long.632. ↗
  80. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631–1642, 2013. URL https://aclanthology.org/D13-1170.pdf. ↗
  81. Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. From light to Rich ERE: Annotation of entities, relations, and events. In Proceedings of the 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Repres…
  82. Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-Tail: How knowledgeable are large language models (LLM)? Aka will LLMs replace knowledge graphs? arXiv preprint arXiv:2308.10168, 2023a. URL https://arxiv.org/pdf/2308.10168.pdf. ↗
  83. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047, 2023b. URL https://arxiv.org/pdf/2305.03047.pdf. ↗
  84. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503, 2021. URL https://arxiv.org/pdf/2102.02503.pdf. ↗
  85. Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. Evaluation of ChatGPT as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992, 2023. URL https://arxiv.org/pdf/2303.07992.pdf. — [arXiv:2303.07992](https://arxiv.org/abs/2303.07992) ↗
  86. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model, 2023. URL https://crfm.stanford.edu/2023/03/13/alpaca.html. ↗
  87. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. UL2: Unifying language learning paradigms. In Proceedings of ICLR, 2022. URL https://openreview.net/pdf?id= 6ruVLB727MC. ↗
  88. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv.org/pdf/2302.13971.… ↗
  89. Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies. In Proceedings of NAACL-HLT, pp. 4363–4374, 2022. URL https://aclanthology.org/2022. naacl-main.324.pdf. ↗
  90. Vicuna. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/. ↗
  91. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 57, 2006. URL https://catalog.ldc.upenn.edu/LDC2006T06. ↗
  92. Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. DeepStruct: Pretraining of language models for structure prediction. In Findings of ACL, pp. 803–823, 2022a. URL https://aclanthology.org/2022.findings-acl.67v2.pdf. ↗
  93. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. MAVEN: A massive general domain event detection dataset. In Proceedings of EMNLP, pp. 1652–1671, 2020. URL https://aclanthology.org/2020. emnlp-main.129.pdf. ↗
  94. Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, et al. MAVEN-ERE: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. arXiv preprint arXiv:2211.07342, 2022b. URL https://arxiv.org/pdf/… ↗
  95. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. Automated concatenation of embeddings for structured prediction. In Proceedings of ACL-IJCNLP, pp. 2643–2660, 2021a. URL https://aclanthology.org/2021.acl-long. 206.pdf. ↗
  96. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022c. URL https://arxiv.org/pdf/2212.10560. pdf. ↗
  97. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-Naturalinstructions: Generalization via declarative instructions on 1600+ tasks. In Proceedings of EMNLP, pp. 5085–51… find
  98. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023. URL https://arxiv.org/pdf/2307.12966.pdf. ↗
  99. Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. CLEVE: Contrastive pre-training for event extraction. In Proceedings of ACL-IJCNLP, pp. 6283–6297, 2021b. URL https://aclanthology.org/2021.acl-long.491.pdf. ↗
  100. Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Proceedings of NAACL-HLT, pp. 2300–2344, 2022. URL https://aclanthology.org/2022.naacl-main.167.pdf. ↗
  101. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proceedings of ICLR, 2021. URL https://openreview.net/pdf?id=gEZrGCozdqR. ↗
  102. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023. URL https://arxiv.org/pdf/2303.03846.pdf. ↗
  103. Lilian Weng. Prompt engineering. lilianweng.github.io, Mar 2023. URL https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/. ↗
  104. Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. LLM-powered data augmentation for enhanced crosslingual performance. arXiv preprint arXiv:2305.14288, 2023. URL https://arxiv.org/pdf/2305.14288.pdf. — [arXiv:2305.14288](https://arxiv.org/abs/2305.14288) ↗
  105. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  106. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts. arXiv preprint arXiv:2305.13300, 2023. URL https://arxiv.org/pdf/2305.13300.pdf. ↗
  107. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? A comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.09841, 2023a. URL https://arxiv.org/ pdf/2306.09841.pdf. ↗
  108. Xin Xu, Yuqi Zhu, Xiaohan Wang, and Ningyu Zhang. How to unleash the power of large language models for few-shot relation extraction? arXiv preprint arXiv:2305.01555, 2023b. URL https://arxiv.org/pdf/2305.01555.pdf. ↗
  109. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of ACL, pp. 764–777, 2019. URL https://aclanthology.org/P19-1074.pdf. ↗
  110. Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. KoLA: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296, 2023a. URL https://arxiv.org/ pdf/2306.09296.pdf. — [arXiv:2306.09296](https://arxiv.org/abs/2306.09296) ↗
  111. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. arXiv preprint arXiv:2306.15895, 2023b. URL https://arxiv.org/pdf/2306. 15895.pdf. ↗
  112. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023. URL https://arxiv.org/pdf/2309.05653.pdf. ↗
  113. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of ACL, pp. 6066–6080, 2020. URL https://aclanthology.org/2020.acl-main. 540.pdf. ↗
  114. Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. Optimizing bi-encoder for named entity recognition via contrastive learning. arXiv preprint arXiv:2208.14565, 2022. URL https://arxiv.org/pdf/2208.14565.pdf. ↗
  115. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP, 2017. URL https://aclanthology.org/D17-1004.pdf. ↗
  116. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Proceedings of ICML, pp. 12697–12706, 2021. URL http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf. ↗
  117. Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795, 2023. URL https://arxiv.org/pdf/2305.14795.pdf. ↗
  118. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023a. URL https://arxiv.org/pdf/2305.11206.pdf. ↗
  119. Jie Zhou, Shenpo Dong, Hongkui Tu, Xiaodong Wang, and Yong Dou. RSGT: Relational structure guided temporal relation extraction. In Proceedings of COLING, pp. 2001–2010, 2022. URL https://aclanthology.org/2022.coling-1.174.pdf. ↗
  120. Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. arXiv preprint arXiv:2303.11315, 2023b. URL https://arxiv.org/pdf/2303.11315.pdf. ↗
  121. Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of llms: An adaptive testing perspective. arXiv preprint arXiv:2306.10512, 2023. URL https://arxiv.org/pdf/2306.10512.pdf. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2023-context-learning-fall-short-why/