Elamur Research
☰
home / papers / 2023 / The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2312.01552

Summary

Alignment tuning through SFT and RLHF is usually seen as the step that turns a pretrained base LLM into a usable assistant. LIMA suggested this step may change surprisingly little, and the authors test that idea directly. They have an aligned model (e.g., Llama-2-chat, Vicuna-7b, Mistral-7b-Instruct) generate an answer greedily. They then give each prefix to the matching base model and check how it ranks the token the aligned model chose. The two models rarely disagree, and the disagreements cluster on stylistic tokens such as greetings, discourse markers, refusals and safety caveats. Knowledge-heavy tokens mostly keep the same top rank. This implies that the knowledge comes from pretraining, and tuning mainly teaches an assistant's tone. Based on this, the authors propose URIAL, which aligns an untuned model purely through in-context learning: a system prompt plus a few fixed, carefully styled example answers. On the new just-eval-instruct benchmark, URIAL applied to base Mistral-7b and Llama-2-70b matches or beats their SFT and RLHF versions. This raises doubts about how much costly fine-tuning is really needed.

Key points

  • Token distribution shift analysis: on average, 77.7% of the aligned model's output tokens are also the base model's top-1 choice, and 92.2% are within its top 3. Shifts are larger at early positions in the response.
  • URIAL (Untuned LLMs with Restyled In-context ALignment): needs no weight updates. It uses only a system prompt and as few as K=3 constant in-context examples. Each example affirms the query, gives structured detailed content and ends with a summary that includes safety notes.
  • just-eval-instruct: 1,000 instructions taken from 9 existing datasets (including AlpacaEval, MT-bench and LIMA). GPT-4 scores answers on six aspects: helpfulness, clarity, factuality, depth, engagement and safety. The last 200 instructions are safety tests.
  • Human pairwise evaluation (100 samples): URIAL on Mistral-7b won 20.0% of comparisons against Mistral-7b-Instruct's 8.9%. URIAL on Llama-2-70b won 16.5% against Llama-2-70b-chat's 8.4%. Most of the remaining comparisons were ties.
  • Implication: these results support the "superficial alignment hypothesis". Tuning-free, inference-time alignment could replace SFT/RLHF in some settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

129 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 124 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.01552 · arXiv 2312.01552

@misc{lin2023unlocking,
  title = {{The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning}},
  author = {Lin, Bill Yuchen and Ravichander, Abhilasha and Lu, Ximing and Dziri, Nouha and Sclar, Melanie and Chandu, Khyathi and Bhagavatula, Chandra and Choi, Yejin},
  year = {2023},
  doi = {10.48550/arxiv.2312.01552},
  eprint = {2312.01552},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.01552}
}

References

54 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of language models: Towards open frontier m… find
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sha… find
  3. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2… find our card
  4. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shan Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. ArXiv, abs/2308.07201, 2023. URL https://api.semanticscholar.org/CorpusID:260887105. ↗ our card
  5. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. ArXiv, abs/2307.08701, 2023a. URL https://api.semanticscholar.org/ CorpusID:259937133. ↗ our card
  6. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. ArXiv, abs/2309.12307, 2023b. URL https://api.semanticscholar.org/CorpusID:262084134. ↗ our card
  7. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ArXiv, abs/2307.08691, 2023. URL https://api.semanticscholar.org/CorpusID:259936734. ↗
  8. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e. Flashattention: Fast and memory-efficient exact attention with io-awareness. ArXiv, abs/2205.14135, 2022. URL https: //api.semanticscholar.org/CorpusID:249151871. find
  9. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. ArXiv, abs/2305.14233, 2023. URL https://api.semanticscholar.org/ CorpusID:258840897. ↗ our card
  10. Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam. Exploring the relationship between in-context learning and instruction tuning. ArXiv, abs/2311.10367, 2023. URL https: //api.semanticscholar.org/CorpusID:265281504. find
  11. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv, abs/2210.17323, 2022. URL https: //api.semanticscholar.org/CorpusID:253237200. find
  12. Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan,… find our card
  13. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. ArXiv, abs/2307.06945, 2023. URL https://api. semanticscholar.org/CorpusID:259847425. find
  14. In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. ArXiv, abs/2311.04934, 2023. URL https://api.semanticscholar.org/CorpusID:265067391. ↗
  15. Arnav Gudibande, Eric Wallace, Charles Burton Snell, Xinyang Geng, Hao Liu, P. Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. ArXiv, abs/2305.15717, 2023. URL https://api.semanticscholar.org/CorpusID:258887629. ↗ our card
  16. Xiaochuang Han. In-context alignment: Chat with vanilla language models before fine-tuning. ArXiv, abs/2308.04275, 2023. URL https://api.semanticscholar.org/CorpusID: 260704721. — [arXiv:2308.04275](https://arxiv.org/abs/2308.04275) ↗
  17. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. ArXiv, abs/2310.06987, 2023. URL https://api. semanticscholar.org/CorpusID:263835408. find our card
  18. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée L… find
  19. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. ArXiv, abs/2310.00752, 2023b. URL https://api.semanticscholar.org/CorpusID:263334281. ↗
  20. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. find our card
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://api. semanticscholar.org/CorpusID:249017743. find our card
  22. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. ArXiv, abs/2308.07317, 2023. URL https://api.semanticscholar.org/CorpusID: 260886870. ↗
  23. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca eval, 2023a. ↗
  24. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. ArXiv, abs/2309.07124, 2023b. URL https: //api.semanticscholar.org/CorpusID:261705563. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) find
  25. Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Unsupervised cross-task generalization via retrieval augmentation. In NeurIPS, 2022. URL http://papers.nips.cc/paper files/paper/2022/hash/ 8a0d3ae989a382ce6e50312bc35bf7e1-Abstract-Conference.html. ↗
  26. Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. ArXiv, abs/2303.16634, 2023. URL https: //api.semanticscholar.org/CorpusID:257804696. find
  27. S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, 2023. URL https://api.semanticschola… ↗ our card
  28. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. ArXiv, abs/2306.08568, 2023. URL https://api.semanticscholar.org/CorpusID: 259164815. ↗
  29. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12837, 2022. URL https://api.semanticscholar.org/CorpusID: 247155069. ↗ our card
  30. Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. ArXiv, abs/2304.08467, 2023. URL https://api.semanticscholar.org/CorpusID: 258179012. ↗ our card
  31. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… find
  32. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http://arxiv.org/abs/ 1908.10084. ↗
  33. Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward (in-)consistency on rlhf, 2023. find
  34. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. ArXiv, abs/2004.09297, 2020. URL https://api. semanticscholar.org/CorpusID:215827489. find our card
  35. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021. URL https://api. semanticscholar.org/CorpusID:233307138. find
  36. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford alpaca, 2023. ↗
  37. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bri… find
  38. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. Focused transformer: Contrastive training for context scaling, 2023. find
  39. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics, 2022a. URL https://api. semanticscholar.org/C… find
  40. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… find
  41. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. ArXiv, abs/2306.04751, 2023. URL https:/… find our card
  42. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. find
  43. OpenReview.net, 2022a. URL https://openreview.net/forum?id=gEZrGCozdqR. ↗
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b. find our card
  45. Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently. ArXiv, abs/2303.03846, 2023. URL https://api.semanticscholar.org/CorpusID: 257378479. ↗ our card
  46. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oğuz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma… find our card
  47. Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable text generation evaluation with automatic feedback. ArXiv, abs/2305.14282, 2023. URL https://api.semanticscholar.org/CorpusID: 258841553. ↗
  48. Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Seo. In-context instruction learning. ArXiv, abs/2302.14691, 2023a. URL https://api. semanticscholar.org/CorpusID:257233109. — [arXiv:2302.14691](https://arxiv.org/abs/2302.14691) find
  49. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. ArXiv, abs/2307.10928, 2023b. URL https://api.semanticscholar. org/CorpusID:259991144. ↗
  50. Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Raghavi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. ArXiv, abs/2311.05657, 2023. URL https://api.semanticscholar.org/ CorpusID:265128672. ↗
  51. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv, abs/2309.05653, 2023. URL https://api.semanticscholar.org/CorpusID:261696697. ↗ our card
  52. Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. ArXiv, abs/2211.04486, 2022. URL https://api.semanticscholar.org/CorpusID:253420743. ↗ our card
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ioan Cristian Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685, 2023. URL https://api.semantic… ↗
  54. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. ArXiv, abs/2305.11206, 2023. URL https: //api.semanticscholar.org/CorpusID:258822910… find