Elamur Research
☰
home / papers / 2024 / Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2403.05530

Summary

This Google technical report presents the Gemini 1.5 family of multimodal models, built to handle very long inputs. Contemporaries such as GPT-4 Turbo (128k) and Claude 3.0 (200k) have much shorter context windows. Two models are described: Gemini 1.5 Flash, a lighter and faster variant, and a May 2024 update to Gemini 1.5 Pro. They rely on sparse and dense scaling plus improved training, distillation and serving infrastructure, and they accept text, audio and video contexts of millions of tokens. Evaluation combines synthetic needle-in-a-haystack retrieval with realistic tasks: long-document and long-video QA, long-context ASR, in-context planning, and standard benchmarks for math, code, vision and multilinguality. Retrieval stays above 99% up to 10M tokens, and the long context does not weaken core skills. Gemini 1.5 Pro beats Gemini 1.0 Ultra on most benchmarks while needing less training compute. A striking demonstration is learning English–Kalamang translation purely from documentation placed in the prompt. For low-resource language work, this suggests that long context can partly substitute for training data.

Key points

  • Needle recall exceeds 99.7% up to 1M tokens in all modalities. Recall holds at about 10M tokens of text (≈7M words), 9.7M tokens of audio (≈107 h) and 9.9M tokens of video (≈10.5 h).
  • Kalamang has fewer than 200 speakers. Given a ~500-page grammar, a dictionary and ~400 parallel sentences in context, the models translate English→Kalamang about as well as a human who studied the same materials. Adding 45 minutes of transcribed speech enables in-context speech recognition for the language.
  • Win rates: 1.5 Pro outperforms 1.0 Pro on 44/50 benchmarks and 1.0 Ultra on 35/45. 1.5 Flash outperforms 1.0 Pro on 41/50.
  • The May update improves on the February version by over 10% relative on average. Examples include MATH (58.5→67.7%), GPQA (41.5→46.2%) and MathVista (52.1→63.9%).
  • In real-world tests, professionals saved 26–75% of task time across 10 job categories. The authors also recommend multiple-needle setups for harder long-context evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1221 papers of our library cite it (of the 140673 whose bibliographies we read); 48 of them have a page here, 1173 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.05530 · arXiv 2403.05530

@misc{team2024gemini,
  title = {{Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context}},
  author = {Team, Gemini and Georgiev, Petko and Lei, Ving Ian and Burnell, Ryan and Bai, Libin and Gulati, Anmol and Tanzer, Garrett and Vincent, Damien and Pan, Zhufeng and Wang, Shibo and Mariooryad, Soroosh and Ding, Yifan and Geng, Xinyang and Alcober, Fred and Frostig, Roy and Omernick, Mark and Walker, Lexi and Paduraru, Cosmin and Sorokin, Christina and Tacchetti, Andrea and Gaffney, Colin and Daruki, Samira and Sercinoglu, Olcan and Gleicher, Zach and Love, Juliette and Voigtlaender, Paul and Jain, Rohan and Surita, Gabriela and Mohamed, Kareem and Blevins, Rory and Ahn, Junwhan and Zhu, Tao and Kawintiranon, Kornraphop and Firat, Orhan and Gu, Yiming and Zhang, Yujing and Rahtz, Matthew and Faruqui, Manaal and Clay, Natalie and Gilmer, Justin and Co-Reyes, JD and Penchev, Ivo and Zhu, Rui and Morioka, Nobuyuki and Hui, Kevin and Haridasan, Krishna and Campos, Victor and Mahdieh, Mahdis and Guo, Mandy and Hassan, Samer and Kilgour, Kevin and Vezer, Arpi and Cheng, Heng-Tze and Liedekerke, Raoul de and Goyal, Siddharth and Barham, Paul and Strouse, DJ and Noury, Seb and Adler, Jonas and Sundararajan, Mukund and Vikram, Sharad and Lepikhin, Dmitry and Paganini, Michela and Garcia, Xavier and Yang, Fan and Valter, Dasha and Trebacz, Maja and Vodrahalli, Kiran and Asawaroengchai, Chulayuth and Ring, Roman and Kalb, Norbert and Soares, Livio Baldini and Brahma, Siddhartha and Steiner, David and Yu, Tianhe and Mentzer, Fabian and He, Antoine and Gonzalez, Lucas and Xu, Bibo and Kaufman, Raphael Lopez and Shafey, Laurent El and Oh, Junhyuk and Hennigan, Tom and Driessche, George van den and Odoom, Seth and Lucic, Mario and Roelofs, Becca and Lall, Sid and Marathe, Amit and Chan, Betty and Ontanon, Santiago and He, Luheng and Teplyashin, Denis and Lai, Jonathan and Crone, Phil and Damoc, Bogdan and Ho, Lewis and Riedel, Sebastian and Lenc, Karel and Yeh, Chih-Kuan and Chowdhery, Aakanksha and Xu, Yang and Kazemi, Mehran and Amid, Ehsan and Petrushkina, Anastasia and Swersky, Kevin and Khodaei, Ali and Chen, Gowoon and Larkin, Chris and Pinto, Mario and Yan, Geng and Badia, Adria Puigdomenech and Patil, Piyush and Hansen, Steven and Orr, Dave and Arnold, Sebastien M. R. and Grimstad, Jordan and Dai, Andrew and Douglas, Sholto and Sinha, Rishika and Yadav, Vikas and Chen, Xi and Gribovskaya, Elena and Austin, Jacob and Zhao, Jeffrey and Patel, Kaushal and Komarek, Paul and Austin, Sophia and Borgeaud, Sebastian and Friso, Linda and Goyal, Abhimanyu and Caine, Ben and Cao, Kris and Chung, Da-Woon and Lamm, Matthew and Barth-Maron, Gabe and Kagohara, Thais and Olszewska, Kate and Chen, Mia and Shivakumar, Kaushik and Agarwal, Rishabh and Godhia, Harshal and Rajwar, Ravi and Snaider, Javier and Dotiwalla, Xerxes and Liu, Yuan and Barua, Aditya and Ungureanu, Victor and Zhang, Yuan and Batsaikhan, Bat-Orgil and Wirth, Mateo and Qin, James and Danihelka, Ivo and Doshi, Tulsee and Chadwick, Martin and Chen, Jilin and Jain, Sanil and Le, Quoc and Kar, Arjun and Gurumurthy, Madhu and Li, Cheng and Sang, Ruoxin and Liu, Fangyu and Lamprou, Lampros and Munoz, Rich and Lintz, Nathan and Mehta, Harsh and Howard, Heidi and Reynolds, Malcolm and Aroyo, Lora and Wang, Quan and Blanco, Lorenzo and Cassirer, Albin and Griffith, Jordan and Das, Dipanjan and Lee, Stephan and Sygnowski, Jakub and Fisher, Zach and Besley, James and Powell, Richard and Ahmed, Zafarali and Paulus, Dominik and Reitter, David and Borsos, Zalan and Joshi, Rishabh and Pope, Aedan and Hand, Steven and Selo, Vittorio and Jain, Vihan and Sethi, Nikhil and Goel, Megha and Makino, Takaki and May, Rhys and Yang, Zhen and Schalkwyk, Johan and Butterfield, Christina and Hauth, Anja and Goldin, Alex and Hawkins, Will and Senter, Evan and Brin, Sergey and Woodman, Oliver and Ritter, Marvin and Noland, Eric and Giang, Minh and Bolina, Vijay and Lee, Lisa and Blyth, Tim and Mackinnon, Ian and Reid, Machel and Sarvana, Obaid and Silver, David and Chen, Alexander and Wang, Lily and Maggiore, Loren and Chang, Oscar and Attaluri, Nithya and Thornton, Gregory and Chiu, Chung-Cheng and Bunyan, Oskar and Levine, Nir and Chung, Timothy and Eltyshev, Evgenii and Si, Xiance and Lillicrap, Timothy and Brady, Demetra and Aggarwal, Vaibhav and Wu, Boxi and Xu, Yuanzhong and McIlroy, Ross and Badola, Kartikeya and Sandhu, Paramjit and Moreira, Erica and Stokowiec, Wojciech and Hemsley, Ross and Li, Dong and Tudor, Alex and Shyam, Pranav and Rahimtoroghi, Elahe and Haykal, Salem and Sprechmann, Pablo and Zhou, Xiang and Mincu, Diana and Li, Yujia and Addanki, Ravi and Krishna, Kalpesh and Wu, Xiao and Frechette, Alexandre and Eyal, Matan and Dafoe, Allan and Lacey, Dave and Whang, Jay and Avrahami, Thi and Zhang, Ye and Taropa, Emanuel and Lin, Hanzhao and Toyama, Daniel and Rutherford, Eliza and Sano, Motoki and Choe, HyunJeong and Tomala, Alex and Safranek-Shrader, Chalence and Kassner, Nora and Pajarskas, Mantas and Harvey, Matt and Sechrist, Sean and Fortunato, Meire and Lyu, Christina and Elsayed, Gamaleldin and Kuang, Chenkai and Lottes, James and Chu, Eric and Jia, Chao and Chen, Chih-Wei and Humphreys, Peter and Baumli, Kate and Tao, Connie and Samuel, Rajkumar and Santos, Cicero Nogueira dos and Andreassen, Anders and Rakićević, Nemanja and Grewe, Dominik and Kumar, Aviral and Winkler, Stephanie and Caton, Jonathan and Brock, Andrew and Dalmia, Sid and Sheahan, Hannah and Barr, Iain and Miao, Yingjie and Natsev, Paul and Devlin, Jacob and Behbahani, Feryal and Prost, Flavien and Sun, Yanhua and Myaskovsky, Artiom and Pillai, Thanumalayan Sankaranarayana and Hurt, Dan and Lazaridou, Angeliki and Xiong, Xi and Zheng, Ce and Pardo, Fabio and Li, Xiaowei and Horgan, Dan and Stanton, Joe and Ambar, Moran and Xia, Fei and Lince, Alejandro and Wang, Mingqiu and Mustafa, Basil and Webson, Albert and Lee, Hyo and Anil, Rohan and Wicke, Martin and Dozat, Timothy and Sinha, Abhishek and Piqueras, Enrique and Dabir, Elahe and Upadhyay, Shyam and Boral, Anudhyan and Hendricks, Lisa Anne and Fry, Corey and Djolonga, Josip and Su, Yi and Walker, Jake and Labanowski, Jane and Huang, Ronny and Misra, Vedant and Chen, Jeremy and Skerry-Ryan, RJ and Singh, Avi and Rijhwani, Shruti and Yu, Dian and Castro-Ros, Alex and Changpinyo, Beer and Datta, Romina and Bagri, Sumit and Hrafnkelsson, Arnar Mar and Maggioni, Marcello and Zheng, Daniel and Sulsky, Yury and Hou, Shaobo and Paine, Tom Le and Yang, Antoine and Riesa, Jason and Rogozinska, Dominika and Marcus, Dror and Badawy, Dalia El and Zhang, Qiao and Wang, Luyu and Miller, Helen and Greer, Jeremy and Sjos, Lars Lowe and Nova, Azade and Zen, Heiga and Chaabouni, Rahma and Rosca, Mihaela and Jiang, Jiepu and Chen, Charlie and Liu, Ruibo and Sainath, Tara and Krikun, Maxim and Polozov, Alex and Lespiau, Jean-Baptiste and Newlan, Josh and Cankara, Zeyncep and Kwak, Soo and Xu, Yunhan and Chen, Phil and Coenen, Andy and Meyer, Clemens and Tsihlas, Katerina and Ma, Ada and Gottweis, Juraj and Xing, Jinwei and Gu, Chenjie and Miao, Jin and Frank, Christian and Cankara, Zeynep and Ganapathy, Sanjay and Dasgupta, Ishita and Hughes-Fitt, Steph and Chen, Heng and Reid, David and Rong, Keran and Fan, Hongmin and Amersfoort, Joost van and Zhuang, Vincent and Cohen, Aaron and Gu, Shixiang Shane and Mohananey, Anhad and Ilic, Anastasija and Tobin, Taylor and Wieting, John and Bortsova, Anna and Thacker, Phoebe and Wang, Emma and Caveness, Emily and Chiu, Justin and Sezener, Eren and Kaskasoli, Alex and Baker, Steven and Millican, Katie and Elhawaty, Mohamed and Aisopos, Kostas and Lebsack, Carl and Byrd, Nathan and Dai, Hanjun and Jia, Wenhao and Wiethoff, Matthew and Davoodi, Elnaz and Weston, Albert and Yagati, Lakshman and Ahuja, Arun and Gao, Isabel and Pundak, Golan and Zhang, Susan and Azzam, Michael and Sim, Khe Chai and Caelles, Sergi and Keeling, James and Sharma, Abhanshu and Swing, Andy and Li, YaGuang and Liu, Chenxi and Bostock, Carrie Grimes and Bansal, Yamini and Nado, Zachary and Anand, Ankesh and Lipschultz, Josh and Karmarkar, Abhijit and Proleev, Lev and Ittycheriah, Abe and Yeganeh, Soheil Hassas and Polovets, George and Faust, Aleksandra and Sun, Jiao and Rrustemi, Alban and Li, Pen and Shivanna, Rakesh and Liu, Jeremiah and Welty, Chris and Lebron, Federico and Baddepudi, Anirudh and Krause, Sebastian and Parisotto, Emilio and Soricut, Radu and Xu, Zheng and Bloxwich, Dawn and Johnson, Melvin and Neyshabur, Behnam and Mao-Jones, Justin and Wang, Renshen and Ramasesh, Vinay and Abbas, Zaheer and Guez, Arthur and Segal, Constant and Nguyen, Duc Dung and Svensson, James and Hou, Le and York, Sarah and Milan, Kieran and Bridgers, Sophie and Gworek, Wiktor and Tagliasacchi, Marco and Lee-Thorp, James and Chang, Michael and Guseynov, Alexey and Hartman, Ale Jakse and Kwong, Michael and Zhao, Ruizhe and Kashem, Sheleem and Cole, Elizabeth and Miech, Antoine and Tanburn, Richard and Phuong, Mary and Pavetic, Filip and Cevey, Sebastien and Comanescu, Ramona and Ives, Richard and Yang, Sherry and Du, Cosmo and Li, Bo and Zhang, Zizhao and Iinuma, Mariko and Hu, Clara Huiyi and Roy, Aurko and Bijwadia, Shaan and Zhu, Zhenkai and Martins, Danilo and Saputro, Rachel and Gergely, Anita and Zheng, Steven and Jia, Dawei and Antonoglou, Ioannis and Sadovsky, Adam and Gu, Shane and Bi, Yingying and Andreev, Alek and Samangooei, Sina and Khan, Mina and Kocisky, Tomas and Filos, Angelos and Kumar, Chintu and Bishop, Colton and Yu, Adams and Hodkinson, Sarah and Mittal, Sid and Shah, Premal and Moufarek, Alexandre and Cheng, Yong and Bloniarz, Adam and Lee, Jaehoon and Pejman, Pedram and Michel, Paul and Spencer, Stephen and Feinberg, Vladimir and Xiong, Xuehan and Savinov, Nikolay and Smith, Charlotte and Shakeri, Siamak and Tran, Dustin and Chesus, Mary and Bohnet, Bernd and Tucker, George and Glehn, Tamara von and Muir, Carrie and Mao, Yiran and Kazawa, Hideto and Slone, Ambrose and Soparkar, Kedar and Shrivastava, Disha and Cobon-Kerr, James and Sharman, Michael and Pavagadhi, Jay and Araya, Carlos and Misiunas, Karolis and Ghelani, Nimesh and Laskin, Michael and Barker, David and Li, Qiujia and Briukhov, Anton and Houlsby, Neil and Glaese, Mia and Lakshminarayanan, Balaji and Schucher, Nathan and Tang, Yunhao and Collins, Eli and Lim, Hyeontaek and Feng, Fangxiaoyu and Recasens, Adria and Lai, Guangda and Magni, Alberto and Cao, Nicola De and Siddhant, Aditya and Ashwood, Zoe and Orbay, Jordi and Dehghani, Mostafa and Brennan, Jenny and He, Yifan and Xu, Kelvin and Gao, Yang and Saroufim, Carl and Molloy, James and Wu, Xinyi and Arnold, Seb and Chang, Solomon and Schrittwieser, Julian and Buchatskaya, Elena and Radpour, Soroush and Polacek, Martin and Giordano, Skye and Bapna, Ankur and Tokumine, Simon and Hellendoorn, Vincent and Sottiaux, Thibault and Cogan, Sarah and Severyn, Aliaksei and Saleh, Mohammad and Thakoor, Shantanu and Shefey, Laurent and Qiao, Siyuan and Gaba, Meenu and Chang, Shuo-yiin and Swanson, Craig and Zhang, Biao and Lee, Benjamin and Rubenstein, Paul Kishan and Song, Gan and Kwiatkowski, Tom and Koop, Anna and Kannan, Ajay and Kao, David and Schuh, Parker and Stjerngren, Axel and Ghiasi, Golnaz and Gibson, Gena and Vilnis, Luke and Yuan, Ye and Ferreira, Felipe Tiengo and Kamath, Aishwarya and Klimenko, Ted and Franko, Ken and Xiao, Kefan and Bhattacharya, Indro and Patel, Miteyan and Wang, Rui and Morris, Alex and Strudel, Robin and Sharma, Vivek and Choy, Peter and Hashemi, Sayed Hadi and Landon, Jessica and Finkelstein, Mara and Jhakra, Priya and Frye, Justin and Barnes, Megan and Mauger, Matthew and Daun, Dennis and Baatarsukh, Khuslen and Tung, Matthew and Farhan, Wael and Michalewski, Henryk and Viola, Fabio and Quitry, Felix de Chaumont and Lan, Charline Le and Hudson, Tom and Wang, Qingze and Fischer, Felix and Zheng, Ivy and White, Elspeth and Dragan, Anca and Alayrac, Jean-baptiste and Ni, Eric and Pritzel, Alexander and Iwanicki, Adam and Isard, Michael and Bulanova, Anna and Zilka, Lukas and Dyer, Ethan and Sachan, Devendra and Srinivasan, Srivatsan and Muckenhirn, Hannah and Cai, Honglong and Mandhane, Amol and Tariq, Mukarram and Rae, Jack W. and Wang, Gary and Ayoub, Kareem and FitzGerald, Nicholas and Zhao, Yao and Han, Woohyun and Alberti, Chris and Garrette, Dan and Krishnakumar, Kashyap and Gimenez, Mai and Levskaya, Anselm and Sohn, Daniel and Matak, Josip and Iturrate, Inaki and Chang, Michael B. and Xiang, Jackie and Cao, Yuan and Ranka, Nishant and Brown, Geoff and Hutter, Adrian and Mirrokni, Vahab and Chen, Nanxin and Yao, Kaisheng and Egyed, Zoltan and Galilee, Francois and Liechty, Tyler and Kallakuri, Praveen and Palmer, Evan and Ghemawat, Sanjay and Liu, Jasmine and Tao, David and Thornton, Chloe and Green, Tim and Jasarevic, Mimi and Lin, Sharon and Cotruta, Victor and Tan, Yi-Xuan and Fiedel, Noah and Yu, Hongkun and Chi, Ed and Neitz, Alexander and Heitkaemper, Jens and Sinha, Anu and Zhou, Denny and Sun, Yi and Kaed, Charbel and Hulse, Brice and Mishra, Swaroop and Georgaki, Maria and Kudugunta, Sneha and Farabet, Clement and Shafran, Izhak and Vlasic, Daniel and Tsitsulin, Anton and Ananthanarayanan, Rajagopal and Carin, Alen and Su, Guolong and Sun, Pei and V, Shashank and Carvajal, Gabriel and Broder, Josef and Comsa, Iulia and Repina, Alena and Wong, William and Chen, Warren Weilun and Hawkins, Peter and Filonov, Egor and Loher, Lucia and Hirnschall, Christoph and Wang, Weiyi and Ye, Jingchen and Burns, Andrea and Cate, Hardie and Wright, Diana Gage and Piccinini, Federico and Zhang, Lei and Lin, Chu-Cheng and Gog, Ionel and Kulizhskaya, Yana and Sreevatsa, Ashwin and Song, Shuang and Cobo, Luis C. and Iyer, Anand and Tekur, Chetan and Garrido, Guillermo and Xiao, Zhuyun and Kemp, Rupert and Zheng, Huaixiu Steven and Li, Hui and Agarwal, Ananth and Ngani, Christel and Goshvadi, Kati and Santamaria-Fernandez, Rebeca and Fica, Wojciech and Chen, Xinyun and Gorgolewski, Chris and Sun, Sean and Garg, Roopal and Ye, Xinyu and Eslami, S. M. Ali and Hua, Nan and Simon, Jon and Joshi, Pratik and Kim, Yelin and Tenney, Ian and Potluri, Sahitya and Thiet, Lam Nguyen and Yuan, Quan and Luisier, Florian and Chronopoulou, Alexandra and Scellato, Salvatore and Srinivasan, Praveen and Chen, Minmin and Koverkathu, Vinod and Dalibard, Valentin and Xu, Yaming and Saeta, Brennan and Anderson, Keith and Sellam, Thibault and Fernando, Nick and Huot, Fantine and Jung, Junehyuk and Varadarajan, Mani and Quinn, Michael and Raul, Amit and Le, Maigo and Habalov, Ruslan and Clark, Jon and Jalan, Komal and Bullard, Kalesha and Singhal, Achintya and Luong, Thang and Wang, Boyu and Rajayogam, Sujeevan and Eisenschlos, Julian and Jia, Johnson and Finchelstein, Daniel and Yakubovich, Alex and Balle, Daniel and Fink, Michael and Agarwal, Sameer and Li, Jing and Dvijotham, Dj and Pal, Shalini and Kang, Kai and Konzelmann, Jaclyn and Beattie, Jennifer and Dousse, Olivier and Wu, Diane and Crocker, Remi and Elkind, Chen and Jonnalagadda, Siddhartha Reddy and Lee, Jong and Holtmann-Rice, Dan and Kallarackal, Krystal and Liu, Rosanne and Vnukov, Denis and Vats, Neera and Invernizzi, Luca and Jafari, Mohsen and Zhou, Huanjie and Taylor, Lilly and Prendki, Jennifer and Wu, Marcus and Eccles, Tom and Liu, Tianqi and Kopparapu, Kavya and Beaufays, Francoise and Angermueller, Christof and Marzoca, Andreea and Sarcar, Shourya and Dib, Hilal and Stanway, Jeff and Perbet, Frank and Trdin, Nejc and Sterneck, Rachel and Khorlin, Andrey and Li, Dinghua and Wu, Xihui and Goenka, Sonam and Madras, David and Goldshtein, Sasha and Gierke, Willi and Zhou, Tong and Liu, Yaxin and Liang, Yannie and White, Anais and Li, Yunjie and Singh, Shreya and Bahargam, Sanaz and Epstein, Mark and Basu, Sujoy and Lao, Li and Ozturel, Adnan and Crous, Carl and Zhai, Alex and Lu, Han and Tung, Zora and Gaur, Neeraj and Walton, Alanna and Dixon, Lucas and Zhang, Ming and Globerson, Amir and Uy, Grant and Bolt, Andrew and Wiles, Olivia and Nasr, Milad and Shumailov, Ilia and Selvi, Marco and Piccinno, Francesco and Aguilar, Ricardo and McCarthy, Sara and Khalman, Misha and Shukla, Mrinal and Galic, Vlado and Carpenter, John and Villela, Kevin and Zhang, Haibin and Richardson, Harry and Martens, James and Bosnjak, Matko and Belle, Shreyas Rammohan and Seibert, Jeff and Alnahlawi, Mahmoud and McWilliams, Brian and Singh, Sankalp and Louis, Annie and Ding, Wen and Popovici, Dan and Simicich, Lenin and Knight, Laura and Mehta, Pulkit and Gupta, Nishesh and Shi, Chongyang and Fatehi, Saaber and Mitrovic, Jovana and Grills, Alex and Pagadora, Joseph and Munkhdalai, Tsendsuren and Petrova, Dessie and Eisenbud, Danielle and Zhang, Zhishuai and Yates, Damion and Mittal, Bhavishya and Tripuraneni, Nilesh and Assael, Yannis and Brovelli, Thomas and Jain, Prateek and Velimirovic, Mihajlo and Akbulut, Canfer and Mu, Jiaqi and Macherey, Wolfgang and Kumar, Ravin and Xu, Jun and Qureshi, Haroon and Comanici, Gheorghe and Wiesner, Jeremy and Gong, Zhitao and Ruddock, Anton and Bauer, Matthias and Felt, Nick and GP, Anirudh and Arnab, Anurag and Zelle, Dustin and Rothfuss, Jonas and Rosgen, Bill and Shenoy, Ashish and Seybold, Bryan and Li, Xinjian and Mudigonda, Jayaram and Erdogan, Goker and Xia, Jiawei and Simsa, Jiri and Michi, Andrea and Yao, Yi and Yew, Christopher and Kan, Steven and Caswell, Isaac and Radebaugh, Carey and Elisseeff, Andre and Valenzuela, Pedro and McKinney, Kay and Paterson, Kim and Cui, Albert and Latorre-Chimoto, Eri and Kim, Solomon and Zeng, William and Durden, Ken and Ponnapalli, Priya and Sosea, Tiberiu and Choquette-Choo, Christopher A. and Manyika, James and Robenek, Brona and Vashisht, Harsha and Pereira, Sebastien and Lam, Hoi and Velic, Marko and Owusu-Afriyie, Denese and Lee, Katherine and Bolukbasi, Tolga and Parrish, Alicia and Lu, Shawn and Park, Jane and Venkatraman, Balaji and Talbert, Alice and Rosique, Lambert and Cheng, Yuchung and Sozanschi, Andrei and Paszke, Adam and Kumar, Praveen and Austin, Jessica and Li, Lu and Salama, Khalid and Perz, Bartek and Kim, Wooyeol and Dukkipati, Nandita and Baryshnikov, Anthony and Kaplanis, Christos and Sheng, XiangHai and Chervonyi, Yuri and Unlu, Caglar and Casas, Diego de Las and Askham, Harry and Tunyasuvunakool, Kathryn and Gimeno, Felix and Poder, Siim and Kwak, Chester and Miecnikowski, Matt and Mirrokni, Vahab and Dimitriev, Alek and Parisi, Aaron and Liu, Dangyi and Tsai, Tomy and Shevlane, Toby and Kouridi, Christina and Garmon, Drew and Goedeckemeyer, Adrian and Brown, Adam R. and Vijayakumar, Anitha and Elqursh, Ali and Jazayeri, Sadegh and Huang, Jin and Carthy, Sara Mc and Hoover, Jay and Kim, Lucy and Kumar, Sandeep and Chen, Wei and Biles, Courtney and Bingham, Garrett and Rosen, Evan and Wang, Lisa and Tan, Qijun and Engel, David and Pongetti, Francesco and Cesare, Dario de and Hwang, Dongseong and Yu, Lily and Pullman, Jennifer and Narayanan, Srini and Levin, Kyle and Gopal, Siddharth and Li, Megan and Aharoni, Asaf and Trinh, Trieu and Lo, Jessica and Casagrande, Norman and Vij, Roopali and Matthey, Loic and Ramadhana, Bramandia and Matthews, Austin and Carey, CJ and Johnson, Matthew and Goranova, Kremena and Shah, Rohin and Ashraf, Shereen and Dasgupta, Kingshuk and Larsen, Rasmus and Wang, Yicheng and Vuyyuru, Manish Reddy and Jiang, Chong and Ijazi, Joana and Osawa, Kazuki and Smith, Celine and Boppana, Ramya Sree and Bilal, Taylan and Koizumi, Yuma and Xu, Ying and Altun, Yasemin and Shabat, Nir and Bariach, Ben and Korchemniy, Alex and Choo, Kiam and Ronneberger, Olaf and Iwuanyanwu, Chimezie and Zhao, Shubin and Soergel, David and Hsieh, Cho-Jui and Cai, Irene and Iqbal, Shariq and Sundermeyer, Martin and Chen, Zhe and Bursztein, Elie and Malaviya, Chaitanya and Biadsy, Fadi and Shroff, Prakash and Dhillon, Inderjit and Latkar, Tejasi and Dyer, Chris and Forbes, Hannah and Nicosia, Massimo and Nikolaev, Vitaly and Greene, Somer and Georgiev, Marin and Wang, Pidong and Martin, Nina and Sedghi, Hanie and Zhang, John and Banzal, Praseem and Fritz, Doug and Rao, Vikram and Wang, Xuezhi and Zhang, Jiageng and Patraucean, Viorica and Du, Dayou and Mordatch, Igor and Jurin, Ivan and Liu, Lewis and Dubey, Ayush and Mohan, Abhi and Nowakowski, Janek and Ion, Vlad-Doru and Wei, Nan and Tojo, Reiko and Raad, Maria Abi and Hudson, Drew A. and Keshava, Vaishakh and Agrawal, Shubham and Ramirez, Kevin and Wu, Zhichun and Nguyen, Hoang and Liu, Ji and Sewak, Madhavi and Petrini, Bryce and Choi, DongHyun and Philips, Ivan and Wang, Ziyue and Bica, Ioana and Garg, Ankush and Wilkiewicz, Jarek and Agrawal, Priyanka and Li, Xiaowei and Guo, Danhao and Xue, Emily and Shaik, Naseer and Leach, Andrew and Khan, Sadh MNM and Wiesinger, Julia and Jerome, Sammy and Chakladar, Abhishek and Wang, Alek Wenjiao and Ornduff, Tina and Abu, Folake and Ghaffarkhah, Alireza and Wainwright, Marcus and Cortes, Mario and Liu, Frederick and Maynez, Joshua and Terzis, Andreas and Samangouei, Pouya and Mansour, Riham and Kępa, Tomasz and Aubet, François-Xavier and Algymr, Anton and Banica, Dan and Weisz, Agoston and Orban, Andras and Senges, Alexandre and Andrejczuk, Ewa and Geller, Mark and Santo, Niccolo Dal and Anklin, Valentin and Merey, Majd Al and Baeuml, Martin and Strohman, Trevor and Bai, Junwen and Petrov, Slav and Wu, Yonghui and Hassabis, Demis and Kavukcuoglu, Koray and Dean, Jeff and Vinyals, Oriol},
  year = {2024},
  doi = {10.48550/arxiv.2403.05530},
  eprint = {2403.05530},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.05530}
}

References

199 references, 57 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. CoRR, abs/2404.11018, 2024a. find our card
  2. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024b. find our card
  3. Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752, 2023. find
  4. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024. find
  5. Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation, 2018. find
  6. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023a. find our card
  7. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 Technical Report, 2023b. find our card
  8. Anthropic. Model Card and Evaluations for Claude Models, 2023a. find
  9. Anthropic. Long context prompting for Claude 2.1, 2023b. find
  10. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. find our card
  11. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… find our card
  12. Ivana Balažević, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Memory consolidation enables long-context video understanding. arXiv preprint arXiv:2402.05861, 2024. — [arXiv:2402.05861](https://arxiv.org/abs/2402.05861) find
  13. Suzanna Becker and Yann LeCun. Improving the convergence of back-propagation learning with second-order methods. 1989. URL https://api.semanticscholar.org/CorpusID: 59695337 . ↗
  14. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. find
  15. Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley. Unlimiformer: Long-range transformers with unlimited length input. arXiv preprint arXiv:2305.01625, 2023. find
  16. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024. find
  17. Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent, 2021. find
  18. Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024. find
  19. Steven Bird. Decolonising speech and language technology. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, pages 3504–3519, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguis… find
  20. Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluis Gomez, Marçal Rusiñol, C.V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4290–4300, 2019. doi: 10.1109/ICCV.2019.00439. — [doi:10.1109/iccv.2019.00439](https://doi.org/10.1109/iccv.2019.00439) find
  21. Bernd Bohnet, Kevin Swersky, Rosanne Liu, Pranjal Awasthi, Azade Nova, Javier Snaider, Hanie Sedghi, Aaron T Parisi, Michael Collins, Angeliki Lazaridou, Orhan Firat, and Noah Fiedel. Long-span question-answering: Automatic question generation and qa-system ranking via side-by-side evaluation, 2024. find
  22. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/ google/jax . ↗
  23. Ralph A. Bradley and Milton E. Terry. The rank analysis of incomplete block designs — I. The method of paired comparisons. Biometrika, 39:324–345, 1952. find
  24. Thorsten Brants, Ashok Popat, Peng Xu, Franz Och, and Jeffrey Dean. Large language models in machine translation. pages 858–867, 01 2007. find our card
  25. Andrei Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, sep 2002. ISSN 0163-5840. doi: 10.1145/792550.792552. URL https://doi.org/10.1145/792550.792552 . — [doi:10.1145/792550.792552](https://doi.org/10.1145/792550.792552) ↗
  26. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  27. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper\_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf . ↗
  28. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Knowledge Discovery and Data Mining, 2006. URL https://api.semanticscholar.org/CorpusID: 11253972 . ↗
  29. Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11079–11091. Curran Associates, Inc., 2022. URL https://proceedings.neuri… ↗
  30. Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling transformer to 1m tokens and beyond with rmt. arXiv preprint arXiv:2304.11062, 2023. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) find
  31. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018. find
  32. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. URL https://arxiv.org/abs/2202.07646 . ↗
  33. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?, 2024. find
  34. Ben Carterette, Paul Clough, Mark Hall, Evangelos Kanoulas, and Mark Sanderson. Evaluating retrieval over sessions: The TREC session track 2011-2014. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, page 685–688, New York,… — [doi:10.1145/2911451.2914675](https://doi.org/10.1145/2911451.2914675) find
  35. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. find our card
  36. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
  37. Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394, 1999. find
  38. Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In International Symposium on Research in Attacks, Intrusions and Defenses, pages 654–668, April 2023a. URL https://arxiv.org/ab… — [arXiv:2304.00409](https://arxiv.org/abs/2304.00409) ↗
  39. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023b. URL https://arxiv.org/abs/2309.12307 . ↗ our card
  40. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023a. find
  41. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1–113, 2023b. URL http://jmlr.org/papers… ↗
  42. Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, Dav… — [arXiv:2202.01169](https://arxiv.org/abs/2202.01169) find
  43. Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. IJCAI, 2020. URL https://www.ijcai.org/proceedings/2020/0537.pdf . ↗ our card
  44. Charles LA Clarke, Saira Rizvi, Mark D Smucker, Maria Maistro, and Guido Zuccon. Overview of the TREC 2020 health misinformation track. In Proceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020), 2020. URL https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.HM. pdf . ↗
  45. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 . ↗ our card
  46. Kevyn Collins-Thompson, Craig Macdonald, Paul N. Bennett, Fernando Diaz, and Ellen M. Voorhees. TREC 2014 web track overview. In Proceedings of the Twenty-Third Text REtrieval Conference (TREC 2014), 2014. URL https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf . ↗
  47. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. find our card
  48. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakra… find our card
  49. Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks, 2014. find
  50. Jeff Dean. tecture, Introducing Pathways: A next-generation AI archi-2021. URL https://blog.google/technology/ai/ introducing-pathways-next-generation-ai-architecture/ . ↗
  51. Dhinakaran, URL https://twitter.com/aparnadhinak/status/1744771295940669689 . Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page… ↗
  52. Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we? arXiv preprint:2403.18624, 2024. — [arXiv:2403.18624](https://arxiv.org/abs/2403.18624) find
  53. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas… find our card
  54. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find our card
  55. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159, 2011. URL http://jmlr.org/papers/v12/duchi11a.html . ↗
  56. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130, 2023. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) find
  57. Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. arXiv preprint arXiv:2406.18321, 2024. find
  58. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. URL https: //arxiv.org/abs/2101.03961 . find
  59. Edward W Felten, Manav Raj, and Robert Seamans. A method to link advances in artificial intelligence to occupational abilities. In AEA Papers and Proceedings, volume 108, pages 54–57. American Economic Association 2014 Broadway, Suite 305, Nashville, TN 37203, 2018. find
  60. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2023. find our card
  61. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390, 2024. find
  62. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. The unreasonable effectiveness of few-shot learning for machine translation, 2023. find
  63. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, November 2020. Association for Computational… find our card
  64. Gemini-Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. URL https://storage. googleapis.com/deepmind-media/g… find our card
  65. Gemma-Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. find
  66. Google. Google’s AI Principles. 2023. find
  67. URL https://ai.google/responsibility/principles/ . Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern reco… ↗ our card
  68. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. URL https://arxiv.org/abs/2312.00752 . ↗
  69. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, 36, 2024. find
  70. Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.07916, 2021. URL https://arxiv.org/abs/2112.07916 . ↗ our card
  71. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929… find
  72. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. find
  73. Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http: //github.com/google/flax . find
  74. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. find our card
  75. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021b. URL https://arxiv.org/abs/2103.03874 . ↗ our card
  76. Tom Heskes. On “Natural” Learning and Pruning in Multilayered Perceptrons. Neural Computation, 12(4):881–901, 04 2000. ISSN 0899-7667. doi: 10.1162/089976600300015637. URL https: //doi.org/10.1162/089976600300015637 . — [doi:10.1162/089976600300015637](https://doi.org/10.1162/089976600300015637) find
  77. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. find our card
  78. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… find
  79. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. find
  80. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. URL https:// arxiv.org/abs/210.… — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
  81. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022. URL https://arxiv.org/ abs/2208.03299 . ↗
  82. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. find
  83. Frederick Jelinek. Statistical methods for speech recognition. MIT press, 1998. find
  84. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  85. Zhengbao Jiang, Luyu Gao, Zhiruo Wang, Jun Araki, Haibo Ding, Jamie Callan, and Graham Neubig. Retrieval as attention: End-to-end learning of retrieval and reading within a single transformer. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirica… find
  86. Alex Jones, Isaac Caswell, Ishank Saxena, and Orhan Firat. Bilex rx: Lexical data augmentation for massively multilingual machine translation. arXiv preprint arXiv:2303.15265, 2023. find
  87. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling, 2016. URL https://arxiv.org/abs/1602.02410 . ↗
  88. URL https://github.com/gkamradt/LLMTest\_NeedleInAHaystack/blob/main/README.md . Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020… ↗
  89. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Nat… find our card
  90. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. find our card
  91. Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465, 2021. — [arXiv:2109.10465](https://arxiv.org/abs/2109.10465) find
  92. Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, et al. Evaluating language-model agents on realistic autonomous tasks. arXiv preprint arXiv:2312.11671, 2023. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) find
  93. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. find our card
  94. R. Kneser and H. Ney. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 181–184 vol.1, 1995. doi: 10.1109/ICASSP.1995.479394. — [doi:10.1109/icassp.1995.479394](https://doi.org/10.1109/icassp.1995.479394) find
  95. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin P… find
  96. Greg Kohs. Alphago. Motion Picture, 2017. Produced by DeepMind Technologies and distributed by Netflix. find
  97. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A multilingual and document-level large audited dataset, 2023. find our card
  98. J. Landeghem, R. Powalski, R. Tito, D. Jurkiewicz, M. Blaschko, L. Borchmann, M. Coustaty, S. Moens, M. Pietruszka, B. Ackaert, T. Stanislawek, P. Joziak, and E. Valveny. Document understanding dataset and evaluation (dude). In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages… find
  99. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020. URL https://openreview… find
  100. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. URL https://arxiv.org/abs/2206.14858… ↗ our card
  101. Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Eisenschlos. MatCha: Enhancing visual language pretraining with math reasoning and chart derendering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proc… find
  102. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention, 2024. find
  103. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. find our card
  104. MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024, February 2024. URL https://maa.org/math-competitions/ american-invitational-mathematics-examination-aime . ↗
  105. Macknight, Aung, and Gomes. Personal Communication. Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In 2nd Workshop o… find
  106. Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, and Chris Alberti. Dolomites: Domain-specific long-form methodical tasks, 2024. find
  107. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. find
  108. Pedro Henrique Martins, Zita Marinho, and Andre Martins. ∞-former: Infinite memory transformer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5468–5485, Dubl… find
  109. Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, 2022. find our card
  110. MATH-AI-2023-Panel. Panel discussion. In MATH-AI Workshop at NeurIPS 2023, New Orleans, Louisiana, USA, December 2023. URL https://mathai2023.github.io/ . ↗
  111. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. find our card
  112. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. find
  113. Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Fengqing Jiang, Aidan O’Gara, Ellie Sakhaee, Zhen Xiang, Arezoo Rajabi, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth. Tdc 2023 (llm edition): The trojan detection challenge. In NeurIPS Competition Track,… find
  114. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. find our card
  115. J. McMurry. Organic Chemistry. International edition. Brooks/Cole Cengage Learning, 2012. ISBN 9780840054531. URL https://books.google.at/books?id=oVv4Az7VJRYC . ↗
  116. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  117. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods i… find our card
  118. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY,… find
  119. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on Fairness, Accountability, and Transparency, pages 220–229, 2019b. find
  120. Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467, 2023. find our card
  121. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023. find our card
  122. OpenAI. GPT-4 Technical Report. 2023a. find our card
  123. OpenAI. GPT-4V(ision) System Card, 2023b. find
  124. OpenAI. Whisper, 2023. URL https://github.com/openai/whisper . ↗
  125. Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349, 2023. find
  126. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. CoRR, abs/2110.08193, 2021. URL https://arxiv.org/abs/2110.08193 . ↗ our card
  127. Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gregoire D… find
  128. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. find
  129. Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–3… find our card
  130. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020. find our card
  131. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. find
  132. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… find
  133. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http: //jmlr.org/papers/v21/2… find
  134. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. find our card
  135. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, 2021. find
  136. William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Be… find
  137. Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555–17566, 2021. find
  138. Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016. find
  139. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of Gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. find
  140. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488, 2021. find our card
  141. Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-… find
  142. Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379–423, 1948. URL http://plan9.bell-labs.com/cm/ms/what/shannonday/ shannon1948.pdf . ↗
  143. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR (Poster). OpenReview.net, 2017. URL http://dblp.uni-trier.de/db/conf/iclr/ iclr2017.html\#ShazeerMMDLHD17 . ↗
  144. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Al… find
  145. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language Models are Multilingual Chain-of-Thought Reasoners. In Proceedings of ICLR 2023, 2023a. URL http: //arxiv.org/abs/2210.03… find our card
  146. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. arXiv preprint arXiv:2310.10638, 2023b. find
  147. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. find
  148. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523–1… find our card
  149. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. URL https://arxiv.org/abs/ 2206.04615 . ↗ our card
  150. Saurabh Srivastava, Annarose M B, Anto P V au2, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap, 2024. find
  151. Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Henryk Michalewski, Łukasz Kuciński, and Piotr Miłoś. Structured packing in llm training improves long context utilization. arXiv preprint arXiv:2312.17296, 2023. — [arXiv:2312.17296](https://arxiv.org/abs/2312.17296) find
  152. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  153. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In Arxiv, 2023. find
  154. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  155. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239 . ↗ our card
  156. Kocmi Tom, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. Findings of the 2023 conference on machine translation (wmt23): Llms are here but not quite there yet. In WMT23-Eighth Conferenc… find
  157. Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset, 2024. find
  158. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint… find
  159. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  160. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36, 2024. find
  161. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762 . ↗
  162. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. find our card
  163. Eline Visser. Kalamang dictionary. Dictionaria, (13):1–2737, 2020a. URL https://dictionaria. clld.org/contributions/kalamang . find
  164. Eline Visser. A grammar of kalamang: The papuan language of the karas islands. 2020b. find
  165. Eline Visser. The Kalamang collection: an archive of linguistic and cultural material from Karas. Lund: Humlab, Lund University Humanities Lab, 2020c. URL http://hdl.handle.net/10050/ 00-0000-0000-0003-C3E8-1 . ↗
  166. Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310, 2020. find
  167. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021. find our card
  168. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VATEX: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019a. find
  169. Xinda Wang, Kun Sun, Archer Batcheller, and Sushil Jajodia. Detecting "0-day" vulnerability: An empirical study of secret security patch in OSS. In IEEE/IFIP International Conference on Dependable Systems and Networks, pages 485–492. IEEE, 2019b. URL https://csis.gmu.edu/ksun/ publications/secretpat… ↗
  170. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. find our card
  171. Laura Weidinger, Joslyn Barnhart, Jenny Brennan, Christina Butterfield, Susie Young, Will Hawkins, Lisa Anne Hendricks, Ramona Comanescu, Oscar Chang, Mikel Rodriguez, et al. Holistic safety and responsibility evaluations of advanced ai models. arXiv preprint arXiv:2404.14068, 2024. — [arXiv:2404.14068](https://arxiv.org/abs/2404.14068) find
  172. Wikipedia contributors. Skeletal formula — Wikipedia, the free encyclopedia, 2024. URL https:// en.wikipedia.org/w/index.php?title=Skeletal\_formula&oldid=1221781655 . [Online; accessed 7-May-2024]. find
  173. World Economic Forum. Jobs of Tomorrow: Large Language Models and Jobs. 2023. URL https: //www3.weforum.org/docs/WEF\_Jobs\_of\_Tomorrow\_Generative\_AI\_2023.pdf . find
  174. Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. arXiv preprint arXiv:2312.14135, 2023. find
  175. Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-augmented transformer for sequence modeling. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, editors, Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 3… find
  176. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022b. wunderwuzzi. Hacking Google Bard - From Prompt Injection to Data Exfiltration. https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/, 2023. x.ai. Gr… ↗
  177. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. find our card
  178. XLA. XLA: Optimizing compiler for TensorFlow. https://www.tensorflow.org/xla , 2019. [Online; accessed December-2023]. ↗
  179. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint arXiv:2105.04663, 2021. — [arXiv:2105.04663](https://arxiv.org/abs/2105.04663) find
  180. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. 2024. find
  181. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2023a. find our card
  182. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Standardizing and benchmarking interactive coding with execution feedback. ArXiv, June 2023b. URL https: //arxiv.org/abs/2306.14898 . find
  183. Kenneth found Yeung. misuse. New google 2024. gemini URL vulnerability enabling pro-https://hiddenlayer.com/ research/new-google-gemini-content-manipulation-vulns-found/ \#Indirect-Injections-are-back! find
  184. Emine Yilmaz, Manisha Verma, Rishabh Mehrotra, Evangelos Kanoulas, Ben Carterette, and Nick Craswell. Overview of the TREC 2015 tasks track. In Proceedings of the Twenty-Fourth Text REtrieval Conference (TREC 2015), 2015. URL https://trec.nist.gov/pubs/trec24/papers/ Overview-T.pdf . ↗
  185. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019. find our card
  186. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… find our card
  187. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297, 2020. find our card
  188. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  189. Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: a case study. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023a. find our card
  190. Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. Advances in Neural Information Processing Systems, 35:21682–21697, 2022. find
  191. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath,… — [arXiv:2303.01037](https://arxiv.org/abs/2303.01037) find
  192. Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14830–14840, 2021. find
  193. Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5657–5673, Abu Dhabi, United Arab Emirates, December 20… find
  194. Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, pages 7590–7598, 2018. find
  195. Yaqin Zhou, Jing Kai Siow, Chenyu Wang, Shangqing Liu, and Yang Liu. SPI: Automated identification of security patches via commits. ACM Transactions on Software Engineering and Methodology, 31 (1):1–27, 2021. find
  196. Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, page 4857–4866, 2022. find
  197. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906, 2022. URL https://arxiv.org/abs/2202.08906 . ↗
  198. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. \#\# 11. Contributions and Acknowledgments Core Contributors Petko Georgiev Ving Ian Lei Ryan Burnell Libin Bai Anmol Gulati Garre… find our card
  199. Within each role, contributions are equal, and are listed in a randomized order. Ordering within each role does not indicate ordering of the contributions. Gemini is a cross-Google effort, with members from Google DeepMind (GDM), Google Research (GR), Bard/Assistant, Knowledge and Information (K&I),… find