Elamur Research
☰
home / papers / 2023 / Gemini: A Family of Highly Capable Multimodal Models

Gemini: A Family of Highly Capable Multimodal Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2312.11805

Summary

This technical report from Google introduces Gemini 1.0, a family of models trained jointly on text, images, audio and video from the start, instead of adding vision or speech to a text-only LLM later. The aim is one system that handles all modalities well and still competes with specialist models in each. The architecture is a Transformer decoder with a 32k-token context and efficient attention. It was trained on TPUv4 and TPUv5e clusters spread across several datacenters. There are three sizes: Ultra, Pro and Nano, which runs on-device. After pre-training, the models were post-trained into chat variants (Gemini Apps) and developer variants (Gemini API). Gemini Ultra sets a new best result on 30 of 32 reported benchmarks, including every image, video and speech benchmark tested. It is described as the first model to pass human-expert level on MMLU. The report also covers safety work: impact assessments, red-teaming and external testing. The models are proprietary and available only through Google services.

Key points

  • Gemini Ultra scores 90.0% on MMLU, above the human-expert level. On MMMU it scores 62.4%, more than 5 points ahead of the previous best model.
  • Ultra's state-of-the-art wins break down as 10/12 text and reasoning tasks, 9/9 image, 6/6 video and 5/5 speech recognition and speech translation benchmarks.
  • Nano comes in two versions, Nano-1 (1.8B parameters) and Nano-2 (3.25B). Both are distilled from larger Gemini models and quantized to 4 bits for phones and other devices.
  • Audio goes in directly as 16kHz Universal Speech Model (USM) features, and video goes in as a sequence of frames. The models can also generate images using discrete image tokens.
  • Training kept redundant copies of the model state in memory instead of saving periodic checkpoints, which raised goodput (useful training throughput) from 85% to 97%. AlphaCode 2, an agent built on Gemini, ranks in the top 15% of Codeforces entrants.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

2130 papers of our library cite it (of the 140673 whose bibliographies we read); 76 of them have a page here, 2054 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.11805 · arXiv 2312.11805

@misc{team2023gemini,
  title = {{Gemini: A Family of Highly Capable Multimodal Models}},
  author = {Team, Gemini and Anil, Rohan and Borgeaud, Sebastian and Alayrac, Jean-Baptiste and Yu, Jiahui and Soricut, Radu and Schalkwyk, Johan and Dai, Andrew M. and Hauth, Anja and Millican, Katie and Silver, David and Johnson, Melvin and Antonoglou, Ioannis and Schrittwieser, Julian and Glaese, Amelia and Chen, Jilin and Pitler, Emily and Lillicrap, Timothy and Lazaridou, Angeliki and Firat, Orhan and Molloy, James and Isard, Michael and Barham, Paul R. and Hennigan, Tom and Lee, Benjamin and Viola, Fabio and Reynolds, Malcolm and Xu, Yuanzhong and Doherty, Ryan and Collins, Eli and Meyer, Clemens and Rutherford, Eliza and Moreira, Erica and Ayoub, Kareem and Goel, Megha and Krawczyk, Jack and Du, Cosmo and Chi, Ed and Cheng, Heng-Tze and Ni, Eric and Shah, Purvi and Kane, Patrick and Chan, Betty and Faruqui, Manaal and Severyn, Aliaksei and Lin, Hanzhao and Li, YaGuang and Cheng, Yong and Ittycheriah, Abe and Mahdieh, Mahdis and Chen, Mia and Sun, Pei and Tran, Dustin and Bagri, Sumit and Lakshminarayanan, Balaji and Liu, Jeremiah and Orban, Andras and Güra, Fabian and Zhou, Hao and Song, Xinying and Boffy, Aurelien and Ganapathy, Harish and Zheng, Steven and Choe, HyunJeong and Weisz, Ágoston and Zhu, Tao and Lu, Yifeng and Gopal, Siddharth and Kahn, Jarrod and Kula, Maciej and Pitman, Jeff and Shah, Rushin and Taropa, Emanuel and Merey, Majd Al and Baeuml, Martin and Chen, Zhifeng and Shafey, Laurent El and Zhang, Yujing and Sercinoglu, Olcan and Tucker, George and Piqueras, Enrique and Krikun, Maxim and Barr, Iain and Savinov, Nikolay and Danihelka, Ivo and Roelofs, Becca and White, Anaïs and Andreassen, Anders and Glehn, Tamara von and Yagati, Lakshman and Kazemi, Mehran and Gonzalez, Lucas and Khalman, Misha and Sygnowski, Jakub and Frechette, Alexandre and Smith, Charlotte and Culp, Laura and Proleev, Lev and Luan, Yi and Chen, Xi and Lottes, James and Schucher, Nathan and Lebron, Federico and Rrustemi, Alban and Clay, Natalie and Crone, Phil and Kocisky, Tomas and Zhao, Jeffrey and Perz, Bartek and Yu, Dian and Howard, Heidi and Bloniarz, Adam and Rae, Jack W. and Lu, Han and Sifre, Laurent and Maggioni, Marcello and Alcober, Fred and Garrette, Dan and Barnes, Megan and Thakoor, Shantanu and Austin, Jacob and Barth-Maron, Gabriel and Wong, William and Joshi, Rishabh and Chaabouni, Rahma and Fatiha, Deeni and Ahuja, Arun and Tomar, Gaurav Singh and Senter, Evan and Chadwick, Martin and Kornakov, Ilya and Attaluri, Nithya and Iturrate, Iñaki and Liu, Ruibo and Li, Yunxuan and Cogan, Sarah and Chen, Jeremy and Jia, Chao and Gu, Chenjie and Zhang, Qiao and Grimstad, Jordan and Hartman, Ale Jakse and Garcia, Xavier and Pillai, Thanumalayan Sankaranarayana and Devlin, Jacob and Laskin, Michael and Casas, Diego de Las and Valter, Dasha and Tao, Connie and Blanco, Lorenzo and Badia, Adrià Puigdomènech and Reitter, David and Chen, Mianna and Brennan, Jenny and Rivera, Clara and Brin, Sergey and Iqbal, Shariq and Surita, Gabriela and Labanowski, Jane and Rao, Abhi and Winkler, Stephanie and Parisotto, Emilio and Gu, Yiming and Olszewska, Kate and Addanki, Ravi and Miech, Antoine and Louis, Annie and Teplyashin, Denis and Brown, Geoff and Catt, Elliot and Balaguer, Jan and Xiang, Jackie and Wang, Pidong and Ashwood, Zoe and Briukhov, Anton and Webson, Albert and Ganapathy, Sanjay and Sanghavi, Smit and Kannan, Ajay and Chang, Ming-Wei and Stjerngren, Axel and Djolonga, Josip and Sun, Yuting and Bapna, Ankur and Aitchison, Matthew and Pejman, Pedram and Michalewski, Henryk and Yu, Tianhe and Wang, Cindy and Love, Juliette and Ahn, Junwhan and Bloxwich, Dawn and Han, Kehang and Humphreys, Peter and Sellam, Thibault and Bradbury, James and Godbole, Varun and Samangooei, Sina and Damoc, Bogdan and Kaskasoli, Alex and Arnold, Sébastien M. R. and Vasudevan, Vijay and Agrawal, Shubham and Riesa, Jason and Lepikhin, Dmitry and Tanburn, Richard and Srinivasan, Srivatsan and Lim, Hyeontaek and Hodkinson, Sarah and Shyam, Pranav and Ferret, Johan and Hand, Steven and Garg, Ankush and Paine, Tom Le and Li, Jian and Li, Yujia and Giang, Minh and Neitz, Alexander and Abbas, Zaheer and York, Sarah and Reid, Machel and Cole, Elizabeth and Chowdhery, Aakanksha and Das, Dipanjan and Rogozińska, Dominika and Nikolaev, Vitaliy and Sprechmann, Pablo and Nado, Zachary and Zilka, Lukas and Prost, Flavien and He, Luheng and Monteiro, Marianne and Mishra, Gaurav and Welty, Chris and Newlan, Josh and Jia, Dawei and Allamanis, Miltiadis and Hu, Clara Huiyi and Liedekerke, Raoul de and Gilmer, Justin and Saroufim, Carl and Rijhwani, Shruti and Hou, Shaobo and Shrivastava, Disha and Baddepudi, Anirudh and Goldin, Alex and Ozturel, Adnan and Cassirer, Albin and Xu, Yunhan and Sohn, Daniel and Sachan, Devendra and Amplayo, Reinald Kim and Swanson, Craig and Petrova, Dessie and Narayan, Shashi and Guez, Arthur and Brahma, Siddhartha and Landon, Jessica and Patel, Miteyan and Zhao, Ruizhe and Villela, Kevin and Wang, Luyu and Jia, Wenhao and Rahtz, Matthew and Giménez, Mai and Yeung, Legg and Keeling, James and Georgiev, Petko and Mincu, Diana and Wu, Boxi and Haykal, Salem and Saputro, Rachel and Vodrahalli, Kiran and Qin, James and Cankara, Zeynep and Sharma, Abhanshu and Fernando, Nick and Hawkins, Will and Neyshabur, Behnam and Kim, Solomon and Hutter, Adrian and Agrawal, Priyanka and Castro-Ros, Alex and Driessche, George van den and Wang, Tao and Yang, Fan and Chang, Shuo-yiin and Komarek, Paul and McIlroy, Ross and Lučić, Mario and Zhang, Guodong and Farhan, Wael and Sharman, Michael and Natsev, Paul and Michel, Paul and Bansal, Yamini and Qiao, Siyuan and Cao, Kris and Shakeri, Siamak and Butterfield, Christina and Chung, Justin and Rubenstein, Paul Kishan and Agrawal, Shivani and Mensch, Arthur and Soparkar, Kedar and Lenc, Karel and Chung, Timothy and Pope, Aedan and Maggiore, Loren and Kay, Jackie and Jhakra, Priya and Wang, Shibo and Maynez, Joshua and Phuong, Mary and Tobin, Taylor and Tacchetti, Andrea and Trebacz, Maja and Robinson, Kevin and Katariya, Yash and Riedel, Sebastian and Bailey, Paige and Xiao, Kefan and Ghelani, Nimesh and Aroyo, Lora and Slone, Ambrose and Houlsby, Neil and Xiong, Xuehan and Yang, Zhen and Gribovskaya, Elena and Adler, Jonas and Wirth, Mateo and Lee, Lisa and Li, Music and Kagohara, Thais and Pavagadhi, Jay and Bridgers, Sophie and Bortsova, Anna and Ghemawat, Sanjay and Ahmed, Zafarali and Liu, Tianqi and Powell, Richard and Bolina, Vijay and Iinuma, Mariko and Zablotskaia, Polina and Besley, James and Chung, Da-Woon and Dozat, Timothy and Comanescu, Ramona and Si, Xiance and Greer, Jeremy and Su, Guolong and Polacek, Martin and Kaufman, Raphaël Lopez and Tokumine, Simon and Hu, Hexiang and Buchatskaya, Elena and Miao, Yingjie and Elhawaty, Mohamed and Siddhant, Aditya and Tomasev, Nenad and Xing, Jinwei and Greer, Christina and Miller, Helen and Ashraf, Shereen and Roy, Aurko and Zhang, Zizhao and Ma, Ada and Filos, Angelos and Besta, Milos and Blevins, Rory and Klimenko, Ted and Yeh, Chih-Kuan and Changpinyo, Soravit and Mu, Jiaqi and Chang, Oscar and Pajarskas, Mantas and Muir, Carrie and Cohen, Vered and Lan, Charline Le and Haridasan, Krishna and Marathe, Amit and Hansen, Steven and Douglas, Sholto and Samuel, Rajkumar and Wang, Mingqiu and Austin, Sophia and Lan, Chang and Jiang, Jiepu and Chiu, Justin and Lorenzo, Jaime Alonso and Sjösund, Lars Lowe and Cevey, Sébastien and Gleicher, Zach and Avrahami, Thi and Boral, Anudhyan and Srinivasan, Hansa and Selo, Vittorio and May, Rhys and Aisopos, Konstantinos and Hussenot, Léonard and Soares, Livio Baldini and Baumli, Kate and Chang, Michael B. and Recasens, Adrià and Caine, Ben and Pritzel, Alexander and Pavetic, Filip and Pardo, Fabio and Gergely, Anita and Frye, Justin and Ramasesh, Vinay and Horgan, Dan and Badola, Kartikeya and Kassner, Nora and Roy, Subhrajit and Dyer, Ethan and Campos, Víctor Campos and Tomala, Alex and Tang, Yunhao and Badawy, Dalia El and White, Elspeth and Mustafa, Basil and Lang, Oran and Jindal, Abhishek and Vikram, Sharad and Gong, Zhitao and Caelles, Sergi and Hemsley, Ross and Thornton, Gregory and Feng, Fangxiaoyu and Stokowiec, Wojciech and Zheng, Ce and Thacker, Phoebe and Ünlü, Çağlar and Zhang, Zhishuai and Saleh, Mohammad and Svensson, James and Bileschi, Max and Patil, Piyush and Anand, Ankesh and Ring, Roman and Tsihlas, Katerina and Vezer, Arpi and Selvi, Marco and Shevlane, Toby and Rodriguez, Mikel and Kwiatkowski, Tom and Daruki, Samira and Rong, Keran and Dafoe, Allan and FitzGerald, Nicholas and Gu-Lemberg, Keren and Khan, Mina and Hendricks, Lisa Anne and Pellat, Marie and Feinberg, Vladimir and Cobon-Kerr, James and Sainath, Tara and Rauh, Maribeth and Hashemi, Sayed Hadi and Ives, Richard and Hasson, Yana and Noland, Eric and Cao, Yuan and Byrd, Nathan and Hou, Le and Wang, Qingze and Sottiaux, Thibault and Paganini, Michela and Lespiau, Jean-Baptiste and Moufarek, Alexandre and Hassan, Samer and Shivakumar, Kaushik and Amersfoort, Joost van and Mandhane, Amol and Joshi, Pratik and Goyal, Anirudh and Tung, Matthew and Brock, Andrew and Sheahan, Hannah and Misra, Vedant and Li, Cheng and Rakićević, Nemanja and Dehghani, Mostafa and Liu, Fangyu and Mittal, Sid and Oh, Junhyuk and Noury, Seb and Sezener, Eren and Huot, Fantine and Lamm, Matthew and Cao, Nicola De and Chen, Charlie and Mudgal, Sidharth and Stella, Romina and Brooks, Kevin and Vasudevan, Gautam and Liu, Chenxi and Chain, Mainak and Melinkeri, Nivedita and Cohen, Aaron and Wang, Venus and Seymore, Kristie and Zubkov, Sergey and Goel, Rahul and Yue, Summer and Krishnakumaran, Sai and Albert, Brian and Hurley, Nate and Sano, Motoki and Mohananey, Anhad and Joughin, Jonah and Filonov, Egor and Kępa, Tomasz and Eldawy, Yomna and Lim, Jiawern and Rishi, Rahul and Badiezadegan, Shirin and Bos, Taylor and Chang, Jerry and Jain, Sanil and Padmanabhan, Sri Gayatri Sundara and Puttagunta, Subha and Krishna, Kalpesh and Baker, Leslie and Kalb, Norbert and Bedapudi, Vamsi and Kurzrok, Adam and Lei, Shuntong and Yu, Anthony and Litvin, Oren and Zhou, Xiang and Wu, Zhichun and Sobell, Sam and Siciliano, Andrea and Papir, Alan and Neale, Robby and Bragagnolo, Jonas and Toor, Tej and Chen, Tina and Anklin, Valentin and Wang, Feiran and Feng, Richie and Gholami, Milad and Ling, Kevin and Liu, Lijuan and Walter, Jules and Moghaddam, Hamid and Kishore, Arun and Adamek, Jakub and Mercado, Tyler and Mallinson, Jonathan and Wandekar, Siddhinita and Cagle, Stephen and Ofek, Eran and Garrido, Guillermo and Lombriser, Clemens and Mukha, Maksim and Sun, Botu and Mohammad, Hafeezul Rahman and Matak, Josip and Qian, Yadi and Peswani, Vikas and Janus, Pawel and Yuan, Quan and Schelin, Leif and David, Oana and Garg, Ankur and He, Yifan and Duzhyi, Oleksii and Älgmyr, Anton and Lottaz, Timothée and Li, Qi and Yadav, Vikas and Xu, Luyao and Chinien, Alex and Shivanna, Rakesh and Chuklin, Aleksandr and Li, Josie and Spadine, Carrie and Wolfe, Travis and Mohamed, Kareem and Das, Subhabrata and Dai, Zihang and He, Kyle and Dincklage, Daniel von and Upadhyay, Shyam and Maurya, Akanksha and Chi, Luyan and Krause, Sebastian and Salama, Khalid and Rabinovitch, Pam G and M, Pavan Kumar Reddy and Selvan, Aarush and Dektiarev, Mikhail and Ghiasi, Golnaz and Guven, Erdem and Gupta, Himanshu and Liu, Boyi and Sharma, Deepak and Shtacher, Idan Heimlich and Paul, Shachi and Akerlund, Oscar and Aubet, François-Xavier and Huang, Terry and Zhu, Chen and Zhu, Eric and Teixeira, Elico and Fritze, Matthew and Bertolini, Francesco and Marinescu, Liana-Eleonora and Bölle, Martin and Paulus, Dominik and Gupta, Khyatti and Latkar, Tejasi and Chang, Max and Sanders, Jason and Wilson, Roopa and Wu, Xuewei and Tan, Yi-Xuan and Thiet, Lam Nguyen and Doshi, Tulsee and Lall, Sid and Mishra, Swaroop and Chen, Wanming and Luong, Thang and Benjamin, Seth and Lee, Jasmine and Andrejczuk, Ewa and Rabiej, Dominik and Ranjan, Vipul and Styrc, Krzysztof and Yin, Pengcheng and Simon, Jon and Harriott, Malcolm Rose and Bansal, Mudit and Robsky, Alexei and Bacon, Geoff and Greene, David and Mirylenka, Daniil and Zhou, Chen and Sarvana, Obaid and Goyal, Abhimanyu and Andermatt, Samuel and Siegler, Patrick and Horn, Ben and Israel, Assaf and Pongetti, Francesco and Chen, Chih-Wei "Louis" and Selvatici, Marco and Silva, Pedro and Wang, Kathie and Tolins, Jackson and Guu, Kelvin and Yogev, Roey and Cai, Xiaochen and Agostini, Alessandro and Shah, Maulik and Nguyen, Hung and Donnaile, Noah Ó and Pereira, Sébastien and Friso, Linda and Stambler, Adam and Kurzrok, Adam and Kuang, Chenkai and Romanikhin, Yan and Geller, Mark and Yan, ZJ and Jang, Kane and Lee, Cheng-Chun and Fica, Wojciech and Malmi, Eric and Tan, Qijun and Banica, Dan and Balle, Daniel and Pham, Ryan and Huang, Yanping and Avram, Diana and Shi, Hongzhi and Singh, Jasjot and Hidey, Chris and Ahuja, Niharika and Saxena, Pranab and Dooley, Dan and Potharaju, Srividya Pranavi and O'Neill, Eileen and Gokulchandran, Anand and Foley, Ryan and Zhao, Kai and Dusenberry, Mike and Liu, Yuan and Mehta, Pulkit and Kotikalapudi, Ragha and Safranek-Shrader, Chalence and Goodman, Andrew and Kessinger, Joshua and Globen, Eran and Kolhar, Prateek and Gorgolewski, Chris and Ibrahim, Ali and Song, Yang and Eichenbaum, Ali and Brovelli, Thomas and Potluri, Sahitya and Lahoti, Preethi and Baetu, Cip and Ghorbani, Ali and Chen, Charles and Crawford, Andy and Pal, Shalini and Sridhar, Mukund and Gurita, Petru and Mujika, Asier and Petrovski, Igor and Cedoz, Pierre-Louis and Li, Chenmei and Chen, Shiyuan and Santo, Niccolò Dal and Goyal, Siddharth and Punjabi, Jitesh and Kappaganthu, Karthik and Kwak, Chester and LV, Pallavi and Velury, Sarmishta and Choudhury, Himadri and Hall, Jamie and Shah, Premal and Figueira, Ricardo and Thomas, Matt and Lu, Minjie and Zhou, Ting and Kumar, Chintu and Jurdi, Thomas and Chikkerur, Sharat and Ma, Yenai and Yu, Adams and Kwak, Soo and Ähdel, Victor and Rajayogam, Sujeevan and Choma, Travis and Liu, Fei and Barua, Aditya and Ji, Colin and Park, Ji Ho and Hellendoorn, Vincent and Bailey, Alex and Bilal, Taylan and Zhou, Huanjie and Khatir, Mehrdad and Sutton, Charles and Rzadkowski, Wojciech and Macintosh, Fiona and Vij, Roopali and Shagin, Konstantin and Medina, Paul and Liang, Chen and Zhou, Jinjing and Shah, Pararth and Bi, Yingying and Dankovics, Attila and Banga, Shipra and Lehmann, Sabine and Bredesen, Marissa and Lin, Zifan and Hoffmann, John Eric and Lai, Jonathan and Chung, Raynald and Yang, Kai and Balani, Nihal and Bražinskas, Arthur and Sozanschi, Andrei and Hayes, Matthew and Alcalde, Héctor Fernández and Makarov, Peter and Chen, Will and Stella, Antonio and Snijders, Liselotte and Mandl, Michael and Kärrman, Ante and Nowak, Paweł and Wu, Xinyi and Dyck, Alex and Vaidyanathan, Krishnan and R, Raghavender and Mallet, Jessica and Rudominer, Mitch and Johnston, Eric and Mittal, Sushil and Udathu, Akhil and Christensen, Janara and Verma, Vishal and Irving, Zach and Santucci, Andreas and Elsayed, Gamaleldin and Davoodi, Elnaz and Georgiev, Marin and Tenney, Ian and Hua, Nan and Cideron, Geoffrey and Leurent, Edouard and Alnahlawi, Mahmoud and Georgescu, Ionut and Wei, Nan and Zheng, Ivy and Scandinaro, Dylan and Jiang, Heinrich and Snoek, Jasper and Sundararajan, Mukund and Wang, Xuezhi and Ontiveros, Zack and Karo, Itay and Cole, Jeremy and Rajashekhar, Vinu and Tumeh, Lara and Ben-David, Eyal and Jain, Rishub and Uesato, Jonathan and Datta, Romina and Bunyan, Oskar and Wu, Shimu and Zhang, John and Stanczyk, Piotr and Zhang, Ye and Steiner, David and Naskar, Subhajit and Azzam, Michael and Johnson, Matthew and Paszke, Adam and Chiu, Chung-Cheng and Elias, Jaume Sanchez and Mohiuddin, Afroz and Muhammad, Faizan and Miao, Jin and Lee, Andrew and Vieillard, Nino and Park, Jane and Zhang, Jiageng and Stanway, Jeff and Garmon, Drew and Karmarkar, Abhijit and Dong, Zhe and Lee, Jong and Kumar, Aviral and Zhou, Luowei and Evens, Jonathan and Isaac, William and Irving, Geoffrey and Loper, Edward and Fink, Michael and Arkatkar, Isha and Chen, Nanxin and Shafran, Izhak and Petrychenko, Ivan and Chen, Zhe and Jia, Johnson and Levskaya, Anselm and Zhu, Zhenkai and Grabowski, Peter and Mao, Yu and Magni, Alberto and Yao, Kaisheng and Snaider, Javier and Casagrande, Norman and Palmer, Evan and Suganthan, Paul and Castaño, Alfonso and Giannoumis, Irene and Kim, Wooyeol and Rybiński, Mikołaj and Sreevatsa, Ashwin and Prendki, Jennifer and Soergel, David and Goedeckemeyer, Adrian and Gierke, Willi and Jafari, Mohsen and Gaba, Meenu and Wiesner, Jeremy and Wright, Diana Gage and Wei, Yawen and Vashisht, Harsha and Kulizhskaya, Yana and Hoover, Jay and Le, Maigo and Li, Lu and Iwuanyanwu, Chimezie and Liu, Lu and Ramirez, Kevin and Khorlin, Andrey and Cui, Albert and LIN, Tian and Wu, Marcus and Aguilar, Ricardo and Pallo, Keith and Chakladar, Abhishek and Perng, Ginger and Abellan, Elena Allica and Zhang, Mingyang and Dasgupta, Ishita and Kushman, Nate and Penchev, Ivo and Repina, Alena and Wu, Xihui and Weide, Tom van der and Ponnapalli, Priya and Kaplan, Caroline and Simsa, Jiri and Li, Shuangfeng and Dousse, Olivier and Yang, Fan and Piper, Jeff and Ie, Nathan and Pasumarthi, Rama and Lintz, Nathan and Vijayakumar, Anitha and Andor, Daniel and Valenzuela, Pedro and Lui, Minnie and Paduraru, Cosmin and Peng, Daiyi and Lee, Katherine and Zhang, Shuyuan and Greene, Somer and Nguyen, Duc Dung and Kurylowicz, Paula and Hardin, Cassidy and Dixon, Lucas and Janzer, Lili and Choo, Kiam and Feng, Ziqiang and Zhang, Biao and Singhal, Achintya and Du, Dayou and McKinnon, Dan and Antropova, Natasha and Bolukbasi, Tolga and Keller, Orgad and Reid, David and Finchelstein, Daniel and Raad, Maria Abi and Crocker, Remi and Hawkins, Peter and Dadashi, Robert and Gaffney, Colin and Franko, Ken and Bulanova, Anna and Leblond, Rémi and Chung, Shirley and Askham, Harry and Cobo, Luis C. and Xu, Kelvin and Fischer, Felix and Xu, Jun and Sorokin, Christina and Alberti, Chris and Lin, Chu-Cheng and Evans, Colin and Dimitriev, Alek and Forbes, Hannah and Banarse, Dylan and Tung, Zora and Omernick, Mark and Bishop, Colton and Sterneck, Rachel and Jain, Rohan and Xia, Jiawei and Amid, Ehsan and Piccinno, Francesco and Wang, Xingyu and Banzal, Praseem and Mankowitz, Daniel J. and Polozov, Alex and Krakovna, Victoria and Brown, Sasha and Bateni, MohammadHossein and Duan, Dennis and Firoiu, Vlad and Thotakuri, Meghana and Natan, Tom and Geist, Matthieu and Girgin, Ser tan and Li, Hui and Ye, Jiayu and Roval, Ofir and Tojo, Reiko and Kwong, Michael and Lee-Thorp, James and Yew, Christopher and Sinopalnikov, Danila and Ramos, Sabela and Mellor, John and Sharma, Abhishek and Wu, Kathy and Miller, David and Sonnerat, Nicolas and Vnukov, Denis and Greig, Rory and Beattie, Jennifer and Caveness, Emily and Bai, Libin and Eisenschlos, Julian and Korchemniy, Alex and Tsai, Tomy and Jasarevic, Mimi and Kong, Weize and Dao, Phuong and Zheng, Zeyu and Liu, Frederick and Yang, Fan and Zhu, Rui and Teh, Tian Huey and Sanmiya, Jason and Gladchenko, Evgeny and Trdin, Nejc and Toyama, Daniel and Rosen, Evan and Tavakkol, Sasan and Xue, Linting and Elkind, Chen and Woodman, Oliver and Carpenter, John and Papamakarios, George and Kemp, Rupert and Kafle, Sushant and Grunina, Tanya and Sinha, Rishika and Talbert, Alice and Wu, Diane and Owusu-Afriyie, Denese and Du, Cosmo and Thornton, Chloe and Pont-Tuset, Jordi and Narayana, Pradyumna and Li, Jing and Fatehi, Saaber and Wieting, John and Ajmeri, Omar and Uria, Benigno and Ko, Yeongil and Knight, Laura and Héliou, Amélie and Niu, Ning and Gu, Shane and Pang, Chenxi and Li, Yeqing and Levine, Nir and Stolovich, Ariel and Santamaria-Fernandez, Rebeca and Goenka, Sonam and Yustalim, Wenny and Strudel, Robin and Elqursh, Ali and Deck, Charlie and Lee, Hyo and Li, Zonglin and Levin, Kyle and Hoffmann, Raphael and Holtmann-Rice, Dan and Bachem, Olivier and Arora, Sho and Koh, Christy and Yeganeh, Soheil Hassas and Põder, Siim and Tariq, Mukarram and Sun, Yanhua and Ionita, Lucian and Seyedhosseini, Mojtaba and Tafti, Pouya and Liu, Zhiyu and Gulati, Anmol and Liu, Jasmine and Ye, Xinyu and Chrzaszcz, Bart and Wang, Lily and Sethi, Nikhil and Li, Tianrun and Brown, Ben and Singh, Shreya and Fan, Wei and Parisi, Aaron and Stanton, Joe and Koverkathu, Vinod and Choquette-Choo, Christopher A. and Li, Yunjie and Lu, TJ and Ittycheriah, Abe and Shroff, Prakash and Varadarajan, Mani and Bahargam, Sanaz and Willoughby, Rob and Gaddy, David and Desjardins, Guillaume and Cornero, Marco and Robenek, Brona and Mittal, Bhavishya and Albrecht, Ben and Shenoy, Ashish and Moiseev, Fedor and Jacobsson, Henrik and Ghaffarkhah, Alireza and Rivière, Morgane and Walton, Alanna and Crepy, Clément and Parrish, Alicia and Zhou, Zongwei and Farabet, Clement and Radebaugh, Carey and Srinivasan, Praveen and Salm, Claudia van der and Fidjeland, Andreas and Scellato, Salvatore and Latorre-Chimoto, Eri and Klimczak-Plucińska, Hanna and Bridson, David and Cesare, Dario de and Hudson, Tom and Mendolicchio, Piermaria and Walker, Lexi and Morris, Alex and Mauger, Matthew and Guseynov, Alexey and Reid, Alison and Odoom, Seth and Loher, Lucia and Cotruta, Victor and Yenugula, Madhavi and Grewe, Dominik and Petrushkina, Anastasia and Duerig, Tom and Sanchez, Antonio and Yadlowsky, Steve and Shen, Amy and Globerson, Amir and Webb, Lynette and Dua, Sahil and Li, Dong and Bhupatiraju, Surya and Hurt, Dan and Qureshi, Haroon and Agarwal, Ananth and Shani, Tomer and Eyal, Matan and Khare, Anuj and Belle, Shreyas Rammohan and Wang, Lei and Tekur, Chetan and Kale, Mihir Sanjay and Wei, Jinliang and Sang, Ruoxin and Saeta, Brennan and Liechty, Tyler and Sun, Yi and Zhao, Yao and Lee, Stephan and Nayak, Pandu and Fritz, Doug and Vuyyuru, Manish Reddy and Aslanides, John and Vyas, Nidhi and Wicke, Martin and Ma, Xiao and Eltyshev, Evgenii and Martin, Nina and Cate, Hardie and Manyika, James and Amiri, Keyvan and Kim, Yelin and Xiong, Xi and Kang, Kai and Luisier, Florian and Tripuraneni, Nilesh and Madras, David and Guo, Mandy and Waters, Austin and Wang, Oliver and Ainslie, Joshua and Baldridge, Jason and Zhang, Han and Pruthi, Garima and Bauer, Jakob and Yang, Feng and Mansour, Riham and Gelman, Jason and Xu, Yang and Polovets, George and Liu, Ji and Cai, Honglong and Chen, Warren and Sheng, XiangHai and Xue, Emily and Ozair, Sherjil and Angermueller, Christof and Li, Xiaowei and Sinha, Anoop and Wang, Weiren and Wiesinger, Julia and Koukoumidis, Emmanouil and Tian, Yuan and Iyer, Anand and Gurumurthy, Madhu and Goldenson, Mark and Shah, Parashar and Blake, MK and Yu, Hongkun and Urbanowicz, Anthony and Palomaki, Jennimaria and Fernando, Chrisantha and Durden, Ken and Mehta, Harsh and Momchev, Nikola and Rahimtoroghi, Elahe and Georgaki, Maria and Raul, Amit and Ruder, Sebastian and Redshaw, Morgan and Lee, Jinhyuk and Zhou, Denny and Jalan, Komal and Li, Dinghua and Hechtman, Blake and Schuh, Parker and Nasr, Milad and Milan, Kieran and Mikulik, Vladimir and Franco, Juliana and Green, Tim and Nguyen, Nam and Kelley, Joe and Mahendru, Aroma and Hu, Andrea and Howland, Joshua and Vargas, Ben and Hui, Jeffrey and Bansal, Kshitij and Rao, Vikram and Ghiya, Rakesh and Wang, Emma and Ye, Ke and Sarr, Jean Michel and Preston, Melanie Moranski and Elish, Madeleine and Li, Steve and Kaku, Aakash and Gupta, Jigar and Pasupat, Ice and Juan, Da-Cheng and Someswar, Milan and M and Chen, Xinyun and Amini, Aida and Fabrikant, Alex and Chu, Eric and Dong, Xuanyi and Muthal, Amruta and Buthpitiya, Senaka and Jauhari, Sarthak and Hua, Nan and Khandelwal, Urvashi and Hitron, Ayal and Ren, Jie and Rinaldi, Larissa and Drath, Shahar and Dabush, Avigail and Jiang, Nan-Jiang and Godhia, Harshal and Sachs, Uli and Chen, Anthony and Fan, Yicheng and Taitelbaum, Hagai and Noga, Hila and Dai, Zhuyun and Wang, James and Liang, Chen and Hamer, Jenny and Ferng, Chun-Sung and Elkind, Chenel and Atias, Aviel and Lee, Paulina and Listík, Vít and Carlen, Mathias and Kerkhof, Jan van de and Pikus, Marcin and Zaher, Krunoslav and Müller, Paul and Zykova, Sasha and Stefanec, Richard and Gatsko, Vitaly and Hirnschall, Christoph and Sethi, Ashwin and Xu, Xingyu Federico and Ahuja, Chetan and Tsai, Beth and Stefanoiu, Anca and Feng, Bo and Dhandhania, Keshav and Katyal, Manish and Gupta, Akshay and Parulekar, Atharva and Pitta, Divya and Zhao, Jing and Bhatia, Vivaan and Bhavnani, Yashodha and Alhadlaq, Omar and Li, Xiaolin and Danenberg, Peter and Tu, Dennis and Pine, Alex and Filippova, Vera and Ghosh, Abhipso and Limonchik, Ben and Urala, Bhargava and Lanka, Chaitanya Krishna and Clive, Derik and Sun, Yi and Li, Edward and Wu, Hao and Hongtongsak, Kevin and Li, Ianna and Thakkar, Kalind and Omarov, Kuanysh and Majmundar, Kushal and Alverson, Michael and Kucharski, Michael and Patel, Mohak and Jain, Mudit and Zabelin, Maksim and Pelagatti, Paolo and Kohli, Rohan and Kumar, Saurabh and Kim, Joseph and Sankar, Swetha and Shah, Vineet and Ramachandruni, Lakshmi and Zeng, Xiangkai and Bariach, Ben and Weidinger, Laura and Vu, Tu and Andreev, Alek and He, Antoine and Hui, Kevin and Kashem, Sheleem and Subramanya, Amar and Hsiao, Sissie and Hassabis, Demis and Kavukcuoglu, Koray and Sadovsky, Adam and Le, Quoc and Strohman, Trevor and Wu, Yonghui and Petrov, Slav and Dean, Jeffrey and Vinyals, Oriol},
  year = {2023},
  doi = {10.48550/arxiv.2312.11805},
  eprint = {2312.11805},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.11805}
}

References

136 references, 56 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… find our card
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… find our card
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv.org/abs/2108.07732 . ↗ our card
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… find our card
  6. Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Dan Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, Brennan Saeta, Parker Schuh, Ryan Sepassi, Laurent El Shafey, Chandramohan A. Thekkath, and Yonghui Wu. Pathways: Asynchronous distributed dataflow for ML. Proc… find
  7. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe… find
  8. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/ google/jax . ↗
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  10. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper\_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf . ↗
  11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
  12. Xi Chen, Xiao Wang, Soravit Changpinyo, A J Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo,… find our card
  13. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paul… — [arXiv:2305.18565](https://arxiv.org/abs/2305.18565) find
  14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  15. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
  16. Jon Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TydiQA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 2020. URL https://storag… find
  17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 . ↗ our card
  18. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. find our card
  19. Jeff Dean. tecture, Introducing Pathways: A next-generation AI archi-URL https://blog.google/technology/ai/ introducing-pathways-next-generation-ai-architecture/ . 2021. ↗
  20. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012. find
  21. Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. Silent data corruptions at scale. arXiv preprint arXiv:2102.11245, 2021. — [arXiv:2102.11245](https://arxiv.org/abs/2102.11245) find
  22. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. find
  23. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find our card
  24. Christian Federmann, Tom Kocmi, and Ying Xin. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21–24, Online, nov 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.s… ↗ our card
  25. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models, 2020. find our card
  26. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fr… find our card
  27. Google. Google’s AI Principles. principles/ . 2023. find
  28. URL https://ai.google/responsibility/Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages… ↗ our card
  29. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 46… find
  30. Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, et al. Can large language models understand real-world complex instructions? arXiv preprint arXiv:2309.09150, 2023. — [arXiv:2309.09150](https://arxiv.org/abs/2309.09150) find
  31. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. find our card
  32. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021b. URL https://arxiv.org/abs/2103.03874 . ↗ our card
  33. Peter H Hochschild, Paul Turner, Jeffrey C Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E Culler, and Amin Vahdat. Cores that don’t count. In Proceedings of the Workshop on Hot Topics in Operating Systems, pages 9–16, 2021. find
  34. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… find
  35. Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. Won’t get fooled again: Answering questions with false premises. arXiv preprint arXiv:2307.02394, 2023. find
  36. EunJeong Hwang and Vered Shwartz. Memecap: A dataset for captioning and interpreting memes, 2023. find
  37. Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020. doi: 10.1145/3360307. URL https://doi.org/10.1145/ 3360307 . — [doi:10.1145/3360307](https://doi.org/10.1145/3360307) find
  38. Norman P Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support fo… find
  39. Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI, 2021. find
  40. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What’s the answer right now? arXiv preprint arXiv:2207.13332, 2022a. find our card
  41. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. RealTime QA: What’s the answer right now?, 2022b. URL https://arxiv.org/abs/2207.13332 . ↗ our card
  42. K Kavukcuoglu, P Kohli, L Ibrahim, D Bloxwich, and S Brown. How our principles helped define AlphaFold’s release. Google DeepMind, 2022. find
  43. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. find our card
  44. Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, et al. Evaluating language-model agents on realistic autonomous tasks. arXiv preprint arXiv:2312.11671, 2023. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) find
  45. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl\_a\_00023. URL https://aclanthology.o… ↗ our card
  46. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović.… find
  47. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199–22213, 2022. find our card
  48. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstratio… find our card
  49. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… find our card
  50. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:… find our card
  51. Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048, Online, November 2020. Association for Computational Linguistic… find our card
  52. Leblond et al. AlphaCode 2 Technical Report. 2023. URL https://storage.googleapis.com/ deepmind-media/AlphaCode2/AlphaCode2\_Tech\_Report.pdf . ↗
  53. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015. find
  54. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022. find our card
  55. Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. find our card
  56. Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. DePlot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2023, p… find
  57. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th… find our card
  58. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. find our card
  59. Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, 2022. find our card
  60. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. find our card
  61. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. find
  62. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. find
  63. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. find our card
  64. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, October-November 201… find
  65. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021. find
  66. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium, October-… find
  67. Középszintű Írásbéli Vizsga, May 2023. find
  68. URL https://dload-oktatas.educatio.hu/erettsegi/feladatok\_2023tavasz\_kozep/ k\_matang\_23maj\_fl.pdf . Angol Nyelven. OpenAI. GPT-4 Technical Report. 2023a. ↗ our card
  69. OpenAI. GPT-4V(ision) System Card, 2023b. find
  70. OpenAI. Whisper, 2023. URL https://github.com/openai/whisper . ↗
  71. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training l… find
  72. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. find our card
  73. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. CoRR, abs/2110.08193, 2021. URL https://arxiv.org/abs/2110.08193 . ↗ our card
  74. Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stepha… — [arXiv:2305.13786](https://arxiv.org/abs/2305.13786) find
  75. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023. find our card
  76. Leon Poutievski, Omid Mashayekhi, Joon Ong, Arjun Singh, Mukarram Tariq, Rui Wang, Jianan Zhang, Virginia Beauregard, Patrick Conner, Steve Gribble, et al. Jupiter evolving: transforming google’s datacenter network via optical circuit switches and software-defined networking. In Proceedings of the A… find
  77. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020. find our card
  78. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language\_ models\_are\_unsupervised\_multitask\_learners.pdf . ↗
  79. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR, 2023. find our card
  80. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… find
  81. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021. find our card
  82. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, pages 1–64, 2023. find our card
  83. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, a… find
  84. Parker Riley, Timothy Dozat, Jan A Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. Frmt: A benchmark for few-shot region-aware machine translation. Transactions of the Association for Computational Linguistics, 2023. find
  85. Hannah Ritchie, Veronika Samborska, and Max Roser. Plastic pollution. Our World in Data, 2023. https://ourworldindata.org/plastic-pollution. ↗
  86. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online, November 2020. Association for Computational Linguistics.… find
  87. William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Be… find
  88. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New… find our card
  89. Candice Schumann, Susanna Ricco, Utsav Prabhu, Vittorio Ferrari, and Caroline Pantofaru. A step toward more inclusive people annotations for fairness. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 916–925, 2021. find
  90. Andrew D. Selbst, Danah Boyd, and Sorelle A. Friedler. Fairness and abstraction in sociotechnical systems. In FFAT* ’19: Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 59–68, January 2019. find
  91. Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-… find
  92. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12… find our card
  93. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150, 2019a. arXiv preprint Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150, 2019b. arXiv preprint Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzad… find
  94. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Al… find
  95. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. ICLR, 2023. find our card
  96. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. find
  97. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. URL https://arxiv.org/abs/ 2206.04615 . ↗ our card
  98. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014. find
  99. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  100. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proof Writer: Generating implications, proofs, and abductive statements over natural language. In Findings, 2020. URL https://api. semanticscholar.org/CorpusID:229371222 . find
  101. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  102. Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In EMNLP, 2022. find
  103. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239 . ↗ our card
  104. Kocmi Tom, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. Findings of the 2023 conference on machine translation (wmt23): Llms are here but not quite there yet. In WMT23-Eighth Conferenc… find
  105. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint… find
  106. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  107. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing… find
  108. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017b. URL http://arxiv.org/abs/1706.03762 . ↗
  109. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. find our card
  110. Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark. arXiv preprint arXiv:2205.15659, 2022. find
  111. Manoj Vishwanathan, Ronak Shah, Kyung Ki Kim, and Minsu Choi. Silent data corruption (sdc) vulnerability of gpu on various gpgpu workloads. In 2015 International SoC Design Conference (ISOCC), pages 11–12, 2015. doi: 10.1109/ISOCC.2015.7401681. — [doi:10.1109/isocc.2015.7401681](https://doi.org/10.1109/isocc.2015.7401681) find
  112. Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310, 2020. find
  113. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021. find our card
  114. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VATEX: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019. find
  115. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. find our card
  116. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. Proceedings of the International Conference on Learning Representations (ICLR), 2022a. URL https://openreview. net/forum?id=gEZrGCo… find
  117. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022b. URL https://arxiv.org/abs/2201.11903 . ↗ our card
  118. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sea… find our card
  119. David Wetherall, Abdul Kabbani, Van Jacobson, Jim Winget, Yuchung Cheng, Brad Morrey, Uma Parthavi Moravapalle, Phillipa Gill, Steven Knight, and Amin Vahdat. Improving network availability with protective reroute. In SIGCOMM 2023, 2023. URL https://dl.acm.org/doi/ 10.1145/3603269.3604867 . — [doi:10.1145/3603269.3604867](https://doi.org/10.1145/3603269.3604867) ↗
  120. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NExT-QA: Next phase of question-answering to explaining temporal actions. In CVPR, 2021. find
  121. XLA. XLA: Optimizing compiler for TensorFlow. https://www.tensorflow.org/xla , 2019. [Online; accessed December-2023]. ↗
  122. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. find our card
  123. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint arXiv:2105.04663, 2021. — [arXiv:2105.04663](https://arxiv.org/abs/2105.04663) find
  124. Chi yao Hong, Subhasree Mandal, Mohammad A. Alfares, Min Zhu, Rich Alimi, Kondapa Naidu Bollineni, Chandan Bhagat, Sourabh Jain, Jay Kaimal, Jeffrey Liang, Kirill Mendelev, Steve Padgett, Faro Thomas Rabe, Saikat Ray, Malveeka Tewari, Matt Tierney, Monika Zahn, Jon Zolla, Joon Ong, and Amin Vahdat.… find
  125. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022a. find our card
  126. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generat… — [arXiv:2206.10789](https://arxiv.org/abs/2206.10789) find
  127. Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988, 2023. — [arXiv:2305.06988](https://arxiv.org/abs/2305.06988) find
  128. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019. find our card
  129. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… find our card
  130. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  131. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath,… — [arXiv:2303.01037](https://arxiv.org/abs/2303.01037) find
  132. Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14830–14840, 2021. find
  133. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018. find our card
  134. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models, 2023. find
  135. Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, pages 7590–7598, 2018. \#\# 9. Contributions and Acknowledgments Gemini Leads Rohan Anil, Co-Lead, Text Sebastian Borgeaud, Co-Lead, Text… find
  136. Overall Gemini Post-Training Lead Slav Petrov Overall Gemini Technical Leads (equal contribution) Jeffrey Dean Oriol Vinyals The roles are defined as below: • Lead: Individual(s) responsible for the sub-team throughout the project. • Core Contributor: Individual that had significant impact throughou… find