Elamur Research
☰
home / papers / 2025 / Humanity's Last Exam

Humanity's Last Exam

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2501.14249

Summary

Popular benchmarks such as MMLU are now close to saturated, so they no longer tell us much about how capable frontier LLMs have become. To fix this, the authors built Humanity's Last Exam (HLE), a multimodal benchmark of very hard, closed-ended academic questions. Each question has a single verifiable answer. Close to a thousand subject experts from many countries and institutions wrote questions across more than a hundred disciplines, with mathematics making up the largest share. Before a question was accepted, it was tested against leading models and dropped if they could solve it. Surviving questions then went through two rounds of expert review: the first improved submissions, and the second approved the best ones for the final set. Evaluation used a standard reasoning-then-answer prompt, with o3-mini judging whether answers were correct. Every model tested scored low and was badly calibrated, often giving wrong answers with high confidence. HLE therefore gives researchers and policymakers a shared, hard-to-game measure of expert-level knowledge and reasoning. The authors note that it does not measure open-ended research ability.

Key points

  • The final public set has about 2,500 questions, a mix of exact-match and multiple-choice. Some include images. A private held-out portion is kept to detect overfitting.
  • Accuracy (%) on HLE: GPT-4o 2.7, Grok 2 3.0, Claude 3.5 Sonnet 4.1, Gemini 1.5 Pro 4.6, Gemini 2.0 Flash Thinking 6.6, o1 8.0. DeepSeek-R1 scored 8.5 and o3-mini (high) 13.4, both on the text-only subset.
  • RMS calibration error ranged from about 73% to 89%. This means the models' stated confidence was far above their real accuracy, a sign of hallucination.
  • Reasoning models produced many more completion tokens than non-reasoning models for their modest gains. The authors call for compute-efficient progress, not only higher accuracy.
  • The authors thought scores above 50% by the end of 2025 were plausible. Even so, they argue that a high HLE score would not show AGI or autonomous research ability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

186 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 176 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41586-025-09962-4 · arXiv 2501.14249

@misc{phan2025humanity,
  title = {{Humanity's Last Exam}},
  author = {Phan, Long and Gatti, Alice and Han, Ziwen and Li, Nathaniel and Hu, Josephina and Zhang, Hugh and Zhang, Chen Bo Calvin and Shaaban, Mohamed and Ling, John and Shi, Sean and Choi, Michael and Agrawal, Anish and Chopra, Arnav and Khoja, Adam and Kim, Ryan and Ren, Richard and Hausenloy, Jason and Zhang, Oliver and Mazeika, Mantas and Dodonov, Dmitry and Nguyen, Tung and Lee, Jaeho and Anderson, Daron and Doroshenko, Mikhail and Stokes, Alun Cennyth and Mahmood, Mobeen and Pokutnyi, Oleksandr and Iskra, Oleg and Wang, Jessica P. and Levin, John-Clark and Kazakov, Mstyslav and Feng, Fiona and Feng, Steven Y. and Zhao, Haoran and Yu, Michael and Gangal, Varun and Zou, Chelsea and Wang, Zihan and Popov, Serguei and Gerbicz, Robert and Galgon, Geoff and Schmitt, Johannes and Yeadon, Will and Lee, Yongki and Sauers, Scott and Sanchez, Alvaro and Giska, Fabian and Roth, Marc and Riis, Søren and Utpala, Saiteja and Burns, Noah and Goshu, Gashaw M. and Naiya, Mohinder Maheshbhai and Agu, Chidozie and Giboney, Zachary and Cheatom, Antrell and Fournier-Facio, Francesco and Crowson, Sarah-Jane and Finke, Lennart and Cheng, Zerui and Zampese, Jennifer and Hoerr, Ryan G. and Nandor, Mark and Park, Hyunwoo and Gehrunger, Tim and Cai, Jiaqi and McCarty, Ben and Garretson, Alexis C and Taylor, Edwin and Sileo, Damien and Ren, Qiuyu and Qazi, Usman and Li, Lianghui and Nam, Jungbae and Wydallis, John B. and Arkhipov, Pavel and Shi, Jack Wei Lun and Bacho, Aras and Willcocks, Chris G. and Cao, Hangrui and Motwani, Sumeet and Santos, Emily de Oliveira and Veith, Johannes and Vendrow, Edward and Cojoc, Doru and Zenitani, Kengo and Robinson, Joshua and Tang, Longke and Li, Yuqi and Vendrow, Joshua and Fraga, Natanael Wildner and Kuchkin, Vladyslav and Maksimov, Andrey Pupasov and Marion, Pierre and Efremov, Denis and Lynch, Jayson and Liang, Kaiqu and Mikov, Aleksandar and Gritsevskiy, Andrew and Guillod, Julien and Demir, Gözdenur and Martinez, Dakotah and Pageler, Ben and Zhou, Kevin and Soori, Saeed and Press, Ori and Tang, Henry and Rissone, Paolo and Green, Sean R. and Brüssel, Lina and Twayana, Moon and Dieuleveut, Aymeric and Imperial, Joseph Marvin and Prabhu, Ameya and Yang, Jinzhou and Crispino, Nick and Rao, Arun and Zvonkine, Dimitri and Loiseau, Gabriel and Kalinin, Mikhail and Lukas, Marco and Manolescu, Ciprian and Stambaugh, Nate and Mishra, Subrata and Hogg, Tad and Bosio, Carlo and Coppola, Brian P and Salazar, Julian and Jin, Jaehyeok and Sayous, Rafael and Ivanov, Stefan and Schwaller, Philippe and Senthilkuma, Shaipranesh and Bran, Andres M and Algaba, Andres and Houte, Kelsey Van den and Sypt, Lynn Van Der and Verbeken, Brecht and Noever, David and Kopylov, Alexei and Myklebust, Benjamin and Li, Bikun and Schut, Lisa and Zheltonozhskii, Evgenii and Yuan, Qiaochu and Lim, Derek and Stanley, Richard and Yang, Tong and Maar, John and Wykowski, Julian and Oller, Martí and Sahu, Anmol and Ardito, Cesare Giulio and Hu, Yuzheng and Kamdoum, Ariel Ghislain Kemogne and Jin, Alvin and Vilchis, Tobias Garcia and Zu, Yuexuan and Lackner, Martin and Koppel, James and Sun, Gongbo and Antonenko, Daniil S. and Chern, Steffi and Zhao, Bingchen and Arsene, Pierrot and Cavanagh, Joseph M and Li, Daofeng and Shen, Jiawei and Crisostomi, Donato and Zhang, Wenjin and Dehghan, Ali and Ivanov, Sergey and Perrella, David and Kaparov, Nurdin and Zang, Allen and Sucholutsky, Ilia and Kharlamova, Arina and Orel, Daniil and Poritski, Vladislav and Ben-David, Shalev and Berger, Zachary and Whitfill, Parker and Foster, Michael and Munro, Daniel and Ho, Linh and Sivarajan, Shankar and Hava, Dan Bar and Kuchkin, Aleksey and Holmes, David and Rodriguez-Romero, Alexandra and Sommerhage, Frank and Zhang, Anji and Moat, Richard and Schneider, Keith and Kazibwe, Zakayo and Clarke, Don and Kim, Dae Hyun and Dias, Felipe Meneguitti and Fish, Sara and Elser, Veit and Kreiman, Tobias and Vilchis, Victor Efren Guadarrama and Klose, Immo and Anantheswaran, Ujjwala and Zweiger, Adam and Rawal, Kaivalya and Li, Jeffery and Nguyen, Jeremy and Daans, Nicolas and Heidinger, Haline and Radionov, Maksim and Rozhoň, Václav and Ginis, Vincent and Stump, Christian and Cohen, Niv and Poświata, Rafał and Tkadlec, Josef and Goldfarb, Alan and Wang, Chenguang and Padlewski, Piotr and Barzowski, Stanislaw and Montgomery, Kyle and Stendall, Ryan and Tucker-Foltz, Jamie and Stade, Jack and Rogers, T. Ryan and Goertzen, Tom and Grabb, Declan and Shukla, Abhishek and Givré, Alan and Ambay, John Arnold and Sen, Archan and Aziz, Muhammad Fayez and Inlow, Mark H and He, Hao and Zhang, Ling and Kaddar, Younesse and Ängquist, Ivar and Chen, Yanxu and Wang, Harrison K and Ramakrishnan, Kalyan and Thornley, Elliott and Terpin, Antonio and Schoelkopf, Hailey and Zheng, Eric and Carmi, Avishy and Brown, Ethan D. L. and Zhu, Kelin and Bartolo, Max and Wheeler, Richard and Stehberger, Martin and Bradshaw, Peter and Heimonen, JP and Sridhar, Kaustubh and Akov, Ido and Sandlin, Jennifer and Makarychev, Yury and Tam, Joanna and Hoang, Hieu and Cunningham, David M. and Goryachev, Vladimir and Patramanis, Demosthenes and Krause, Michael and Redenti, Andrew and Aldous, David and Lai, Jesyin and Coleman, Shannon and Xu, Jiangnan and Lee, Sangwon and Magoulas, Ilias and Zhao, Sandy and Tang, Ning and Cohen, Michael K. and Paradise, Orr and Kirchner, Jan Hendrik and Ovchynnikov, Maksym and Matos, Jason O. and Shenoy, Adithya and Wang, Michael and Nie, Yuzhou and Sztyber-Betley, Anna and Faraboschi, Paolo and Riblet, Robin and Crozier, Jonathan and Halasyamani, Shiv and Verma, Shreyas and Joshi, Prashant and Meril, Eli and Ma, Ziqiao and Andréoletti, Jérémy and Singhal, Raghav and Platnick, Jacob and Nevirkovets, Volodymyr and Basler, Luke and Ivanov, Alexander and Khoury, Seri and Gustafsson, Nils and Piccardo, Marco and Mostaghimi, Hamid and Chen, Qijia and Singh, Virendra and Khánh, Tran Quoc and Rosu, Paul and Szlyk, Hannah and Brown, Zachary and Narayan, Himanshu and Menezes, Aline and Roberts, Jonathan and Alley, William and Sun, Kunyang and Patel, Arkil and Lamparth, Max and Reuel, Anka and Xin, Linwei and Xu, Hanmeng and Loader, Jacob and Martin, Freddie and Wang, Zixuan and Achilleos, Andrea and Preu, Thomas and Korbak, Tomek and Bosio, Ida and Kazemi, Fereshteh and Chen, Ziye and Bálint, Biró and Lo, Eve J. Y. and Wang, Jiaqi and Nunes, Maria Inês S. and Milbauer, Jeremiah and Bari, M Saiful and Wang, Zihao and Ansarinejad, Behzad and Sun, Yewen and Durand, Stephane and Elgnainy, Hossam and Douville, Guillaume and Tordera, Daniel and Balabanian, George and Wolff, Hew and Kvistad, Lynna and Milliron, Hsiaoyun and Sakor, Ahmad and Eron, Murat and O and Shah, Shailesh and Zhou, Xiaoxiang and Kamalov, Firuz and Abdoli, Sherwin and Santens, Tim and Barkan, Shaul and Tee, Allison and Zhang, Robin and Tomasiello, Alessandro and Luca, G. Bruno De and Looi, Shi-Zhuo and Le, Vinh-Kha and Kolt, Noam and Pan, Jiayi and Rodman, Emma and Drori, Jacob and Fossum, Carl J and Muennighoff, Niklas and Jagota, Milind and Pradeep, Ronak and Fan, Honglu and Eicher, Jonathan and Chen, Michael and Thaman, Kushal and Merrill, William and Firsching, Moritz and Harris, Carter and Ciobâcă, Stefan and Gross, Jason and Pandey, Rohan and Gusev, Ilya and Jones, Adam and Agnihotri, Shashank and Zhelnov, Pavel and Mofayezi, Mohammadreza and Piperski, Alexander and Zhang, David K. and Dobarskyi, Kostiantyn and Leventov, Roman and Soroko, Ignat and Duersch, Joshua and Taamazyan, Vage and Ho, Andrew and Ma, Wenjie and Held, William and Xian, Ruicheng and Zebaze, Armel Randy and Mohamed, Mohanad and Leser, Julian Noah and Yuan, Michelle X and Yacar, Laila and Lengler, Johannes and Olszewska, Katarzyna and Fratta, Claudio Di and Oliveira, Edson and Jackson, Joseph W. and Zou, Andy and Chidambaram, Muthu and Manik, Timothy and Haffenden, Hector and Stander, Dashiell and Dasouqi, Ali and Shen, Alexander and Golshani, Bita and Stap, David and Kretov, Egor and Uzhou, Mikalai and Zhidkovskaya, Alina Borisovna and Winter, Nick and Rodriguez, Miguel Orbegozo and Lauff, Robert and Wehr, Dustin and Tang, Colin and Hossain, Zaki and Phillips, Shaun and Samuele, Fortuna and Ekström, Fredrik and Hammon, Angela and Patel, Oam and Farhidi, Faraz and Medley, George and Mohammadzadeh, Forough and Peñaflor, Madellene and Kassahun, Haile and Friedrich, Alena and Perez, Rayner Hernandez and Pyda, Daniel and Sakal, Taom and Dhamane, Omkar and Mirabadi, Ali Khajegili and Hallman, Eric and Okutsu, Kenchi and Battaglia, Mike and Maghsoudimehrabani, Mohammad and Amit, Alon and Hulbert, Dave and Pereira, Roberto and Weber, Simon and Handoko and Peristyy, Anton and Malina, Stephen and Mehkary, Mustafa and Aly, Rami and Reidegeld, Frank and Dick, Anna-Katharina and Friday, Cary and Singh, Mukhwinder and Shapourian, Hassan and Kim, Wanyoung and Costa, Mariana and Gurdogan, Hubeyb and Kumar, Harsh and Ceconello, Chiara and Zhuang, Chao and Park, Haon and Carroll, Micah and Tawfeek, Andrew R. and Steinerberger, Stefan and Aggarwal, Daattavya and Kirchhof, Michael and Dai, Linjie and Kim, Evan and Ferret, Johan and Shah, Jainam and Wang, Yuzhou and Yan, Minghao and Burdzy, Krzysztof and Zhang, Lixin and Franca, Antonio and Pham, Diana T. and Loh, Kang Yong and Robinson, Joshua and Jackson, Abram and Giordano, Paolo and Petersen, Philipp and Cosma, Adrian and Colino, Jesus and White, Colin and Votava, Jacob and Vinnikov, Vladimir and Delaney, Ethan and Spelda, Petr and Stritecky, Vit and Shahid, Syed M. and Mourrat, Jean-Christophe and Vetoshkin, Lavr and Sponselee, Koen and Bacho, Renas and Yong, Zheng-Xin and Rosa, Florencia de la and Cho, Nathan and Li, Xiuyu and Malod, Guillaume and Weller, Orion and Albani, Guglielmo and Lang, Leon and Laurendeau, Julien and Kazakov, Dmitry and Adesanya, Fatimah and Portier, Julien and Hollom, Lawrence and Souza, Victor and Zhou, Yuchen Anna and Degorre, Julien and Yalın, Yiğit and Obikoya, Gbenga Daniel and Rai and Bigi, Filippo and Boscá, M. C. and Shumar, Oleg and Bacho, Kaniuar and Recchia, Gabriel and Popescu, Mara and Shulga, Nikita and Tanwie, Ngefor Mildred and Lux, Thomas C. H. and Rank, Ben and Ni, Colin and Brooks, Matthew and Yakimchyk, Alesia and Huanxu and Liu and Cavalleri, Stefano and Häggström, Olle and Verkama, Emil and Newbould, Joshua and Gundlach, Hans and Brito-Santana, Leonor and Amaro, Brian and Vajipey, Vivek and Grover, Rynaa and Wang, Ting and Kratish, Yosi and Li, Wen-Ding and Gopi, Sivakanth and Caciolai, Andrea and Witt, Christian Schroeder de and Hernández-Cámara, Pablo and Rodolà, Emanuele and Robins, Jules and Williamson, Dominic and Cheng, Vincent and Raynor, Brad and Qi, Hao and Segev, Ben and Fan, Jingxuan and Martinson, Sarah and Wang, Erik Y. and Hausknecht, Kaylie and Brenner, Michael P. and Mao, Mao and Demian, Christoph and Kassani, Peyman and Zhang, Xinyu and Avagian, David and Scipio, Eshawn Jessica and Ragoler, Alon and Tan, Justin and Sims, Blake and Plecnik, Rebeka and Kirtland, Aaron and Bodur, Omer Faruk and Shinde, D. P. and Labrador, Yan Carlos Leyva and Adoul, Zahra and Zekry, Mohamed and Karakoc, Ali and Santos, Tania C. B. and Shamseldeen, Samir and Karim, Loukmane and Liakhovitskaia, Anna and Resman, Nate and Farina, Nicholas and Gonzalez, Juan Carlos and Maayan, Gabe and Anderson, Earth and Pena, Rodrigo De Oliveira and Kelley, Elizabeth and Mariji, Hodjat and Pouriamanesh, Rasoul and Wu, Wentao and Finocchio, Ross and Alarab, Ismail and Cole, Joshua and Ferreira, Danyelle and Johnson, Bryan and Safdari, Mohammad and Dai, Liangti and Arthornthurasuk, Siriphan and McAlister, Isaac C. and Moyano, Alejandro José and Pronin, Alexey and Fan, Jing and Ramirez-Trinidad, Angel and Malysheva, Yana and Pottmaier, Daphiny and Taheri, Omid and Stepanic, Stanley and Perry, Samuel and Askew, Luke and Rodríguez, Raúl Adrián Huerta and Minissi, Ali M. R. and Lorena, Ricardo and Iyer, Krishnamurthy and Fasiludeen, Arshad Anil and Clark, Ronald and Ducey, Josh and Piza, Matheus and Somrak, Maja and Vergo, Eric and Qin, Juehang and Borbás, Benjámin and Chu, Eric and Lindsey, Jack and Jallon, Antoine and McInnis, I. M. J. and Chen, Evan and Semler, Avi and Gloor, Luk and Shah, Tej and Carauleanu, Marc and Lauer, Pascal and Huy, Tran Đuc and Shahrtash, Hossein and Duc, Emilien and Lewark, Lukas and Brown, Assaf and Albanie, Samuel and Weber, Brian and Vaz, Warren S. and Clavier, Pierre and Fan, Yiyang and Silva, Gabriel Poesia Reis e and Long and Lian and Abramovitch, Marcus and Jiang, Xi and Mendoza, Sandra and Islam, Murat and Gonzalez, Juan and Mavroudis, Vasilios and Xu, Justin and Kumar, Pawan and Goswami, Laxman Prasad and Bugas, Daniel and Heydari, Nasser and Jeanplong, Ferenc and Jansen, Thorben and Pinto, Antonella and Apronti, Archimedes and Galal, Abdallah and Ze-An, Ng and Singh, Ankit and Jiang, Tong and Xavier, Joan of Arc and Agarwal, Kanu Priya and Berkani, Mohammed and Zhang, Gang and Du, Zhehang and Junior, Benedito Alves de Oliveira and Malishev, Dmitry and Remy, Nicolas and Hartman, Taylor D. and Tarver, Tim and Mensah, Stephen and Loume, Gautier Abou and Morak, Wiktor and Habibi, Farzad and Hoback, Sarah and Cai, Will and Gimenez, Javier and Montecillo, Roselynn Grace and Łucki, Jakub and Campbell, Russell and Sharma, Asankhaya and Meer, Khalida and Gul, Shreen and Gonzalez, Daniel Espinosa and Alapont, Xavier and Hoover, Alex and Chhablani, Gunjan and Vargus, Freddie and Agarwal, Arunim and Jiang, Yibo and Patil, Deepakkumar and Outevsky, David and Scaria, Kevin Joseph and Maheshwari, Rajat and Dendane, Abdelkader and Shukla, Priti and Cartwright, Ashley and Bogdanov, Sergei and Mündler, Niels and Möller, Sören and Arnaboldi, Luca and Thaman, Kunvar and Siddiqi, Muhammad Rehan and Saxena, Prajvi and Gupta, Himanshu and Fruhauff, Tony and Sherman, Glen and Vincze, Mátyás and Usawasutsakorn, Siranut and Ler, Dylan and Radhakrishnan, Anil and Enyekwe, Innocent and Salauddin, Sk Md and Muzhen, Jiang and Maksapetyan, Aleksandr and Rossbach, Vivien and Harjadi, Chris and Bahaloohoreh, Mohsen and Sparrow, Claire and Sidhu, Jasdeep and Ali, Sam and Bian, Song and Lai, John and Singer, Eric and Uro, Justine Leon and Bateman, Greg and Sayed, Mohamed and Menshawy, Ahmed and Duclosel, Darling and Bezzi, Dario and Jain, Yashaswini and Aaron, Ashley and Tiryakioglu, Murat and Siddh, Sheeshram and Krenek, Keith and Shah, Imad Ali and Jin, Jun and Creighton, Scott and Peskoff, Denis and EL-Wasif, Zienab and P, Ragavendran and Richmond, Michael and McGowan, Joseph and Patwardhan, Tejal and Sun, Hao-Yu and Sun, Ting and Zubić, Nikola and Sala, Samuele and Ebert, Stephen and Kaddour, Jean and Schottdorf, Manuel and Wang, Dianzhuo and Petruzella, Gerol and Meiburg, Alex and Medved, Tilen and ElSheikh, Ali and Hebbar, S Ashwin and Vaquero, Lorenzo and Yang, Xianjun and Poulos, Jason and Zouhar, Vilém and Bogdanik, Sergey and Zhang, Mingfang and Sanz-Ros, Jorge and Anugraha, David and Dai, Yinwei and Nhu, Anh N. and Wang, Xue and Demircali, Ali Anil and Jia, Zhibai and Zhou, Yuyin and Wu, Juncheng and He, Mike and Chandok, Nitin and Sinha, Aarush and Luo, Gaoxiang and Le, Long and Noyé, Mickaël and Perełkiewicz, Michał and Pantidis, Ioannis and Qi, Tianbo and Purohit, Soham Sachin and Parcalabescu, Letitia and Nguyen, Thai-Hoa and Winata, Genta Indra and Ponti, Edoardo M. and Li, Hanchen and Dhole, Kaustubh and Park, Jongee and Abbondanza, Dario and Wang, Yuanli and Nayak, Anupam and Caetano, Diogo M. and Wong, Antonio A. W. L. and Rio-Chanona, Maria del and Kondor, Dániel and Francois, Pieter and Chalstrey, Ed and Zsambok, Jakob and Hoyer, Dan and Reddish, Jenny and Hauser, Jakob and Rodrigo-Ginés, Francisco-Javier and Datta, Suchandra and Shepherd, Maxwell and Kamphuis, Thom and Zhang, Qizheng and Kim, Hyunjun and Sun, Ruiji and Yao, Jianzhu and Dernoncourt, Franck and Krishna, Satyapriya and Rismanchian, Sina and Pu, Bonan and Pinto, Francesco and Wang, Yingheng and Shridhar, Kumar and Overholt, Kalon J. and Briia, Glib and Nguyen, Hieu and David and Bartomeu, Soler and Pang, Tony CY and Wecker, Adam and Xiong, Yifan and Li, Fanfei and Huber, Lukas S. and Jaeger, Joshua and Maddalena, Romano De and Lù, Xing Han and Zhang, Yuhui and Beger, Claas and Kon, Patrick Tser Jern and Li, Sean and Sanker, Vivek and Yin, Ming and Liang, Yihao and Zhang, Xinlu and Agrawal, Ankit and Yifei, Li S. and Zhang, Zechen and Cai, Mu and Sonmez, Yasin and Cozianu, Costin and Li, Changhao and Slen, Alex and Yu, Shoubin and Park, Hyun Kyu and Sarti, Gabriele and Briański, Marcin and Stolfo, Alessandro and Nguyen, Truong An and Zhang, Mike and Perlitz, Yotam and Hernandez-Orallo, Jose and Li, Runjia and Shabani, Amin and Juefei-Xu, Felix and Dhingra, Shikhar and Zohar, Orr and Nguyen, My Chiffon and Pondaven, Alexander and Yilmaz, Abdurrahim and Zhao, Xuandong and Jin, Chuanyang and Jiang, Muyan and Todoran, Stefan and Han, Xinyao and Kreuer, Jules and Rabern, Brian and Plassart, Anna and Maggetti, Martino and Yap, Luther and Geirhos, Robert and Kean, Jonathon and Wang, Dingsu and Mollaei, Sina and Sun, Chenkai and Yin, Yifan and Wang, Shiqi and Li, Rui and Chang, Yaowen and Wei, Anjiang and Bizeul, Alice and Wang, Xiaohan and Arrais, Alexandre Oliveira and Mukherjee, Kushin and Chamorro-Padial, Jorge and Liu, Jiachen and Qu, Xingyu and Guan, Junyi and Bouyamourn, Adam and Wu, Shuyu and Plomecka, Martyna and Chen, Junda and Tang, Mengze and Deng, Jiaqi and Subramanian, Shreyas and Xi, Haocheng and Chen, Haoxuan and Zhang, Weizhi and Ren, Yinuo and Tu, Haoqin and Kim, Sejong and Chen, Yushun and Marjanović, Sara Vera and Ha, Junwoo and Luczyna, Grzegorz and Ma, Jeff J. and Shen, Zewen and Song, Dawn and Zhang, Cedegao E. and Wang, Zhun and Gendron, Gaël and Xiao, Yunze and Smucker, Leo and Weng, Erica and Lee, Kwok Hao and Ye, Zhe and Ermon, Stefano and Lopez-Miguel, Ignacio D. and Knights, Theo and Gitter, Anthony and Park, Namkyu and Wei, Boyi and Chen, Hongzheng and Pai, Kunal and Elkhanany, Ahmed and Lin, Han and Siedler, Philipp D. and Fang, Jichao and Mishra, Ritwik and Zsolnai-Fehér, Károly and Jiang, Xilin and Khan, Shadab and Yuan, Jun and Jain, Rishab Kumar and Lin, Xi and Peterson, Mike and Wang, Zhe and Malusare, Aditya and Tang, Maosen and Gupta, Isha and Fosin, Ivan and Kang, Timothy and Dworakowska, Barbara and Matsumoto, Kazuki and Zheng, Guangyao and Sewuster, Gerben and Villanueva, Jorge Pretel and Rannev, Ivan and Chernyavsky, Igor and Chen, Jiale and Banik, Deepayan and Racz, Ben and Dong, Wenchao and Wang, Jianxin and Bashmal, Laila and Gonçalves, Duarte V. and Hu, Wei and Bar, Kaushik and Bohdal, Ondrej and Patlan, Atharv Singh and Dhuliawala, Shehzaad and Geirhos, Caroline and Wist, Julien and Kansal, Yuval and Chen, Bingsen and Tire, Kutay and Yücel, Atak Talay and Christof, Brandon and Singla, Veerupaksh and Song, Zijian and Chen, Sanxing and Ge, Jiaxin and Ponkshe, Kaustubh and Park, Isaac and Shi, Tianneng and Ma, Martin Q. and Mak, Joshua and Lai, Sherwin and Moulin, Antoine and Cheng, Zhuo and Zhu, Zhanda and Zhang, Ziyi and Patil, Vaidehi and Jha, Ketan and Men, Qiutong and Wu, Jiaxuan and Zhang, Tianchi and Vieira, Bruno Hebling and Aji, Alham Fikri and Chung, Jae-Won and Mahfoud, Mohammed and Hoang, Ha Thi and Sperzel, Marc and Hao, Wei and Meding, Kristof and Xu, Sihan and Kostakos, Vassilis and Manini, Davide and Liu, Yueying and Toukmaji, Christopher and Paek, Jay and Yu, Eunmi and Demircali, Arif Engin and Sun, Zhiyi and Dewerpe, Ivan and Qin, Hongsen and Pflugfelder, Roman and Bailey, James and Morris, Johnathan and Heilala, Ville and Rosset, Sybille and Yu, Zishun and Chen, Peter E. and Yeo, Woongyeong and Jain, Eeshaan and Yang, Ryan and Chigurupati, Sreekar and Chernyavsky, Julia and Reddy, Sai Prajwal and Venugopalan, Subhashini and Batra, Hunar and Park, Core Francisco and Tran, Hieu and Maximiano, Guilherme and Zhang, Genghan and Liang, Yizhuo and Shiyu, Hu and Xu, Rongwu and Pan, Rui and Suresh, Siddharth and Liu, Ziqi and Gulati, Samaksh and Zhang, Songyang and Turchin, Peter and Bartlett, Christopher W. and Scotese, Christopher R. and Cao, Phuong M. and Wu, Ben and Karwowski, Jacek and Scaramuzza, Davide and Nattanmai, Aakaash and McKellips, Gordon and Cheraku, Anish and Suhail, Asim and Luo, Ethan and Deng, Marvin and Luo, Jason and Zhang, Ashley and Jindel, Kavin and Paek, Jay and Halevy, Kasper and Baranov, Allen and Liu, Michael and Avadhanam, Advaith and Zhang, David and Cheng, Vincent and Ma, Brad and Fu, Evan and Do, Liam and Lass, Joshua and Yang, Hubert and Sunkari, Surya and Bharath, Vishruth and Ai, Violet and Leung, James and Agrawal, Rishit and Zhou, Alan and Chen, Kevin and Kalpathi, Tejas and Xu, Ziqi and Wang, Gavin and Xiao, Tyler and Maung, Erik and Lee, Sam and Yang, Ryan and Yue, Roy and Zhao, Ben and Yoon, Julia and Sun, Sunny and Singh, Aryan and Luo, Ethan and Peng, Clark and Osbey, Tyler and Wang, Taozhi and Echeazu, Daryl and Yang, Hubert and Wu, Timothy and Patel, Spandan and Kulkarni, Vidhi and Sundarapandiyan, Vijaykaarti and Zhang, Ashley and Le, Andrew and Nasim, Zafir and Yalam, Srikar and Kasamsetty, Ritesh and Samal, Soham and Yang, Hubert and Sun, David and Shah, Nihar and Saha, Abhijeet and Zhang, Alex and Nguyen, Leon and Nagumalli, Laasya and Wang, Kaixin and Zhou, Alan and Wu, Aidan and Luo, Jason and Telluri, Anwith and Dillmann, Steven and Wang, Zhengxiang and Luo, Junyu and Lunn, Hugo and Gazizov, Artem and Borde, Haitz Sáez de Ocáriz and Trus, Ivan and Hervault, Morgan and Zhang, Zheyu and Chen, Bo and Wu, Yuchen and Cordier, Christopher J. and Kaynar, Gün and Ayvaz, Cansin and Avdiunina, Polina and Brust, Johannes and Diao, Xingjian and Meaney, K. D. and Gu, Yifan and Wang, Chenyu and Dong, Chenzhuo and Wright, William and Brave, Simon and Root, Owen and Liu, Jiayuan and Lok, Chow Chun and Li, Tianqin and Du, Shiyi and He, Dailan and Liu, Lufeiya and Jamalzadegan, Sina and Ramakrishna, Anil and Xu, Xuanqing and Qing, Xin and Luo, Xin and Li, Wenkai and Bo, Shi and Gusev, Filipp and Skandalis, Maximos and Ma, Desheng and Zhang, Chunhui and Qiu, Haoran and Hart, Allen G and Gabrielsson, Rickard Brüel and Akov, Ido and Lukoianov, Artem and Yue, Summer and Wang, Alexandr and Hendrycks, Dan},
  year = {2025},
  doi = {10.1038/s41586-025-09962-4},
  eprint = {2501.14249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41586-025-09962-4}
}

References

63 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Alberti, K. Lee, and M. Collins. A bert baseline for the natural questions, 2019. URL https://arxiv.org/abs/1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) ↗
  2. M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y. Gal, and X. Davies. Agentharm: A benchmark for measuring harmfulness of llm agents, 2024. URL https://arxiv.org/abs/ 2410.09024. ↗
  3. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://api. semanticscholar.org/CorpusID:268232499. find
  4. Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet, 2024. URL https://assets.anthropic.com/m/1cd9d098ac3e6467/original/ Claude-3-Model-Card-October-Addendum.pdf. ↗
  5. Anthropic. Responsible scaling policy updates, 2024. URL https://www.anthropic.com/ rsp-updates. ↗
  6. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL https://arxiv.org/abs/ 2505.08775. ↗
  7. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732. find our card
  8. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J.… find
  9. P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL https://arxiv.org/ abs/1611.09268. ↗ our card
  10. M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe. Purple llama cyberseceval: A secure coding b… find
  11. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M adry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024. URL https://arxiv.org/abs/2410.07095. ↗ our card
  12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… find our card
  13. F. Chollet, M. Knoop, G. Kamradt, and B. Landers. Arc prize 2024: Technical report, 2024. URL https://arxiv.org/abs/2412.04604. ↗
  14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
  15. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://github.com/ deepseek-ai/DeepSeek-V3/blob/main/DeepSeek\_V3.pdf. ↗
  16. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https: //arxiv.org/abs/1903.00161. find our card
  17. A. Dubey et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783. ↗
  18. B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024. URL https://arxiv.org/abs/2410.07… ↗ our card
  19. E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. de Oliveira Santos, O. Järviniemi, M. Barnett, R. Sandler, J. Sevilla, Q. Ren, E. Pratt, L. Levine, G. Barkley, N. Stewart, B. Grechuk, T. Grechuk, and S. V. Enugandla. Frontiermath: A benchma… find
  20. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024. URL https://arxiv.org/ abs/2402.14008. ↗ our card
  21. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021. URL https://arxiv.org/abs/2105.09938. ↗ our card
  22. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009. 03300. ↗ our card
  23. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv. org/abs/2103.03874. find our card
  24. D. Hendrycks, A. Zou, M. Mazeika, L. Tang, B. Li, D. Song, and J. Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures, 2022. URL https://arxiv. org/abs/2112.05135. — [arXiv:2112.05135](https://arxiv.org/abs/2112.05135) find
  25. A. Hosseini, A. Sordoni, D. Toyama, A. Courville, and R. Agarwal. Not all llm reasoners are created equal, 2024. URL https://arxiv.org/abs/2410.01748. — [arXiv:2410.01748](https://arxiv.org/abs/2410.01748) ↗
  26. A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. W. andMadhu Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, R. Sims, Z. Zhang, S. Goldshtein, and D.… find
  27. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/ abs/2310.06770. ↗ our card
  28. D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in nlp, 2021. URL https://arxiv. org/abs/2104.14337. find
  29. P. Kumar, E. Lau, S. Vijayakumar, T. Trinh, S. R. Team, E. Chang, V. Robinson, S. Hendryx, S. Zhou, M. Fredrikson, S. Yue, and Z. Wang. Refusal-trained llms are easily jailbroken as browser agents, 2024. URL https://arxiv.org/abs/2410.13886. — [arXiv:2410.13886](https://arxiv.org/abs/2410.13886) ↗
  30. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Pon-napati, A. D. White, and S. G. Rodriques. Lab-bench: Measuring capabilities of language models for biology research, 2024. URL https://arxiv.org/abs/2407.10362. ↗
  31. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A.… find
  32. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255. ↗ our card
  33. T. R. McIntosh, T. Susnjak, N. Arachchilage, T. Liu, P. Watters, and M. N. Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence, 2024. URL https://arxiv.org/abs/2402.09880. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) ↗
  34. Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial nli: A new benchmark for natural language understanding, 2020. URL https://arxiv.org/abs/1910. 14599. Openai o1 system card, 2024. ↗
  35. OpenAI. o1-system-card-20240917.pdf. URL https://cdn.openai.com/ ↗
  36. OpenAI. Openai and los alamos national laboratory announce bioscience research partnership, 2024. URL https://openai.com/index/ openai-and-los-alamos-national-laboratory-work-together/. ↗
  37. OpenAI. Introducing swe-bench verified, 2024. introducing-swe-bench-verified/. URL https://openai.com/index/ ↗
  38. OpenAI et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. ↗ our card
  39. S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, and M. Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13(1): 6793, 2022. find
  40. D. Owen. How predictable is language model benchmark performance?, 2024. URL https: //arxiv.org/abs/2401.04757. find
  41. E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. M… find our card
  42. M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, H. Howard, T. Lieberum, R. Kumar, M. A. Raad, A. Webson, L. Ho, S. Lin, S. Farquhar, M. Hutter, G. Deletang, A. Ruoss, S. El-Sayed, S. Brown, A. Dragan, R. Shah, A. Dafoe, and T. She… find
  43. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250. ↗ our card
  44. P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad, 2018. URL https://arxiv.org/abs/1806.03822. ↗ our card
  45. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv. org/abs/2311.12022. find our card
  46. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620 (7972):172–180, 2023. find our card
  47. M. Skarlinski, J. Laurent, A. Bou, and A. White. About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong, July 2025. URL https://www.futurehouse. org/research-announcements/hle-exam. ↗
  48. V. K. Srinivasan, Z. Dong, B. Zhu, B. Yu, H. Mao, D. Mosk-Aoyama, K. Keutzer, J. Jiao, and J. Zhang. Nexusraven: A commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https: //openreview.net/forum?id=5lcPe6DqfI. find
  49. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slon… find our card
  50. S. A. Taghanaki, A. Khani, and A. Khasahmadi. Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in llms, 2024. URL https://arxiv.org/abs/2409.02257. — [arXiv:2409.02257](https://arxiv.org/abs/2409.02257) ↗
  51. G. Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530. ↗ our card
  52. G. Tsoukalas, J. Lee, J. Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri. Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition, 2024. URL https://arxiv.org/abs/2407.11214. ↗
  53. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https: //arxiv.org/abs/1804.07461. find our card
  54. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020. URL https://arxiv.org/abs/1905.00537. ↗ our card
  55. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks), 2024… find our card
  56. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/ abs/2411.04368. ↗ our card
  57. H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai r&d capabilities of langua… find
  58. xAI. Grok-2 beta release, 2024. URL https://x.ai/blog/grok-2. ↗
  59. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8\_berkeley\_ function\_calling\_leaderboard.html, 2024. ↗
  60. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600. ↗ our card
  61. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045. ↗ our card
  62. A. K. Zhang, N. Perry, R. Dulepet, J. Ji, J. W. Lin, E. Jones, C. Menders, G. Hussein, S. Liu, D. Jasper, P. Peetathawatchai, A. Glenn, V. Sivashankar, D. Zamoshchin, L. Glikbarg, D. Askar-yar, M. Yang, T. Zhang, R. Alluri, N. Tran, R. Sangpisit, P. Yiorkadjis, K. Osele, G. Raghupathi, D. Boneh, D.… — [arXiv:2408.08926](https://arxiv.org/abs/2408.08926) find
  63. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https: //arxiv.org/abs/2304.06364. find our card