Elamur Research
☰
home / papers / 2024 / GPT-4o System Card

GPT-4o System Card

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2410.21276

Summary

The GPT-4o System Card explains how OpenAI assessed and reduced the risks of GPT-4o. GPT-4o is a single end-to-end network that accepts text, audio, image and video and can produce text, audio and image output. Because the model holds real-time spoken conversations, the report focuses on hazards specific to speech-to-speech use, but it also checks text and vision.

Pre-training data (cutoff October 2023) came from public web crawls, ML datasets, code and math, multimodal material and licensed partner content. It was filtered with the Moderation API, safety classifiers, personal-data reduction and fingerprints of DALL-E 3 opt-out images.

Risks were found through staged external red teaming and then turned into measurable evaluations. Existing text benchmarks were converted to audio with Voice Engine (TTS) so they could be reused. Mitigations combine post-training refusals with deployed classifiers, for example:

  • limiting output to preset voices,
  • refusing speaker identification,
  • blocking music,
  • hedging on sensitive traits.

Preparedness Framework and third-party results also appear. The card serves as a practical template for safety-testing voice-enabled multimodal models.

Key points

  • Audio response latency is as low as 232 ms, with a 320 ms average, which is close to human conversational timing. The model matches GPT-4 Turbo on English text and code, improves on non-English languages, and costs 50% less in the API.
  • Red teaming involved more than 100 external testers covering 45 languages and 29 countries. It ran in four phases (10, 30, 65 and 65 people) from early March to late June 2024, and the final phase used advanced voice mode in the iOS app.
  • TTS-based evaluation has known limits. Equations and code convert poorly to speech, and accents, background noise, cross-talk and intonation are not fully represented.
  • Moderation classifiers run on transcripts of both audio prompts and generations, and high-severity outputs are blocked. In the limited alpha of Advanced Voice Mode, the model was told not to sing.
  • External contributors included METR, Apollo Research and Virtue AI, plus the Uhura Evals group, which is relevant for low-resource languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

2045 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 2020 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.21276 · arXiv 2410.21276

@misc{openai2024gpt,
  title = {{GPT-4o System Card}},
  author = {OpenAI and Hurst, Aaron and Lerer, Adam and Goucher, Adam P. and Perelman, Adam and Ramesh, Aditya and Clark, Aidan and Ostrow, AJ and Welihinda, Akila and Hayes, Alan and Radford, Alec and Mądry, Aleksander and Baker-Whitcomb, Alex and Beutel, Alex and Borzunov, Alex and Carney, Alex and Chow, Alex and Kirillov, Alex and Nichol, Alex and Paino, Alex and Renzin, Alex and Passos, Alex Tachard and Kirillov, Alexander and Christakis, Alexi and Conneau, Alexis and Kamali, Ali and Jabri, Allan and Moyer, Allison and Tam, Allison and Crookes, Amadou and Tootoochian, Amin and Tootoonchian, Amin and Kumar, Ananya and Vallone, Andrea and Karpathy, Andrej and Braunstein, Andrew and Cann, Andrew and Codispoti, Andrew and Galu, Andrew and Kondrich, Andrew and Tulloch, Andrew and Mishchenko, Andrey and Baek, Angela and Jiang, Angela and Pelisse, Antoine and Woodford, Antonia and Gosalia, Anuj and Dhar, Arka and Pantuliano, Ashley and Nayak, Avi and Oliver, Avital and Zoph, Barret and Ghorbani, Behrooz and Leimberger, Ben and Rossen, Ben and Sokolowsky, Ben and Wang, Ben and Zweig, Benjamin and Hoover, Beth and Samic, Blake and McGrew, Bob and Spero, Bobby and Giertler, Bogo and Cheng, Bowen and Lightcap, Brad and Walkin, Brandon and Quinn, Brendan and Guarraci, Brian and Hsu, Brian and Kellogg, Bright and Eastman, Brydon and Lugaresi, Camillo and Wainwright, Carroll and Bassin, Cary and Hudson, Cary and Chu, Casey and Nelson, Chad and Li, Chak and Shern, Chan Jun and Conger, Channing and Barette, Charlotte and Voss, Chelsea and Ding, Chen and Lu, Cheng and Zhang, Chong and Beaumont, Chris and Hallacy, Chris and Koch, Chris and Gibson, Christian and Kim, Christina and Choi, Christine and McLeavey, Christine and Hesse, Christopher and Fischer, Claudia and Winter, Clemens and Czarnecki, Coley and Jarvis, Colin and Wei, Colin and Koumouzelis, Constantin and Sherburn, Dane and Kappler, Daniel and Levin, Daniel and Levy, Daniel and Carr, David and Farhi, David and Mely, David and Robinson, David and Sasaki, David and Jin, Denny and Valladares, Dev and Tsipras, Dimitris and Li, Doug and Nguyen, Duc Phong and Findlay, Duncan and Oiwoh, Edede and Wong, Edmund and Asdar, Ehsan and Proehl, Elizabeth and Yang, Elizabeth and Antonow, Eric and Kramer, Eric and Peterson, Eric and Sigler, Eric and Wallace, Eric and Brevdo, Eugene and Mays, Evan and Khorasani, Farzad and Such, Felipe Petroski and Raso, Filippo and Zhang, Francis and Lohmann, Fred von and Sulit, Freddie and Goh, Gabriel and Oden, Gene and Salmon, Geoff and Starace, Giulio and Brockman, Greg and Salman, Hadi and Bao, Haiming and Hu, Haitang and Wong, Hannah and Wang, Haoyu and Schmidt, Heather and Whitney, Heather and Jun, Heewoo and Kirchner, Hendrik and Pinto, Henrique Ponde de Oliveira and Ren, Hongyu and Chang, Huiwen and Chung, Hyung Won and Kivlichan, Ian and O'Connell, Ian and O'Connell, Ian and Osband, Ian and Silber, Ian and Sohl, Ian and Okuyucu, Ibrahim and Lan, Ikai and Kostrikov, Ilya and Sutskever, Ilya and Kanitscheider, Ingmar and Gulrajani, Ishaan and Coxon, Jacob and Menick, Jacob and Pachocki, Jakub and Aung, James and Betker, James and Crooks, James and Lennon, James and Kiros, Jamie and Leike, Jan and Park, Jane and Kwon, Jason and Phang, Jason and Teplitz, Jason and Wei, Jason and Wolfe, Jason and Chen, Jay and Harris, Jeff and Varavva, Jenia and Lee, Jessica Gan and Shieh, Jessica and Lin, Ji and Yu, Jiahui and Weng, Jiayi and Tang, Jie and Yu, Jieqi and Jang, Joanne and Candela, Joaquin Quinonero and Beutler, Joe and Landers, Joe and Parish, Joel and Heidecke, Johannes and Schulman, John and Lachman, Jonathan and McKay, Jonathan and Uesato, Jonathan and Ward, Jonathan and Kim, Jong Wook and Huizinga, Joost and Sitkin, Jordan and Kraaijeveld, Jos and Gross, Josh and Kaplan, Josh and Snyder, Josh and Achiam, Joshua and Jiao, Joy and Lee, Joyce and Zhuang, Juntang and Harriman, Justyn and Fricke, Kai and Hayashi, Kai and Singhal, Karan and Shi, Katy and Karthik, Kavin and Wood, Kayla and Rimbach, Kendra and Hsu, Kenny and Nguyen, Kenny and Gu-Lemberg, Keren and Button, Kevin and Liu, Kevin and Howe, Kiel and Muthukumar, Krithika and Luther, Kyle and Ahmad, Lama and Kai, Larry and Itow, Lauren and Workman, Lauren and Pathak, Leher and Chen, Leo and Jing, Li and Guy, Lia and Fedus, Liam and Zhou, Liang and Mamitsuka, Lien and Weng, Lilian and McCallum, Lindsay and Held, Lindsey and Ouyang, Long and Feuvrier, Louis and Zhang, Lu and Kondraciuk, Lukas and Kaiser, Lukasz and Hewitt, Luke and Metz, Luke and Doshi, Lyric and Aflak, Mada and Simens, Maddie and Boyd, Madelaine and Thompson, Madeleine and Dukhan, Marat and Chen, Mark and Gray, Mark and Hudnall, Mark and Zhang, Marvin and Aljubeh, Marwan and Litwin, Mateusz and Zeng, Matthew and Johnson, Max and Shetty, Maya and Gupta, Mayank and Shah, Meghan and Yatbaz, Mehmet and Yang, Meng Jia and Zhong, Mengchao and Glaese, Mia and Chen, Mianna and Janner, Michael and Lampe, Michael and Petrov, Michael and Wu, Michael and Wang, Michele and Fradin, Michelle and Pokrass, Michelle and Castro, Miguel and Castro, Miguel Oom Temudo de and Pavlov, Mikhail and Brundage, Miles and Wang, Miles and Khan, Minal and Murati, Mira and Bavarian, Mo and Lin, Molly and Yesildal, Murat and Soto, Nacho and Gimelshein, Natalia and Cone, Natalie and Staudacher, Natalie and Summers, Natalie and LaFontaine, Natan and Chowdhury, Neil and Ryder, Nick and Stathas, Nick and Turley, Nick and Tezak, Nik and Felix, Niko and Kudige, Nithanth and Keskar, Nitish and Deutsch, Noah and Bundick, Noel and Puckett, Nora and Nachum, Ofir and Okelola, Ola and Boiko, Oleg and Murk, Oleg and Jaffe, Oliver and Watkins, Olivia and Godement, Olivier and Campbell-Moore, Owen and Chao, Patrick and McMillan, Paul and Belov, Pavel and Su, Peng and Bak, Peter and Bakkum, Peter and Deng, Peter and Dolan, Peter and Hoeschele, Peter and Welinder, Peter and Tillet, Phil and Pronin, Philip and Tillet, Philippe and Dhariwal, Prafulla and Yuan, Qiming and Dias, Rachel and Lim, Rachel and Arora, Rahul and Troll, Rajan and Lin, Randall and Lopes, Rapha Gontijo and Puri, Raul and Miyara, Reah and Leike, Reimar and Gaubert, Renaud and Zamani, Reza and Wang, Ricky and Donnelly, Rob and Honsby, Rob and Smith, Rocky and Sahai, Rohan and Ramchandani, Rohit and Huet, Romain and Carmichael, Rory and Zellers, Rowan and Chen, Roy and Chen, Ruby and Nigmatullin, Ruslan and Cheu, Ryan and Jain, Saachi and Altman, Sam and Schoenholz, Sam and Toizer, Sam and Miserendino, Samuel and Agarwal, Sandhini and Culver, Sara and Ethersmith, Scott and Gray, Scott and Grove, Sean and Metzger, Sean and Hermani, Shamez and Jain, Shantanu and Zhao, Shengjia and Wu, Sherwin and Jomoto, Shino and Wu, Shirong and Shuaiqi and Xia and Phene, Sonia and Papay, Spencer and Narayanan, Srinivas and Coffey, Steve and Lee, Steve and Hall, Stewart and Balaji, Suchir and Broda, Tal and Stramer, Tal and Xu, Tao and Gogineni, Tarun and Christianson, Taya and Sanders, Ted and Patwardhan, Tejal and Cunninghman, Thomas and Degry, Thomas and Dimson, Thomas and Raoux, Thomas and Shadwell, Thomas and Zheng, Tianhao and Underwood, Todd and Markov, Todor and Sherbakov, Toki and Rubin, Tom and Stasi, Tom and Kaftan, Tomer and Heywood, Tristan and Peterson, Troy and Walters, Tyce and Eloundou, Tyna and Qi, Valerie and Moeller, Veit and Monaco, Vinnie and Kuo, Vishal and Fomenko, Vlad and Chang, Wayne and Zheng, Weiyi and Zhou, Wenda and Manassra, Wesam and Sheu, Will and Zaremba, Wojciech and Patil, Yash and Qian, Yilei and Kim, Yongjik and Cheng, Youlong and Zhang, Yu and He, Yuchen and Zhang, Yuchen and Jin, Yujia and Dai, Yunxing and Malkov, Yury},
  year = {2024},
  doi = {10.48550/arxiv.2410.21276},
  eprint = {2410.21276},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.21276}
}

References

59 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Stivers, N. J. Enfield, P. Brown, C. Englert, M. Hayashi, T. Heinemann, G. Hoymann, F. Rossano, J. P. de Ruiter, K. E. Yoon, and S. C. Levinson, “Universals and cultural variation in turn-taking in conversation,” Proceedings of the National Academy of Sciences, vol. 106, no. 26, pp. 10587–10592,… find
  2. The White House, “Fact sheet: Biden-harris administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by ai,” 2023. find
  3. OpenAI, “Openai preparedness framework openai-preparedness-framework-beta.pdf. beta,” 2023. https://cdn.openai.com/ ↗
  4. Shutterstock, “Shutterstock press release,” 2023. find
  5. OpenAI, “Gpt-4 technical report,” 2023. find our card
  6. OpenAI, “Gpt-4v(ision) system card.” https://openai.com/index/gpt-4v-system-card/, 2023. Accessed: 2024-07-22. ↗
  7. OpenAI, “Navigating the challenges and opportunities of synthetic voices.” https://openai.com/index/ navigating-the-challenges-and-opportunities-of-synthetic-voices/, 2024. Accessed: 2024-07-22. ↗
  8. K. T. Mai, S. Bray, T. Davies, and L. D. Griffin, “Warning: Humans cannot reliably detect speech deepfakes,” PLoS One, vol. 18, p. e0285333, Aug. 2023. find
  9. M. Mori, K. F. MacDorman, and N. Kageki, “The uncanny valley [from the field],” IEEE Robotics & automation magazine, vol. 19, no. 2, pp. 98–100, 2012. find
  10. OpenAI, “How the voices for chatgpt were chosen,” 2024. find
  11. I. Solaiman, Z. Talat, W. Agnew, L. Ahmad, D. Baker, S. L. Blodgett, C. Chen, H. D. I. au2, J. Dodge, I. Duan, E. Evans, F. Friedrich, A. Ghosh, U. Gohar, S. Hooker, Y. Jernite, R. Kalluri, A. Lusoli, A. Leidinger, M. Lin, X. Lin, S. Luccioni, J. Mickel, M. Mitchell, J. Newman, A. Ovalle, M.-T. Png,… find
  12. R. Shelby, S. Rismani, K. Henne, A. Moon, N. Rostamzadeh, P. Nicholas, N. Yilla, J. Gallegos, A. Smart, E. Garcia, and G. Virk, “Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction,” 2023. find
  13. S. L. Blodgett, Q. V. Liao, A. Olteanu, R. Mihalcea, M. Muller, M. K. Scheuerman, C. Tan, and Q. Yang, “Responsible language technologies: Foreseeing and mitigating harms,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, (New York, NY, USA), Associ… find
  14. H. Suresh and J. Guttag, “A framework for understanding sources of harm throughout the machine learning life cycle,” in Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’21, ACM, Oct. 2021. find
  15. S. Shahriar, S. Allana, S. M. Hazratifard, and R. Dara, “A survey of privacy risks and mitigation strategies in the artificial intelligence life cycle,” IEEE Access, vol. 11, pp. 61829–61854, 2023. find
  16. OpenAI, “Moderation overview,” 2024. find
  17. A. Tamkin, M. Brundage, J. Clark, and D. Ganguli, “Understanding the capabilities, limitations, and societal impact of large language models,” 2021. find
  18. B. Buchanan, A. Lohn, M. Musser, and K. Sedova, “Truth, lies, and automation: How language models could change disinformation,” May 2021. find
  19. OpenAI, “Openai usage policies,” 2023. https://openai.com/policies/usage-policies/. ↗
  20. OpenAI, “Building an early warning system for llm-aided biological threat creation,” 2024. https://openai.com/ index/building-an-early-warning-system-for-llm-aided-biological-threat-creation/. ↗
  21. Deloitte, “Deloitte acquires gryphon scientific business to expand security science and public health capabilities,” 2024. https://www2.deloitte.com/us/en/pages/about-deloitte/articles/press-releases/ deloitte-acquires-gryphon-scientific-business-to-expand-security-science-and-public-health-capabili… ↗
  22. L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, and W. Isaac, “Sociotechnical safety evaluation of generative ai systems,” 2023. find
  23. A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, and D. Ganguli, “Evaluating and mitigating discrimination in language model decisions,” 2023. find
  24. J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” 2023. find
  25. I. Pentina, T. Hancock, and T. Xie, “Exploring relationship development with social chatbots: A mixed-method study of replika,” Computers in Human Behavior, vol. 140, p. 107600, 2023. find
  26. Y. Bengio, G. Hinton, A. Yao, D. Song, P. Abbeel, T. Darrell, Y. N. Harari, Y.-Q. Zhang, L. Xue, S. Shalev-Shwartz, G. Hadfield, J. Clune, T. Maharaj, F. Hutter, A. G. Baydin, S. McIlraith, Q. Gao, A. Acharya, D. Krueger, A. Dragan, P. Torr, S. Russell, D. Kahneman, J. Brauner, and S. Mindermann, “M… find
  27. S. B. Johnson, J. R. Clark, M. C. Luetke, N. M. Butala, A. T. Pearson, J. M. Shapiro, D. M. Aleman, J. M. Lee, M. M. Beil, C. V. Winkle, M. C. Boudreaux, R. C. D’Cunha, H. J. Krouse, and C. Li, “Chatgpt in medical education: a workshop-based large language model-powered intervention for evidence-bas… find
  28. K. Kavukcuoglu, “Real-world challenges for agi,” Nov 2021. find
  29. S. Altman, “Planning for agi and beyond,” OpenAI, 2023. find
  30. T. Eloundou, S. Manning, P. Mishkin, and D. Rock, “Gpts are gpts: An early look at the labor market impact potential of large language models,” arXiv preprint arXiv:2303.10130, 2023. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) find
  31. L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, et al., “Sociotechnical safety evaluation of generative ai systems,” arXiv preprint arXiv:2310.11986, 2023. — [arXiv:2310.11986](https://arxiv.org/abs/2310.11986) find
  32. S. Cox, M. Hammerling, J. Lála, J. Laurent, S. Rodriques, M. Rubashkin, and A. White, “Wikicrow: Automating synthesis of human scientific knowledge,” Future House, 2023. find
  33. S. A. Athaluri, S. V. Manthena, V. S. R. K. M. Kesapragada, V. Yarlagadda, T. Dave, and R. T. S. Duddumpudi, “Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through chatgpt references,” Cureus, vol. 15, no. 4, p. e3743… find
  34. Z. Li, “The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination,” 2023. find
  35. M. Dubiel, A. Sergeeva, and L. A. Leiva, “Impact of voice fidelity on decision making: A potential dark pattern?,” 2024. find
  36. B. Waber, M. Williams, J. S. Carroll, and A. S. Pentland, “A voice is worth a thousand words: The implications of the micro-coding of social signals in speech for trust research,” in Handbook of Research Methods on Trust (G. M. Fergus Lyon and M. N. Saunders, eds.), ch. 23, p. 320, New York: Edward… find
  37. I. Pentina, B. Guo, and W. P. Fan, “Friend, mentor, lover: Does chatbot engagement lead to psychological dependence?,” Journal of Service Management, 2023. find
  38. H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” arXiv preprint arXiv:2303.13375, 2023. find our card
  39. H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu, et al., “Can generalist foundation models outcompete special-purpose tuning? case study in medicine,” arXiv preprint arXiv:2311.16452, 2023. find
  40. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, N. Scharli, A. Chowdhery, P. Mansfield, B. A. y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Ra… find our card
  41. K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, M. Schaekermann, A. Wang, M. Amin, S. Lachgar, P. Mansfield, S. Prakash, B. Green, E. Dominowska, B. A. y Arcas, N. Tomasev, Y. Liu, R. Wong, C. Semturs, S. S. Mahdavi, J. Barral, D. Webster, G… find our card
  42. K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, J. Z. Chaves, S.-Y. Hu, M. Schaekermann, A. Kamath, Y. Cheng, D. G. T. Barrett, C. Cheung, B. Mustafa, A. Palepu, D. McDuff, L. Hou, T. Golany, L. Liu, J. baptiste Alayrac, N. Houlsby, N. Tomasev,… find
  43. Epic Systems Corporation, “Epic and microsoft bring gpt-4 to ehrs,” Epic, 2023. find
  44. D. Van Veen, C. Van Uden, L. Blankemeier, J.-B. Delbrouck, A. Aali, C. Bluethgen, A. Pareek, M. Polacin, E. P. Reis, A. Seehofnerová, et al., “Adapted large language models can outperform medical experts in clinical text summarization,” Nature medicine, vol. 30, no. 4, pp. 1134–1142, 2024. find
  45. Epic, “Epic and microsoft bring gpt-4 to ehrs,” 2023. find
  46. P. Garcia, S. P. Ma, S. Shah, M. Smith, Y. Jeong, A. Devon-Sand, M. Tai-Seale, K. Takazawa, D. Clutter, K. Vogt, C. Lugtu, M. Rojo, S. Lin, T. Shanafelt, M. A. Pfeffer, and C. Sharp, “Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages,” JAMA Network Open, vol. 7, pp. e243201–e… find
  47. OpenAI, “Paradigm: Improving patient access to clinical trials.” https://openai.com/index/paradigm/, 2024. Accessed: 2024-08-07. ↗
  48. M. Hutson, “How ai is being used to accelerate clinical trials,” Nature, vol. 627, pp. S2–S5, 2024. find
  49. OpenAI, “Using gpt-4o reasoning to transform cancer care.” https://openai.com/index/color-health/, 2024. Accessed: 2024-08-07. ↗
  50. J. Varghese and J.-L. Chapiro, “Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks,” Nature Communications, vol. 15, no. 1, p. 46411, 2024. Accessed: 2024-08-07. find
  51. E. Schmidt, “Ai will transform science.” https://www.technologyreview.com/2023/07/05/1075865/ eric-schmidt-ai-will-transform-science/, 2023. Accessed: 2024-08-07. ↗
  52. N. Rosenberg, “Science, invention and economic growth,” The Economic Journal, vol. 84, no. 333, pp. 90–108, 1974. find
  53. R. M. Atlas and M. Dando, “The dual-use dilemma for the life sciences: Perspectives, conundrums, and global solutions,” Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science, vol. 4, no. 3, pp. 276–286, 2006. PMID: 16999588. find
  54. X. Gu and M. Krenn, “Generation and human-expert evaluation of interesting research ideas using knowledge graphs and large language models,” 2024. find
  55. A. Ghafarollahi and M. J. Buehler, “Atomagents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence,” 2024. find
  56. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques, “Lab-bench: Measuring capabilities of language models for biology research,” 2024. find
  57. H. Cai, X. Cai, J. Chang, S. Li, L. Yao, C. Wang, Z. Gao, H. Wang, Y. Li, M. Lin, S. Yang, J. Wang, M. Xu, J. Huang, F. Xi, J. Zhuang, Y. Yin, Y. Li, C. Chen, Z. Cheng, Z. Zhao, L. Zhang, and G. Ke, “Sciassess: Benchmarking llm proficiency in scientific literature analysis,” 2024. find
  58. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the AI2 reasoning challenge,” CoRR, vol. abs/1803.05457, 2018. find
  59. S. Lin, J. Hilton, and O. Evans, “Truthfulqa: vol. abs/2109.07958, 2021. find our card