Elamur Research⌕
☰
home / papers / 2024 / Large language models surpass human experts in predicting neuroscience results

Large language models surpass human experts in predicting neuroscience results

Paper · 2024 · Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun et al.

Summary

The paper asks whether large language models can forecast the outcomes of neuroscience experiments, a task that may exceed what human experts can do as the literature keeps growing. Existing science benchmarks such as MMLU, PubMedQA and MedMCQA mostly test recall of established facts. The authors instead built BrainBench, a forward-looking test based on recent Journal of Neuroscience abstracts. Each item pairs the original abstract with a version whose results were rewritten to say something different but still read coherently. Both human experts and LLMs must identify the real one.

Models choose the option with lower perplexity, and their confidence is measured by the perplexity gap between the two versions. LLMs outperformed human experts. BrainGPT, a LLaMa2-7B model adapted to neuroscience abstracts with LoRA, did better still. For both models and humans, higher confidence went with higher accuracy, which suggests humans and LLMs could work well as teams. The approach could transfer to other knowledge-heavy fields.

Key points

  • BrainBench spans five subfields: Behavioral/Cognitive, Systems/Circuits, Neurobiology of Disease, Cellular/Molecular and Developmental/Plasticity/Repair. Behavioral/Cognitive items are the most frequent, at about 38% of the GPT-4-generated cases.
  • Fifteen LLMs were evaluated, including the Galactica, Falcon, Llama-2 and Mistral families. They beat human experts in every subfield, and base models did better than their chat and instruct variants.
  • LLMs largely agreed with one another about which items were hard (mean Spearman 0.83 ± 0.06). Their agreement with human experts was much weaker (about 0.27 ± 0.05).
  • BrainGPT was built with LoRA, a parameter-efficient method that freezes the pretrained LLaMa2 weights and trains only small low-rank matrices.
  • Confidence was well calibrated for both humans and all LLMs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom395 works · 81,691 citations🎓University of PennsylvaniaPhiladelphia · United States241 works · 66,904 citations🎓New York UniversityNew York · United States233 works · 104,003 citations🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations🎓Massachusetts Institute of TechnologyCambridge · United States274 works · 82,013 citations🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations🎓University of OxfordOxford · United Kingdom138 works · 161,448 citations🎓Georgia Institute of TechnologyAtlanta · United States159 works · 28,096 citations🎓Brown UniversityProvidence · United States85 works · 26,322 citations🎓University College LondonLondon · United Kingdom119 works · 23,545 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany116 works · 57,070 citations🎓Yale UniversityNew Haven · United States112 works · 14,009 citations🎓University of ZurichZurich · Switzerland83 works · 12,449 citations🎓University of TübingenTübingen · Germany81 works · 41,480 citations🎓University of GenevaGeneva · Switzerland30 works · 3,317 citations🎓Technical University of MunichMunich · Germany66 works · 49,493 citations🎓Emory UniversityAtlanta · United States43 works · 40,058 citations🔬The Alan Turing InstituteLondon · United Kingdom35 works · 5,111 citations🎓Ghent UniversityGhent · Belgium26 works · 3,108 citations🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany36 works · 1,654 citations🎓Johannes Gutenberg University MainzMainz · Germany8 works · 837 citations🎓Osnabrück UniversityOsnabrück · Germany8 works · 409 citations🎓University of BristolBristol · United Kingdom20 works · 985 citations🎓Bilkent UniversityAnkara · Türkiye6 works · 649 citations🔬Forschungszentrum JülichJülich · Germany6 works · 1,460 citations🎓University of St AndrewsSt Andrews · United Kingdom6 works · 200 citations🎓University of NottinghamNottingham · United Kingdom5 works · 185 citations🎓Aarhus UniversityAarhus · Denmark8 works · 171 citations🎓Novosibirsk State Technical UniversityNovosibirsk · Russia2 works · 160 citations🎓University of Newcastle AustraliaNewcastle · Australia3 works · 287 citations🎓University of BaselBasel · Switzerland3 works · 230 citations🏥University Medical Center of the Johannes Gutenberg University MainzMainz · Germany2 works · 160 citations🔬National Institute of Mental HealthBethesda · United States2 works · 160 citations

Also on the byline: Instituto Cajal · Novosibirsk State Medical University · Centre Hospitalier Universitaire Vaudois · Institut Universitaire de Gériatrie de Montréal · Max Planck Institute for Human Cognitive and Brain Sciences · Federal Neurosurgical Center Novosibirsk · Max Planck Institute for Neurobiology of Behavior – caesar.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksBiochemistry, Genetics and Molecular Biology · 273 worksMolecular Biology · 270 worksMedicine · 288 worksHealth Informatics · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello, Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N Apurva Ratan Murty, Leyla Loued-Khenissi, Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao and Bradley C. Love. 2024. Large language models surpass human experts in predicting neuroscience results. arXiv:2403.03230.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41562-024-02046-9 · arXiv 2403.03230

@misc{luo2024large,
  title = {{Large language models surpass human experts in predicting neuroscience results}},
  author = {Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello, Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N Apurva Ratan Murty, Leyla Loued-Khenissi, Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao and Bradley C. Love},
  year = {2024},
  doi = {10.1038/s41562-024-02046-9},
  eprint = {2403.03230},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41562-024-02046-9}
}

References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Bornmann, R. Mutz, Journal of the Association for Information Science and Technology 66, 2215 (2015). find
  2. J. S. G. Chu, J. A. Evans, Proceedings of the National Academy of Sciences 118, e2021636118 (2021). find
  3. K. Tunyasuvunakool, et al., Nature 596, 590 (2021). find
  4. A. Zhavoronkov, et al., Nature Biotechnology 37, 1038 (2019). find
  5. V. Tshitoyan, et al., Nature 571, 95 (2019). find
  6. R. M. Mok, B. C. Love, Science Advances 9, eade6903 (2023). find
  7. R. Botvinik-Nezer, et al., Nature 582, 84 (2020). Publisher: Nature Publishing Group. find
  8. Y. Liu, et al. (2023). Publisher: arXiv Version Number: 3. find
  9. A. Vaswani, et al. (2017). Publisher: arXiv Version Number: 5. find
  10. W. Fedus, B. Zoph, N. Shazeer (2021). Publisher: arXiv Version Number: 3. find
  11. A. Srivastava, et al. (2022). Publisher: arXiv Version Number: 3. find
  12. S. Gunasekar, et al. (2023). Publisher: arXiv Version Number: 1. find
  13. R. Strack, Nature Methods 20, 1868 (2023). find
  14. D. Hendrycks, et al., Measuring Massive Multitask Language Understanding (2021).
  15. ArXiv:2009.03300 [cs]. Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, X. Lu, PubMedQA: A Dataset for Biomedical Research Question Answering (2019). ArXiv:1909.06146 [cs, q-bio].
  16. A. Pal, L. K. Umapathi, M. Sankarasubbu, MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering (2022). ArXiv:2203.14371 [cs].
  17. P. Lewis, et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2021). ArXiv:2005.11401 [cs].
  18. N. Carlini, et al., Extracting Training Data from Large Language Models (2021).
  19. ArXiv:2012.07805 [cs]. E. J. Hu, et al., LoRA: Low-Rank Adaptation of Large Language Models (2021).
  20. ArXiv:2106.09685 [cs]. R. Taylor, et al., Galactica: A Large Language Model for Science (2022). ArXiv:2211.09085
  21. [cs, stat]. M. Steyvers, H. Tejeda, G. Kerrigan, P. Smyth, Proceedings of the National Academy of Sciences 119, e2111547119 (2022). find
  22. L. Zheng, et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023).
  23. ArXiv:2306.05685 [cs]. J. Hu, R. Levy, Prompting is not a substitute for probability measurements in large language models (2023). ArXiv:2305.13264 [cs].
  24. A. Azaria, T. Mitchell, The Internal State of an LLM Knows When It’s Lying (2023). — The Internal State of an LLM Knows When It's Lying
  25. ArXiv:2304.13734 [cs]. L. Gao, et al., A framework for few-shot language model evaluation (2023). Version Number: v0.4.0.
  26. J. Wei, et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2023). ArXiv:2201.11903 [cs].
  27. M. Nasr, et al., Scalable Extraction of Training Data from (Production) Language Models (2023). ArXiv:2311.17035 [cs].
  28. Gailly, Adler, zlib compression library. (2024). find
  29. A. L. Anwyl-Irvine, J. Massonnié, A. Flitton, N. Kirkham, J. K. Evershed, Behavior Research Methods 52, 388 (2020). find
  30. A. Q. Jiang, et al., Mistral 7B (2023). ArXiv:2310.06825 [cs]. find
  31. I. Loshchilov, F. Hutter, Decoupled Weight Decay Regularization (2019). ArXiv:1711.05101 find
  32. [cs, math]. X. Luo, G. Sun, B. C. Love, Matching domain experts by training from scratch on domain knowledge (2024). ArXiv:2405.09395 [cs, q-bio]. — [arXiv:2405.09395](https://arxiv.org/abs/2405.09395) find
  33. E. Almazrouei, et al., The Falcon Series of Open Language Models (2023).
  34. H. Touvron, et al., LLaMA: Open and Efficient Foundation Language Models (2023).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2024-large-language-models-surpass/