Elamur Research⌕
☰
home / papers / 2024 / Eureka: Evaluating and Understanding Large Foundation Models

Eureka: Evaluating and Understanding Large Foundation Models

Paper · 2024 · Vidhisha Balachandran, Jingya Chen, Neel Joshi et al.

Summary

Evaluating today's large foundation models has become unreliable. Many popular benchmarks are nearly maxed out, measurement setups are rarely disclosed, scoring generative outputs takes complex processing, and leaderboard numbers say little about how models fail. Microsoft Research addresses this with EUREKA, an open-source library for building modular evaluation pipelines covering preprocessing, prompt templates, inference, postprocessing, metrics and reporting. The authors release the code and the experiment logs to support reproducibility. With it they assemble EUREKA-BENCH, a set of language and vision benchmarks chosen because they remain hard for frontier systems and target basic skills that standard reports often skip, such as spatial and geometric reasoning. They test 12 leading models, breaking results down by data subcategory and experimental condition rather than reporting one score. They also study run-to-run non-determinism and backward compatibility between model versions. No model wins everywhere, though some lead more often than others. Shared weak spots remain. The work matters because it gives developers and researchers a transparent, extensible way to find specific failures and decide where improvement is needed.

Key points

  • A benchmark is included in EUREKA-BENCH if performance, overall or on a key condition, falls below 80% for all models or roughly half of them. By contrast, saturated suites such as MMLU, GSM8K, HumanEval, DROP and ChartQA show scores above 85%.
  • EUREKA-BENCH contents:
    • Multimodal: GeoMeter (1,086 prompts), MMMU (900), Image Understanding (10,240) and Vision Language Understanding (13,500).
    • Text: IFEval (541), FlenQA (12,000), Kitab (34,217) and Toxigen (10,500).
  • Evaluated models come from six families: Claude 3 Opus/3.5 Sonnet, Gemini 1.5 Pro, GPT-4 variants and GPT-4o, Llama 3/3.1 (70B, 405B), Llava 1.6 34B and Mistral Large 2407.
  • Common weaknesses:
    • fine-grained image understanding;
    • making real use of visual input instead of leaning on text;
    • factuality and grounding in information retrieval;
    • excessive refusals.
  • Several datasets (GeoMeter, Image Understanding, Vision Language Understanding, Kitab) can be procedurally regenerated, which reduces contamination concerns. The code is available at github.com/microsoft/eureka-ml-insights.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey and Safoora Yousefi. 2024. Eureka: Evaluating and Understanding Large Foundation Models. arXiv:2409.10566.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.10566 · arXiv 2409.10566

@misc{balachandran2024eureka,
  title = {{Eureka: Evaluating and Understanding Large Foundation Models}},
  author = {Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey and Safoora Yousefi},
  year = {2024},
  doi = {10.48550/arxiv.2409.10566},
  eprint = {2409.10566},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.10566}
}

References

136 references, 51 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet, 2024. Accessed: 2024-08-13. ↗
  2. Llava. https://huggingface.co/docs/transformers/en/model_doc/llava, 2024. ↗
  3. Mistral large 2. 2024-08-13. https://mistral.ai/news/mistral-large-2407/, 2024. Accessed: ↗
  4. Mmmu evaluation. https://github.com/MMMU-Benchmark/MMMU, 2024. ↗
  5. Openai evals. https://github.com/openai/evals, 2024. ↗
  6. M. I. Abdin, S. Gunasekar, V. Chandrasekaran, J. Li, M. Yüksekgönül, R. G. Peshawaria, R. Naik, and B. Nushi. KITAB: evaluating llms on constraint satisfaction for information retrieval. In International Conference on Learning Representations, 2024. find
  7. K. Ahuja, H. Diddee, R. Hada, M. Ochieng, K. Ramesh, P. Jain, A. U. Nambi, T. Ganu, S. Segal, M. Ahmed, K. Bali, and S. Sitaram. MEGA: multilingual evaluation of generative AI. In H. Bouamor, J. Pino, and K. Bali, editors, Conference on Empirical Methods in Natural Language Processing, pages 4232–42…
  8. S. Ahuja, D. Aggarwal, V. Gumma, I. Watts, A. Sathe, M. Ochieng, R. Hada, P. Jain, M. Axmed, K. Bali, et al. Megaverse: Benchmarking large language models across languages, modalities, models and tasks. arXiv preprint arXiv:2311.07463, 2023.
  9. S. Azad, Y. Jain, R. Garg, Y. Rawat, , and V. Vineet. Dh-bench: Probing depth and height perception of large visual-language models. arXiv preprint arXiv:2408.11748, 2024. — [arXiv:2408.11748](https://arxiv.org/abs/2408.11748) find
  10. G. Bansal, B. Nushi, E. Kamar, D. S. Weld, W. S. Lasecki, and E. Horvitz. Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff. In AAAI Conference on Artificial Intelligence, 2019. find
  11. S. Barocas, A. Guo, E. Kamar, J. Krones, M. R. Morris, J. W. Vaughan, W. D. Wadsworth, and H. Wallach. Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In AAAI/ACM Conference on AI, Ethics, and Society, pages 368–378, 2021. find
  12. M. S. Bernstein, J. Teevan, S. Dumais, D. Liebling, and E. Horvitz. Direct answers for search queries in the long tail. In SIGCHI conference on human factors in computing systems, pages 237–246, 2012. find
  13. M. Bertran, N. Martinez, A. Oesterling, and G. Sapiro. Distributionally robust group backwards compatibility. arXiv preprint arXiv:2112.10290, 2021. — [arXiv:2112.10290](https://arxiv.org/abs/2112.10290) find
  14. D. Bohus, S. Andrist, Y. Bao, E. Horvitz, and A. Paradiso. Is this it?: Towards ecologically valid benchmarks for situated collaboration. In International Conference on Multimodal Interaction (ICMI Companion ’24), 2024. find
  15. D. Bohus, S. Andrist, N. Saw, A. Paradiso, I. Chakraborty, and M. Rad. Sigma: An open-source interactive system for mixed-reality task assistance research–extended abstract. In IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 889–890. IEEE, 2024. find
  16. S. Bordt, H. Nori, V. Rodrigues, B. Nushi, and R. Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. In Conference on Language Modeling, 2024. find
  17. F. Brahman, S. Kumar, V. Balachandran, P. Dasigi, V. Pyatkin, A. Ravichander, S. Wiegreffe, N. Dziri, K. Chandu, J. Hessel, et al. The art of saying no: Contextual noncompliance in language models. arXiv preprint arXiv:2407.12043, 2024.
  18. Z. Buçinca, M. B. Malaya, and K. Z. Gajos. To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making. ACM on Human-computer Interaction, 5(CSCW1):1– 21, 2021. find
  19. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024.
  20. B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12168, 2024. find
  21. L. Chen, M. Zaharia, and J. Zou. How is chatgpt’s behavior changing over time? arXiv:2307.09009, 2023. arXiv preprint find
  22. L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation, 2017. find
  23. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  24. Y. Chen, Y. Ge, Y. Ge, M. Ding, B. Li, R. Wang, R. Xu, Y. Shan, and X. Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models. arXiv preprint arXiv:2312.06722, 2023. — [arXiv:2312.06722](https://arxiv.org/abs/2312.06722) find
  25. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
  26. J. Cho, A. Zala, and M. Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In IEEE/CVF International Conference on Computer Vision, pages 3043–3054, 2023. find
  27. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  28. J. Cunningham, S. L. Blodgett, M. Madaio, H. D. Iii, C. Harrington, and H. Wallach. Understanding the impacts of language technologies’ performance disparities on african american language speakers. In Findings of the Association for Computational Linguistics ACL, pages 12826–12833, 2024. find
  29. J. Dekoninck, M. N. Müller, and M. Vechev. Constat: Performance-based contamination detection in large language models. arXiv preprint arXiv:2405.16281, 2024. — [arXiv:2405.16281](https://arxiv.org/abs/2405.16281) find
  30. C. Deng, Y. Zhao, X. Tang, M. Gerstein, and A. Cohan. Investigating data contamination in modern benchmarks for large language models. arXiv preprint arXiv:2311.09783, 2023.
  31. Y. Deng, W. Zhang, S. J. Pan, and L. Bing. Multilingual jailbreak challenges in large language models. In International Conference on Learning Representations, 2024.
  32. A. Diwan, L. Berry, E. Choi, D. Harwath, and K. Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. arXiv preprint arXiv:2211.00768, 2022. find
  33. D. Downey, S. Dumais, and E. Horvitz. Heads and tails: studies of web search with common and rare queries. In ACM SIGIR conference on Research and development in information retrieval, pages 847– 848, 2007. find
  34. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  35. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  36. J. Echterhoff, F. Faghri, R. Vemulapalli, T.-Y. Hu, C.-L. Li, O. Tuzel, and H. Pouransari. Muscle: A model update strategy for compatible llm evolution. arXiv preprint arXiv:2407.09435, 2024. find
  37. A. E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. find
  38. S. Eyuboglu, M. Varma, K. K. Saab, J. Delbrouck, C. Lee-Messer, J. Dunnmon, J. Zou, and C. Ré. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations, 2022. find
  39. R. Fogliato, S. Chappidi, M. Lungren, P. Fisher, D. Wilson, M. Fitzke, M. Parkinson, E. Horvitz, K. Inkpen, and B. Nushi. Who goes first? influences of human-ai workflow on decision making in clinical imaging. In ACM Conference on Fairness, Accountability, and Transparency, pages 1362–1374, 2022. find
  40. C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024. ↗
  41. C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024.
  42. L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 2021. find
  43. S. Golchin and M. Surdeanu. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.06233, 2023. find
  44. T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Association for Computational Linguistics, pages 3309–3326, 2022.
  45. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
  46. T. Hosking, P. Blunsom, and M. Bartolo. Human feedback is not gold standard. In International Conference on Learning Representations, 2024. find
  47. E. Jahani, B. S. Manning, J. Zhang, H.-Y. TuYe, M. Alsobay, C. Nicolaides, S. Suri, and D. Holtz. As generative models improve, people adapt their prompts. arXiv preprint arXiv:2407.14333, 2024. — [arXiv:2407.14333](https://arxiv.org/abs/2407.14333) find
  48. B. Jayaraman, C. Guo, and K. Chaudhuri. D\\’ej\\a vu memorization in vision-language models. arXiv preprint arXiv:2402.02103, 2024. — [arXiv:2402.02103](https://arxiv.org/abs/2402.02103) find
  49. C. Jin, H. Peng, S. Zhao, Z. Wang, W. Xu, L. Han, J. Zhao, K. Zhong, S. Rajasekaran, and D. N. Metaxas. Apeer: Automatic prompt engineering enhances large language model reranking. arXiv preprint arXiv:2406.14449, 2024. — [arXiv:2406.14449](https://arxiv.org/abs/2406.14449) find
  50. J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  51. A. Kembhavi, M. Salvato, E. Kolve, M. J. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, The European Conference on Computer Vision, pages 235–251, 2016.
  52. R. Koo, M. Lee, V. Raheja, J. I. Park, Z. M. Kim, and D. Kang. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics ACL 2024, pages 517–545, 2024. find
  53. T. Lee, M. Yasunaga, C. Meng, Y. Mai, J. S. Park, A. Gupta, Y. Zhang, D. Narayanan, H. Teufel, M. Bellagente, M. Kang, T. Park, J. Leskovec, J.-Y. Zhu, F.-F. Li, J. Wu, S. Ermon, and P. S. Liang. Holistic evaluation of text-to-image models. In Advances in Neural Information Processing Systems, 2023. find
  54. M. Levy, A. Jacoby, and Y. Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024.
  55. C. Li, M. Chen, J. Wang, S. Sitaram, and X. Xie. Culturellm: Incorporating cultural differences into large language models. arXiv preprint arXiv:2402.10946, 2024.
  56. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024.
  57. T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen. Long-context llms struggle with long in-context learning, 2024.
  58. X. L. Li, E. Z. Liu, P. Liang, and T. Hashimoto. Autobencher: Creating salient, novel, difficult datasets for language models. arXiv preprint arXiv:2407.08351, 2024. find
  59. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. A. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L.… find
  60. Q. V. Liao, M. Vorvoreanu, H. Subramonyam, and L. Wilcox. Ux matters: The critical role of ux in responsible ai. Interactions, 31(4):22–27, 2024. find
  61. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Association for Computational Linguistics, pages 3214–3252, 2022.
  62. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014.
  63. F. Liu, G. Emerson, and N. Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. find
  64. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin. Mmbench: Is your multi-modal model an all-around player? ECCV, 2023.
  65. Y. L. Liu, S. L. Blodgett, J. C. K. Cheung, Q. V. Liao, A. Olteanu, and Z. Xiao. Ecbd: Evidence-centered benchmark design for nlp. arXiv preprint arXiv:2406.08723, 2024. find
  66. S. Luccioni, C. Akiki, M. Mitchell, and Y. Jernite. Stable bias: Evaluating societal representations in diffusion models. Advances in Neural Information Processing Systems, 36, 2024. find
  67. L. Lucy, S. L. Blodgett, M. Shokouhi, H. Wallach, and A. Olteanu. “one-size-fits-all”? examining expectations around what constitute “fair” or “good” nlg system behaviors. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages… find
  68. W. Ma, C. Yang, and C. Kästner. (why) is my prompt getting worse? rethinking regression testing for evolving llm apis. In IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pages 166–171, 2024. find
  69. K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, 2023.
  70. A. Masry, D. X. Long, J. Q. Tan, S. R. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics, pages 2263–2279, 2022.
  71. R. Matsuno and K. Sakuma. A robust backward compatibility metric for model retraining. In ACM International Conference on Information and Knowledge Management, pages 4190–4194, 2023. find
  72. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  73. M. Moayeri, E. Tabassi, and S. Feizi. Worldbench: Quantifying geographic disparities in llm factual recall. In ACM Conference on Fairness, Accountability, and Transparency, pages 1211–1228, 2024. find
  74. I. Momennejad, H. Hasanbeig, F. Vieira Frujeri, H. Sharma, N. Jojic, H. Palangi, R. Ness, and J. Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 2024. find
  75. R. Naik and B. Nushi. Social biases through the text-to-image generation lens. In AAAI/ACM Conference on AI, Ethics, and Society, pages 786–808, 2023. find
  76. H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz. Capabilities of gpt-4 on medical challenge problems, 2023.
  77. H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452, 2023. find
  78. B. Nushi, E. Kamar, and E. Horvitz. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. In AAAI Conference on Human Computation and Crowdsourcing, volume 6, pages 126–135, 2018. find
  79. B. Nushi, E. Kamar, E. Horvitz, and D. Kossmann. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In AAAI Conference on Artificial Intelligence, 2017. find
  80. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2024.
  81. Y. Oren, S. Sagawa, T. B. Hashimoto, and P. Liang. Distributionally robust language modeling. In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, pages 4226–4236. Association for Computa… find
  82. A. Parnami and M. Lee. Learning from few examples: A summary of approaches to few-shot learning. arXiv preprint arXiv:2203.04291, 2022. — [arXiv:2203.04291](https://arxiv.org/abs/2203.04291) find
  83. S. Passi and M. Vorvoreanu. Overreliance on ai literature review. Microsoft Research, 2022. find
  84. V. Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y. Yang, C. Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024. find
  85. Y. Perlitz, A. Gera, O. Arviv, A. Yehudai, E. Bandel, E. Shnarch, M. Shmueli-Scheuer, and L. Choshen. Benchmark agreement testing done right: A guide for llm benchmark evaluation. arXiv preprint arXiv:2407.13696, 2024. — [arXiv:2407.13696](https://arxiv.org/abs/2407.13696) find
  86. K. Ramesh, S. Sitaram, and M. Choudhury. Fairness in language models beyond english: Gaps and challenges. In A. Vlachos and I. Augenstein, editors, Association for Computational Linguistics, pages 2061–2074, 2023. find
  87. K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. D. Reid, and N. Sünderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In J. Tan, M. Toussaint, and K. Darvish, editors, Conference on Robot Learning, volume 229 of Machine Learning Research, pag… find
  88. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. find
  89. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  90. S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  91. M. Research. Image understanding benchmark. https://aka.ms/image-understanding-benchmark, 2024. ↗
  92. T. Schnabel and J. Neville. Prompts as programs: A structure-aware approach to efficient compile-time prompt optimization. arXiv preprint arXiv:2404.02319, 2024. — [arXiv:2404.02319](https://arxiv.org/abs/2404.02319) find
  93. P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac-Arnold, S. Maddineni, N. J. Joshi, P. Florence, W. Han, R. Baruch, Y. Lu, S. Mirchandani, P. Xu, P. Sanketi, K. Hausman, I. Shafran, B. Ichter, and Y. Cao. Robovqa: Multimodal long-horizon reasoning for robotics.… find
  94. P. Seshadri, S. Singh, and Y. Elazar. The bias amplification paradox in text-to-image generation. arXiv preprint arXiv:2308.00755, 2023. find
  95. U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, 2023.
  96. Y. Shen, Y. Xiong, W. Xia, and S. Soatto. Towards backward-compatible representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6368–6377, 2020. find
  97. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In International Conference on Learning Representations, 2023.
  98. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In International Conference on Learning Representations, 2023.
  99. T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano-Pérez, and L. P. Kaelbling. Pddl planning with pretrained large language models. In NeurIPS 2022 foundation models for decision making workshop, 2022. find
  100. S. Singla, B. Nushi, S. Shah, E. Kamar, and E. Horvitz. Understanding failures of deep networks via robust feature extraction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12853–12862, 2021. find
  101. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  102. M. Srivastava, B. Nushi, E. Kamar, S. Shah, and E. Horvitz. An empirical analysis of backward compatibility in machine learning systems. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3272–3280, 2020. find
  103. S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension, 2022. find
  104. T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022. find
  105. Y. Tian, A. Ravichander, L. Qin, R. L. Bras, R. Marjieh, N. Peng, Y. Choi, T. L. Griffiths, and F. Brahman. Macgyver: Are large language models creative problem solvers? arXiv preprint arXiv:2311.09682, 2023. find
  106. F. Träuble, J. Von Kügelgen, M. Kleindessner, F. Locatello, B. Schölkopf, and P. Gehler. Backward-compatible prediction updates: A probabilistic approach. Advances in Neural Information Processing Systems, 34:116–128, 2021. find
  107. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. — ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
  108. K. Valmeekam, M. Marquez, A. O. Hernandez, S. Sreedharan, and S. Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informatio…
  109. B. Vidgen, A. Agrawal, A. M. Ahmed, V. Akinwande, N. Al-Nuaimi, N. Alfaraj, E. Alhajjar, L. Aroyo, T. Bavalatti, B. Blili-Hamelin, et al. Introducing v0. 5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241, 2024.
  110. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537, 2019.
  111. B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In Advances in Neural Information Processing Systems, 2023.
  112. J. Wang, Y. Ming, Z. Shi, V. Vineet, X. Wang, and N. Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024. find
  113. P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics,… find
  114. S. Wang, Z. Long, Z. Fan, Z. Wei, and X. Huang. Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. arXiv preprint arXiv:2402.11443, 2024.
  115. Y. Wang, H. Li, X. Han, P. Nakov, and T. Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023.
  116. M. Wu and A. F. Aji. Style over substance: Evaluation biases for large language models. arXiv preprint arXiv:2307.03025, 2023.
  117. T. Xie, X. Qi, Y. Zeng, Y. Huang, U. M. Sehwag, K. Huang, L. He, B. Wei, D. Li, Y. Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598, 2024.
  118. S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, and I. Stoica. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850, 2023.
  119. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Conference on Empirical Methods in Natural Language Processing, pages 236…
  120. Y. Yao, A. Zhang, Z. Zhang, Z. Liu, T.-S. Chua, and M. Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models, 2022. find
  121. Z.-X. Yong, C. Menghini, and S. H. Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2023.
  122. X. Yuan, J. Li, D. Wang, Y. Chen, X. Mao, L. Huang, H. Xue, W. Wang, K. Ren, and J. Wang. S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models. arXiv preprint arXiv:2405.14191, 2024. — [arXiv:2405.14191](https://arxiv.org/abs/2405.14191) find
  123. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024.
  124. M. Yüksekgönül, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. In International Conference on Learning Representations, 2024. find
  125. A. K. Zhang, N. Perry, R. Dulepet, E. Jones, J. W. Lin, J. Ji, C. Menders, G. Hussein, S. Liu, D. Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risk of language models. arXiv preprint arXiv:2408.08926, 2024. find
  126. G. Zhang and M. Hardt. Inherent trade-offs between diversity and stability in multi-task benchmark. arXiv preprint arXiv:2405.01719, 2024. — [arXiv:2405.01719](https://arxiv.org/abs/2405.01719) find
  127. H. Zhang, J. Da, D. Lee, V. Robinson, C. Wu, W. Song, T. Zhao, P. Raja, D. Slack, Q. Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024.
  128. W. Zhang, M. Aljunied, C. Gao, Y. K. Chia, and L. Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484–5505, 2023.
  129. Y. Zhang, K. Zhou, and Z. Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022. — [arXiv:2206.04673](https://arxiv.org/abs/2206.04673) find
  130. H. S. Zheng, S. Mishra, H. Zhang, X. Chen, M. Chen, A. Nova, L. Hou, H.-T. Cheng, Q. V. Le, E. H. Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520, 2024. find
  131. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
  132. B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. find
  133. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
  134. K. Zhu, J. Chen, J. Wang, N. Z. Gong, D. Yang, and X. Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In International Conference on Learning Representations, 2024. find
  135. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015.
  136. Z. Zong, G. Song, and Y. Liu. Detrs with collaborative hybrid assignments training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6725–6735. IEEE Computer Society, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/balachandran-2024-eureka-evaluating-understanding-large-foundation/