Elamur Research⌕
☰
home / papers / 2026 / BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks

BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks

Paper · 2026 · Anna Sokol, Inge Vejsbjerg, Elizabeth M. Daly et al.

Summary

The number of LLM benchmarks keeps growing, and information about each one is spread across papers, repositories and dataset cards that use different fields, terms and levels of detail. This makes it hard to pick a benchmark that fits a real use case. Earlier efforts such as BenchmarkCards, BenchHub and BetterBench document or grade individual benchmarks, but none gives a consistent way to compare candidates while searching.

The authors combine a practitioner survey, which builds on ten earlier interviews, with an analysis of metadata from several sources to find which attributes matter for discovery. They then build BenchNavigator, a prototype that:

  • maps existing metadata onto a shared schema with controlled vocabularies,
  • reconciles conflicting or missing values,
  • records the provenance of each value,
  • presents everything through one filterable interface.

The prototype shows that comparable presentation is achievable without asking benchmark creators to adopt new reporting standards. The authors frame the system as discovery infrastructure, not a quality score. They treat gaps in reporting, such as on contamination and validity, as an explicit finding.

Key points

  • The paper poses three research questions: current selection workflows and pain points, the attributes practitioners prioritize, and whether those attributes can be unified across heterogeneous sources in one interface.
  • The method has three layers: schema alignment with controlled vocabularies, normalization of conflicting or absent fields, and a uniform presentation layer that keeps provenance.
  • Benchmark growth is illustrated using ArXiv, GitHub and Hugging Face. The authors cite Liang et al.'s study of over 32,000 model cards, which found that voluntary documentation is often incomplete.
  • BenchNavigator does not rank a single "best" benchmark. It shows metadata and filters so users can choose based on their own context. Survey respondents reported little trust in leaderboards and vendor-reported scores.
  • Planned extensions include adding EvalCards and similar formats, and showing adoption trends such as paper mentions over time and saturation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓University of Notre DameNotre Dame · United States51 works · 36,717 citations🔬IBM Research - IrelandDublin · Ireland11 works · 904 citations🏢IBM (Ireland)Dublin · Ireland6 works · 274 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksInformation Systems and Management · 41 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Anna Sokol, Inge Vejsbjerg, Elizabeth M. Daly, David Piorkowski, Michael Hind, Nuno Moniz and Nitesh V. Chawla. 2026. BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.evaleval-1.29

@inproceedings{sokol2026benchnavigator,
  title = {{BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks}},
  author = {Anna Sokol, Inge Vejsbjerg, Elizabeth M. Daly, David Piorkowski, Michael Hind, Nuno Moniz and Nitesh V. Chawla},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.29},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.29}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. NIST AI. 2023. Artificial intelligence risk management framework (ai rmf 1.0). URL: https://nvlpubs.nist. gov/nistpubs/ai/nist. ai, pages 100–1. ↗
  2. Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, and 1 others. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of… find
  3. Frank Bagehorn, Kristina Brimijoin, Elizabeth M Daly, Jessica He, Michael Hind, Luis Garces-Erice, Christopher Giblin, Ioana Giurgiu, Jacquelyn Martino, Rahul Nair, and 1 others. 2025. Ai risk atlas: Taxonomy and tooling for navigating ai risks and resources. arXiv preprint arXiv:2503.05780. — [arXiv:2503.05780](https://arxiv.org/abs/2503.05780) find
  4. Vanessa Bracamonte, Sebastian Pape, Sascha Löbner, and Frederic Tronnier. 2023. Effectiveness and information quality perception of an ai model card: a study among non-experts. In 2023 20th Annual International Conference on Privacy, Security and Trust (PST), pages 1–7. IEEE. find
  5. Ryan Burnell, Wout Schellaert, John Burden, Tomer D. Ullman, Fernando Martinez-Plumed, Joshua B. Tenenbaum, Danaja Rutar, Lucy G. Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, Douwe Kiela, Murray Shanahan, Ellen M. Voorhees, Anthony G. Cohn, Joel Z. Leibo, and Jose Hernandez-Orallo. 2023. Rethink… find
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.
  7. Ruchira Dhar, Danae Sanchez Villegas, Antonia Karamolegkou, Alice Schiavone, Yifei Yuan, Xinyi Chen, Jiaang Li, Stella Frank, Laura De Grazia, Monorama Swain, and 1 others. 2025. Evalcards: A framework for standardized evaluation reporting. arXiv preprint arXiv:2511.21695. — [arXiv:2511.21695](https://arxiv.org/abs/2511.21695) find
  8. Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation. Preprint, arXiv:2502.06559. — [arXiv:2502.06559](https://arxiv.org/abs/2502.06559) find
  9. Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the nlp evaluation landscape. arXiv preprint arXiv:2101.04840. find
  10. Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M Asmar, Sanmi Koyejo, Michael S Bernstein, and Mykel John Kochenderfer. 2025. More than marketing? on the information value of ai benchmarks for practitioners. In Proceedings of the 30th International Conference on I… find
  11. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, and 1 others. 2021. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337. find
  12. Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel, Amit Agarwal, and Alice Oh. 2025. Benchhub: A unified benchmark suite for holistic and customizable llm evaluation. arXiv preprint arXiv:2506.00482. — [arXiv:2506.00482](https://arxiv.org/abs/2506.00482) find
  13. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  14. Weixin Liang, Nazneen Rajani, Xinyu Yang, Ezinwanne Ozoani, Eric Wu, Yiqun Chen, Daniel Scott Smith, and James Zou. 2024. Systematic analysis of 32,111 ai model cards characterizes documentation practice in ai. Nature Machine Intelligence, 6(7):744–753. find
  15. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. find
  16. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366. find
  17. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. 2024. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. Advances in Neural Information Processing Systems, 37:21763–21813. find
  18. Anna Sokol, Elizabeth Daly, Michael Hind, David Piorkowski, Xiangliang Zhang, Nuno Moniz, and Nitesh V. Chawla. 2024. Benchmarkcards: Standardized documentation for large language model benchmarks. ArXiv. find
  19. Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, and 1 others. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244.
  20. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. *Figure 1: Cumulative growth of language model benchmarks across arXiv, GitHub, and Hugging F… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sokol-2026-benchnavigator-discovery-interface-comparing-llm/