Elamur Research⌕
☰
home / papers / 2022 / Mapping global dynamics of benchmark creation and saturation in artificial intelligence

Mapping global dynamics of benchmark creation and saturation in artificial intelligence

Paper · 2022 · Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec et al.

Summary

Benchmarks not only measure progress in AI but also set its direction. At the same time, many of them quickly reach saturation or are barely used. The authors proposed a way to concisely map the emergence, use, and saturation of benchmarks across entire fields. To do this, they took a Papers With Code dump and built from it the ontology and knowledge graph ITO (Intelligence Task Ontology) in OWL format. They manually curated the hierarchy of tasks and metrics. The analysis covered 3765 benchmarks for 947 computer vision and NLP tasks. To make different metrics comparable, the authors normalized SOTA trajectories via relative improvement, calculated from the first result to the best one. The shapes of the curves were clustered using self-organizing maps (SOM). It turned out that a substantial share of benchmarks approach saturation very early. Many of them are hardly used by anyone, and sharp jumps in performance on individual tasks occur unpredictably. The practical conclusion: a new benchmark, including one for the Ukrainian language, has a better chance of being adopted if it is versatile, broad in coverage, and useful for real-world applications.

Key points

  • Data volume. 3765 benchmarks and 947 tasks were analyzed. Of the 1318 NLP benchmarks that have at least one result, only 661 (50%) have results at three or more distinct points in time.
  • Three curve types. SOM grouped 1079 metric trajectories from 654 benchmarks into three clusters: steady growth (460), early saturation or stagnation (378), and stagnation followed by a late breakthrough (241).
  • Dynamics over time. The number of benchmarks began to grow in 2013. It increased fastest in 2017–2018, after which growth slowed, and in 2020 it stagnated, most noticeably in NLP. The number of active benchmarks peaked in 2020: 432 in NLP versus 1100 in computer vision.
  • Popularity. The most popular and least used datasets were compared using one-sided t-tests. According to the authors, popularity is associated with versatility, breadth of coverage, and practical usefulness.
  • Open resources. The ITO knowledge graph is published on Zenodo (doi:10.5281/zenodo.7097305). The code and the interactive "Global Map of AI Benchmark Dynamics" are available in the GitHub repository OpenBioLink/ITO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Medical University of ViennaVienna · Austria3 works · 161 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksDecision Sciences · 209 worksStatistics, Probability and Uncertainty · 61 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Artificial intelligence · 49 works

Cite

Cite as Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner and Matthias Samwald. 2022. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. arXiv:2203.04592 · Nature Communications volume 13, Article number: 6793 (2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41467-022-34591-0 · arXiv 2203.04592

@misc{ott2022mapping,
  title = {{Mapping global dynamics of benchmark creation and saturation in artificial intelligence}},
  author = {Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner and Matthias Samwald},
  year = {2022},
  doi = {10.1038/s41467-022-34591-0},
  eprint = {2203.04592},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41467-022-34591-0}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Deng, J. et al. ImageNet: A large-scale hierarchical image database. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009). doi:10.1109/CVPR.2009.5206848. find
  2. Soomro, K., Zamir, A. R. & Shah, M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv (2012). find
  3. Borgwardt, K. M. et al. Protein function prediction via graph kernels. Bioinformatics 21 Suppl 1, i47-56 (2005). find
  4. Kiela, D. et al. Dynabench: rethinking benchmarking in NLP. in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 4110–4124 (Association for Computational Linguistics, 2021). doi:10.18653/v1/2021.naacl-main.3… find
  5. Bowman, S. R. & Dahl, G. What will it take to fix benchmarking in natural language understanding? in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 4843–4855 (Association for Computational Linguistics, 20… find
  6. Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020). find
  7. Krizhevsky, A. Learning multiple layers of features from tiny images. (2009). find
  8. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D. & Parikh, D. Making the V in VQA matter: elevating the role of image understanding in visual question answering. in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 6325–6334 (IEEE, 2017). doi:10.1109/CVPR.2017.670.
  9. Zhang, D. et al. The AI Index 2021 Annual Report. Preprint at arXiv: https://arxiv.org/abs/2103.06312 (2021). — [arXiv:2103.06312](https://arxiv.org/abs/2103.06312) ↗
  10. Martínez-Plumed, F., Barredo, P., hÉigeartaigh, S. Ó. & Hernández-Orallo, J. Research community dynamics behind popular AI benchmarks. Nat. Mach. Intell. (2021) doi:10.1038/s42256-021-00339-6. — [doi:10.1038/s42256-021-00339-6](https://doi.org/10.1038/s42256-021-00339-6) find
  11. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250 [cs] (2016).
  12. Koch, B., Denton, E., Hanna, A. & Foster, J. G. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research. (2021). \#\# 13. Dehghani, M. et al. The Benchmark Lottery. arXiv (2021). \#\# 14. Nie, Y. et al. Adversarial NLI: A new benchmark for natural language understanding. in Proc…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ott-2022-mapping-global-dynamics-benchmark-creation/