📏 Evaluation & benchmarks

684 papers · page 2 of 8 · 26–125 · the most cited first · all topics

How systems are measured: benchmarks, metrics, human judgements and what the numbers mean.

📄 Papers (684)

← prev 1 2 3 … 8 next →

26. One Billion Word Benchmark for Measuring Progress in Statistical Language ModelingInterspeechCiprian Chelba, Tomas Mikolov, Mike Schuster, et al.2013 · 949 citations
27. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIIEEE Computer Society Conference on Computer Vision and Pattern RecognitionXiang Yue, Yuansheng Ni, Kai Zhang, et al.2023 · 948 citations
28. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over ParagraphsDheeru Dua, Yizhong Wang, Pradeep Dasigi, et al.2019 · 943 citations
29. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram StatisticsAnnual Meeting of the Association for Computational Linguistics (ACL)Chin-Yew Lin, Franz Josef Och2004 · 940 citations
30. Findings of the 2016 Conference on Machine TranslationConference on Machine Translation: Shared Task PapersOndřej Bojar, Rajen Chatterjee, Christian Federmann, et al.2016 · 938 citations
31. Findings of the 2014 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Ondřej Bojar, Christian Buck, Christian Federmann, et al.2014 · 929 citations
32. Collecting Highly Parallel Data for Paraphrase EvaluationAnnual Meeting of the Association for Computational Linguistics (ACL)David Chen, William Dolan2011 · 929 citations
33. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and InteractivityarXiv:2302.04023Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, et al.2023 · 876 citations
34. On Coreference Resolution Performance MetricsXiaoqiang Luo2005 · 855 citations
35. Findings of the 2019 Conference on Machine Translation (WMT19)Conference on Machine Translation (WMT)Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, et al.2019 · 841 citations
36. Automatic Evaluation of Topic CoherenceConference of the North American Chapter of the Association for Computational Linguistics (NAACL)David Newman, Jey Han Lau, Karl Grieser, et al.2010 · 825 citations
37. ERASER: A Benchmark to Evaluate Rationalized NLP ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, et al.2020 · 821 citations
38. Findings of the 2018 Conference on Machine Translation (WMT18)Conference on Machine Translation: Shared Task PapersOndřej Bojar, Christian Federmann, Mark Fishel, et al.2018 · 750 citations
39. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningFindings of the Association for Computational LinguisticsAhmed Masry, Do Xuan Long, Jia Qing Tan, et al.2022 · 745 citations
40. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning StepsInternational Conference on Computational Linguistics (COLING)Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, et al.2020 · 735 citations
41. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding BenchmarkConference on Neural Information Processing Systems (NeurIPS)Yubo Wang, Xueguang Ma, Ge Zhang, et al.2024 · 731 citations
42. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian LanguagesFindings of the Association for Computational LinguisticsDivyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, et al.2020 · 730 citations
43. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday TasksIEEE Computer Society Conference on Computer Vision and Pattern RecognitionMohit Shridhar; Jesse Thomason; Daniel Gordon; Yonatan Bisk; Winson Han; Roozbeh Mottaghi; Luke Zettlemoyer;, Dieter Fox2019 · 726 citations
44. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesTransactions of the Association for Computational Linguistics (TACL)Jonathan H. Clark, Eunsol Choi, Michael Collins, et al.2020 · 692 citations
45. Findings of the 2017 Conference on Machine Translation (WMT17)Conference on Machine Translation (WMT)Ondřej Bojar, Rajen Chatterjee, Christian Federmann, et al.2017 · 689 citations
46. Why We Need New Evaluation Metrics for NLGConference on Empirical Methods in Natural Language Processing (EMNLP)Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, et al.2017 · 688 citations
47. Evaluation methods for unsupervised word embeddingsConference on Empirical Methods in Natural Language Processing (EMNLP)Tobias Schnabel, Igor Labutov, David Mimno, et al.2015 · 674 citations
48. KILT: a Benchmark for Knowledge Intensive Language TasksConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Fabio Petroni, Aleksandra Piktus, Angela Fan, et al.2020 · 665 citations
49. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationMarkus Freitag, George Foster, David Grangier, et al.2021 · 660 citations
50. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, et al.2023 · 647 citations
51. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationConference on Neural Information Processing Systems (NeurIPS)Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, et al.2023 · 632 citations
52. A Critique and Improvement of an Evaluation Metric for Text SegmentationComputational LinguisticsLev Pevzner, Marti A. Hearst2002 · 595 citations
53. Evaluating Gender Bias in Machine TranslationAnnual Meeting of the Association for Computational Linguistics (ACL)Gabriel Stanovsky, Noah A. Smith, Luke Zettlemoyer2019 · 589 citations
54. Instruction-Following Evaluation for Large Language ModelsarXiv:2311.07911Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, et al.2023 · 586 citations
55. Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style TransferSudha Rao, Joel Tetreault2018 · 582 citations
56. AGIEval: A Human-Centric Benchmark for Evaluating Foundation ModelsFindings of the Association for Computational LinguisticsWanjun Zhong, Ruixiang Cui, Yiduo Guo, et al.2024 · 564 citations
57. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and DetectionarXiv:2301.07597Biyang Guo, Xin Zhang, Ziyuan Wang, et al.2023 · 553 citations
58. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralizationarXiv:2003.11080Junjie Hu, Sebastian Ruder, Aditya Siddhant, et al.2020 · 552 citations
59. Edinburgh System Description for the 2005 IWSLT Speech Translation EvaluationInternational Conference on Spoken Language Translation (IWSLT)Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, et al.2005 · 545 citations
60. A Structured Review of the Validity of BLEUEhud Reiter2018 · 541 citations
61. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation SystemsConference on Machine Translation (WMT)Michael Denkowski, Alon Lavie2011 · 541 citations
62. Automatic Annotation and Evaluation of Error Types for Grammatical Error CorrectionChristopher Bryant, Mariano Felice, Ted Briscoe2017 · 534 citations
63. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language ModelsConference on Neural Information Processing Systems (NeurIPS)Chaoyou Fu, Peixian Chen, Yunhang Shen, et al.2023 · 529 citations
64. Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmov, et al.2020 · 522 citations
65. LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingarXiv:2308.14508Yushi Bai, Xin Lv, Jiajie Zhang, et al.2023 · 506 citations
66. A-OKVQA: A Benchmark for Visual Question Answering using World KnowledgeLecture notes in computer scienceDustin Schwenk, Apoorv Khandelwal, Christopher Clark, et al.2022 · 490 citations
67. BBQ: A hand-built bias benchmark for question answeringFindings of the Association for Computational LinguisticsAlicia Parrish, Angelica Chen, Nikita Nangia, et al.2022 · 488 citations
68. Evaluating Large Language Models in Theory of Mind TasksProceedings of the National Academy of SciencesMichal Kosinski2023 · 467 citations
69. Findings of the 2015 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Ondřej Bojar, Rajen Chatterjee, Christian Federmann, et al.2015 · 462 citations
70. Text-to-SQL Empowered by Large Language Models: A Benchmark EvaluationarXiv:2308.15363Dawei Gao, Haibin Wang, Yaliang Li, et al.2023 · 461 citations
71. Automatic Evaluation of Translation Quality for Distant Language PairsConference on Empirical Methods in Natural Language Processing (EMNLP)Hideki Isozaki, Tsutomu Hirao, Kevin Duh, et al.2010 · 461 citations
72. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific ProblemsAnnual Meeting of the Association for Computational Linguistics (ACL)Chaoqun He, Renjie Luo, Yuzhuo Bai, et al.2024 · 448 citations
73. Has Machine Translation Achieved Human Parity? A Case for Document-level EvaluationConference on Empirical Methods in Natural Language Processing (EMNLP)Samuel Läubli, Rico Sennrich, Martin Volk2018 · 443 citations
74. Findings of the 2013 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Ondřej Bojar, Christian Buck, Chris Callison-Burch, et al.2013 · 443 citations
75. MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkIEEE Computer Society Conference on Computer Vision and Pattern RecognitionKunchang Li, Yali Wang, Yinan He, et al.2023 · 442 citations
76. Neural versus Phrase-Based Machine Translation Quality: a Case StudyConference on Empirical Methods in Natural Language Processing (EMNLP)Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, et al.2016 · 437 citations
77. Manual and Automatic Evaluation of Machine Translation between European LanguagesConference on Machine Translation (WMT)Philipp Koehn, Christof Monz2006 · 434 citations
78. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared TaskConference on Machine Translation (WMT)Ricardo Rei, José G. C. de Souza, Duarte Alves, et al.2022 · 433 citations
79. How we BLESSed distributional semantic evaluationGEMS Workshop on GEometrical Models of Natural Language SemanticsMarco Baroni, Alessandro Lenci2011 · 429 citations
80. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality MetricsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Artidoro Pagnoni, Vidhisha Balachandran, Yulia Tsvetkov2021 · 418 citations
81. Lattice-based Minimum Error Rate Training for Statistical Machine TranslationConference on Empirical Methods in Natural Language Processing (EMNLP)Wolfgang Macherey, Franz Och, Ignacio Thayer, et al.2008 · 410 citations
82. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video AnalysisIEEE Computer Society Conference on Computer Vision and Pattern RecognitionChaoyou Fu, Yuhan Dai, Yongdong Luo, et al.2024 · 404 citations
83. Findings of the 2020 Conference on Machine Translation (WMT20)Conference on Machine Translation (WMT)Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, et al.2020 · 401 citations
84. QMSum: A New Benchmark for Query-based Multi-domain Meeting SummarizationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Ming Zhong, Da Yin, Tao Yu, et al.2021 · 400 citations
85. Long Range Arena: A Benchmark for Efficient TransformersarXiv:2011.04006Yi Tay, Mostafa Dehghani, Samira Abnar, et al.2020 · 399 citations
86. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in FinanceAnnual Meeting of the Association for Computational Linguistics (ACL)Fengbin Zhu, Wenqiang Lei, Youcheng Huang, et al.2021 · 397 citations
87. Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language TasksConference on Empirical Methods in Natural Language Processing (EMNLP)Rion Snow, Brendan O'Connor, Daniel Jurafsky, et al.2008 · 392 citations
88. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human JudgmentsConference on Machine Translation (WMT)Alon Lavie, Abhaya Agarwal2007 · 389 citations
89. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation MetricsNitika Mathur, Timothy Baldwin, Trevor Cohn2020 · 386 citations
90. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language ModelsarXiv:2308.11462Neel Guha; Julian Nyarko; Daniel E. Ho; Christopher Re; Adam Chilton; Aditya Narayana; Alex Chohlas-Wood; Austin Peters; Brandon Waldon; Daniel N. Rockmore; Diego Zambrano; Dmitry Talisman; Enam Hoque; Faiz Surani; Frank Fagan; Galit Sarfaty; Gregory M. Dickinson; Haggai Porat; Jason Hegland; Jessica Wu; Joe Nudell; Joel Niklaus; John Nay; Jonathan H. Choi; Kevin Tobia; Margaret Hagan; Megan Ma; Michael Livermore; Nikon Rasumov-Rahe; Nils Holzenberger; Noam Kolt; Peter Henderson; Sean Rehaag; Sharad Goel; Shang Gao; Spencer Williams; Sunny Gandhi; Tom Zur; Varun Iyer;, Zehua Li2023 · 381 citations
91. An Evaluation Tool for Machine Translation: Fast Evaluation for MT ResearchLanguage Resources and Evaluation Conference (LREC)Sonja Nießen, Franz Josef Och, Gregor Leusch, et al.2000 · 368 citations
92. Latent Hatred: A Benchmark for Understanding Implicit Hate SpeechConference on Empirical Methods in Natural Language Processing (EMNLP)Mai ElSherief, Caleb Ziems, David Muchlinski, et al.2021 · 365 citations
93. The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–EnglishConference on Empirical Methods in Natural Language Processing (EMNLP)Francisco Guzmán, Peng-Jen Chen, Myle Ott, et al.2019 · 363 citations
94. How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular BenchmarksDivyansh Kaushik, Zachary C. Lipton2018 · 361 citations
95. XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and GenerationConference on Empirical Methods in Natural Language Processing (EMNLP)Yaobo Liang, Nan Duan, Yeyun Gong, et al.2020 · 356 citations
96. Improving Text-to-SQL Evaluation MethodologyAnnual Meeting of the Association for Computational Linguistics (ACL)Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, et al.2018 · 350 citations
97. Problems With Evaluation of Word Embeddings Using Word Similarity TasksWorkshop on Evaluating Vector Space Representations for NLPManaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, et al.2016 · 334 citations
98. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error DetectionTransactions of the Association for Computational Linguistics (TACL)Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, et al.2024 · 333 citations
99. An Evaluation Exercise for Word AlignmentProceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and BeyondRada Mihalcea, Ted Pedersen2003 · 326 citations
100. Algorithms for Scoring Coreference ChainsAmit Bagga, Breck Baldwin1998 · 325 citations
101. Paraphrasing for Automatic EvaluationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)David Kauchak, Regina Barzilay2006 · 323 citations
102. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine TranslationConference on Machine Translation: Research PapersAntonio Toral, Sheila Castilho, Ke Hu, et al.2018 · 321 citations
103. On Some Pitfalls in Automatic Evaluation and Significance Testing for MTACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or SummarizationStefan Riezler, John T. Maxwell2005 · 318 citations
104. Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI)Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final WorkshopElisabetta Fersini, Debora Nozza, Paolo Rosso2018 · 315 citations
105. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Annual Meeting of the Association for Computational Linguistics (ACL)Peter Hase, Mohit Bansal2020 · 309 citations
106. Discovering Language Model Behaviors with Model-Written EvaluationsarXiv:2212.09251Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, et al.2022 · 307 citations
107. An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense DisambiguationConference on Empirical Methods in Natural Language Processing (EMNLP)Yoong Keok Lee, Hwee Tou Ng2002 · 303 citations
108. Improving Translation Quality by Discarding Most of the PhrasetableConference on Empirical Methods in Natural Language Processing (EMNLP)Howard Johnson, Joel Martin, George Foster, et al.2007 · 300 citations
109. BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in BanglaFindings of the Association for Computational LinguisticsAbhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, et al.2022 · 296 citations
110. Re-evaluating the Role of Bleu in Machine Translation ResearchConference of the European Chapter of the Association for Computational Linguistics (EACL)Chris Callison-Burch, Miles Osborne, Philipp Koehn2006 · 295 citations
111. Evaluating Verifiability in Generative Search EnginesFindings of the Association for Computational LinguisticsNelson Liu, Tianyi Zhang, Percy Liang2023 · 294 citations
112. A Benchmark Dataset for Learning to Intervene in Online Hate SpeechConference on Empirical Methods in Natural Language Processing (EMNLP)Jing Qian, Anna Bethke, Yinyin Liu, et al.2019 · 292 citations
113. Diverse Few-Shot Text Classification with Multiple MetricsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Mo Yu, Xiaoxiao Guo, Jinfeng Yi, et al.2018 · 291 citations
114. USR: An Unsupervised and Reference Free Evaluation Metric for Dialog GenerationAnnual Meeting of the Association for Computational Linguistics (ACL)Shikib Mehri, Maxine Eskenazi2020 · 284 citations
115. Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big ChallengesConference on Machine Translation (WMT)Qingsong Ma, Johnny Wei, Ondřej Bojar, et al.2019 · 284 citations
116. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMsConference on Empirical Methods in Natural Language Processing (EMNLP)Minghao Li, Yingxiu Zhao, Bowen Yu, et al.2023 · 282 citations
117. A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEUConference on Machine Translation (WMT)Boxing Chen, Colin Cherry2014 · 282 citations
118. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language ModelsConference on Neural Information Processing Systems (NeurIPS)Patrick Chao, Edoardo Debenedetti, Alexander Robey, et al.2024 · 273 citations
119. WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive SummarizationFindings of the Association for Computational LinguisticsFaisal Ladhak, Esin Durmus, Claire Cardie, et al.2020 · 273 citations
120. Re-examining Machine Translation Metrics for Paraphrase IdentificationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Nitin Madnani, Joel Tetreault, Martin Chodorow2012 · 272 citations
121. A Challenge Set Approach to Evaluating Machine TranslationPierre Isabelle, Colin Cherry, George Foster2017 · 269 citations
122. Prometheus: Inducing Fine-grained Evaluation Capability in Language ModelsarXiv:2310.08491Seungone Kim, Jamin Shin, Yejin Cho, et al.2023 · 267 citations
123. Handling Divergent Reference Texts when Evaluating Table-to-Text GenerationAnnual Meeting of the Association for Computational Linguistics (ACL)Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, et al.2019 · 267 citations
124. IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language UnderstandingConference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL)Bryan Wilie, Karissa Vincentio, Genta Indra Winata, et al.2020 · 265 citations
125. Evaluating the Underlying Gender Bias in Contextualized Word EmbeddingsWorkshop on Gender Bias in Natural Language ProcessingChristine Basta, Marta R. Costa-jussà, Noe Casas2019 · 265 citations

← prev 1 2 3 … 8 next →

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/2/