Elamur Research⌕
☰
home / topics / Evaluation & benchmarks / page 3

📏 Evaluation & benchmarks

684 papers · page 3 of 8 · 126–225 · the most cited first · all topics

How systems are measured: benchmarks, metrics, human judgements and what the numbers mean.

 Papers (684)

1 2 3 4 … 8

  1. Are We on the Right Way for Evaluating Large Vision-Language Models?Conference on Neural Information Processing Systems (NeurIPS)Lin Chen, Jinsong Li, Xiaoyi Dong, et al.2024 · 263 citations
  2. Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More RobustMarkus Freitag, Ricardo Rei, Nitika Mathur, et al.2022 · 263 citations
  3. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval ModelsNeurIPS 2021 Datasets and BenchmarksNandan Thakur, Nils Reimers, Andreas Rücklé, et al.2021 · 262 citations
  4. Findings of the VarDial Evaluation Campaign 2017Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, et al.2017 · 261 citations
  5. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language VariantsAnnual Meeting of the Association for Computational Linguistics (ACL)Lucas Bandarkar, Davis Liang, Benjamin Muller, et al.2023 · 259 citations
  6. Automatic Machine Translation Evaluation in Many Languages via Zero-Shot ParaphrasingConference on Empirical Methods in Natural Language Processing (EMNLP)Brian Thompson, Matt Post2020 · 259 citations
  7. How Can We Accelerate Progress Towards Human-like Linguistic Generalization?Annual Meeting of the Association for Computational Linguistics (ACL)Tal Linzen2020 · 258 citations
  8. Re-evaluating Evaluation in Text SummarizationConference on Empirical Methods in Natural Language Processing (EMNLP)Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, et al.2020 · 258 citations
  9. MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicsarXiv:2109.00110Kunhao Zheng, Jesse Michael Han, Stanislas Polu2021 · 255 citations
  10. BLEU is Not Suitable for the Evaluation of Text SimplificationConference on Empirical Methods in Natural Language Processing (EMNLP)Elior Sulem, Omri Abend, Ari Rappoport2018 · 255 citations
  11. LexGLUE: A Benchmark Dataset for Legal Language Understanding in EnglisharXiv:2110.00976Ilias Chalkidis; Abhik Jana; Dirk Hartung; Michael Bommarito; Ion Androutsopoulos; Daniel Martin Katz;, Nikolaos Aletras2021 · 254 citations
  12. Evaluating Discourse Phenomena in Neural Machine TranslationarXiv:1711.00513Rachel Bawden, Rico Sennrich, Alexandra Birch, et al.2017 · 252 citations
  13. Syntactic Features for Evaluation of Machine TranslationDing Liu, Daniel Gildea2005 · 251 citations
  14. Overview of the EVALITA 2018 Hate Speech Detection TaskEvaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final WorkshopCristina Bosco, Felice Dell'Orletta, Fabio Poletto, et al.2018 · 248 citations
  15. Automatic Evaluation of Information Ordering: Kendall's TauComputational LinguisticsMirella Lapata2006 · 246 citations
  16. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkFindings of the Association for Computational LinguisticsOscar Sainz, Jon Campos, Iker García-Ferrero, et al.2023 · 245 citations
  17. Overview of the SPMRL 2013 Shared Task: A Cross-Framework Evaluation of Parsing Morphologically Rich LanguagesWorkshop on Statistical Parsing of Morphologically-Rich LanguagesDjamé Seddah, Reut Tsarfaty, Sandra Kübler, et al.2013 · 244 citations
  18. CLUE: A Chinese Language Understanding Evaluation BenchmarkarXiv:2004.05986Liang Xu, Hai Hu, Xuanwei Zhang, et al.2020 · 243 citations
  19. Comparing Automatic and Human Evaluation of Local Explanations for Text ClassificationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Dong Nguyen2018 · 240 citations
  20. MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing BenchmarkConference of the European Chapter of the Association for Computational Linguistics (EACL)Haoran Li, Abhinav Arora, Shuohui Chen, et al.2021 · 239 citations
  21. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A SurveyFindings of the Association for Computational LinguisticsLin Long, Rui Wang, Ruixuan Xiao, et al.2024 · 236 citations
  22. Evaluating Open-Domain Question Answering in the Era of Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Ehsan Kamalloo, Nouha Dziri, Charles Clarke, et al.2023 · 233 citations
  23. Evaluating Automated and Manual Acquisition of Anaphora Resolution StrategiesAnnual Meeting of the Association for Computational Linguistics (ACL)Chinatsu Aone, Scott William Bennett1995 · 233 citations
  24. The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future ApproachesConference of the Association for Machine Translation in the Americas (AMTA)John S. White, Theresa A. O'Connell, Francis E. O'Mara1994 · 229 citations
  25. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language ModelsarXiv:2308.01263Paul Rottger, Hannah Rose Kirk, Bertie Vidgen, et al.2023 · 227 citations
  26. Findings of the Third Shared Task on Multimodal Machine TranslationConference on Machine Translation: Shared Task PapersLoïc Barrault, Fethi Bougares, Lucia Specia, et al.2018 · 227 citations
  27. Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging TasksarXiv:1801.07772 · IJCNLP 8 (2017), volume 1, 1-10Yonatan Belinkov, Lluis Marquez, Hassan Sajjad, et al.2017 · 225 citations
  28. Overview of the TREC 2019 deep learning trackarXiv:2003.07820Nick Craswell, Bhaskar Mitra, Emine Yilmaz, et al.2020 · 224 citations
  29. BLEU might be Guilty but References are not InnocentConference on Empirical Methods in Natural Language Processing (EMNLP)Markus Freitag, David Grangier, Isaac Caswell2020 · 223 citations
  30. Unsupervised Evaluation of Interactive Dialog with DialoGPTAnnual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)Shikib Mehri, Maxine Eskenazi2020 · 223 citations
  31. GLUECoS: An Evaluation Benchmark for Code-Switched NLPAnnual Meeting of the Association for Computational Linguistics (ACL)Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, et al.2020 · 222 citations
  32. Beyond BLEU: Training Neural Machine Translation with Semantic SimilarityAnnual Meeting of the Association for Computational Linguistics (ACL)John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, et al.2019 · 221 citations
  33. Break It Down: A Question Understanding BenchmarkTransactions of the Association for Computational Linguistics (TACL)Tomer Wolfson, Mor Geva, Ankit Gupta, et al.2020 · 218 citations
  34. Automatically Evaluating Text Coherence Using Discourse RelationsAnnual Meeting of the Association for Computational Linguistics (ACL)Ziheng Lin, Hwee Tou Ng, Min-Yen Kan2011 · 218 citations
  35. YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available ResourcesConference on Machine Translation (WMT)Chi-kiu Lo2019 · 217 citations
  36. Better Summarization Evaluation with Word Embeddings for ROUGEConference on Empirical Methods in Natural Language Processing (EMNLP)Jun-Ping Ng, Viktoria Abrecht2015 · 217 citations
  37. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4arXiv:2304.03439Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, et al.2023 · 215 citations
  38. Data-Driven Sentence Simplification: Survey and BenchmarkComputational LinguisticsFernando Alva-Manchego, Carolina Scarton, Lucia Specia2020 · 213 citations
  39. Results of the WMT17 Metrics Shared TaskConference on Machine Translation (WMT)Ondřej Bojar, Yvette Graham, Amir Kamran2017 · 212 citations
  40. GAIA: a benchmark for General AI AssistantsarXiv:2311.12983Gregoire Mialon, Clementine Fourrier, Craig Swift, et al.2023 · 211 citations
  41. Answers Unite! Unsupervised Metrics for Reinforced Summarization ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, et al.2019 · 211 citations
  42. Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware MetricNafise Sadat Moosavi, Michael Strube2016 · 210 citations
  43. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about ChangeConference on Neural Information Processing Systems (NeurIPS)Karthik Valmeekam, Matthew Marquez, Alberto Olmo, et al.2022 · 209 citations
  44. Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGEConference on Empirical Methods in Natural Language Processing (EMNLP)Yvette Graham2015 · 209 citations
  45. (Meta-) Evaluation of Machine TranslationConference on Machine Translation (WMT)Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, et al.2007 · 209 citations
  46. Findings of the 2012 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Chris Callison-Burch, Philipp Koehn, Christof Monz, et al.2012 · 204 citations
  47. Evaluating Factuality in Generation with Dependency-level EntailmentFindings of the Association for Computational LinguisticsTanya Goyal, Greg Durrett2020 · 203 citations
  48. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark DatasetsFindings of the Association for Computational LinguisticsMd Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, et al.2023 · 200 citations
  49. Results of the WMT20 Metrics Shared TaskConference on Machine Translation (WMT)Nitika Mathur, Johnny Wei, Markus Freitag, et al.2020 · 195 citations
  50. A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial TopicsWorkshop on Argumentation MiningEhud Aharoni, Anatoly Polnarov, Tamar Lavee, et al.2014 · 195 citations
  51. Report on the 11th IWSLT evaluation campaignInternational Conference on Spoken Language Translation (IWSLT)Mauro Cettolo, Jan Niehues, Sebastian Stüker, et al.2014 · 195 citations
  52. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning OptimizationarXiv:2306.05087Yidong Wang, Zhuohao Yu, Zhengran Zeng, et al.2023 · 192 citations
  53. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder PipelinearXiv:2406.11939Tianle Li, Wei-Lin Chiang, Evan Frick, et al.2024 · 191 citations
  54. GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4Conference on Machine Translation (WMT)Tom Kocmi, Christian Federmann2023 · 191 citations
  55. Collecting Diverse Natural Language Inference Problems for Sentence Representation EvaluationConference on Empirical Methods in Natural Language Processing (EMNLP)Adam Poliak, Aparajita Haldar, Rachel Rudinger, et al.2018 · 190 citations
  56. Findings of the 2021 Conference on Machine Translation (WMT21)Conference on Machine Translation (WMT)Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, et al.2021 · 189 citations
  57. EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for ItalianEvaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final WorkshopValerio Basile, Danilo Croce, Maria Di Maro, et al.2020 · 188 citations
  58. We Need to Consider Disagreement in EvaluationProceedings of the 1st Workshop on Benchmarking: Past, Present and FutureValerio Basile, Michael Fell, Tommaso Fornaciari, et al.2021 · 185 citations
  59. GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical ReasoningFindings of the Association for Computational LinguisticsJiaqi Chen, Jianheng Tang, Jinghui Qin, et al.2021 · 183 citations
  60. Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question AnsweringBioNLP Shared Task WorkshopAsma Ben Abacha, Chaitanya Shivade, Dina Demner-Fushman2019 · 179 citations
  61. Evaluating Large Language Models: A Comprehensive SurveyarXiv:2310.19736Zishan Guo, Renren Jin, Chuang Liu, et al.2023 · 178 citations
  62. Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved YetConference on Machine Translation (WMT)Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, et al.2024 · 177 citations
  63. Findings of the IWSLT 2022 Evaluation CampaignInternational Conference on Spoken Language Translation (IWSLT)Antonios Anastasopoulos, Loïc Barrault, Luisa Bentivogli, et al.2022 · 177 citations
  64. LILA: A Unified Benchmark for Mathematical ReasoningConference on Empirical Methods in Natural Language Processing (EMNLP)Swaroop Mishra, Matthew Finlayson, Pan Lu, et al.2022 · 177 citations
  65. Results of the WMT16 Metrics Shared TaskConference on Machine Translation: Shared Task PapersOndřej Bojar, Yvette Graham, Amir Kamran, et al.2016 · 177 citations
  66. Results of the WMT14 Metrics Shared TaskConference on Machine Translation (WMT)Matouš Macháček, Ondřej Bojar2014 · 176 citations
  67. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic LanguagesAnnual Meeting of the Association for Computational Linguistics (ACL)Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, et al.2023 · 174 citations
  68. Findings of the WMT 2019 Shared Tasks on Quality EstimationConference on Machine Translation (WMT)Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, et al.2019 · 173 citations
  69. CDER: Efficient MT Evaluation Using Block MovementsConference of the European Chapter of the Association for Computational Linguistics (EACL)Gregor Leusch, Nicola Ueffing, Hermann Ney2006 · 172 citations
  70. MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn DialoguesAnnual Meeting of the Association for Computational Linguistics (ACL)Ge Bai, Jie Liu, Xingyuan Bu, et al.2024 · 171 citations
  71. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMsConference of the European Chapter of the Association for Computational Linguistics (EACL)Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, et al.2024 · 170 citations
  72. Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and InclusionWorkshop on Language Technology for Equality, Diversity, InclusionBharathi Raja Chakravarthi, Vigneshwaran Muralidaran2021 · 170 citations
  73. Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News DomainConference on Machine Translation (WMT)Markus Freitag, Ricardo Rei, Nitika Mathur, et al.2021 · 169 citations
  74. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy OptimizationarXiv:2210.01241Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiante Brantley, et al.2022 · 168 citations
  75. Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performanceConference on Machine Translation: Shared Task PapersQingsong Ma, Ondřej Bojar, Yvette Graham2018 · 165 citations
  76. Overview of the DSL Shared Task 2015LT4VarDial 2015Marcos Zampieri, Liling Tan, Nikola Ljubešić, et al.2015 · 164 citations
  77. On Evaluation of Adversarial Perturbations for Sequence-to-Sequence ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Paul Michel, Xian Li, Graham Neubig, et al.2019 · 162 citations
  78. Chart-to-Text: A Large-Scale Benchmark for Chart SummarizationAnnual Meeting of the Association for Computational Linguistics (ACL)Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, et al.2022 · 161 citations
  79. Findings of the WMT 2020 Shared Task on Quality EstimationConference on Machine Translation (WMT)Lucia Specia, Frédéric Blain, Marina Fomicheva, et al.2020 · 160 citations
  80. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language ModelsarXiv:2410.07985Bofei Gao, Feifan Song, Zhe Yang, et al.2024 · 159 citations
  81. Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors PersistConference on Machine Translation (WMT)Marzena Karpinska, Mohit Iyyer2023 · 156 citations
  82. The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation EvaluationConference on Machine Translation (WMT)Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, et al.2023 · 156 citations
  83. The (Un)Suitability of Automatic Evaluation Metrics for Text SimplificationComputational LinguisticsFernando Alva-Manchego, Carolina Scarton, Lucia Specia2021 · 156 citations
  84. Question Answering as an Automatic Evaluation Metric for News Article SummarizationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Matan Eyal, Tal Baumel, Michael Elhadad2019 · 156 citations
  85. Accurate Evaluation of Segment-level Machine Translation MetricsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Yvette Graham, Timothy Baldwin, Nitika Mathur2015 · 156 citations
  86. Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not InnocentConference on Machine Translation (WMT)Markus Freitag, Nitika Mathur, Chi-kiu Lo, et al.2023 · 155 citations
  87. Evaluating Style Transfer for TextConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Remi Mir, Bjarke Felbo, Nick Obradovich, et al.2019 · 155 citations
  88. LLM-based NLG Evaluation: Current Status and ChallengesComputational LinguisticsMingqi Gao, Xinyu Hu, Xunjian Yin, et al.2025 · 152 citations
  89. FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGNInternational Conference on Spoken Language Translation (IWSLT)Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, et al.2023 · 152 citations
  90. A Perspective on Word Sense Disambiguation Methods and Their EvaluationTagging Text with Lexical Semantics: Why, What, and How?Philip Resnik, David Yarowsky1997 · 151 citations
  91. Statistical Power and Translationese in Machine Translation EvaluationConference on Empirical Methods in Natural Language Processing (EMNLP)Yvette Graham, Barry Haddow, Philipp Koehn2020 · 150 citations
  92. Overview of the IWSLT 2017 Evaluation CampaignInternational Conference on Spoken Language Translation (IWSLT)Mauro Cettolo, Marcello Federico, Luisa Bentivogli, et al.2017 · 150 citations
  93. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a SummaryTransactions of the Association for Computational Linguistics (TACL)Daniel Deutsch, Tania Bedrax-Weiss, Dan Roth2021 · 148 citations
  94. BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and LanguagesConference on Neural Information Processing Systems (NeurIPS)Junho Myung, Nayeon Lee, Yi Zhou, et al.2024 · 147 citations
  95. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation PairsConference of the European Chapter of the Association for Computational Linguistics (EACL)Rico Sennrich2017 · 147 citations
  96. Evaluating Machine Translation Output with Automatic Sentence SegmentationInternational Conference on Spoken Language Translation (IWSLT)Evgeny Matusov, Gregor Leusch, Oliver Bender, et al.2005 · 147 citations
  97. Evaluating Correctness and Faithfulness of Instruction-Following Models for Question AnsweringTransactions of the Association for Computational Linguistics (TACL)Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, et al.2024 · 145 citations
  98. Self-Evaluation Guided Beam Search for ReasoningConference on Neural Information Processing Systems (NeurIPS)Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, et al.2023 · 145 citations
  99. AfriSenti: A Twitter Sentiment Analysis Benchmark for African LanguagesConference on Empirical Methods in Natural Language Processing (EMNLP)Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, et al.2023 · 143 citations
  100. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine TranslationInternational Conference on Computational Linguistics (COLING)Chin-Yew Lin, Franz Josef Och2004 · 143 citations

1 2 3 4 … 8

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/3/