Elamur Research⌕
☰
home / topics / Evaluation & benchmarks / page 4

📏 Evaluation & benchmarks

684 papers · page 4 of 8 · 226–325 · the most cited first · all topics

How systems are measured: benchmarks, metrics, human judgements and what the numbers mean.

 Papers (684)

1 2 3 4 5 … 8

  1. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and BeyondIEEE Transactions on Knowledge and Data EngineeringFangzhi Xu, Qika Lin, Jiawei Han, et al.2023 · 141 citations
  2. RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation EvaluationConference on Machine Translation: Shared Task PapersHiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi2018 · 141 citations
  3. A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text GenerationAnnual Meeting of the Association for Computational Linguistics (ACL)Tianyu Liu, Yizhe Zhang, Chris Brockett, et al.2022 · 140 citations
  4. TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text GenerationFindings of the Association for Computational LinguisticsAdaku Uchendu, Zeyu Ma, Thai Le, et al.2021 · 140 citations
  5. On The Evaluation of Machine Translation Systems Trained With Back-TranslationAnnual Meeting of the Association for Computational Linguistics (ACL)Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, et al.2020 · 140 citations
  6. FaithDial: A Faithful Benchmark for Information-Seeking DialogueTransactions of the Association for Computational Linguistics (TACL)Nouha Dziri, Ehsan Kamalloo, Sivan Milton, et al.2022 · 138 citations
  7. Investigating Data Contamination in Modern Benchmarks for Large Language ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Chunyuan Deng, Yilun Zhao, Xiangru Tang, et al.2024 · 137 citations
  8. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation TasksarXiv:2406.18403Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, et al.2024 · 137 citations
  9. Findings of the 2009 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Chris Callison-Burch, Philipp Koehn, Christof Monz, et al.2009 · 137 citations
  10. Findings of the 2011 Workshop on Statistical Machine TranslationConference on Machine Translation (WMT)Chris Callison-Burch, Philipp Koehn, Christof Monz, et al.2011 · 136 citations
  11. Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language ModelsFindings of the Association for Computational LinguisticsQingyu Lu, Baopu Qiu, Liang Ding, et al.2024 · 135 citations
  12. Semantic Structural Evaluation for Text SimplificationElior Sulem, Omri Abend, Ari Rappoport2018 · 135 citations
  13. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Paul Röttger, Valentin Hofmann, Valentina Pyatkin, et al.2024 · 134 citations
  14. Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesarXiv:1904.01172Shane Storks, Qiaozi Gao, Joyce Y. Chai2019 · 134 citations
  15. Findings of the WMT 2018 Shared Task on Quality EstimationConference on Machine Translation: Shared Task PapersLucia Specia, Frédéric Blain, Varvara Logacheva, et al.2018 · 134 citations
  16. Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image DescriptionConference on Machine Translation (WMT)Desmond Elliott, Stella Frank, Loïc Barrault, et al.2017 · 134 citations
  17. Learning Compact Metrics for MTConference on Empirical Methods in Natural Language Processing (EMNLP)Amy Pu, Hyung Won Chung, Ankur Parikh, et al.2021 · 130 citations
  18. An Analysis of Natural Language Inference Benchmarks through the Lens of NegationMd Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, et al.2020 · 130 citations
  19. Findings of the WMT 2023 Shared Task on Quality EstimationConference on Machine Translation (WMT)Frederic Blain, Chrysoula Zerva, Ricardo Rei, et al.2023 · 129 citations
  20. ZeroSCROLLS: A Zero-Shot Benchmark for Long Text UnderstandingFindings of the Association for Computational LinguisticsUri Shaham, Maor Ivgi, Avia Efrat, et al.2023 · 129 citations
  21. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text AnonymizationComputational LinguisticsIldikó Pilán, Pierre Lison, Lilja Øvrelid, et al.2022 · 129 citations
  22. GO FIGURE: A Meta Evaluation of Factuality in SummarizationFindings of the Association for Computational LinguisticsSaadia Gabriel, Asli Celikyilmaz, Rahul Jha, et al.2021 · 129 citations
  23. Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction BenchmarkNatural Legal Language Processing WorkshopJoel Niklaus, Ilias Chalkidis, Matthias Stürmer2021 · 129 citations
  24. Task-oriented Evaluation of Syntactic Parsers and Their RepresentationsYusuke Miyao, Rune Sætre, Kenji Sagae, et al.2008 · 129 citations
  25. Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationFindings of the Association for Computational LinguisticsYang Deng, Lizi Liao, Liang Chen, et al.2023 · 128 citations
  26. A Comprehensive Assessment of Dialog Evaluation MetricsThe First Workshop on Evaluations and Assessments of Neural Conversation SystemsYi-Ting Yeh, Maxine Eskenazi, Shikib Mehri2021 · 128 citations
  27. A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilityarXiv:2303.13547Aiwei Liu, Xuming Hu, Lijie Wen, et al.2023 · 127 citations
  28. A Corpus-Based Evaluation of Centering and Pronoun ResolutionComputational LinguisticsJoel R. Tetreault2001 · 125 citations
  29. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsarXiv:2406.12045Shunyu Yao, Noah Shinn, Pedram Razavi, et al.2024 · 123 citations
  30. LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal DomainFindings of the Association for Computational LinguisticsJoel Niklaus, Veton Matoshi, Pooja Rani, et al.2023 · 123 citations
  31. A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine TranslationConference on Machine Translation: Research PapersMathias Müller, Annette Rios, Elena Voita, et al.2018 · 123 citations
  32. EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language InferenceAbhilasha Ravichander, Aakanksha Naik, Carolyn Rose, et al.2019 · 122 citations
  33. Generative Judge for Evaluating AlignmentarXiv:2310.05470Junlong Li, Shichao Sun, Weizhe Yuan, et al.2023 · 121 citations
  34. The Moral Integrity Corpus: A Benchmark for Ethical Dialogue SystemsAnnual Meeting of the Association for Computational Linguistics (ACL)Caleb Ziems, Jane A. Yu, Yi-Chia Wang, et al.2022 · 121 citations
  35. Automated evaluation of written discourse coherence using GPT-4Workshop on Innovative Use of NLP for Building Educational Applications (BEA)Ben Naismith, Phoebe Mulcaire, Jill Burstein2023 · 120 citations
  36. Overview of the WANLP 2022 Shared Task on Propaganda Detection in ArabicArabic Natural Language Processing Workshop (WANLP)Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, et al.2022 · 119 citations
  37. X-Fact: A New Benchmark Dataset for Multilingual Fact CheckingAnnual Meeting of the Association for Computational Linguistics (ACL)Ashim Gupta, Vivek Srikumar2021 · 119 citations
  38. MetricX-24: The Google Submission to the WMT 2024 Metrics Shared TaskConference on Machine Translation (WMT)Juraj Juraska, Daniel Deutsch, Mara Finkelstein, et al.2024 · 118 citations
  39. MetricX-23: The Google Submission to the WMT 2023 Metrics Shared TaskConference on Machine Translation (WMT)Juraj Juraska, Mara Finkelstein, Daniel Deutsch, et al.2023 · 118 citations
  40. Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the AmericasWorkshop on NLP for Indigenous Languages of the Americas (AmericasNLP)Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, et al.2021 · 118 citations
  41. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine TranslationConference on Machine Translation (WMT)Chris Callison-Burch, Philipp Koehn, Christof Monz, et al.2010 · 118 citations
  42. Constructing Corpora for the Development and Evaluation of Paraphrase SystemsComputational LinguisticsTrevor Cohn, Chris Callison-Burch, Mirella Lapata2008 · 118 citations
  43. How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVENTransactions of the Association for Computational Linguistics (TACL)R. Thomas McCoy, Paul Smolensky, Tal Linzen, et al.2023 · 116 citations
  44. Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human EvaluationarXiv:2212.07981Yixin Liu, Alexander R. Fabbri, Pengfei Liu, et al.2022 · 116 citations
  45. Intrinsic Bias Metrics Do Not Correlate with Application BiasAnnual Meeting of the Association for Computational Linguistics (ACL)Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, et al.2021 · 116 citations
  46. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling MethodsTransactions of the Association for Computational Linguistics (TACL)Daniel Deutsch, Rotem Dror, Dan Roth2021 · 115 citations
  47. Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT MetricConference on Machine Translation (WMT)Matthew Snover, Nitin Madnani, Bonnie Dorr, et al.2009 · 115 citations
  48. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language ModelsFindings of the Association for Computational LinguisticsLijun Li, Bowen Dong, Ruohui Wang, et al.2024 · 114 citations
  49. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical TurkConference on Empirical Methods in Natural Language Processing (EMNLP)Chris Callison-Burch2009 · 114 citations
  50. CRUXEval: A Benchmark for Code Reasoning, Understanding and ExecutionarXiv:2401.03065Alex Gu, Baptiste Rozière, Hugh Leather, et al.2024 · 113 citations
  51. Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical StudyFindings of the Association for Computational LinguisticsYi Chen, Rui Wang, Haiyun Jiang, et al.2023 · 113 citations
  52. LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form SummarizationConference of the European Chapter of the Association for Computational Linguistics (EACL)Kalpesh Krishna, Erin Bransom, Bailey Kuehl, et al.2023 · 113 citations
  53. A Critical Evaluation of Evaluations for Long-form Question AnsweringAnnual Meeting of the Association for Computational Linguistics (ACL)Fangyuan Xu, Yixiao Song, Mohit Iyyer, et al.2023 · 112 citations
  54. Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in ArabicArabic Natural Language Processing Workshop (WANLP)Ibrahim Abu Farha, Wajdi Zaghouani, Walid Magdy2021 · 112 citations
  55. XL-WiC: A Multilingual Benchmark for Evaluating Semantic ContextualizationConference on Empirical Methods in Natural Language Processing (EMNLP)Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, et al.2020 · 111 citations
  56. Corpus Annotation for Parser EvaluationJohn Carroll, Guido Minnen, Ted Briscoe1999 · 111 citations
  57. RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text DetectorsAnnual Meeting of the Association for Computational Linguistics (ACL)Liam Dugan, Alyssa Hwang, Filip Trhlík, et al.2024 · 110 citations
  58. Evaluating the Performance of Large Language Models on GAOKAO BenchmarkarXiv:2305.12474Xiaotian Zhang, Chunyang Li, Yi Zong, et al.2023 · 109 citations
  59. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent EvaluationAnnual Meeting of the Association for Computational Linguistics (ACL)Quan Tu, Shilong Fan, Zihang Tian, et al.2024 · 108 citations
  60. GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem SolversAnnual Meeting of the Association for Computational Linguistics (ACL)Qintong Li, Leyang Cui, Xueliang Zhao, et al.2024 · 107 citations
  61. Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption ContestAnnual Meeting of the Association for Computational Linguistics (ACL)Jack Hessel, Ana Marasovic, Jena D. Hwang, et al.2023 · 107 citations
  62. Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?Transactions of the Association for Computational Linguistics (TACL)Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, et al.2022 · 107 citations
  63. APE at Scale and Its Implications on MT Evaluation BiasesConference on Machine Translation (WMT)Markus Freitag, Isaac Caswell, Scott Roy2019 · 107 citations
  64. Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared TaskConference on Machine Translation (WMT)Markus Freitag, Nitika Mathur, Daniel Deutsch, et al.2024 · 106 citations
  65. NTREX-128 – News Test References for MT Evaluation of 128 LanguagesProceedings of the First Workshop on Scaling Up Multilingual EvaluationChristian Federmann, Tom Kocmi, Ying Xin2022 · 104 citations
  66. LENS: A Learnable Evaluation Metric for Text SimplificationAnnual Meeting of the Association for Computational Linguistics (ACL)Mounica Maddela, Yao Dou, David Heineman, et al.2023 · 103 citations
  67. Evaluating the Evaluation of Diversity in Natural Language GenerationConference of the European Chapter of the Association for Computational Linguistics (EACL)Guy Tevet, Jonathan Berant2021 · 103 citations
  68. Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and KannadaProceedings of the First Workshop on Speech and Language Technologies for Dravidian LanguagesBharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, et al.2021 · 103 citations
  69. On the Safety of Conversational Models: Taxonomy, Dataset, and BenchmarkFindings of the Association for Computational LinguisticsHao Sun, Guangxuan Xu, Jiawen Deng, et al.2022 · 102 citations
  70. Findings of the WMT 2019 Shared Task on Automatic Post-EditingConference on Machine Translation (WMT)Rajen Chatterjee, Christian Federmann, Matteo Negri, et al.2019 · 102 citations
  71. MMTEB: Massive Multilingual Text Embedding BenchmarkarXiv:2502.13595Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, et al.2025 · 101 citations
  72. Rethinking Benchmark and Contamination for Language Models with Rephrased SamplesarXiv:2311.04850Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, et al.2023 · 100 citations
  73. On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language RepresentationsAnnual Meeting of the Association for Computational Linguistics (ACL)Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, et al.2022 · 100 citations
  74. Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering EvaluationConference on Empirical Methods in Natural Language Processing (EMNLP)Jannis Bulian, Christian Buck, Wojciech Gajewski, et al.2022 · 100 citations
  75. UniTE: Unified Translation EvaluationAnnual Meeting of the Association for Computational Linguistics (ACL)Yu Wan, Dayiheng Liu, Baosong Yang, et al.2022 · 100 citations
  76. Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE CorpusAnnual Meeting of the Association for Computational Linguistics (ACL)Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, et al.2020 · 100 citations
  77. Evaluating Attribution in Dialogue Systems: The BEGIN BenchmarkTransactions of the Association for Computational Linguistics (TACL)Nouha Dziri, Hannah Rashkin, Tal Linzen, et al.2022 · 99 citations
  78. Unbabel's Participation in the WMT20 Metrics Shared TaskConference on Machine Translation (WMT)Ricardo Rei, Craig Stewart, Ana C Farinha, et al.2020 · 98 citations
  79. DialFact: A Benchmark for Fact-Checking in DialogueAnnual Meeting of the Association for Computational Linguistics (ACL)Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, et al.2022 · 97 citations
  80. Cross-Linguistic Syntactic Evaluation of Word Prediction ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, et al.2020 · 97 citations
  81. Evaluation of machine translation and its evaluationMachine Translation Summit (MT Summit)Joseph P. Turian, Luke Shen, I. Dan Melamed2003 · 97 citations
  82. LinCE: A Centralized Benchmark for Linguistic Code-switching EvaluationLanguage Resources and Evaluation Conference (LREC)Gustavo Aguilar, Sudipta Kar, Thamar Solorio2020 · 96 citations
  83. Overview of the EVALITA 2018 Task on Irony Detection in Italian Tweets (IronITA)Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final WorkshopAlessandra Teresa Cignarella, Simona Frenda, Valerio Basile, et al.2018 · 95 citations
  84. Using Test Suites in Evaluation of Machine Translation SystemsMargaret King, Kirsten Falkedal1990 · 95 citations
  85. SPARTQA: A Textual Question Answering Benchmark for Spatial ReasoningRoshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, et al.2021 · 93 citations
  86. Findings of the WMT 2022 Shared Task on Quality EstimationConference on Machine Translation (WMT)Chrysoula Zerva, Frédéric Blain, Ricardo Rei, et al.2022 · 91 citations
  87. Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence TasksConference on Empirical Methods in Natural Language Processing (EMNLP)Andrea Sottana, Bin Liang, Kai Zou, et al.2023 · 89 citations
  88. A Pairwise Event Coreference Model, Feature Impact and Evaluation for Event Coreference ResolutionProceedings of the Workshop on Events in Emerging Text TypesZheng Chen, Heng Ji, Robert Haralick2009 · 88 citations
  89. Multidimensional quality metrics: a flexible system for assessing translation qualityArle Richard Lommel, Aljoscha Burchardt, Hans Uszkoreit2013 · 87 citations
  90. Evaluation Metrics For End-to-End Coreference Resolution SystemsSIGDIAL ConferenceJie Cai, Michael Strube2010 · 87 citations
  91. Error Analysis of Statistical Machine Translation OutputDavid Vilar, Jia Xu, Luis Fernando D'Haro, et al.2006 · 87 citations
  92. Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)Workshop on Discourse in Machine TranslationLesly Miculicich Werlen, Andrei Popescu-Belis2017 · 86 citations
  93. Unbounded Dependency Recovery for Parser EvaluationLaura Rimell, Stephen Clark, Mark Steedman2009 · 86 citations
  94. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Yuxin Jiang, Yufei Wang, Xingshan Zeng, et al.2024 · 85 citations
  95. BlonDe: An Automatic Evaluation Metric for Document-level Machine TranslationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Yuchen Jiang, Tianyu Liu, Shuming Ma, et al.2022 · 85 citations
  96. Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared TaskConference on Machine Translation (WMT)Ricardo Rei, Ana C Farinha, Chrysoula Zerva, et al.2021 · 85 citations
  97. Evaluation of Dependency Parsers on Unbounded DependenciesJoakim Nivre, Laura Rimell, Ryan McDonald, et al.2010 · 85 citations
  98. Evaluating Mathematical Reasoning Beyond AccuracyProceedings of the AAAI Conference on Artificial IntelligenceShijie Xia, Xuefeng Li, Yixin Liu, et al.2024 · 84 citations
  99. Error Span Annotation: A Balanced Approach for Human Evaluation of Machine TranslationConference on Machine Translation (WMT)Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, et al.2024 · 83 citations
  100. ORCA: A Challenging Benchmark for Arabic Language UnderstandingFindings of the Association for Computational LinguisticsAbdelRahim Elmadany, ElMoatez Billah Nagoudi, Muhammad Abdul-Mageed2023 · 83 citations

1 2 3 4 5 … 8

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/4/