Elamur Research⌕
☰
home / topics / Evaluation & benchmarks / page 5

📏 Evaluation & benchmarks

684 papers · page 5 of 8 · 326–425 · the most cited first · all topics

How systems are measured: benchmarks, metrics, human judgements and what the numbers mean.

 Papers (684)

1 … 4 5 6 7 8

  1. MEANT 2.0: Accurate semantic MT evaluation for any output languageChi-kiu Lo2017 · 82 citations
  2. Findings of the WMT 2021 Shared Task on Quality EstimationConference on Machine Translation (WMT)Lucia Specia, Frédéric Blain, Marina Fomicheva, et al.2021 · 80 citations
  3. Generating Benchmarks for Factuality Evaluation of Language ModelsConference of the European Chapter of the Association for Computational Linguistics (EACL)Dor Muhlgay, Ori Ram, Inbal Magar, et al.2024 · 79 citations
  4. Formalism-Independent Parser Evaluation with CCG and DepBankStephen Clark, James Curran2007 · 79 citations
  5. KoBBQ: Korean Bias Benchmark for Question AnsweringTransactions of the Association for Computational Linguistics (TACL)Jiho Jin, Jiseon Kim, Nayeon Lee, et al.2024 · 77 citations
  6. Evaluating prose style transfer with the BibleKeith Carlson, Allen Riddell, Daniel Rockmore2017 · 77 citations
  7. A survey on Recognizing Textual Entailment as an NLP EvaluationWorkshop on Evaluation and Comparison of NLP SystemsAdam Poliak2020 · 76 citations
  8. A Dependency Perspective on RST Discourse Parsing and EvaluationMathieu Morey, Philippe Muller, Nicholas Asher2018 · 76 citations
  9. AudioBench: A Universal Benchmark for Audio Large Language ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Bin Wang, Xunlong Zou, Geyu Lin, et al.2025 · 75 citations
  10. PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics CapabilitiesSettaluri Sravanthi, Meet Doshi, Pavan Tankala, et al.2024 · 75 citations
  11. Extracting and evaluating general world knowledge from the Brown CorpusLenhart Schubert, Matthew Tong2003 · 75 citations
  12. MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model EvaluationConference on Empirical Methods in Natural Language Processing (EMNLP)Weihao Xuan, Rui Yang, Heli Qi, et al.2025 · 74 citations
  13. Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test SetsConference on Machine Translation (WMT)Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, et al.2025 · 73 citations
  14. Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing TasksLanguage Resources and Evaluation Conference (LREC)Ruiyang Zhou, Lu Chen, Kai Yu2024 · 73 citations
  15. OR-Bench: An Over-Refusal Benchmark for Large Language ModelsarXiv:2405.20947Justin Cui, Wei-Lin Chiang, Ion Stoica, et al.2024 · 73 citations
  16. ScandEval: A Benchmark for Scandinavian Natural Language ProcessingNordic Conference on Computational Linguistics (NoDaLiDa)Dan Nielsen2023 · 73 citations
  17. CLiMP: A Benchmark for Chinese Language Model EvaluationConference of the European Chapter of the Association for Computational Linguistics (EACL)Beilei Xiang, Changbing Yang, Yu Li, et al.2021 · 73 citations
  18. Evaluating BERT for natural language inference: A case study on the CommitmentBankConference on Empirical Methods in Natural Language Processing (EMNLP)Nanjiang Jiang, Marie-Catherine de Marneffe2019 · 72 citations
  19. CBLUE: A Chinese Biomedical Language Understanding Evaluation BenchmarkarXiv:2106.08087Ningyu Zhang, Mosha Chen, Zhen Bi, et al.2021 · 71 citations
  20. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive RetrievalarXiv:2407.12883Hongjin Su, Howard Yen, Mengzhou Xia, et al.2024 · 70 citations
  21. MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical ReasoningConference on Neural Information Processing Systems (NeurIPS)Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, et al.2024 · 70 citations
  22. StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in TextsZhengxiang Shi, Qiang Zhang, Aldo Lipani2022 · 70 citations
  23. IrokoBench: A New Benchmark for African Languages in the Age of Large Language ModelsConference of the North American Chapter of the Association for Computational Linguistics (NAACL)David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, et al.2025 · 69 citations
  24. Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?arXiv:2309.07462Rishav Hada, Varun Gumma, Adrian de Wynter, et al.2023 · 69 citations
  25. Findings of the WMT 2017 Biomedical Translation Shared TaskConference on Machine Translation (WMT)Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, et al.2017 · 69 citations
  26. AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMsInternational Conference on Computational Linguistics (COLING)Basel Mousi, Nadir Durrani, Fatema Ahmad, et al.2025 · 68 citations
  27. Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level MetricConference on Machine Translation (WMT)Giorgos Vernikos, Brian Thompson, Prashant Mathur, et al.2022 · 68 citations
  28. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use CapabilitiesFindings of the Association for Computational LinguisticsJiarui Lu, Thomas Holleis, Yizhe Zhang, et al.2025 · 67 citations
  29. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, et al.2024 · 66 citations
  30. Findings of the WMT 2018 Shared Task on Automatic Post-EditingConference on Machine Translation: Shared Task PapersRajen Chatterjee, Matteo Negri, Raphael Rubino, et al.2018 · 66 citations
  31. Evaluating Topic Coherence Using Distributional SemanticsInternational Conference on Computational Semantics – Long PapersNikolaos Aletras, Mark Stevenson2013 · 64 citations
  32. A Fine-Grained Analysis of BERTScoreConference on Machine Translation (WMT)Michael Hanna, Ondřej Bojar2021 · 62 citations
  33. Large Language Model Routing with Benchmark DatasetsarXiv:2309.15789Tal Shnitzer, Anthony Ou, Mirian Silva, et al.2023 · 61 citations
  34. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language ModelsLanguage Resources and Evaluation Conference (LREC)Zican Dong, Tianyi Tang, Junyi Li, et al.2024 · 59 citations
  35. HUME: Human UCCA-Based Evaluation of Machine TranslationAlexandra Birch, Omri Abend, Ondřej Bojar, et al.2016 · 58 citations
  36. Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared TaskConference on Machine Translation (WMT)Thibault Sellam, Amy Pu, Hyung Won Chung, et al.2020 · 55 citations
  37. CMB: A Comprehensive Medical Benchmark in ChinesearXiv:2308.08833Xidong Wang, Guiming Hardy Chen, Dingjie Song, et al.2023 · 54 citations
  38. Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial ExamplesXiaoqing Zheng, Jiehang Zeng, Yi Zhou, et al.2020 · 54 citations
  39. Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluation for MEDLINE Abstracts and Biomedical TerminologiesConference on Machine Translation (WMT)Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, et al.2019 · 54 citations
  40. Parser Evaluation over Local and Non-Local Deep Dependencies in a Large CorpusEmily M. Bender, Dan Flickinger, Stephan Oepen, et al.2011 · 53 citations
  41. Overview of the IWPT 2020 Shared Task on Parsing into Enhanced Universal DependenciesGosse Bouma, Djamé Seddah, Daniel Zeman2020 · 52 citations
  42. Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary InductionEMNLP-IJCNLP 2019 (Hong Kong)Yova Kementchedjhieva, Mareike Hartmann, Anders Sgaard2019 · 52 citations
  43. Evaluation Tool for Rule-based Anaphora Resolution MethodsAnnual Meeting of the Association for Computational Linguistics (ACL)Catalina Barbu, Ruslan Mitkov2001 · 52 citations
  44. XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and RetrievalAnnual Meeting of the Association for Computational Linguistics (ACL)Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, et al.2024 · 51 citations
  45. Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMRJuri Opitz, Anette Frank2021 · 51 citations
  46. Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality TransferEleftheria Briakou, Sweta Agrawal, Joel Tetreault, et al.2021 · 49 citations
  47. Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction SystemsInternational Joint Conference on Natural Language Processing (IJCNLP)Hiroki Asano, Tomoya Mizumoto, Kentaro Inui2017 · 49 citations
  48. How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechAnnual Meeting of the Association for Computational Linguistics (ACL)Aditya Yedetore, Tal Linzen, Robert Frank, et al.2023 · 48 citations
  49. Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan LanguagesarXiv:2112.09301Thomas Mandl, Sandip Modha, Gautam Kishore Shahi, et al.2021 · 48 citations
  50. Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph SimilarityJuri Opitz, Angel Daza, Anette Frank2021 · 47 citations
  51. Evaluating Dependency Parsing: Robust and Heuristics-Free Cross-Annotation EvaluationReut Tsarfaty, Joakim Nivre, Evelina Andersson2011 · 46 citations
  52. Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human JudgmentsComputational LinguisticsMarina Fomicheva, Lucia Specia2019 · 45 citations
  53. Findings of the WMT 2023 Shared Task on Low-Resource Indic Language TranslationConference on Machine Translation (WMT)Santanu Pal, Partha Pakray, Sahinur Rahman Laskar, et al.2023 · 41 citations
  54. Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNetInternational Conference on Computational Linguistics (COLING)Alexis Palmer, Caroline Sporleder2010 · 41 citations
  55. Evaluating a Focus-Based Approach to Anaphora ResolutionAnnual Meeting of the Association for Computational Linguistics (ACL)Saliha Azzam, Kevin Humphreys, Robert Gaizauskas1998 · 41 citations
  56. Automatic Evaluation and Analysis of Idioms in Neural Machine TranslationConference of the European Chapter of the Association for Computational Linguistics (EACL)Christos Baziotis, Prashant Mathur, Eva Hasler2023 · 40 citations
  57. xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and RetrievalarXiv:2303.03004Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, et al.2023 · 40 citations
  58. Discourse Structure in Machine Translation EvaluationComputational LinguisticsShafiq Joty, Francisco Guzmán, Lluís Màrquez, et al.2017 · 40 citations
  59. Findings of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23)Conference on Machine Translation (WMT)Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, et al.2023 · 39 citations
  60. ASAD: A Twitter-based Benchmark Arabic Sentiment Analysis DatasetarXiv:2011.00578Basma Alharbi, Hind Alamro, Manal Alshehri, et al.2020 · 39 citations
  61. Linguistic Evaluation of German-English Machine Translation Using a Test SuiteConference on Machine Translation (WMT)Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, et al.2019 · 38 citations
  62. Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysisACM Transactions on Asian and Low-Resource Language Information ProcessingAli Bou Nassif, Abdollah Masoud Darya, Ashraf Elnagar2021 · 34 citations
  63. FRMT: A Benchmark for Few-Shot Region-Aware Machine TranslationTransactions of the Association for Computational Linguistics (TACL)Parker Riley, Timothy Dozat, Jan A. Botha, et al.2023 · 33 citations
  64. Fine-Tuned Machine Translation Metrics Struggle in Unseen DomainsAnnual Meeting of the Association for Computational Linguistics (ACL)Vilém Zouhar, Shuoyang Ding, Anna Currey, et al.2024 · 32 citations
  65. MaTESe: Machine Translation Evaluation as a Sequence Tagging ProblemConference on Machine Translation (WMT)Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, et al.2022 · 32 citations
  66. Realistic Evaluation Principles for Cross-document Coreference ResolutionJoint Conference on Lexical and Computational Semantics (*SEM)Arie Cattan, Alon Eirew, Gabriel Stanovsky, et al.2021 · 31 citations
  67. Observational Scaling Laws and the Predictability of Language Model PerformancearXiv:2405.10938Yangjun Ruan, Chris J. Maddison, Tatsunori Hashimoto2024 · 30 citations
  68. MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding MethodsarXiv:2309.10966Mara Finkelstein; Subhajit Naskar; Mehdi Mirzazadeh; Apurva Shah;, Markus Freitag2023 · 30 citations
  69. Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker CommitmentAnnual Meeting of the Association for Computational Linguistics (ACL)Nanjiang Jiang, Marie-Catherine de Marneffe2019 · 30 citations
  70. Revisiting the Evaluation for Cross Document Event CoreferenceInternational Conference on Computational Linguistics (COLING)Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, et al.2016 · 30 citations
  71. ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation MetricsChantal Amrhein, Nikita Moghe, Liane Guillou2022 · 29 citations
  72. Findings of the WMT 2020 Shared Task on Chat TranslationConference on Machine Translation (WMT)M. Amin Farajian, António V. Lopes, André F. T. Martins, et al.2020 · 29 citations
  73. Machine Translation Meta Evaluation through Translation Accuracy Challenge SetsComputational LinguisticsNikita Moghe, Arnisa Fazla, Chantal Amrhein, et al.2025 · 28 citations
  74. SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language InferenceMartin Schmitt, Hinrich Schütze2019 · 28 citations
  75. RED: A Reference Dependency Based MT Evaluation MetricHui Yu, Xiaofeng Wu, Jun Xie, et al.2014 · 28 citations
  76. Findings of the Second Shared Task on Multilingual Coreference ResolutionWorkshop on Computational Models of Reference, Anaphora and Coreference (CRAC)Zdeněk Žabokrtský, Miloslav Konopik, Anna Nedoluzhko, et al.2023 · 27 citations
  77. The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency ParsersMurhaf Fares, Stephan Oepen, Lilja Øvrelid, et al.2018 · 27 citations
  78. Description, Evaluation and Stance: Exploring the Forms and Functions of Speech Descriptors in Early Modern EnglishNordic Journal of English StudiesPeter J. Grund2017 · 27 citations
  79. The Multidimensional Quality Metrics (MQM) Framework: a new framework for translation quality assessmentValerie Mariana, Troy Cox, Alan Melby2015 · 27 citations
  80. How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled BenchmarkMinglai Yang, Ethan Huang, Liang Zhang, et al.2025 · 24 citations
  81. M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation EvaluationAnnual Meeting of the Association for Computational Linguistics (ACL)Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, et al.2025 · 24 citations
  82. Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!Annual Meeting of the Association for Computational Linguistics (ACL)Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, et al.2024 · 24 citations
  83. Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal NegationConference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL)Hung Thinh Truong, Yulia Otmakhova, Timothy Baldwin, et al.2022 · 24 citations
  84. A Human Evaluation of AMR-to-English Generation SystemsEmma Manning, Shira Wein, Nathan Schneider2020 · 24 citations
  85. Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and modelsarXiv:2005.14709Viktor Schlegel, Goran Nenadic, Riza Batista-Navarro2020 · 24 citations
  86. A New Amharic Speech Emotion Dataset and Classification BenchmarkACM Transactions on Asian and Low-Resource Language Information ProcessingEphrem A. Retta, Eiad Almekhlafi, Richard Sutcliffe, et al.2022 · 23 citations
  87. Taking the Correction Difficulty into Account in Grammatical Error Correction EvaluationInternational Conference on Computational Linguistics (COLING)Takumi Gotou, Ryo Nagata, Masato Mita, et al.2020 · 23 citations
  88. Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of ClaimsarXiv:2109.15118 · CLEF-2019Tamer Elsayed, Preslav Nakov, Alberto Barron-Cedeno, et al.2019 · 23 citations
  89. Coreference Resolution Evaluation for Higher Level ApplicationsConference of the European Chapter of the Association for Computational Linguistics (EACL)Don Tuggener2014 · 23 citations
  90. Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation BankLanguage Resources and Evaluation Conference (LREC)Sudheer Kolachina, Rashmi Prasad, Dipti Misra Sharma, et al.2012 · 23 citations
  91. Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language ModelsFindings of the Association for Computational LinguisticsYeeun Kim, Youngrok Choi, Eunkyung Choi, et al.2024 · 22 citations
  92. Singletons and Coreference Resolution EvaluationSandra Kübler, Desislava Zhekova2011 · 22 citations
  93. A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and NewsWorkshop on Computational Models of Reference, Anaphora and Coreference (CRAC)Corbèn Poot, Andreas van Cranenburgh2020 · 21 citations
  94. BLiMP: The Benchmark of Linguistic Minimal Pairs for EnglishAlex Warstadt, Alicia Parrish, Haokun Liu, et al.2020 · 21 citations
  95. Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment AnalysisFindings of the Association for Computational LinguisticsDaniela Teodorescu, Saif Mohammad2023 · 20 citations
  96. KoBE: Knowledge-Based Machine Translation EvaluationZorik Gekhman, Roee Aharoni, Genady Beryozkin, et al.2020 · 20 citations
  97. Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary DetectionNafise Sadat Moosavi, Leo Born, Massimo Poesio, et al.2019 · 20 citations
  98. Evaluating a Statistical CCG Parser on WikipediaMatthew Honnibal, Joel Nothman, James R. Curran2009 · 19 citations
  99. Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and EleganceConference on Empirical Methods in Natural Language Processing (EMNLP)Andong Chen, Lianzhang Lou, Kehai Chen, et al.2025 · 18 citations
  100. CamelEval: Advancing Culturally Aligned Arabic Language Models and BenchmarksarXiv:2409.12623Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, et al.2024 · 18 citations

1 … 4 5 6 7 8

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/5/