📏 Evaluation & benchmarks

A survey · about 18 minutes to read · every paper named here has its own page in this library

Evaluation is the part of language technology that decides what counts as progress. Every system — a translator, a parser, a question-answering model, a chat assistant — is eventually reduced to numbers: a score on a test set, a correlation with human ratings, a rank on a leaderboard. This topic gathers the papers in this library that build those numbers and examine them: automatic metrics such as BLEU, METEOR, chrF and COMET; benchmark suites such as GLUE, Natural Questions and MMLU-Pro; shared-task campaigns that organise whole communities around common data; and the methodological studies that ask whether any of it measures what it claims to.

The topic is unusual because it is about the instruments rather than the phenomena. A paper on machine translation asks how to translate better; a paper here asks how anyone would know. That makes the field reflexive and often sceptical. Many of its most influential works are critiques — of a metric that rewards the wrong thing, of a benchmark that can be solved without understanding, of a human study too small to support its conclusions.

In this library the subject is dominated by machine translation, where automatic evaluation was invented and refined over two decades, but it reaches far beyond it: reading comprehension, summarisation, coreference, word embeddings, multimodal models, code generation, and the broad and fast-moving evaluation of large language models.

What it is

Evaluation in natural language processing answers three related questions. What is being measured? — a capability such as translation adequacy, factual accuracy, or reasoning. How is it measured? — by human judgement, by an automatic metric, or by a test whose answers can be checked mechanically. How much should the result be trusted? — which depends on test-set size, annotator agreement, statistical significance, and whether the system may already have seen the test data.

Several distinctions organise the field:

• Intrinsic versus extrinsic evaluation. An intrinsic evaluation scores the output directly (is this summary good?); an extrinsic one measures its usefulness in a downstream task (does it help a reader answer questions?).
• Reference-based versus reference-free metrics. Classic metrics compare system output with one or more human references; newer ones judge output against the source or an image, or use a learned model that needs no reference at all.
• Metrics versus benchmarks. A metric is a scoring function; a benchmark is a fixed collection of inputs, expected outputs and rules for comparison, often with a public leaderboard and a hidden test set.
• Evaluation versus meta-evaluation. Meta-evaluation asks how well a metric agrees with human judgement, usually by correlation at the level of whole systems or individual segments. A great deal of the work here is meta-evaluation.

The topic borders closely on machine translation, which supplies most of its metrics and campaigns, and on language models & pretraining, whose rapid growth created a demand for ever harder and broader benchmarks. It differs from both in that its object is the measurement itself.

History and milestones

Early evaluation programmes

Systematic evaluation of language technology began in large government-funded programmes in the United States in the late 1980s and 1990s. The Message Understanding Conferences defined information-extraction tasks with fixed scoring, and the ARPA machine translation programme experimented with ways of rating translations by fluency, adequacy and informativeness; The ARPA MT Evaluation Methodologies records what that programme learned. In coreference, Algorithms for Scoring Coreference Chains introduced the B-cubed measure as an alternative to the MUC link-based score. Human judgement was the gold standard, and it was slow and expensive.

The automatic-metric era

The turning point came in 2002 with the IBM paper introducing BLEU, which showed that a cheap n-gram overlap score could rank systems roughly as human judges did. It is by far the most cited work in this topic, and it set the template for a decade. Summarisation followed with recall-oriented n-gram scores (Lin and Hovy, 2003) that grew into ROUGE, and translation gained alternatives built on longest common subsequences (Lin and Och, 2004) and on explicit word alignment with stemming and synonyms (METEOR, 2005). Large shared datasets became benchmarks in their own right: RCV1 for text categorisation and the ACE programme for entity, relation and event extraction.

Shared tasks and meta-evaluation

From 2006 the annual Workshop on Statistical Machine Translation — from 2016 the Conference on Machine Translation, WMT — combined a translation competition with large-scale human evaluation and a metrics task that ranked the metrics themselves. Its yearly findings papers, from Koehn and Monz (2006) through 2014, 2016 and 2019 to 2024, form a continuous record of how human evaluation practice changed: from relative ranking of a few outputs, to direct assessment on a 0–100 scale, to source-based and document-level judgement, and finally to expert error annotation. Similar campaigns ran for spoken translation (IWSLT), parsing of morphologically rich languages (SPMRL 2013) and dialect identification (VarDial).

Benchmarks for general understanding

With pretrained neural models came a new style of benchmark: a suite of many tasks with a single aggregate score. GLUE (2018) was overtaken by models within about a year, prompting the harder SuperGLUE. The same pattern produced multilingual suites (XTREME, XGLUE), language-specific ones (IndicNLPSuite, CLUE, IndoNLU) and domain ones (LexGLUE, TweetEval). At the same time, adversarial and diagnostic work — Jia and Liang (2017), contrast sets — showed how far high scores could rest on shallow cues.

Learned metrics and the LLM era

Around 2020 machine translation evaluation moved from string overlap to learned metrics such as COMET, and WMT's metrics task concluded in 2022 that neural metrics should replace BLEU (Freitag et al., 2022). Large language models then changed the field twice over: they became the systems to be evaluated, on benchmarks such as MMLU-Pro, GPQA and MMMU, and they became evaluators themselves, as in GEMBA-MQM and Prometheus. Most papers on the topic in this library date from this most recent period.

Main problems and tasks

The work collected here falls into several recurring problems.

Designing automatic metrics. The basic task is a function that maps a system output (and possibly a source and references) to a quality score. Families include n-gram precision (BLEU), recall-weighted alignment (METEOR), character n-gram F-scores (chrF), edit distances such as TER and CDER, semantic and syntactic matching (YiSi, syntactic features), and trained regressors over pretrained encoders.

Meta-evaluating metrics. Before a metric is trusted, its agreement with human judgement must be checked. This raises its own questions: whether to correlate at system or segment level (Graham et al., 2015), how outliers distort correlations (Tangled up in BLEU), and whether a metric validated for one task transfers to another — it often does not, as studies of text simplification and summarisation found.

Collecting reliable human judgements. Human evaluation is the reference point, yet it is noisy. Work here covers crowdsourcing quality (Snow et al., 2008), scales and protocols, the role of document context (Läubli et al., 2018), expert error annotation (Freitag et al., 2021), statistical power (Graham et al., 2020) and annotator disagreement as signal rather than noise (Basile et al., 2021).

Building benchmarks. A benchmark needs representative inputs, correct labels, a fair split, and a way to prevent overfitting to the test set. Questions written from real information needs (Natural Questions, TyDi QA) behave differently from questions written while looking at the answer.

Targeted and diagnostic testing. Instead of a single average, challenge sets probe specific phenomena: discourse in translation, contrastive grammatical pairs, gender bias, object hallucination, or exaggerated safety refusals.

Quality estimation. Predicting the quality of a translation without any reference — useful when output must be checked at the moment of use — has its own WMT task series (2019, 2020, 2023).

Contamination and reproducibility. When models are trained on web-scale data, test sets leak into training data, and published numbers become hard to compare. Sainz et al. (2023) and Balloccu et al. (2024) treat this as a central threat; Post (2018) addressed the older problem of incomparable metric configurations.

Methods and schools

Surface-overlap metrics

The first school treats quality as similarity to a human reference, measured on surface strings. BLEU uses clipped n-gram precision with a length penalty; ROUGE flips the emphasis to recall for summaries; METEOR aligns words in stages and penalises fragmented matches; chrF works on characters and so needs no tokeniser, which suits morphologically rich languages. These metrics are transparent, fast and language-agnostic, and their weaknesses are well documented: they reward lexical match rather than meaning, penalise valid paraphrases, and are unreliable for single sentences. Proposals to soften them with paraphrases include Kauchak and Barzilay (2006), Meteor Universal and smoothing for sentence-level BLEU.

Learned and embedding-based metrics

A second school learns the metric from human judgements. COMET feeds source, hypothesis and reference through a multilingual encoder and regresses a score; its successors COMET-22, xCOMET and Google's MetricX-23 and MetricX-24 add error spans and larger backbones. Related ideas appear in paraphrase-based scoring, in CLIPScore for image captions, and in unsupervised dialogue metrics such as USR. The debate over these metrics is about opacity and robustness: a learned scorer may correlate well on average yet be fooled by inputs unlike its training data, and WMT asked in 2024 whether LLM-generated translations were breaking MT metrics.

Language models as judges

The newest school prompts a large language model to grade outputs, either holistically or by listing errors. GEMBA-MQM and Fernandes et al. (2023) ask models to mark translation errors in the MQM scheme; Prometheus trains an open evaluator model on rubrics; PandaLM and Arena-Hard build judge-based benchmarks for instruction-tuned models. The open questions concern judge bias, self-preference and reproducibility when the judge is a closed commercial model.

Execution and verifiable checks

Where outputs can be run or checked mechanically, evaluation can avoid both references and judges. HumanEval scored generated code by unit tests and introduced pass@k; EvalPlus showed that thin test suites inflate such scores. IFEval applies the same idea to instruction following by using only constraints a short program can verify, and text-to-SQL work examines how to compare queries fairly.

Human evaluation protocols

Human evaluation has its own schools. Relative ranking asks judges to order several outputs; direct assessment, championed by Yvette Graham and adopted by WMT, asks for an absolute score on a continuous scale, standardised per annotator; MQM-style annotation asks trained professionals to mark and classify every error. The shift toward MQM, documented by Freitag et al. (2021), reflected the finding that crowd ratings could not separate strong systems. The debate over human parity in translation — answered by Läubli et al. (2018) and Toral et al. (2018) — turned largely on these protocol choices.

Adversarial and behavioural testing

A further tradition attacks benchmarks rather than systems. Adversarial distractors, contrast sets, partial-input baselines and model-written evaluations all ask whether a high score reflects the intended skill. Linzen (2020) argued more broadly that leaderboards built on i.i.d. test sets reward the wrong kind of generalisation.

Datasets, resources and benchmarks

The benchmarks in this library can be grouped by what they test.

Beyond named benchmarks, the WMT campaigns are themselves a resource: every year they release test sets, system outputs and human judgements, which later metric papers reuse as training and validation data. The MQM annotations of Freitag et al. and the human parity data from Microsoft play the same role at a finer grain.

Landmark papers in this library

BLEU (2002)

Papineni and colleagues at IBM proposed scoring a translation by how many of its word sequences of length one to four also occur in reference translations, with counts capped so that repetition is not rewarded and a penalty for output that is too short. Its value was practical: developers could test a change in minutes rather than wait for human raters. BLEU shaped two decades of machine translation and was borrowed, often uncritically, for other generation tasks — a habit that much later work in this topic set out to correct.

METEOR (2005)

Banerjee and Lavie answered BLEU's lack of recall and its poor sentence-level behaviour with a metric based on explicit word alignment, matching exact forms, then stems, then WordNet synonyms, and penalising scattered matches. Later versions (Meteor 1.3, Meteor Universal) added paraphrase tables, function-word weighting and a way to support new target languages without hand-built resources.

ROUGE precursors (2003–2004)

Lin and Hovy tested whether BLEU-style scoring could judge summaries and found that plain unigram recall tracked human coverage ratings best. With Lin and Och the same line produced subsequence- and skip-bigram measures. Together they underlie ROUGE, still the default summarisation metric despite the doubts raised in Graham (2015) and Bhandari et al. (2020).

GLUE (2018) and SuperGLUE (2019)

GLUE bundled nine English sentence-level tasks behind a single score, a public leaderboard and a diagnostic set, aiming to reward general-purpose rather than task-specific systems. When pretrained transformers passed its human estimate within months, the same team released SuperGLUE with harder tasks and human baselines for each. The rapid saturation of both became the standard example of how short a benchmark's useful life can be.

A Call for Clarity in Reporting BLEU Scores (2018)

Matt Post showed that "BLEU" in different papers often meant different computations, because authors tokenised references in their own ways, and that the resulting differences could exceed the gains being claimed. The accompanying SacreBLEU tool, which handles preprocessing itself and prints a signature of its settings, became the standard way to report the metric.

Natural Questions (2019)

Kwiatkowski and colleagues built a question-answering benchmark from real search queries paired with Wikipedia pages, annotated with long and short answers. Because the questions were not written by people looking at the text, the dataset avoided the lexical overlap that made earlier sets easy; and by collecting several annotations per test item, it treated legitimate disagreement as part of the target.

Adversarial Examples for Reading Comprehension (2017)

Jia and Liang inserted a single misleading sentence into SQuAD paragraphs and watched the accuracy of published models collapse while human readers were barely affected. The paper made adversarial evaluation a standard tool and helped motivate benchmarks such as DROP, whose questions were collected against a live model.

Evaluating Large Language Models Trained on Code (2021)

The Codex paper by Chen and colleagues replaced text similarity with functional correctness: a generated function counts as right only if it passes hidden unit tests. Its HumanEval set and pass@k estimator became the default for code models, and EvalPlus later showed how much a richer test suite lowers those scores.

COMET (2020)

Rei, Stewart, Farinha and Lavie trained a metric on human quality judgements using a cross-lingual pretrained encoder that sees the source, the hypothesis and the reference. It correlated better with people than any surface metric, and within two years learned metrics of this kind had become the recommended replacement for BLEU in WMT.

Experts, Errors, and Context (2021)

Freitag and colleagues had professional translators annotate errors in top WMT systems using the MQM framework, with full document context. The resulting ranking differed sharply from the crowd-based one and placed human translations clearly ahead of machines. The study shifted WMT toward expert error annotation and became the reference human data for judging metrics.

Human parity claims (2018)

Microsoft's claim of human parity for Chinese–English news used careful source-based direct assessment. Two responses — Läubli, Sennrich and Volk and Toral and colleagues — showed that with document context, professional raters and attention to translationese, the gap reappeared. The exchange is a compact lesson in how evaluation design determines conclusions.

A Survey on Evaluation of Large Language Models (2023)

Chang and colleagues organised the sudden flood of LLM evaluation work by what is evaluated, where, and how, and catalogued general, domain and multimodal benchmarks. It is the most cited LLM-era overview here, alongside Guo et al. (2023), and it called for dynamic tests that resist contamination.

MMLU-Pro (2024) and GPQA (2023)

MMLU-Pro responded to the saturation of MMLU by expanding answer options to ten, filtering easy items and cleaning labels, which also made scores less sensitive to prompt wording. GPQA took a different route: questions written by doctoral-level specialists that skilled non-experts with web access cannot reliably answer, designed as a testbed for supervising systems that may exceed their overseers.

CEAF and coreference scoring (2005)

Luo proposed aligning gold and predicted entities one-to-one and scoring the best alignment, fixing MUC's tendency to reward over-merging. CEAF was later averaged with MUC and B-cubed to form the CoNLL coreference score; Moosavi and Strube (2016) later reopened the question.

WindowDiff (2002)

Pevzner and Hearst analysed the Pk measure for text segmentation, showed that it treats misses and false alarms unequally and punishes near misses too harshly, and proposed WindowDiff, which compares boundary counts in a sliding window. It is a model of how a careful critique of a metric can change practice in a subfield.

People

• Ondřej Bojar (Charles University) is the most prolific author on this topic in the library, having led or co-written the WMT findings papers through the 2010s, including WMT14, WMT16, WMT17 and WMT18, as well as the WMT17 metrics results.
• Philipp Koehn (Johns Hopkins University) co-founded the WMT evaluation tradition with Christof Monz (University of Amsterdam), beginning with their 2006 comparison of manual and automatic evaluation, and both remained organisers of the campaign for many years.
• Markus Freitag led the MQM study of human evaluation, the argument that references themselves are a source of error, and the WMT metrics tasks from 2021 to 2024.
• Alon Lavie (Carnegie Mellon University) connects two generations of metrics: he co-created METEOR and its successors, and later co-authored COMET.
• Ricardo Rei is the lead author of COMET and COMET-22, working with André F. T. Martins (Instituto de Telecomunicações and the University of Lisbon), who also co-authored xCOMET and quality-estimation shared tasks.
• Christian Federmann built and ran much of the human evaluation infrastructure behind WMT and the human parity study.
• Yvette Graham (Trinity College Dublin) developed direct assessment and studied the statistics of evaluation, including segment-level metric testing, summarisation metrics and statistical power and translationese.
• Barry Haddow (University of Edinburgh) co-organised WMT across the period covered here.
• Lucia Specia (Imperial College London) shaped quality estimation as a task and co-organised multimodal translation and simplification benchmarking.
• Tom Kocmi brought language models into MT evaluation with GEMBA-MQM and led the WMT24 general task findings.
• Matt Post (Microsoft) is known here for SacreBLEU and for co-organising several WMT campaigns.
• Nitika Mathur showed in Tangled up in BLEU how outlier systems inflate metric correlations, and led the WMT20 metrics task.
• Chris Callison-Burch organised early WMT evaluations, including the (meta-)evaluation of 2007 and the 2012 findings.
• Samuel R. Bowman co-led GLUE, SuperGLUE and GPQA, with Alex Wang as first author of both GLUE papers.
• Reut Tsarfaty (Bar-Ilan University) contributed to parsing evaluation, from robust dependency-parsing evaluation to the SPMRL shared task and the HeQ Hebrew reading benchmark.
• Nizar Habash (New York University Abu Dhabi) works on Arabic evaluation, including recent studies of diacritics and medical question answering.

Institutions and places

The institutions most represented reflect the two halves of the topic. European machine translation groups — Charles University in Prague, the University of Edinburgh, the University of Amsterdam, ETH Zurich, the German Research Centre for Artificial Intelligence and the Lisbon cluster around the Instituto de Telecomunicações — carry the WMT tradition and the learned-metric work. Large North American universities and industrial labs — Carnegie Mellon, Johns Hopkins, Stanford, the University of Washington, New York University, the Allen Institute for AI, Microsoft, Google and Amazon — dominate the benchmark side, especially for question answering and large language models. Chinese institutions, among them the Chinese Academy of Sciences and Zhejiang University, figure prominently in recent LLM and multimodal benchmarks, and the Mohamed bin Zayed University of Artificial Intelligence in multilingual evaluation.

By country, the United States leads, followed by the United Kingdom and China, then Germany, Czechia and Switzerland — the last two largely because of the WMT organisers.

The proceedings of the Language Resources and Evaluation Conference (LREC), whose very name names the topic, carry more of these papers than any other venue. They are followed by Computational Linguistics, the Transactions of the ACL, the main ACL and EMNLP conferences, and the yearly proceedings of the Conference on Machine Translation.

Open questions

Validity. Does a metric or benchmark measure the property it names? Reiter's structured review concluded that BLEU is supported for diagnosing machine translation systems but not for single texts or other tasks. Every newer metric faces the same question.

Saturation and shelf life. Benchmarks are overtaken ever faster. Building harder ones — expert-written, adversarial, multimodal — buys time but raises costs, and harder is not the same as more meaningful.

Contamination. When training data is undisclosed and drawn from the web, it is difficult to know whether a model has seen a test set. Proposed remedies include held-out canary strings, post-cutoff test material, dynamic benchmarks and contamination audits, none of them complete.

Who judges the judges. Learned metrics and LLM judges are trained on, or prompted with, data whose biases they inherit; they may prefer outputs from models like themselves. Their reliability has to be checked against human judgement, which is itself expensive and variable.

Human disagreement. For many tasks — offensive language, sentiment, summary quality — there is no single right answer. Treating disagreement as noise to be averaged away may discard exactly the information an evaluation should keep.

References. Reference-based metrics are only as good as their references. Work such as Freitag et al. (2020) and Dhingra et al. (2019) shows that poor or divergent references distort scores, which pushes the field toward reference-free and source-based assessment.

Languages beyond English. Most benchmarks and metrics were built for English or a few high-resource languages. Efforts such as FLORES, Belebele, BLEnD and IndicNLPSuite extend coverage, but translated test sets carry translationese and cultural mismatch.

What a single number hides. Aggregate scores conceal trade-offs between accuracy, robustness, fairness and cost. Multi-metric reporting, as in ERASER, and fine-grained error typing, as in ERRANT, are partial answers.

Neighbouring topics

Evaluation touches almost every topic on the site, since each needs its own measures.

• Machine translation is the closest neighbour, with the largest overlap; the machine translation shelf holds the wider literature.
• Language models & pretraining supplies the systems that most recent benchmarks test, and Reasoning supplies many of the hardest tasks.
• Question answering, Summarization, Dialogue systems and Text generation each have metric debates of their own.
• Multilingual & cross-lingual NLP and Low-resource languages contribute multilingual benchmarks; Semitic languages and Hebrew & Aramaic add Arabic and Hebrew test sets.
• Coreference resolution and Dependency parsing have long histories of scoring disputes.
• Vision & language and Speech & spoken language extend evaluation to images, video and audio.
• Bias, fairness & safety and Sentiment & emotion raise questions of disagreement and harm; Legal AI brings domain benchmarks; Interpretability evaluates explanations.

On the site's shelves, the topic stands mostly on Linguistics & languages, Machine translation and Natural language processing, with smaller parts on AI & machine learning and Mathematics & statistics.

Key works in this library

• BLEU (Papineni et al.) — the n-gram precision metric that started automatic MT evaluation.
• METEOR — alignment-based metric with stems, synonyms and a recall emphasis.
• chrF — character n-gram F-score that needs no tokeniser.
• Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics — the recall-based groundwork for ROUGE.
• A Call for Clarity in Reporting BLEU Scores — SacreBLEU and standardised metric signatures.
• A Structured Review of the Validity of BLEU — what the evidence does and does not support.
• COMET — a learned metric over a cross-lingual encoder.
• Results of WMT22 Metrics Shared Task — the recommendation to move from BLEU to neural metrics.
• Experts, Errors, and Context — expert MQM annotation as a new human gold standard.
• Has Machine Translation Achieved Human Parity? — the case for document-level human evaluation.
• Findings of the 2016 Conference on Machine Translation — the first WMT as a full conference.
• GLUE — the multi-task understanding benchmark with a single score.
• SuperGLUE — its harder successor after rapid saturation.
• Natural Questions — question answering from real search queries.
• Adversarial Examples for Evaluating Reading Comprehension Systems — distractor sentences that exposed shallow matching.
• DROP — reading comprehension that requires counting and arithmetic.
• Evaluating Large Language Models Trained on Code — HumanEval and the pass@k measure.
• XTREME — zero-shot cross-lingual transfer across forty languages.
• MTEB — a unified benchmark for text embeddings.
• A Survey on Evaluation of Large Language Models — a map of LLM evaluation.
• MMLU-Pro — a harder, less prompt-sensitive knowledge benchmark.
• GPQA — expert questions for oversight research.
• MMMU — college-level multimodal reasoning.
• Evaluating Object Hallucination in Large Vision-Language Models — the POPE probe for invented objects.
• On Coreference Resolution Performance Metrics — the CEAF entity-alignment score.