🔡 chrF
The character n-gram F-score (chrF) is an automatic metric for machine translation and other text generation. It compares a system output with a human reference by counting shared character sequences and combines character precision and recall into a recall-weighted F-score.
What it is
chrF is a way to score a translation automatically. You give it a machine output and one or more human reference translations. It returns a number that is higher the more short character sequences the two texts share. It belongs to the family of overlap metrics in machine translation evaluation, next to the older BLEU score (Papineni et al. 2002) and METEOR (Denkowski & Lavie 2014).
The difference is the unit of comparison. BLEU matches whole words. chrF matches character n-grams, such as "hou", "ous" and "use". A translation that gets a word's stem right but its ending wrong therefore still earns partial credit. Maja Popović proposed the metric in 2015. She presented it as a simple scorer that needs no external tools or linguistic resources and does not depend on the language or on how the text is tokenized (Popović 2015).
The key idea
The F-beta formula
For every n-gram order n from 1 up to a maximum N, the hypothesis and the reference are cut into overlapping character n-grams. Counts are clipped as in BLEU, so a hypothesis n-gram is matched at most as often as it occurs in the reference (Papineni et al. 2002). Precision and recall are then averaged over the orders and combined:
chrP = (1/N) · Σₙ matchesₙ / hyp_ngramsₙ
chrR = (1/N) · Σₙ matchesₙ / ref_ngramsₙ
chrFβ = (1 + β²) · chrP · chrR / (β² · chrP + chrR)
The symbols mean the following:
- matchesₙ is the clipped number of character n-grams of order n that the hypothesis shares with the reference.
- hyp_ngramsₙ and ref_ngramsₙ are the totals of such n-grams in the hypothesis and in the reference.
- chrP and chrR are the averaged character precision and recall.
- β is the weight that sets how much more recall matters than precision, following the classic F-measure of van Rijsbergen (1979).
The 2015 paper tested 4-, 6- and 10-grams. It also tried a variable length tied to the average word length. 6-grams worked best, and treating the space as a separate character gave no gain (Popović 2015).
The paper compared β = 1 with β = 3, and chrF3 correlated best with the human rankings of the time (Popović 2015). The Conference on Machine Translation (WMT) then switched to direct assessment, in which human raters give each translation an absolute quality score. β was re-tested against these scores. β = 2 and β = 3 tied on average, and β = 2 became the default because a heavier recall weight can push tuned systems towards overly long outputs (Popović 2017).
Length bias from tuning can also run the other way. The Edinburgh system for the 2005 International Workshop on Spoken Language Translation (IWSLT) was tuned towards BLEU, which favours precision. Its output came out too short, and this badly hurt its scores under NIST, a related n-gram metric (Koehn et al. 2005).
Why characters help with rich morphology
In languages with rich morphology, one lemma appears in many word forms. A word-level metric treats "domu" and "dom" in Czech as unrelated. Character n-grams see that they share most of their material. The 2015 study covered Hindi, Czech, Russian, German, French, Spanish and English. It argued that an inexpensive character-based metric is especially valuable when translating into morphologically rich languages (Popović 2015).
The same intuition underlies character n-gram representations outside evaluation: CHARAGRAM embeddings handled rare words, misspellings and morphological variants well (Wieting et al. 2016).
A small worked example
Take the reference "houses" and the hypothesis "house". To keep the arithmetic short, use only orders 1 and 2 (real chrF goes to 6).
| Order | Hypothesis n-grams | Reference n-grams | Matches | Precision | Recall |
|---|---|---|---|---|---|
| 1 | h o u s e (5) | h o u s e s (6) | 5 | 5/5 | 5/6 |
| 2 | ho ou us se (4) | ho ou us se es (5) | 4 | 4/4 | 4/5 |
Averaging gives chrP = 1 and chrR ≈ 0.817. With β = 2, chrF2 = 5 · 1 · 0.817 / (4 · 1 + 0.817) ≈ 0.85.
An exact-match word metric such as BLEU would give this pair zero, because "house" ≠ "houses". chrF instead registers a near miss that is penalised mainly on the recall side.
chrF++: adding word n-grams
In 2017 Popović scored character and word n-gram orders separately against direct assessment on WMT-15 and WMT-16 data. Character 3- and 4-grams were the strongest single orders. Word unigrams and bigrams correlated well, while word 3- and 4-grams tended to underrate good translations (Popović 2017).
chrF++ therefore averages character 1–6-grams with word 1–2-grams under uniform weights. On that WMT-15/16 data, this raised segment-level correlation with direct assessment above plain chrF. A unigram-only variant, chrF+, did best for English→Russian, though that test set was small (Popović 2017).
History and founding papers
Before 2015, character n-grams appeared only inside larger combined metrics such as BEER and MTeRater. They had not been tested as a metric in their own right (Popović 2015). The field's reference points were BLEU, with its clipped word precisions combined by a geometric mean (Papineni et al. 2002), and paired bootstrap resampling for judging whether score differences are significant (Koehn 2004). A year earlier, Meteor Universal had tried to carry a richer metric to languages without stemmers or WordNet (Denkowski & Lavie 2014).
The founding paper scored all WMT 12–14 systems. In system-level Pearson correlation with human judgments on WMT 14, chrF3 reached 0.857, against 0.845 for BLEU. For translation out of English, its average segment-level Kendall τ rank correlation beat even the best WMT 14 metrics (Popović 2015). Compared direction by direction, chrF3 agreed with human judgments better than BLEU in 80% of translation directions (Popović 2015).
Meta-evaluation itself was changing during these years. Graham and colleagues replaced relative rankings with repeated absolute adequacy scores. They also used the Williams test, which tells whether one metric's correlation with humans is significantly higher than a rival's (Graham et al. 2015). chrF++ came out of re-testing chrF against direct assessment scores, the newer official WMT judgment type (Popović 2017). Discourse-aware metrics explored a different axis. DISCOTKparty added discourse-tree similarity to lexical metrics and ranked first in the WMT14 metrics task at both system and segment level, as a later journal account describes (Joty et al. 2017).
How it is used today
In WMT and toolkits
By WMT18, chrF++ was one of the standard n-gram baselines against which learned metrics such as RUSE were compared (Shimanaka et al. 2018). Popović released a Python script that reads raw text and supports multiple references. It uses β = 2 by default and reports segment scores plus micro- and macro-averaged document scores (Popović 2017).
Later meta-evaluations scored chrF side by side with BLEU computed through the standardised sacreBLEU implementation (Mathur et al. 2020). One reanalysis of WMT 2019 found that BLEU and TER make wrong pairwise system decisions more often than chrF. Its authors recommended replacing BLEU with chrF, YiSi-1 or ESIM (Mathur et al. 2020). chrF also sits inside composite baselines: one domain-robustness study averaged BLEU, chrF and TER as its surface-form reference point (Zouhar et al. 2024).
Beside neural metrics
Trained neural metrics now set the pace in evaluation benchmarks. xCOMET reported state-of-the-art results at segment, system and error-span level (Guerreiro et al. 2024). LLM judges have long trailed trained metrics such as MetricX and xCOMET, most clearly per segment (Feng et al. 2025). This line runs from multilingual zero-shot paraphrase scoring with Prism (Thompson & Post 2020) and unified source/reference evaluators such as UniTE (Wan et al. 2022) to span-tagging metrics such as MaTESe (Perrella et al. 2022) and xCOMET (Guerreiro et al. 2024).
Large language models have added generative evaluators:
- AutoMQM, a prompt that has the model list and grade error spans (Fernandes et al. 2023)
- EAPrompt (Error Analysis Prompting), which first identifies errors and then counts them (Lu et al. 2024)
- M-MAD, a multi-agent debate method (Feng et al. 2025)
- Remedy-R, an evaluator trained with reinforcement learning rather than prompted (Tan et al. 2026)
chrF remains useful because it is deterministic, fast and free of a trained model. That matters for low-resource languages, for reproducibility, and as a sanity check next to neural scores. Overlap metrics also travel to neighbouring tasks. Older MT metrics such as BLEU, TER and METEOR, used as the only features, made a strong paraphrase-identification baseline (Madnani et al. 2012).
Open questions
- Semantic depth. Lexical metrics, chrF included, lack the semantic depth needed to judge non-literal translation of slang, idioms and poetry (Tian et al. 2026). In chat translation, reference-based neural metrics correlated best with human Multidimensional Quality Metrics (MQM) scores, which are derived from annotated errors (Agrawal et al. 2024).
- Small score gaps. In WMT 2019 data, a 1–2 point BLEU gain matched a real human-judged improvement only about half the time (Mathur et al. 2020). Whether chrF differences of similar size are more trustworthy is not settled, so significance testing remains advisable (Koehn 2004).
- Test-set artefacts. Some past WMT test sets were partly reverse-created, with their sources translated from the target side. When these sets were split into forward- and reverse-created halves, the correlation between the BLEU system rankings on the two halves fell as low as 0.106 (Graham et al. 2020). Benchmark contamination can also inflate surface-overlap scores (Tan et al. 2026). A string-matching metric cannot detect either problem by itself.
- Modality and granularity. Speech translation is still scored mostly with text-only metrics (Zarzu & Zouhar 2026). Error-based human schemes measure effort in ways a single F-score does not capture. One example, tested on Arabic, orders post-editing error classes by the mental effort needed to fix them (Temnikova et al. 2016).
- The role of β and word n-grams. It is unclear whether these settings should change for scripts written without spaces, or for systems built on subword tokenization. The question has received little systematic study since the 2017 re-tuning (Popović 2017).
Key papers
Related concepts
- BLEU score — BLEU score (Bilingual Evaluation Understudy) is an automatic metric for machine translation. It rates a candidate text by how many of its…
- Morphology — Morphology is the branch of linguistics that studies the internal structure of words. It covers how words are built from meaningful units…
More papers
Topics: 🔁 Machine translation · 📏 Evaluation & benchmarks
Shelves: 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/chrf/