chrF++: words helping character n-grams
What it is about
The paper asks whether word n-grams add anything to the character chrF score, and whether chrF tracks direct human assessment (absolute scores per segment) as well as it tracks the relative rankings used earlier. It uses the WMT 2015 and 2016 outputs with direct assessments for translation into English and into Russian, and Pearson correlation at segment level. First, β = 2 is confirmed as the best recall weight for both character and word F-scores, with β = 2 and β = 3 close. Then single n-gram orders are compared: word 1-grams and 2-grams correlate well, while word 3- and 4-grams make the word score too pessimistic, and character 3- to 6-grams are the strongest character orders. Adding word unigrams and bigrams to the six character n-grams raises the mean correlation over the test sets (about 0.58 against 0.57 for plain chrF and 0.53 for the word score). The paper names the combination chrF++ and notes that unigrams alone (chrF+) are best for Russian, and that the test sets are too small to decide between the two.
Ideas
- I1. Add word unigrams and bigrams to the character counts in
tools/translate/src/measure.rsto get the chrF++ form. Drop word 3- and 4-grams, since the paper shows they hurt. Split words on whitespace and keep the counts inBTreeMap. - I2. Word bigrams test word order, which the character score mostly misses. For the translator this is a direct probe of the linearizer in
en/src/lin.rs: a change in word order should move the bigram part of the score and leave the character part alone. - I3. Report chrF, chrF+ and chrF++ per target language. The gain in the paper was clearest for Russian, a morphologically rich target, so Hebrew-to-English and English-to-Hebrew results may differ and should not be averaged together.
- I4. Build a small direct-assessment set of about one hundred verses scored by a person, as the paper's preliminary evidence suggests. Without one, the claim that chrF++ tracks quality is untested for Mova.
For Mova
The code is tools/translate/src/measure.rs, with the word part reading the output of the linearizer. The paper gives a cheap normal-mode measure that is sensitive to word order, which the translator currently lacks outside exact mode. The first run is chrF++ against chrF on the dev split, run twice, with the word orders named in the output.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-4770.pdf
Summary
Character n-gram F-score (chrF) is a known strong automatic metric for machine translation, particularly for morphologically rich target languages, but earlier validation relied on relative rankings (RR). Its behaviour against direct human assessment (DA), the official WMT-17 judgment type, had not been studied systematically. Popović computes segment-level Pearson correlations between DA and several n-gram F-scores on all WMT-15 and WMT-16 system outputs. The data consist of English-target translations whose source texts were in Turkish, Romanian, Russian, Czech, Finnish, or German, along with the English→Russian direction. She first re-tests the recall weight β, then scores each character and word n-gram order separately, and finally tries combinations. Word F-score built on 3- and 4-grams tends to underrate good translations, while short word n-grams correlate well. Averaging word unigrams and bigrams together with character 1–6-grams gives chrF++, which improves on plain chrF. A unigram-only variant, chrF+, works best for Russian. The work yields a simple, tokenisation-free, language-independent metric aligned with WMT's newer evaluation protocol.
Key points
- Choice of β: β = 2 is confirmed for DA. Mean Pearson for chrF is .562 at β=1, .574 at β=2, and .574 at β=3. For wordF the means are .519, .526, and .528. β=2 was preferred because heavier recall weighting can cause overly long outputs in tuning.
- Individual n-grams: Character 3- and 4-grams are strongest (mean .579 each). Among word n-grams, unigrams (.537) and bigrams (.531) clearly beat 3-grams (.481) and 4-grams (.414).
- Definition of chrF++: Character n-grams up to N=6 are combined with word n-grams up to N=2 and averaged uniformly. Other weighting schemes gave no consistent gains.
- Russian and chrF+: For English→Russian, chrF+ (word unigrams only) correlated best. The test set was too small to draw firm conclusions.
- Tool: A Python script is available at github.com/m-popovic/chrF. It takes raw text, supports multiple references, uses β=2 by default (adjustable), and reports segment-level scores plus micro- and macro-averaged document-level scores.
Cite
@inproceedings{popovic2017chrf,
title = {{chrF++: words helping character n-grams}},
author = {Popović, Maja},
booktitle = {Proceedings of the Second Conference on Machine Translation},
year = {2017},
doi = {10.18653/v1/w17-4770},
url = {https://doi.org/10.18653/v1/w17-4770}
}References
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In Proceedings of the ACL-05 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. Ann Arbor, MI, pages 65–72. find
- Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation (WMT-16). Berlin, Germany, pages 199–231. find our card