Multidimensional quality metrics: a flexible system for assessing translation quality

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/2013.tc-1.6.pdf

Summary

Assessment of translation quality has traditionally depended on inflexible, one-size-fits-all standards, for example SAE J2450 and the LISA QA Model. Human and machine translation were judged with unrelated methods: error counts on one side, reference-similarity scores like BLEU on the other. Defects in source texts were also often blamed on translators. Drawing on business quality theory (Garvin), the authors adopt a functionalist definition: a translation is good when it delivers the accuracy and fluency its audience and purpose require and meets the agreed specifications.

Within the EU-funded QTLaunchPad project, they built Multidimensional Quality Metrics (MQM). MQM is an open, hierarchical catalog of issue types, compiled by comparing more than twenty quality metrics, QA tools and MT evaluation approaches. Users pick the issues relevant to a task, assign severities and compute penalty scores normalized by word count.

Preliminary testing suggests three results:

• MQM can describe many existing error-count metrics.
• It works for both human and MT output.
• For MT, its results matched post-editing results while adding diagnostic detail.

The framework matters because it offers a shared, adaptable vocabulary for evaluating translation quality.

Key points

• The catalog held 108 issue types as of November 2013, arranged as a hierarchy. Child nodes are illustrative examples of their parent, not a complete list. Issues that fit no child can therefore be tagged at the parent level (e.g., Grammar).
• Sources for the catalog:
  • Metrics: LISA QA Model 3.1, SAE J2450, ISO CD 14080, SDL TMS Classic, ATA grading criteria and TAUS DQF.
  • Tools: Acrocheck, ApSIC XBench, CheckMate, LanguageTool, QA Distiller and XLIFF:doc.
  • MT measures: BLEU, METEOR, TER and others. These contributed little, because they rarely name specific error types.
• Five branches: Fluency, Accuracy, Verity (fit with real-world requirements), Design and Internationalization. Internationalization was still underspecified at the time.
• Scoring: the penalty is calculated as (minor + 5×major + 10×critical) ÷ word count. Penalties are summed within each branch, and both the multipliers and the severity levels are customizable. Two scorecards are offered: a Tabular Scorecard and a Simple Scorecard.
• Implementation and governance:
  • Tools: demo tools are planned under the Eclipse Public License. MQM markup is supported in the open-source editor translate5, and MQM code is available under GPL 3.0 with FLOSS exceptions.
  • Stewardship: the specification is being handed over to GALA's CRISP program.
  • Standardization: the authors plan to pursue it through W3C, OASIS and ISO.

Interesting ideas

• Master catalog with per-task selection. Rather than one fixed metric, keep a superset of issue types and assemble a task-specific subset. This approach suits any annotation scheme (e.g., NLP error taxonomies) that must serve different use cases while staying comparable.
• Fallback to the parent category. Treating children as non-exhaustive lets annotators label an error at the parent level when it fits no child. This keeps annotation consistent without forcing false precision.
• A "Verity" dimension. A text can be faithfully translated yet wrong for the target locale's real-world conditions. Separating such errors from translator errors assigns responsibility correctly and could apply to localization of data, trade documents or geographic content.
• Word-normalized, severity-weighted penalties with transparent math. Exposing every intermediate calculation and the adjustable weights in a scorecard makes quality scores auditable and tunable per project. The authors admit the default weights are not yet validated.
• Applying the same issue types to source texts. Running the same Fluency and Accuracy checks on the source flags upstream defects before translation. This could help clean corpora before building parallel data.