📄 Evaluating Machine Translation Output with Automatic Sentence Segmentation

Summary

Standard MT metrics such as WER, PER, BLEU and NIST assume that the system output has exactly as many segments as the reference, aligned one-to-one. In speech translation this rarely holds, because ASR segments the input automatically and the MT system may split or merge units. Matusov, Leusch, Bender and Ney (RWTH Aachen) address this by re-segmenting the MT output to match the reference boundaries. Their algorithm computes a Levenshtein alignment between the whole candidate document and the references, then transfers the reference segment boundaries onto the candidate words. A dynamic-programming formulation, borrowed from connected-word speech recognition, extends this to multiple references. The result is a new measure, AS-WER, and the re-segmented output can also be scored with AS-BLEU, AS-NIST and similar metrics. On TC-STAR and IWSLT data, scores computed on automatic segmentation are close to those computed on manual segmentation and correlate with human judgement at least as well. Naive document-level scoring, by contrast, inflates quality. This makes realistic evaluation of speech-to-text translation possible without forcing ASR to follow manual boundaries.

Key points

• The method accepts output with wrong segment boundaries or completely unsegmented text, and it handles several reference translations.
• Multiple references are equalized for the alignment step:
  • Each shorter reference segment is padded with "empty word" symbols ($) to the length of the longest reference.
  • An artificial segment-end symbol is inserted at every boundary, so the method does not depend on punctuation.
• On TC-STAR (Table 3), BLEU vs. AS-BLEU is almost identical: 51.1 vs. 50.9 for system A and 40.6 vs. 40.6 for system D. NIST behaves the same way (10.34 vs. 10.29 for system A).
• Document-level scoring without segmentation overestimates quality, for example BLEU 55.3 and NIST 11.57 for system A. It also shrinks differences between systems: the relative B–D gap is about 6% at document level, compared with about 15% under correct segmentation.
• The segmentation error rate is defined as the WER between the manually and automatically segmented versions of the same candidate. It stays below 10% for all systems and grows only slightly as translation WER rises, even for systems with WER around 50% or higher.

Interesting ideas

• I1. Project boundaries through an alignment instead of predicting them. Aligning the full output to the reference at document level and copying the reference boundaries across could help evaluate any generation task whose output units do not match the gold units. Examples include ASR-then-summarize pipelines and subtitle generation.
• I2. Pad references to a common length. Filling shorter references with empty-word tokens and adding explicit end-of-segment symbols lets one alignment pass handle many references at once. It also removes reliance on punctuation, which is useful for noisy or unpunctuated text.
• I3. Measure the evaluation tool itself. The segmentation error rate checks how far the automatic boundaries drift from the manual ones. Validating an auxiliary preprocessing step against gold data before trusting downstream scores is a habit worth copying.
• I4. Document-level scoring is a misleading shortcut. Bag-of-words and n-gram metrics computed over an entire document match words across distant, unrelated positions. This inflates scores and hides differences between systems, a warning that applies to any corpus-level metric.
• I5. Reuse dynamic-programming designs from neighbouring fields. The algorithm separates within-segment alignment from recombination at segment boundaries, much like connected-word recognition. This suggests that classic ASR decoding structures can solve alignment problems in other NLP evaluation settings.