Elamur Research
☰
home / papers / 2005 / Syntactic Features for Evaluation of Machine Translation

Syntactic Features for Evaluation of Machine Translation

What it is about

The paper adds syntax to n-gram evaluation. The headword-chain metric (HWCM) matches the head-modifier paths of the reference against the hypothesis. The subtree metric (STM) matches constituent subtrees, and a dependency subtree variant (DSTM) matches head subtrees. All of them need a parser on the hypothesis. At sentence level, most syntactic metrics predict fluency better than BLEU, and HWCM predicts overall sentence judgments better too. At corpus level the kernel-based subtree metric reaches a correlation of about 0.88 with human overall judgments on the 2005 test systems, against about 0.78 for BLEU at its best n-gram length. The authors note that the kernel metric is weak at sentence level, so the two levels can disagree.

Ideas

For Mova

The places are tools/roundtrip (UD trees) and en/src/lin.rs (order). Mova does not need a parser on the output: the paper's cost is the parsing of noisy MT, and Mova can compare meta trees built from the references instead.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W05-0904.pdf

Summary

The paper addresses a known weakness of n-gram metrics such as BLEU and NIST. They align reasonably with human rankings of whole systems, but they judge individual sentences poorly and largely ignore grammaticality. Liu and Gildea parse both the MT hypotheses and the reference translations, then measure how much syntactic structure they share. They propose several metrics:

  • STM: overlap of constituent subtrees, grouped by depth.
  • TKM: a tree-kernel cosine similarity over all subtree shapes.
  • HWCM: overlap of headword chains, which are paths in dependency trees obtained from the constituent parses using Collins' head rules.
  • DSTM and DTKM: dependency-tree versions of STM and TKM.

The metrics are tested on Chinese-English MT output with human scores from the 2003 Johns Hopkins workshop and on the ACL05 MT workshop systems. They are compared with BLEU at both the sentence level and the corpus level. Syntactic metrics, especially the dependency-based ones, generally beat BLEU, and the kernel-based dependency metric gives the strongest corpus-level correlation. The work shows that structural similarity to references captures fluency that word n-grams miss.

Key points

  • One test set has 665 English sentences produced by a Chinese-English MT system. Each sentence has three reference translations and human judgments on a 1–5 scale from the 2003 JHU workshop.
  • At the corpus level, seven ACL05 workshop systems (E9, E11, E12, E14, E15, E17, E22) were scored.
    • The kernel version of DSTM reached a correlation of 0.875.
    • BLEU's best was 0.781, at n-gram length 5.
    • HWCM stayed close to 0.78 across chain lengths.
  • Sentence-level and corpus-level correlations did not always agree. The kernel-based metrics were weak per sentence but best over whole corpora.
  • In one example, a fluent hypothesis scored 0.511 with HWCM but only 0.084 with BLEU, and both human judges gave it a 4.
  • A parser is needed only for the target language, because hypotheses and references are always compared within the same language.

Interesting ideas

  • Arithmetic mean for sparse features. The authors average precision across subtree depths with an arithmetic mean instead of BLEU's geometric mean. A single empty level therefore cannot zero out the score, which suits any metric built on rare, structured features.
  • Dependency paths as n-grams. Treating paths in a dependency tree as "n-grams" captures head-modifier relations even when the words are far apart or reordered. This is attractive for free-word-order languages such as Ukrainian.
  • Compare two parses, not one. The method scores the similarity between a hypothesis parse and a reference parse rather than judging a single parse on its own. This sidesteps the parser's habit of assigning a plausible tree to bad input, and the same trick applies wherever parsers are used on noisy text.
  • Report both evaluation levels. Sentence-level correlation ignores any uniform shift in a system's scores, while corpus-level correlation does not. Metrics should therefore be validated at both levels before being trusted.
  • Tree kernels for structure similarity. Collins-Duffy convolution kernels with cosine normalisation give a structure similarity score in [0,1] over all subtree shapes without listing them explicitly. This makes them a reusable tool for comparing any kind of tree.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{liu2005syntactic,
  title = {{Syntactic Features for Evaluation of Machine Translation}},
  author = {Liu, Ding and Gildea, Daniel},
  booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
  year = {2005},
  url = {https://aclanthology.org/W05-0904/}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hiyan Alshawi, Srinivas Bangalore, and Shona Douglas. 2000. Learning dependency translation models as collections of finite state head transducers. Computational Linguistics, 26(1):45–60. find
  2. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence estimation for machine translation. Technical report, Center for Language and Speech Processing, Johns Hopkins University, Baltimore. Summer Workshop Final Report. find
  3. Eugene Charniak, Kevin Knight, and Kenji Yamada. 2003. Syntax-based language models for machine translation. In Proc. MT Summit IX. find
  4. Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in Neural Information Processing Systems. find
  5. Michael John Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. find our card
  6. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In In HLT 2002, Human Language Technology Conference, San Diego, CA. find
  7. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 304–311. find
  8. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI), Baltimore, MD, October. find
  9. Dekang Lin. 2004. A path-based transfer model for machine translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), pages 625–630, Geneva, Switzerland. find
  10. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the 2004 Meeting of the North American chapter of the Association for Computational Linguistics (NAACL-04), Boston. find
  11. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41th Annual Conference of the Association for Computational Linguistics (ACL-03). find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Conference of the Association for Computational Linguistics (ACL-02), Philadelphia, PA. find our card