Elamur Research⌕
☰
home / papers / 2004 / A Smorgasbord of Features for Statistical Machine Translation

A Smorgasbord of Features for Statistical Machine Translation

Summary

The paper asks whether richer linguistic knowledge, especially syntax, can fix typical errors of statistical Chinese-English MT. Such errors include missing verbs or content words, bad word order and wrong function words. The team started from a strong alignment template (phrase-based) system. They did not change the decoder. Instead they had it output large n-best lists and reranked these candidates with a log-linear model, adding new feature functions one at a time. The features came from many levels: words, POS tags, chunks, and full Treebank-style parses of source and target, including TAG elementary trees. Feature weights were tuned directly for BLEU on a development set. The best single feature was the IBM Model 1 score, which raised BLEU from 31.6 to 32.5. A greedy combination of features reached 32.9. None of the syntactic features gave a statistically significant gain on its own. The work matters for two reasons. It was the first large-scale attempt to plug multi-level syntax into a state-of-the-art phrase-based system. It also showed that reranking makes a convenient shared testbed for a big team.

Key points

  • Data:
    • Chinese-English data from recent evaluations.
    • Training corpus of about 170M English words.
    • Dev set of 993 sentences and test set of 878 sentences, each about 25K words.
    • Four English reference translations per sentence.
  • Baseline features:
    • Alignment template and word selection probabilities.
    • A phrase-level monotonicity feature.
    • Word and phrase penalties.
    • A count of LDC lexicon entries used.
    • Four trigram language models trained on different news corpora.
  • N-best lists:
    • The baseline produced 16,384 candidates per sentence with A* search.
    • Experiments used the top 1000 candidates.
  • Oracle analysis on 1000-best lists:
    • Optimal oracle reached 113% of the human BLEU score.
    • A round-robin oracle, which selects and scores with different references, reached 88.5%.
    • The 1-best output reached 79.2%.
  • Results:
    • Baseline BLEU was 31.6.
    • IBM Model 1 p(f|e) gave 32.5, the only clearly significant single feature.
    • Greedy feature combination gave 32.9 (+1.3 absolute).
    • Bootstrap 95% confidence intervals were about ±0.8.

Interesting ideas

  • I1.

    Rerank instead of changing the decoder. Rescoring n-best lists avoids redesigning search for every new long-range or sentence-level feature. Many people can then test many features in parallel on one common platform, which suits any team prototyping features for a generation task.

  • I2.

    Check the ceiling with oracles first. Before investing in features, measure how much a perfect reranker could gain. The round-robin variant picks candidates with one reference and scores them with the others, which gives a more honest bound than scoring with the same references used for selection.

  • I3.

    Run off-the-shelf taggers on system output with care. POS taggers applied to ungrammatical MT output tend to "repair" it, for example by inventing a verb where none exists. Tools trained on clean text can therefore mislead feature extraction on noisy or generated text.

  • I4.

    Combine features greedily. At each step, add whichever feature most improves the metric. This is a simple and transparent way to build an ensemble from many weak features, though the authors note that tuning many weights on a small dev set risks overfitting.

  • I5.

    Look for the real reasons behind negative results. The authors list plausible explanations for why syntax did not help:

    • mismatch between parser training data and MT output;
    • limited n-best size;
    • low sensitivity of BLEU to grammaticality;
    • parallel training data roughly a hundred times (two orders of magnitude) larger than the available treebanks.

    This checklist is useful when diagnosing why linguistic features fail in other pipelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Treebanks · Part-of-speech tagging · Machine translation

Cited by

143 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 134 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{och2004smorgasbord,
  title = {{A Smorgasbord of Features for Statistical Machine Translation}},
  author = {Och, Franz Josef and Gildea, Daniel and Khudanpur, Sanjeev and Sarkar, Anoop and Yamada, Kenji and Fraser, Alex and Kumar, Shankar and Shen, Libin and Smith, David and Eng, Katherine and Jain, Viren and Jin, Zhen and Radev, Dragomir},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1021/}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alshawi, Hiyan, Srinivas Bangalore, and Shona Douglas. 2000. Learning dependency translation models as collections of finite state head transducers. Computational Linguistics, 26(1):45–60. find
  2. Brown, Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  3. Gildea, Daniel. 2003. Loosely tree-based alignment for machine translation. In Proc. of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), Sapporo, Japan. find our card
  4. Ney, Hermann, M. Generet, and Frank Wessel. 1995. Extensions of absolute discounting for language modeling. In Proc. of the Fourth European Conf. on Speech Communication and Technology, Madrid, Spain. find
  5. Och, Franz Josef. 2003. Minimum error rate training in statistical machine translation. In Proc. of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), Sapporo, Japan. find
  6. Och, Franz Josef and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA. find our card
  7. Och, Franz Josef and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics. Accepted for Publication. find
  8. Och, Franz Josef, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In Proc. of the Joint SIGDAT Conf. on Empirical Methods in Natural Language Processing and Very Large Corpora, College Park, MD. find our card
  9. Schafer, Charles and David Yarowsky. 2003. Statistical machine translation using coercive two-level syntactic transduction. In Proc. of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP), Philadelphia, PA. find
  10. Vogel, Stephan, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In COLING ’96: The 16th Int. Conf. on Computational Linguistics, Copenhagen, Denmark. find
  11. Wu, Dekai and H. Wong. 1998. Machine translation with a stochastic grammatical channel. In COLING-ACL ’98: 36th Annual Meeting of the Association for Computational Linguistics and 17th Int. Conf. on Computational Linguistics, Montreal, Canada. find
  12. Yamada, Kenji and Kevin Knight. 2001. A syntax-based statistical translation model. In Proc. of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), Toulouse, France. find our card
  13. Yamada, Kenji and Kevin Knight. 2002. A decoder for syntax-based MT. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/och-2004-smorgasbord-features-statistical-machine-translation/