Elamur Research⌕
☰
home / papers / 2007 / Improving Translation Quality by Discarding Most of the Phrasetable

Improving Translation Quality by Discarding Most of the Phrasetable

Summary

Phrase-based statistical machine translation systems extract huge phrasetables from word-aligned parallel corpora. Many of the entries look wrong or are never used, yet crude size reductions usually cost BLEU. Johnson, Martin, Foster and Kuhn (NRC Canada) propose filtering each phrase pair with a significance test.

For every pair they build a 2×2 contingency table from sentence-level co-occurrence counts. They then compute a p-value with Fisher's exact test, made practical with Moore's (2004) computational shortcuts. Pairs whose negative-log-p-value falls below a threshold are discarded.

Experiments use the WMT06 Europarl shared-task data (French, Spanish and German into English) plus larger Chinese–English corpora. They show that up to about 90% of a phrasetable can be dropped without hurting BLEU. Without phrasetable smoothing, pruning even raises BLEU. With strong smoothing the gain mostly disappears, but the size savings remain. On bigger corpora the fraction that can be removed shrinks.

The work matters because smaller tables need less memory and processing time. Those resources can then go into richer features or better search.

Key points

  • Baseline system:
    • Phrases come from symmetrized IBM model 2 alignments using the Koehn et al. (2003) induction procedure, with at most 8 tokens per side.
    • A log-linear model is tuned with Och's method on BLEU.
    • The language model is a 4-gram Kneser-Ney model built with SRILM.
    • For each source phrase, only the top 30 translations by p(t̃|s̃) are kept.
  • Pruning criterion: Fisher's exact test (hypergeometric distribution) is applied to counts of sentence pairs containing s̃, t̃, or both. Thresholds are expressed as negative-log p-values, so a larger number means more pruning.
  • Special case: a "1-1-1" pair occurs once, and each of its sides also occurs only once. Its p-value is 1/N, which gives a natural threshold α = log(N).
    • A threshold just above α removes every pair with C(s̃,t̃)=1, which roughly reproduces the common practice of dropping singleton pairs.
  • Results:
    • Up to about 90% of the table can be removed with no BLEU loss.
    • Removing 80% did no harm even when smoothing was used.
    • BLEU improved when phrasetable smoothing (Foster et al., 2006) was absent.
  • Scale effect: in Chinese–English large-corpus runs, aggressive pruning still preserved quality, but a smaller percentage of the table was prunable.

Interesting ideas

  • I1. Test association against chance, per item. Asking whether an aligned pair could plausibly co-occur at random is a simple, principled filter. The same idea could clean bilingual lexicons, cross-lingual term lists, or collocation candidates extracted from Ukrainian corpora.
  • I2. Use an exact test for small counts. Fisher's exact test is preferable to chi-squared or log-likelihood when counts are tiny, as in sparse, long-tail data. Log-gamma approximations and the hypergeometric recurrence keep it cheap. Moore's results suggest log-likelihood as a cheaper approximation.
  • I3. Work in negative-log p-values. This avoids underflow with extremely small probabilities and gives an intuitive "bigger means stricter" knob. The derived threshold α = log(N) ties the cutoff directly to corpus size.
  • I4. Treat pruning and smoothing as related. Both push the model away from rare long phrases toward combinations of frequent short ones. Comparing a pruning method with and without smoothing reveals whether its gains overlap with existing regularization.
  • I5. Reuse the statistic beyond filtering. The significance score can serve as a diagnostic of phrasetable quality, for example for morphologically rich languages. It can also act as an extra model feature, or be applied to patterns with gaps such as skip-n-grams.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

107 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{johnson2007improving,
  title = {{Improving Translation Quality by Discarding Most of the Phrasetable}},
  author = {Johnson, Howard and Martin, Joel and Foster, George and Kuhn, Roland},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1103/}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Agresti. 1996. An Introduction to Categorical Data Analysis. Wiley. find
  2. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter estimation. Computational Linguistics, 19(2):263–312, June. find
  3. Philipp Koehn 2003. Europarl: A Multilingual Corpus for Evaluation of Machine Translation. Unpublished draft. see http://www.iccs.inf.ed.ac.uk/˜pkoehn/publications/europarl.pdf George Foster, Roland Kuhn, and Howard Johnson. 2006. Phrasetable Smoothing for Statistical Machine Translation. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia. ↗
  4. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 1995, pages 181–184, Detroit, Michigan. IEEE. find
  5. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Eduard Hovy, editor, Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 127–133, Edmonton, Alberta, Canada, May. NAACL. ↗
  6. Robert C. Moore. 2004. On Log-Likelihood-Ratios and the Significance of Rare Events. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain. find
  7. NIST. 2006. NIST MT Benchmark Test. see http://www.nist.gov/speech/tests/mt/ ↗
  8. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41th Annual Meeting of the Association for Computational Linguistics(ACL), Sapporo, July. find
  9. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: A method for automatic evaluation of Machine Translation. Technical Report RC22176, IBM, September. find our card
  10. NAACL Workshop on Statistical Machine Translation. 2006. see http://www.statmt.org/wmt06/ ↗
  11. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP) 2002, Denver, Colorado, September. find
  12. Richard Zens and Hermann Ney. 2004. Improvements in phrase-based statistical machine translation. In Proceedings of Human Language Technology Conference / North American Chapter of the ACL, Boston, May. Table 5: WMT06 Results: BLEU by type of smoothing and pruning threshold fr −→ en es −→ en threshold phrasetable % relative frequency: no smoothing 100% 25.39 27.26 20.74 84–88% 25.97 27.81 21.08 63–68% 26.32 28.00 21.27 14–17% 26.34 28.27 21.22 13–15% 26.36 28.50 21.14 11–13% 26.51 28.45 21.36 8–10% 26.50 28.38 21.28 4–5% 26.26 27.88 20.87 2% 25.66 27.07 20.07 0.2% 20.49 21.66 15.23 none 10 α −  α +  15 20 25 50 100 1000 Good-Turing 28.14 28.33 28.63 28.49 28.03 28.45 28.56 27.78 27.07 21.66 none 10 α −  α +  15 20 25 50 100 1000 100% 84–88% 63–68% 14–17% 13–15% 11–13% 8–10% 4–5% 2% 0.2% 25.96 26.33 26.54 26.24 26.48 26.65 26.54 26.26 25.70 20.49 Kneser-Ney (3 parameter) 26.89 28.70 21.78 26.79 28.78 21.71 26.49 28.69 21.34 26.73 28.67 21.54 26.84 28.70 21.29 26.44 28.16 20.93 25.72 27.27 20.11 20.48 21.70 15.28 none 10 15 20 25 50 100 1000 100% 84–88% 13–15% 11–13% 8–10% 4–5% 2% 0.2% Zens-Ney 29.07 29.00 28.67 28.47 28.79 27.96 27.34 21.76 26.87 26.81 26.92 26.93 26.85 26.51 25.82 20.50 none 10 15 20 25 50 100 1000 100% 84–88% 13–15% 11–13% 8–10% 4–5% 2% 0.2% Table 6: Chinese Results: BLEU by pruning threshold threshold phrasetable % nist04 Zens-Ney Smoothing applied to all phrasetables 100% 32.14 30.69 13.06 40–65% 32.66 31.14 13.11 22–38% 32.73 30.97 13.14 21–36% 31.56 30.45 12.49 20–35% 32.00 30.73 12.50 18–31% 30.54 29.58 11.68 none 14 16 18 20 25 also pruning C(s̃, t̃) = 1 23–39% 32.08 30.99 12.75 de −→ en en −→ fr en −→ es en −→ de 27.29 27.82 28.11 28.16 28.20 28.28 28.32 28.05 27.38 22.51 27.17 27.71 28.09 28.08 28.18 28.06 27.97 27.90 27.11 22.31 14.71 15.09 15.19 15.24 15.29 15.28 15.25 15.08 14.66 11.36 21.17 21.38 21.50 21.15 21.21 21.41 21.31 20.94 20.12 15.52 27.84 28.18 28.36 28.22 28.27 28.36 28.35 28.07 27.41 22.53 27.95 28.27 28.39 28.16 28.21 28.14 28.04 27.95 27.13 22.31 15.13 15.22 15.31 15.28 15.31 15.25 15.28 15.08 14.66 11.37 28.64 28.63 28.60 28.56 28.54 28.17 27.50 22.58 28.71 28.41 28.57 28.44 28.21 28.05 27.26 22.36 15.50 15.35 15.52 15.41 15.42 15.17 14.58 11.33 28.75 28.72 28.79 28.69 28.59 28.30 27.57 22.68 28.54 28.52 28.32 28.42 28.27 27.96 27.30 22.33 15.50 15.54 15.44 15.45 15.37 15.27 14.51 11.56 21.55 21.65 21.74 21.72 21.58 20.96 20.02 15.46 nist05 nist06-GALE nist06-NIST 27.97 28.35 28.00 27.03 27.33 26.12 27.66 find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/johnson-2007-improving-translation-quality-discarding-most/