Elamur ResearchβŒ•
☰
home / papers / 2009 / Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases

Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases

Summary

Statistical machine translation (SMT) systems often leave source words and phrases untranslated when the parallel training data is small. Earlier work filled these gaps with paraphrases obtained by "pivoting" through other languages, but pivoting itself needs extra bitexts. Marton, Callison-Burch and Resnik instead derive paraphrases from monolingual source-language text using distributional similarity. For each out-of-vocabulary phrase, they build a co-occurrence profile from a 6-token flat window. They then look for other phrases that appear in similar contexts and score them with a vector similarity measure such as cosine. The best-scoring candidates above a threshold are added to the phrase table. In Spanish-to-English experiments with small bitext subsets (10k, 20k, 80k), they used monolingual corpora built from wmt09, Acquis and afp data. On the 10k setting the approach gave a statistically significant BLEU gain, and TER also dropped. Gains shrank as the bitext grew. The authors present this as the first successful use of collocational, monolingually derived paraphrases in an end-to-end state-of-the-art SMT system. It matters for low-resource languages, where large monolingual text exists but parallel corpora do not.

Key points

  • Motivation from earlier work. Callison-Burch et al. (2006) showed severe vocabulary gaps: a 10,000-word training corpus covered only about 10% of the source vocabulary, and even about 10M words reached only about 90% coverage.
  • Building phrase profiles. Profiles count all collocates inside a 6-token sliding window without position information. Very frequent phrases are capped at t = 10000 sampled occurrences, and the counts are rescaled by count(phr)/t.
  • Main result (Spanish-to-English, 10k bitext).
    • Baseline: 23.78 BLEU.
    • With wmt09+Acquis paraphrases (1–4-grams, minScore .6): 24.13 BLEU, significant at p < 0.0001 under Koehn's pair-wise bootstrap test.
    • Using unigram paraphrases only gave 24.11.
  • Larger bitexts.
    • At 20k, BLEU moved slightly, from 24.68 to 24.75–24.77.
    • At 80k, BLEU stayed roughly flat (27.89 vs. 27.82–27.84), while TER decreased slightly.
  • Paraphrase quality issues (qualitative analysis).
    • The top candidates from the monolingual method sometimes include antonyms, e.g. "en contra del informe" for "a favor del informe". This improves with bigger corpora.
    • Pivoting paraphrases tend to rank frequent function words highly.

Interesting ideas

  • I1. Paraphrase lookup as an OOV fallback. Instead of finding translations directly in comparable data, the method maps unknown source phrases to known source phrases that have similar contexts, then reuses their existing translations. This only requires monolingual text, so it suits languages with almost no parallel data, such as many low-resource language pairs.
  • I2. Capped sampling for very frequent phrases. Collecting at most a fixed number of occurrences and scaling the counts back up keeps profile construction cheap. It avoids the huge precomputed collocation matrices that made earlier distributional approaches hard to scale, and the same trick fits any large-corpus co-occurrence statistic.
  • I3. Treating the similarity threshold as a tunable parameter. The authors stress a trade-off: a high cutoff loses coverage, while a low cutoff lets in bad paraphrases. They suggest tuning the cutoff on held-out data, which is a sensible pattern for any pipeline that injects automatically mined resources.
  • I4. Watching for antonyms. Distributional similarity tends to rank antonyms highly, so filtering them out matters wherever context vectors stand in for synonymy. Remedies the authors propose include sense disambiguation (WordNet, thesauri, context), syntax, and larger corpora.
  • I5. Evaluating on artificially small bitext subsets. Simulating low-resource conditions with 10k/20k/80k subsets shows where an augmentation technique helps and where its benefit fades. This is a cheap evaluation design for any low-density language project.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation Β· Natural language processing

Topics: Machine translation

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 63 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marton2009improved,
  title = {{Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases}},
  author = {Marton, Yuval and Callison-Burch, Chris and Resnik, Philip},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  doi = {10.3115/1699510.1699560},
  url = {https://doi.org/10.3115/1699510.1699560}
}

References

37 references, 9 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of ACL-2001. find our card
  2. P.F. Brown, S.A.D. Pietra, V.J.D. Pietra, and R.L. Mercer. 1993. The mathematics of statistical machine translation. Computational Linguistics, 19(2):263– 313. find
  3. Lou Burnard. 2000. Reference Guide for the British National Corpus. Oxford University Computing Services, Oxford, England, world edition edition. find
  4. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings NAACL-2006. find
  5. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of EMNLP 2008, Waikiki, Hawai’i. find our card
  6. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL-05, pages 263–270. find our card
  7. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. β†— our card
  8. Mona Diab and Steve Finch. 2000. A statistical word-level translation model for comparable corpora. In Proceedings of the Conference on Content-Based Multimedia Information Access (RIAO). β†—
  9. B. Dolan, C. Quirk, and C. Brockett. 2004. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics of the Association for Computational Linguistics, Geneva, Switzerland. find our card
  10. T. Dunning. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find our card
  11. Katrin Erk and Sebastian PadΓ³. 2008. A structured vector space model for word meaning in context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2086), pages 897–906, Honolulu, HI. find
  12. John R. Firth. 1957. A synopsis of linguistic theory 1930UΜ‹55. Studies in Linguistic Analysis, (special volume of the Philological Society):1–32. Distributional Hypothesis. find
  13. Pascale Fung and Percy Cheung. 2004. Multi-level bootstrapping for extracting parallel sentences from a quasi-comparable corpus. In Proceedings of the 20th international conference on Computational Linguistics, page 1051, Geneva, Switzerland. Association for Computational Linguistics. find
  14. Pascale Fung and Lo Yuen Yee. 1998. An ir approach for translating new words from nonparallel, comparable texts. In Proceedings of COLING-ACL98, pages 414–420, Montreal, Canada. find
  15. Zellig S. Harris. 1940. Review of louis h. gray, foundations of language (new york: Macmillan, 1939). Language, 16(3):216UΜ‹231. find
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT-NAACL, pages 127–133. β†—
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session, Prague, Czech Republic. find
  18. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. EMNLP. find
  19. Philipp Koehn. 2004b. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of AMTA. find
  20. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit. find
  21. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, pages 296–304, San Francisco, CA. find
  22. Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation. find
  23. Yuval Marton, Saif Mohammad, and Philip Resnik. 2009. Estimating semantic distance using soft semantic constraints in knowledge-source / corpus hybrid models. In Procedings of EMNLP, Singapore. find
  24. S. McDonald. 2000. Environmental determinants of lexical processing effort. Ph.D. thesis, University of Edinburgh. find
  25. Saif Mohammad and Graeme Hirst. 2006. Distributional measures of concept-distance: A task-oriented evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2006), Sydney, Australia. find
  26. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477–504. β†— our card
  27. Doug Oard, David Doermann, Bonnie Dorr, Daqing He, Phillip Resnik, William Byrne, Sanjeeve Khudanpur, David Yarowsky, Anton Leuski, Philipp Koehn, and Kevin Knight. 2003. Desperately seeking cebuano. In Proceedings of HLT-NAACL. find
  28. Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting of the ACL, pages 440–447. find
  29. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of ACL. find our card
  30. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the ACL, pages 160–167. find
  31. Kishore Papineni, Salim Roukos, Todd Ward, John Henderson, and Florence Reeder. 2002. Corpus-based comprehensive and diagnostic MT evaluation: Initial Arabic, Chinese, French, and Spanish results. In Proceedings of the ACL Human Language Technology Conference, pages 124–127, San Diego, CA. find
  32. Reinhard Rapp. 1999. Automatic identification of word translations from unrelated english and german corpora. In Proceedings of the 37th Annual Conference of the Association for Computational Linguistics., pages 519–525. find
  33. Philip Resnik and Noah Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349–380. β†— our card
  34. Philip Resnik. 1999. Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research (JAIR), 11:95–130. β†—
  35. Hinrich Schuetze and Jan O. Pedersen. 1997. A cooccurrence-based thesaurus and two applications to information retreival. Information Processing and Management, 33(3):307UΜ‹318. find
  36. Matthew Snover, Bonnie J. Dorr, Richard Schwartz, John Makhoul, Linnea Micciulla, and Ralph Weischedel. 2005. A study of translation error rate with targeted human annotation. Technical Report LAMP-TR-126, CS-TR-4755, UMIACS-TR-2005-58, University of Maryland, July, 2005. find
  37. Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter. In Fourth SIGHAN Workshop on Chinese Language Processing. find

↑ back to top

This page in Markdown β†’ https://md.research.elamur.ai/free/papers/marton-2009-improved-statistical-machine-translation-monolingually/