Elamur ResearchβŒ•
☰
home / papers / 2017 / Paraphrasing Revisited with Neural Machine Translation

Paraphrasing Revisited with Neural Machine Translation

Summary

Bilingual pivoting is a long-standing way to acquire paraphrases. If two English expressions share a foreign translation, they are treated as equivalent. Phrase-based versions of this idea produced PPDB. This paper moves pivoting into neural machine translation. The authors build ParaNet, which uses an attentional encoder-decoder to translate an English sentence into a pivot language and then translates the result back into English. Because each step is conditioned on the whole input, paraphrase choices reflect full-sentence context rather than isolated phrase matches. A single pivot translation can lose meaning, so the model draws on the K-best translations and on several foreign languages at once, merging the decoders' predictions with a weighted form of late averaging. The same system can score a candidate pair of any length, generate paraphrases with beam search, and provide continuous representations for semantic similarity. The authors evaluate it on paraphrase detection/similarity and on generation, using automatic metrics and human judges. In both settings it outperforms phrase-based pivoting. It needs only parallel data and standard NMT, with no phrase table and no syntax.

Key points

  • The baseline paradigm is Bannard and Callison-Burch (2005) pivoting and its syntactic extensions. These led to PPDB, which holds over a billion paraphrase pairs across 23 languages.
  • ParaNet is built from a bidirectional RNN encoder, an attention mechanism, and a conditional RNN decoder. Paraphrasing works as English β†’ pivot (e.g., French) β†’ English.
  • The simplest variant, one-to-one back-translation, reuses existing NMT systems with no extra training. Its weakness is that the output depends only on one pivot sentence, so anything that sentence misses is lost.
  • Multi-pivoting decodes from K-best pivot translations, optionally in multiple languages. It extends the late averaging approach of Firat et al. (2016b) with per-pivot weights.
  • Evaluation covers paraphrase detection/semantic similarity (intrinsic) and generation (extrinsic), with example outputs on the Wikianswers, Leagues, and MTC datasets. The exact scores were in the omitted part of the extracted text, so they are not reproduced here.

Interesting ideas

  • I1. Round-trip translation as a ready-made paraphraser. Any pair of off-the-shelf MT models can produce paraphrases without paraphrase-specific training. This is handy for data augmentation in lower-resource languages such as Ukrainian, where parallel data exists but paraphrase corpora are scarce.
  • I2. Pivoting through many translations instead of one. Combining the K-best pivots, or pivots in different languages, guards against one bad intermediate translation. Each pivot can also contribute different semantic or syntactic aspects of the source, which suits any pipeline sensitive to single-path errors.
  • I3. Weighted late averaging of several decoders. Each pivot is encoded separately, and the next-word distributions are mixed with weights at every decoding step. This simple ensembling trick also works for multi-source translation or for combining inputs of varying reliability.
  • I4. One model for scoring, generation, and embeddings. The same network both generates paraphrases and assigns a probability to a given pair. That probability can serve as a similarity measure for phrases or sentences of any length, avoiding separate tools for each task.
  • I5. Attention as a source of alignments. The authors propose, as future work, mining attention scores to extract PPDB-style paraphrase pairs or to supply features for entailment. This suggests a way to build lexical resources as a by-product of a trained neural model.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation Β· Natural language processing

Topics: Machine translation

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mallinson2017paraphrasing,
  title = {{Paraphrasing Revisited with Neural Machine Translation}},
  author = {Mallinson, Jonathan and Sennrich, Rico and Lapata, Mirella},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  doi = {10.18653/v1/e17-1083},
  url = {https://doi.org/10.18653/v1/e17-1083}
}

References

53 references, 16 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. β†— our card
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Associa-Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875, San Diego, California. tion for Computational Linguistics, pages 597–604, Ann Arbor, Michigan. find our card
  3. Regina Barzilay. 2003. Information Fusion for Multi-Document Summarization: Praphrasing and Generation. Ph.D. thesis, Columbia University. find
  4. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Baltimore, Maryland. find
  5. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. CoRR, abs/1606.04164. β†—
  6. Juri Ganitkevitch and Chris Callison-Burch. 2014. The multilingual paraphrase database. In The 9th edition of the Language Resources and Evaluation Conference, Reykjavik, Iceland. find
  7. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 17–24, New York City, USA. find
  8. Juri Ganitkevitch, Chris Callison-Burch, Courtney Napoles, and Benjamin Van Durme. 2011. Learning sentential paraphrases from bilingual parallel corpora for text-to-text generation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1168–1179, Edinburgh, Scotland, UK. find
  9. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 196– 205, Honolulu, Hawaii. find our card
  10. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. find our card
  11. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1723–1732, Beijing, China. find our card
  12. Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine Learning, 63(1):3–42. β†—
  13. Nadir Durrani, Helmut Schmid, and Alexander Fraser. 2011. A joint sequence translation model with integrated reordering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1045–1054, Portland, Oregon, USA. find
  14. Weiwei Guo and Mona Diab. 2012. A simple unsupervised latent semantics based approach for sentence similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 586–590. find
  15. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. find
  16. Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197. find our card
  17. Shudong Huang, David Graff, George Doddington, Linguistic Data Consortium, et al. 2002. Multiple-translation Chinese corpus. Linguistic Data Consortium, University of Pennsylvania. β†—
  18. Asli Eyecioglu and Bill Keller. 2015. Twitter paraphrase identification with simple overlap features and svms. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 64–69, Denver, Colorado. find
  19. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1–10, Beijing, China. β†— our card
  20. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608–1618, Sofia, Bulgaria. find our card
  21. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015b. Montreal neural machine translation systems for wmt15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisbon, Portugal. find
  22. Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 120–127, Prague, Czech Republic. find
  23. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle, Washington, USA. find
  24. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. β†—
  25. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 102–109, Edmonton, Canada. find our card
  26. Martin Kay. 1997. The proper place of men and machines in language translation. Machine Translation, 12(1-2):3–23. β†—
  27. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54, Sapporo, Japan. β†—
  28. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425–430, Beijing, China. find our card
  29. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. find
  30. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. β†—
  31. Philip Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86, Phuket, Thailand. find
  32. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremb. 2016. Sequence level training with recurrent neural networks. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico. β†— our card
  33. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, Main Volume, pages 605–612, Barcelona, Spain. find our card
  34. Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal, and Yi Liu. 2007. Statistical machine translation for query expansion in answer retrieval. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 464–471, Prague, Czech Republic. find
  35. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question answering. Natural Language Engineering, 7(4):342–360. β†—
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. find
  37. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. β†—
  38. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. find
  39. Nitin Madnani and Bonnie J. Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics, 36(3):341–387. β†— our card
  40. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. find
  41. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701. β†— our card
  42. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. DLS@CU: Sentence similarity from word alignment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 241–246. find
  43. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 30–34, San Diego, California. find
  44. Shiqi Zhao, Haifeng Wang, Ting Liu, and Sheng Li. 2008. Pivot approach for extracting paraphrase patterns from bilingual corpora. In Proceedings of ACL-08: HLT, pages 780–788, Columbus, Ohio. find
  45. Hong Sun and Ming Zhou. 2012. Joint learning of a dual smt system for paraphrase generation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38–42, Jeju Island, Korea. find
  46. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. Curran Associates, Inc. β†—
  47. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 484–491, Rochester, New York. find
  48. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the Association of Computational Linguistics – Volume 3, Issue 1, pages 345–358. β†— our card
  49. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In Proceedings of 4th International Conference on Learning Representations, San Juan, Puerto Rico. β†— our card
  50. Kristian Woodsend and Mirella Lapata. 2014. Text rewriting improves semantic role labeling. Journal of Artificial Intelligence Research, 51:133–164. β†—
  51. Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. Machine Translation, 21(3):165–181. β†—
  52. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics, 2:435–448. β†—
  53. Guido Zarrella, John Henderson, Elizabeth M. Merkhofer, and Laura Strickhart. 2015. Mitre: Seven systems for semantic similarity in tweets. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 12–17, Denver, Colorado. find

↑ back to top

This page in Markdown β†’ https://md.research.elamur.ai/free/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/