Elamur Research
☰
home / papers / 2018 / Adversarial Example Generation with Syntactically Controlled Paraphrase Networks

Adversarial Example Generation with Syntactically Controlled Paraphrase Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1170.pdf

Summary

NLP datasets often lack linguistic variety, so models trained on them can fail when the same meaning appears in a different sentence structure. Earlier data augmentation mostly relied on rule-based word substitutions, and meaning-preserving structural rewrites remained an open problem. The authors introduce syntactically controlled paraphrase networks (SCPNs), the first learned approach to producing a paraphrase that follows a requested constituency structure.

Training pairs come from ParaNMT-50M, built by backtranslating the Czech side of CzEng. The syntactic differences within each pair are labeled automatically by parsing both sentences. The model uses a bidirectional LSTM encoder and an LSTM decoder with attention and copying, and the decoder also attends over an encoding of the target parse. A second network with the same design expands a simple two-level template into a full parse.

Automatic and human evaluations show that outputs nearly always match the target syntax, with paraphrase quality close to uncontrolled backtranslation. The generated examples mislead pretrained sentiment and entailment classifiers. Adding them to training data makes these models more robust without lowering accuracy on the original test sets.

Key points

  • Training data comes from ParaNMT-50M, which has over 50 million paraphrase pairs. They were produced by backtranslating CzEng with a pretrained Czech–English Nematus model, and both sides were parsed with the Stanford shift-reduce parser. Reversed pairs were also used to reduce NMT bias.
  • Syntactic control is specified either as a full linearized constituency parse without tokens or as a parse template. A template is the top two levels of the parse, e.g. S → NP VP.
  • Generation from a template is a two-step pipeline. A separately trained parse generator first predicts a full parse, and SCPN then uses that parse to produce the paraphrase. This design was chosen because conditioning directly on templates produced short, formulaic outputs.
  • Adversarial examples were generated with the twenty most frequent ParaNMT-50M templates and tested on sentiment analysis and textual entailment. Removing the copy mechanism made the model reproduce the input syntax more often, with a 47.7% exact template match.
  • Compared with uncontrolled paraphrasers, SCPN makes mostly syntactic rather than lexical changes. Code, labeled data and pretrained models are available at github.com/miyyer/scpn.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

321 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 295 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1170

@inproceedings{iyyer2018adversarial,
  title = {{Adversarial Example Generation with Syntactically Controlled Paraphrase Networks}},
  author = {Iyyer, Mohit and Wieting, John and Gimpel, Kevin and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1170},
  url = {https://doi.org/10.18653/v1/n18-1170}
}

References

51 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ion Androutsopoulos and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research 38. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  3. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the Association for Computational Linguistics. find our card
  4. Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  5. Ondřej Bojar, Ondřej Dušek, Tom Kocmi, Jindřich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Text, Speech, and Dialogue: 19th International Conference, TSD 2016. find
  6. Igor Bolshakov and Alexander Gelbukh. 2004. Synonymous paraphrasing using WordNet and Internet. Natural Language Processing and Information Systems . find
  7. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of Empirical Methods in Natural Language Processing. find
  8. Michel Carl, Paul Schmidt, and Jörg Schütz. 2005. Reversible template-based shake & bake generation. In MT Summit X. find
  9. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of Empirical Methods in Natural Language Processing. find
  10. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for nlp. arXiv preprint arXiv:1712.06751 . ↗
  11. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable nlp systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems. find
  12. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Characterizing stylistic elements in syntactic structure. In Proceedings of Empirical Methods in Natural Language Processing. find
  13. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633 . ↗ our card
  14. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations. find our card
  15. Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2017. A deep generative framework for paraphrase generation. arXiv preprint arXiv:1709.05074 . ↗
  16. Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2017. Generating sentences by editing prototypes. arXiv preprint arXiv:1709.08878 . ↗ our card
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation . find
  18. Mohit Iyyer, Jordan Boyd-Graber, and Hal Daumé III. 2014. Generating sentences from semantic vector space representations. In NIPS Workshop on Learning Semantics. find
  19. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  20. Stanley Kok and Chris Brockett. 2010. Hitting the right paraphrases in good time. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  21. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural conversation model. In Proceedings of the Association for Computational Linguistics. find our card
  22. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562 . ↗
  23. Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2017. Paraphrase generation with deep reinforcement learning. arXiv preprint arXiv:1711.00279 . ↗ our card
  24. Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2017. Deep text classification can be fooled. arXiv preprint arXiv:1704.08006 . ↗
  25. Nitin Madnani and Bonnie J Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics 36(3). find our card
  26. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the European Chapter of the Association for Computational Linguistics. find
  27. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics System Demonstrations. find our card
  28. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval 2014 . find our card
  29. Siwei Shen, Dragomir R Radev, Agam Patel, and Güneş Erkan. 2006. Adding syntax to dynamic programming for aligning comparable texts for the generation of paraphrases. In Proceedings of International Conference on Computational Linguistics. find
  30. Kathleen R McKeown. 1983. Paraphrasing questions using given and new information. Computational Linguistics 9(1). find
  31. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Proceedings of Advances in Neural Information Processing Systems. find our card
  32. Courtney Napoles, Chris Callison-Burch, and Matt Post. 2016. Sentential paraphrasing as black-box machine translation. In Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. find
  33. David A Smith and Jason Eisner. 2006. Quasi-synchronous grammars: Alignment by soft projection of syntactic dependencies. In Proceedings of the Workshop on Statistical Machine Translation. find our card
  34. Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output. In Proceedings of Empirical Methods in Natural Language Processing. find
  35. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods in Natural Language Processing. find
  36. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  37. Yui Suzuki, Tomoyuki Kajiwara, and Mamoru Komachi. 2017. Building a non-trivial paraphrase corpus using multiple machine translation systems. In Proceedings of ACL Student Research Workshop. find
  38. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing. find
  39. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the Association for Computational Linguistics. find
  40. Aaditya Prakash, Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. In Proceedings of International Conference on Computational Linguistics. find
  41. Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Nyberg. 2017. Steering output style and topic in neural response generation. In Proceedings of Empirical Methods in Natural Language Processing. find
  42. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  43. John Wieting and Kevin Gimpel. 2017. Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. arXiv preprint arXiv:1711.05732 . ↗
  44. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Association for Computational Linguistics. find
  45. John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of Empirical Methods in Natural Language Processing. find
  46. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, et al. 2017. Nematus: a toolkit for neural machine translation. arXiv preprint arXiv:1703.04357 . ↗ our card
  47. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  48. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. In Proceedings of the Association for Computational Linguistics. find
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  50. Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. arXiv preprint arXiv:1502.01710 . ↗
  51. Shiqi Zhao, Xiang Lan, Ting Liu, and Sheng Li. 2009. Application-driven statistical paraphrase generation. In Proceedings of the Association for Computational Linguistics. find