Elamur Research
☰
home / papers / 2017 / Generating Sentences by Editing Prototypes

Generating Sentences by Editing Prototypes

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1709.08878

Summary

Standard neural language models build each sentence from nothing, usually word by word. They often drift toward bland, generic output, and simple tricks to add variety tend to damage grammaticality. Guu, Hashimoto, Oren and Liang propose a different generative story inspired by how people draft and revise. The model first picks a real sentence from the training corpus as a prototype. It then samples a latent edit vector, and an attention-based neural editor rewrites the prototype according to that vector. Training maximizes a lower bound on likelihood with two approximations. Only lexically close prototypes are considered, found efficiently with locality sensitive hashing. An inverse editor acts as a variational encoder, giving an ELBO in VAE style. On the Yelp corpus and the One Billion Word Benchmark, the approach lowers perplexity, and human judges rate its outputs higher. The learned edit vectors also beat a sentence VAE on similarity, controlled generation and analogy tasks. The work shows that retrieving and editing corpus examples is a strong inductive bias for unconditional text generation.

Key points

  • Compared with from-scratch neural LMs, perplexity is 13 points lower on Yelp, and the One Billion Word Benchmark shows a 7-point reduction.
  • On Yelp, 70% of test sentences lie within word-token Jaccard distance 0.5 of some training sentence, although exact repeats are almost absent.
  • Prototype neighborhoods are restricted to Jaccard distance below 0.5 and precomputed with LSH plus minhashing.
  • Human review of 250 neighbor pairs from Yelp found:
    • 35.2% were full paraphrases.
    • 84% were at least roughly equivalent.
    • Only 7.2% involved negation or a topic shift.
  • Architecture:
    • The editor uses a 3-layer bidirectional LSTM encoder with GloVe inputs and a 3-layer attentive LSTM decoder, with the edit vector fed in at every decoding step.
    • Light interpolation with a plain NLM covers test sentences that have no close prototype.
    • Code and data are released on CodaLab.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 105 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.08878 · arXiv 1709.08878

@misc{guu2017generating,
  title = {{Generating Sentences by Editing Prototypes}},
  author = {Guu, Kelvin and Hashimoto, Tatsunori B. and Oren, Yonatan and Liang, Percy},
  year = {2017},
  doi = {10.48550/arxiv.1709.08878},
  eprint = {1709.08878},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.08878}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic textual similarity. In International Conference on Computational Linguistics (COLING), pages 81–91. find our card
  2. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). find our card
  3. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(0):1137–1155. find
  4. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. 2016. Generating sentences from a continuous space. In Computational Natural Language Learning (CoNLL), pages 10–21. find our card
  5. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  6. C. Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908. — [arXiv:1606.05908](https://arxiv.org/abs/1606.05908) find
  7. J. Gu, Y. Wang, K. Cho, and V. O. Li. 2017. Search engine guided non-parametric neural machine translation. arXiv preprint arXiv:1705.07267. — [arXiv:1705.07267](https://arxiv.org/abs/1705.07267) find
  8. U. Hahn and I. Mani. 2000. The challenges of automatic summarization. Computer, 33. find
  9. T. B. Hashimoto, D. Alvarez-Melis, and T. S. Jaakkola. 2016. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics (TACL), 4:273–286. find
  10. J. R. Hayes and L. S. Flower. 1986. Writing research and the writer. American psychologist, 41(10):1106–1113. find
  11. H. He, A. G. II, J. Boyd-Graber, and H. D. III. 2015. Syntax-based rewriting for simultaneous machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 55–64. find
  12. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Association for Computational Linguistics (ACL), pages 690–696. find our card
  13. M. Hodosh, P. Young, and J. Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research (JAIR), 47:853–899. find
  14. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37:183–233. find
  15. N. Kalchbrenner and P. Blunsom. 2013. Recurrent continuous translation models. In Empirical Methods in Natural Language Processing (EMNLP), pages 1700– 1709. find
  16. D. P. Kingma and M. Welling. 2014. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114. find our card
  17. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. 2013. Generalizing image captions for image-text parallel corpus. In Association for Computational Linguistics (ACL), pages 790–796. find
  18. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110–119. find our card
  19. R. Mason and E. Charniak. 2014. Domain-specific image captioning. In Computational Natural Language Learning (CoNLL), pages 2–10. find
  20. T. Mikolov, K. Chen, G. Corrado, and Jeffrey. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  21. T. Mikolov, W. Yih, and G. Zweig. 2013b. Linguistic regularities in continuous space word representations. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), volume 13, pages 746–751. find
  22. E. Parzen. 1962. On estimation of a probability density function and mode. Annals of Mathematical Statistics, 33:1065–1076. find
  23. J. Pennington, R. Socher, and C. D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  24. A. Ritter, C. Cherry, and W. B. Dolan. 2011. Data-driven response generation in social media. In Empirical Methods in Natural Language Processing (EMNLP), pages 583–593. find
  25. L. Shao, S. Gouws, D. Britz, A. Goldie, B. Strope, and R. Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Empirical Methods in Natural Language Processing (EMNLP), pages 2210–2219. find
  26. Y. Song, R. Yan, X. Li, D. Zhao, and M. Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149. — [arXiv:1610.07149](https://arxiv.org/abs/1610.07149) find
  27. I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 3104–3112. find
  28. J. B. Tenenbaum, V. D. Silva, and J. C. Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. Science, pages 2319–2323. find
  29. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  30. A. T. Wood. 1994. Simulation of the von mises fisher distribution. Communications in statistics-simulation and computation, pages 157–164. find
  31. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  32. Yelp. 2017. Yelp Dataset Challenge, Round 8. https: //www.yelp.com/dataset\_challenge. find