Elamur Research⌕
☰
home / papers / 2014 / Chinese Poetry Generation with Recurrent Neural Networks

Chinese Poetry Generation with Recurrent Neural Networks

Summary

The paper addresses the automatic composition of classical Chinese quatrains. These are four-line poems with five or seven characters per line, and they must obey rhyme and Ping/Ze tonal patterns while still reading as coherent, evocative text. Earlier systems relied on templates, genetic search, SMT-style line-to-line translation or summarization, and they usually conditioned only on the preceding line or on n-grams. Zhang and Lapata propose RNNPG, a neural generator that handles content selection and surface realization together.

User keywords are expanded through the ShiXueHanYing phrase taxonomy, and the first line is assembled from those phrases under tonal constraints. Each subsequent line is produced character by character, conditioned on the whole poem so far. Three components do this work: a convolutional sentence model encodes earlier lines, a recurrent context model summarizes them, and a recurrent generation model predicts the next character.

On automatic metrics (perplexity, BLEU) and in a human study with Chinese poetry experts, RNNPG beats competitive SMT-based, summarization-based and random baselines. This was an early demonstration that neural networks can directly generate constrained creative text.

Key points

  • Architecture. The model has three parts:
    • a convolutional sentence model (CSM) turns each previous line into a vector;
    • a recurrent context model (RCM) builds position-specific context vectors from all prior lines;
    • a recurrent generation model (RGM) outputs the next-character distribution.
  • First-line generation. Keywords are restricted to the ShiXueHanYing taxonomy, which has 1,016 manually built phrase clusters. Candidate first lines are ranked by a character-based RNN language model interpolated with a Kneser-Ney trigram and decoded with a stack decoder.
  • Automatic evaluation. The BLEU-2 comparison is against a re-implementation of He et al.'s (2012) SMT generator. That SMT baseline was trained on 284,899 poems, whereas Yan et al. used 61,960. RNNPG also incorporates three of the SMT features.
  • Human evaluation.
    • Design: 30 experts rated poems on fluency, coherence, meaningfulness and poeticness, and also gave an overall ranking.
    • Against baselines: for 5-char quatrains, RNNPG was significantly better than Random, SUM (Yan et al., 2013) and SMT.
    • Against human poems: RNNPG's overall rank was not significantly different from human poems, although human poems won on every other dimension.
  • Difficulty by form. 7-char quatrains proved harder, and the gap between human and machine poems was wider for them.

Interesting ideas

  • I1. Encode all earlier text, not just the last line. Summarizing every previously generated line into a context vector, instead of conditioning on the immediately preceding one, gives a simple recipe for long-range coherence. It could carry over to multi-sentence generation such as summaries or dialogue.
  • I2. Separate hard constraints from soft fluency. The system enumerates only candidates that satisfy the tonal pattern, then lets a language model rank them. This pattern suits any generation task with strict formal rules, such as meter, morphology agreement or templates.
  • I3. Use a curated resource as a strong random baseline. The Random system picks phrases from human-edited clusters tied to the keywords. A baseline like this checks whether a model adds value beyond its lexical resource.
  • I4. Rank-based human scoring. Converting each system's position in a ranked list into a normalized pairwise-win score complements per-dimension ratings. It helps when absolute scales are noisy across annotators.
  • I5. Treat BLEU with caution for creative output. The authors note that n-gram overlap can be overly conservative for creative tasks. They pair it with perplexity and expert judgments, which is a sensible evaluation mix for any open-ended generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1074

@inproceedings{zhang2014chinese,
  title = {{Chinese Poetry Generation with Recurrent Neural Networks}},
  author = {Zhang, Xingxing and Lapata, Mirella},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1074},
  url = {https://doi.org/10.3115/v1/d14-1074}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manex Agirrezabal, Bertol Arrieta, Aitzol Astigarraga, and Mans Hulden. 2013. POS-Tag Based Poetry Generation with WordNet. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 162–166, Sofia, Bulgaria. find
  2. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint Language and Translation Modeling with Recurrent Neural Networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1044– 1054, Seattle, Washington, USA. find
  3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the 7th Workshop on Statistical Machine Translation, pages 10– 51, Montréal, Canada. find our card
  4. Simon Colton, Jacob Goodwin, and Tony Veale. 2012. Full-FACE Poetry Generation. In Proceedings of the International Conference on Computational Creativity, pages 95–102, Dublin, Ireland. find
  5. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
  6. Erica Greene, Tugba Bodrumlu, and Kevin Knight. 2010. Automatic Analysis of Rhythmic Poetry with Applications to Generation and Translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 524– 533, Cambridge, MA. find
  7. Jing He, Ming Zhou, and Long Jiang. 2012. Generating Chinese Classical Poems with Statistical Machine Translation Models. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, pages 1650–1656, Toronto, Canada. find
  8. Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, United Kingdom, July. find our card
  9. Long Jiang and Ming Zhou. 2008. Generating Chinese Couplets using a Statistical MT Approach. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 377–384, Manchester, UK, August. find
  10. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle, Washington. find our card
  11. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical Phrase-based Translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54, Edmonton, Canada. find
  12. Ruli Manurung, Graeme Ritchie, and Henry Thompson. 2012. Using Genetic Algorithms to Create Meaningful Poetic Text. Journal of Experimental Theoretical Artificial Intelligence, 24(1):43–64. find
  13. Ruli Manurung. 2003. An Evolutionary Algorithm Approach to Poetry Generation. Ph.D. thesis, University of Edinburgh. find
  14. Tomas Mikolov and Geoffrey Zweig. 2012. Context Dependent Recurrent Neural Network Language Model. In Proceedings of 2012 IEEE Workshop on Spoken Language Technology, pages 234– 239, Miami, Florida. find
  15. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent Neural Network based Language Model. In Proceedings of INTERSPEECH, pages 1045–1048, Makuhari, Japan. find
  16. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget, and Jan Cernocky. 2011a. Strategies for Training Large Scale Neural Network Language Models. In Proceedings of ASRU 2011, pages 196– 201, Hilton Waikoloa Village, Big Island, Hawaii, US. find
  17. Tomas Mikolov, Stefan Kombrink, Lukas Burget, JH Cernocky, and Sanjeev Khudanpur. 2011b. Extensions of Recurrent Neural Network Language Model. In Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5528–5531, Prague, Czech Republic. find
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119, Lake Tahoe, Nevada, United States. find
  19. Jeff Mitchell and Mirella Lapata. 2010. Composition in Distributional Models of Semantics. Cognitive Science, 34(8):1388–1439. find
  20. Yael Netzer, David Gabay, Yoav Goldberg, and Michael Elhadad. 2009. Gaiku: Generating Haiku with Word Associations Norms. In Proceedings of the Workshop on Computational Approaches to Linguistic Creativity, pages 32–39, Boulder, Colorado. find
  21. Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 160–167, Sapporo, Japan. find
  22. Hugo Gonçalo Oliveira. 2012. PoeTryMe: a Versatile Platform for Poetry Generation. Computational Creativity, Concept Invention, and General Intelligence, 1:21. find
  23. David Rumelhart, Geoffrey Hinton, and Ronald Williams. 1988. Learning Representations by Backpropagating Errors. MIT Press, Cambridge, MA, USA. find
  24. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211,… find our card
  25. Naoko Tosa, Hideto Obara, and Michihiko Minoh. 2008. Hitch Haiku: An Interactive Supporting System for Composing Haiku Poem How I Learned to Love the Bomb: Defcon and the Ethics of Computer Games. In Proceedings of the 7th International Conference on Entertainment Computing, pages 209–216, Pittsburg… find
  26. Li Wang. 2002. A Summary of Rhyming Constraints of Chinese Poems (Shi Ci Ge Lv Gai Yao). find
  27. Xiaofeng Wu, Naoko Tosa, and Ryohei Nakatsu. 2009. New Hitch Haiku: An Interactive Renku Poem Composition Supporting Tool Applied for Sightseeing Navigation System. In Proceedings of the 8th International Conference on Entertainment Computing, pages 191–196, Paris, France. find
  28. Rui Yan, Han Jiang, Mirella Lapata, Shou-De Lin, Xueqiang Lv, and Xiaoming Li. 2013. I, Poet: Automatic Chinese Poetry Composition Through a Generative Summarization Framework Under Constrained Optimization. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence, pages… find
  29. Cheng-Le Zhou, Wei You, and Xiaojun Ding. 2010. Genetic Algorithm and its Implementation of Automatic Generation of Chinese SongCi. Journal of Software, pages 427–437. find
  30. Geoffrey Zweig and Konstantin Makarychev. 2013. Speed Regularization and Optimality in Word Classing. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 8237–8241, Florence, Italy. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2014-chinese-poetry-generation-recurrent-neural/