Elamur Research
☰
home / papers / 2016 / Neural Text Generation from Structured Data with Application to the Biography Domain

Neural Text Generation from Structured Data with Application to the Biography Domain

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1603.07771

Summary

The paper applies concept-to-text generation to a much larger and more varied domain than earlier benchmarks such as WEATHERGOV and ROBOCUP. The task is to write the opening sentence of a Wikipedia biography from that person's infobox. The authors built a new corpus of over 700k biographies paired with their infoboxes, with a vocabulary of more than 400k word types. Their model is a feed-forward neural language model conditioned on the table in two ways. With local conditioning, each previously generated word is tagged with the fields where it occurs in the infobox and its position in each field, counted from both the start and the end. With global conditioning, the model also sees the full set of fields and words in the table. Copy actions in the output layer let the model take tokens directly from a field, which handles rare and out-of-vocabulary words. The resulting Table NLM beats a Templated Kneser-Ney baseline by almost 15 BLEU. This shows that table-aware neural language models built on embeddings work on large, open-domain data-to-text tasks where count-based models run into sparsity.

Key points

  • Dataset scale: over 700k Wikipedia biographies with a vocabulary of more than 400k words. That is about ten times more samples than earlier datasets, and their vocabularies were only about 350 words.
  • Gains from each part: global conditioning on fields adds more than 7 BLEU, and adding table words adds another 1.3 BLEU. In total the model beats Template KN by about 15 BLEU, 15 ROUGE and 2.8 NIST.
  • Copy mechanism: the output space is the fixed vocabulary W plus the table tokens Q, with tokens such as name_2. This lets the model produce names that are not in its vocabulary.
  • Decoding: beam search works best with K = 5. At that setting the neural model needs about 200 ms per sentence and is several times faster than the Kneser-Ney baseline.
  • Limitations: the training loss does not penalize wrong facts, such as an incorrect nationality or occupation, more than other errors. BLEU, ROUGE and NIST also do not measure factual accuracy, and the infobox data itself sometimes contradicts the article text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

252 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 237 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.07771 · arXiv 1603.07771

@misc{lebret2016neural,
  title = {{Neural Text Generation from Structured Data with Application to the Biography Domain}},
  author = {Lebret, Remi and Grangier, David and Auli, Michael},
  year = {2016},
  doi = {10.48550/arxiv.1603.07771},
  eprint = {1603.07771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.07771}
}

References

44 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Angeli, P. Liang, and D. Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 502–512. Association for Computational Linguistics. find
  2. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  3. R. Barzilay and M. Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 331–338. find
  4. R. Barzilay and M. Lapata. 2006. Aggregation via set partitioning for natural language generation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 359–366. Association for Computational… find
  5. A. Belz and E. Reiter. 2006. Comparing automatic and human evaluation of nlg systems. In In Proc. EACL06, pages 313–320. find
  6. A. Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering, 14(04):431–455. find
  7. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  8. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In International Conference on Management of Data, pages 1247–1250. ACM. find
  9. R. Dale, S. Geldof, and J.-P. Prost. 2003. Coral: Using natural language generation for navigational assistance. In Proceedings of the 26th Australasian computer science conference-Volume 16, pages 35–44. Australian Computer Society, Inc. find
  10. J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz, and J. Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1370–1380. find
  11. P. A. Duboue and K. R. McKeown. 2002. Content planner construction via evolutionary algorithms and a corpus-based fitness function. In Proceedings of INLG 2002, pages 89–96. find
  12. H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollar, J. Gao, X. He, M. Mitchell, J. C. Platt, L. C. find
  13. Zitnick, and G. Zweig. 2015. From captions to visual concepts and back. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. find
  14. D. Ferrucci. 2012. Introduction to this is watson. IBM Journal of Research and Development, 56(3.4):1–1. find
  15. D. Galanis and I. Androutsopoulos. 2007. Generating multilingual descriptions from linguistically annotated owl ontologies: the naturalowl system. In Proceedings of the Eleventh European Workshop on Natural Language Generation, pages 143–146. Association for Computational Linguistics. find
  16. N. Green. 2006. Generation of biomedical arguments for lay readers. In Proceedings of the Fourth International Natural Language Generation Conference, pages 114– 121. Association for Computational Linguistics. find
  17. B. Gyawali and C. Gardent. 2014. Surface realisation from knowledge-bases. In Proc. of ACL. find
  18. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696, Sofia, Bulgaria, August. find our card
  19. A. Karpathy and L. Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. find our card
  20. J. Kim and R. J. Mooney. 2010. Generative alignment and semantic parsing for learning from ambiguous supervision. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 543–551. Association for Computational Linguistics. find
  21. V. Kindratenko. 2014. Numerical Computations with GPUs. Springer. find
  22. R. Kiros, R. Salakhutdinov, and R. S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539. find our card
  23. I. Konstas and M. Lapata. 2013. A global model for concept-to-text generation. J. Artif. Int. Res., 48(1):305–346, October. find
  24. I. Langkilde and K. Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proc. ACL, pages 704–710. find
  25. R. Lebret and R. Collobert. 2014. Word embeddings through hellinger pca. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–490, Gothenburg, Sweden, April. Association for Computational Linguistics. find
  26. Y. A LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. 2012. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–48. Springer. find
  27. P. Liang, M. I. Jordan, and D. Klein. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 91–99. As… find
  28. W. Lu and H. T. Ng. 2011. A probabilistic forest-to-string model for language generation from typed lambda calculus expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1611–1622. Association for Computational Linguistics. find
  29. M.-T. Luong, I. Sutskever, Q. V Le, O. Vinyals, and W. Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proc. ACL, pages 11–19. find our card
  30. F. Mairesse and M. Walker. 2011. Controlling user perceptions of linguistic style: Trainable generation of personality traits. Comput. Linguist., 37(3):455–488. find
  31. C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  32. H. Mei, M. Bansal, and M. R. Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. In Proceedings of Human Language Technologies: The 2016 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  33. A. Oh and A. Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In ANLP/NAACL Workshop on Conversational Systems, pages 27–32. find
  34. A. Ratnaparkhi. 2002. Trainable approaches to surface natural language generation and their application to conversational dialog systems. Computer Speech & Language, 16(3):435–455. find
  35. E. Reiter, R. Dale, and Z. Feng. 2000. Building natural language generation systems, volume 33. MIT Press. find
  36. E. Reiter, S. Sripada, J. Hunter, J. Yu, and I. Davy. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence, 167(1):137–169. find
  37. L. Shang, Z. Lu, and H. Li. 2015. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364. find our card
  38. Radu Soricut and Daniel Marcu. 2006. Stochastic language generation using widl-expressions and its application in machine translation and summarization. In Proc. ACL, pages 1105–1112. find
  39. R. Turner, S. Sripada, and E. Reiter. 2010. Generating approximate geographic descriptions. In Empirical methods in natural language generation, pages 121– 140. Springer. find
  40. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2015. Show and tell: A neural image caption generator. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. find our card
  41. T. Wen, M. Gasic, N. Mrkšić, P. Su, D. Vandyke, and S. Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, Lisbon, Portugal, September.… find
  42. Y. W. Wong and R. J. Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In HLT-NAACL, pages 172–179. find our card
  43. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of The 32nd International Conference on Machine Learning, volume 37, July. find our card
  44. K. Yao, G. Zweig, and B. Peng. 2015. Attention with intention for a neural network conversation model. arXiv preprint arXiv:1510.08565. — [arXiv:1510.08565](https://arxiv.org/abs/1510.08565) find