Elamur Research
☰
home / papers / 2017 / Challenges in Data-to-Document Generation

Challenges in Data-to-Document Generation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1707.08052

Summary

Neural generators perform well on older data-to-text benchmarks such as WeatherGov, RoboCup and WikiBio. However, those benchmarks pair small record sets with short, simple outputs. The authors argue that a harder test is needed: producing multi-sentence documents from large structured tables. To support this, they release a corpus of NBA game write-ups aligned with box- and line-score tables, drawn from two sources, RotoWire and SBNation. Because BLEU mostly measures fluency, they also introduce an extractive evaluation. An information-extraction model reads the generated text, recovers (entity, value, type) records and compares them with the source table and the reference summary. Using this setup, they benchmark encoder-decoder models with joint or conditional copying, reconstruction losses and a template system. The neural models write fluent, stylistically convincing prose, but they often hallucinate facts, choose content poorly and lack coherent long-range structure. Templates outperform them on several content measures. Copy and reconstruction extensions help but leave a large gap to human writing. The work matters because it exposes weaknesses that surface metrics hide and offers a reusable benchmark for long-form generation.

Key points

  • Dataset scale: RotoWire contains 4.9K examples, about 1.6M tokens and an 11.3K vocabulary. Its summaries average 337.1 words, and each game has 628 records spanning 39 record types. SBNation is larger (10.9K examples, 8.8M tokens, 805.4-word average) but its fan-written text is far noisier. The data is hosted at github.com/harvardnlp/boxscore-data.
  • Extractive metrics: The proposed metrics are relation generation (RG), content selection (CS) and content ordering (CO), with CO measured via DLD. They are computed from records that a relation classifier extracts from the output text.
  • Human evaluation of factual support: Conditional Copy achieved the most box-score-supported facts per sentence (3.05). Contradicted facts were 0.70 for gold texts, compared with 1.16–2.31 for the models. Adding reconstruction plus TVD to Joint Copy significantly reduced contradictions.
  • Human evaluation of ordering: Raters scored the naturalness of fact ordering at 5.19 for gold summaries and 3.90–4.43 for models on a 1–7 scale. These differences were not statistically significant.
  • Typical model errors: Generated texts swapped the teams' win/loss records, produced self-contradictory statements and invented details that the tables do not support.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

328 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 313 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.08052 · arXiv 1707.08052

@misc{wiseman2017challenges,
  title = {{Challenges in Data-to-Document Generation}},
  author = {Wiseman, Sam and Shieber, Stuart M. and Rush, Alexander M.},
  year = {2017},
  doi = {10.48550/arxiv.1707.08052},
  eprint = {1707.08052},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.08052}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 502–512. Association for Computational Linguistics. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 331–338. Association for Computational Linguistics. find
  4. Anja Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering, 14(04):431–455. find
  5. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions, pages 69–72. Association for Computational Linguistics. find
  6. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL, pages 10–21. find our card
  7. Eric Brill and Robert C Moore. 2000. An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 286–293. Association for Computational Linguistics. find
  8. David L Chen and Raymond J Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In Proceedings of the 25th international conference on Machine learning, pages 128–135. ACM. find
  9. Andrew Chisholm, Will Radford, and Ben Hachey. 2017. Learning to generate one-sentence biographies from wikidata. CoRR, abs/1702.06235. find
  10. KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. find our card
  11. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  12. Pablo A Duboue and Kathleen R McKeown. 2003. Statistical acquisition of content selection rules for natural language generation. In EMNLP, pages 121– 128. Association for Computational Linguistics. find
  13. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  14. Eli Goldberg, Norbert Driedger, and Richard I Kittredge. 1994. Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45– 53. find
  15. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. find
  16. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In ACL. find our card
  17. Çaglar Gülçehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In ACL. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9:1735–1780. find
  19. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In ACL Volume 1: Long Papers. find
  20. Dan Jurafsky and James H Martin. 2014. Speech and language processing, volume 3. Pearson London. find
  21. Anjuli Kannan and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training. find
  22. Joohyun Kim and Raymond J Mooney. 2010. Generative alignment and semantic parsing for learning from ambiguous supervision. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 543–551. Association for Computational Linguistics. find
  23. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746– 1751. find our card
  24. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. CoRR, abs/1701.02810. find our card
  25. Ioannis Konstas and Mirella Lapata. 2013. A global model for concept-to-text generation. J. Artif. Intell. Res.(JAIR), 48:305–346. find
  26. Karen Kukich. 1983. Design of a knowledge-based report generator. In ACL, pages 145–150. find
  27. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In EMNLP, pages 1203–1213. find our card
  28. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. CoRR, abs/1701.06547. find
  29. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In ACL, pages 91–99. Association for Computational Linguistics. find
  30. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP, pages 2122–2132. find
  31. Wei Lu and Hwee Tou Ng. 2011. A probabilistic forest-to-string model for language generation from typed lambda calculus expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1611–1622. Association for Computational Linguistics. find
  32. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 1412– 1421. find
  33. Kathleen McKeown. 1992. Text generation - using discourse strategies and focus constraints to generate natural language text. Studies in natural language processing. Cambridge University Press. find
  34. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. In NAACL HLT, pages 720–730. find
  35. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843. find our card
  36. T. Mikolov, M. Karafit, L. Burget, J. Cernock, and S. Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH. find
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  38. Ehud Reiter. 2017. You need to understand your corpora! the weathergov example. https://ehudr eiter.com/2017/05/09/weathergov/. find
  39. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering, 3(1):57–87. find
  40. Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu, and Ian Davy. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence, 167(1-2):137–169. find
  41. Jacques Robin. 1994. Revision-based generation of Natural Language Summaries providing historical Background. Ph.D. thesis, Citeseer. find
  42. Cı́cero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In ACL, pages 626– 634. find
  43. Radu Soricut and Daniel Marcu. 2006. Stochastic language generation using widl-expressions and its application in machine translation and summarization. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational L… find
  44. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML), pages 1017–1024. find
  45. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 3104–3112. find
  46. Kumiko Tanaka-Ishii, Kôiti Hasida, and Itsuki Noda. 1998. Reactive content selection in the generation of real-time soccer commentary. In COLING-ACL, pages 1282–1288. find
  47. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI, pages 3097–3103. find
  48. Michael White, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface realization with ccg. In Proceedings of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation (UCNLG+ MT), pages 267– 276. find our card
  49. Yuk Wah Wong and Raymond J Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In HLT-NAACL, pages 172–179. find our card
  50. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  51. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2016. Reference-aware language models. CoRR, abs/1611.01628. find
  52. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In COLING, pages 2335–2344. find
  53. Zhu Zhang. 2004. Weakly-supervised relation classification for information extraction. In Proceedings of the thirteenth ACM international conference on Information and knowledge management, pages 581– 588. ACM. find
  54. GuoDong Zhou, JunHui Li, LongHua Qian, and Qiaoming Zhu. 2008. Semi-supervised learning for relation extraction. In Third International Joint Conference on Natural Language Processing, page 32. find