Elamur Research
☰
home / papers / 2019 / Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation

Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1236.pdf

Summary

End-to-end neural data-to-text systems produce fluent text, but they often drop, repeat or invent facts, and they offer no way to steer the structure of the output. The authors split generation into two stages and test the idea on the WebNLG benchmark, where inputs are sets of DBPedia RDF triplets. First, a symbolic planner builds a text plan: a sequence of ordered trees. The plan fixes how facts are grouped into sentences, the order of sentences and facts, and the direction in which each relation is verbalized. By construction, every input triplet appears in the plan exactly once. The plan is then linearized and given to an NMT-based realizer that only has to choose the wording. To train the realizer, the authors automatically pair reference texts with plans that match them. The same annotated data drives a scoring method that selects good plans for new inputs. The resulting system matches or exceeds a strong neural baseline on BLEU and human fluency ratings, and it is clearly more faithful to the input. It can also produce diverse, controllable outputs. This shows that the classic split between planning and realization still pays off in neural NLG.

Key points

  • Plan representation: each sentence plan is a labeled ordered tree whose edges carry an RDF relation and a direction (→/←). Facts sharing an entity are linked as chains or siblings. For small WebNLG inputs, all candidate plans can be enumerated exhaustively by grouping the facts and running DFS traversals.
  • Human fluency evaluation:
    • The plan-based system (BestPlan) performed on par with the end-to-end StrongNeural baseline.
    • It beat the earlier human-evaluation leader, UPF-FORGe.
    • It was rated below the human References, partly because it does no referring expression generation.
  • Realizer consistency with plans (Table 4):
    • For the best plan, all requested entities were realized in 98.9% of sentence plans on the seen test set and 66.7% on the unseen set (93.6% / 41.3% for the top 10% of plans).
    • Whenever all entities were realized, their order followed the plan in essentially 100% of cases.
  • Diversity: realizing random plans from the top 10% (RandomPlan-1/2/3) produced quite different texts. Automatic scores dropped only modestly and stayed stable across random seeds.
  • Released resources: the code and the WebNLG corpus extended with matching plans are available at github.com/AmitMY/chimera.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

114 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1236

@inproceedings{moryossef2019step,
  title = {{Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation}},
  author = {Moryossef, Amit and Goldberg, Yoav and Dagan, Ido},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1236},
  url = {https://doi.org/10.18653/v1/n19-1236}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  3. Regina Barzilay and Mirella Lapata. 2006. Aggregation via set partitioning for natural language generation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 359–366. Association for Comp… find
  4. Steven Bird and Edward Loper. 2004. Nltk: the natural language toolkit. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 31. Association for Computational Linguistics. find
  5. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  6. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. arXiv preprint arXiv:1703.00955. find our card
  7. Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 329–339. find
  8. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. find our card
  9. Ioannis Konstas and Mirella Lapata. 2012. Unsupervised concept-to-text generation with hypergraphs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 752–761. Association for Computational Linguist… find
  10. Emilie Colin, Claire Gardent, Yassine Mrabet, Shashi Narayan, and Laura Perez-Beltrachini. 2016. The webnlg challenge: Generating text from dbpedia data. In Proceedings of the 9th International Natural Language Generation conference, pages 163– 167. find
  11. Ioannis Konstas and Mirella Lapata. 2013. Inducing document plans for concept-to-text generation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1503–1514. find
  12. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. arXiv preprint arXiv:1810.01170. find
  13. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 545– 552. Association for Computational Linguistics. find
  14. Thiago Castro Ferreira, Diego Moussallem, Ákos Kádár, Sander Wubben, and Emiel Krahmer. 2018. Neuralreg: An end-to-end approach to referring expression generation. arXiv preprint arXiv:1805.08093. find
  15. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. find
  16. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633. find our card
  17. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133. find our card
  18. Albert Gatt and Emiel Krahmer. 2017. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. CoRR, abs/1703.09902. — [arXiv:1703.09902](https://arxiv.org/abs/1703.09902) find
  19. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. arXiv preprint arXiv:1603.08148. find
  20. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437. find our card
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. find
  22. Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2015. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. arXiv preprint arXiv:1509.00838. find
  23. Simon Mille, Roberto Carlini, Alicia Burga, and Leo Wanner. 2017. Forge at semeval-2017 task 9: Deep sentence generation based on a sequence of graph transducers. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 920–923. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  25. Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. 2017. Adversarially regularized autoencoders for generating discrete structures. CoRR, abs/1706.04223. — [arXiv:1706.04223](https://arxiv.org/abs/1706.04223) find
  26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  27. Ratish Puduppully, Li Dong, and Mirella Lapata. 2018. Data-to-text generation with content selection and planning. arXiv preprint arXiv:1809.00582. — [arXiv:1809.00582](https://arxiv.org/abs/1809.00582) find
  28. Yevgeniy Puzikov and Iryna Gurevych. 2018. E2e nlg challenge: Neural models vs. templates. In Proceedings of the 11th International Conference on Natural Language Generation, pages 463–471. find
  29. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press. find
  30. Robert E Schapire. 1999. A brief introduction to boosting. In Ijcai, volume 99, pages 1401–1406. find
  31. Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799. — [arXiv:1706.09799](https://arxiv.org/abs/1706.09799) find
  32. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems, pages 6830–6841. find our card
  33. Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd annual meeting on association for computational linguistics, page 79. Association for Computational Linguistics. find
  34. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575. find our card
  35. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745. find
  36. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. arXiv preprint arXiv:1707.08052. find our card
  37. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2018. Learning neural templates for text generation. arXiv preprint arXiv:1808.10122. find