Elamur Research
☰
home / papers / 2019 / Text Generation from Knowledge Graphs with Graph Transformers

Text Generation from Knowledge Graphs with Graph Transformers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1238.pdf

Summary

The paper asks whether automatically extracted knowledge, rather than costly hand-built tables or document plans, can guide the generation of coherent multi-sentence text. The authors run the SciIE information extraction system over scientific abstracts. Each abstract becomes a knowledge graph: entities are nodes, coreferent mentions are merged into one node, and extracted relations become labeled edges. The resulting graph–text pairs form a new resource, the AGENDA dataset. Their model, GraphWriter, is an encoder-decoder. It encodes the title with a bidirectional RNN. It encodes the knowledge graph with a new Graph Transformer, which adapts the Transformer to graph inputs by building on Graph Attention Networks. Before encoding, each graph is converted into an unlabeled, connected form that still keeps the relation information. During decoding, the model attends over both the graph and the title, and it can either produce a vocabulary word or copy an entity from the graph. In automatic and human evaluations, GraphWriter produced more informative and better-structured abstracts than competing encoder-decoder baselines, including GAT, EntityWriter and Rewriter. This shows that noisy IE output, when encoded as a graph, is useful input for long-form generation.

Key points

  • AGENDA dataset:
    • Roughly 40k paper titles paired with their abstracts, taken from the Semantic Scholar Corpus and covering 12 leading AI conferences.
    • Split into 38,720 training, 1,000 validation and 1,000 test instances.
    • Abstracts average about 141 tokens; graphs average about 12.4 vertices and 4.4 edges.
  • Graph content: SciIE (Luan et al., 2018) supplies the entity types Task, Method, Metric, Material and Other Scientific Term. It also supplies coreference and relations such as Compare, Used-for, Feature-of, Hyponym-of, Evaluate-for and Conjunction.
  • Encoder comparison: GraphWriter's transformer-style graph encoder works better than a plain Graph Attention Network (GAT) on this task, and its outputs repeat less.
  • Knowledge without a graph: Even entities inferred from the title alone help. Infer-EntityWriter scores 3.60 BLEU and 12.2 METEOR, compared with 1.05 BLEU and 8.38 METEOR for the title-only Rewriter.
  • Remaining errors:
    • About 40% of graph entities never appear in the generated text.
    • About 18% of generated sentences repeat earlier content and have to be pruned in post-processing.
    • Data and code are released at github.com/rikdz/GraphWriter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

121 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1238

@inproceedings{koncelkedziorski2019text,
  title = {{Text Generation from Knowledge Graphs with Graph Transformers}},
  author = {Koncel-Kedziorski, Rik and Bekal, Dhanush and Luan, Yi and Lapata, Mirella and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1238},
  url = {https://doi.org/10.18653/v1/n19-1238}
}

References

35 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. Construction of the Literature Graph in Semantic Scholar. In NAACL. find
  2. Regina Barzilay and Mirella Lapata. 2005. Collective Content Selection for Concept-to-Text Generation. In EMNLP, pages 331–338. Association for Computational Linguistics. find
  3. Daniel Edward Robert Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-Sequence Learning using Gated Graph Neural Networks. In ACL. find
  4. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS. find our card
  5. Michael J. Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Workshop on Statistical Machine Translation. find our card
  6. Alex Graves. 2012. Sequence Transduction with Recurrent Neural Networks. arXiv preprint arXiv:1211.3711. ↗ our card
  7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput., 9(8):1735– 1780. find
  9. Joohyun Kim and Raymond J Mooney. 2010. Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 543–551. find
  10. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR. find our card
  11. Svetlana Kiritchenko and Saif Mohammad. 2016. Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling. In NAACL-HLT. find
  12. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke S. Zettlemoyer. 2017. Neural AMR: Sequence-to-Sequence Models for Parsing and Generation. In ACL. find our card
  13. Ioannis Konstas and Mirella Lapata. 2013. Inducing Document Plans for Concept-to-Text Generation. In EMNLP, pages 1503–1514. find
  14. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural Text Generation from Structured Data with Application to the Biography Domain. In EMNLP. find our card
  15. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A Graph-to-Sequence Model for AMR-to-Text Generation. In ACL. find our card
  16. Percy Liang, Michael I. Jordan, and Dan Klein. 2009. Learning Semantic Correspondences with Less Supervision. In ACL/AFNLP, pages 91–99. find
  17. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res., 15(1):1929–1958. find
  18. Jordan J Louviere, Terry N Flynn, and Anthony Alfred John Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  19. Jordan J Louviere and George G Woodworth. 1991. Best-Worst Scaling: A Model for the Largest Difference Judgments. University of Alberta: Working Paper. find
  20. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In NeurIPS, pages 3104–3112. find
  21. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling Coverage for Neural Machine Translation. arXiv preprint arXiv:1601.04811. ↗ our card
  22. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In EMNLP, pages 3219–3232. find our card
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In NeurIPS, pages 5998–6008. find
  24. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In EMNLP. find
  25. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR. find
  26. Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep Graph Convolutional Encoders for Structured Data to Text Generation. INLG. find
  27. Qingyun Wang, Zhihao Zhou, Lifu Huang, Spencer Whitehead, Boliang Zhang, Heng Ji, and Kevin Knight. 2018. Paper Abstract Writing through Editing Mechanism. In Proceedings of NAACL-HLT. find
  28. Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. In NAACL-HLT, pages 720–730. find
  29. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in Data-to-document Generation. In EMNLP. find our card
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In ACL. find our card
  31. Lutz Prechelt. 1998. Early Stopping - but when? In Neural Networks: Tricks of the Trade, pages 55–69, London, UK, UK. Springer-Verlag. find
  32. Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-Text Generation with Content Selection and Planning. In AAAI. find
  33. Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural networks : the official journal of the International Neural Network Society, 12 1:145–151. find
  34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Accessed at https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/ language-unsupervised/language_ understanding_paper.pdf. ↗
  35. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the Point: Summarization with Pointer-Generator Networks. arXiv preprint arXiv:1704.04368. ↗