Elamur Research
☰
home / papers / 2020 / AMR-To-Text Generation with Graph Transformer

AMR-To-Text Generation with Graph Transformer

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.2.pdf

Summary

The paper addresses AMR-to-text generation: producing an English sentence from an Abstract Meaning Representation graph. This is hard because AMR leaves out function words, tense, number and definiteness. Earlier graph-to-sequence encoders (graph state LSTM, GGNN, GCN) had not clearly beaten linearized sequence-to-sequence baselines. The authors adapt the Transformer into a fully attention-based Graph Transformer. In each encoder layer, a node attends separately to its incoming and to its outgoing neighbours, and a pairwise interaction function models the labelled relation between two concepts. A fusion layer then merges both directions, and stacking layers spreads information beyond immediate neighbours. The decoder follows the standard Transformer and adds a copy mechanism for rare tokens such as names and numbers. On the LDC2015E86 and LDC2017T10 benchmarks the model set new state-of-the-art BLEU scores. It was strongest when training data was scarce and when graphs contained many reentrancies. This suggests that pure attention over graph structure is an effective encoder for semantic graphs.

Key points

  • Without external data, Graph Transformer beats the previous best neural model, g-GCNSEQ (Damonte and Cohen, 2019), by 1.5 BLEU on LDC2015E86 and 4.8 BLEU on LDC2017T10.
  • Extra training data from the Gigaword corpus, automatically annotated by a pretrained AMR parser, raises the score to 36.4 BLEU on LDC2015E86, the best reported result on that dataset.
  • On graphs with more than 3 reentrancies, the model outperforms an S2S baseline by 2.9 BLEU, while GraphLSTM performs roughly the same as S2S.
  • The architecture has three main parts:
    • direction-specific graph attentions, plus a fusion layer and residual connections;
    • final node vectors formed by concatenating the representations of the two directions;
    • a Transformer decoder with a copy mechanism.
  • The code is released at github.com/sodawater/GraphTransformer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

40 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 33 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00297

@article{wang2020amr,
  title = {{AMR-To-Text Generation with Graph Transformer}},
  author = {Wang, Tianming and Wan, Xiaojun and Jin, Hanqi},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00297},
  url = {https://doi.org/10.1162/tacl_a_00297}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Normunds Gruzitis, Didzis Gosko, and Guntis Barzdins. 2017. Rigotrio at SemEval-2017 task 9: combining machine learning and grammar engineering for amr parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics. find our card
  2. Peter Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchezgonzalez, Vinicius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, and others. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint: 1806.01261. ↗ our card
  3. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. International Conference on Learning Representations. find our card
  4. Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 273–283. find
  5. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393. ↗ our card
  6. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. 2014. Spectral networks and locally connected networks on graphs. International Conference on Learning Representations. find
  7. Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare R. Voss, Jiawei Han, and Avirup Sil. 2016. Liberal event extraction and event schema induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 258–268. find
  8. Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for amr-to-text generation. arXiv preprint arXiv:1903.11410v1. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Bevan Jones, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. 2012. Semantics-based machine translation with hyperedge replacement grammars. Proceedings of COLING 2012, pages 1359–1376. find our card
  11. David K. Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems, pages 2224–2232. find
  12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations. find
  13. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations. find our card
  14. Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 731–739. find our card
  15. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural amr: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 146–157. find our card
  16. Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729–734. IEEE. find
  17. Gerasimos Lampouras and Andreas Vlachos. 2017. Sheffield at SemEval-2017 task 9: Transition-based language generation from AMR. In Proceedings of the 11th International neural machine translation using AMR. Transactions of the Association for Computational Linguistics, 7:19–31. find
  18. Workshop on Semantic Evaluation (SemEval-2017), pages 586–591. find
  19. Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493. ↗
  20. Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. AMR-to-text generation with synchronous node replacement grammar. Meeting of the Association for Computational Linguistics, 2:7–13. find our card
  21. Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman M. Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. North American Chapter of the Association for Computational Linguistics, pages 1077–1086. find
  22. Linfeng Song, Yue Zhang, Xiaochang Peng, Zhiguo Wang, and Daniel Gildea. 2016. Amr-to-text generation as a traveling salesman problem. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2084–2089. find
  23. Arindam Mitra and Chitta Baral. 2016. Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. In Thirtieth AAAI Conference on Artificial Intelligence. find
  24. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1616–1626. find our card
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  26. Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata. 2016. Neural headline generation on abstract meaning representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1054–1059. find
  27. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  28. Maja Popović. 2017. CHRF++: Words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. find our card
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  30. Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016. Generating English from abstract meaning representations. In Proceedings of the 9th International Natural Language Generation Conference, pages 21–25. find our card
  31. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. International Conference on Learning Representations. find
  32. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80. find
  33. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542. find our card
  34. Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic find