Elamur Research
☰
home / papers / 2016 / Discourse Parsing with Attention-based Hierarchical Neural Networks

Discourse Parsing with Attention-based Hierarchical Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1035.pdf

Summary

The paper tackles document-level discourse parsing under Rhetorical Structure Theory (RST), where a text is organized into a tree. The leaves of this tree are Elementary Discourse Units (EDUs), and its inner nodes carry nuclearity and rhetorical relations. Strong earlier systems relied heavily on hand-engineered features, and the only neural approach was a recursive network that struggles with long inputs. The authors build span representations in two levels. A bi-LSTM over words and POS tags encodes each EDU, and a second bi-LSTM over EDUs encodes a whole span. An attention layer then highlights the most informative EDUs. For a pair of adjacent spans, a factorized tensor layer captures multiplicative interactions between their features. Three classifiers then decide whether to merge the spans, which side is the nucleus, and what relation holds between them. A probabilistic CKY-style parser, sped up with two levels of caching, assembles the tree. On RST-DT the system matches state-of-the-art results with minimal manual features and achieves the top span score. This suggests that neural models can compete in a task long dominated by feature engineering.

Key points

  • Evaluated on RST-DT, which contains 385 Wall Street Journal articles split into 347 for training and 38 for testing.
  • The full model scores S 85.8, N 71.1 and R 58.9, the best span result among the compared systems. Human agreement is S 88.7, N 77.7 and R 65.8.
  • Without handcrafted features, the model reaches S 84.2, N 70.4 and R 56.3. The recursive model of Li et al. (2014a) scores S 82.4, N 69.2 and R 56.8 under the same condition, so the new model is better on span and nucleus but slightly lower on relation.
  • To limit parameters and computation, each tensor slice is approximated by the product of two low-rank matrices, following Pei et al. (2014).
  • A qualitative attention example on a Contrast relation shows the model giving the most weight (0.49) to the EDU that carries the main meaning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

50 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1035

@inproceedings{li2016discourse,
  title = {{Discourse Parsing with Attention-based Hierarchical Neural Networks}},
  author = {Li, Qi and Li, Tianshi and Chang, Baobao},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1035},
  url = {https://doi.org/10.18653/v1/d16-1035}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. ↗ our card
  2. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral Presentation. find
  3. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. ↗ our card
  4. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In ACL (1), pages 511–521. find our card
  5. Michael Heilman and Kenji Sagae. 2015. Fast rhetorical structure theory discourse parsing. CoRR, abs/1505.02425. ↗
  6. Karl Moritz Hermann, Tomá s Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. CoRR, abs/1506.03340. ↗ our card
  7. Hugo Hernault, Helmut Prendinger, David A DuVerle, and Mitsuru Ishizuka. 2010. Hilda: a discourse parser using support vector machine classification. Dialogue and Discourse, 1(3):1–33. find our card
  8. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580. ↗
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  10. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In ACL (1), pages 13–24. find our card
  11. Shafiq R. Joty, Giuseppe Carenini, Raymond T. Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In ACL. find
  12. Daniel Jurafsky and James H Martin. 2008. Speech and language processing, chapter 14. In Prentice Hall. find
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  14. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. CoRR, abs/1506.06726. ↗
  15. Jiwei Li, Rumeng Li, and Eduard H Hovy. 2014a. Recursive deep models for discourse parsing. In EMNLP, pages 2061–2069. find
  16. Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014b. Text-level discourse dependency parsing. In ACL (1), pages 25–35. find
  17. Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 147–156. Association for Computational Linguistics. find
  18. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. ↗ our card
  19. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In ACL. find our card
  20. Daniel Marcu. 2000. The theory and practice of discourse parsing and summarization. MIT press. find
  21. Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Max-margin tensor neural network for chinese word segmentation. In ACL (1), pages 293–303. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. Citeseer. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomá s Kociský, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. CoRR, abs/1509.06664. ↗ our card
  24. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer. find
  25. Swapna Somasundaran. 2010. Discourse-level relations for Opinion Analysis. Ph.D. thesis, University of Pittsburgh. find
  26. Ilya Sutskever, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. 2009. Modelling relational data using bayesian clustered tensor factorization. In NIPS. find
  27. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 735–736. ACM. find
  28. Matthew D. Zeiler. 2012. Adadelta: An adaptive learning rate method. CoRR, abs/1212.5701. ↗ our card