Elamur Research⌕
☰
home / papers / 2017 / A Systematic Study of Neural Discourse Models for Implicit Discourse Relation

A Systematic Study of Neural Discourse Models for Implicit Discourse Relation

Summary

Identifying the sense of an implicit discourse relation, where no connective signals the link between two text spans, is the hardest part of discourse parsing. Earlier neural approaches were difficult to compare because they used different label sets and evaluation setups. The authors run a controlled comparison of three neural designs that range from no structure to full syntactic structure:

  • a bag-of-words feedforward network that pools word vectors with max, sum or mean pooling;
  • a sequential LSTM;
  • a tree LSTM built over binarized constituency parses.

They test these on the Penn Discourse TreeBank (PDTB), the Chinese Discourse Treebank (CDTB) and the CoNLL 2015–2016 Shared Task datasets. Unexpectedly, the tuned feedforward model usually matches or beats the LSTM variants and also stands up well against competitive convolutional and recurrent systems. It needs no hand-crafted surface features. The authors argue that large, high-dimensional pretrained word vectors make up for the missing structure when annotated data is scarce. They also release outputs from many earlier systems so that future work can use a shared benchmark.

Key points

  • The fine-grained English experiments cover 11 PDTB second-level senses, with 12,930 training, 515 development and 766 test instances. The split follows Lin et al. (2009) and Ji and Eisenstein (2015).
  • The word vectors compared include 50- and 100-dimensional Skip-gram vectors trained on WSJ text and off-the-shelf word2vec Google News vectors. For Chinese, Skip-gram vectors consistently outperform CBOW vectors.
  • On the CoNLL 2016 blind set, the summation-pooling feedforward model without manual features scores 0.3767, the best result among the systems compared. Convolutional systems do better in-domain (e.g., Wang & Lan 2016 reach 0.4091 on WSJ Test) but fall to 0.3418 on the blind data.
  • In Chinese, a model using only word vectors outperforms MaxEnt baselines built on feature sets that are known to work well for English.
  • All models were implemented in Theano. The authors recommend the CoNLL 2015–2016 evaluation scheme as the standard for comparing future systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

27 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rutherford2017systematic,
  title = {{A Systematic Study of Neural Discourse Models for Implicit Discourse Relation}},
  author = {Rutherford, Attapol and Demberg, Vera and Xue, Nianwen},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  doi = {10.18653/v1/e17-1027},
  url = {https://doi.org/10.18653/v1/e17-1027}
}

References

41 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akanksha and Jacob Eisenstein. 2016. Shallow discourse parsing using distributed argument representations and bayesian optimization. CoRR, abs/1606.04503. ↗
  2. Ozan İrsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 720–728. find
  3. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Berg-eron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. ↗
  4. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  5. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. ↗ our card
  6. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ↗
  7. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437– 478. Springer. find
  8. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral Presentation. find
  9. Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. ↗
  10. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. find
  11. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 546–556. Association for Computational Linguistics. find
  12. Alex Lascarides and Nicholas Asher. 2007. Segmented discourse representation theory: Dynamic semantics with discourse structure. In Computing meaning, pages 87–124. Springer. find
  13. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Empirical Methods in Natural Language Processing (EMNLP 2015). find our card
  14. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053. ↗ our card
  15. Jason P.C. Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308. ↗ our card
  16. Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. 2010. Convolutional networks and applications in vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253–256. IEEE. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 433– 440. Association for Computational Linguistics. find
  17. Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2304–2314, Lisbon, Portugal, September. Association for Computational Linguistics. find
  18. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical Reports (CIS), page 884. find our card
  19. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 343–351. Association for Computational Linguistics. find our card
  20. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 683–691. Association for Computational Linguistics. find our card
  21. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
  22. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L. Webber. 2008. The penn discourse treebank 2.0. In LREC. find our card
  23. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 368–375. Association for Computational Linguistics. find our card
  24. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow discourse parsing using convolutional neural network. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, page 70. find our card
  25. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. ↗ our card
  26. Attapol T. Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014), Gothenburg, Sweden, April. find our card
  27. Todor Mihaylov and Anette Frank. 2016. Discourse relation sense classification using cross-argument semantic similarity based on word embeddings. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, page 100. find our card
  28. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 799–808, Denver, Colorado, May–June. Association for Computational Linguistics. find our card
  29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. ↗
  30. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  31. Niko Schenk, Christian Chiarcos, Kathrin Donandt, Samuel Rönnqvist, Evgeny A. Stepanov, and Giuseppe Riccardi. 2016. Do we really need all those rich linguistic features? a neural network-based approach to implicit sense labeling. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, page 41. find
  32. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112. Association for Computational Linguistics. find our card
  33. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2014), 12:1532–1543. find
  34. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, Beijing, China, July. Association for Computational Linguistics. find
  35. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2755–2763. Curran Associates, Inc. find our card
  36. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 17–24, Beijing, China, July. Association for Computational Linguistics. find our card
  37. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for english and chinese in conll-2016 shared task. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, page 33. find our card
  38. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16, Beijing, China, July. Association for Computational Linguistics. find our card
  39. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Bonnie Webber, Attapol Rutherford, Chuan Wang, and Hongmin Wang. 2016. The conll-2016 shared task on multilingual shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, Berlin, Germany, August. Association for Computational Linguistics. find our card
  40. Yuping Zhou and Nianwen Xue. 2012. Pdtb-style discourse annotation of chinese text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 69–77. Association for Computational Linguistics. find
  41. Yuping Zhou and Nianwen Xue. 2015. The chinese discourse treebank: A chinese corpus annotated with discourse relations. Language Resources and Evaluation, 49(2):397–431. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rutherford-2017-systematic-study-neural-discourse-models/