Elamur Research
☰
home / papers / 2014 / Representation Learning for Text-level Discourse Parsing

Representation Learning for Text-level Discourse Parsing

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P14-1002.pdf

Summary

RST discourse parsing builds a tree of rhetorical relations over a whole document. It is hard because telling relations apart takes semantic judgment. Lexical and syntactic features capture this poorly, especially with small annotated corpora. Before this paper, the best relation F-measure on the RST Treebank was only a little above 55%. Ji and Eisenstein propose DPLP (Discourse Parsing from Linear Projection). It is a shift-reduce parser that maps bag-of-words features of elementary discourse units (EDUs) into a low-dimensional latent space and makes its parsing decisions there. The classifier weights are trained with a multi-class large-margin (SVM) objective. Training alternates between fitting these weights and taking gradient steps on the projection matrix, so the parser and the representation are learned together. Sharing parameters this way reduces feature sparsity. On the RST Treebank, DPLP clearly beats the previous state of the art on relation and nuclearity detection. The learned space also groups discourse-relevant words in a sensible way, which shows that discourse annotations can serve as supervision for learning representations.

Key points

  • The parser looks at three EDUs at a time: the top two stack elements and the front of the queue. It uses large-margin transition-based structured prediction instead of the decision trees or perceptrons used in earlier shift-reduce work.
  • Three shapes of the projection matrix A are compared: general, concatenation (one shared block B applied to each EDU) and difference (encodes differences between EDUs to capture semantic similarity).
  • The parameter count falls from 3VC for a plain linear classifier to (3V+C)K for DPLP, where K is the small latent dimension.
  • Labeled F-measure improves by roughly 6 points on relations and about 2.5 points on nuclearity over the prior best system (Joty et al., 2013).
  • A t-SNE visualization of the learned word representations (K = 20) groups modal verbs, reporting verbs and PDTB precedence/succession connectives together. Unsupervised NMF clusters show no such discourse structure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Treebanks

Cited by

116 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1002

@inproceedings{ji2014representation,
  title = {{Representation Learning for Text-level Discourse Parsing}},
  author = {Ji, Yangfeng and Eisenstein, Jacob},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1002},
  url = {https://doi.org/10.3115/v1/p14-1002}
}

References

55 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jason Baldridge and Alex Lascarides. 2005. Probabilistic head-driven parsing for discourse structure. In Proceedings of the Ninth Conference on Computational Natural Language Learning, pages 96–103. find
  2. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828. find our card
  3. Ezra Black, Steve Abney, Dan Flickinger, Claudia Gdaniec, Ralph Grishman, Phil Harrison, Don Hindle, Robert Ingria, Fred Jelinek, Judith Klavans, Mark Liberman, Mitchell Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991, pages 306–311. find
  4. Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013. Holistic discourse coherence annotation for noisy essay writing. Dialogue & Discourse, 4(2):34–52. find
  5. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2001. Building a Discourse-tagged Corpus in the Framework of Rhetorical Structure Theory. In Proceedings of Second SIGdial Workshop on Discourse and Dialogue. find our card
  6. Michael Collins and Brian Roark. 2004. Incremental parsing with the perceptron algorithm. In Proceedings of ACL, page 111. Association for Computational Linguistics. find our card
  7. R. Collobert and J. Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In ICML. find
  8. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  9. Simon Corston-Oliver. 1998. Beyond string matching and cue phrases: Improving efficiency and coverage in discourse analysis. In The AAAI Spring Symposium on Intelligent Text Summarization, pages 9–15. find
  10. Koby Crammer and Yoram Singer. 2001. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. Journal of Machine Learning Research, 2:265–292. find
  11. Georgiana Dinu and Mirella Lapata. 2010. Measuring Distributional Similarity in Context. In EMNLP, pages 1162–1172. find
  12. Vanessa Wei Feng and Graeme Hirst. 2012. Text-level Discourse Parsing with Rich Linguistic Features. In Proceedings of ACL. find our card
  13. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building Watson: An overview of the DeepQA project. AI magazine, 31(3):59–79. find
  14. Katherine Forbes-Riley, Bonnie Webber, and Aravind Joshi. 2006. Computing discourse semantics: The predicate-argument semantics of discourse connectives in D-LTAG. Journal of Semantics, 23(1):55– 106. ↗
  15. Weiwei Guo and Mona Diab. 2012. Modeling Sentences in the Latent Space. In Proceedings of ACL, pages 864–872, Jeju Island, Korea, July. Association for Computational Linguistics. find
  16. Hugo Hernault, Helmut Prendinger, David A. duVerle, and Mitsuru Ishizuka. 2010. HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue and Discourse, 1(3):1–33. find our card
  17. Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative Improvements to Distributional Sentence Similarity. In EMNLP, pages 891–896, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  18. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining Intra- and Multi-sentential Rhetorical Parsing for Document-level Discourse Analysis. In Proceedings of ACL. find
  19. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
  20. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple Semi-supervised Dependency Parsing. In Proceedings of ACL-HLT, pages 595–603, Columbus, Ohio, June. Association for Computational Linguistics. find our card
  21. Ralf Krestel, Sabine Bergler, and René Witte. 2008. Minding the Source: Automatic Tagging of Reported Speech in Newspaper Articles. In LREC, Marrakech, Morocco, May. European Language Resources Association (ELRA). find
  22. Quoc V. Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y. Ng. 2011. On Optimization Methods for Deep Learning. In ICML. find
  23. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing Implicit Discourse Relations in the Penn Discourse Treebank. In EMNLP. find our card
  24. Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 147–156. Association for Computational Linguistics. find
  25. Daniel Marcu and Abdessamad Echihabi. 2002. An Unsupervised Approach to Recognizing Discourse Relations. In Proceedings of ACL, pages 368–375, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. find our card
  26. Daniel Marcu. 1996. Building Up Rhetorical Structure Trees. In Proceedings of AAAI. find
  27. Daniel Marcu. 1999. A Decision-Based Approach to Rhetorical Parsing. In Proceedings of ACL, pages 365–372, College Park, Maryland, USA, June. Association for Computational Linguistics. find
  28. Daniel Marcu. 2000a. The Rhetorical Parsing of Unrestricted Texts: A Surface-based Approach. Computational Linguistics, 26:395–448. find our card
  29. Daniel Marcu. 2000b. The Theory and Practice of Discourse Parsing and Summarization. MIT Press. find
  30. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name Tagging with Word Clusters and Discriminative Training. In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL, pages 337–342, Boston, Massachusetts, USA, May 2 - May 7. Association for Computational Linguistics. find
  31. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(1):25–55. find
  32. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  33. Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. Constrained Decoding for Text-Level Discourse Parsing. In Coling, pages 1883–1900, Mumbai, India, December. find
  34. The COL-ING 2012 Organizing Committee. find
  35. Anil Kumar Nelakanti, Cedric Archambeau, Julien Mairal, Francis Bach, and Guillaume Bouchard. 2013. Structured Penalties for Log-Linear Language Models. In EMNLP, pages 233–243, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  36. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. find
  37. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In LREC. find our card
  38. Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2010. Realization of discourse relations by other means: alternative lexicalizations. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1023–1031. Association for Computational Linguistics. find
  39. Kenji Sagae. 2009. Analysis of Discourse Structure with Syntactic Dependencies and Data-Driven Shift-Reduce Parsing. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT), pages 81–84, Paris, France, October. Association for Computational Linguistics. find
  40. Frank Schilder. 2002. Robust discourse parsing via discourse markers, topicality and position. Natural Language Engineering, 8(3):235–255. find
  41. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS. find
  42. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In EMNLP. find
  43. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  44. Swapna Somasundaran, Galileo Namata, Janyce Wiebe, and Lise Getoor. 2009. Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In Proceedings of EMNLP. find
  45. Radu Soricut and Daniel Marcu. 2003. Sentence Level Discourse Parsing using Syntactic and Lexical Information. In NAACL. find our card
  46. Rajen Subba and Barbara Di Eugenio. 2009. An effective Discourse Parser that uses Rich Linguistic Information. In NAACL-HLT, pages 566–574, Boulder, Colorado, June. Association for Computational Linguistics. find
  47. K. Sumita, K. Ono, T. Chino, T. Ukita, and S. Amano. 1992. A discourse structure analyzer for Japanese text. In Proceedings International Conference on Fifth Generation Computer Systems, pages 1133– 1140. find
  48. Maite Taboada and William C Mann. 2006. Applications of rhetorical structure theory. Discourse studies, 8(4):567–588. find
  49. Benjamin Taskar, Carlos Guestrin, and Daphne Koller. 2003. Max-margin markov networks. In NIPS. find
  50. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representation: A Simple and General Method for Semi-Supervised Learning. In Proceedings of ACL, pages 384–394. find
  51. Tim Van de Cruys and Marianna Apidianaki. 2011. Latent Semantic Word Sense Induction and Disambiguation. In Proceedings of ACL, pages 1476– 1485, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  52. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9:2759–2605, November. find
  53. Kimberly Voll and Maite Taboada. 2007. Not all words are created equal: Extracting semantic orientation as a function of adjective relevance. In Proceedings of Australian Conference on Artificial Intelligence. find
  54. Ngo Xuan Bach, Nguyen Le Minh, and Akira Shimazu. 2012. A Reranking Model for Discourse Segmentation using Subtree Features. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 160–168. find
  55. Chun-Nam John Yu and Thorsten Joachims. 2009. Learning structural SVMs with latent variables. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1169–1176. ACM. find