Elamur Research⌕
☰
home / papers / 2015 / One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations

One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations

Summary

Ji and Eisenstein (TACL, 2015) tackle the problem of determining which relation holds between arguments when a discourse relation is implicit, using data from the Penn Discourse Treebank (PDTB). This task is hard because few surface cues exist, and the relation often depends on the meaning of the arguments and on how entities are shared between them. The authors argue that a single bottom-up vector per argument cannot separate cases like "She was hungry" from "He was hungry," since he and she have nearly identical embeddings. Their model has two parts. A Recursive Neural Network first builds argument representations bottom-up over binarized syntactic parse trees. A second, downward pass then gives each node a vector built from its parent's downward vector and its sibling's upward vector, similar to the inside-outside algorithm. Relations are scored with bilinear products of the root vectors, plus products of the downward vectors of coreferent entity mentions shared by the two arguments. Combined with a few surface features and trained with a large-margin objective, the model outperforms earlier work. This suggests that discourse semantics needs representations of entities, not just whole sentences.

Key points

  • On multiclass implicit relation classification in the PDTB, accuracy is about 4% higher than the previous best system (Lin et al., 2009), whose performance was roughly 40%. The model also beats more recent work on binary classification.
  • The up-down network stays feedforward, so all upward and downward vectors are computed in linear time. Matrix-vector composition needs on the order of K² parameters, compared with K³ for a tensor alternative.
  • To reduce overfitting, each classification matrix is approximated by a rank-one term plus a diagonal term.
  • Entity vectors give only a small gain overall, because the coreference resolver has limited recall. On argument pairs where coreference is detected, the gain from entity semantics exceeds 2%.
  • The composition procedure matches the Inside-Outside Recursive Neural Network of Le and Zuidema (2014), but that work applies it to dependency parsing rather than discourse relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Coreference resolution

Cited by

64 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 26 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00142

@misc{ji2015one,
  title = {{One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations}},
  author = {Ji, Yangfeng and Eisenstein, Jacob},
  year = {2015},
  doi = {10.1162/tacl\_a\_00142},
  url = {https://doi.org/10.1162/tacl_a_00142}
}

References

75 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Raffaella Bernardi, and Roberto Zamparelli. 2014a. Frege in space: A program for compositional distributional semantics. Linguistic Issues in Language Technologies. find
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014b. Dont count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, MD. find our card
  3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An Entity-Based approach. Computational Linguistics, 34(1):1–34, March. find our card
  4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  5. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. Springer. find
  6. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437– 478. Springer. find
  7. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the Association for Computational Linguistics (ACL), pages 69–73, Sophia, Bulgaria. find our card
  8. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 546–556. find
  9. Léon Bottou. 1998. Online learning and stochastic approximations. On-line learning in neural networks, 17:9. find
  10. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 740–750. find our card
  11. Freddy YY Choi, Peter Wiemer-Hastings, and Johanna Moore. 2001. Latent semantic analysis for text segmentation. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). find
  12. Stephen Clark, Bob Coecke, and Mehrnoosh Sadrzadeh. 2011. Mathematical foundations for a compositional distributed model of meaning. Linguistic Analysis, 36(1-4):345–384. find
  13. Shay B Cohen, Karl Stratos, Michael Collins, Dean P Foster, and Lyle Ungar. 2014. Spectral learning of latent-variable PCFGs: Algorithms and sample complexity. Journal of Machine Learning Research, 15:2399–2449. find
  14. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  15. Simon Corston-Oliver. 1998. Beyond string matching and cue phrases: Improving efficiency and coverage in discourse analysis. In The AAAI Spring Symposium on Intelligent Text Summarization, pages 9–15. find
  16. Dan Cristea, Nancy Ide, and Laurent Romary. 1998. Veins theory: A model of global discourse cohesion and coherence. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1, pages 281– 285. Asso… find
  17. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  18. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  19. J. R. Firth. 1957. Papers in Linguistics 1934-1951. Oxford University Press. find
  20. Peter W Foltz, Walter Kintsch, and Thomas K Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse processes, 25(2-3):285–307. find
  21. Katherine Forbes-Riley, Bonnie Webber, and Aravind Joshi. 2006. Computing discourse semantics: The predicate-argument semantics of discourse connectives in d-ltag. Journal of Semantics, 23(1):55–106. find
  22. Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, IEEE International Conference on, pages 347–352. IEEE. find
  23. Edward Grefenstette. 2013. Towards a Formal Distributional Semantics: Simulating Logical Calculi with Tensors, April. find
  24. Barbara J Grosz, Scott Weinstein, and Aravind K Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Computational linguistics, 21(2):203–225. find our card
  25. Ozan İrsoy and Claire Cardie. 2013. Bidirectional recursive neural networks for token-level labeling with structure. CoRR (presented at the 2013 NIPS Workshop on Deep Learning), abs/1312.0493. — [arXiv:1312.0493](https://arxiv.org/abs/1312.0493) find
  26. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, MD. find
  27. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, MD. find our card
  28. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
  29. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In Proceedings of NAACL, pages 455–462. Association for Computational Linguistics. find our card
  30. Andrew Kehler and Hannah Rohde. 2013. A probabilistic reconciliation of coherence-driven and centering-driven theories of pronoun interpretation. Theoretical Linguistics, 39(1-2):1–37. find
  31. Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the Association for Computational Linguistics (ACL), pages 423–430. find our card
  32. Alistair Knott. 1996. A data-driven methodology for motivating a set of coherence relations. Ph.D. thesis, The University of Edinburgh. find
  33. Thomas Landauer, Peter W. Foltz, and Darrel Laham. 1998. Introduction to Latent Semantic Analysis. Discource Processes, 25:259–284. find
  34. Karim Lari and Steve J Young. 1990. The estimation of stochastic context-free grammars using the inside-outside algorithm. Computer speech & language, 4(1):35–56. find
  35. Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 729–739. find
  36. Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. 2012. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–48. Springer. find
  37. Mike Lewis and Mark Steedman. 2013. Combined distributional and logical semantics. Transactions of the Association for Computational Linguistics, 1:179–192. find
  38. Junyi Jessy Li and Ani Nenkova. 2014. Reducing sparsity improves the recognition of implicit discourse relations. In Proceedings of SIGDIAL, pages 199– 207. find our card
  39. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). find our card
  40. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 343–351, Singapore. find our card
  41. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically Evaluating Text Coherence Using Discourse Relations. In Proceedings of the Association for Computational Linguistics (ACL), pages 997–1006, Portland, OR. find our card
  42. Ziheng Lin, Hwee T. Ng, and Min Y. Kan. 2012. A PDTB-styled end-to-end discourse parser. Natural Language Engineering, FirstView:1–34, November. find
  43. Annie Louis, Aravind Joshi, and Ani Nenkova. 2010a. Discourse indicators for content selection in summarization. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 147–156. Association for Computational Linguistics. find
  44. Annie Louis, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010b. Using entity features to classify implicit discourse relations. In Proceedings of the SIGDIAL, pages 59–62, Tokyo, Japan, September. Association for Computational Linguistics. find our card
  45. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. find our card
  46. William Mann. 1984. Discourse structures for text generation. In Proceedings of the 10th International Conference on Computational Linguistics and 22nd annual meeting on Association for Computational Linguistics, pages 367–375. Association for Computational Linguistics. find
  47. Daniel Marcu. 1999. A decision-based approach to rhetorical parsing. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, pages 365–372. Association for Computational Linguistics. find
  48. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. 2006. Corpus-based and knowledge-based measures of text semantic similarity. In AAAI. find
  49. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 746–751, Stroudsburg, Pennsylvania. Association for Computational Lingu… find
  50. Raymond J. Mooney. 2014. Semantic parsing: Past, present, and future. Presentation slides from the ACL Workshop on Semantic Parsing. find
  51. Joonsuk Park and Claire Cardie. 2012. Improving Implicit Discourse Relation Recognition Through Feature Set Optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112, Seoul, South Korea, July. Association for Computational Linguist… find our card
  52. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  53. Romain Paulus, Richard Socher, and Christopher D Manning. 2014. Global belief recursive neural networks. In Neural Information Processing Systems (NIPS), pages 2888–2896, Montréal. find
  54. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 1532–1543. find
  55. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the Association for Computational Linguistics (ACL). find our card
  56. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90, Manchester, UK, August. find our card
  57. Coling 2008 Organizing Committee. find
  58. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic Sense Prediction for Implicit Discourse Relations in Text. In Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP . find our card
  59. Hoifung Poon and Pedro Domingos. 2009. Unsupervised semantic parsing. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 1–10, Singapore. find
  60. Sameer S Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. International Journal of Semantic Computing, 1(04):405–419. find
  61. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of LREC. find our card
  62. Attapol T Rutherford and Nianwen Xue. 2014. Discovering Implicit Discourse Relations Through Brown Cluster Pair Representation and Coreference Patterns. In Proceedings of the European Chapter of the Association for Computational Linguistics (EACL), Stroudsburg, Pennsylvania. Association for Computat… find our card
  63. Paul Smolensky. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1):159–216. find
  64. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML), Seattle, WA. find
  65. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). find
  66. Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2014. Reasoning With Neural Tensor Networks For Knowledge Base Completion. In Neural Information Processing Systems (NIPS). Lake Tahoe. find
  67. Swapna Somasundaran, Galileo Namata, Janyce Wiebe, and Lise Getoor. 2009. Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In Proceedings of EMNLP. find our card
  68. Ivan Titov and Alexandre Klementiev. 2011. A bayesian model for unsupervised semantic parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1445– 1455. Association for Computational Linguistics. find
  69. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representation: A Simple and General Method for Semi-Supervised Learning. In Proceedings of the Association for Computational Linguistics (ACL), pages 384–394, Uppsala, Sweden. find
  70. Peter D. Turney and Patrick Pantel. 2010. From Frequency to Meaning: Vector Space Models of Semantics. JAIR, 37:141–188. find
  71. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9:2759–2605, November. find
  72. Bonnie Webber, Alistair Knott, Matthew Stone, and Aravind Joshi. 1999. Discourse relations: A structural and presuppositional account using lexicalised tag. In Proceedings of the Association for Computational Linguistics (ACL), pages 41–48. find our card
  73. Bonnie Webber. 2004. D-LTAG: extending lexicalized TAG to discourse. Cognitive Science, 28(5):751– 779, September. find
  74. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based Discourse Parser for Single-Document Summarization. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). find our card
  75. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1507– 1514. find our card

↑ back to top