Elamur Research⌕
☰
home / papers / 2016 / Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network

Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network

Summary

The paper addresses implicit discourse relation recognition: deciding how two text segments relate logically (e.g., Contingency vs. Comparison) when no connective such as "because" appears. Earlier systems relied heavily on hand-crafted word-pair features, which suffer from sparse data, cannot measure how semantically related two words are, and ignore context. The authors replace raw words with embeddings and run a bidirectional LSTM over each segment, so every token becomes a context-aware vector. For each token pair across the two segments, a gated relevance network (GRN) computes an interaction score. It blends a bilinear tensor, which captures linear interactions, with a single-layer network, which captures nonlinear ones, using a learned sigmoid gate. Max-pooling over the resulting score matrix keeps the strongest interactions, and an MLP with softmax then predicts the relation. Trained end to end on PDTB 2.0 without manual features, the Bi-LSTM+GRN model beats previous feature-based methods on all relation types. This shows that learned pairwise interactions can stand in for engineered linguistic features.

Key points

  • Data: Penn Discourse Treebank 2.0, which has 40,600 annotated relations over 2,312 WSJ articles. The split is sections 2–20 for training, 21–22 for testing, and the rest for validation.
  • Architecture: embedding layer (rare words mapped to an OOV vector), Bi-LSTM, GRN score matrix, non-overlapping max-pooling, MLP with softmax. The model is trained with cross-entropy loss and diagonal AdaGrad on minibatches.
  • Unlike Socher et al.'s Neural Tensor Network, the GRN learns a gate that weighs the linear and nonlinear interaction terms. It also applies no extra nonlinearity to the combined score.
  • The paper analyses the "Psyllium's not a good crop" example (gold relation Contingency, implicit "because"). With the Bi-LSTM, pairs involving "not"/"good" and "ruined" receive high scores. Plain word embeddings fail to focus on these pairs.
  • The authors report gains over prior methods on every top-level relation. The exact figures are in the results section, which was missing from the supplied text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1163

@inproceedings{chen2016implicit,
  title = {{Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network}},
  author = {Chen, Jifan and Zhang, Qi and Liu, Pengfei and Qiu, Xipeng and Huang, Xuanjing},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1163},
  url = {https://doi.org/10.18653/v1/p16-1163}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. Springer. find
  2. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Empirical Methods in Natural Language Processing (EMNLP 2015). find our card
  3. Jifan Chen, Qi Zhang, Pengfei Liu, and Xuanjing Huang. 2016. Discourse relations detection via a mixed generative-discriminative framework. In AAAI. find
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  5. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  7. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems, pages 2042–2050. find
  8. Rodolphe Jenatton, Nicolas L Roux, Antoine Bordes, and Guillaume R Obozinski. 2012. A latent factor model for highly multi-relational data. In Advances in Neural Information Processing Systems, pages 3167–3175. find
  9. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  10. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In AAAI. find
  11. Kathleen McKeown and Or Biran. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 69–73. The Association for Computational Linguistics. find our card
  12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  13. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112. Association for Computational Linguistics. find our card
  14. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical Reports (CIS), page 884. find our card
  15. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume… find our card
  16. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. Citeseer. find our card
  17. Xipeng Qiu and Xuanjing Huang. 2015. Convolutional neural tensor network architecture for community-based question answering. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), pages 1305–1311. find
  18. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In EACL, volume 645, page 2014. find our card
  19. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  20. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems, pages 926–934. find
  21. Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 149–156. Association for Co… find our card
  22. Ilya Sutskever, Joshua B Tenenbaum, and Ruslan R Salakhutdinov. 2009. Modelling relational data using bayesian clustered tensor factorization. In Advances in neural information processing systems, pages 1821–1828. find
  23. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  24. Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2015. A deep architecture for semantic matching with multiple positional sentence representations. arXiv preprint arXiv:1511.08277. — [arXiv:1511.08277](https://arxiv.org/abs/1511.08277) find
  25. Bonnie Webber. 2004. D-ltag: Extending lexicalized tag to discourse. Cognitive Science, 28(5):751–779. find
  26. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi PrasadO Christopher Bryant, and Attapol T Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of CoNLL, page 2. find our card
  27. Rong Zhao and William I Grosky. 2002. Narrowing the semantic gap-improved text-based web document retrieval using visual features. Multimedia, IEEE Transactions on, 4(2):189–200. find
  28. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1507–1514. Association for Computational Linguis… find our card

↑ back to top