Elamur Research⌕
☰
home / papers / 2018 / Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning

Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning

Summary

The paper tackles implicit discourse relation recognition: deciding how two adjacent text spans (Arg1 and Arg2) relate, for example by cause or contrast, when no explicit connective marks the link. The authors argue that earlier neural approaches have two weaknesses. They either encode the arguments independently, or they model pair interactions in only one direction. They also overlook that the word-pair patterns that signal a relation are sparse.

Their framework, TIASL, works in two stages:

  • Encoding with interactive attention. Both arguments are encoded with a Bi-LSTM. A word-by-word similarity matrix is then normalized column-wise and row-wise to produce attention in both directions, loosely mimicking a reader who moves back and forth between the two spans.
  • Pair-pattern extraction. The attended representations go into a neural tensor network (NTN). l1 regularization on the NTN prunes redundant or noisy interactions and keeps the informative pair patterns.

Experiments on PDTB show the approach is effective. Ablations indicate that each component helps. The authors present this as the first use of sparse learning for this task.

Key points

  • Data and setup: The task is framed as classification on the PDTB corpus, evaluated in both binary and four-way settings. The four-way results follow the same trends as the binary ones.
  • Interactive attention: Dot products of Bi-LSTM hidden states form a pair-wise matrix. Column-wise softmax yields Arg2-to-Arg1 attention, and row-wise softmax yields Arg1-to-Arg2 attention. Both are averaged before reweighting each argument's representation.
  • Interaction modeling: A 3-way tensor with k slices, following Socher et al. (2013a), acts as a set of feature extractors for Arg1–Arg2 interactions. Its output passes through a hidden layer and a softmax classifier trained with cross-entropy.
  • Sparse learning: An l1 penalty pushes the model to retain only the most indicative pair patterns.
  • Qualitative evidence: Heat maps on a Contingency example ("You are really lucky" / an earthquake sentence) show that interactive attention spreads high scores over more relevant word pairs than a plain Bi-LSTM. The plain Bi-LSTM fixates on pairs such as (lucky, earthquake), which could mislead it toward Comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{guo2018implicit,
  title = {{Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning}},
  author = {Guo, Fengyu and He, Ruifang and Jin, Di and Dang, Jianwu and Wang, Longbiao and Li, Xiangang},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1046/}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2201–2211. find our card
  2. Deng Cai and Hai Zhao. 2017. Pair-aware neural sentence modeling for implicit discourse relation classification. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 458–466. Springer. find
  3. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1726–1735. find our card
  4. Maxwell D Collins and Pushmeet Kohli. 2014. Memory bounded deep convolutional networks. arXiv preprint arXiv:1412.1442. — [arXiv:1412.1442](https://arxiv.org/abs/1412.1442) find
  5. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 160–167. ACM. find
  6. Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1602–1613. find
  7. Herzog, Kammer Michael H., and Scharnowski Frank Thomas. 2016. Time slices: What is the duration of a percept? PLOS Biology. find
  8. Ryuichiro Higashinaka, Kenji Imamura, Toyomi Meguro, Chiaki Miyazaki, Nozomi Kobayashi, Hiroaki Sugiyama, Toru Hirano, Toshiro Makino, and Yoshihiro Matsuo. 2014. Towards an open-domain conversational system fully based on natural language processing. In Proceedings of the 25th International Confere… find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  10. Rodolphe Jenatton, Nicolas L Roux, Antoine Bordes, and Guillaume R Obozinski. 2012. A latent factor model for highly multi-relational data. In Advances in Neural Information Processing Systems (NIPS), pages 3167–3175. find
  11. Yangfeng Ji and Jacob Eisenstein. 2014. One vector is not enough: Entity-augmented distributional semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find
  12. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1299–1… find our card
  13. Wenqiang Lei, Xuancong Wang, Meichun Liu, Ilija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: A simple word interaction model for implicit discourse relation recognition. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4026–4032. find
  14. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Emirical Methods in Natural Language Processing (EMNLP), pages 362–371. find our card
  15. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1224–1233. find our card
  16. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI), pages 2750–2756. find
  17. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems (NIPS), pages 2204–2212. find
  18. Neal Parikh and Stephen Boyd. 2014. Proximal algorithms. Foundations and Trends in Optimization., 1(3):127– 239. find
  19. Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Max-margin tensor neural network for chinese word segmentation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 293– 303. find
  20. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  21. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical Reports (CIS). find our card
  22. Emily Pitler, Annie Louis, and Ani and Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–69… find our card
  23. Rashmi Prasad, Nikhil Diesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC). find our card
  24. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Shallow discourse parsing using convolutional neural network. In CoNLL Shared Task, pages 70–77. find our card
  25. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2263–2270. find our card
  26. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 645–654. find our card
  27. Attapol T Rutherford, Vera Demberg, and Nianwen Xue. 2016. Neural network models for implicit discourse relation classification in english and chinese without surface features. arXiv preprint arXiv:1606.01990. — [arXiv:1606.01990](https://arxiv.org/abs/1606.01990) find
  28. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013a. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pages 926– 934. find
  29. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag… find
  30. Wei Wen, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. 2017. Learning intrinsic sparse structures within long short-term memory. arXiv preprint arXiv:1709.05027. — [arXiv:1709.05027](https://arxiv.org/abs/1709.05027) find
  31. Jaehong Yoon and Sung Ju Hwang. 2017. Combined group and exclusive sparsity for deep neural networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR), pages 3958–3966. find
  32. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2230–2235. find our card
  33. Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Neural discourse relation recognition with semantic memory. arXiv preprint arXiv:1603.03873. — [arXiv:1603.03873](https://arxiv.org/abs/1603.03873) find
  34. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1507–1514. Association for Computational Linguis… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2018-implicit-discourse-relation-recognition-neural/