Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention

Summary

Implicit discourse relations, such as contrast or causality between two text spans (arguments) that lack an overt connective like "because", are hard to classify automatically. Liu and Li argue that earlier feature-based and neural classifiers effectively scan the argument pair only once, while people facing difficult text reread it with sharper focus each time. They model this behaviour with NNMA (Neural Networks with Multi-Level Attention). A general level encodes each argument with a bi-LSTM over GloVe embeddings and mean-pools the outputs. Several stacked attention levels then reweight the words of both arguments, guided by an external short-term memory vector that is refreshed recurrently from the previous level. The top level's output feeds a softmax classifier. On the PDTB, using the four top-level relation classes, the authors report state-of-the-art results. Attention visualizations show the model homing in on relation-bearing words over successive passes. The work is an early demonstration that memory-guided, multi-pass attention benefits discourse parsing.

Key points

Task and data: implicit relation classification on the PDTB, using the top-level senses Temporal, Contingency, Comparison and Expansion.
General level: two separate bi-LSTMs (one for Arg-1, one for Arg-2) initialized with pre-trained GloVe vectors, followed by mean pooling to obtain global argument vectors.
First memory: built by a nonlinear transform of the two argument vectors and their difference. At each later level, the memory also takes the previous memory as input and drives softmax attention for attention-weighted pooling of the bi-LSTM states.
Final prediction: a softmax over the top-level argument representations and their difference. The design extends to any number of attention levels.
Analysis: per-level attention heatmaps for Comparison, Contingency and Expansion examples, plus KL-divergences between levels, show attention becoming progressively more focused. Exact scores fall in the portion of the paper missing from the extracted text and are not reproduced here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

43 papers of our library cite it (of the 140673 whose bibliographies we read); 30 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1130 · arXiv 1609.06380

@inproceedings{liu2016recognizing,
  title = {{Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention}},
  author = {Liu, Yang and Li, Sujian},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1130},
  eprint = {1609.06380},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1130}
}

References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
Jason Baldridge and Alex Lascarides. 2005. Probabilistic head-driven parsing for discourse structure. In Proceedings of CoNLL. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of EMNLP. Ronan Collobert, Jason Weston, L… find
Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. Hugo Hernault, Helmut Prendinger, David A duVerle, Mitsuru Ishizuka, et al. 2010. Hilda: a discourse parser using support vector machine classification. Dialogue and Discourse, 1(3):1–33. find
Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329– 344. find our card
Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. arXiv preprint arXiv:1603.01913. — [arXiv:1603.01913](https://arxiv.org/abs/1603.01913) find
Shafiq Joty, Giuseppe Carenini, and Raymond T Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of EMNLP. find
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. find our card
Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285. find our card
Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of EMNLP. find our card
Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural network. In Proceedings of AAAI. find
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of NAACL. find
Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of SigDial. find our card
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP 2014. find
Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of ACL. find our card
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L. Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of LREC. find our card
Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of EACL. find our card
Attapol T Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of NAACL. find our card
Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of NAACL. find our card
Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In Proceedings of NAACL. find our card
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Proceedings of NIPS. find
Ming Tan, Bing Xiang, and Bowen Zhou. 2015. Lstm-based deep learning models for non-factoid answer selection. arXiv preprint arXiv:1511.04108. — [arXiv:1511.04108](https://arxiv.org/abs/1511.04108) find
Di Wang and Eric Nyberg. 2015. A long short-term memory model for answer sentence selection in question answering. In Proceedings of EMNLP. find
Xun Wang, Sujian Li, Jiwei Li, and Wenjie Li. 2012. Implicit Discourse Relation Recognition by Selecting Typical Training Examples. In Proceedings of COLING. find our card
Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. Proceedings of ICLR. find
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2015. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274. find
Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615. find
Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of EMNLP. find our card
Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the ICCL, pages 1507–1514. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2016-recognizing-implicit-discourse-relations-repeated/

📄 Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention