Elamur Research⌕
☰
home / papers / 2017 / Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification

Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification

Summary

The paper addresses implicit discourse relation recognition: deciding how two adjacent text spans (Arg-1 and Arg-2) are logically related, such as cause or contrast, when no explicit connective links them. The authors argue that simply concatenating independently encoded arguments misses cross-argument cues. They also note that labeled training data is scarce. Their model combines two ideas. First, Bi-LSTM encoders feed an attention mechanism in which each argument's summary vector weights the words of the other argument, so each pair representation depends on both spans. Second, this pair encoder is shared in a multi-task setup that pairs the main task with an auxiliary task drawing on unannotated text. Knowledge passes between tasks through equal loss sharing, weighted loss sharing, or a sigmoid-gated interaction layer. The model is evaluated on PDTB 2.0 and the CoNLL-2016 shared task data. It reaches state-of-the-art or competitive scores on most settings, which suggests that combining argument-level and task-level interaction helps a low-resource discourse task.

Key points

  • The architecture uses word embeddings and a Bi-LSTM, whose summed hidden states are re-weighted by cross-argument attention. A softmax classifier is trained with cross-entropy loss.
  • The multi-task variants are Equal Share, Weighted Share (auxiliary loss scaled by w ∈ (0, 1]) and Sigmoid (Gated) Interaction. Small values of w worked best, because they limit noise from the auxiliary task.
  • On PDTB binary classification, the reported F1 scores are 40.73 for Comparison, 58.96 for Contingency, 72.47 for Expansion, 81.36 for Expansion+EntRel and 38.50 for Temporal.
  • On PDTB 4-way classification, the model reaches F1 47.80 with accuracy 57.39. Accuracy is slightly below the best compared system (57.57).
  • On CoNLL-2016, the model scores 39.40 and 40.12, most likely on the test and blind test sets. It beats Wang and Lan (2016) on the second set but not the first. The model also outperforms Liu et al. (2016b) and the authors' earlier traditional multi-task system, Lan et al. (2013).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

39 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1134

@inproceedings{lan2017multi,
  title = {{Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification}},
  author = {Lan, Man and Wang, Jianxiang and Wu, Yuanbin and Niu, Zheng-Yu and Wang, Haifeng},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1134},
  url = {https://doi.org/10.18653/v1/d17-1134}
}

References

30 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Empirical Methods in Natural Language Processing (EMNLP 2015). find our card
  2. Chloé Braud and Pascal Denis. 2016. Learning connective-based word representations for implicit discourse relation identification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 203–213, Austin, Texas. Association for Computational Linguistics. find our card
  3. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of ACL. find our card
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  5. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  6. Robert Fisher and Reid Simmons. 2015. Spectral semi-supervised discourse relation classification. Volume 2: Short Papers. find our card
  7. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  8. Man Lan, Yu Xu, and Zheng-Yu Niu. 2013. Leveraging synthetic discourse data via multi-task learning for implicit discourse relation recognition. In ACL (1), pages 476–485. Citeseer. find our card
  9. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 343– 351. Association for Computational Linguistics. find our card
  10. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016a. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. — [arXiv:1605.05101](https://arxiv.org/abs/1605.05101) find
  11. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224–1233, Austin, Texas. Association for Computational Linguistic… find our card
  12. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016b. Implicit discourse relation classification via multi-task neural networks. arXiv preprint arXiv:1603.02776. find
  13. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume… find our card
  14. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In LREC. Citeseer. find our card
  15. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Implicit discourse relation recognition with con-textaware character-enhanced embeddings. In the 26th International Conference on Computational Linguistics (COLING), Osaka, Japan, December. find
  16. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, USA, November. find our card
  17. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the NAACL-HLT. find our card
  18. Attapol T Rutherford and Nianwen Xue. 2016. Robust non-explicit neural discourse parser in english and chinese. find our card
  19. Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2016. Improved representation learning for question answer matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 464–473, Berlin, Germany. Association for Computatio… find
  20. Bingning Wang, Kang Liu, and Jun Zhao. 2016a. Inner attention based recurrent neural networks for answer selection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1288–1297, Berlin, Germany. Association for Computational Ling… find
  21. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for english and chinese in conll-2016 shared task. Proceedings of the CoNLL-16 shared task, pages 33–40. find our card
  22. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016b. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1298– 1307, Berlin, Germany. Association for Computational L… find
  23. WenTing Wang, Jian Su, and Chew Lim Tan. 2010. Kernel based discourse relation recognition with temporal ordering information. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 710–719. Association for Computational Linguistics. find our card
  24. Yequan Wang, Minlie Huang, xiaoyan zhu, and Li Zhao. 2016c. Attention-based lstm for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606–615, Austin, Texas. Association for Computational Linguistics. find
  25. Changxing Wu, xiaodong shi, Yidong Chen, Yanzhou Huang, and jinsong su. 2016. Bilingually-constrained synthetic data for implicit discourse relation recognition. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2306–2312, Austin, Texas. Association for… find our card
  26. Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and Hinrich Schütze. 2016. Simple question answering by attentive convolutional neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1746–1756, Osaka, Japan. find
  27. The COLING 2016 Organizing Committee. find
  28. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. find our card
  29. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20… find our card
  30. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1507– 1514, Stroudsburg, PA, USA. Association for Computa… find our card

↑ back to top