Elamur Research⌕
☰
home / papers / 2020 / Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition

Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.coling-main.282.pdf

Summary

The paper addresses implicit discourse relation recognition (IDRR), which means deciding how two arguments relate when no explicit connective such as "because" marks the link. Earlier systems used either self-attention, which looks only within one argument, or interactive attention, which uses the paired argument as evidence. These two signals were always computed separately. The authors propose IPAL (Interactively-Propagative Attention Learning), a cross-coupled two-channel network. Each argument is first encoded with BiLSTM over GloVE vectors, or with fine-tuned BERT, and receives its own self-attentive summary. That summary is then passed to the opposite channel and steers the interactive attention there. In this way, salient cues found in one argument help pick out key words in the other. On PDTB v2.0, with one binary classifier for each top-level relation, IPAL beats its BiLSTM and BERT baselines as well as standalone self- and interactive attention on most relations. Further analysis suggests that IPAL gives sharper weights to the words that truly signal a relation. This indicates that chaining the two attention types is a cheap and useful refinement for discourse parsers.

Key points

  • Data: PDTB v2.0 with the standard split (sections 02–20 for training, 00–01 for development, 21–22 for testing). There are 12,345, 1,156 and 1,011 positive instances, and evaluation uses one-vs-rest F1 for Comparison, Contingency, Expansion and Temporality.
  • Architecture: the self-attention follows Lin et al. (2017). The mean-pooled self-attentive vector of one argument conditions the interactive attention over the other, in both directions, and the two channel outputs are concatenated and fed to an MLP with softmax.
  • With GloVE + BiLSTM, IPAL lifts F1 from 33.33/50.75/67.84/29.81 to 37.09/54.21/70.86/37.56 (COM/CON/EXP/TEM).
  • With fine-tuned BERT, the baseline scores 41.14/55.67/73.39/35.34. IPAL reaches 46.88/57.98/75.27/36.31, and IPAL combined with multi-head attention reaches 46.75/59.56/75.83/39.35.
  • Analysis with NDCG@k and attention-weight divergence shows that IPAL concentrates attention more strongly on the relation-signalling words. The authors propose adding common-sense knowledge graphs in future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.282

@inproceedings{ruan2020interactively,
  title = {{Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition}},
  author = {Ruan, Huibin and Hong, Yu and Xu, Yang and Huang, Zhen and Zhou, Guodong and Zhang, Min},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.282},
  url = {https://doi.org/10.18653/v1/2020.coling-main.282}
}

References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, pages 571–583. find our card
  2. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, pages 1726–1735. find our card
  3. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling interdependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, N… find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhikers guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392. find
  6. Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Longbiao Wang, and Xiangang Li. 2018. Implicit discourse relation recognition using neural tensor network with interactive attention and sparse learning. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa… find our card
  7. Ruifang He, Jian Wang, Fengyu Guo, and Yugui Han. 2020. Transs-driven joint learning architecture for implicit discourse relation recognition. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Li… find our card
  8. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422–446. find
  9. Douglas H Johnson. 1999. The insignificance of statistical significance testing. The journal of wildlife management, pages 763–772. find
  10. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhag… find our card
  11. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intel… find
  12. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In EMNLP, pages 343–351. find our card
  13. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  14. Zhouhan Lin, Minwei Feng, Cı́cero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. find our card
  15. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, pages 1224–1233. find our card
  16. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, Arizona, USA, pages 2750–2756. find
  17. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, pages 4068–4074. find
  18. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In COLING, pages 368–375. find our card
  19. Fandong Meng, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Interactive attention for neural machine translation. In Proceedings of the 26th International Conference on Computational Linguistics, COLING 2016, Osaka, Japan, pages 2174–2185. find
  20. Linh The Nguyen, Ngo Van Linh, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence,… find our card
  21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, pages 1532–1543. find
  22. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, AC… find our card
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L. Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008, 26 May - 1 June 2008, Marrakech, Morocco. find our card
  24. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, pages 2263–2270. find our card
  25. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric P. Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, pages 1006–101… find our card
  26. Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45(11):2673–2681. find
  27. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929–1958. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  29. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, pages 2230–2235. find our card

↑ back to top