Elamur Research⌕
☰
home / papers / 2018 / Deep Enhanced Representation for Implicit Discourse Relation Recognition

Deep Enhanced Representation for Implicit Discourse Relation Recognition

Summary

This paper tackles implicit discourse relation recognition: deciding how two text spans relate when no connective such as "but" or "so" signals the link. Because surface cues are missing and training data is limited, the authors argue that richer text representations are the main lever. Their model builds representations at several levels. At the word level, fixed pre-trained word vectors are combined with subword embeddings and frozen ELMo contextual vectors. The subwords come from BPE segmentation and are encoded with a CNN plus a highway layer. At the sentence level, each argument passes through stacked CNN or RNN encoder blocks with residual connections. At the pair level, bi-attention links Arg1 and Arg2 at every layer, and MLP classifiers make the final prediction. On PDTB 2.0, the model is the first to exceed 48% accuracy for 11-way classification and 50% macro F1 for 4-way classification. This shows that deep, multi-granular representations can push past earlier state-of-the-art results on this task.

Key points

  • Evaluation data: Penn Discourse Treebank 2.0 (PDTB 2.0), using the PDTB-Ji split for 11-way, 4-way and one-vs-others binary classification.
  • Cumulative ablation (11-way accuracy):
    • baseline: 41.48%
      • bi-attention: 42.25%
      • residual connections: 46.29%
      • subword embeddings: 47.03%
      • ELMo: 48.22%
    • Without ELMo, the model already beats the previous state of the art (Qin et al., 2017) under identical data settings.
  • Subword vs. character input: BPE subwords outperform character-level embeddings (47.03% vs 46.37%). Argument-specific encoder parameters beat shared ones (46.29% vs 45.97%).
  • Residual connections: Those inside the encoder blocks (Res 1, 45.33%) help much more than those on the pair representation (Res 2, 43.57%). Using both works best (46.29%).
  • First-level results:
    • 4-way macro F1: 51.06
    • Binary F1: Comparison 47.85, Contingency 54.47, Expansion 70.60, Temporal 36.87
    • CNN and RNN encoder blocks reach similar top accuracy, and CNN blocks train faster.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

51 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 25 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1807.05154

@inproceedings{bai2018deep,
  title = {{Deep Enhanced Representation for Implicit Discourse Relation Recognition}},
  author = {Bai, Hongxiao and Zhao, Hai},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  eprint = {1807.05154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.05154}
}

References

45 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning: Theory and Applications, pages 137–186. Berlin, Heidelberg. find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 632–642, Lisbon, Portugal, September. find
  3. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2201–2211, Lisbon, Portugal, September. find our card
  4. Deng Cai and Hai Zhao. 2016. Neural Word Segmentation Learning for Chinese. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 409–420, Berlin, Germany, August. find
  5. Deng Cai and Hai Zhao. 2017. Pair-aware neural sentence modeling for implicit discourse relation classification. In Advances in Artificial Intelligence: From Theory to Practice, pages 458–466, Cham. find
  6. Changge Chen, Peilu Wang, and Hai Zhao. 2015. Shallow discourse parsing using constituent parsing tree. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning (CoNLL) - Shared Task, pages 37–41, Beijing, China, July. find our card
  7. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1726–17… find our card
  8. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670–680, Cop… find our card
  10. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. Language modeling with gated convolutional networks. arXiv preprint arXiv:1612.08083. find our card
  11. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2017. AllenNLP: A deep semantic natural language processing platform. find our card
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, Las Vegas, USA. find
  14. Shexia He, Zuchao Li, Hai Zhao, Hongxiao Bai, and Gongshen Liu. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), Melbourne, Australia, July. find
  15. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 977–986, Baltimore, Maryland, June. find
  16. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics (TACL), 3:329–344. find our card
  17. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse-driven language models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL:HLT),… find
  18. Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. 2016. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence (AAAI), pages 2741–2749, Phoenix, USA. find our card
  19. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1299–1… find our card
  20. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL:HLT), New Orleans, Louisiana. find our card
  21. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), New… find
  22. Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014. Assessing the discourse factors that influence the quality of machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers), pages 283–288, Baltimore, Maryland, Jun… find
  23. Zhongyi Li, Hai Zhao, Chenxi Pang, Lili Wang, and Huan Wang. 2016. A constituent syntactic parse tree based discourse parser. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL) - Shared Task, pages 60–64, Berlin, Germany, August. find our card
  24. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 343–351, Singapore, August. find our card
  25. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In International Conference on Learning Representations (ICLR). find our card
  26. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1224–1233, Austin, Texas, November. find our card
  27. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In Spoken Language Technology (SLT) Workshop, pages 234–239. find
  28. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. find
  29. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  30. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of he Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Proc… find our card
  31. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth conference on International Language Resources and Evaluation (LREC-2008), pages 2961–2968, Marrakech, Morocco, May. find our card
  32. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceedings of the 26th International Conference on Computational Linguistics (COLING): Technical Papers, pages 1914–1924. find our card
  33. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. Shallow discourse parsing using convolutional neural network. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL) - Shared Task, pages 70–77, Berlin, Germany, August. find our card
  34. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016c. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2263–2270, Austin, Texas, November. find our card
  35. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1006–1017,… find our card
  36. Samuel Rönnqvist, Niko Schenk, and Christian Chiarcos. 2017. A recurrent neural model with attention for the recognition of Chinese implicit discourse relations. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers), pages 256–262,… find our card
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. find
  38. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. arXiv preprint arXiv:1507.06228. — [arXiv:1507.06228](https://arxiv.org/abs/1507.06228) find
  39. Will Williams, Niranjani Prasad, David Mrva, Tom Ash, and Tony Robinson. 2015. Scaling recurrent neural network language models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5391–5395, South Brisbane, Queensland, Australia. find
  40. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. find
  41. Changxing Wu, Xiaodong Shi, Yidong Chen, Yanzhou Huang, and Jinsong Su. 2016. Bilingually-constrained synthetic data for implicit discourse relation recognition. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2306–2312, Austin, Texas, Novembe… find our card
  42. Changxing Wu, Xiaodong Shi, Yidong Chen, Jinsong Su, and Boli Wang. 2017. Improving implicit discourse relation recognition with discourse-specific word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers), pages 269–27… find our card
  43. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2230–2235, Lisbon, Portugal, Sep… find our card
  44. Zhisong Zhang, Hai Zhao, and Lianhui Qin. 2016. Probabilistic graph-based dependency parsing with convolutional neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1382–1392, Berlin, Germany, August. find
  45. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1507–1514, Beijing, China, August. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bai-2018-deep-enhanced-representation-implicit-discourse/