Elamur Research⌕
☰
home / papers / 2016 / A Stacking Gated Neural Architecture for Implicit Discourse Relation Classification

A Stacking Gated Neural Architecture for Implicit Discourse Relation Classification

Summary

Implicit discourse relations hold between adjacent sentences that have no connective such as "because." They are the hardest part of discourse parsing because the sense must be inferred from the content of the two arguments alone. The authors treat the task as sentence-pair classification and build a stacked neural model. A shared-parameter CNN, fed with word and POS embeddings, encodes each argument separately. The two resulting vectors are concatenated and passed to a new collaborative gated neural network (CGNN), followed by a softmax layer. CGNN is inspired by the Highway Network. It computes two independent sigmoid gates and applies them one after the other to an inner cell, so a feature dimension passes through only when both gates are open. This works like a soft logical AND that picks out the most relevant interaction features. The model is evaluated on the Penn Discourse Treebank (PDTB) as four one-versus-others binary tasks over the top-level senses. It beats earlier feature-based and neural systems on every class, with the largest gain on TEMPORAL. This shows that refined gating helps capture interactions between the two arguments.

Key points

  • Data: PDTB, split into sections 2–20 for training, 0–1 for development and 21–22 for testing. The four binary tasks are COMPARISON, CONTINGENCY, EXPANSION and TEMPORAL, with an extra variant that merges EntRel into EXPANSION.
  • Setup: inputs are padded or truncated to 80 tokens. Word embeddings are 300-dimensional and initialized from word2vec, POS embeddings are 50-dimensional, and the CNN has three filter groups of 1024 filters each. Training uses cross-entropy with L2 regularization and AdaGrad.
  • Ablation: average F1 is 47.48 for CNN alone and 51.45 for CNN+CGNN. CNN+CGNN also outperforms CNN combined with MLP, LSTM or Highway layers, and it is the only variant that improves COMPARISON.
  • Results without EntRel: per-class F1 is 41.55 for COMPARISON, 57.32 for CONTINGENCY, 71.50 for EXPANSION and 35.43 for TEMPORAL. The average of 51.45 compares with 48.26 for Rutherford & Xue (2014).
  • With EntRel merged into EXPANSION, average F1 reaches 53.82, against 51.72 for Chen et al. (2016) and 49.09 for Ji & Eisenstein (2015). TEMPORAL improves by 4.89 F1 points over the previous best.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

36 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1246

@inproceedings{qin2016stacking,
  title = {{A Stacking Gated Neural Architecture for Implicit Discourse Relation Classification}},
  author = {Qin, Lianhui and Zhang, Zhisong and Zhao, Hai},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1246},
  url = {https://doi.org/10.18653/v1/d16-1246}
}

References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 69–73, Sofia, Bulgaria, August. Chloé Braud and Pascal Denis. 2015. Comparing wo… find our card
  2. Deng Cai and Hai Zhao. 2016. Neural Word Segmentation Learning for Chinese. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 409–420, Berlin, Germany, August. find
  3. Changge Chen, Peilu Wang, and Hai Zhao. 2015. Shallow discourse parsing using constituent parsing tree. In Proceedings of the CoNLL-15 shared task, pages 37–41, Beijing, China, July. find our card
  4. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1726–1735, Berlin, Germany, Aug… find our card
  5. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. find
  6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation, 9(8):1735– 1780. find
  7. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 977–986, Baltimore, Maryland, June. find
  8. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics (TACL), 3:329– 344. find our card
  9. Zhongye Jia and Hai Zhao. 2014. A Joint Graph Model for Pinyin-to-Chinese Conversion with Typo Correction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1512–1523, Baltimore, Maryland, June. find
  10. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 655–665, Baltimore, Maryland, June. find our card
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October. find our card
  12. Zhongyi Li, Hai Zhao, Chenxi Pang, Lili Wang, and Huan Wang. 2016. A constituent syntactic parse tree based discourse parser. In Proceedings of the CoNLL-16 shared task, pages 60–64, Berlin, Germany, August. find our card
  13. Maria Liakata, Simon Dobnik, Shyamasree Saha, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2013. A discourse-driven content model for summarising scientific articles evaluated in a complex question answering task. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pro… find
  14. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(02):151–184. find
  15. Thomas Meyer and Andrei Popescu-Belis. 2012. Using sense-labeled discourse connectives for statistical machine translation. In Proceedings of the Joint Workshop on Exploiting Synergies between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra)… find our card
  16. Thomas Meyer and Bonnie Webber. 2013. Implicitation of discourse connectives in (machine) translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 19–26, Sofia, Bulgaria, August. find our card
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (NIPS), pages 3111–3119, South Lake Tahoe, Nevada, US, December. find
  18. Gabriel Murray, Steve Renals, Jean Carletta, and Johanna Moore. 2006. Incorporating speaker and discourse features into speech summarization. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (N… find
  19. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 108–112, Seoul, South Korea, July. find our card
  20. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of he Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Proc… find our card
  21. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth conference on International Language Resources and Evaluation (LREC-2008), pages 2961–2968, Marrakech, Morocco, May. find our card
  22. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In the 26th International Conference on Computational Linguistics (COLING), Osaka, Japan, December. find our card
  23. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. Shallow discourse parsing using convolutional neural network. In Proceedings of the CoNLL-16 shared task, pages 70–77, Berlin, Germany, August. find our card
  24. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 645–654, Gothenburg, Swe… find our card
  25. Jianxiang Wang and Man Lan. 2016. Two End-to-end Shallow Discourse Parsers for English and Chinese in CoNLL-2016 Shared Task. In Proceedings of the CoNLL-16 shared task, pages 33–40, Berlin, Germany, August. find our card
  26. Rui Wang, Hai Zhao, Bao-Liang Lu, Masao Utiyama, and Eiichiro Sumita. 2014. Neural network based bilingual language model growing for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 189–195, Doha, Qatar, Octobe… find
  27. Peilu Wang, Yao Qian, Frank K Soong, Lei He, and Hai Zhao. 2015. Word embedding for recurrent neural network based TTS synthesis. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879–4883, Brisbane, Australia. find
  28. Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2016a. Learning distributed word representations for bidirectional LSTM recurrent neural network. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  29. Rui Wang, Masao Utiyama, Isao Goto, Eiichiro Sumita, Hai Zhao, and Bao-Liang Lu. 2016b. Converting continuous-space language models into n-gram language models with efficient bilingual pruning for statistical machine translation. ACM Transactions on Asian and Low-Resource Language Information Proces… find
  30. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 Shared Task on Shallow Discourse Parsing. In Proceedings of the CoNLL-15 shared task, pages 1–16, Beijing, China, July. find our card
  31. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  32. CoNLL 2016 Shared Task on Multilingual Shallow Discourse Parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19, Berlin, Germany, August. find our card
  33. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1834–1839, Doha, Qatar, October. find our card
  34. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2230–2235, Lisbon, Portugal, Sep… find our card
  35. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Hai Zhao, Graham Neubig, and Satoshi Nakamura. 2016a. Learning local word reorderings for hierarchical phrase-based statistical machine translation. Machine Translation, pages 1–18. find
  36. Zhisong Zhang, Hai Zhao, and Lianhui Qin. 2016b. Probabilistic graph-based dependency parsing with convolutional neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1382–1392, Berlin, Germany, August. find
  37. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1507– 1514, Beijing, China, August. find our card

↑ back to top