Elamur Research⌕
☰
home / papers / 2015 / Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition

Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition

Summary

Implicit discourse relations are hard to classify because no explicit connective such as "because" or "but" signals the link between two text spans. Zhang et al. argue that deep networks are a poor fit for this task because PDTB training data is small and the relation classes are skewed, so deep models tend to overfit. They propose a Shallow Convolutional Neural Network (SCNN) with a single hidden layer. For each argument, SCNN pools Word2Vec-initialised word vectors with average, min and max operations. It then joins the features from both arguments, applies tanh and length normalisation, and feeds the result to a softmax classifier.

The model is tested on PDTB 2.0 as four separate one-vs-rest binary tasks over the top-level senses. SCNN beats SVM, TSVM, Brown-cluster-enhanced SVMs and a recursive autoencoder on three of the four relations, mostly because its recall is higher. The work shows that a very simple neural model can match or surpass heavily feature-engineered systems when training data is scarce.

Key points

  • Data split: The authors use PDTB 2.0, which covers 2,312 Wall Street Journal articles. Sections 2–20 are for training, 0–1 for development and 21–22 for testing. Training sets were balanced by sampling equal numbers of positive and negative instances.
  • Setup: Embeddings have d=128 and were pre-trained with Word2Vec on 1.02M unlabeled sentences (33.5M words). The parameters were tuned with L-BFGS using λ=1e-4.
  • F1 scores: SCNN reached 52.04 for CONTINGENCY, 69.59 for EXPANSION and 30.54 for TEMPORAL, which were the best results reported. For COMPARISON it scored 33.22, slightly behind No-Cro at 34.22.
  • Effect of normalisation: Length normalisation clearly helped. Measured against the SVM baseline, SCNN gained 0.71, 2.17, 6.52 and 4.93 F1 points on the four relations, and on most of them it also beat SCNN-No-Norm.
  • Other baselines: The RAE and unlabeled-data TSVM baselines mostly underperformed. Brown cluster pair features helped only for COMPARISON and CONTINGENCY.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1266

@inproceedings{zhang2015shallow,
  title = {{Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition}},
  author = {Zhang, Biao and Su, Jinsong and Xiong, Deyi and Lu, Yaojie and Duan, Hong and Yao, Junfeng},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1266},
  url = {https://doi.org/10.18653/v1/d15-1266}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155. find
  2. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. Springer Berlin Heidelberg. find
  3. Chloé Braud and Pascal Denis. 2014. Combining natural and artificial examples to improve implicit discourse relation identification. In Proc. of COLING, pages 1694–1705. find our card
  4. Philipp Cimiano, Uwe Reyle, and Jasmin Šarić. 2005. Ontology-driven discourse analysis for information extraction. Data & Knowledge Engineering, pages 59–83. find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, pages 2493–2537. find our card
  6. Francisco Guzmán, Shafiq Joty, Lluı́s Màrquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In Proc. of ACL, pages 687–698. Association for Computational Linguistics. find
  7. T. Joachims. 1999. Making large-scale SVM learning practical. In Advances in Kernel Methods - Support Vector Learning, chapter 11, pages 169–184. MIT Press, Cambridge, MA. find
  8. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. of EMNLP, pages 1746–1751. Association for Computational Linguistics. find our card
  9. Man Lan, Yu Xu, and Zhengyu Niu. 2013. Leveraging synthetic discourse data via multi-task learning for implicit discourse relation recognition. In Proc. of ACL, pages 476–485. Association for Computational Linguistics. find our card
  10. Peng Li, Yang Liu, and Maosong Sun. 2013. Recursive autoencoders for ITG-based translation. In Proc. of EMNLP, pages 567–577. Association for Computational Linguistics. find
  11. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proc. of EMNLP, pages 2061–2069. Association for Computational Linguistics. find our card
  12. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proc. of EMNLP, pages 343–351. Association for Computational Linguistics. find our card
  13. Annie Louis, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In Proc. of SIGDIAL, pages 59–62. Association for Computational Linguistics. find our card
  14. Kathleen McKeown and Or Biran. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proc. of ACL, pages 69–73. The Association for Computational Linguistics. find our card
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. CoRR, abs/1310.4546. find
  16. Eleni Miltsakaki, Nikhil Dinesh, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2005. Experiments on sense annotations and sense disambiguation of discourse connectives. In Proc. of TLT2005. find
  17. Joonsuk Park and Claire Cardie. 2012. Improving Implicit Discourse Relation Recognition Through Feature Set Optimization. In Proc. of SIGDIAL, pages 108–112. Association for Computational Linguistics. find our card
  18. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical Reports (CIS). find our card
  19. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proc. of ACL-AFNLP, pages 683– 691. Association for Computational Linguistics. find our card
  20. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In Proc. of LREC. Citeseer. find our card
  21. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proc. of EACL, pages 645–654. Association for Computational Linguistics. find our card
  22. Richard Socher, Eric H. Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y. Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc. of NIPS, pages 801–809. Curran Associates, Inc. find
  23. Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. 2011b. Parsing natural scenes and natural language with recursive neural networks. In Proc. of ICML, pages 129–136. Omnipress. find
  24. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP, pages 1631–1642. Association for Computational Linguistics. find
  25. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proc. of SIGIR, pages 735–736. ACM. find
  26. Yannick Versley. 2013. Subgraph-based classification of explicit and implicit discourse relations. In Proc. of IWCS, pages 264–275. Association for Computational Linguistics. find
  27. Xun Wang, Sujian Li, Jiwei Li, and Wenjie Li. 2012. Implicit discourse relation recognition by selecting typical training examples. In Proc. of COLING, pages 2757–2772. find our card

↑ back to top