Elamur Research⌕
☰
home / papers / 2016 / Variational Neural Discourse Relation Recognizer

Variational Neural Discourse Relation Recognizer

Summary

Implicit discourse relations hold between two adjacent text spans (Arg1 and Arg2) that have no explicit connective such as "because". They are hard to detect because the surface text gives almost no cue. Earlier systems treat this as discriminative classification over hand-crafted features or learned sentence embeddings, which can overfit given limited and unbalanced data.

The authors take a generative approach instead. Their model, VarNDRR, assumes that a continuous latent semantic variable produces both the argument pair and its relation label. The design borrows from the variational autoencoder. A neural "recognizer" reconstructs bag-of-words versions of both arguments and predicts the relation. Two neural "approximators" estimate a Gaussian posterior and a Gaussian prior over the latent variable. Training uses a reparameterized variational lower bound with ordinary stochastic gradient methods.

On the PDTB-style English benchmark, the model reaches F1 scores comparable to strong state-of-the-art baselines while using no manual features. The authors present this as the first generative graphical model for implicit discourse relation recognition (DRR).

Key points

  • Data and setup: Experiments use the Penn Discourse Treebank (PDTB). Implicit DRR is split into four separate tasks, one per relation type, and analysis suggests this split works better than classifying a mixture of relations.
  • Input and reconstruction:
    • Arg1 and Arg2 are encoded as bag-of-words vectors.
    • Reconstruction treats each argument as a multivariate Bernoulli distribution.
    • The relation is predicted by a softmax layer on top of a four-layer MLP applied to the latent variable z.
  • Inference networks: The posterior q(z|x,y) and prior q(z|x) are both diagonal Gaussians produced by feed-forward networks. The reparameterization trick connects the inference networks to the generative part so the whole model trains end to end.
  • Relation to prior work: The closest work is the latent variable RNN of Ji et al. (2016). VarNDRR differs in three ways:
    • it uses simple feed-forward encoders rather than recurrent ones;
    • its latent variable is a semantic representation, not the relation label itself;
    • it is optimized with variational inference rather than plain likelihood.
  • Future directions: The authors propose using explicit-connective instances, either converted into pseudo-implicit examples or exploited through semi-supervised learning. They also suggest applying the model to tasks like natural language inference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1037 · arXiv 1603.03876

@inproceedings{zhang2016variational,
  title = {{Variational Neural Discourse Relation Recognizer}},
  author = {Zhang, Biao and Xiong, Deyi and Su, Jinsong and Liu, Qun and Ji, Rongrong and Duan, Hong and Zhang, Min},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1037},
  eprint = {1603.03876},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d16-1037}
}

References

27 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chloé Braud and Pascal Denis. 2014. Combining natural and artificial examples to improve implicit discourse relation identification. In Proc. of COLING, pages 1694–1705, August. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Pr… find our card
  2. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Proc. of NIPS. Philipp Cimiano, Uwe Reyle, and Jasmin Šarić. 2005. Ontology-driven discourse analysis for information extraction. Data &… find
  3. Robert Fisher and Reid Simmons. 2015. Spectral semi-supervised discourse relation classification. In Proc. of ACL-IJCNLP, pages 89–93, July. find our card
  4. Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. CoRR, abs/1502.04623. find
  5. Ryuichiro Higashinaka, Kenji Imamura, Toyomi Me-guro, Chiaki Miyazaki, Nozomi Kobayashi, Hiroaki Sugiyama, Toru Hirano, Toshiro Makino, and Yoshihiro Matsuo. 2014. Towards an open-domain conversational system fully based on natural language processing. In Proc. of COLING, pages 928–939. find
  6. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. TACL, pages 329–344. find our card
  7. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse-driven language models. In Proc. of NAACL, pages 332–342, June. find
  8. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980. find
  9. Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In Proc. of ICLR. find our card
  10. Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Proc. of NIPS, pages 3581–3589. find
  11. Man Lan, Yu Xu, and Zhengyu Niu. 2013. Leveraging Synthetic Discourse Data via Multi-task Learning for Implicit Discourse Relation Recognition. In Proc. of ACL, pages 476–485, Sofia, Bulgaria, August. find our card
  12. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proc. of EMNLP, pages 343– 351. find our card
  13. Annie Louis, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In Proc. of SIGDIAL, pages 59–62, Tokyo, Japan, September. find our card
  14. Eleni Miltsakaki, Nikhil Dinesh, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2005. Experiments on sense annotations and sense disambiguation of discourse connectives. In Proc. of TLT2005. find
  15. Joonsuk Park and Claire Cardie. 2012. Improving Implicit Discourse Relation Recognition Through Feature Set Optimization. In Proc. of SIGDIAL, pages 108–112, Seoul, South Korea, July. find our card
  16. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proc. of EMNLP, pages 914–923. find our card
  17. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical Reports (CIS), page 884. find our card
  18. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proc. of ACL-AFNLP, pages 683–691, August. find our card
  19. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. find our card
  20. Citeseer. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proc. of ICML, pages 1278–1286. find our card
  21. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proc. of EACL, pages 645–654, April. find our card
  22. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proc. of NAACL-HLT, pages 799–808, May–June. find our card
  23. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proc. of SIGIR, pages 735–736. find
  24. Xun Wang, Sujian Li, Jiwei Li, and Wenjie Li. 2012. Implicit discourse relation recognition by selecting typical training examples. In Proc. of COLING, pages 2757–2772. find our card
  25. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In Proc. of EMNLP, pages 1834–1839, October. find our card
  26. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proc. of EMNLP, September. find our card
  27. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proc. of COLING, pages 1507–1514. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2016-variational-neural-discourse-relation-recognizer/