Elamur Research⌕
☰
home / papers / 2019 / Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification

Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification

Summary

Identifying implicit discourse relations, where no connective such as "because" or "but" signals the link between two text spans, is a major bottleneck in discourse parsing. Shi and Demberg tackle this with a model that copies how PDTB annotators worked: annotators first inserted a fitting connective and only then chose a relation label. The model pairs a classifier with a seq2seq network. The network reads [Arg1; Arg2] and learns to output [Arg1; implicit connective; Arg2], using the connectives annotated in PDTB 2.0. Its parts are a bidirectional LSTM encoder over Word2Vec embeddings, a separate LSTM decoder with Luong-style global attention, a gated-interaction layer, and a context memory network that retrieves similar training instances. The classifier combines the original argument encoding, the "explicitated" decoder representation, and the memory-derived knowledge vector. The authors report state-of-the-art results on the standard 11-way PDTB classification, including under cross-validation, and strong results on top-level 4-way and binary tasks. This suggests that generating a connective as an auxiliary task produces more discriminative representations of the arguments.

Key points

  • Auxiliary task: reconstruct the arguments with the annotated implicit connective inserted, so the input and output sequences differ. This distinguishes the model from a plain autoencoder and from the adversarial connective approach of Qin et al. (2017).
  • Data: PDTB 2.0 implicit relations, with 12,345 training, 1,156 development and 1,011 test instances at the top level. Only 55 Temporal instances appear in the test set.
  • Top-level 4-way results: 61.42% accuracy (the best reported) and 46.40 macro-F1 (second best).
  • One-vs-all binary F1: Comparison 41.83, Contingency 62.07, Expansion 69.58, Temporal 35.72. The model beats prior systems on Comparison and Contingency.
  • Analysis and outlook: memory attention tends to select training examples with the same relation. The gains are not explained merely by a larger parameter count. The authors see potential for connective prediction and argument generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-0416 · arXiv 1811.01697

@inproceedings{shi2019learning,
  title = {{Learning to Explicitate Connectives with Seq2Seq Network for Implicit Discourse Relation Classification}},
  author = {Shi, Wei and Demberg, Vera},
  booktitle = {Proceedings of the 13th International Conference on Computational Semantics - Long Papers},
  year = {2019},
  doi = {10.18653/v1/w19-0416},
  eprint = {1811.01697},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w19-0416}
}

References

35 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, S., O. Vinyals, N. Jaitly, and N. Shazeer (2015). Scheduled sampling for sequence prediction with recurrent neural networks. In Proceeding of NIPS, pp. 1171–1179. find
  2. Cai, D. and H. Zhao (2017). Pair-aware neural sentence modeling for implicit discourse relation classification. In IEA-AIE, pp. 458–466. Springer. find
  3. Dai, A. M. and Q. V. Le (2015). Semi-supervised sequence learning. In Proceedings of NIPS, pp. 3079–3087. find
  4. Hochreiter, S. and J. Schmidhuber (1997). Long short-term memory. Neural computation 9(8), 1735– 1780. find
  5. Ji, Y. and J. Eisenstein (2015). One vector is not enough: Entity-augmented distributional semantics for discourse relations. TACL 3, 329–344. find
  6. Ji, Y., G. Haffari, and J. Eisenstein (2016). A latent variable recurrent neural network for discourse relation language models. In Proceedings of NAACL, pp. 332–342. find
  7. Kalchbrenner, N., E. Grefenstette, and P. Blunsom (2014). A convolutional neural network for modelling sentences. In Proceedings of ACL. find our card
  8. Kingma, D. and J. Ba (2014). arXiv:1412.6980. find
  9. Adam: A method for stochastic optimization. arXiv preprint Lan, M., J. Wang, Y. Wu, Z.-Y. Niu, and H. Wang (2017). Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of EMNLP, pp. 1299–1308. find our card
  10. Lin, Z., M.-Y. Kan, and H. T. Ng (2009). Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of EMNLP, pp. 343–351. find our card
  11. Liu, Q., Y. Zhang, and J. Liu (2018). Learning domain representation for multi-domain sentiment classification. In Proceedings of NAACL, pp. 541–550. find
  12. Liu, Y. and S. Li (2016). Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of EMNLP, pp. 1224–1233. find our card
  13. Liu, Y., S. Li, X. Zhang, and Z. Sui (2016). Implicit discourse relation classification via multi-task neural networks. In Proceedings of AAAI, pp. 2750–2756. find
  14. Luong, M.-T., H. Pham, and C. D. Manning (2015). Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP, pp. 1412–1421. find
  15. Mikolov, T., I. Sutskever, K. Chen, G. S. Corrado, and J. Dean (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pp. 3111–3119. find
  16. Miltsakaki, E., N. Dinesh, R. Prasad, A. Joshi, and B. Webber (2005). Experiments on sense annotations and sense disambiguation of discourse connectives. In Proceedings of the Fourth Workshop TLT-2005. find
  17. Prasad, R., N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber (2008). The penn discourse treebank 2.0. In Proceedings of LREC. find our card
  18. Qin, L., Z. Zhang, and H. Zhao (2016a). Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceedings of COLING. find our card
  19. Qin, L., Z. Zhang, and H. Zhao (2016b). A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of EMNLP, pp. 2263–2270. find our card
  20. Qin, L., Z. Zhang, H. Zhao, Z. Hu, and E. Xing (2017). Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of ACL, pp. 1006–1017. find our card
  21. Reimers, N. and I. Gurevych (2018). Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. arXiv preprint arXiv:1803.09578. — [arXiv:1803.09578](https://arxiv.org/abs/1803.09578) find
  22. Rutherford, A., V. Demberg, and N. Xue (2017). A systematic study of neural discourse models for implicit discourse relation. In Proceedings of EACL, pp. 281–291. find our card
  23. Rutherford, A. and N. Xue (2014). Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of EACL, pp. 645–654. find our card
  24. Rutherford, A. and N. Xue (2015). Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of NAACL, pp. 799–808. find our card
  25. Shi, W. and V. Demberg (2017). On the need of cross validation for discourse relation classification. In Proceedings of EACL, pp. 150–156. find our card
  26. Shi, W., F. Yung, and V. Demberg (2018). Acquiring annotated data with cross-lingual explicitation for implicit discourse relation classification. arXiv preprint arXiv:1808.10290. find our card
  27. Shi, W., F. Yung, R. Rubino, and V. Demberg (2017). Using explicit discourse connectives in translation for implicit discourse relation classification. In Proceedings of IJCNLP, pp. 484–495. find our card
  28. Sutskever, I., O. Vinyals, and Q. V. Le (2014). Sequence to sequence learning with neural networks. In Proceedings of NIPS, pp. 3104–3112. find
  29. Wu, C., X. Shi, Y. Chen, Y. Huang, and J. Su (2016). Bilingually-constrained synthetic data for implicit discourse relation recognition. In Proceedings of EMNLP, pp. 2306–2312. find our card
  30. Xue, N., H. T. Ng, S. Pradhan, R. Prasad, C. Bryant, and A. Rutherford (2015). The conll-2015 shared task on shallow discourse parsing. In Proceedings of CoNLL-15 Shared Task, pp. 1–16. find our card
  31. Xue, N., H. T. Ng, A. Rutherford, B. Webber, C. Wang, and H. Wang (2016). find
  32. Conll 2016 shared task on multilingual shallow discourse parsing. In Proceedings of CoNLL-16 shared task, pp. 1–19. find our card
  33. Zhang, B., J. Su, D. Xiong, Y. Lu, H. Duan, and J. Yao (2015). Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of EMNLP, pp. 2230–2235. find our card
  34. Zhou, P., W. Shi, J. Tian, Z. Qi, B. Li, H. Hao, and B. Xu (2016). Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of ACL, pp. 207–212. find our card
  35. Zhou, Z.-M., Y. Xu, Z.-Y. Niu, M. Lan, J. Su, and C. L. Tan (2010). Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of COLING, pp. 1507–1514. find our card

↑ back to top

This page in Markdown →