Elamur Research⌕
☰
home / papers / 2019 / A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing

A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing

Summary

Implicit discourse relations in PDTB, which lack an explicit connective, remain hard to classify. Earlier systems reached macro F1 below 50%, and the authors attribute much of this gap to data sparsity. Dai and Huang propose adding external event knowledge (for example, causal or temporal links between event words) and entity coreference links to a neural parser without letting these signals override context. They build on their paragraph-level hierarchical BiLSTM model with a CRF output layer. Between the word-level and DU-level BiLSTMs, they insert a knowledge layer that transforms the hidden states of words involved in a relation using relation-type-specific feedforward networks. A TransE-style regularization term added to the training loss constrains these transformed vectors, which balances attention between knowledge and context. On PDTB v2.0, the approach improves both implicit and explicit relation classification and outperforms previous systems. Removing the regularizer makes results worse than the base model. This indicates that external knowledge needs controlled integration to help rather than hurt.

Key points

  • The base model is a re-implementation of Dai and Huang (2018) with character-level CNN features. It supports 300D GloVe or 1024D ELMo embeddings. POS, NER and coreference annotations come from Stanford CoreNLP.
  • With ELMo, the full model improves over the base model by 2.1 macro-F1 and 3.2 accuracy points on top-level implicit relations. It also beats Bai and Zhao (2018), the strongest earlier system, by more than 1.8 macro-F1.
  • Several alternatives gained little over the base model: adding knowledge as discrete word-level or DU-level features, or using a two-step approach with fixed knowledge embeddings plus attention. The model also improves second-level fine-grained classification.
  • On the dev set, the gains come mostly from the 507 implicit instances that carry event or coreference constraints. Errors on these instances fell from 217 to 193, with 60 corrections against 36 false reversions. On the 717 unconstrained instances, results were similar (407 vs. 402 correct).
  • About one third of the reversions came from noisy event knowledge or coreference errors, and the rest from over-reliance on constraints. The authors suggest antonym knowledge as future work to help with Comparison relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Coreference resolution

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1295

@inproceedings{dai2019regularization,
  title = {{A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing}},
  author = {Dai, Zeyu and Huang, Ruihong},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1295},
  url = {https://doi.org/10.18653/v1/d19-1295}
}

References

59 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In COLING, pages 571–583. find our card
  3. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In ACL, volume 2, pages 69–73. find our card
  4. Or Biran and Kathleen McKeown. 2015. Pdtb discourse parsing as a tagging task: The two taggers approach. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 96–104. find our card
  5. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NeurIPS, pages 2787–2795. find
  6. Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. 2018. Discourse-aware neural rewards for coherent text generation. In NAACL-HLT, pages 173–184. find
  7. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In EMNLP, pages 2201–2211. find our card
  8. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. find our card
  9. Bhuwan Dhingra, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017. Linguistic knowledge as memory for recurrent neural networks. arXiv preprint arXiv:1703.02620. — [arXiv:1703.02620](https://arxiv.org/abs/1703.02620) find
  10. Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In EMNLP, pages 1602–1613. find
  11. Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Long-biao Wang, and Xiangang Li. 2018. Implicit discourse relation recognition using neural tensor network with interactive attention and sparse learning. In COLING. find our card
  12. Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In ACL, volume 1, pages 687–696. find our card
  13. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. TACL, pages 329–344. find our card
  14. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In NAACL-HLT, pages 332–342. find
  15. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2018. A knowledge-augmented neural network model for implicit discourse relation classification. In COLING, pages 584–595. find
  16. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In ACL. find our card
  17. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001, pages 282–289. find
  18. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018. Neural natural language inference models enhanced with external knowledge. In ACL, pages 2406–2417. find
  19. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In EMNLP, pages 1310–1319. find our card
  20. Timothy Chklovski and Patrick Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In EMNLP. find
  21. Kevin Clark and Christopher D Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In ACL, volume 1, pages 643–653. find our card
  22. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, pages 681–691. find our card
  23. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In NAACL, volume 1, pages 141–151. find our card
  24. Wenqiang Lei, Xuancong Wang, Meichun Liu, Ilija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: A simple word interaction model for implicit discourse relation recognition. In IJCAI, pages 4026–4032. find
  25. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In AAAI. find
  26. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago press. find
  27. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI. find
  28. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In EMNLP, pages 343–351. find our card
  29. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  30. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In EMNLP, pages 1224–1233. find our card
  31. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In AAAI, pages 2750– 2756. find
  32. Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm. In AAAI. find
  33. William C Mann. 1984. Discourse structures for text generation. In 10th International Conference on Computational Linguistics and 22nd Annual Meeting of the Association for Computational Linguistics. find
  34. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL System Demonstrations, pages 55–60. find our card
  35. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112. find our card
  36. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. A stacking gated neural architecture for implicit discourse relation classification. In EMNLP, pages 2263–2270. find our card
  37. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric P. Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In ACL, pages 1006–1017. find our card
  38. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In EACL, pages 281–291. find our card
  39. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In EACL, pages 645–654. find our card
  40. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In NAACL-HLT, pages 799–808. find our card
  41. Attapol T Rutherford and Nianwen Xue. 2016. Robust non-explicit neural discourse parser in english and chinese. ACL, page 55. find our card
  42. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML, pages 1818–1826. find
  43. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
  44. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In NeurIPS, pages 926–934. find
  45. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  46. Robert Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679–3686. find
  47. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT, volume 1, pages 2227– 2237. find
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008. find
  49. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In ACL, pages 683–691. find our card
  50. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In ACL-IJCNLP, pages 13–16. find our card
  51. R. Prasad, N. Dinesh, Lee A., E. Miltsakaki, L. Robaldo, Joshi A., and B. Webber. 2008. The Penn Discourse Treebank 2.0. In lrec2008. find our card
  52. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In COL-ING, pages 1914–1924. find our card
  53. Bonnie Webber, Andrei Popescu-Belis, and Jörg Tiedemann. 2017. Proceedings of the third workshop on discourse in machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation. find
  54. Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, and Guodong Zhou. 2018. Using active learning to expand training data for implicit discourse relation recognition. In EMNLP, pages 725– 731. find our card
  55. Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2017. Incorporating loose-structured knowledge into conversation modeling via recall-gate lstm. In IJCNN, pages 3506–3513. IEEE. find
  56. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi PrasadO Christopher Bryant, and Attapol T Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. CoNLL, page 1. find our card
  57. Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in lstms for improving machine reading. In ACL, volume 1, pages 1436–1446. find
  58. Wenlin Yao and Ruihong Huang. 2018. Temporal event knowledge acquisition via identifying narratives. In ACL, volume 1, pages 537–547. find
  59. Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In IJCAI, pages 4623–4629. find

↑ back to top