Elamur Research⌕
☰
home / papers / 2021 / CVAE-based Re-anchoring for Implicit Discourse Relation Classification

CVAE-based Re-anchoring for Implicit Discourse Relation Classification

Summary

Implicit discourse relation classification asks a model to label how two arguments relate when no connective signals it. Annotated data for this task is scarce. The authors place a VAE between a RoBERTa encoder and an MLP classifier. The VAE resamples latent variants of the argument-pair representation in every epoch, which should act as data augmentation. In practice, this setup performed worse than plain RoBERTa+MLP. The authors trace the drop to erroneous sampling: the VAE's sampling region can cross class boundaries, so some variants receive the wrong relation label.

To fix this, they use a Conditional VAE during training. It shifts the anchor of the sampling region toward a relation-type embedding, and an added risk term penalises variants that land nearer to other relations. Because gold labels are unavailable at test time, RoBERTa is trained to absorb this re-anchoring, and inference uses an ordinary VAE.

On PDTB v2.0, re-anchoring beats the RoBERTa baseline on every main relation class. Combined with transfer learning and interactive attention, the model achieves state-of-the-art 4-way results. This shows that augmentation from generative models must account for label-preserving sampling.

Key points

  • Data split: PDTB v2.0, with sections 02–20 for training, 00–01 for development and 21–22 for testing. Performance is reported as binary F1 for Comparison, Contingency, Expansion and Temporal, plus 4-way F1 and accuracy.
  • Plain VAE hurts: adding a VAE lowered binary F1 on all four classes. For example, Comparison fell from 53.71 to 48.22 and Expansion from 75.90 to 70.07.
  • Re-anchoring helps: CVAE re-anchoring raised the scores to 56.60 (COM), 62.60 (CON), 77.74 (EXP) and 37.13 (TEM). This roughly matches transfer learning, which relies on 18,459 external explicit argument pairs.
  • Risk term: the risk loss combines mean-square divergences to the true and to competing relation embeddings, weighted by α = β = 0.5. It is added to the cross-entropy loss and back-propagated into RoBERTa.
  • Full model results: the full model (re-anchoring + transfer learning + interactive attention) reaches 65.06 4-way F1 and 70.17 accuracy. This exceeds Liu et al. (2020), who reported 63.39 and 69.06.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.110

@inproceedings{dou2021cvae,
  title = {{CVAE-based Re-anchoring for Implicit Discourse Relation Classification}},
  author = {Dou, Zujun and Hong, Yu and Sun, Yu and Zhou, Guodong},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.110},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.110}
}

References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We verify the effectiveness of re-anchoring by mea- Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recog-suring the percentage of salvaging the mistakenly-nition. In COLING, pages 571–583. determined semantically-similar argument pairs. Given two groups… find
  2. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling There are 5,165 groups of semantically-similar inter-dependencies of discourse units in a paragraph. argument pairs found in the test set, each of which In NAACL-HLT, pages 141–151. hold different type… find
  3. IJCNN 2005. 2,220 for 4-way classification, occupying 43%. By Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Long-contrast, the CVAE-based re-anchoring salvaged biao Wang, and Xiangang Li. 2018. Implicit dis-1,962 and 1,601 cases for binary and 4-way classi-course relation recognition using neural ten… find
  4. Ruifang He, Jian Wang, Fengyu Guo, and Yugui Han. 2020. Transs-driven joint learning architecture for implicit discourse relation recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 139–148. find our card
  5. Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. find
  6. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In EMNLP, pages 1299–1308. find our card
  7. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In AAAI, pages 4848–4855. find
  8. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3830–3836. International J… find
  9. Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In EMNLP, pages 1224–1233. find our card
  10. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In AAAI, pages 2750– 2756. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  12. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th annual meeting of the association for computational linguistics, pages 368–375. find our card
  13. Linh The Nguyen, Ngo Van Linh, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In ACL, pages 4201–4207. find our card
  14. Allen Nie, Erin Bennett, and Noah Goodman. 2019. Dissent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510. find our card
  15. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. find our card
  16. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. A stacking gated neural architecture for implicit discourse relation classification. In EMNLP, pages 2263–2270. find our card
  17. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric P. Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -Augus… find our card
  18. Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, and Min Zhang. 2020. Interactively-propagative attention learning for implicit discourse relation recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3168–3178. find our card
  19. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7… find our card
  20. Wei Shi and Vera Demberg. 2019. Learning to explic-itate connectives with seq2seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics, IWCS 2019, Long Papers, Gothenburg, Sweden, May 23-27 May, 2019, pages 188–199. As… find
  21. Wei Shi, Frances Yung, and Vera Demberg. 2018. Acquiring annotated data with cross-lingual explicitation for implicit discourse relation classification. CoRR, abs/1808.10290. find our card
  22. Wei Shi, Frances Yung, Raphael Rubino, and Vera Demberg. 2017. Using explicit discourse connectives in translation for implicit discourse relation classification. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 484–495, Taipei… find our card
  23. Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. find
  24. Siddharth Varia, Christopher Hidey, and Tuhin Chakrabarty. 2019. Discourse relation prediction: Revisiting word pairs with convolutional networks. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 442–452, Stockholm, Sweden. Association for Computational Linguistics. find our card
  25. Changxing Wu, Xiaodong Shi, Yidong Chen, Yanzhou Huang, and Jinsong Su. 2016. Bilingually-constrained synthetic data for implicit discourse relation recognition. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2306–2312, Austin, Texas. Association for… find our card
  26. Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, and Guodong Zhou. 2018. Using active learning to expand training data for implicit discourse relation recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 725–731, Brussels, Belgium. Assoc… find our card
  27. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In EMNLP, pages 2230–2235. find our card
  28. Ye Zhang and Byron Wallace. 2016. A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. find
  29. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In COLING, pages 1507–1514. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dou-2021-cvae-based-re-anchoring-implicit/