Elamur Research⌕
☰
home / papers / 2020 / Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing

Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing

Summary

Explicit discourse relations are marked by connectives such as "however" or "while." These words are hard to handle for two reasons: they can also appear in a purely sentential role, and they can signal different senses. Knaebel and Stede address the first problem, binary connective disambiguation, without hand-crafted syntactic features. Their model is a small multilayer perceptron with one hidden layer. Its input is the averaged embeddings of a connective-head candidate and its neighbouring words. They compare non-contextual vectors (GloVe and Wikipedia-trained fastText) with contextual BERT representations and vary the size of the context window. They also train a joint variant that predicts explicit senses as an auxiliary task. Because this variant needs neither argument extraction nor earlier pipeline outputs, it avoids error propagation. On PDTB2, under the CoNLL 2016 Shared Task setup, the BERT-based model beats the strongest shared-task systems at connective disambiguation. The joint model trails them on sense classification but gives more stable results. The work shows that a feature-free design is competitive and suggests embeddings may help transfer discourse parsing to languages without PDTB-scale corpora.

Key points

  • Data: PDTB2 contains about 43,000 relations, of which roughly 18,000 are explicit. Candidates are normalized to connective heads following the CoNLL convention. Non-connective (NoConn) candidates make up about 70% of the training data.
  • Connective disambiguation (F1): BERT with a one-word context window (bert-ctx-1) scores 97.32 on the test set and 96.98 on the blind set. The best CoNLL 2016 systems reached 94.74 (Soochow, test) and 91.79 (OPT, blind).
  • Non-contextual embeddings: fastText scores around 92 on test and 89–90 on blind, which is roughly on par with older feature-based systems. This supports the authors' hypothesis that contextual embeddings work better.
  • Joint fine-sense classification (F1): The joint model scores 86.26 on test and 71.69 on blind with BERT. The best competitors, Ecnuc and OPT, scored 90.13 on test and about 77 on blind.
  • Evaluation and auxiliary task: The authors introduce a more sensitive evaluation measure. Using it, they show that training sense classification jointly improves the stability of connective disambiguation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.codi-1.7

@inproceedings{knaebel2020contextualized,
  title = {{Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing}},
  author = {Knaebel, René and Stede, Manfred},
  booktitle = {Proceedings of the First Workshop on Computational Approaches to Discourse},
  year = {2020},
  doi = {10.18653/v1/2020.codi-1.7},
  url = {https://doi.org/10.18653/v1/2020.codi-1.7}
}

References

37 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571– 583, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  2. Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299–1308, Cop… find our card
  3. Jonathan Baxter. 2000. A model of inductive bias learning. Journal of artificial intelligence research, 12:149–198. find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  5. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20:151–184. find
  6. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2201–2211, Lisbon, Portugal. Association for Computational Linguistics. find our card
  7. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496, Florence, Italy. Association for Computational Linguistics… find our card
  8. Richard A. Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Machine Learning Proceedings 1993, pages 41 – 48. Morgan Kaufmann, San Francisco (CA). find
  9. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Assoc… find
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12(null):2493–2537. find our card
  11. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 2750–2756. AAAI Press. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  13. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. find
  14. Ziwei Fan, Zhenghua Li, and Min Zhang. 2016. Finding arguments as sequence labeling in discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 150–157. Association for Computational Linguistics. find
  15. Stephan Oepen, Jonathon Read, Tatjana Scheffler, Uladzimir Sidarenka, Manfred Stede, Erik Velldal, Evgeny Stepanov and Giuseppe Riccardi. 2016a. Unitn end-to-end discourse parser for conll 2016 shared task. In Proceedings of the CoNLL-16 shared task, pages 85–91. Association for Computational Lingui… find our card
  16. Evgeny Stepanov and Giuseppe Riccardi. 2016b. UniTN end-to-end discourse parser for CoNLL 2016 shared task. In Proceedings of the CoNLL-16 shared task, pages 85–91, Berlin, Germany. Association for Computational Linguistics. find our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  18. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of syntax. Computational Linguistics, 38(2):369–410. find
  19. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  20. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 17–24, Beijing, China. Association for Computational Linguistics. find our card
  21. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16, Suntec, Singapore. Association for Computational Linguistics. find our card
  22. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for english and chinese in conll-2016 shared task. In Proceedings of the CoNLL-16 shared task, pages 33–40. Association for Computational Linguistics. find our card
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In In Proceedings of LREC. find our card
  24. Longyue Wang, Chris Hokamp, Tsuyoshi Okita, Xiaojun Zhang, and Qun Liu. 2015. The dcu discourse parser for connective, argument identification and explicit sense classification. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 89–94. Associa… find our card
  25. Rashmi Prasad, Bonnie Webber, and Alan Lee. 2018. Discourse annotation in the PDTB: The next generation. In Proceedings 14th Joint ACL - ISO Workshop on Interoperable Semantic Annotation, pages 87–97, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  26. Bonnie Webber, Rashmi Prasad, and Alan Lee. 2019. Ambiguity in explicit discourse connectives. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 134– 141, Gothenburg, Sweden. Association for Computational Linguistics. find our card
  27. L. Prechelt. 1998. Automatic early stopping using cross validation: quantifying the criteria. Neural networks : the official journal of the International Neural Network Society, 11 4:761–767. find
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  29. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow discourse parsing using convolutional neural network. In Proceedings of the CoNLL-16 shared task, pages 70–77, Berlin, Germany. Association for Computational Linguistics. find our card
  30. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006–1017, Vanco… find our card
  31. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  32. Karin Sim Smith. 2017. On integrating discourse in machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 110–121, Copenhagen, Denmark. Association for Computational Linguistics. find
  33. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. CoRR, abs/1804.07036. — [arXiv:1804.07036](https://arxiv.org/abs/1804.07036) find
  34. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16. Association for Comput… find our card
  35. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  36. Conll 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19. Association for Computational Linguistics. find our card
  37. Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. In AAAI. find

↑ back to top

This page in Markdown →