Elamur Research⌕
☰
home / papers / 2019 / Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains

Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains

Summary

Classifying implicit discourse relations is hard because no connective signals the link between two text segments. A model must judge whether the second argument meets or violates what the first one leads a reader to expect, and annotated data is too sparse to learn such expectations directly. Shi and Demberg propose that BERT is well suited to this problem because it is pre-trained with Next Sentence Prediction (NSP), which teaches it what a plausible following sentence looks like. They treat the task as BERT sentence-pair classification, with the two arguments separated by [SEP]. They fine-tune the uncased base model and optionally continue pre-training on unannotated in-domain text: WSJ for PDTB and GENIA for BioDRB. On 11-way PDTB level-2 classification, BERT clearly beats earlier systems. It also transfers well to the biomedical BioDRB corpus. Ablations that drop NSP from the extra pre-training lower performance. This supports the view that modelling the likely next sentence helps the classifier form discourse expectations.

Key points

  • PDTB results: BERT beats the ELMo-based state of the art (Bai and Zhao, 2018) by 7.3 points on PDTB-Lin and 5.5 points on PDTB-Ji. It also beats Shi and Demberg (2019) by about 8 points in 10-fold cross-validation (p<0.01). BERT+WSJ reaches 54.82% accuracy on PDTB-Lin.
  • Benefit of NSP: Leaving NSP out of the additional WSJ pre-training reduces accuracy, for example to 53.39% on PDTB-Lin. The relations that gain most from NSP include Temporal.Synchrony, Comparison.Contrast, Expansion.Conjunction and Expansion.Alternative.
  • BioDRB, multi-class: BioDRB annotations were mapped to PDTB relations. The authors report a gain of roughly 15–16 points over the previous best system. Further pre-training on about 18k unannotated GENIA sentences adds about 3 more points.
  • BioDRB, binary level-1 classification: BERT achieves 43.03% average F1 and 77.34% average accuracy. This exceeds the naïve Bayes and MaxEnt baselines of Xu et al. (2012) by about 16 F1 points and 19 accuracy points.
  • BioBert comparison: BioBert was pre-trained on about 18B tokens of PubMed/PMC text. It has only limited advantage over BERT+GENIA. The authors therefore suggest adding external knowledge, such as ontologies or causal graphs, in future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

40 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 28 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1586

@inproceedings{shi2019next,
  title = {{Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains}},
  author = {Shi, Wei and Demberg, Vera},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1586},
  url = {https://doi.org/10.18653/v1/d19-1586}
}

References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fatemeh Torabi Asr and Vera Demberg. 2015. Uniform surprisal at the level of discourse relations: Negation markers and discourse connective omission. In Proceedings of the 11th International Conference on Computational Semantics, pages 118–128. find
  2. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571–583. find our card
  3. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 69–73, Sofia, Bulgaria. Association for Computational Linguistics. find our card
  4. Deng Cai and Hai Zhao. 2017. Pair-aware neural sentence modeling for implicit discourse relation classification. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 458–466. Springer. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Bhuwan Dhingra, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017. Linguistic knowledge as memory for recurrent neural networks. arXiv preprint arXiv:1703.02620. — [arXiv:1703.02620](https://arxiv.org/abs/1703.02620) find
  7. Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 977–986. find
  8. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association of Computational Linguistics, 3(1):329–344. find our card
  9. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In Proceedings of NAACL-HLT, pages 332–342. find
  10. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Junichi Tsujii. 2003. Genia corpusa semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl 1):i180–i182. find
  11. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2018. A knowledge-augmented neural network model for implicit discourse relation classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 584– 595. find
  12. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746. find our card
  13. Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014. Assessing the discourse factors that influence the quality of machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 283–288. find
  14. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 343–351. Association for Computational Linguistics. find our card
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  16. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 108. find our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  18. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  19. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90. find our card
  20. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  21. Rashmi Prasad, Susan McRoy, Nadya Frid, Aravind Joshi, and Hong Yu. 2011. The biomedical discourse relation bank. BMC bioinformatics, 12(1):188. find
  22. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, pages 1914–1924. find our card
  23. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1006–1… find our card
  24. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 281–291. find our card
  25. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 645–654. Association for Comput… find our card
  26. Shared Task, pages 1–16. Association for Computational Linguistics. find
  27. Nianwen Xue, Hwee Tou Ng, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  28. Conll 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19. Association for Computational Linguistics. find our card
  29. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  30. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1834–1839. find our card
  31. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. find our card
  32. Wei Shi and Vera Demberg. 2017. Do we need cross validation for discourse relation classification? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 150–156. find
  33. Wei Shi and Vera Demberg. 2019. Learning to explicitate connectives with seq2seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics-Long Papers, pages 188–199. find our card
  34. Wei Shi, Frances Yung, and Vera Demberg. 2019. Acquiring annotated data with cross-lingual explicitation for implicit discourse relation classification. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 12–21. find our card
  35. Wei Shi, Frances Yung, Raphael Rubino, and Vera Demberg. 2017. Using explicit discourse connectives in translation for implicit discourse relation classification. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 484–495. find our card
  36. Yu Xu, Man Lan, Yue Lu, Zheng Yu Niu, and Chew Lim Tan. 2012. Connective prediction using machine learning for implicit discourse relation classification. In Neural Networks (IJCNN), The 2012 International Joint Conference on, pages 1–8. IEEE. find
  37. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the CoNLL-15 find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2019-next-sentence-prediction-helps-implicit/