Elamur Research⌕
☰
home / papers / 2019 / DisSent: Learning Sentence Representations from Explicit Discourse Relations

DisSent: Learning Sentence Representations from Explicit Discourse Relations

Summary

Good general-purpose sentence embeddings usually require either huge unlabeled corpora (as with BERT) or expensive hand-labeled sentence-pair data such as the entailment sets behind InferSent. Nie, Bennett and Goodman propose a middle path. Writers routinely signal how two ideas relate with connectives like because, but or so, and the authors use these naturally occurring discourse markers as free supervision. They apply the Stanford CoreNLP dependency parser and marker-specific dependency patterns to BookCorpus. This lets them automatically pull out pairs of sentence-like clauses in their correct conceptual order, even when the marker's surface position varies. A model then has to guess the connecting marker from the two clauses alone. Training a BiLSTM encoder on this DisSent task yields fixed embeddings that roughly match InferSent on SentEval and beat it on some tasks. Using the task as an intermediate fine-tuning step for BERT-base gives state-of-the-art results on implicit relation classification in the Penn Discourse Treebank (PDTB). The approach is cheap, needs no manual annotation, and could in principle carry over to other languages through universal dependencies.

Key points

  • Corpus: The dataset is built from BookCorpus and contains about 4.7M clause pairs (4,706,292) covering 15 markers. Markers were chosen if they occur in more than 1% of PDTB cases. but and and each account for roughly 22% of pairs. Experiments use three marker subsets: ALL, 5 and 8.
  • Data efficiency: DisSent Books 5 uses about 3.2M examples and 63.5M tokens. This is far less text than SkipThought (about 800M tokens) or BERT pre-training (about 3300M tokens).
  • Encoder architecture: The encoder is a bidirectional LSTM with global max pooling over time. The two sentence vectors are combined with their average, difference and element-wise product, then passed to a softmax classifier over markers.
  • BERT fine-tuning: BERT-base (12-layer Transformer) is trained on DisSent using the [CLS] representation, then fine-tuned further on PDTB tasks.
  • Extraction quality: Pairs were aligned against PDTB gold annotations to check quality. Precision differs by marker, being higher for because and so and lower for and and still. When an aligned match is found, the extracted pair is generally close to the gold pair.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

46 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 31 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1442

@inproceedings{nie2019dissent,
  title = {{DisSent: Learning Sentence Representations from Explicit Discourse Relations}},
  author = {Nie, Allen and Bennett, Erin and Goodman, Noah},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1442},
  url = {https://doi.org/10.18653/v1/p19-1442}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. arXiv:1705.00557 [cs, stat]. ArXiv: 1705.00557. — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) find
  2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. International Conference on Learning Representations. find
  3. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find
  5. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  6. Alistair Knott. 1996. A data-driven methodology for motivating a set of coherence relations. find
  7. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. find our card
  8. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 343–351. Association for Computational Linguistics. find our card
  9. Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2018. Automatic prediction of discourse connectives. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association. find our card
  10. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  11. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 914–923. find our card
  12. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  13. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  14. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, Adversarial connective-and Eric Xing. 2017. exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006– 1017, Vanc… find
  15. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  16. Prasad Rashmi, Dinesh Nihkil, Lee Alan, Miltsakaki Eleni, Robaldo Livio, Joshi Aravind, Webber Bonnie, et al. 2008. The penn discourse treebank 2.0. In Lexical Resources and Evaluation Conference. find our card
  17. Sebastian Schuster and Christopher D Manning. 2016. Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In Lexical Resources and Evaluation Conference. find our card
  18. Caroline Sporleder and Alex Lascarides. 2008. Using automatically labelled examples to classify rhetorical relations: An assessment. Natural Language Engineering, 14(3):369–416. find
  19. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations. find
  20. Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003. Anaphora and discourse structure. Computational linguistics, 29(4):545–587. find our card
  21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  22. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  23. Frances Yung, Kevin Duh, Taku Komura, and Yuji Matsumoto. 2017. A psycholinguistic model for the marking of discourse relations. Dialogue & Discourse, 8(1):106–131. find
  24. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card

↑ back to top