Elamur Research⌕
☰
home / papers / 2019 / Mining Discourse Markers for Unsupervised Sentence Representation Learning

Mining Discourse Markers for Unsupervised Sentence Representation Learning

Summary

Supervised sentence encoders depend on costly annotated corpora such as SNLI. Earlier attempts to use discourse markers as free supervision covered only a few markers or coarse categories and drew on the small BookCorpus. This paper presents a way to mine sentence pairs linked by explicit markers from web-scale text. It uses a Rutherford and Xue-style rule: the second sentence begins with a single-word adverbial or conjunction followed by a comma. The rule is not limited to a fixed inventory, so it surfaces new markers, and PDTB markers are added on top. Applied to the Depcc corpus (Common Crawl), the method produces the Discovery dataset: 174 English markers, each with at least 10K examples, including rare ones like "coincidentally." Encoders are trained in the InferSent framework to predict the marker for a sentence pair. On SentEval transfer and probing tasks they match or beat strong baselines on most tasks. Analysis shows that many markers are predictable from shallow lexical cues alone. Shuffling sentence pairs barely hurts the models, which suggests they do not truly exploit inter-sentence relations.

Key points

  • From 8.5 billion consecutive sentence pairs sampled from Depcc, filtering kept 53%. About 3.77% of those (≈170M pairs) contained a candidate marker. After keeping markers with ≥10K occurrences and capping each at 200K, the dataset has 19M pairs.
  • 243 candidate markers were found, giving a curated set of 174. The rule recovered 31 of the 42 single-word PDTB markers that precede a comma.
  • A Fasttext classifier was used to measure how predictable each marker is from shallow features. Removing these "easy" examples (DiscoveryHard) did not improve transfer results.
  • On the SentEval linguistic probing tasks, DiscoveryBig averages 75.5. This compares with 70.8 for InferSent, 72.7 for SkipThought and 71.4 for QuickThought.
  • The dataset and code are released at github.com/synapse-developpement/Discovery. The authors argue that the method could be adapted to other languages with similar discourse usage, such as French or German.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 36 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1351 · arXiv 1903.11850

@inproceedings{sileo2019mining,
  title = {{Mining Discourse Markers for Unsupervised Sentence Representation Learning}},
  author = {Sileo, Damien and Cruys, Tim Van De and Pradel, Camille and Muller, Philippe},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1351},
  eprint = {1903.11850},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1351}
}

References

41 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Mirella Lapata. 2008. Modeling Local Coherence: An Entity-Based Approach. Computational Linguistics, 34(1):1–34. find our card
  2. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python, volume 43. find
  3. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), (3):1–6. find our card
  4. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  5. Siddhartha Brahma. 2018. Unsupervised learning of sentence representations using sequence consistency. CoRR, abs/1808.04217. — [arXiv:1808.04217](https://arxiv.org/abs/1808.04217) find
  6. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. find
  7. Chloé Braud and Pascal Denis. 2016. Learning connective-based word representations for implicit discourse relation identification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 203–213, Austin, Texas. Association for Computational Linguistics. find our card
  8. Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. find
  9. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the SIGDIAL 2001 Workshop, The 2nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saturday, September 1, 2001 to… find our card
  10. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, pages 670–680. Association for Computational Linguistics. find our card
  11. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find our card
  12. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating Compositionality in Sentence Embeddings. (2011). find
  13. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. Cite arxiv:1607.01759. find
  14. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  15. Omer Levy, Samuel R Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. Proceedings of NAACL-HLT 2018, pages 107–112. find
  16. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. pages 1–16. find our card
  17. Lajanugen Logeswaran, Honglak Lee, and Dragomir Radev. 2016. Sentence Ordering using Recurrent Neural Networks. pages 1–15. find
  18. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579–2605. find
  19. Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2017. Automatic Prediction of Discourse Connectives. find our card
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  21. Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2018. Automatic Prediction of Discourse Connectives. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find our card
  22. Bjarke Felbo, Alan Mislove, Anders Sogaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In EMNLP. find our card
  23. Daniel Marcu and Abdessamad Echihabi. 2001. An unsupervised approach to recognizing discourse relations. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, (July):368. find our card
  24. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 368–375. find our card
  25. Allen Nie, Erin D. Bennett, and Noah D. Goodman. 2017. DisSent: Sentence Representation Learning from Explicit Discourse Relations. find
  26. Alexander Panchenko, Eugen Ruppert, Stefano Far-alli, Simone P Ponzetto, and Chris Biemann. 2017. Building a Web-Scale Dependency-Parsed Corpus from Common Crawl. pages 1816–1823. find
  27. Christian S. Perone, Roberto Silveira, and Thomas S. Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. CoRR, abs/1806.06259. — [arXiv:1806.06259](https://arxiv.org/abs/1806.06259) find
  28. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088. find our card
  29. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In ACL 2009, Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 2-7 Au… find our card
  30. Caroline Sporleder and Alex Lascarides. 2005. Exploiting linguistic cues to classify rhetorical relations. In Proceedings of Recent Advances in Natural Langauge Processing (RANLP), Bulgaria. find
  31. Caroline Sporleder and Alex Lascarides. 2008. Using automatically labelled examples to classify rhetorical relations: an assessment. Natural Language Engineering, 14(3):369–416. find
  32. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. International Conference on Learning Representations. find
  33. Nianwen Xue, Vera Demberg, and Attapol Rutherford. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume… find our card
  34. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Coling 2010: Posters, pages 1507–1514. find our card
  35. Coling 2010 Organizing Committee. find
  36. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90. find our card
  37. Coling 2008 Organizing Committee. find
  38. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. Proceedings of the 7th Joint Conference on Lexical and Computational Semantics, (1):180–191. find
  39. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  40. Alec Radford. 2018. Improving language understanding by generative pre-training. find
  41. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver… find our card

↑ back to top

This page in Markdown →