Elamur Research⌕
☰
home / papers / 2023 / Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling

Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling

Paper · 2023 · Rastislav Hronsky and Emmanuel Keuleers

Summary

The paper asks whether learning sentence embeddings from surrounding sentences, the Skip-Thought idea borrowed from word embeddings, actually beats learning them from the sentence's own content. The authors argue that context matters for words because word form says little about meaning. That reasoning is weaker for full sentences.

They used the Switchboard Dialog Act Corpus (SwDA) of telephone conversations. Several content-based utterance encoders were built: LSA, LDA, averaged GloVe, an LSTM auto-encoder, and averaged random word vectors. Each was then re-projected by small linear layers trained to predict neighbouring utterances. An end-to-end Skip-Thought-like LSTM was also trained. All vectors were scored with a frozen linear probe that predicts dialog act tags.

Context-projected versions showed no consistent gain over their content-only sources. Only the end-to-end Skip-Thought model edged ahead, by about 3 points over the best auto-encoder. Sentence vector dimensionality, by contrast, mattered almost everywhere. The work challenges a widely held but rarely tested assumption in representation learning.

Key points

  • Data and splits: SwDA, split 80/10/10 by utterance, with about 212K training utterances. Long utterances were cut into windows of 20 words with a step of 18.
  • Probe setup: a ridge-regression classifier with 10-fold cross-validation. The 10 most frequent of the 42 dialog acts were kept and the rest merged into one class. Ridge was chosen over logistic regression because the features were highly collinear.
  • Varied factors: vector size D ∈ {6, 12, 25} and context window of 1 or 2 utterances per side. GloVe used the pre-trained glove-twitter-25 vectors, truncated as needed.
  • Accuracy figures: the majority baseline was 35.00%. The best score was 57.54%, from Skip-Thought with D=25 and 2-utterance context. Content-only GloVe at D=25 was nearly as good at 57.29%.
  • Tag entropy: a dialog act tag carries 2.1 bits of entropy unconditioned and 1.86 bits given the previous tag. This confirms that context does hold some signal for the task.

Interesting ideas

  • I1. Before building a context model, compare a label's unconditional entropy with its entropy given the previous label, to measure cheaply how much context could help at all.
  • I2. Keep the content encoder fixed and add only a thin trained projection toward neighbours. Any gain can then be credited to the context objective rather than to a new encoder.
  • I3. Use averaged arbitrarily assigned word vectors as a no-knowledge baseline. At higher dimensions it already clears the majority baseline by a wide margin, which shows how much a probe rewards plain lexical identity.
  • I4. Dimensionality turned out to be a bigger driver than the training signal, so methods should be compared at matched vector sizes.
  • I5. Score frozen representations with a purely linear probe, picking a regularised variant when features are collinear, so accuracy reflects the representation rather than extra learning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Run this check on speech-act labels in dialogues to decide whether crate prag should condition act classification on the preceding act or rely on the utterance alone.
  • I2 (learn): When testing whether discourse context improves a component, freeze the base features and add context as a separate small layer, so the ablation cleanly isolates its contribution.
  • I3 (learn): Add such random-feature baselines as negative controls in evaluations, so scores of real features are judged against what lexical memorisation alone achieves.
  • I4 (learn): When comparing representations or parser feature sets, report results at equal capacity so a size effect is not mistaken for a better method.
  • I5 (tree): Probe gparse's transformer states linearly for UD relations or dialog-related tags to check what is encoded before trusting downstream gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tilburg UniversityTilburg · Netherlands51 works · 6,798 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 654 worksSpeech act theory · 229 works

Cite

Cite as Rastislav Hronsky and Emmanuel Keuleers. 2023. Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.588

@inproceedings{hronsky2023role,
  title = {{Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling}},
  author = {Rastislav Hronsky and Emmanuel Keuleers},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.588},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.588}
}

References

47 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International conference on learning representations. find
  2. Yonatan Belinkov. 2022a. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. ↗
  3. Yonatan Belinkov. 2022b. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. ↗
  4. David M Blei. 2012. Probabilistic topic models. Communications of the ACM, 55(4):77–84. ↗
  5. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. ↗
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. ↗
  7. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, pages 169–174.
  8. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR. ↗
  9. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. arXiv preprint arXiv:1905.06401. ↗
  10. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DiffCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207–4218, Seattle, United States. Association for Computational Linguistics. ↗
  11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. ↗
  12. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070. ↗
  13. Ferdinand De Saussure. 1916. Nature of the linguistic sign. Course in general linguistics, 1:65–70. find
  14. Jacob Devlin, Ming-Wei Chang, and Kenton Lee. 2019. Google, kt, language, ai: Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171– 4186.
  15. William Dolan, Chris Quirk, Chris Brockett, and Bill Dolan. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources.
  16. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  17. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. ↗
  18. Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2). ↗
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  20. Thomas Hofmann. 2001. Unsupervised learning by probabilistic latent semantic analysis. Machine learning, 42(1-2):177. ↗
  21. Dan Jurafsky. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual. Technical report. find
  22. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. arXiv preprint arXiv:1306.3584. ↗
  23. Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 28. ↗
  24. Thomas K Landauer and Susan T Dumais. 1997. A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211. ↗
  25. Thomas K Landauer, Peter W Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse processes, 25(2-3):259–284. ↗
  26. Wei Li and Yunfang Wu. 2019. Multi-level gated recurrent neural network for dialog act classification. arXiv preprint arXiv:1910.01822. ↗
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  28. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893. ↗
  29. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 1–8.
  30. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  32. Minsik Oh, Jiwei Li, and Guoyin Wang. 2023. Tadse: Template-aware dialogue sentence embeddings. find
  33. Richard Socher, Eric Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in neural information processing systems, 24. find
  34. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 151–161.
  35. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  36. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075. ↗
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Sami Virpioja, Peter Smit, Stig-Arne Grönroos, Mikko Kurimo, et al. 2013. Morfessor 2.0: Python implementation and extensions for morfessor baseline. Technical report. find
  39. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  40. Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917–929, Online. Association for Computational Linguistics. ↗
  41. Christian S Perone, Roberto Silveira, and Thomas S Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv preprint arXiv:1806.06259. ↗
  42. Weihao Zeng, Keqing He, Yejie Wang, Chen Zeng, Jin-gang Wang, Yunsen Xian, and Weiran Xu. 2023. Fu-turetod: Teaching future knowledge to pre-trained language model for task-oriented dialogue. find
  43. Jordan B Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46(1-2):77–105. ↗
  44. Zhihan Zhou, Dejiao Zhang, Wei Xiao, Nicholas Ding-wall, Xiaofei Ma, Andrew Arnold, and Bing Xiang. 2022. Learning dialogue representations from consecutive utterances. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 754–768, Seattle, United States. Association for Computational Linguistics. ↗
  45. Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint arXiv:1904.02594. ↗
  46. Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 29–48. Citeseer. find
  47. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. *Figure 1: Correlation matrices of feature vectors across representation variants with dimension of 6.* *Figure 2: Correlation matrices of feature vectors across representation variants with dimension of 12.* *Figure 3: Correlation matrices of feature vectors across representation variants with dimension of 25.* ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hronsky-2023-role-context-unsupervised-sentence-representation/