Comparing Word Representations for Implicit Discourse Relation Classification
Summary
Implicit discourse relations lack an explicit connective such as "because" and account for roughly half of the relations in the Penn Discourse Treebank. They are hard to classify, and earlier systems relied on parsers, lexicons and heavy feature engineering. Braud and Denis ask how far cheap lexical information can go if words are represented well. They build a controlled comparison along three dimensions:
- Word representations: sparse one-hot vectors, Brown clusters, and three dense embeddings (Collobert and Weston, hierarchical log-bilinear, and Hellinger PCA).
- Words used: only the head words of the two segments, or all their tokens.
- Combination: concatenation, or a Kronecker (outer) product of the vectors.
Separate binary classifiers are trained for the top-level PDTB relations. Denser representations consistently beat sparser ones and reach or exceed state-of-the-art scores without hand-crafted features. Adding traditional features yields only small further gains, which suggests the embeddings already encode most of the useful syntactic and semantic information. Because the representations can be learned from raw text alone, the approach transfers readily to languages with few resources.
Key points
- Representations compared: one-hot word pairs, Brown clusters, Collobert and Weston, HLBL and H-PCA embeddings, all publicly available for English.
- Brown clusters questioned: a reproduction of Rutherford and Xue (2014) performs about the same as raw one-hot word pairs with extra features. Their reported gains seem to come from a tuned filtering threshold and coreference features, not from the clusters themselves.
- Best system with added features: it sets new top F1 scores for Temporal (29.30, previously 28.69) and Contingency (55.76, previously 54.42) on the PDTB.
- Gaps that remain: the best system stays below earlier work on Comparison (36.36 vs 39.70) and Expansion. Ji and Eisenstein (2014) report 79.91 on Expansion.
- Head words: the authors find head words distinctly relevant. This contradicts Park and Cardie (2012), who concluded that word-pair features add little value.
Cited by
- Deep Enhanced Representation for Implicit Discourse Relation Recognition
- Cross-lingual RST Discourse Parsing
- Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network
- A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing
- Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning
- TransS-Driven Joint Learning Architecture for Implicit Discourse Relation Recognition
- Implicit Discourse Relation Classification: We Need to Talk about Evaluation
- Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing
- Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification
- Discourse Relation Sense Classification Using Cross-argument Semantic Similarity Based on Word Embeddings
- Employing the Correspondence of Relations and Connectives to Identify Implicit Discourse Relations via Label Embeddings
- Implicit Discourse Relation Recognition with Context-aware Character-enhanced Embeddings
- Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification
- A Systematic Study of Neural Discourse Models for Implicit Discourse Relation
- On the Need of Cross Validation for Discourse Relation Classification
- Two End-to-end Shallow Discourse Parsers for English and Chinese in CoNLL-2016 Shared Task
Cite
@inproceedings{braud2015comparing,
title = {{Comparing Word Representations for Implicit Discourse Relation Classification}},
author = {Braud, Chloé and Denis, Pascal},
booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
year = {2015},
doi = {10.18653/v1/d15-1262},
url = {https://doi.org/10.18653/v1/d15-1262}
}References
- N. Asher and A. Lascarides. 2003. Logics of Conversation. Cambridge University Press. find
- Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
- Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find our card
- Chloé Braud and Pascal Denis. 2014. Combining natural and artificial examples to improve implicit discourse relation identification. In Proceedings of the 25th International Conference on Computational Linguistics. find our card
- Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18:467–479. find
- Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning. find
- Yangfeng Ji and Jacob Eisenstein. 2014. One vector is not enough: Entity-augmented distributional semantics for discourse relations. Transactions of the Association for Computational Linguistics. find
- Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find our card
- Igor Labutov and Hod Lipson. 2013. Re-embedding words. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics. find
- Rémi Lebret and Ronan Collobert. 2014. Word emdeddings through hellinger PCA. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. find
- Junyi Jessy Li and Ani Nenkova. 2014a. Addressing class imbalance for improved recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
- Junyi Jessy Li and Ani Nenkova. 2014b. Reducing sparsity improves the recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find our card
- Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. find our card
- Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. find our card
- Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th International Conference on Machine Learning. find
- Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find our card
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
- Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find our card
- Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find our card
- Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation. find our card
- Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. find our card
- Philip J. Stone and Earl B. Hunt. 1963. A computer approach to content analysis: Studies using the general inquirer system. In Proceedings of the Spring Joint Computer Conference. find
- Karl Stratos and Michael Collins. 2005. Simple semi-supervised pos tagging. In Proceedings of NAACL Workshop on Vector Space Modeling for NLP. find
- Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. find our card
- Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. find