Elamur Research⌕
☰
home / papers / 2016 / Discourse Relation Sense Classification Using Cross-argument Semantic Similarity Based on Word Embeddings

Discourse Relation Sense Classification Using Cross-argument Semantic Similarity Based on Word Embeddings

Summary

The paper describes a system built for the supplementary track of the 2016 CoNLL shared task, which focused on shallow parsing of discourse. The track asks systems to assign one of 15 hierarchically organized senses to an English discourse relation when its two arguments are already known, along with the connective if the relation is Explicit. The authors build separate models for Explicit and Non-Explicit relations. At test time, they route each item by checking whether a connective is present. The core model is a Logistic Regression classifier whose features come only from pre-trained word2vec Google News vectors. It uses averaged (centroid) embeddings of Arg1, Arg2 and the connective, plus cosine-based comparisons between the arguments: whole-argument similarity, top-N and word-alignment similarities, POS-restricted centroids, and similarity to 103 connectives from the Penn Discourse Treebank manual. For Non-Explicit senses, they also test two CNNs: Kim's sentence classifier and a modified ARC-1 matching network called ARC-1M. The lightweight LR system placed first overall. This suggests that dense embeddings combined with simple cross-argument similarity can compete with more complex designs. The code is public on GitHub.

Key points

  • Official overall F-scores were 64.13 (Dev), 63.31 (Test) and 54.69 (Blind). The system ranked 1st overall, 2nd on Explicit (0.07 behind the leader) and 4th on Non-Explicit (2.75 behind). The Explicit F-score on Dev was 0.912.
  • Non-Explicit results on Dev:
    • Embeddings-only LR scored 35.54. Embeddings-only CNN scored 34.34 and ARC-1M scored 36.21.
    • Adding similarity features raised LR to 40.32 (the submitted system).
    • Adding connective similarities brought it to 40.99.
    • Adding modal-verb (MD) POS similarities brought it to 41.66.
  • The post-deadline additions also improved Non-Explicit on Blind by 0.36, to 34.92.
  • Data sizes were as follows, with evaluation run on the TIRA platform:
    • Train: 15,500 Explicit and 18,115 Non-Explicit relations.
    • Dev: 740 Explicit and 782 Non-Explicit relations.
    • Test: 990 Explicit and 1,026 Non-Explicit relations.
    • Blind: 608 Explicit and 661 Non-Explicit relations.
  • The neural models showed complementary strengths:
    • ARC-1M was clearly best on Contingency.Cause.Reason and Contingency.Cause.Result.
    • Kim's CNN did better on Contrast, EntRel, Conjunction and Instantiation but never predicted Expansion.Restatement.
    • Rare senses such as Concession and Synchrony scored 0 for all models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-2014

@inproceedings{mihaylov2016discourse,
  title = {{Discourse Relation Sense Classification Using Cross-argument Semantic Similarity Based on Word Embeddings}},
  author = {Mihaylov, Todor and Frank, Anette},
  booktitle = {Proceedings of the CoNLL-16 shared task},
  year = {2016},
  doi = {10.18653/v1/k16-2014},
  url = {https://doi.org/10.18653/v1/k16-2014}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 32–36, Beijing, China, July. Association for Computational Linguistics. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey… find
  2. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. Quality estimation from scratch (quetch): Deep learning for word-level translation quality estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisbon, Portugal, September. Association for Co… find
  3. Sobha Lalitha Devi, Sindhuja Gopalan, Lakshmi S, Pattabhi RK Rao, Vijay Sundar Ram, and Malarkodi C.S. 2015. A hybrid discourse relation parser in conll 2015. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 50–55, Beijing, China, July. Asso… find
  4. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2201–2211, Lisbon, Portugal, September. Association for Computational Linguistics. find our card
  5. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland, June. Association for Computational Linguistics. find
  6. Christian Chiarcos and Niko Schenk. 2015. A minimalist approach to shallow discourse parsing and implicit relation recognition. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 42–49, Beijing, China, July. Association for Computational Lingu… find
  7. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. CoRR, abs/1603.02776. find
  8. Ana Marasović and Anette Frank. 2016. Multilingual Modal Sense Classification using a Convolutional Neural Network. In Proceedings of the 1st Workshop on Representation Learning for NLP, Berlin, Germany, August. find
  9. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. J. Mach. Learn. Res., 9:1871–1874, June. find
  10. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 368–375, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. find our card
  11. Francisco Guzmán, Lluı́s Màrquez, and Preslav Nakov. 2016a. Machine translation evaluation meets community question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL ’16, Berlin, Germany. find
  12. Francisco Guzmán, Lluı́s Màrquez, and Preslav Nakov. 2016b. MTE-NN at SemEval-2016 Task 3: Can machine translation evaluation help community question answering? In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, USA. find
  13. Todor Mihaylov and Preslav Nakov. 2016. SemanticZ at SemEval-2016 Task 3: Ranking relevant answers in community question answering using semantic similarity based on fine-tuned word embeddings. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, Californ… find
  14. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2015. Convolutional neural network architectures for matching natural language sentences. CoRR, abs/1503.03244. — [arXiv:1503.03244](https://arxiv.org/abs/1503.03244) find
  15. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746–751, Atl… find
  16. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746– 1751, Doha, Qatar, October. Association for Computational Linguistics. find our card
  17. Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alha Freihat, Jim Glass, and Bilal Randeree. 2016. SemEval-2016 task 3: Community question answering. In Fang Kong, Sheng Li, and Guodong Zhou. 2015. The sonlp-dp system in the conll-2015 shared task. In Beijing,… find
  18. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, USA. find
  19. Quan Hung Tran, Vu Tran, Tu Vu, Minh Nguyen, and Son Bao Pham. 2015. JAIST: Combining multiple features for answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’15, pages 215–219, Denver, Colorado, USA. find
  20. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  21. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 17–24, Beijing, China, July. Association for Computational Linguistics. find our card
  22. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90, Manchester, UK, August. find our card
  23. Coling 2008 Organizing Committee. find
  24. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16, Beijing, China, July.… find our card
  25. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the Reproducibility of PAN’s Shared Tasks: Plagiarism Detection, Author Identification, and Author Profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sander-son, Mark Hall… find
  26. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Bonnie Webber, Attapol Rutherford, Chuan Wang, and Hongmin Wang. 2016. The conll-2016 shared task on multilingual shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task, Berlin, Germany,… find our card
  27. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In In Proceedings of LREC. find our card
  28. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Lisbon, Portugal, September.… find our card
  29. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 645–654, Gothenburg, Sweden, Ap… find our card
  30. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7… find our card
  31. Caroline Sporleder and Alex Lascarides. 2008. Using automatically labelled examples to classify rhetorical relations: An assessment. Nat. Lang. Eng., 14(3):369–416, July. find
  32. Evgeny Stepanov, Giuseppe Riccardi, and Ali Orkan Bayer. 2015. The unitn discourse parser in conll 2015 shared task: Token-level sequence labeling with argument-specific models. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 25–31, find our card

↑ back to top

This page in Markdown →