Elamur Research
☰
home / papers / 2015 / The CoNLL-2015 Shared Task on Shallow Discourse Parsing

The CoNLL-2015 Shared Task on Shallow Discourse Parsing

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/K15-2001.pdf

Summary

The CoNLL-2015 shared task introduced Shallow Discourse Parsing as a community benchmark. Given English news text, a system must find individual discourse relations. This involves locating any explicit connective, marking the two argument spans as Arg1 and Arg2, and assigning a sense label. Relations may be explicit (signalled by words like "because" or "however") or non-explicit (Implicit, AltLex, EntRel, NoRel). The organizers adapted the Penn Discourse TreeBank (PDTB 2.0), built on the WSJ corpus, for training and development. For a blind test, they newly annotated Wikinews articles following PDTB guidelines. Sixteen teams took part. For the first time in CoNLL, systems were deployed on remote virtual machines and run through the TIRA web platform, so participants never saw the test data. Full end-to-end parsing proved very hard: the strongest system reached only 24.0% F1 on the blind set, largely because errors compound across pipeline stages. The released data and scorer provide a lasting benchmark for discourse parsing research.

Key points

  • Data splits: PDTB 2.0 Sections 2–21 were used for training, Section 22 for development, and Section 23 as a standard test set. Official rankings relied on a separate blind test set of 30,158 words drawn from English Wikinews.
  • PDTB 2.0 relation counts: the corpus holds 40,600 relations, broken down as follows:
    • 18,459 Explicit
    • 16,053 Implicit
    • 624 AltLex
    • 5,210 EntRel
    • 254 NoRel
  • Sense labels: senses follow a three-level hierarchy with 4 top classes and 16 second-level types. Attribution and supplementary-span annotations were left out of the task.
  • Evaluation setup: 16 teams from three continents submitted systems through the TIRA platform. This kept the test data hidden and the evaluation reproducible.
  • Results: the best end-to-end score on the blind test was 24.0% F1. Non-Explicit relations were much harder to handle than Explicit ones, and some teams tackled only the Explicit subtask.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

81 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k15-2001

@inproceedings{xue2015conll,
  title = {{The CoNLL-2015 Shared Task on Shallow Discourse Parsing}},
  author = {Xue, Nianwen and Ng, Hwee Tou and Pradhan, Sameer and Prasad, Rashmi and Bryant, Christopher and Rutherford, Attapol},
  booktitle = {Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task},
  year = {2015},
  doi = {10.18653/v1/k15-2001},
  url = {https://doi.org/10.18653/v1/k15-2001}
}

References

43 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
  2. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current Directions in Discourse and Dialogue. find our card
  3. Christian Chiarcos and Niko Schenk. 2015. A minimalist approach to shallow discourse parsing and implicit relation recognition. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  4. Sobha Lalitha Devi, Sindhuja Gopalan, Lakshmi S, Pattabhi RK Rao, Vijay Sundar Ram R., and Malarkodi C.S. 2015. A hybrid discourse relation parser in CoNLL 2015. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  5. David A Duverle and Helmut Prendinger. 2009. A novel discourse parser based on support vector machine classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association of Computational Lingustics and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find
  6. Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find our card
  7. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find our card
  8. Sucheta Ghosh, Giuseppe Riccardi, and Richard Johansson. 2012. Global features for shallow discourse parsing. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  9. Tim Gollub, Benno Stein, and Steven Burrows. 2012. Ousting Ivory Tower Research: Towards a Web Framework for Providing Experiments as a Service. In Bill Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson, editors, 35th International ACM Conference on Research and Development in Information Retrieval (SIGIR 12), pages 1125–1126. ACM, August. find
  10. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find our card
  11. Fang Kong, Sheng Li, and Guodong Zhou. 2015. The SoNLP-DP system in the CoNLL-2015 shared task. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  12. Man Lan, Yu Xu, and Zhengyu Niu. 2013. Leveraging synthetic discourse data via multi-task learning for implicit discourse relation recognition. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
  13. Junyi Jessy Li and Ani Nenkova. 2014. Addressing class imbalance for improved recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  14. Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find
  15. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. find our card
  16. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  17. William C Mann and Sandra A Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
  18. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find our card
  19. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗ our card
  20. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. Annotating discourse connectives and their arguments. In Proceedings of the Human Language Technology/North American Chapter of the Association for Computational Linguistics Workshop on Frontiers in Corpus Annotation. find
  21. Truong Son Nguyen, Bao Quoc Ho, and Le Minh Nguyen. 2015. JAIST: A two-phase machine learning approach for identifying discourse relations in newswire texts. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  22. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗ our card
  23. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  24. Slav Petrov and Dan Klein. 2007. Improved inferencing for unlexicalized parsing. In Proceedings of the Human Language Technology/North American Chapter of the Association for Computational Linguistics. find
  25. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K Joshi. 2008. Easily identifiable discourse relations. Technical report, University of Pennsylvania. find
  26. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. find our card
  27. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the Reproducibility of PAN’s Shared Tasks: Plagiarism Detection, Author Identification, and Author Profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5th International Conference of the CLEF Initiative (CLEF 14), pages 268–299, Berlin Heidelberg New York, September. Springer. find
  28. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computation Natural Language Learining: Shared Task. find our card
  29. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computation Natural Language Learining: Shared Task. find our card
  30. Rashmi Prasad and Harry Bunt. 2015. Semantic relations in discourse: The current state of ISO 24617-8. In Proceedings of the 11th Joint ACL-ISO Workshop on Interoperable Semantic Annotation. find
  31. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation. find our card
  32. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse Treebank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. ↗ our card
  33. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through Brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. find
  34. Yangqiu Song, Haoruo Peng, Parisa Kordjamshidi, Mark Sammons, and Dan Roth. 2015. Improving a pipeline architecture for shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  35. Manfred Stede. 2012. Discourse Processing. Morgan & Claypool Publishers. find
  36. Evgeny Stepanov, Giuseppe Riccardi, and Ali Orkan Bayer. 2015. The UniTN discourse parser in CoNLL 2015 shared task: Token-level sequence labeling with argument-specific models. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  37. Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  38. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  39. Xun Wang, Sujian Li, Jiwei Li, and Wenjie Li. 2012. Implicit discourse relation recognition by selecting typical training examples. In Proceedings of the 24th International Conference on Computational Linguistics. find
  40. Longyue Wang, Chris Hokamp, Tsuyoshi Okita, Xiaojun Zhang, and Qun Liu. 2015. The DCU discourse parser for connective, argument identification and explicit sense classification. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  41. Bonnie Webber, Markus Egg, and Valia Kordoni. 2012. Discourse structure and language technology. Natural Language Engineering, 18(4):437–490. find
  42. Yasuhisa Yoshida, Katsuhiko Hayashi, Tsutomu Hirao, and Masaaki Nagata. 2015. Hybrid approach to PDTB-styled discourse parsing for CoNLL-2015. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning: Shared Task. find
  43. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics. find