Elamur Research⌕
☰
home / papers / 2021 / The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification

The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification

Summary

DISRPT 2021 was the second edition of a shared task on the basic building blocks of discourse parsing. It aimed to make results comparable across three theoretical frameworks: RST, SDRT and PDTB. The organizers kept and updated the 2019 tracks on EDU segmentation and connective detection. They also added a new track on discourse relation classification. For that track, relations from every framework were converted into a common format of directed unit pairs (*.rels files), with surrounding sentence context and token offsets. The benchmark contains 16 datasets in 11 languages, about 3 million hand-annotated tokens in total. Segmentation and connective data come in two versions: treebanked CoNLL-U with gold UD-style parses, and plain tokens without sentence splits. All submissions were scored with the same scripts and inputs.

The best systems reached roughly the low 90s on segmentation and connective detection. Relation classification remained much harder, with scores in the low 60s. The benchmark matters because it gives researchers from different frameworks a single, uniform resource for multilingual discourse parsing.

Key points

  • Data composition: 16 datasets, 11 languages, about 3M tokens.
    • 13 datasets are for EDU segmentation (RST and SDRT corpora).
    • 3 PDTB-style datasets are for connective detection: eng.pdtb.pdtb, tur.pdtb.tdb and zho.pdtb.cdtb.
    • All 16 datasets are used for relation classification.
  • Surprise dataset: the Persian RST Corpus (fas.rst.prstc) was released together with the test sets, about one month before the deadline.
  • Licensing: five corpora (e.g., RST-DT, PDTB, CDTB, TDB, and part of GUM) are distributed as annotations only, with scripts to rebuild the text.
  • Scoring: segmentation and connective detection use span-based F-score. Connectives count only on exact span match, with no partial credit.
  • Relation classification results:
    • DisCoDisCo was best on 12 datasets and averaged more than 7 points above DiscRel. It relied on language-specific contextual embeddings, with BERT-style NSP models working well.
    • DiscRel, which uses stacked Random Forests over SBERT features, won on both Chinese datasets, spa.rst.sctb and Persian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

34 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 26 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.disrpt-1.1

@inproceedings{zeldes2021disrpt,
  title = {{The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification}},
  author = {Zeldes, Amir and Liu, Yang Janet and Iruskieta, Mikel and Muller, Philippe and Braud, Chloé and Badene, Sonia},
  booktitle = {Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021)},
  year = {2021},
  doi = {10.18653/v1/2021.disrpt-1.1},
  url = {https://doi.org/10.18653/v1/2021.disrpt-1.1}
}

References

47 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stergos Afantenos, Nicholas Asher, Farah Benamara, Myriam Bras, Cécile Fabre, Mai Ho-dac, Anne Le Draoulec, Philippe Muller, Marie-Paule Péry-Woodley, Laurent Prévot, Josette Rebeyrolles, Ludovic Tanguy, Marianne Vergez-Couret, and Laure Vieu. 2012. An empirical resource for discovering cognitive principles of discourse organisation: the ANNODIS corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2727–2734, Istanbul, Turkey. European Language Resources Association (ELRA). ↗
  2. Iria da Cunha, Juan-Manuel Torres-Moreno, and Gerardo Sierra. 2011. On the development of the RST Spanish treebank. In Proceedings of the 5th Linguistic Annotation Workshop, pages 1–10, Portland, Oregon, USA. Association for Computational Linguistics. find
  3. Vera Demberg, Fatemeh Torabi Asr, and Merel Scholman. 2019. How consistent are our discourse annotations? Insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse, 10(1):87––135. find
  4. Nicholas Asher. 1993. Reference to abstract objects in discourse, volume 50. Springer Science & Business Media. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2721– 2727, Portorož, Slovenia. European Language Resources Association (ELRA). ↗ our card
  7. Tillmann Dönicke. 2021. Delexicalised Multilingual Discourse Segmentation for DISRPT 2021 and Tense, Mood, Voice and Modality Tagging for 11 Languages. In Proceedings of Discourse Relation Parsing and Treebanking 2021 (DISRPT 2021), Punta Cana, Dominican Republic. find
  8. Giuseppe Attardi, Daniele Sartiano, and Maria Simi. 2021. Biaffine dependency and semantic graph parsing for EnhancedUniversal dependencies. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 184–188, Online. Association for Computational Linguistics. find our card
  9. Carla Parra Escartín, Teresa Lynn, J. Moorkens, and Jane Dunne. 2021. Towards transparency in NLP shared tasks. ArXiv, abs/2105.05020. ↗
  10. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304, Valencia, Spain. Association for Computational Linguistics. ↗ our card
  11. Morteza Kamaladdini Ezzabady, Philippe Muller, and Chloé Braud. 2021. Multi-lingual Discourse Segmentation and Connective Identification: MELODI at DISRPT2021. In Proceedings of Discourse Relation Parsing and Treebanking 2021 (DISRPT 2021), Punta Cana, Dominican Republic. find
  12. Shuyuan Cao, Iria da Cunha, and Mikel Iruskieta. 2018. The RST Spanish-Chinese treebank. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 156–166, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  13. Luke Gessler, Shabnam Behzad, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2021. Dis-CoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detection. In Proceedings of Discourse Relation Parsing and Treebanking 2021 (DISRPT 2021), Punta Cana, Dominican Republic. find
  14. Paula Christina Figueira Cardoso, Erick Galani Maziero, Maria Lucía del Rosario Castro Jorge, M. Eloize, R. Kibar Aji Seno, Ariani Di Felippo, Lucia Helena Machado Rino, Maria das Graças Volpe Nunes, and Thiago Alexandre Salgueiro Pardo. 2011. CSTNews - a discourse-annotated corpus for single and multi-document summarization of news texts in Brazilian Portuguese. In Proceedings of the 3rd RST Brazilian Meeting, pages 88–105, Cuiabá, Brazil. find
  15. Luke Gessler, Siyao Peng, Yang Liu, Yilun Zhu, Shabnam Behzad, and Amir Zeldes. 2020. AMALGUM – a free, balanced, multilayer English web corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5267–5275, Marseille, France. European Language Resources Association. ↗ our card
  16. Grigorii Guz and Giuseppe Carenini. 2020. Coreference for discourse parsing: A neural approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 160–167, Online. Association for Computational Linguistics. ↗ our card
  17. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. ↗ our card
  18. Jérémy Perret, Stergos Afantenos, Nicholas Asher, and Mathieu Morey. 2016. Integer linear programming for discourse parsing. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 99–109, San Diego, California. Association for Computational Linguistics. find our card
  19. Mikel Iruskieta, María Jesús Aranzabe, Arantza Diaz de Ilarraza, Itziar Gonzalez-Dios, Mikel Lersundi, and Oier Lopez de Lacalle. 2013. The RST Basque TreeBank: An online search interface to check rhetorical relations. In 4th Workshop on RST and Discourse Studies, pages 40–49, Fortaleza, Brasil. find
  20. Yusuke Kido and Akiko Aizawa. 2016. Discourse relation sense classification with two-step classifiers. In Proceedings of the CoNLL-16 shared task, pages 129–135, Berlin, Germany. Association for Computational Linguistics. find
  21. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find our card
  22. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414, Online. Association for Computational Linguistics. ↗ our card
  23. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. ↗ our card
  24. Murathan Kurfali. 2020. Labeling explicit discourse relations using pre-trained language models. In Text, Speech, and Dialogue. TSD 2020, pages 79–86, Brno, Czech Republic. Springer. find
  25. Gisela Redeker, Ildikó Berzlánovich, Nynke van der Vliet, Gosse Bouma, and Markus Egg. 2012. Multilayer discourse annotation of a Dutch text corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2820–2825, Istanbul, Turkey. European Language Resources Association (ELRA). find
  26. Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas. Association for Computational Linguistics. find our card
  27. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics. find our card
  28. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3830–3836. International Joint Conferences on Artificial Intelligence Organization. Main track. find
  29. Sara Shahmohammadi, Hadi Veisi, and Ali Darzi. 2021. Persian Rhetorical Structure Theory. arXiv preprint arXiv:2106.13833. ↗
  30. Manfred Stede and Arne Neumann. 2014. Potsdam commentary corpus 2.0: Annotation for discourse research. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 925–929, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692. ↗
  32. Svetlana Toldova, Dina Pisarevskaya, Margarita Ananyeva, Maria Kobozeva, Alexander Nasedkin, Sofia Nikiforova, Irina Pavlova, and Alexey Shelepov. 2017. Rhetorical relations markers in Russian RST treebank. In Proceedings of the 6th Workshop on Recent Advances in RST and Related Formalisms, pages 29–33, Santiago de Compostela, Spain. Association for Computational Linguistics. ↗
  33. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
  34. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2018. A dependency perspective on RST discourse parsing and evaluation. Computational Linguistics, 44(2):197–235. ↗ our card
  35. Hanna Varachkina and Franziska Pannach. 2021. A Unified Approach to Discourse Relation Classification in nine Languages. In Proceedings of Discourse Relation Parsing and Treebanking 2021 (DISRPT 2021), Punta Cana, Dominican Republic. find our card
  36. Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510, Florence, Italy. Association for Computational Linguistics. find our card
  37. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 17–24, Beijing, China. Association for Computational Linguistics. find our card
  38. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for English and Chinese in CoNLL-2016 shared task. In Proceedings of the CoNLL-16 shared task, pages 33–40, Berlin, Germany. Association for Computational Linguistics. find our card
  39. Yue Yu, Yilun Zhu, Yang Liu, Yan Liu, Siyao Peng, Mackenzie Gong, and Amir Zeldes. 2019. Gum-Drop at the DISRPT2019 shared task: A model stacking approach to discourse unit segmentation and connective detection. In Proceedings of Discourse Relation Treebanking and Parsing (DISRPT 2019), pages 133–143, Minneapolis, MN. find
  40. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  41. Amir Zeldes, Debopam Das, Erick Galani Maziero, Juliano Antonio, and Mikel Iruskieta. 2019. find
  42. The DIS-RPT 2019 shared task on elementary discourse unit segmentation and connective detection. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 97–104, Minneapolis, MN. Association for Computational Linguistics. find
  43. Deniz Zeyrek and Murathan Kurfalı. 2017. TDB 1.1: Extensions on Turkish discourse bank. In Proceedings of the 11th Linguistic Annotation Workshop, pages 76–81, Valencia, Spain. Association for Computational Linguistics. ↗
  44. Deniz Zeyrek and Bonnie Webber. 2008. A discourse resource for Turkish: Annotating discourse connectives in the METU corpus. In Proceedings of the 6th Workshop on Asian Language Resources. find our card
  45. Liwen Zhang, Ge Wang, Wenjuan Han, and Kewei Tu. 2021a. Adapting unsupervised syntactic parsing methodology for discourse dependency parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5782–5794, Online. Association for Computational Linguistics. find
  46. Longyin Zhang, Fang Kong, and Guodong Zhou. 2021b. Adversarial learning for discourse rhetorical structure parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3946–3957, Online. Association for Computational Linguistics. find our card
  47. Yuping Zhou, Jill Lu, Jennifer Zhang, and Nianwen Xue. 2014. Chinese Discourse Treebank 0.5 LDC2014T21. find

↑ back to top

This page in Markdown →