Elamur Research⌕
☰
home / papers / 2021 / Where Are We in Discourse Relation Recognition?

Where Are We in Discourse Relation Recognition?

Summary

This position paper asks why discourse relation parsers, despite their theoretical appeal, rarely improve downstream NLP tasks. The authors suspect two causes: parsers are weak, and they are applied to text that differs from their WSJ training data. They test this with gradual domain shift. First, they compare implicit relations across PDTB-2, PDTB-3 and the intra-sentential subset of PDTB-3, using the BERT-based DiscoEval classifier (Chen et al., 2019). Second, they run PDTB and RST parsers on unannotated text: WSJ articles outside the Penn Treebank, other news, and the GUM corpus. Linguists annotated these outputs by hand. Errors appear even on in-domain WSJ text. Accuracy did not differ notably between in-domain and out-of-domain samples, but the types of errors did. Inter- and intra-sentential implicit relations behave differently. Many errors stem from ignoring context beyond the two arguments, and RST parsers suffer from attachment errors that cascade through the tree. The authors call for context-aware, uncertainty-calibrated parsers and release their annotated data.

Key points

  • Over more than a decade, English implicit relation recognition improved by less than 8 F1 points: Lin et al. (2009) reported 40.2, while Lee et al. (2020) reached only 47.8.
  • Accuracy of the BERT-based DiscoEval classifier (base / large):
    • PDTB-2: 0.4236 / 0.4358
    • PDTB-3: 0.4897 / 0.5094
    • PDTB-3 intra-sentential: 0.6251 / 0.6251
  • Plain-text PDTB parsing combined the Wang and Lan (2015) end-to-end parser with the DiscoEval implicit sense classifier. Among the evaluation texts, only GUM has gold RST labels, and none have gold PDTB labels.
  • RST parsers frequently mislabel relations as Same-Unit or Elaboration and make attachment mistakes. The paper also criticizes the standard practice of collapsing relations into 18 top-level classes, which turns Elaboration into a very large class.
  • The annotated dataset is available at github.com/katherine-atwell/discourse-domain-shift.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

20 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.sigdial-1.34

@inproceedings{atwell2021where,
  title = {{Where Are We in Discourse Relation Recognition?}},
  author = {Atwell, Katherine and Li, Junyi Jessy and Alikhani, Malihe},
  booktitle = {Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2021},
  doi = {10.18653/v1/2021.sigdial-1.34},
  url = {https://doi.org/10.18653/v1/2021.sigdial-1.34}
}

References

58 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press. find
  2. David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2138–2144. find
  3. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212–2218, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  4. Chloé Braud, Maximin Coavoux, and Anders Søgaard. 2017. Cross-lingual RST discourse parsing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 292–304. ↗ our card
  5. Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. ISI Technical Report ISI-TR-545, 54:56. find
  6. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. ↗ our card
  7. Muthu Kumar Chandrasekaran, Carrie Epp, Min-Yen Kan, and Diane Litman. 2017. Using discourse signals for robust instructor intervention prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. ↗
  8. Eugene Charniak. 1973. Jack and Janet in search of a theory of knowledge. In IJCAI, pages 337–343. find
  9. Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 649–662. ↗
  10. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  11. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  12. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). find our card
  13. Laurence Danlos, Diégo Antolinos-Basso, Chloé Braud, and Charlotte Roze. 2012. Vers le fdtb: French discourse tree bank. In TALN 2012: 19ème conférence sur le Traitement Automatique des Langues Naturelles, volume 2, pages 471–478. ATALA/AFCP. find
  14. Mark Davies. 2012. Expanding horizons in historical linguistics with the 400-million word Corpus of Historical American English. Corpora, 7(2):121–157. ↗
  15. Aravind Joshi Bonnie Webber Eleni Miltsakaki, Rashmi Prasad. 2004. The Penn Discourse Treebank. Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). find our card
  16. Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511–521. find our card
  17. Hugo Hernault, Helmut Prendinger, Mitsuru Ishizuka, et al. 2010. HILDA: A discourse parser using support vector machine classification. Dialogue & Discourse, 1(3). ↗ our card
  18. Jerry R. Hobbs. 1985. On the coherence and structure of discourse. Technical report, Center for the Study of Language and Information, Stanford University. find
  19. Yuta Kikuchi, Tsutomu Hirao, Hiroya Takamura, Manabu Okumura, and Masaaki Nagata. 2014. Single document summarization based on nested tree structure. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 315–320. find
  20. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414. ↗ our card
  21. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1152– 1158. find our card
  22. Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down RST parsing utilizing granularity levels in documents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8099– 8106. find
  23. Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split or merge: Which is better for unsupervised RST parsing? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5801–5806. find
  24. Murathan Kurfalı and Robert Östling. 2019. Zero-shot transfer for implicit discourse relation classification. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 226–231. find our card
  25. Haejun Lee, Drew A Hudson, Kangwook Lee, and Christopher D Manning. 2020. SLM: Learning a discourse language representation with sentence un-shuffling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1551–1562. find
  26. Li Liang, Zheng Zhao, and Bonnie Webber. 2020. Extending implicit discourse relation recognition to the PDTB-3. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 135–147. find our card
  27. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and M Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4190– 4200. find our card
  28. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 343–351. find our card
  29. Linlin Liu, Xiang Lin, Shafiq Joty, Simeng Han, and Lidong Bing. 2019. Hierarchical pointer net parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1006– 1016. Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2284–2295. find
  30. William C. Mann and Sandra A. Thompson. 1987. Rhetorical Structure Theory: a theory of text organization. Technical Report RS-87-190, USC/Information Sciences Institute. Reprint series. find
  31. Daniel Marcu. 1999. Discourse trees are good indicators of importance in text. Advances in automatic text summarization, 293:123–136. find
  32. Daniel Marcu. 2000. The Theory and Practice of Discourse Parsing and Summarization. MIT press. ↗
  33. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. ↗ our card
  34. Thomas Meyer, Andrei Popescu-Belis, Sandrine Zufferey, and Bruno Cartoni. 2011. Multilingual annotation and disambiguation of discourse connectives for machine translation. In Association for Computational Linguistics-Proceedings of 12th SIGdial Meeting on Discourse and Dialogue, CONF. find our card
  35. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. find our card
  36. Karthik Narasimhan and Regina Barzilay. 2015. Machine comprehension with discourse relations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1253– 1262. ↗ our card
  37. Noriki Nishida and Hideki Nakayama. 2020. Unsupervised discourse constituency parsing using Viterbi EM. Transactions of the Association for Computational Linguistics, 8:215–230. ↗
  38. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 186–195. find
  39. Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, and Aravind Joshi. 2007. The Penn Discourse Treebank 2.0 annotation manual. find our card
  40. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. find
  41. Rashmi Prasad, Bonnie Webber, and Alan Lee. 2018. Discourse annotation in the PDTB: The next generation. In Proceedings 14th Joint ACL - ISO Workshop on Interoperable Semantic Annotation, pages 87–97, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  42. Amandla Mabona, Laura Rimell, Stephen Clark, and Andreas Vlachos. 2019. Neural generative rhetorical structure parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Alan Lee Eleni Miltsakaki Livio Robaldo Aravind Joshi Bonnie Webber Rashmi Prasad, Nikhil Dinesh. 2008. The Penn Discourse Treebank 2.0. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). ↗ our card
  43. RC Schank and RP Abelson. 1977. Plans, goals aid understanding: An inquiry into human knowledge structures. find
  44. Wei Shi and Vera Demberg. 2019. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5794–5800. ↗ our card
  45. Maite Taboada. 2016. Sentiment analysis: overview from linguistics. find
  46. An Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou. 2020. A top-down neural architecture towards text-level parsing of discourse rhetorical structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6386–6395. find our card
  47. Yingxue Zhang, Fandong Meng, Peng Li, Ping Jian, and Jie Zhou. 2021. Context tracking network: Graph-based context modeling for implicit discourse relation recognition. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1592–1599. find our card
  48. Jianxiang Wang and Man Lan. 2015. A refined end-to-end discourse parser. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning-Shared Task, pages 17–24. find our card
  49. Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9709–9716. ↗
  50. Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184–188. ↗ our card
  51. Yuping Zhou, Jill Lu, Jennifer Zhang, and Nianwen Xue. 2014. Chinese Discourse Treebank 0.5 ldc2014t21. Web Download. Philadelphia: Linguistic Data Consortium. find
  52. Florian Wolf and Edward Gibson. 2005. Representing discourse coherence: A corpus-based study. Computational linguistics, 31(2):249–287. ↗ our card
  53. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive text summarization. arXiv preprint arXiv:1910.14142. ↗ our card
  54. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1–16, Beijing, China. Association for Computational Linguistics. find our card
  55. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. ConLL 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19. find our card
  56. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  57. Deniz Zeyrek, Amália Mendes, Yulia Grishina, Murathan Kurfalı, Samuel Gibbon, and Maciej Ogrodniczuk. 2020. TED Multilingual Discourse Bank (TED-MDB): A parallel corpus annotated in the PDTB style. Language Resources and Evaluation, 54:587–613. ↗
  58. Deniz Zeyrek and Bonnie Webber. 2008. A discourse resource for Turkish: Annotating discourse connectives in the METU corpus. In Proceedings of the 6th workshop on Asian language resources. find our card

↑ back to top

This page in Markdown →