Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation
Paper · 2022 · Noriki Nishida and Yuji Matsumoto
Summary
Discourse dependency parsers trained on one annotated treebank lose much of their accuracy on text from a different domain, and labeled data for new domains is scarce and expensive. Nishida and Matsumoto test whether bootstrapping can narrow this gap without any target-domain annotation. They adapt BERT-based graph-based and transition-based discourse dependency parsers using self-training, co-training, tri-training and asymmetric tri-training. They also systematically compare confidence measures and rules for selecting pseudo-labeled documents. Two settings are examined: monologue transfer between scientific disciplines and transfer across dialogue genres. The authors also release COVID19-DTB, a new manually annotated treebank of biomedical paper abstracts. Bootstrapping gives significant and consistent improvements in every setting. However, reliable pseudo labels concentrate in simpler documents, so complex texts stay hard to parse, and feeding in more unlabeled data does not solve this. Annotating a few carefully chosen documents through active learning helps where bootstrapping stalls. The work offers practical guidance for applying discourse parsing in domains that lack a treebank.
Key points
- New resource: COVID19-DTB, discourse dependency annotations for COVID-19-related biomedical abstracts. The data is available in the biomedical-discourse-treebanks GitHub repository, and the code is in norikinishida/discourse-parsing.
- Other corpora mentioned include RST-DT (only 385 labeled documents, news), SciDTB (NLP abstracts), and STAC and Molweni (multi-party dialogues).
- The model's own predictive probability is a usable filter for noisy pseudo labels because it correlates with label accuracy. The authors recommend the Rank-above-k selection criterion over Rank-diff-k, since its k does not depend on the size of the unlabeled pool.
- Active learning simulation: the 100 least-confident dialogues from the Molweni training set (about 9K dialogues) were manually labeled and used for fine-tuning. This beat randomly chosen samples (averaged over 5 trials), and combining it with co-training outperformed active learning on the source-only model.
- Limitation: all experiments use English only, and extending the study to other languages would require discourse treebanks in those languages.
Institutions
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 2Research subjects and languages the work is about.Discourse analysis · 466 worksDependency parsing · 131 works
Cite
Cite as Noriki Nishida and Yuji Matsumoto. 2022. Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation. Transactions of the Association for Computational Linguistics.
@article{nishida2022out,
title = {{Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation}},
author = {Noriki Nishida and Yuji Matsumoto},
journal = {Transactions of the Association for Computational Linguistics},
year = {2022},
doi = {10.1162/tacl\_a\_00451},
url = {https://doi.org/10.1162/tacl_a_00451}
}References
- Steven Abney. 2002. Bootstrapping. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.3115/1073083 .1073143 — [doi:10.3115/1073083](https://doi.org/10.3115/1073083) ↗
- Stergos Afantenos, Eric Kow, Nicholas Asher, and Jérémy Perret. 2015. Discourse parsing for multi-party chat dialogues. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 928–937. https://doi.org/10 .18653/v1/D15-1109
- Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory (COLT), pages 92–100. https://doi.org/10.1145/279943.279962 ↗
- Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. Technical Report ISI-TR-545. University California Information Sciences Institute. find
- Nicholas Asher, Julie Hunter, Mathieu Morey, Farah Benamara, and Afantenos Stergos. 2016. Discourse structure and dialogue acts in multi-Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Proceedings of t… — [doi:10.3115/1118078](https://doi.org/10.3115/1118078)
- Hugo Hernault, Danushka Bollegala, and Mitsuru Ishizuka. 2010a. A semi-supervised approach to improve classification of infrequent discourse relations using feature vector extension. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 399–409.
- Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. https://doi.org/10.3115/v1/D14-1082 ↗
- Hugo Hernault, Helmut Prendinger, David a. DuVerle, and Mitsuru Ishizuka. 2010b. HILDA: A discourse parser using support vector machine classification. Dialogue & Discourse, 1(3):1–33. https://doi.org/10.5087/dad.2010.003 ↗
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
- Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013. Single-document summarization as a tree knapsack problem. In Proceedings of the 2013 Conference of Empirical Methods in Natural Language Processing (EMNLP), pages 1515–1520. find
- Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the 5th International Conference on Learning Representations (ICLR).
- Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, and Rui Yan. 2019. GSN: A graph-structured network for multi-party dialogues. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 5010–5016. find
- Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL2016), pages 1998–2008. https://doi.org/10 .18653/v1/…
- Zhongqiang Huang and Mary Harper. 2009. Self-training PCFG grammars with latent annotations across languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 832–841. https://doi.org/10 .3115/1699571.1699621 find
- Vanessa Wei Feng and Graema Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 511–521.
- Patrick Huber and Giuseppe Carenini. 2019. Predicting discourse structure using distant supervision from sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint conference on Natural Language Processing (EMNLP-IJCNLP), page…
- Elisa Ferracane, Su Wang, and Raymond J. Mooney. 2017. Leveraging discourse information effectively for authorship attribution. In Proceedings of the The 8th International Joint Conference on Natural Language Processing (IJCNLP), pages 584–593. find
- Ofir Israeli, Adi Beth-Din, Nir Paran, Dana Stein, Shirley Lazar, Shay Weiss, Elad Milrot, Yafit Atiya-Nasagi, Shmuel Yitzhaki, Orly Laskar, and Ofir Schuster. 2020. find
- Evaluating Grigorii Guz and Giuseppe Carenini. 2020. Coreference for discourse parsing: A neural approach. In Proceedings of the First Workshop the efficacy of RT-qPCR SARS-CoV-2 direct approaches in comparison to RNA extraction. BioRxiv preprint 2020.06.10.144196v1. https://doi.org/10.1101/2020.06… ↗
- Peter Jansen, Mihai Surdeanu, and Peter Clark. 2014. Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 977–986. https://doi.org/10 .3115/v1/P14-1092 find
- Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. https://doi.org/10.1162/tacl_a_00101 ↗
- Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 13–24.
- Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down RST parsing utilizing granularity levels in documents. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 8099–8106. https://doi.org/10.1609/aaai.v34i05 .6… — [doi:10.1609/aaai.v34i05](https://doi.org/10.1609/aaai.v34i05) ↗
- Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 996–1005.
- Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2021. Improving neural RST parsing model with silver agreement subtrees. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
- Kailang Jiang, Giuseppe Carenini, and Raymond T. Ng. 2016. Training data enrichment for infrequent discourse relations. In Proceedings of the 26th International Conference on Computational Linguistics (COLING), pages 2603–2614. find
- Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10.1162/tacl_a_00300 ↗
- Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split of merge: Which is better for unsupervised RST parsing? In Proceedings of the 2019 Conference of Empirical Methods in Natural Language Processing (EMNLP), pages 5797–5802. https://doi.… — [doi:10.18653/v1/d19](https://doi.org/10.18653/v1/d19) ↗
- Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385–435. https://doi.org/10.1162/COLI a 00226 ↗
- Shafiq Joty, Giuseppe Carenini, Raymond T. Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 486–496.
- Fajri Koto, Jey Han Lan, and Timothy Baldwin. 2021. Top-down discourse parsing via sequence labeling. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 715–726. https://doi.org/10 .18653/v1/2021.eacl-main.60
- Jungo Kasai, Kun Qian, Sairam Gurajada, Yunyao Li, and Lucian Popa. 2019. find
- Low-Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, and Bing Qin. 2020. Molweni: A challenge multiparty dialogue-based machine reading comprehension dataset with discourse structure. In Proceedings of the 28th International Conference on Computational Linguistics (C… find
- Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. https://doi.org/10.1515/text.1.1988.8.3.243 ↗
- Daniel Marcu. 1999. A decision-based approach to rhetorical parsing. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL), pages 365–372. find
- Jiwei Li, Rumeng Li, and Eduard Hovy. 2014a. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference of Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069.
- David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 152–159.
- Qi Li, Tianshi Li, and Baobao Chang. 2016a. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 Conference of Empirical Methods in Natural Language Processing (EMNLP), pages 362–371. https://doi.org/10 .18653/v1/D16-1035
- Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (EMNLP-HLT), pages 523–530. https://doi.o… ↗
- Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014b. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 25–35.
- Mathieu Morey, Philippe Muller, and Nicholas Asher. 2018. A dependency perspective on RST discourse parsing and evaluation. Computational Linguistics, 44(2):197–235. https://doi.org/10.1162/coli a 00314 ↗
- Zhenghua Li, Min Zhang, Yue Zhang, Zhanyi Liu, Wenliang Chen, Hua Wu, and Heifeng Wang. 2016b. Active learning for dependency parsing with partial annotation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 344–354. find
- Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. Constrained decoding for text-level discourse parsing. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), pages 1883–1900.
- Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Assocication for Computational Linguistics, 6:63–75. https://doi.org/10 .1162/tacl_a_00005
- Noriki Nishida and Hideki Nakayama. 2020. Unsupervised discourse constituency parsing using Viterbi EM. Transactions of the Association for Computational Linguistics, 8:215–230. https://doi.org/10.1162/tacl_a_00312 ↗
- Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In Proceedings of the SIGDIAL 2010 Conference, pages 147–156. find
- Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In Proceedings of the ACL Workshop Incremental Parsing: Bringing Engineering and Cognition Together, pages 50–57. https://doi.org/10.3115/1613148.1613156 ↗
- Ryan Lowe, Nissan Pow, Lulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIDDIAL), pages 285–294.
- Jérémy Perret, Stergos Afantenos, Nicholas Asher, and Mathieu Morey. 2016. Integer linear programming for discourse parsing. In Proceedings of the 2016 Conference of the North American William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Towards a functional theory of text or…
- Zhouxing Shi and Minlie Huang. 2019. A deep sequential model for discourse parsing on multi-party dialogues. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 7007–7014. https://doi.org/10.1609/aaai.v33i01 .33017007 — [doi:10.1609/aaai.v33i01](https://doi.org/10.1609/aaai.v33i01) ↗
- Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation.
- Anders Søaard and Christian Rishøj. 2010. Semi-supervised dependency parsing using generalized tri-training. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1065–1073. find
- Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 1171–1182. https://doi.org/10.18653/v1/E17-1110 ↗
- Mark Steedman, Rebecca Hwa, Stephen Clark, Miles Osborne, Anoop Sarkar, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003a. Example selection for bootstrapping statistical parsers. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computat… find
- Alexander J. Ratner, Cristopher M. De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data programming: Creating large training sets, quickly. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), pages 3574–3582. find
- Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adap-tationa of statistical parsers train on small datasets. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 616–623. find
- Mark Steedman, Miles Osborne, Anoop Sarkar, Stephen Clark, Rebecca Hwa, Julia Hockenmaier, Paula Ruhlen, Steven Baker, and Jeremiah Crim. 2003b. Bootstrapping statistical parsers from small datasets. In Proceedings of the 10th Conference of the European Chapter of the Association for Computational L… find
- Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1044–1054. https://doi.org/10 .18653/v1/P18-1096
- Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), pages 665–673. find
- Kenji Sagae. 2009. Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In Proceedings of the 11th International Workshop on Parsing Technology (IWPT), pages 81–84. https://doi.org/10.3115/1697236.1697253 ↗
- Kuniaki Saito, Toshitaka Ushiku, and Tatsuya Harada. 2017. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of The 34th International Conference on Machine Learning (ICML), pages 2988–2997. find
- Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 735–736. https:/… — [doi:10.1145/1277741.1277883](https://doi.org/10.1145/1277741.1277883) find
- Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648. University of Wisconsin–Madison.
- Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Douglas Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuans… — CORD-19: The COVID-19 Open Research Dataset
- (ACL), pages 444–449. https://doi.org/10.18653/v1/P18-2071 — SciDTB: Discourse Dependency TreeBank for Scientific Abstracts ↗
- David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 189–196. https://doi.org/10.3115/981658.981684 ↗
- Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1834–1839. https://doi.org/10.3115/v1/D14-1196 ↗
- Yizhong Wang, Sujian Li, and Honfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 184–188. https://doi.org/10.18653/v1/P17 -2029 ↗
- Liwen Zhang, Ge Wang, Wenjuan Han, and Kewei Tu. 2021. Adapting unsupervised syntactic parsing methodology for discourse dependency parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… find
- Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/D18-1116 ↗
- David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (A… find
- Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou. 2020. A top-down neural architecture towards text-level parsing of discourse rhetorical structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6386–6395. https://doi.org/1…
- Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5021–5031.
- Yan Zhou and Sally Goldman. 2004. Democratic co-learning. In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence (ICTAI). find
- Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17(11):1529–1541. https://doi.org/10.1109/TKDE.2005.186 — [doi:10.1109/tkde.2005.186](https://doi.org/10.1109/tkde.2005.186) ↗
- An Yang and Sujian Li. 2018. SciDTB: Discourse dependency treebank for scientific abstracts. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/nishida-2022-out-domain-discourse-dependency-parsing/