Elamur Research⌕
☰
home / papers / 2023 / A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

Paper · 2023 · Bruce W. Lee, BongSeok Yang and Jason Hyung-Jong Lee

Summary

The authors address a gap in implicit discourse relation classification, which means labeling the coherence relation between two text spans when no connective such as "but" or "because" signals it. No one had yet compared many pre-trained transformers on this task, so researchers had little guidance on which public model to pick. The authors fine-tune seven large Huggingface models: RoBERTa, ALBERT, BigBird-RoBERTa, BART, Longformer, SpanBERT and DeBERTa. They train on PDTB-3 and evaluate with the cross-validation protocol of Kim et al. (2020). The best system reaches 0.671 mean accuracy on 14 Level-2 senses, a new state of the art for this protocol.

The comparison also yields several findings:

  • Sentence-level pre-training objectives (NSP, SBO, SOP) did not give the top models.
  • Adapting RoBERTa for long documents with sparse attention made results worse.
  • The simple pairing of MLM with full attention worked best.
  • Feeding whole source sentences instead of the annotated argument spans lowered accuracy.

The study suggests that a careful search over existing models can move a benchmark as much as new architectures do.

Key points

  • Data and protocol: PDTB-3 is annotated over the roughly 1M-word Wall Street Journal Corpus. Its 25 sections are split into 12 folds by a sliding window of two sections, each fold holding 21 training, 2 development and 2 test sections. The label set is the 14 Level-2 senses that have at least 100 instances.
  • Training setup: AdamW with a linear schedule and no warm-up, batch size 8, max input length 256. Training runs up to 10 epochs and stops early once validation loss stops falling. A learning rate of 5e-6 worked best for almost all models, compared with the common default of 2e-5.
  • Multiply-annotated relations: each label becomes its own training instance. At test time, predicting any one of the gold labels counts as correct.
  • Results: the best mean test accuracy is 0.671, reported with its variance across folds. Earlier milestones were about 60% for XLNet large and below 50% for pre-transformer methods. ALBERT was an exception to the pattern that sentence-level objectives hurt.
  • Full-sentence input: the authors built an argument matcher to recover the source sentence of each span. Full-sentence input generally scored lower on the test set. However, existing argument extractors reach only about 50–60% accuracy, which limits the gold-span advantage in end-to-end use.

Interesting ideas

  • I1. Cross-validating over whole corpus sections with a sliding window keeps documents intact, eases sparse labels, and lets results be reported as a mean with variance instead of one lucky split.
  • I2. When a gold item carries several valid labels, each label is trained as a separate instance, and at test time a match with any of them counts as correct.
  • I3. Long-document attention variants lost to plain full attention because the task depends on dense cues packed into a few tokens. Attention design should therefore follow how local the needed evidence is.
  • I4. The natural guess that more context (whole sentences) would help was tested and failed. The authors still noted that span-level gains depend on how accurate the upstream span extractor is.
  • I5. Before designing a new architecture, sweep several off-the-shelf pretrained models under one fixed protocol and tune mainly the learning rate. This alone set a new state of the art.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Our evaluation runs could rotate held-out treebank or fable sections the same way, so that a small gain is judged against fold-to-fold variance.
  • I2 (frames): Ambiguous speech acts or discourse senses in prag could be scored this way, so the system is not penalized for picking one legitimate reading.
  • I3 (tree): For sentence-level dependency parsing in gparse, full attention is the safer default, and sparse or windowed attention should only be adopted after a direct comparison.
  • I4 (frames): When linking clauses into frames, we should measure the cost of using predicted argument spans versus gold spans, so that pipeline error compounding is visible rather than assumed away.
  • I5 (learn): Witness parsers or LLM teacher tiers could be chosen by such a uniform side-by-side sweep rather than by reputation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDiscourse analysis · 466 works

Cite

Cite as Bruce W. Lee, BongSeok Yang and Jason Hyung-Jong Lee. 2023. A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification. arXiv:2307.03378.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.03378 · arXiv 2307.03378

@misc{lee2023side,
  title = {{A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification}},
  author = {Bruce W. Lee, BongSeok Yang and Jason Hyung-Jong Lee},
  year = {2023},
  doi = {10.48550/arxiv.2307.03378},
  eprint = {2307.03378},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.03378}
}

References

45 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571–583. ↗
  2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. ↗
  3. Peter Bourgonje and Manfred Stede. 2019. Explicit discourse argument extraction for german. In International Conference on Text, Speech, and Dialogue, pages 32–44. Springer. find
  4. Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. arXiv preprint arXiv:1804.05918. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. James Fiacco, Shiyan Jiang, David Adamson, and Carolyn Rosé. 2022. Toward automatic discourse parsing of student writing motivated by neural interpretation. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 204–215, Seattle, Washington. Association for Computational Linguistics. find
  7. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020a. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. ↗
  8. Ruifang He, Jian Wang, Fengyu Guo, and Yugui Han. 2020b. TransS-driven joint learning architecture for implicit discourse relation recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 139–148, Online. Association for Computational Linguistics.
  9. Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020. Pretraining with contrastive sentence objectives improves discourse performance of language models. arXiv preprint arXiv:2005.10389. ↗
  10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. ↗
  11. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542. ↗
  12. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414, Online. Association for Computational Linguistics. ↗
  13. Yudai Kishimoto, Yugo Murawaki, and Sadao Kurohashi. 2020. Adapting bert to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1152– 1158.
  14. René Knaebel. 2021. discopy: A neural system for shallow discourse parsing. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 128–133, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. ↗
  15. Murathan Kurfalı. 2022. Contributions to Shallow Discourse Parsing: To English and beyond. Ph.D. thesis, Department of Linguistics, Stockholm University. find
  16. Murathan Kurfalı and Robert Östling. 2021a. Let’s be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction. arXiv preprint arXiv:2106.03192. ↗
  17. Murathan Kurfalı and Robert Östling. 2021b. Probing multilingual language models for discourse. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 8–19. find
  18. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. ↗
  19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.
  21. Jiaqi Li, Ming Liu, Bing Qin, and Ting Liu. 2022. A survey of discourse parsing. Frontiers of Computer Science, 16(5):1–12. ↗
  22. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  24. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  25. Kashif Munir, Hongxiao Bai, Hai Zhao, and Junhan Zhao. 2021. Memorizing all for implicit discourse relation recognition. Transactions on Asian and Low-Resource Language Information Processing, 21(3):1– 20. ↗
  26. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  27. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16, Suntec, Singapore. Association for Computational Linguistics.
  28. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). ↗
  29. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006–1017, Vancouver, Canada. Association for Computational Linguistics. ↗
  30. Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 281–291. ↗
  31. Wei Shi and Vera Demberg. 2017. On the need of cross validation for discourse relation classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 150–156, Valencia, Spain. Association for Computational Linguistics. ↗
  32. Wei Shi and Vera Demberg. 2019a. Learning to explicitate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden. Association for Computational Linguistics. ↗
  33. Wei Shi and Vera Demberg. 2019b. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5790–5796. ↗
  34. Damien Sileo, Tim Van-de Cruys, Camille Pradel, and Philippe Muller. 2019. Discourse-based evaluation of language understanding. arXiv preprint arXiv:1907.08672. ↗
  35. Wassiliki Siskou, Clara Giralt Mirón, Sarah Molina-Raith, and Miriam Butt. 2022. Automatized detection and annotation for calls to action in Latin-American social media postings. In Proceedings of the 6th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 65–69, Gyeongju, Republic of Korea. International Conference on Computational Linguistics. find
  36. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China national conference on Chinese computational linguistics, pages 194–206. Springer. ↗
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465–4476, Florence, Italy. Association for Computational Linguistics. ↗
  39. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania. find
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  41. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The conll-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning-Shared Task, pages 1–16.
  42. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. Conll 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1–19.
  43. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  44. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297. ↗
  45. Zheng Zhao and Bonnie Webber. 2022. Revisiting shallow discourse parsing in the pdtb-3: Handling intra-sentential implicits. arXiv preprint arXiv:2204.00350. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2023-side-side-comparison-transformers-english/