Elamur Research⌕
☰
home / papers / 2023 / A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis

A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis

Summary

Research on discourse coherence usually treats its dimensions separately: scoring a whole text, tracking how topics develop, or labelling relations between text segments. Few resources combine these dimensions, and Chinese data is especially scarce. The authors address this with CEDCC (Chinese Essay Discourse Coherence Corpus), a set of 501 Chinese secondary-school essays drawn from exams and everyday practice. Linguistic experts annotated each essay on three layers:

  • an overall coherence grade, together with the specific problems that lowered it,
  • topic sentences for each paragraph and for the essay as a whole,
  • discourse relations between sentences and between paragraphs.

The paper reports baselines with pre-trained encoders and ChatGPT. In the experiments, XLNet recognises discourse relations best in the most coherent essays, while implicit relations remain hard at every coherence level. Adding the fine-grained annotations to ChatGPT prompts raises its coherence-grading accuracy. The resource matters because it makes coherence scores interpretable and supports Chinese automated essay scoring. It also offers real student writing rather than artificially shuffled sentences.

Key points

  • Corpus composition: CEDCC has 501 essays covering eight genres, including narrative, argument, letter and prose. Essay length ranges from 603 to 1,600 tokens, averaging about 713. Teacher-assigned scores are included.

  • Coherence grading: grades run from 0 (poor) to 2. Annotators flag four kinds of errors: sections that stray from the topic, wrongly used connectives, unsuitable clauses, and logic that does not hold together.

  • Relation recognition baselines: BERT, RoBERTa, XLNet and ChatGPT variants were tested. No single model was best across all settings.

  • Fine-grained annotations in ChatGPT grading:

    SettingAccuracyRecallF1
    Zero-shot55%–31.15%
    3-shot54%–41.49%
    With fine-grained annotations60%53.4%28.60%

    Accuracy and recall improve with the annotations, but F1 is lower than in the other two settings.

  • Availability and limitations: data and code are released at github.com/cubenlp/CEDCC_corpus. The authors note three limitations: the modest corpus size, subjectivity in manual annotation, and the restriction to student essays.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.412

@inproceedings{wu2023multi,
  title = {{A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis}},
  author = {Wu, Hongyi and Shen, Xinshu and Lan, Man and Mao, Shaoguang and Bai, Xiaopeng and Wu, Yuanbin},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.412},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.412}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hesam Amoualian, Wei Lu, Eric Gaussier, Georgios Ba-likas, Massih R. Amini, and Marianne Clausel. 2017. Topical coherence in LDA-based models through induced segmentation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1799–… find
  2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 141–148, Ann Arbor, Michigan. Association for Computational Linguistics. find
  3. Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… find
  4. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Youmna Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HL… find
  7. Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ire… find
  8. Linea Flansmose Mikkelsen, Oliver Kinch, Anders Jess Pedersen, and Ophélie Lacroix. 2022. DDisCo: A discourse coherence dataset for Danish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2440–2445, Marseille, France. European Language Resources Association. find
  9. Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. 2021. Long text generation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… find
  10. Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 93–103, Sofia, Bulgaria. Association for Computational Linguistics. find
  11. Andrew T. Jebb, Vincent Ng, and Louis Tay. 2021. A review of key likert scale development advances: 1995–2019. Frontiers in Psychology, 12. find
  12. Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What substitutes tell us - analysis of an “all-words” lexical substitution corpus. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 540–549, Gothenburg, Sweden. A… find
  13. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223, Melbourne, Australia. Association for Computational Linguistics. find our card
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  15. Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198–209, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  16. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 9… find our card
  17. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019… find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  19. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  20. WILLIAM C. MANN and SANDRA A. THOMPSON. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243– 281. find our card
  21. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004, Barcelona, Spa… find
  22. Elham Mohammadi, Timothe Beiko, and Leila Kosseim. 2020. On the creation of a corpus for coherence evaluation of discursive units. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1067–1072, Marseille, France. European Language Resources Association. find
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). find our card
  24. Association for Computational Linguistics, 9:621– 640. find
  25. Disha Shrivastava, Abhijit Mishra, and Karthik Sankaranarayanan. 2018. Modeling topical coherence in discourse without supervision. CoRR, abs/1809.00410. — [arXiv:1809.00410](https://arxiv.org/abs/1809.00410) find
  26. Hongyi Wu, Hao Zhou, Man Lan, Yuanbin Wu, and Yadong Zhang. 2023. Connective prediction for implicit discourse relation recognition via knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5908–5923, Toronto… find our card
  27. Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2019. Modeling coherence for discourse neural machine translation. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Sy… find
  28. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Ne… find our card
  29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe-view.net. find
  30. Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. arXiv preprint arXiv:2210.07032. find our card
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. find
  32. Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie L. Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long… find our card
  33. Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin, and Jianzhong Qi. 2021. Evaluating document coherence modeling. Transactions of the find

↑ back to top

This page in Markdown →