Elamur Research⌕
☰
home / papers / 2024 / DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting

DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting

Paper · 2024 · Xuanming Zhang, Anthony Diaz, Zixun Chen et al.

Summary

Second-language (L2) English learners often struggle with discourse coherence. Existing automated writing evaluation tools mostly assign a coherence score based on shallow linguistic features and do not help learners fix the problem. The authors present DECOR, a benchmark built from 100 medium-level TOEFL-11 essays. The essays were first cleaned with a grammar-correction model so that surface errors would not hide the intended meaning. Following the CoheSentia protocol, each essay was split into incremental context-sentence pairs, where every target sentence is judged against all text before it. Two expert annotators then labelled each pair for three tasks: whether the sentence is incoherent, why (using a reason taxonomy derived from Reinhart's cohesion, consistency and relevance conditions), and how to rewrite it. The authors fine-tuned smaller open models, including Llama2-7B and Llama3-8B-Instruct, on GPT-4-synthesized task data. Detection quality was comparable to zero- and few-shot GPT-4. Conditioning rewrites on the incoherence reason gave better results in both automatic and human evaluation. This matters because it offers an affordable path to feedback that explains and repairs learner writing.

Key points

  • DECOR statistics:
    • 100 essays and 26,376 words.
    • 1,352 context-sentence pairs: 906 coherent and 446 incoherent.
    • 213 expert rewrites, which the authors describe as the first parallel incoherent/rewritten corpus for L2 writing.
  • Reason labels:
    • R1 covers meaning links between sentences, R2 how entities are referred to, and R3 relations between discourse units (missing marker). R4 is consistency, R5 contextual relevance, R6 tangential relevance and R7 others.
    • These are grouped into four high-level types; Consistency is the rarest.
  • Training data and baseline: models were fine-tuned on GPT-4-generated synthetic data, and GPT-4 also served as the rewrite baseline. The authors note that GPT-4 tends toward invasive, unnecessary edits.
  • Human evaluation:
    • Human references beat GPT-4 rewrites with a win rate of up to 74%.
    • Llama 2 trained with reasons outperformed Llama 2 trained without them (chi-square, p < 0.01).
  • Stated limitations:
    • The reason types are unbalanced.
    • The data covers only TOEFL-11 prompts.
    • Context is left-only, although some coherence breaks depend on later sentences.

Interesting ideas

  • I1. Each sentence is judged only against the text before it, which turns document coherence into a series of local, inspectable decisions
  • I2. Detection is split into named failure reasons (unresolved entity, missing discourse relation, contradiction, irrelevance) instead of a single score, so every verdict comes with an explanation
  • I3. Telling the rewriter why a sentence is incoherent yields better repairs than asking it to rewrite blindly
  • I4. Over-editing is treated as a failure: a good rewrite keeps the learner's text and changes only what is needed, as the human rewrites did compared with GPT-4
  • I5. Systems are compared pairwise against one fixed baseline (win/loss rates), with significance tested and judging done by someone other than the annotators

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): the world builder could flag the exact sentence where a new entity, contradiction or irrelevant state first fails to attach to the accumulated text world.
  • I2 (frames): coreference and frame linking could emit these same reason codes when a pronoun or argument cannot be anchored, giving explainable diagnostics.
  • I3 (back): generation from the meaning graph could take the detected break type (e.g. a missing connective, an ambiguous referent) as an explicit input and produce a targeted fix.
  • I4 (back): the text → graph → text compressor could be scored on minimal edit distance as well as meaning preservation.
  • I5 (learn): evaluation of parser or teacher-tier variants could use the same fixed-baseline win/loss design with a significance test to separate real gains from noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓University of California, DavisDavis · United States28 works · 41,916 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Xuanming Zhang, Anthony Diaz, Zixun Chen, Qingyang Wu, Kun Qian, Erik Voss and Zhou Yu. 2024. DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.639

@inproceedings{zhang2024decor,
  title = {{DECOR: Improving Coherence in L2 English Writing with a Novel Benchmark for Incoherence Detection, Reasoning, and Rewriting}},
  author = {Xuanming Zhang, Anthony Diaz, Zixun Chen, Qingyang Wu, Kun Qian, Erik Voss and Zhou Yu},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.639},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.639}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. ↗
  3. John Bitchener and Helen Basturkmen. 2006. Perceptions of the difficulties of postgraduate l2 thesis students writing the discussion section. Journal of English for Academic Purposes, 5(1):4–18. ↗
  4. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. Toefl11: A corpus of non-native english. ETS Research Report Series, 2013(2):i–15. ↗
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. ↗
  6. Hannan Cao, Wenmian Yang, and Hwee Tou Ng. 2023. Mitigating exposure bias in grammatical error correction with data augmentation and reweighting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2123–2135. find
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  8. Linda Cooley and Jo Lewkowicz. 1995. The writing needs of postgraduate students at the university of hong kong: A project report. Hong Kong papers in Linguistics and language teaching, 18:121–123. find
  9. Scott A Crossley, Kristopher Kyle, and Danielle S Mc-Namara. 2016. The tool for the automatic analysis of text cohesion (taaco): Automatic assessment of local, global, and text cohesion. Behavior research methods, 48:1227–1237. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36.
  12. Muhammad Fatihul Firdaus, Joseph Nugraha Wibawa, and Fajri Fathur Rahman. 2023. Utilization of gpt-4 to improve education quality through personalized learning for generation z in indonesia. IT for Society, 8(1). ↗
  13. Melanie C González. 2017. The contribution of lexical diversity to college-level writing. TESOL Journal, 8(4):899–919. ↗
  14. Michael Alexander Kirkwood Halliday and Ruqaiya Hasan. 1976. Cohesion in english. Routledge. find
  15. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗
  16. Mark D Johnson, Anthony Acevedo, and Leonardo Mercado. 2016. Vocabulary knowledge and vocabulary use in second language writing. TESOL Journal, 7(3):700–715. ↗
  17. Ute Knoch. 2007. ‘little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing writing, 12(2):108–128. find
  18. Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. arXiv preprint arXiv:1805.04993. ↗
  19. Liisa Lautamatti. 1978. Observations on the development of the topic in simplified discourse. AFinLAn vuosikirja, pages 71–104. find
  20. Gunter Lorenz. 1999. Learning to cohere: Causal links in native vs. non-native argumentative writing. Pragmatics and Beyond New Series, pages 55–76. ↗
  21. Aviya Maimon and Reut Tsarfaty. 2023. Cohesentia: A novel benchmark of incremental versus holistic assessment of coherence in generated texts. arXiv preprint arXiv:2310.16329. ↗
  22. Danielle S McNamara, Max M Louwerse, Philip M McCarthy, and Arthur C Graesser. 2010. Coh-metrix: Capturing linguistic features of cohesion. Discourse Processes, 47(4):292–330. ↗
  23. Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. Automated evaluation of written discourse coherence using gpt-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403.
  24. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector–grammatical error correction: tag, not rewrite. arXiv preprint arXiv:2005.12592. ↗
  25. OpenAI. 2023. Gpt-4 technical report.
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. ↗
  27. Tanya Reinhart. 1980. Conditions for text coherence. Poetics today, 1(4):161–180. ↗
  28. Melanie Schneider and Ulla Connor. 1990. Analyzing topical structure in esl essays: Not all topics are equal. Studies in second language acquisition, 12(4):411– 427. find
  29. Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error correction. arXiv preprint arXiv:2203.13064. ↗
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  31. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. Lm-critic: Language models for unsupervised grammatical error correction. arXiv preprint arXiv:2109.06822. ↗
  32. Xuanming Zhang, Zixun Chen, and Zhou Yu. 2024. Prolex: A benchmark for language proficiency-oriented lexical substitution. arXiv preprint arXiv:2401.11356. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. ↗
  33. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685. ↗
  34. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2024-decor-improving-coherence-l2-english/