Elamur Research⌕
☰
home / papers / 2009 / An Exploratory Application of Rhetorical Structure Theory to Detect Coherence Errors in L2 English Writing: Possible Implications for Automated Writing Evaluation Software

An Exploratory Application of Rhetorical Structure Theory to Detect Coherence Errors in L2 English Writing: Possible Implications for Automated Writing Evaluation Software

Paper · 2009 · Sophia Skoufaki

Summary

Discourse-level mistakes in second-language English essays are hard to spot and rarely catalogued. This pilot study asks whether Rhetorical Structure Theory (RST) can expose such coherence errors in writing by Taiwanese low-intermediate learners whose L1 is Chinese. One researcher built RST diagrams for 45 short texts from the LTTC English Learner Corpus, a learner corpus still under construction. The working assumption was that any spot where a diagram breaks the RST formation rules marks a coherence problem.

The most frequent anomalies were dangling structures, which mostly reflected irrelevant content. Next came misused relations (background, motivation, conclusion) and a single case of crossed dependencies. Misused background relations corresponded to inductive ordering, where the main point arrives late. Several error types clustered on particular essay prompts, which suggests strong topic effects.

The paper compares these findings with what ETS's Criterion software can detect. Criterion's vocabulary-based off-topic checks and its discourse-element tagging would likely miss most of these errors. No reliability test was run, so all results are preliminary.

Key points

  • Data: 45 pieces of writing drawn from the LTTC English Learner Corpus, produced by low-intermediate L1 Chinese learners in Taiwan. The analysis was done by a single annotator, with no inter- or intra-judge reliability check.
  • Method: RST (Mann & Thompson, 1988) diagrams. Rule-violating diagram configurations are treated as indicators of coherence errors.
  • Error types found:
    • Dangling structures were the most frequent. All but two of them signalled irrelevant content.
    • Inappropriate background relations ranked second and signal inductive order. They were concentrated on the "idol" prompt.
    • Unwarranted motivation relations appeared only on the "nearsightedness" prompt.
    • An inappropriate conclusion relation occurred twice.
    • Crossed dependencies occurred once.
  • Criterion tags Introductory Material, Thesis Statement, Main Point, Support and Conclusion, and puts everything else under "Other." Its three off-topic methods all compare vocabulary, either against training essays, against cross-topic word frequencies, or against the prompt.
  • RST was chosen because earlier bottom-up methods correlated poorly with teacher judgments (Watson Todd et al., 2007). Topical structure analysis and genre analysis are examples of those methods.

Interesting ideas

  • I1. If a text's structure cannot be assembled into a well-formed relational graph under explicit rules, the failure itself becomes a diagnostic signal that pinpoints the incoherent segment.
  • I2. Off-topic content can reuse the topic's vocabulary, so lexical-overlap checks miss it, while a structural "nothing attaches here" test catches it.
  • I3. A system should be validated by whether it finds the same kinds of errors humans find, not only by score agreement with human graders.
  • I4. Lumping rare cases into a catch-all "Other" label hides how often the system simply failed to classify a unit, and those failures may be meaningful (e.g., unusual ordering).
  • I5. Error statistics can be driven by the prompt or topic rather than the writer, so results should be broken down by topic before general claims are made.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): When a sentence's entities and events cannot be linked to anything already in the world of the text, it could be flagged as a possible coherence break, with the reason stated explicitly.
  • I2 (answer): In fable QA or reading tests, unconnected world fragments could help reject distractor answers that share words with the story but have no link to its events.
  • I3 (learn): Evaluations could report per-error-type agreement against human judgments, alongside negative controls, so that high aggregate scores cannot hide blind spots.
  • I4 (tree): During UD dialect conversion and parsing, unmappable relations could be counted separately rather than collapsed into a generic label, keeping conversion failures visible.
  • I5 (learn): Silver-data quality and parser error rates could be reported per genre or source, to separate domain artefacts from real model weaknesses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan UniversityTaipei · Taiwan29 works · 2,116 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksLiterature and Literary Theory · 519 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksDiscourse analysis · 466 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Sophia Skoufaki. 2009. An Exploratory Application of Rhetorical Structure Theory to Detect Coherence Errors in L2 English Writing: Possible Implications for Automated Writing Evaluation Software. International Journal of Computational Linguistics & Chinese Language Processing, Volume 14, Number 2, June 2009-Special Issue on Computer Assisted Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{skoufaki2009exploratory,
  title = {{An Exploratory Application of Rhetorical Structure Theory to Detect Coherence Errors in L2 English Writing: Possible Implications for Automated Writing Evaluation Software}},
  author = {Sophia Skoufaki},
  journal = {International Journal of Computational Linguistics \& Chinese Language Processing, Volume 14, Number 2, June 2009-Special Issue on Computer Assisted Language Learning},
  year = {2009},
  url = {https://aclanthology.org/O09-4003/}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Azar, M. (1999). Argumentative Text as Rhetorical Structure: An Application of Rhetorical Structure Theory. Argumentation, 13(1), 97-114. ↗
  2. Burstein, J. (2009). Opportunities for Natural Language Processing Research in Education. In Gebulkh, A. (Ed.), Springer lecture notes in computer science (Vol. 5449, pp. 6-27). Springer: New York, NY. find
  3. Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The Criterion Online Writing Evaluation service. AI Magazine, 25(3), 27-36. find
  4. Burstein, J., Marcu, D., & Knight, K. (2003). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. In Harabagiu, S. & Ciravegna, F. (Eds.), IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing, 18(1), 32-39. ↗
  5. Carlson, L. & Marcu, D. (2001). Discourse Tagging Manual. ISI Tech Report ISI-TR-545. find
  6. Chen, J.P. (2001). Markedness in intercultural discourse: a study of Chinese EFL students’ discourse patterns. PhD thesis, Guangdong University of Foreign Studies, in ERIC, RIE June 2001. find
  7. Chen, J.P. (2008). An investigation into the preference for discourse patterns in the Chinese EFL learning context. International Journal of Applied Linguistics, 18(2), 188-211. find
  8. Díaz-Negrillo, A. & Fernández-Domínguez, J. (2006). Error tagging systems for learner corpora. Revista Española de Lingüística Aplicada, 19, 83-102. find
  9. González, V., Chen, C-Y., & Sanchez, C. (2001). Cultural Thinking and Discourse Organizational Patterns Influencing Writing Skills in a Chinese English-as-a-Foreign-Language (EFL) Learner. Bilingual Research Journal, 25(4), 417-442. ↗
  10. Herrington, A., & Moran, C. (2001). What happens when machines read our students’ writing? College English, 63(4), 480-499. ↗
  11. Higgins, D., Burstein, J., Marcu, D., & Gentile, C. (2004). Evaluating multiple aspects of coherence in student essays. In Dumais, S., Marcu, D. & Roukos, S. (Eds.), Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT-NAACL) 2004: Main Proceedings (pp. 185-192). Boston, MA: Association for Computational Linguistics. find
  12. Kaplan, R. (1966). Cultural thought patterns in intercultural education. Language Learning, 16(1), 1-20. find
  13. Mann, W.C. & Thompson, S.A. (1987). Rhetorical Structure Theory: A Theory of Text Organization (No. ISI/RS-87-190). Marina del Rey, CA: Information Sciences Institute. find
  14. Matalene, C. (1985). Contrastive rhetoric: an American writing teacher in China. College English, 47(8), 789-808. ↗
  15. Mohan, B.A. & Lo, W.A.-Y. (1985). Academic writing and Chinese students: transfer and developmental factors. TESOL Quarterly, 19(3), 515-34. ↗
  16. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2002). Stamping e-rater: Challenging the validity of automated essay scoring. Computers in Human Behavior, 18(2), 103-134. find
  17. Scollon, R. & Wong-Scollon, S. (1995). Intercultural communication: A discourse approach. Blackwell: Oxford, U.K. and Cambridge, MA, U.S.A. find
  18. Taboada, M. & Mann, W.C. (2006a). Rhetorical Structure Theory: looking back and moving ahead. Discourse Studies, 8(3), 423-459. ↗
  19. Taboada, M. and Mann, W.C. (2006b). Applications of Rhetorical Structure Theory. Discourse Studies, 8(4), 567-588. ↗
  20. Vantage Learning. (2007). MY Access! Efficacy Report. Newtown, PA: Vantage Learning. Retrieved on August 6 2009 from http://www.vantagelearning.com/docs/myaccess/myaccess.research.efficacy.report.2007 09.pdf. ↗
  21. Ware, P. (2005). Automated writing evaluation as a pedagogical tool for writing assessment. In A. Pandian, G. Chakravarthy, P. Kell, & S. Kaur (Eds.), Strategies and practices for improving learning literacy (pp. 174-184). Selangor, Malaysia: Universiti Putra Malaysia Press. find
  22. Watson Todd, R., Khongput, S., & Drasawang, P. (2007). Coherence, cohesion and comments on students’ academic essays. Assessing Writing, 12(1), 10-25. in L2 English Writing: Possible Implications for Automated Writing Evaluation Software Weir, C. J. (2005). Language testing and validation: An evidence-based approach. New York: Palgrave MacMillan. ↗
  23. Wolf, F. & Gibson, E. (2004). Representing Discourse Coherence: A Corpus-based Analysis. In Proceedings of the 20th International Conference on Computational Linguistics (COLING) (article no. 134), Geneva, Switzerland.
  24. Wolf, F. & Gibson, E. (2005). Representing Discourse Coherence: A Corpus-based Analysis. Computational Linguistics, 31(2), 249-287.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/skoufaki-2009-exploratory-application-rhetorical-structure-theory/