Elamur Research⌕
☰
home / papers / 2010 / Modeling Organization in Student Essays

Modeling Organization in Student Essays

Paper · 2010 · Isaac Persing, Alan Davis and Vincent Ng

Summary

The paper tackles automatic scoring of essay organization: how well an essay introduces a topic, takes a stance, backs it up and wraps up. Most essay scorers give one holistic grade, and organization has rarely been modeled. The authors annotated 1003 argumentative essays from the International Corpus of Learner English (ICLE) with organization scores from 1 to 4 in half-point steps, and released the data. They first assign discourse roles heuristically, to sentences and then to paragraphs, using paragraph position and the types of sentences a paragraph contains. The resulting label sequences are then used to predict a score in two ways: through heuristic sequence-alignment systems, and through SVM regressors. The SVMs use a linear kernel over flat features, a string kernel, and a newly proposed alignment kernel. Each kernel contributes useful information. Combining all three gives the best system, which significantly beats the strongest heuristic baseline. The work matters because it sets a first baseline and a public benchmark for a discourse-level task that previously had little annotated data.

Key points

  • Data: 1003 argumentative ICLE essays covering twelve topics and fifteen native languages. Scores range from 1.0 to 4.0 in half-point steps, and most essays fall at 3.0 (416) or 3.5 (289).
  • Inter-annotator agreement: 846 essays were scored by two annotators. They agreed exactly only 29% of the time, but came within 0.5 points on 71% and within 1.0 point on 93% of essays. Treating one annotator as gold and the other as predictions gives a mean error of 0.54 and an MSE of 0.50.
  • Label schemas:
    • Paragraphs get four labels: Introduction, Body, Rebuttal, Conclusion.
    • Sentences get ten labels, such as Thesis, Main Idea, Support, Rebuttal, Solution and Suggestion. These extend the scheme of Burstein et al. (2003).
  • Model comparison:
    • The best single-kernel model is the linear kernel (Rl nps). Its main gain over the best heuristic (H p) is that it removes the need to pick among 12 heuristic variants.
    • The string and alignment kernels are weak alone, but their combination (Rsa) is the best two-kernel system.
    • The three-kernel model significantly beats H p (p < 0.05 on S1, p < 0.01 on S3).
  • Feature analysis: Backward elimination on the linear-kernel features shows nearest-neighbor features matter most and flat paragraph-sequence features matter least. Removing any feature group still hurts performance.

Interesting ideas

  • I1. Labels as a symbol string: Each essay becomes a short string of discourse labels, which can then be compared to other essays with bioinformatics-style sequence alignment. The same trick could suit any task where the order of structural units matters, such as dialogue flow or document templates.
  • I2. Stacking heuristic labels: Without gold paragraph-role annotations, the authors infer sentence roles first and then derive paragraph roles from position plus sentence content. This is a cheap way to get structure for a downstream task when annotating intermediate layers is too costly.
  • I3. Combining weak kernels: The string and alignment kernels perform poorly on their own but excel when combined. Evaluating complementary representations together, not just individually, can therefore reveal value that ablations of single components would hide.
  • I4. Agreement within a tolerance: Reporting agreement within 0.5 and 1.0 points, plus error measures, gives a more honest ceiling for fine-grained ordinal scoring than exact agreement alone. This is useful for any subjective graded annotation.
  • I5. Learning to avoid choosing: Feeding many heuristic signals into a learner can match the best hand-picked heuristic, so no single heuristic has to be selected up front. This is a practical pattern whenever several plausible rule sets compete.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States47 works · 5,363 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

Cite

Cite as Isaac Persing, Alan Davis and Vincent Ng. 2010. Modeling Organization in Student Essays. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{persing2010modeling,
  title = {{Modeling Organization in Student Essays}},
  author = {Isaac Persing, Alan Davis and Vincent Ng},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1023/}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). find
  2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the ACL, pages 141–148. find
  3. Regina Barzilay and Lillian Lee. 2002. Bootstrapping lexical choice via multiple-sequence alignment. In Proceedings of EMNLP, pages 164–171. find
  4. Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In HLT-NAACL 2003: Main Proceedings, pages 16–23.
  5. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In HLT-NAACL 2004: Main Proceedings, pages 113–120.
  6. Razvan Bunescu and Raymond Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of HLT/EMNLP, pages 724–731.
  7. Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004. Automated essay evaluation: The Criterion online writing evaluation service. AI Magazine, 25(3), 27–36. find
  8. Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the write stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18(1):32–39. find
  9. Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In Proceedings of the ACL, pages 263–270. find
  10. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273–297. find
  11. Scott Elliot. 2001. IntelliMetric: From here to validity. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. find
  12. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In NAACL HLT 2007: Proceedings of the Main Conference, pages 436–443.
  13. Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. International Corpus of Learner English (Version 2). Presses universitaires de Louvain. find
  14. Derrick Higgins, Jill Burstein, Daniel Marcu, and Claudia Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In HLT-NAACL 2004: Main Proceedings, pages 185–192. find
  15. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Bernhard Scholkopf and Alexander Smola, editors, Advances in Kernel Methods - Support Vector Learning, pages 44–56. MIT Press. find
  16. Thomas K. Landauer, Darrell Laham, and Peter W. Foltz. 2003. Automated scoring and annotation of essays with the Intelligent Essay Assessor TM . In Automated Essay Scoring: A Cross-Disciplinary Perspective, pages 87–112. find
  17. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Christopher J. C. H. Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2:419–444. find
  18. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(1):25–55. find
  19. Alessandro Moschitti. 2004. A study on convolution kernels for shallow statistic parsing. In Proceedings of the ACL, pages 335–342. find
  20. Saul Ben Needleman and Christian Dennis Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3):443–453, March. find
  21. Mark Shermis and Jill Burstein. 2003. Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. find
  22. Mark Shermis, Jill Burstein, Derrick Higgins, and Klaus Zechner. 2010. Automated essay scoring: Writing assessment and instruction. In International Encyclopedia of Education (3rd edition), pages 20–26. find
  23. Tony Silva. 1993. Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. 27(4):657–677. find
  24. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 803–810. find
  25. Yannick Versley, Alessandro Moschitti, Massimo Poesio, and Xiaofeng Yang. 2008. Coreference systems based on kernels methods. In Proceedings of COL-ING, pages 961–968.
  26. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of ICML, pages 412–420. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/persing-2010-modeling-organization-student-essays/