Elamur Research⌕
☰
home / papers / 2015 / Modeling Argument Strength in Student Essays

Modeling Argument Strength in Student Essays

Summary

Persing and Ng address automatically scoring the argument strength of argumentative student essays, meaning how convincingly an essay defends its thesis. Most essay-scoring work either gives one holistic grade or targets other dimensions such as coherence or prompt relevance. The authors selected 1000 essays from the ICLE learner corpus, written in response to 10 argumentative prompts, and had trained annotators score each one from 1 to 4 in half-point steps. They treat prediction as regression with a linear SVM regressor (LIBSVM). Their feature set combines a reimplemented, learning-based version of Ong et al.'s sentence-labeling heuristics with seven new feature types, including:

  • POS n-grams,
  • prompt agreement,
  • argument component predictions,
  • transitional phrases,
  • semantic frames.

The full system significantly beats the Ong-style baseline on three of four error metrics, by up to 16.1%. The annotations are released publicly, which addresses the lack of a shared corpus for this dimension of essay quality.

Key points

  • Corpus: 1000 essays from ICLE covering 10 prompts and 15 native languages. Scores range from 1.0 to 4.0 in half-point increments and cluster heavily at 2.5 (342 essays) and 3.0 (372 essays).
  • Agreement: 846 essays were double-annotated. Annotators matched exactly only 26% of the time, but their scores fell within 0.5 points for 67% of essays and within 1.0 point for 89%. When annotators disagreed, the gold label was their average, rounded down to the nearest half point.
  • Baselines:
    • Baseline 1 always predicts the most frequent training score.
    • Baseline 2 converts Ong et al.'s rule-based sentence labels (HYPOTHESIS, OPPOSES, CLAIM, SUPPORTS) into three regression features.
  • Model setup: a linear SVM regressor from LIBSVM, with the regularization parameter c tuned on validation data over 10^0 to 10^6. Results are reported with four metrics: S1, S2, S3 and PC.
  • Feature importance: ablation shows that POS n-grams, prompt agreement and argument component predictions matter most. Semantic frame features (SFR) rank as most important only under the S1 metric.

Interesting ideas

  • I1. Regression for ordinal grades: modeling half-point scores as a continuous target means a 2.5 counts as closer to 3.0 than to 1.0. A classifier ignores that ordering, so the regression framing is a cheap fit for any ordinal rating task.
  • I2. Rules as features: the authors take hand-crafted heuristic rules from earlier work and use their outputs as features for a learner rather than as final decisions. This lets a supervised model decide how much each heuristic should count.
  • I3. Greedy backward ablation per metric: they repeatedly drop the least useful feature type and record the result separately for each metric. This showed that which features look "best" depends on the metric, a reminder to fix the evaluation goal before designing the system.
  • I4. Prompt-independent discourse cues: transitional-phrase lists grouped by function (for example, restating points) are normalized per sentence. This yields features that should carry over across topics, which helps whenever training and test prompts differ.
  • I5. Quantile analysis of predictions: they tabulate quartiles of predicted scores for each gold score. This exposes a model's pull toward frequent labels while still showing whether predictions rise as gold scores rise, a useful diagnostic for skewed regression data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1053

@inproceedings{persing2015modeling,
  title = {{Modeling Argument Strength in Student Essays}},
  author = {Persing, Isaac and Ng, Vincent},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1053},
  url = {https://doi.org/10.3115/v1/p15-1053}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with E-rater v.2.0. Journal of Technology, Learning, and Assessment, 4(3). find
  2. Avrim Blum and Pat Langley. 1997. Selection of relevant features and examples in machine learning. Artificial Intelligence, 97(1–2):245–271. find
  3. Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18(1):32–39. find
  4. Chih-Chung Chang and Chih-Jen Lin, 2001. LIB-SVM: A library for support vector machines. Software available at http://www.csie.ntu. edu.tw/˜cjlin/libsvm. ↗
  5. Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A. Smith. 2010. Probabilistic frame-semantic parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 948–956. find our card
  6. Mohammad Hassan Falakmasir, Kevin D. Ashley, Christian D. Schunn, and Diane J. Litman. 2014. Identifying thesis and conclusion statements in student essays to scaffold peer review. In Intelligent Tutoring Systems, pages 254–259. Springer International Publishing. find
  7. Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. International Corpus of Learner English (Version 2). Presses universitaires de Louvain. find
  8. Derrick Higgins, Jill Burstein, Daniel Marcu, and Claudia Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In Human Language Technologies: The 2004 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 185–192. find
  9. Thomas K. Landauer, Darrell Laham, and Peter W. Foltz. 2003. Automated scoring and annotation of essays with the Intelligent Essay Assessor TM . In Automated Essay Scoring: A Cross-Disciplinary Perspective, pages 87–112. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. find
  10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  11. Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu. ↗
  12. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(1):25–55. find
  13. Nathan Ong, Diane Litman, and Alexandra Brusilovsky. 2014. Ontology-based argument mining and automatic essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 24–28. find
  14. Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 260–269. find
  15. Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534–1543. find
  16. Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229– 239. find
  17. Mark D. Shermis and Jill C. Burstein. 2003. Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. find
  18. Mark D. Shermis, Jill Burstein, Derrick Higgins, and Klaus Zechner. 2010. Automated essay scoring: Writing assessment and instruction. In International Encyclopedia of Education (3rd edition). Elsevier, Oxford, UK. find
  19. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69–78. find
  20. Christian Stab and Iryna Gurevych. 2014a. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers, pages 1501–1510. find our card
  21. Christian Stab and Iryna Gurevych. 2014b. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 46–56. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/persing-2015-modeling-argument-strength-student-essays/