Elamur Research⌕
☰
home / papers / 2016 / End-to-End Argumentation Mining in Student Essays

End-to-End Argumentation Mining in Student Essays

Paper · 2016 · Isaac Persing and Vincent Ng

Summary

The paper addresses the task of mining arguments from persuasive essays written by students, under conditions that are entirely realistic. The system receives raw essays and must find where argument components sit, label them as Major Claim, Claim or Premise, and decide which pairs are linked by Support or Attack. Earlier work by Stab and Gurevych (S&G) started from gold component spans, which sidesteps the hardest part.

Using S&G's 90-essay corpus, the authors first build a pipeline baseline:

  • parse-tree heuristics over-generate candidate spans;
  • a maximum entropy classifier types each span;
  • a second classifier labels relations between span pairs.

Errors from the first stage flow into the second. To counter this, they combine both classifiers' outputs with joint inference in an Integer Linear Programming (ILP) framework, adding constraints that keep the predicted argument structure globally consistent. They also design a new ILP objective that maximizes F-score directly, replacing the usual objective.

The joint system cuts F-score error by 18.5% relative to the pipeline. The F-score objective is general enough to apply to other ILP-based NLP tasks.

Key points

  • Data: S&G's corpus of 90 essays, with 417 paragraphs, 1,673 sentences, 90 Major Claims, 429 Claims, 1,033 Premises, 1,312 Support relations and 161 Attack relations.
  • Candidate extraction relies on Stanford CoreNLP parses and boundary rules over S, SBAR and PP nodes. The candidates cover 92.1% of gold components exactly and 98.4% approximately (more than half of the tokens shared).
  • Relation identification is a five-way task: no relation, plus support or attack in either direction. Both classifiers use MALLET maximum entropy models with S&G's structural, lexical, syntactic, indicator and contextual features.
  • Gains over the pipeline baseline:
    • 18.5% relative error reduction in F-score overall;
    • F-score improvements of 13.2% for component identification and 14.6% for relation identification.
  • Ablation results (average F of the full system is 38.6):
    • dropping the ILP inference lowers it to 27.0;
    • dropping the F-score objective lowers it to 33.4.

Interesting ideas

  • I1. Over-generate, then classify. Cheap, recall-oriented syntactic rules produce many candidate spans, and a classifier filters them. This turns span detection into ordinary classification and suits any clause-level extraction task where boundaries are fuzzy.
  • I2. Global constraints across subtasks. Separate local classifiers are reconciled by an ILP with structural consistency constraints. Removing a constraint group aimed at one subtask also hurt the other subtask, which shows the two tasks really help each other through shared inference.
  • I3. Optimize the metric you report. Rewriting the ILP objective so the solver maximizes F-score, instead of summing classifier confidences, aligns decoding with evaluation. This can carry over to any constrained decoding problem judged by F-score.
  • I4. Two matching regimes. Scoring with both exact-boundary and approximate-overlap matching separates boundary errors from typing errors. This is a useful evaluation habit for span-based annotation.
  • I5. Error analysis that points to features. The authors trace failures to specific gaps and suggest matching fixes: - missed components often contain topic-related words, so topic similarity features could help; - major claims use stance-taking words, so an arguing-language lexicon could help; - missed links often involve shared entities, so coreference features could help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States47 works · 5,363 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Philosophy · 211 works

Cite

Cite as Isaac Persing and Vincent Ng. 2016. End-to-End Argumentation Mining in Student Essays. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1164

@inproceedings{persing2016end,
  title = {{End-to-End Argumentation Mining in Student Essays}},
  author = {Isaac Persing and Vincent Ng},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1164},
  url = {https://doi.org/10.18653/v1/n16-1164}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18(1):32–39. find
  2. Yejin Choi, Eric Breck, and Claire Cardie. 2006. Joint extraction of entities and relations for opinion recognition. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 431–439. find
  3. Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39:1–38. find
  4. Mohammad Hassan Falakmasir, Kevin D. Ashley, Christian D. Schunn, and Diane J. Litman. 2014. Identifying thesis and conclusion statements in student essays to scaffold peer review. In Intelligent Tutoring Systems, pages 254–259. find
  5. Springer International Publishing. Eirini Florou, Stasinos Konstantopoulos, Antonis Koukourikos, and Pythagoras Karampiperis. 2013. Argument extraction for supporting public policy formulation. In Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Huma… find
  6. Theodosios Goudas, Christos Louizos, Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news, blogs, and the social web. International Journal on Artificial Intelligence Tools, 24(5). find
  7. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of the 25th International Conference on Computational Linguistics, pages 1489–1500. our card
  8. Marco Lippi and Paolo Torroni. 2015. Context-independent claim detection for argument mining. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, pages 185–191. find
  9. Marco Lippi and Paolo Torroni. 2016. Argument mining from speech: Detecting claims in political debates. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. find
  10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. our card
  11. Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu. ↗
  12. David Milne and Ian Witten. 2008. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceedings of AAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy, pages 25–30. find
  13. Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, pages 225–230. find
  14. Nathan Ong, Diane Litman, and Alexandra Brusilovsky. 2014. Ontology-based argument mining and automatic essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 24–28. find
  15. Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: The detection, classification and structure of arguments in text. In Proceedings of the 12th International Conference on Artificial Intelligence and Law, pages 98–107. find
  16. Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38. our card
  17. Andreas Peldszus and Manfred Stede. 2015. Joint prediction in mst-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 938–948. our card
  18. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450. our card
  19. Niall Rooney, Hui Wang, and Fiona Browne. 2012. Applying kernel methods to argumentation mining. In Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference. find
  20. Dan Roth and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning, pages 1–8. our card
  21. Christos Sardianos, Ioannis Manousos Katakis, Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news. In Proceedings of the Second Workshop on Argumentation Mining, pages 56–66. find
  22. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. our card
  23. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69–78. find
  24. Christian Stab and Iryna Gurevych. 2014a. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on Computational Linguistics, pages 1501–1510. our card
  25. Christian Stab and Iryna Gurevych. 2014b. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 46–56. our card
  26. Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 217–226. find
  27. Simone Teufel. 1999. Argumentative Zoning: Information Extraction from Scientific Text. Ph.D. thesis, University of Edinburgh. find
  28. Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1640–1649. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/persing-2016-end-end-argumentation-mining-student/