Elamur Research⌕
☰
home / papers / 2014 / Annotating Argument Components and Relations in Persuasive Essays

Annotating Argument Components and Relations in Persuasive Essays

Summary

Supervised argument mining requires corpora that are annotated reliably for both argument components and the links between them. Earlier corpora covered other genres, such as legal texts, reviews and debates, and seldom reported a careful agreement analysis. Stab and Gurevych address this for English persuasive essays. Their annotation scheme has three component types (major claim, claim, premise) and two directed relation types (support, attack). Annotators choose span boundaries freely rather than working from predefined units. Guidelines were refined after a pre-study, and then three annotators labeled 90 essays. Agreement was substantial: αU = 0.72 for components and α = 0.81 for relations. The authors examine disagreements with confusion probability matrices. These show that most confusion falls between claims and premises, and that relations are mainly confused with "no relation." The consolidated corpus and the guidelines are freely released. The work gives educational NLP, such as argumentative writing support, a benchmark with documented reliability. It is also groundwork for training classifiers that detect components and relations.

Key points

  • Scheme: each essay has one major claim, which states the author's stance and usually sits in the introduction. Claims are marked as for or against the major claim. Premises are linked to their targets by support or attack relations.
  • Agreement: three annotators on 90 essays reached αU = 0.72 for component spans and labels and α = 0.81 for relations.
  • Final corpus: 1,673 sentences, 34,917 tokens, 90 major claims, 429 claims (365 for, 64 against), 1,033 premises, 1,312 support and 161 attack relations. This averages about 388 tokens and 11 premises per essay.
  • Consolidation: an annotation entered the final corpus only when at least two annotators agreed on both the category and the boundaries. The cases this rule could not settle (7 for components, 10 for relations) were resolved by group discussion.
  • Availability: the corpus and guidelines are distributed through the UKP Lab argumentation-mining page (TU Darmstadt).

Interesting ideas

  • I1. Two-stage annotation with consolidation in between: components are merged before anyone annotates relations, so every annotator links the same units. This keeps relation agreement from being polluted by span disagreements, and it could help any pipeline that annotates entities and then relations, such as NER followed by relation extraction.
  • I2. Confusion probability matrix (CPM) for more than two annotators: the matrix gives the probability that one annotator picks a category given another annotator's choice. It extends the ordinary confusion matrix to multi-annotator studies and is a cheap way to see which label pairs cause disagreement in any annotation project.
  • I3. Unitizing-aware agreement (αU) instead of fixed units: annotators may mark sub-sentence spans, and agreement is measured in a way that accounts for differing boundaries. This fits tasks where meaningful units do not line up with sentences.
  • I4. Role depends on structure, not only content: in a reasoning chain, the same statement can be a premise or a claim depending on what it supports. This explains why claim/premise labels are hard to agree on, and it suggests that modeling relations jointly with, or before, component types could help both annotation and automatic systems. The authors themselves propose trying relations first.
  • I5. Pre-study to shape the guidelines: running a small pilot and revising the guidelines from what it reveals is a transferable practice for building new corpora, for example Ukrainian argumentation or discourse resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

162 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 155 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stab2014annotating,
  title = {{Annotating Argument Components and Relations in Persuasive Essays}},
  author = {Stab, Christian and Gurevych, Iryna},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  url = {https://aclanthology.org/C14-1142/}
}

References

30 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Yigal Attali, Will Lewis, and Michael Steier. 2013. Scoring with the computer: Alternative procedures for improving the reliability of holistic essay scoring. Language Testing, 30(1):125–141. find
  3. Beata Beigman Klebanov and Michael Flor. 2013. Argumentation-Relevant Metaphors in Test-Taker Essays. In Proceedings of the First Workshop on Metaphor in NLP, pages 11–20, Atlanta, GA, USA. find
  4. Beata Beigman Klebanov and Derrick Higgins. 2012. Measuring the use of factual information in test-taker essays. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 63– 72, Montreal, Quebec, Canada. find
  5. Jamal Bentahar, Bernard Moulin, and Micheline Bélanger. 2010. A taxonomy of argumentation models used for knowledge representation. Artificial Intelligence Review, 33(3):211–259. find
  6. Jill Burstein and Magdalena Wolska. 2003. Toward evaluation of writing style: finding overly repetitive word use in student essays. In Proceedings of the tenth conference of European chapter of the Association for Computational Linguistics, EACL ’03, pages 35–42, Budapest, Hungary. find
  7. Jill Burstein, Karen Kukich, Susanne Wolff, Ji Lu, and Martin Chodorow. 1998. Enriching Automated Essay Scoring Using Discourse Marking. In Proceedings of the Workshop on Discourse Relations and Discourse Markers, pages 15–21, Montreal, Quebec, Canada. find
  8. Jodie A. Butler and M. Anne Britt. 2011. Investigating Instruction for Improving Revision of Argumentative Essays. Written Communication, 28(1):70–96. find
  9. Elena Cabrio and Serena Villata. 2012. Natural language arguments: A combined approach. In Proceedings of the 20th European Conference on Artificial Intelligence, ECAI ’12, pages 205–210, Montpellier, France. find
  10. Jean Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2):249–254. find our card
  11. Silvie Cinková, Martin Holub, and Vincent Krı́ž. 2012. Managing uncertainty in semantic tagging. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL ’12, pages 840–850, Avignon, France. find
  12. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. find
  13. Vanessa Wei Feng and Graeme Hirst. 2011. Classifying arguments by scheme. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 987–996, Portland, OR, USA. find
  14. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  15. James B. Freemen. 2011. Argument Structure: Representation and Theory, volume 18 of Argumentation Library. Springer. find
  16. Klaus Krippendorff. 1980. Content Analysis: An Introduction to its Methodology. Sage. find
  17. Klaus Krippendorff. 2004. Measuring the Reliability of Qualitative Text Analysis Data. Quality & Quantity, 38(6):787–800. find
  18. Nitin Madnani, Michael Heilman, Joel Tetrault, and Martin Chodorow. 2012. Identifying High-Level Organizational Elements in Argumentative Discourse. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAAC… find
  19. Raquel Mochales-Palau and Marie-Francine Moens. 2008. Study on the Structure of Argumentation in Case Law. In JURIX the twenty-first annual conference on legal knowledge and information systems, pages 11–20, Florence, Italy. find
  20. Raquel Mochales-Palau and Marie-Francine Moens. 2011. Argumentation mining. Artificial Intelligence and Law, 19(1):1–22. find
  21. Andreas Peldszus and Manfred Stede. 2013. From Argument Diagrams to Argumentation Mining in Texts: A Survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1):1–31. find
  22. Chris Reed, Raquel Mochales-Palau, Glenn Rowe, and Marie-Francine Moens. 2008. Language resources for studying argument. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC ’08, pages 2613–2618, Marrakech, Morocco. find
  23. Niall Rooney, Hui Wang, and Fiona Browne. 2012. Applying kernel methods to argumentation mining. In Proceedings of the Twenty-Fifth International Florida Artificial Intelligence Research Society Conference, FLAIRS ’12, pages 272–275, Marco Island, FL, USA. find
  24. Victor D. Sampson and Douglas B. Clark. 2006. Assessment of argument in science education: A critical review of the literature. In Proceedings of the 7th International Conference on Learning Sciences, ICLS ’06, pages 655–661, Bloomington, IN, USA. find
  25. William A. Scott. 1955. Reliability of Content Analysis: The Case of Nominal Scale Coding. Public Opinion Quarterly, 19(3):321–325. find
  26. Alan Sergeant. 2013. Automatic argumentation extraction. In Proceedings of the 10th European Semantic Web Conference, ESWC ’13, pages 656–660, Montpellier, France. find
  27. Mark D. Shermis and Jill Burstein. 2013. Handbook of Automated Essay Evaluation: Current Applications and New Directions. Routledge Chapman & Hall. find
  28. Stephen E. Toulmin. 1958. The uses of Argument. Cambridge University Press. find
  29. Maria Paz Garcia Villalba and Patrick Saint-Dizier. 2012. Some facets of argument mining for opinion analysis. In Proceeding of the 2012 conference on Computational Models of Argument, COMMA ’12, pages 23–34, Vienna, Austria. find
  30. Douglas Walton, Chris Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cambridge University Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stab-2014-annotating-argument-components-relations-persuasive/