Elamur Research⌕
☰
home / papers / 2021 / Parsing Argumentative Structure in English-as-Foreign-Language Essays

Parsing Argumentative Structure in English-as-Foreign-Language Essays

Paper · 2021 · Jan Wira Gotama Putra, Simone Teufel and Takenobu Tokunaga

Summary

The paper applies argument mining to essays by learners of English as a foreign language. Their writing is noisier and less coherent than the polished texts that most argument-mining corpora assume. The authors use ICNALE-AS2R, a sentence-level annotation of 434 ICNALE essays by Asian college students. Each sentence is a unit, and the goal is to recover a tree rooted at the essay's major claim. The work is split into a pipeline: first decide which sentence each sentence attaches to, then name the relation on that link. Several neural architectures are compared, including a BiLSTM sequence tagger and a biaffine model borrowed from syntactic dependency parsing, with BERT and Sentence-BERT as encoders. Biaffine with SBERT links best, while fine-tuned BERT labels relations best. The corpus also stores annotator-improved versions of every essay, so the authors can test training on cleaner text. Mixing it in reaches 94% of the in-domain end-to-end accuracy. This suggests that corpora of well-written text can help parse learner writing.

Key points

  • ICNALE-AS2R covers 434 essays on two prompts (a smoking ban and students' part-time jobs). Sentences are marked as argumentative or not, linked into a tree, and labelled with one of four relations: SUPPORT, ATTACK, DETAIL, RESTATEMENT.
  • Annotators also reordered sentences into a more logical arrangement and minimally repaired the result. For example, they replaced pronouns with their referents, fixed connectives and made the major claim self-contained. This gives a parallel original/improved corpus.
  • Sentence linking: biaffine + SBERT was best, with F1-macro .323 on individual links. Relation labelling: fine-tuned BERT was best, with F1-macro .595. Models that were not fine-tuned generally did better with SBERT than with BERT. Multi-task learning helped the sequence tagger in some respects.
  • End-to-end accuracy over 20 runs was .341 for the in-domain system and .321 for the cross-domain system (trained on noisy plus improved essays). The difference is significant, and both remain clearly below human agreement.
  • Errors cluster around:
    • major claims that do not open the essay;
    • reasons placed before the claim they support;
    • attaching a sub-argument as a sibling instead of a child;
    • overusing RESTATEMENT when a claim shares most words with the major claim.

Interesting ideas

  • I1. Discourse structure is cast as dependency parsing over sentences: each sentence picks a head, the major claim acts as root, and a biaffine scorer chooses links, mirroring syntactic parsing one level up.
  • I2. Evaluation pairs a strict per-sentence score with per-subtask agreement. A sentence counts only if its argumentative status, target distance and relation are all right, while Cohen's κ against gold is reported for each subtask separately.
  • I3. A parallel corpus of noisy texts and their cleaned versions makes it possible to measure the domain gap directly: train with clean data, test on noisy, and compare to in-domain training.
  • I4. The error analysis suspects the model learned RESTATEMENT from surface word overlap, a shortcut rather than real paraphrase understanding.
  • I5. After reordering sentences, annotators did minimal repair: they resolved pronouns to referents, dropped or replaced connectives that no longer fit, and rewrote dependent claims to stand alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The same biaffine machinery in gparse could be run over sentence embeddings to produce an explainable inter-sentence tree alongside the UD tree.
  • I2 (learn): Pipeline evaluations could report both an all-or-nothing score per unit and per-stage agreement, showing where errors enter rather than averaging them away.
  • I3 (learn): Pairing learner-style or garbled inputs with repaired versions would let Mova quantify how much clean-text training transfers to messy input before collecting more noisy annotation.
  • I4 (world): Restatement or paraphrase detection could compare entities and states in the text world, using lexical-overlap-only cases as a negative control.
  • I5 (back): When the compressor regenerates text from the meaning graph in a new order, it could apply the same repairs so references and connectives stay valid.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Tokyo Institute of TechnologyTokyo · Japan22 works · 2,137 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Jan Wira Gotama Putra, Simone Teufel and Takenobu Tokunaga. 2021. Parsing Argumentative Structure in English-as-Foreign-Language Essays. Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{putra2021parsing,
  title = {{Parsing Argumentative Structure in English-as-Foreign-Language Essays}},
  author = {Jan Wira Gotama Putra, Simone Teufel and Takenobu Tokunaga},
  booktitle = {Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2021},
  url = {https://aclanthology.org/2021.bea-1.10/}
}

References

56 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khalid Al-Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. A news editorial corpus for mining argumentation strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3433–3443, Osaka, Japan.
  2. The COLING 2016 Organizing Committee. find
  3. Kevin D. Ashley. 1990. Modeling legal argument - reasoning with cases and hypotheticals. Artificial intelligence and legal reasoning. MIT Press. find
  4. Nahla Nola Bacha. 2010. Teaching the academic argument in a university efl environment. Journal of English for Academic Purposes, 9(3):229 – 241. ↗
  5. Betty Bamberg. 1983. What makes a text coherent. College Composition and Communication, 34(4):417–429. ↗
  6. Elena Cabrio and Serena Villata. 2012. Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 208–212, Jeju Island, Korea. Association for Computational Linguistics. find
  7. Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018. Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621– 631, Melbourne, Australia. Association for Computational Linguistics. find
  8. Y. Chu and T. Liu. 1965. On the shortest arborescence of a directed graph. find
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  10. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  11. Ulla Connor. 2002. New directions in contrastive rhetoric. TESOL Quarterly, 36(4):493–510. ↗
  12. Mamoru Deguchi and Kazunori Yamaguchi. 2019. Argument component classification by relation identification by neural network and TextRank. In Proceedings of the 6th Workshop on Argument Mining, pages 83–91, Florence, Italy. Association for Computational Linguistics. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  14. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the International Conference on Learning Representations (ICLR).
  15. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards - B. Mathematics and Mathematical Physics, 71B:233– 240. find
  16. Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017. Neural end-to-end learning for computational argumentation mining. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11–22, Vancouver, Canada. Association for Computational Linguistics.
  17. Ronald Aylmer Fisher. 1937. The design of experiments. Oliver and Boyd, Edinburgh. find
  18. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. CoRR, abs/1803.07640. ↗
  19. Debanjan Ghosh, Aquila Khanam, Yubo Han, and Smaranda Muresan. 2016. Coarse-grained argumentation features for scoring persuasive essays. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 549–554, Berlin, Germany. Association for Computational Linguistics. find
  20. Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204. ↗
  21. Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125–179. ↗
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  23. Eduard H. Hovy. 1991. Approaches to the Planning of Coherent Text, pages 83–102. Springer US, Boston, MA. find
  24. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991. ↗
  25. Ryu Iida and Takenobu Tokunaga. 2014. Building a corpus of manually revised texts from discourse perspective. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 936–941, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  26. Roz Invanic. 2004. Discourses of writing and learning to write. Language and Education, 18:220–245. find
  27. Shinichiro Ishikawa. 2013. The icnale and sophisticated contrastive interlanguage analysis of asian learners of english. Learner Corpus Studies in Asia and the World, 1:91–118. ↗
  28. Shinichiro Ishikawa. 2018. The icnale edited essays: A dataset for analysis of l2 english learner essays based on a new integrative viewpoint. English Corpus Linguistics, 25:1–14. find
  29. Ann M. Johns. 1986. The esl student and the revision process: Some insights from schema theory. Journal of Basic Writing, 5(2):70 – 80. ↗
  30. Gaku Morio, Hiroaki Ozaki, Terufumi Morishita, Yuta Koreeda, and Kohsuke Yanai. 2020. Towards better non-tree argument mining: Proposition-level biaffine parsing with task-specific parameterization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3259–3266, Online. Association for Computational Linguistics. find
  31. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  32. Robert B. Kaplan. 1966. Cultural thought patterns in inter-cultural education. Language Learning, 16(1-2):1–20. ↗
  33. Andreas Peldszus and Manfred Stede. 2013. From argument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence, 7(1):1–31. ↗
  34. Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the International Conference on Learning Representations (ICLR). find
  35. Andreas Peldszus and Manfred Stede. 2016. An annotated corpus of argumentative microtexts. In Argumentation and Reasoned Action - Proceedings of the 1st European Conference on Argumentation, Lisbon, 2015. find
  36. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: a method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR). ↗
  37. Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229–239, Cambridge, MA. Association for Computational Linguistics.
  38. Christian Kirschner, Judith Eckle-Kohler, and Iryna Gurevych. 2015. Linking the thoughts: Analysis of argumentation structures in scientific publications. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 1–11, Denver, CO. Association for Computational Linguistics. find
  39. Marco Lippi and Paolo Torroni. 2016. Argumentation mining: State of the art and emerging trends. ACM Trans. Internet Technol., 16(2). find
  40. Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105–1116, Berlin, Germany. Association for Computational Linguistics.
  41. Gaku Morio and Katsuhide Fujita. 2018. End-to-end argument mining for discussion threads based on parallel constrained pointer architecture. In Proceedings of the 5th Workshop on Argument Mining, pages 11–21, Brussels, Belgium. Association for Computational Linguistics. find
  42. Jan Wira Gotama Putra, Simone Teufel, Kana Matsumura, and Takenobu Tokunaga. 2020. TIARA: A tool for annotating discourse relations and sentence reordering. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6912–6920, Marseille, France. European Language Resources Association. find
  43. Ella Rabinovich, Sergiu Nisioi, Noam Ordan, and Shuly Wintner. 2016. On the similarities between native, non-native and translated texts. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1881, Berlin, Germany. Association for Computational Linguistics. ↗
  44. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. ↗
  45. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS. find
  46. Tony Silva. 1993. Toward an understanding of the distinct nature of l2 writing: The esl research and its implications. TESOL Quarterly, 27(4):657–677. ↗
  47. Maria Skeppstedt, Andreas Peldszus, and Manfred Stede. 2018. More or less controlled elicitation of argumentative text: Enlarging a microtext corpus via crowdsourcing. In Proceedings of the 5th Workshop on Argument Mining, pages 155–163, Brussels, Belgium. Association for Computational Linguistics. find
  48. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69– 78, Baltimore, Maryland. Association for Computational Linguistics. find
  49. Christian Stab and Iryna Gurevych. 2014. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1501–1510, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  50. Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619–659. ↗
  51. Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating hand-crafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077–6088, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  52. Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680–1691, Osaka, Japan. find
  53. The COLING 2016 Organizing Committee. find
  54. Bonnie Webber and Aravind Joshi. 2012. Discourse structure and computation: Past, present and future. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 42– 54, Jeju Island, Korea. Association for Computational Linguistics. find
  55. Hiroaki Yamada, Simone Teufel, and Takenobu Tokunaga. 2019. Building a corpus of legal argumentation in japanese judgement documents: towards structure-based summarisation. Artificial Intelligence and Law, 27(2):141–170. ↗
  56. Fan Zhang and Diane Litman. 2015. Annotation and classification of argumentative writing revisions. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 133–143. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/putra-2021-parsing-argumentative-structure-english-as/