Elamur Research⌕
☰
home / papers / 2023 / Span Identification of Epistemic Stance-Taking in Academic Written English

Span Identification of Epistemic Stance-Taking in Academic Written English

Paper · 2023 · Masaki Eguchi and Kristopher Kyle

Summary

Most automated writing evaluation (AWE) tools judge vocabulary, grammar and organization. They largely ignore how writers position their claims relative to other voices. Eguchi and Kyle address this gap with a stance-analysis system built on the engagement system of Appraisal theory (Martin & White, 2005). They created the Engagement Discourse Treebank (EDT): 4,688 sentences of academic English, drawn from student assignments and timed essays by first- and second-language writers. The sentences are annotated with spans for eight rhetorical stance categories, such as PROCLAIM and ATTRIBUTION, plus some extra discourse elements. They frame the problem as span identification and compare three architectures in 5-fold cross-validation: a spaCy spancat baseline, a Dual-RoBERTa model, and RoBERTa with a Bi-LSTM layer. The RoBERTa + LSTM model performed best, reaching macro-F1 of .7208, slightly above the agreement between human annotators before adjudication (.6629). This matters because stance features can make essay scoring richer and easier to interpret. They can also be shown to learners as highlighted feedback.

Key points

  • EDT size: 4,688 sentences, 126,411 tokens and 11,856 annotated spans. It is released on GitHub (LCR-ADS-Lab/Engagement-Discourse-Treebank), with a public annotation guideline and a Hugging Face demo.
  • Sources: two collections of university student writing, the British Academic Written English corpus (BAWE) and MICUSP (Michigan's corpus of papers written by students in upper-level courses), plus timed L2 essay corpora (e.g., ICNALE). Sampling used three-sentence windows rather than whole documents.
  • Best model, RoBERTa + LSTM: macro-F1 .7208, Cohen's Kappa .6738, MCC .6755. Dual-RoBERTa was close behind (Kappa .6694, MCC .6710), and both beat the default spaCy spancat.
  • The added Bi-LSTM made the model more stable. The authors suggest two possible reasons: extra sequential information, or a dimension-reduction effect on top of the span-enumeration setup.
  • The two annotators were undergraduate linguistics majors trained over 10 weeks. Their agreement was only moderate, but it improved on the earlier benchmark by Read and Carroll (2012).

Interesting ideas

  • I1. The same word can do different rhetorical jobs depending on context: "suggest" attributes an idea in "The authors suggest" but hedges the writer's own view in "We suggest." A function label therefore needs the surrounding context, not a dictionary lookup.
  • I2. The authors compare model output against inter-annotator agreement measured before adjudication, which gives a realistic ceiling for a subjective task instead of treating gold labels as perfect.
  • I3. Sampling short three-sentence windows from many documents, rather than a few full texts, covers more writers and styles for the same annotation budget.
  • I4. Engagement theory separates bare assertions (monogloss) from claims that acknowledge other voices (attributed, countered, conceded). This amounts to labeling who stands behind each proposition.
  • I5. The authors argue that interpretable stance features can sit beside opaque end-to-end scorers to keep grades explainable, with highlighted spans shown to learners.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could decide whether a reporting verb is attribution or hedging by checking its UD subject (first person vs. external source) and record this in the speech-act frame.
  • I2 (learn): when we grade silver data, agreement among the witness parsers could serve the same role, so we can tell whether model errors are real or only reflect noise in the task itself.
  • I3 (learn): when building silver or LLM-teacher datasets, spreading small excerpts across many sources should help the parser and frame extractors generalize beyond a few authors' habits.
  • I4 (world): facts in the world of a text could carry a source-and-commitment tag, so the reader can tell "X is true" apart from "someone says X" or "X, although disputed."
  • I5 (answer): when Mova grades or answers school-course tasks, it could show the stance spans and frames that drove its judgment as an explicit justification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Masaki Eguchi and Kristopher Kyle. 2023. Span Identification of Epistemic Stance-Taking in Academic Written English. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.35

@inproceedings{eguchi2023span,
  title = {{Span Identification of Epistemic Stance-Taking in Academic Written English}},
  author = {Masaki Eguchi and Kristopher Kyle},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.35},
  url = {https://doi.org/10.18653/v1/2023.bea-1.35}
}

References

61 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alsop, S., & Nesi, H. (2009). Issues in the development of the British Academic Written English (BAWE) corpus. Corpora, 4(1), 71–83. https://doi.org/10.3366/E1749503209000227 ↗
  2. Attali, Y. (2007). Construct validity of e-rater® in scoring TOEFL® essays. ETS Research Report Series, 2007(1), i–22. https://doi.org/10.1002/j.2333-8504.2007.tb02063.x ↗
  3. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press. find
  4. Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford University Press. find
  5. Bax, S., Nakatsuhara, F., & Waller, D. (2019). Researching L2 writers’ use of metadiscourse markers at intermediate and advanced levels. System, 83, 79–95. https://doi.org/10.1016/j.system.2019.02.010 ↗
  6. Biber, D. (2006). Stance in spoken and written university registers. Journal of English for Academic Purposes, 5(2), 97–116. https://doi.org/10.1016/j.jeap.2006.05.001 ↗
  7. Biber, D., & Finegan, E. (1988). Adverbial stance types in English. Discourse Processes, 11(1), 1–34. https://doi.org/10.1080/01638538809544689 ↗
  8. Biber, D., Johansso, S., Leech, G., Conrad, S., & Finegan, E. (Eds.). (1999). Longman grammar of spoken and written English (10. impression). Longman. find
  9. Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., & Chodorow, M. (2013). TOEFL11: A Corpus of Non-Native English. ETS Research Report Series, 2013(2), i–15. https://doi.org/10.1002/j.2333-8504.2013.tb02331.x ↗
  10. Burstein, J., Elliot, N., Klebanov, B. B., Madnani, N., Napolitano, D., Schwartz, M., Houghton, P., & Molloy, H. (2018). Writing MentorTM: Writing Progress Using Self-Regulated Writing Support. The Journal of Writing Analytics, 2(1), 285–313. https://doi.org/10.37514/JWA-J.2018.2.1.12 ↗
  11. Burstein, J., Elliot, N., & Molloy, H. (2016). Informing Automated Writing Evaluation Using the Lens of Genre: Two Studies. CALICO Journal, 33(1), 117– 141. https://doi.org/10.1558/cj.v33i1.26374 ↗
  12. Carr, N. T. (2013). Computer-automated scoring of written responses. In A. J. Kunnan (Ed.), The Companion to Language Assessment (pp. 1063– 1078). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118411360.wbcla124 ↗
  13. Chang, P., & Schleppegrell, M. (2011). Taking an effective authorial stance in academic writing: Making the linguistic resources explicit for L2 writers in the social sciences. Journal of English for Academic Purposes, 10(3), 140–151. https://doi.org/10.1016/j.jeap.2011.05.005 ↗
  14. Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT’s Attention. ArXiv:1906.04341 [Cs]. http://arxiv.org/abs/1906.04341 ↗
  15. Cotos, E. (2014). Genre-Based Automated Writing Evaluation for L2 Research Writing. Palgrave Macmillan UK. https://doi.org/10.1057/9781137333377 ↗
  16. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs]. http://arxiv.org/abs/1810.04805 ↗
  17. Eguchi, M. (2023). Automatic Analysis of Epistemic Stance-Taking in Academic English Writing: A Systemic Functional Approach [Doctoral dissertation]. University of Oregon. find
  18. Fiacco, J., Jiang, S., Adamson, D., & Rosé, C. (2022). Toward Automatic Discourse Parsing of Student Writing Motivated by Neural Interpretation. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 204–215. https://doi.org/10.18653/v1/2022.bea-1.25 ↗
  19. Fu, J., Huang, X., & Liu, P. (2021). SpanNER: Named Entity Re-/Recognition as Span Prediction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7183–7195. https://doi.org/10.18653/v1/2021.acl-long.558 ↗
  20. Fuoli, M. (2018). A stepwise method for annotating appraisal. Functions of Language, 25(2), 229–258. https://doi.org/10.1075/fol.15016.fuo ↗
  21. Gu, W., Zheng, B., Chen, Y., Chen, T., & Van Durme, B. (2022). An Empirical Study on Finding Spans (arXiv:2210.06824). arXiv. http://arxiv.org/abs/2210.06824 ↗
  22. Han, X., & Eisenstein, J. (2019). Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4238–4248. https://doi.org/10.18653/v1/D19-1433 ↗
  23. Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735 ↗
  24. Honnibal, M., Ines, M., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python (3.3). https://spacy.io ↗
  25. Huawei, S., & Aryadoust, V. (2023). A systematic review of automated writing evaluation systems. Education and Information Technologies, 28(1), 771–795. https://doi.org/10.1007/s10639-022-11200-7 ↗
  26. Hunston, S. (2004). Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In A. S. Partington (Ed.), Corpora and Discourse. find
  27. Hunston, S., & Thompson, G. (2000). Evaluation in Text: Authorial Stance and the Construction of Discourse. Oxford University Press, UK. find
  28. Hunt, K. W. (1965). Grammatical structures written at three grade levels. NCTE Research Report No. 3. http://eric.ed.gov/?id=ED113735 ↗
  29. Hyland, K. (2005). Metadiscourse: interaction in writing. Continuum. find
  30. Exploring Ishikawa, S. (2013). The ICNALE and Sophisticated Contrastive Interlanguage Analysis of Asian Learners of English. 神戸大学国際コミュニケー シ ョ ン セ ン タ ー. https://doi.org/10.24546/81006678 ↗
  31. Kyle, K. (2020). The relationship between features of source text use and integrated writing quality. Assessing Writing, 45, 100467. https://doi.org/10.1016/j.asw.2020.100467 ↗
  32. Lam, S. L., & Crosthwaite, P. (2018). APPRAISAL resources in L1 and L2 argumentative essays: A contrastive learner corpus-informed study of evaluative stance. Journal of Corpora and Discourse Studies, 1(1), 8. https://doi.org/10.18573/jcads.1 ↗
  33. Lancaster, Z. (2014). Exploring valued patterns of stance in upper-level student writing in the disciplines. Written Communication, 31(1), 27–57. https://doi.org/10.1177/0741088313515170 ↗
  34. Lee, K., He, L., Lewis, M., & Zettlemoyer, L. (2017). End-to-end Neural Coreference Resolution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 188–197. https://doi.org/10.18653/v1/D17-1018 ↗
  35. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Michael Lewis, Michael Lewis, Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv: Computation and Language.
  36. Lu, X. (2021). Directions for future automated analyses of L2 written texts. In R. M. Manchón & C. Polio, The Routledge handbook of second language acquisition and writing (1st ed., pp. 370– 382). Routledge. https://doi.org/10.4324/9780429199691-36 ↗
  37. Martin, J. R., & White, P. R. R. (2005). The language of evaluation: Appraisal in English. Palgrave Macmillan. find
  38. Nesi, H. (2021). Sources for courses: Metadiscourse and the role of citation in student writing. Lingua, 253, 103040. https://doi.org/10.1016/j.lingua.2021.103040 ↗
  39. Papay, S., Klinger, R., & Padó, S. (2020). Dissecting Span Identification Tasks with Performance Prediction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4881–4895. https://doi.org/10.18653/v1/2020.emnlp-main.396 ↗
  40. Pareti, S. (2016). PARC 3.0: A Corpus of Attribution Relations. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 3914–3920. https://aclanthology.org/L16-1619 ↗
  41. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1), 140:5485-140:5551. ↗
  42. Ramponi, A., & Plank, B. (2020). Neural Unsupervised Domain Adaptation in NLP — A Survey (arXiv:2006.00632). arXiv. https://doi.org/10.48550/arXiv.2006.00632 ↗
  43. Rao, A. R. (2022). ASRtrans at SemEval-2022 Task 4: Ensemble of Tuned Transformer-based Models for PCL Detection. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), 344–351. https://doi.org/10.18653/v1/2022.semeval-1.44 ↗
  44. Read, J., & Carroll, J. (2012). Annotating expressions of Appraisal in English. Language Resources and Evaluation, 46(3), 421–447. https://doi.org/10.1007/s10579-010-9135-7 ↗
  45. Römer, U., & O’Donnell, M. B. (2011). From student hard drive to web corpus (part 1): The design, compilation and genre classification of the Michigan Corpus of Upper-level Student Papers (MICUSP). Corpora, 6(2), 159–177. ↗
  46. Schiller, B., Daxenberger, J., & Gurevych, I. (2021). Stance Detection Benchmark: How Robust is Your Stance Detection? KI - Künstliche Intelligenz, 35(3), 329–341. https://doi.org/10.1007/s13218-021-00714-w ↗
  47. Schmuhl, E., Miranda, L., Kádár, Á., Van Landeghem, S., & Boyd, A. (2022, June 14). Spancat: A new approach for span labeling · Explosion. Explosion. https://explosion.ai/blog/spancat ↗
  48. Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11), 2673–2681. https://doi.org/10.1109/78.650093 ↗
  49. Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook on automated essay evaluation: Current applications and new directions. Routledge. find
  50. Sparks, J. R., Song, Y., Brantley, W., & Liu, O. L. (2014). Assessing Written Communication in Higher Education: Review and Recommendations for Next-Generation Assessment. ETS Research Report Series, 2014(2), 1–52. https://doi.org/10.1002/ets2.12035 ↗
  51. Wang, X., & Wang, Y. (2022). Sentence-Level Resampling for Named Entity Recognition. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2151–2165. https://doi.org/10.18653/v1/2022.naacl-main.156 ↗
  52. Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., & Franchini, M. (2013). Ontonotes release 5.0. Linguistic Data Consortium. ↗
  53. White, P. R. R. (2003). Beyond modality and hedging: A dialogic view of the language of intersubjective stance. Text - Interdisciplinary Journal for the Study of Discourse, 23(2). https://doi.org/10.1515/text.2003.011 ↗
  54. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., … Rush, A. M. (2020). HuggingFace’s Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771). arXiv. http://arxiv.org/abs/1910.03771 ↗
  55. Wu, S. M. (2007). The use of engagement resources in high- and low-rated undergraduate geography essays. Journal of English for Academic Purposes, 6(3), 254–271. https://doi.org/10.1016/j.jeap.2007.09.006 ↗
  56. Xie, J. (2020). A review of research on authorial evaluation in English academic writing: A methodological perspective. Journal of English for Academic Purposes, 47, 100892. https://doi.org/10.1016/j.jeap.2020.100892 ↗
  57. Xu, X., & Nesi, H. (2017). An analysis of the evaluation contexts in academic discourse. Functional Linguistics, 4(1). https://doi.org/10.1186/s40554-016-0037-x ↗
  58. Xu, Y. (2020). SECOND LANGUAGE WRITING COMPLEXITY IN ACADEMIC LEGAL DISCOURSE: DEVELOPMENT AND ASSESSMENT UNDER A CURRICULAR LENS [PhD Dissertation]. Georgetown University. find
  59. Yannakoudakis, H., Briscoe, T., & Medlock, B. (2011). A New Dataset and Method for Automatically Grading ESOL Texts. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 180–189. https://aclanthology.org/P11-1019 ↗
  60. Yoon, H.-J. (2017). Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 66, 130– 141. https://doi.org/10.1016/j.system.2017.03.007 ↗
  61. Zhu, Q., Lin, Z., Zhang, Y., Sun, J., Li, X., Lin, Q., Dang, Y., & Xu, R. (2021). HITSZ-HLT at SemEval-2021 Task 5: Ensemble Sequence Labeling and Span Boundary Detection for Toxic Span Detection. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), 521–526. https://doi.org/10.18653/v1/2021.semeval-1.63 DENY: ENTERTAIN: MONOGLOSS: PROCLAIM: ENDOPHORIC: JUSTIFYING: CITATION: SOURCES: COUNTER: sum ATTRIBUTION: Train Dev Fold 1 Test Train Dev Fold 2 Test ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eguchi-2023-span-identification-epistemic-stance-taking/