Elamur Research⌕
☰
home / papers / 2025 / Enhancing the Automatic Classification of Metadiscourse in Low-Proficiency Learners' Spoken and Written English Texts Using XLNet

Enhancing the Automatic Classification of Metadiscourse in Low-Proficiency Learners' Spoken and Written English Texts Using XLNet

Paper · 2025 · Wenwen Guan, Marijn Alta and Jelke Bloem

Summary

Metadiscourse (MD) covers the expressions speakers and writers use to steer their audience through a text. Examples are transitions, frame markers, hedges, boosters and direct appeals to the listener. Labelling MD automatically is hard because the markers are ambiguous out of context and span boundaries are fuzzy. Interpersonal MD is the hardest part. Earlier SVM/CNN systems handled it poorly. The authors built an annotated sample of the ICNALE corpus containing spoken and written English, mostly from moderately proficient Asian learners. The annotation scheme rests on Ädel's (2010) function-based taxonomy, extended with categories from Hyland (2005). They framed the task as token classification and fine-tuned seven encoder models, including models chosen because they are pretrained with span-oriented objectives. They compared these against a spaCy baseline and an untuned Llama 3.3 70B-Instruct baseline. XLNet-large performed best, with accuracy 0.91 and weighted F1 0.93. Only the tuned encoders could handle every category, including the interpersonal ones. Rare categories remain weak. The work suggests that a modest annotated set is enough to bootstrap semi-automatic MD annotation, which matters for second-language research and assessment.

Key points

  • Dataset: 575,541 tokens annotated with 24 labels, sampled from ICNALE. It contains both speech and writing, from learners and native speakers.
  • Seven encoders were tuned: BERT, RoBERTa, ELECTRA, SpanBERT, ERNIE 2.0, XLNet and ModernBERT.
    • XLNet-large had the best accuracy and weighted F1.
    • ERNIE had the best macro-F1, meaning it did better on rare classes.
    • Large model variants gave only minimal gains.
  • Baselines:
    • spaCy reached a weighted F1 of 0.81, and only on the most frequent categories.
    • Few-shot Llama-3.3-70B-Instruct reached a weighted F1 of 0.41, even with examples for each category.
  • XLNet-large results by category:
    • F1 is above 0.8 for 13 of 21 categories.
    • F1 is below 0.6 for 5 of 21 categories.
    • Four infrequent categories stay below 0.5.
    • Anticipating the audience's response (AAR) improves from 0.3 in prior work to 0.5.
  • Known limitation: the scheme allows only one label per token, so nested MD spans cannot be fully represented. An example is the booster "fully" inside "I fully agree that".

Interesting ideas

  • I1. Some discourse markers change role with context: "so" links clauses causally in one sentence and is an anaphoric object in "I don't think so", so classification must look beyond the word itself.
  • I2. The taxonomy grounds categories in discourse function and in spoken language, then adds hedges, boosters and attitude markers from a writing-based scheme, so it covers both learner speech and essays.
  • I3. The authors admit that single-label token tagging cannot tell one enclosing span apart from two split spans around a nested one, which argues for layered or multi-label span representations.
  • I4. A tuned small encoder clearly beat a 70B instruction model used few-shot on a fine-grained span-labelling task. This suggests large generative teachers are unreliable for detailed taxonomies without tuning.
  • I5. When a classifier pre-annotates data, its errors bias the resulting corpus toward its weak classes, so human checking should be steered to the worst-performing categories.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Use the UD relation of the marker (mark/cc/advmod versus obj) as an explicit, explainable cue for whether a token is discourse-structuring or propositional.
  • I2 (frames): Treat hedges, boosters and epistemic-attitude markers as speaker-stance annotations on frames, so the world layer can mark a claim as tentative or emphatic rather than plain fact.
  • I3 (frames): Store discourse/stance spans as separate layered annotations over the tree instead of one tag per token, so nested markers stay unambiguous and traceable.
  • I4 (learn): Gate LLM-teacher labels for fine-grained categories behind agreement with a tuned model or parser witness, rather than accepting them as silver data directly.
  • I5 (learn): Route silver-data review effort by per-category error rates, and do targeted extra annotation only for rare, low-F1 labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Wenwen Guan, Marijn Alta and Jelke Bloem. 2025. Enhancing the Automatic Classification of Metadiscourse in Low-Proficiency Learners' Spoken and Written English Texts Using XLNet. Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.codi-1.3

@inproceedings{guan2025enhancing,
  title = {{Enhancing the Automatic Classification of Metadiscourse in Low-Proficiency Learners' Spoken and Written English Texts Using XLNet}},
  author = {Wenwen Guan, Marijn Alta and Jelke Bloem},
  booktitle = {Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.codi-1.3},
  url = {https://doi.org/10.18653/v1/2025.codi-1.3}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Annelie Ädel. 2006. Metadiscourse in L1 and L2 English. John Benjamins. ↗
  2. Annelie Ädel. 2010. Just to give you kind of a map of where we are going: A taxonomy of metadiscourse in spoken and written academic English. Nordic Journal of English Studies, 9(2):69–97. ↗
  3. Ghada Alharbi. 2016. Metadiscourse tagging in academic lectures. Ph.D. thesis, University of Sheffield. find
  4. Martin Juan José Bucher and Marco Martini. 2024. Fine-tuned ’small’ llms (still) significantly outperform zero-shot generative ai models in text classification. arXiv preprint arXiv:2406.08660. ↗
  5. Sathena Chan, Manoranjan Sathyamurthy, Chihiro Inoue, Michael Bax, Johnathan Jones, and John Oyekan. 2024. Integrating metadiscourse analysis with transformer-based models for enhancing construct representation and discourse competence assessment in L2 writing: A systemic multidisciplinary approach. Journal of Measurement and Evaluation in Education and Psychology, 15(Special Issue):318– 347. ↗
  6. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. Preprint, arXiv:2003.10555. ↗
  7. Avon Crismore, Raija Markkanen, and Margrat S. Steffensen. 1993. Metadiscourse in persuasive writing: A study of texts written by american and finnish university students. Written Communication, 10(1):39–71. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Rui Pedro dos Santos Correia. 2018. Automatic Classification of Metadiscourse. Ph.D. thesis, Language Technologies Institute, School of Computer Science, Carnegie Mellon University. find
  10. Masaki Eguchi and Kristopher Kyle. 2023. Span identification of epistemic stance-taking in academic written English. In The 61st Annual Meeting Of The Association For Computational Linguistics.
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  12. M.A.K. Halliday. 1994. An Introduction to Functional Grammar. Hodder Arnold. find
  13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. ↗
  14. Matthew Honnibal, Ines Montani, et al. 2024. Prodigy: An annotation tool for AI, Machine Learning & NLP. https://prodi.gy. ↗
  15. Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3477–3486. ↗
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276. ↗
  17. Ken Hyland. 2005. Metadiscourse: Exploring interaction in writing. Continuum. spaCy. 2024. Span categorizer: Pipeline component. https://spacy.io/api/spancategorizer. Accessed: 2024-10-14. ↗
  18. Shin’ichiro Ishikawa. 2023. The ICNALE guide: An introduction to a learner corpus study on Asian learners’ L2 English. Routledge. find
  19. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64–77.
  20. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8968– 8975.
  21. Imed Keraghel, Stanislas Morbieu, and Mohamed Nadif. 2024. A survey on recent advances in named entity recognition. arXiv preprint arXiv:2401.10825. ↗
  22. Darinka Verdonik. 2023. Annotating dialogue acts in speech data: Problematic issues and basic dialogue act categories. International Journal of Corpus Linguistics, 28(2):144–171. ↗
  23. William Vande Kopple. 1985. Some exploratory discourse on metadiscourse. College Composition & Communication, 36(1):82–93. ↗
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  25. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275.
  26. Andrea Martinenghi, Gregor Donabauer, Simona Amenta, Sathya Bursic, Mathyas Giudici, Udo Kruschwitz, Franca Garzotto, Dimitri Ognibene, et al. 2024. LLMs of Catan: Exploring pragmatic capabilities of generative chatbots through prediction and classification of dialogue acts in boardgames’ multi-party dialogues. In Proceedings of the 10th Workshop on Games and Natural Language Processing@ LREC-COLING 2024, pages 107–118. ELRA and ICCL.
  27. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547, Vienna, Austria. Association for Computational Linguistics.
  28. Anna Mauranen. 1993. Cultural differences in academic discourse–problems of a linguistic and cultural minority. AFinLAn vuosikirja, pages 157–174. find
  29. ED Milne. 2003. Metadiscourse revisited: a contrastive study of persuasive writing in professional discourse. regreso al metadiscurso: estudio contrastivo de la persuasión en el discurso profesional. Estudios ingleses de la Universidad Complutense, 11:29–52. ↗
  30. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  31. Ayesha Qamar, Jonathan Tong, and Ruihong Huang. 2025. Do LLMs understand dialogues? a case study on dialogue acts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26219– 26237.
  32. Frances Yung, Mansoor Ahmad, Merel Scholman, and Vera Demberg. 2024. Prompting implicit discourse relation annotation. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 150–165. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guan-2025-enhancing-automatic-classification-metadiscourse-low/