Elamur Research⌕
☰
home / papers / 2026 / Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation

Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation

Paper · 2026 · Hakyung Sung and Gyu-Ho Shin

Summary

Manual UD annotation of learner corpora is reliable but slow and costly. Fully automatic parsing of L2 Korean is still not accurate enough to replace it. Sung and Shin test a lightweight human-in-the-loop setup in which two parsers act as cross-checks on each other. Stanza and Trankit, both fine-tuned on the UD-KSL training data, annotated 2,208 sentences from argumentative essays by Japanese- and English-speaking learners of Korean. The comparison covered four layers: LEMMA, XPOS, HEAD and DEPREL. Tokens where the parsers matched were provisionally accepted. Mismatches went to two annotators, and a third person resolved any remaining conflicts. To check whether agreement is a valid signal of correctness, the authors also had humans annotate every sentence independently. Human judgments largely confirmed the parsers' consensus labels, which supports semi-automatic annotation for L2 Korean. The remaining disagreements were not random. Some, such as argument-role confusions, could plausibly be fixed through targeted retraining. Others, involving clause boundaries and learner-specific forms, reflect deeper representational ambiguity. The study gives a practical triage strategy for building learner treebanks more cheaply.

Key points

  • Stanza and Trankit produced matching outputs for 82% of decisions at the token level, and within those consensus cases the human annotators were in agreement with one another 93% of the time.
  • 31% of tokens needed human review on at least one feature, and 8% still required third-party adjudication.
  • Fine-tuned Trankit outperformed Stanza on the UD-KSL test set in LAS (89.13 vs. 80.36) and UAS (92.28 vs. 85.53), while Stanza was stronger on LEMMA (95.64 vs. 88.84).
  • Dependency-level disagreements clustered around argument roles, complementizer-headed complements, clause boundaries and modifier attachment.
  • XPOS mismatches mostly involved case particles (JKS, JKB, JKO, JX, etc.), noun versus root confusion (NNG/XR), tags for ill-formed forms (NF, NV, NA) and segmentation differences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States61 works · 20,107 citations🎓Rochester Institute of TechnologyRochester · United States15 works · 5,233 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Universal Dependencies · 184 works

Cite

Cite as Hakyung Sung and Gyu-Ho Shin. 2026. Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.2

@inproceedings{sung2026parser,
  title = {{Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation}},
  author = {Hakyung Sung and Gyu-Ho Shin},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.2},
  url = {https://doi.org/10.18653/v1/2026.law-main.2}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kristopher Kyle and Scott A Crossley. 2018. Measuring syntactic complexity in l2 writing using fine-grained clausal and phrasal indices. The Modern Language Journal, 102(2):333–349. find
  2. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI magazine, 35(4):105–120. find
  3. Kristopher Kyle and Masaki Eguchi. 2024. Evaluating nlp models with written and spoken l2 samples. Research Methods in Applied Linguistics, 3(2):100120. find
  4. Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016. Universal Dependencies for learner English. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  5. Kristopher Kyle, Masaki Eguchi, Aaron Miller, and Theodore Sither. 2022. A dependency treebank of spoken second language English. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 39–45, Seattle, Washington. Association for Computati…
  6. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational Linguistics, 47(2):255– 308.
  7. John Lee, Herman Leung, and Keying Li. 2017. Towards universal dependencies for learner chinese. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), volume 135 of Linköping Electronic Conference Proceedings, pages 67–71. Linköping University Electronic Press. find
  8. Elisa Di Nuovo, Cristina Bosco, Alessandro Mazzei, and Manuela Sanguinetti. 2019. Towards an italian learner treebank in universal dependencies. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), pages 151–158, Bari, Italy. CEUR Workshop Proceedings. find
  9. Sun-Hee Lee, Seok Bae Jang, and Sang-Kyu Seo. 2009. Annotation of korean learner corpora for particle error detection. Calico Journal, 26(3):529–544. find
  10. María Díez-Ortega and Kristopher Kyle. 2024. Measuring the development of lexical richness of l2 spanish: A longitudinal learner corpus study. Studies in Second Language Acquisition, 46(1):169–199. find
  11. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In International conference on intelligent text processing and computational linguistics, pages 171–189. Springer.
  12. Sylviane Granger and Yves Bestgen. 2014. The use of collocations by intermediate vs. advanced non-native writers: A bigram-based study. International Review of Applied Linguistics in Language Teaching, 52(3):229–252. find
  13. Arianna Masciolini, Aleksandrs Berdičevskis, Maria Irena Szawerna, and Elena Volodina. 2025. Annotating second language in universal dependencies: a review of current practices and directions for harmonized guidelines. In Proceedings of the Eighth Workshop on Universal Dependencies (UDW, SyntaxFest…
  14. Yuxin Hao, Xuelin Wang, Shuai Bin, Qihao Yang, and Haitao Liu. 2024. How syntactic complexity indices predict chinese l2 writing quality: An analysis of unified dependency syntactically-annotated corpus. Assessing Writing, 61:100847. find
  15. Detmar Meurers. 2015. Learner corpora and natural language processing. In Sylviane Granger, Gaëtanelle Gilquin, and Fanny Meunier, editors, The Cambridge Handbook of Learner Corpus Research, pages 537– 566. Cambridge University Press, Cambridge. find
  16. Lars Holmberg, Paul Davidsson, and Per Linde. 2020. A feature space focus in machine teaching. In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 1–2. find
  17. Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Ríos, José Bobes-Bascarán, and Ángel Fernández-Leal. 2023. Human-in-the-loop machine learning: a state of the art. Artificial Intelligence Review, 56(4):3005–3054. find
  18. Andreas Holzinger. 2016. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain informatics, 3(2):119–131. find
  19. Magali Paquot. 2019. The phraseological dimension in interlanguage complexity research. Second language research, 35(1):121–145. find
  20. Olesya Kisselev, Rossina Soyan, Dmitrii Pastushenkov, and Jason Merrill. 2022. Measuring writing development and proficiency gains using indices of lexical and syntactic complexity: Evidence from longitudinal russian learner corpus data. The Modern Language Journal, 106(4):798–817. find
  21. Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in nlp. arXiv:1608.07836 [cs]. — [arXiv:1608.07836](https://arxiv.org/abs/1608.07836) find
  22. Kristopher Kyle and Scott Crossley. 2017. Assessing syntactic sophistication in l2 writing: A usage-based approach. Language Testing, 34(4):513–535. find
  23. Emiliana Pulido, Robert Pugh, and Zoey Liu. 2025. I speak for the árboles: Developing a dependency treebank for spanish l2 and heritage speakers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 814–822. find
  24. Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 115–125, Vienna, Austria. Association for Computational Linguistics. find
  25. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082.
  26. Mihai Surdeanu and Christopher D Manning. 2010. Ensemble models for dependency parsing: cheap and good? In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 649–652. find
  27. Eric Ringger, Marc Carmen, Robbie Haertel, Kevin Seppi, Deryle Lonsdale, Peter McClanahan, James Carroll, and Noel Ellison. 2008. Assessing the costs of machine-assisted corpus annotation through a user study. In Proceedings of the Sixth International Conference on Language Resources and Evaluation… find
  28. Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System De…
  29. Alla Rozovskaya. 2024. Universal Dependencies for learner Russian. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17112–17119, Torino, Italia. ELRA and ICCL. find
  30. Elena Volodina, Arianna Masciolini, Beáta Megyesi, Julia Prentice, Lisa Rudebeck, Gunlög Sundberg, and Mats Wirén. 2025. SweLL with pride: How to put a learner corpus to good use. Huminfra Handbook (forthcoming). find
  31. Rachel Rubin, Bram Bulté, Magali Paquot, and Alex Housen. 2025. Exploring complexity at the lexis-grammar interface: Diversity and sophistication of verb-argument structures in l2 dutch writing. Journal of Second Language Writing, 67:101183. find
  32. Daniel Zeman. 2025. Corpus-based language comparison: From morphology to dependencies and beyond. find
  33. Estudos Linguísticos (São Paulo. 1978), 54(1):259– 275. find
  34. A Burr Settles. 2009. Active learning literature survey. (1648). Computer Sciences Technical Report.
  35. Gloss abbreviations Abbreviations used in interlinear glosses follow standard Leipzig conventions. find
  36. Hakyung Sung, Sooyeon Cho, and Kristopher Kyle. 2024. An empirical evaluation of lexical diversity indices in l2 korean writing assessment. Language Assessment Quarterly, 21(2):159–180. find
  37. Abbreviation ACC ADN ADV CONN COP DECL FML LOC NOM PASS PL TOP Hakyung Sung and Gyu-Ho Shin. 2024. Constructing a dependency treebank for second language learners of korean. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRE… find
  38. Hakyung Sung and Gyu-Ho Shin. 2025a. Second language korean universal dependency treebank v1. 2: Focus on data augmentation and annotation scheme refinement. In Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025), pages 13–1… find
  39. Hakyung Sung and Gyu-Ho Shin. 2025b. Towards robust morphosyntactic analysis of L2 korean: Evaluating and fine-tuning a korean language model. ACM Transactions on Asian and Low-Resource Language Information Processing, 24(4):1–20. find
  40. Hakyung Sung, Gyu-Ho Shin, Chanyoung Lee, You Kyung Sung, and Boo Kyung Jung. 2025. UD-KSL treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags. In Meaning Accusative Adnominal Adverbial Connective ending Copula Declarative Formal speech level Locative Nominativ… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sung-2026-parser-agreement-disagreement-l2-korean/