Elamur Research⌕
☰
home / papers / 2026 / LLM as a Morphological Disambiguator for Belarusian: A Preliminary Study

LLM as a Morphological Disambiguator for Belarusian: A Preliminary Study

Summary

Belarusian has very little publicly available text with resolved morphological annotation. Its main UD resource, UD_Belarusian-HSE, holds about 305K tokens, and larger corpora keep ambiguous analyses. The authors test whether a general-purpose LLM can resolve this ambiguity when a rule-based tool supplies the options. In their two-stage pipeline, the YABC morphological analyzer proposes possible lemmas and tags. An LLM then selects the right ones in context, handling several paragraphs per request. Each batch is processed three times, and a majority vote decides the final choice. When evaluated on a manually annotated article of about 1000 tokens, Gemini 3 Flash Preview and Claude 4.5 Sonnet each achieved roughly 99% accuracy on lemmas and 93–94% on tags, with ChatGPT trailing behind. Gemini was then applied to a 375K-word newspaper subset of YABC, and its output was compared with Stanza trained on UD_Belarusian-HSE. Manual review of the cases where they disagreed generally favoured the LLM. This points to an affordable way of producing better silver-standard annotation for an under-resourced language, and eventually better training data for taggers.

Key points

  • Results on the gold sample (majority vote): lemma accuracy was 0.989 for Claude, 0.988 for Gemini and 0.973 for ChatGPT. Tag accuracy was 0.935, 0.944 and 0.889. Tags proved harder than lemmas for every model.
  • Large-scale run: the data were 476 articles from Čyrvonaja źmiena (2008–2009), split into 188 batches of about 2000 tokens. These were processed through the free Gemini API, and 85.5% of requests succeeded. The analyzer failed to recognize 1.37% of tokens.
  • Manual sampling (100 instances per class): when choosing among analyzer candidates, the LLM's tags were correct in 90 cases and its lemmas in 73. For tokens unknown to the analyzer, its own lemmas were correct in 91 cases.
  • Comparison with Stanza:
    • Lemma disagreements affected 6.29% of tokens; the LLM was judged better in 55 cases versus 15 for Stanza.
    • Tag disagreements affected 10.53% of tokens; the split was 68 versus 15.
    • A fine-tuned TinyBART lemmatizer fared worse, with the LLM better in 59 cases versus 4.
  • Typical problems and next steps:
    • Some tokens were corrupted in the output, e.g., transliterated or translated into English, Russian or Ukrainian.
    • The model made errors with pronouns, suppletive nouns, and case/gender agreement.
    • Planned work includes testing open-weight LLMs, annotating the whole YABC corpus, and mapping its tags to UD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3skazxbd27m8

@inproceedings{poritski2026llm,
  title = {{LLM as a Morphological Disambiguator for Belarusian: A Preliminary Study}},
  author = {Poritski, Vladislav and Volchek, Oksana and Afanasev, Ilia},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/3skazxbd27m8},
  url = {https://doi.org/10.63317/3skazxbd27m8}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ilia Afanasev and Olga Lyashevskaya. 2023. From web to dialects: how to enhance non-standard Russian lects lemmatisation? In Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 167–175, Gothenburg, Sweden. Association for Computational Linguistics. find
  2. Savita Bhat and Vasudeva Varma. 2023. Large language models as annotators: A preliminary evaluation for annotating low-resource language content. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 100–107, Bali, Indonesia. Association for Computational Linguistics. find
  3. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for… find our card
  4. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. find our card
  5. Google DeepMind. 2025. Gemini 3 Flash model card. Accessed: 2026-02-17. find
  6. Mika Hämäläinen. 2024. DAG: Dictionary-augmented generation for disambiguation of sentences in endangered Uralic languages using ChatGPT. In Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, pages 36–40, Helsinki, Finland. Association for Computational… find
  7. Nataliia Kholodna, Sahib Julka, Mohammad Khodadadi, Muhammed Nurullah Gumus, and Michael Granitzer. 2024. LLMs in the loop: Leveraging large language model annotations for active learning in low-resource languages. find
  8. Anton Korikov, Pan Du, Scott Sanner, and Navid Rekabsaz. 2025. Batched self-consistency improves LLM relevance assessment and ranking. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32687–32703, Suzhou, China. Association for Computational Linguistic… find
  9. Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2024, pages 209–215. find
  10. Yana Shishkina and Olga Lyashevskaya. 2021. Sculpting enhanced dependencies for Belarusian. In Analysis of Images, Social Networks and Texts, pages 137–147, Cham. Springer International Publishing. find
  11. Christopher D. Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Proceedings of the 12th International Conference on Computational Linguistics and Intelligent Text Processing - Volume Part I, CICLing’11, page 171–189, Berlin, Heidelberg. Springer-Verlag. find our card
  12. Chahan Vidal-Gorène, Bastien Kindt, and Florian Cafiero. 2026. Under-resourced studies of under-resourced languages: lemmatization and POS-tagging with LLM annotators for historical Armenian, Georgian, Greek and Syriac. In Proceedings of the Second Workshop on Language Models for Low-Resource Langua… find
  13. Cameron Morin and Matti Marttinen Larsson. 2025. Large corpora and large language models: a replicable method for automating grammatical annotation. Linguistics Vanguard, 11(1):501– 510. find
  14. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. find our card
  15. Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor, and Roi Reichart. 2025. Are LLMs better than reported? Detecting label errors and mitigating their effect on model performance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26782–26809, Suzho… find
  16. Lars Wissler, Mohammed Almashraee, Dagmar Monett, and Adrian Paschke. 2014. The gold standard in corpus annotation. In 5th IEEE Germany Student Conference. find
  17. Danni Yu, Luyang Li, Hang Su, and Matteo Fuoli. 2024. Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis. International Journal of Corpus Linguistics, 29(4):534–561. find
  18. Arbi Haza Nasution and Aytuğ Onan. 2024. Chat-GPT label: Comparing the quality of human-generated and LLM-generated annotations in low-resource language NLP tasks. IEEE Access, 12:71876–71900. find
  19. Judit Ács. 2025. Morphology in the Age of Pre-trained Language Models. Ph.D. dissertation, Budapest University of Technology and Economics. find
  20. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evalu… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poritski-2026-llm-as-morphological-disambiguator-belarusian/