Elamur Research⌕
☰
home / papers / 2026 / Russian Generative Spelling, Punctuation and Capitalization Correction

Russian Generative Spelling, Punctuation and Capitalization Correction

Summary

Fixing typos, punctuation and letter case in Russian is difficult because the language has rich morphology and flexible word order. Prompting large LLMs for this job is expensive and depends heavily on how the instruction is worded. Such models also tend to "fix" rare words that are actually correct. The authors introduce SAGE, an openly released family of encoder-decoder models that handle spelling (SEC), punctuation (PEC) and capitalization (CEC) correction in a single pass. The model takes noisy text as input and returns the corrected version, with no instruction needed. The Russian variants are built on FRED-T5, and a Russian-English variant uses mT5. Training has two stages. First, the models are pre-trained on millions of synthetic error pairs produced by the Augmentex corruption algorithm, with two auxiliary objectives added. Second, they are fine-tuned on human-annotated multi-domain data that was newly labeled for punctuation and case. According to the authors, SAGE outperforms open and proprietary LLMs as well as existing spellcheckers on Russian benchmarks. A small distilled model comes close to GPT-4o and can run on a CPU. This makes SAGE a cheap, private and reproducible option for normalizing Russian text.

Key points

  • Models: FRED-T5 at 95M, 820M and 1.7B parameters, plus mT5-large (1.2B) for Russian-English. The best Russian model is distilled to 95M. Everything is released under the MIT license on HuggingFace (ai-forever/sage), with a web demo running on one A100 80GB GPU.
  • Pre-training data:
    • Russian: Wikipedia plus video transcripts, corrupted with Augmentex, giving about 6.6M source–target pairs.
    • mT5: an extra 6 GB from the Leipzig Corpora news section and English Wikipedia, for about 25M pairs in total.
  • Auxiliary objectives: one task identifies which kind of error is present (ETR), and the other predicts the right token to substitute (CTP). Both are trained with cross-entropy loss.
  • New annotation: the RUSpellRU and MultidomainGold splits were relabeled for punctuation and capitalization. Seven professional editors annotated about 12K samples in 211 hours, at a cost of $3,149.
  • Evaluation and model selection:
    • Russian test sets: RUSpellRU, MultidomainGold, MedSpellCheck and GitHubTypoCorpusRu, extending the earlier benchmark to cover PEC/CEC.
    • English test sets (for mT5): BEA60K and JFLEG.
    • Model choice: FRED-T5 was preferred after comparison with the decoder-only mGPT-1.3B.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2gv3b9npuo2s

@inproceedings{martynov2026russian,
  title = {{Russian Generative Spelling, Punctuation and Capitalization Correction}},
  author = {Martynov, Nikita and Astafurov, Danil and Isaeva, Ulyana and Maksimov, Ivan Vasil'yevich and Azocar, Joqsan and Kosenko, Dmitrii and Fenogenova, Alena},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2gv3b9npuo2s},
  url = {https://doi.org/10.63317/2gv3b9npuo2s}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. vLLM Issue 187 - Adding support for encoder-decoder models, like T5 or BART. 2024. Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available. 2024. vLLM Issue 7366 - [RFC]: Encoder/decoder models & feature compatibility. find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… find our card
  3. Andrey Bout, Alexander Podolskiy, Sergey Nikolenko, and Irina Piontkovskaya. 2023. Efficient grammatical error correction via multi-task training and optimized training schedule. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5800–5816, Singapore. As… find
  4. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, 49(3):643–701. find
  5. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is chatgpt’s behavior changing over time? find
  6. Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu. 2024. An empirical study on challenges for llm developers. arXiv preprint arXiv:2408.05002. — [arXiv:2408.05002](https://arxiv.org/abs/2408.05002) find
  7. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on M… find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  9. Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023. Analyzing the performance of gpt-3.5 and gpt-4 in grammatical error correction. arXiv preprint arXiv:2303.14342. find
  10. Fred J. Damerau. 1964. A technique for computer detection and correction of spelling errors. Commun. ACM, 7(3):171–176. find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  12. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. find
  13. Tao Fang, Shu Yang, Kaixin Lan, Derek F Wong, Jinpeng Hu, Lidia S Chao, and Yue Zhang. 2023. Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation. arXiv preprint arXiv:2304.01746. find
  14. Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia M… find
  15. Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  16. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. find
  17. V. I. Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10:707. find
  18. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35. find our card
  19. Renjie Liu, Yanxiang Zhang, Yun Zhu, Haicheng Sun, Yuanbo Zhang, Michael Xuelin Huang, Shanqing Cai, Lei Meng, and Shumin Zhai. 2024. Proofread: Fixes all errors with one tap. arXiv preprint arXiv:2406.04523. find
  20. Nikita Martynov, Mark Baushenko, Alexander Abramov, and Alena Fenogenova. 2023. Augmentation methods for spelling corruptions. In Proceedings of the International Conference “Dialogue”, volume 2023. find
  21. Nikita Martynov, Mark Baushenko, Anastasia Kozlova, Katerina Kolomeytseva, Aleksandr Abramov, and Alena Fenogenova. 2024. A methodology for generative spelling correction via natural spelling errors emulation across multiple domains and languages. In Findings of the Association for Computational Lin… find
  22. Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov, Igor Kiselev, and Artem Shelmanov. 2026. Vikhr: The family of open-source instruction-tuned large language models for russian. find
  23. Marina Panina, Alexey Baitin, and Irina Galinskaya. 2013. Context-independent autocorrection of query spelling errors.[avtomaticheskoe ispravlenie opechatok v poiskovykh zaprosakh bez ucheta konteksta]. In In Computational Linguistics and Intellectual Technologies: Proceedings of the International C… find
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. find
  25. Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  26. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In International Conference on Machine Learning, pages 4596–4604. PMLR. find
  27. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mgpt: Few-shot learners go multilingual. Transactions of the Association for Computational Linguistics, 12:58–79. find our card
  28. Alexey Sorokin. 2017. Spelling correction for morphologically rich language: a case study of russian. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 45–53. find
  29. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. 2023. Ul2: Unifying language learning paradigms. In International Conference on Learning Representations (ICLR 2023). find our card
  30. Mikhail Tikhomirov and Daniil Chernyshev. 2023. Impact of tokenization on llama russian adaptation. In 2023 Ivannikov Ispras Open Conference (ISPRAS), pages 163–168. IEEE. find
  31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. find
  33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language pr… find our card
  34. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648. find
  35. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  36. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  37. Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. A family of pre-trained transformer language models for… find
  38. Language Resource References CJ Bryant, Mariano Felice, and Edward Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Va… find our card
  39. Anisia Katinskaia, Maria Lebedeva, Jue Hou, and Roman Yangarber. 2022. Semi-automatically annotated learner corpus for russian. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 832–839. find
  40. Nikita Martynov, Mark Baushenko, Anastasia Kozlova, Katerina Kolomeytseva, Aleksandr Abramov, and Alena Fenogenova. 2024. A methodology for generative spelling correction via natural spelling errors emulation across multiple domains and languages. In Findings of the Association for Computational Lin… find
  41. Sai Muralidhar Jayanthi, Danish Pruthi, and Graham Neubig. 2020. Neuspell: A neural spelling correction toolkit. arXiv e-prints, pages arXiv– 2010. find
  42. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), Volume 2: Short Papers, pages 229–234, Va… find
  43. Tatiana Shavrina, Alexey Sorokin, Alexey Baitin, Irina Galinskaya, and Elena Rykunova. 2016. Spellrueval : the first competition on automatic spelling correction for russian. In Proceedings of the Annual International Conference “Dialogue”, volume 15. find
  44. Alexey Sorokin, Alexey Baytin, Irina Galinskaya, and Tatiana Shavrina. 2016. Spellrueval: The first competition on automatic spelling correction for russian. In Proceedings of the Annual International Conference “Dialogue”, volume 15. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/martynov-2026-russian-generative-spelling-punctuation-capitalization/