Elamur Research⌕
☰
home / papers / 2026 / Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

Summary

Neural MT systems often depend on punctuation to work out sentence structure. User-written text frequently lacks commas or uses them wrongly, which can yield Marathi output that reads smoothly but means something else. The authors study this weakness for English→Marathi. They built Virām, a small hand-curated diagnostic set. Each item pairs an ambiguous unpunctuated English sentence with its intended English reading and a Marathi translation of that reading. The authors then test two fixes. The first is a pipeline that restores English punctuation before translating, using token-classification or text-to-text restorers. The second fine-tunes IndicTrans2 directly on data with and without punctuation. Both fixes clearly improve robustness over the unmodified model. Qualitatively, direct fine-tuning gives the most stable translations, while the pipeline sometimes carries restoration errors into the output. Prompted LLMs perform worse than both task-specific methods, though few-shot prompting helps. On standard benchmarks the robust models score slightly lower. This matters for low-resource MT applied to noisy real-world input.

Key points

  • Virām benchmark: 54 punctuation-ambiguous items in the format English (Written) – English (Meant) – Marathi (Meant). The English sentences come from Kirkman (2006), and two native Marathi speakers translated them. The benchmark is released on GitHub (KaustubhShejole/Viram_Marathi).
  • Restore-then-translate: the token-classification restorers are bert-large-uncased and microsoft-mpnet-base. For text-to-text restoration, google-t5-base was fine-tuned, and AI4Bharat's Cadence was used without fine-tuning.
  • Direct fine-tuning: IndicTrans2 (en-indic-dist-200M) was fine-tuned on an in-house professionally translated corpus in four variants:
    • With Punct
    • Without Punct
    • Combined 2x
    • Combined x (punctuation alternately kept or removed per sentence)
  • LLMs tested: Sarvam-2b-v0.5, Gemma-2-9b and LLaMA-3.1-8b, each with zero-shot and three-shot prompting, in both restore-then-translate and direct modes. All lagged behind the task-specific approaches.
  • Metrics: the authors note that BLEU and chrF barely reflect punctuation-driven meaning shifts. Typical baseline failures were wrong clause attachment in headlines, conditions merged into actions in instructions, and coordination misread as disjunction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loresmt-1.13

@inproceedings{shejole2026assessing,
  title = {{Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation}},
  author = {Shejole, Kaustubh Shivshankar and Deoghare, Sourabh and Bhattacharyya, Pushpak},
  booktitle = {Proceedings for the Ninth Workshop on Technologies for Machine Translation of Low Resource Languages (LoResMT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loresmt-1.13},
  url = {https://doi.org/10.18653/v1/2026.loresmt-1.13}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We thank the anonymous reviewers and the meta-reviewer of LoResMT 2026 for their constructive comments and suggestions, which substantially improved this work. The first author gratefully acknowledges Shalaka Thorat for insightful discussions and significant assistance in developing the evaluation pipelines. We also thank seniors at CFILT, Satyam Kumar and Dhara Gorasiya, for their helpful feedback on the manuscript. We are grateful to the senior linguists at the CFILT Lab, Dr. Nilesh Joshi and Dr. Irawati Kulkarni, for their valuable remarks. Finally, the first author thanks his friends for their continued support and motivation. find
  2. CUP Varnith Chordia. 2021. Punktuator: A multilingual punctuation restoration system for spoken and written text. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 312–320. find
  3. Raj Dabre, Mary Dabre, and Teresa Pereira. 2024. Machine translation of Marathi dialects: A case study of kadodi. In Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024), pages 36–44, Miami, Florida, USA. Association for Computational Linguistics. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  5. Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and 1 others. 2023. Indicmt eval: A dataset to meta-evaluate machine translation metrics for indian languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14210–14228. find
  6. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 878–891. find our card
  7. Saurabh Sampatrao Gaikwad, Tharindu Ranasinghe, Marcos Zampieri, and Christopher Homan. 2021. Cross-lingual offensive language identification for low resource languages: The case of marathi. In Proceedings of the international conference on recent advances in natural language processing (RANLP 2021), pages 437–443. find
  8. Jay Gala, Pranjal A Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, and 1 others. 2023. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. arXiv preprint arXiv:2305.16307. ↗ our card
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  10. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, and 1 others. 2021. MuRIL: Multilingual representations for indian languages. arXiv preprint arXiv:2103.10730. ↗ our card
  11. John Kirkman. 2006. Punctuation matters: Advice on punctuation for scientific and technical writing. Routledge. find
  12. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kenton Murray, and 10 others. 2025. Findings of the WMT25 general machine translation shared task: Time to stop evaluating on easy test sets. In Proceedings of the Tenth Conference on Machine Translation, pages 355–413, Suzhou, China. Association for Computational Linguistics. find our card
  13. Pawan Lahoti, Namita Mittal, and Girdhari Singh. 2022. A survey on nlp resources, tools, and techniques for marathi language processing. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(2):1–34. ↗
  14. Vasile Păiş and Dan Tufiş. 2022. Capitalization and punctuation restoration: a survey. Artificial Intelligence Review, 55(3):1681–1722. ↗
  15. Partha Pakray, Reddi Krishna, Santanu Pal, Advaitha Vetagiri, Sandeep Dash, Arnab Kumar Maji, Saralin A. Lyngdoh, Lenin Laitonjam, Anupam Jamatia, Koj Sambyo, Ajit Das, and Riyanka Manna. 2025. Findings of WMT 2025 shared task on low-resource Indic languages translation. In Proceedings of the Tenth Conference on Machine Translation, pages 532–553, Suzhou, China. Association for Computational Linguistics. find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  17. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. find our card
  18. Sidharth Pulipaka, Sparsh Jain, Ashwin Sankar, and Raj Dabre. 2025. Mark my words: A robust multilingual model for punctuation in text and speech transcripts. arXiv preprint arXiv:2506.03793. ↗
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  20. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. find our card
  21. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892. find
  22. Thibault Sellam, Amy Pu, Hyung Won Chung, and 1 others. 2020b. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921–927. find our card
  23. Mogahed M Mogahed. 2012. Punctuation marks make a difference in translation: Practical examples. Online Submission. find
  24. Anushka Singh, Ananya Sai, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. How good is zero-shot mt evaluation for low resource indian languages? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 640–649. ↗
  25. Attila Nagy, Bence Bial, and Judit Ács. 2021. Automatic punctuation restoration with bert models. arXiv preprint arXiv:2101.07343. ↗
  26. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in Noah Lukeman. 2011. The art of punctuation. OUP Oxford. neural information processing systems, 33:16857– 16867. ↗ our card
  27. Ottokar Tilk and Tanel Alumäe. 2015. Lstm for punctuation restoration in speech transcripts. In Interspeech, pages 683–687. find
  28. Ottokar Tilk and Tanel Alumäe. 2016. Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In Interspeech, volume 3, page 9. find
  29. Robert Lawrence Trask. 2019. The Penguin guide to punctuation. Penguin UK. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/shejole-2026-assessing-improving-punctuation-robustness-english/