Elamur Research⌕
☰
home / papers / 2026 / Beyond Accuracy: A Structured Error Analysis of Multilingual LLMs on Marathi Script Variation and Syntax

Beyond Accuracy: A Structured Error Analysis of Multilingual LLMs on Marathi Script Variation and Syntax

Paper · 2026 · Tejas Patil and Barnali Chetia

Summary

Multilingual LLM benchmarks rarely test Marathi beyond surface-level tasks. When they do include it, they often use machine-translated data that ignores its rich morphology, its frequent Romanized spelling, and its mixing with English. The authors built a hand-written probing set of 1,020 Marathi sentences with no machine translation. It covers three probes: Devanagari vs. Romanized script, Marathi-English code-mixing, and syntax (SOV order, vibhakti case markers, verb gender agreement, postpositions). They tested Llama-3.1-8B, Llama-3.3-70B, Mistral-7B and Qwen3-32B on translation, similarity, grammaticality and case-marker tasks, with prompts in both English and Marathi. Translation was scored with token-level F1 and BERTScore. Qwen3-32B performed best overall, but every model lost accuracy on Romanized input. Errors were systematic rather than random:

  • Hindi case markers replaced Marathi ones.
  • Negation in counterfactual clauses was dropped.
  • Romanized sentences were often translated into unrelated, invented content.

These results suggest that models do not separate Marathi from Hindi internally, and that Romanized-script users get a noticeably weaker system. The authors argue this points to a need for dedicated Marathi and Romanized training data.

Key points

  • Dataset: 1,020 sentences, split into transliteration (367), code-mixing (307) and syntax (346) probes.
    • 500 sentences are verified by native speakers; 520 still await full verification.
    • Each sentence is run under English and Marathi instructions, giving 4,080 prompt conditions.
  • Overall scores: Qwen3-32B 0.790, Llama-3.3-70B 0.640, Mistral-7B 0.624, Llama-3.1-8B 0.526. Model size alone does not explain the ranking.
  • Romanized vs. Devanagari input: performance drops by 20.5% for Llama-3.1-8B and by 7.9% for Qwen3-32B. Of the outputs with zero lexical overlap with the reference, 91.5% came from Romanized input.
  • Systematic errors across all models:
    • Hindi ko is chosen over Marathi la for the dative case.
    • The negative subjunctive nasta is lost in every model and condition.
    • On vibhakti, Mistral-7B scores only 0.467, close to chance on this binary task.
  • Further findings:
    • Qwen3-32B improves with Marathi instructions (0.873 vs. 0.709 with English instructions).
    • For Llama-3.1-8B, BERTScore (0.921) is far above token F1 (0.448). This means many of its translations are correct paraphrases rather than wrong translations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Indian Institute of Information Technology Vadodara.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksSyntax & grammar formalisms · 214 works

Cite

Cite as Tejas Patil and Barnali Chetia. 2026. Beyond Accuracy: A Structured Error Analysis of Multilingual LLMs on Marathi Script Variation and Syntax. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.11

@inproceedings{patil2026beyond,
  title = {{Beyond Accuracy: A Structured Error Analysis of Multilingual LLMs on Marathi Script Variation and Syntax}},
  author = {Tejas Patil and Barnali Chetia},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.11},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.11}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association.
  2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in N…
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arX…
  4. BigScience Workshop. 2022. BLOOM: A 176Bparameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  5. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online. Association for Computational Linguistics. find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  7. Marta R. Costa-jussà, James Cross, Onur Çelebi, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  8. Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, and Pratyush Kumar. 2022. IndicBART: A pre-trained model for Indic natural language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1849–1863, Dublin, Ireland. Association…
  9. Asang Dani and Shailesh R. Sathe. 2024. A review of the Marathi natural language processing. arXiv preprint arXiv:2412.15471. — [arXiv:2412.15471](https://arxiv.org/abs/2412.15471) find
  10. Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. find
  11. Raviraj Joshi. 2022. L3Cube-MahaCorpus and MahaBERT: Marathi monolingual corpus, Marathi BERT language models, and resources. arXiv preprint arXiv:2202.01159. find
  12. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computation…
  13. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575–3585, Online. Association for Computati…
  14. Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. AI4Bharat-IndicNLP corpus: Monolingual corpora and word embeddings for Indic languages. arXiv preprint arXiv:2005.00085. — [arXiv:2005.00085](https://arxiv.org/abs/2005.00085) find
  15. Pawan Lahoti, Namita Mittal, and Girdhari Singh. 2022. A survey on NLP resources, tools, and techniques for Marathi language processing. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(2). find
  16. Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv preprint arXiv:2307.16039.
  17. Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. IndicTrans2: Towards high-quality and accessible machine translatio…
  18. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mbert is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu…
  19. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  20. Qwen Team. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, D… — Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
  22. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6984–6996. Association for Computational Linguistics.
  23. Task Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/patil-2026-beyond-accuracy-structured-error-analysis/