Elamur Research⌕
☰
home / papers / 2026 / Under the Surface: Probing Tamil Paraphrase Intelligence

Under the Surface: Probing Tamil Paraphrase Intelligence

Summary

Tamil has rich morphology and agglutinative word formation, yet it has very little annotated data for deciding whether two sentences mean the same thing. To fill this gap, the authors built a unified Tamil paraphrase benchmark by translating QQP, PAWS and MRPC with NLLB-200 1.3B, which they used in both translation directions. They then checked the translations in several ways:

  • LaBSE similarity scores,
  • COMET scoring,
  • round-trip back-translation,
  • label agreement from a RoBERTa paraphrase classifier,
  • length-based filtering and manual correction,
  • human review of a random sample.

On this benchmark they fine-tuned five multilingual encoders and TLMR, their own compact 6-layer DeBERTa model trained on Tamil text. They also probed [CLS] embeddings with shallow classifiers. A separate efficiency score combined masked-LM accuracy, vocabulary use and Tamil script purity.

MuRIL gave the best paraphrase detection results. TLMR stayed close behind despite its small size and topped the efficiency metric. The work provides a reusable resource and evaluation protocol for semantic tasks in Tamil and similar low-resource languages.

Key points

  • Source data: about 187.2K English pairs before translation (QQP 116K, PAWS 65.4K, MRPC 5.8K). Quality was assessed on a 75,000-pair automatic subset and a 6,000-pair human subset.
  • Compared models: mBERT, XLM-R, IndicBERT, MuRIL, DistilmBERT and TLMR (TLM-DeBERTa). TLMR was pretrained on 525M Tamil tokens with a Tamil-oriented BPE tokenizer.
  • Fine-tuning results (152,464 pairs, F1):
    • MuRIL 86.03%
    • TLMR 85.00%
    • XLM-R 84.42%
    • IndicBERT 83.72%
    • DistilmBERT 75.44%
  • Embedding-based classifiers (Logistic Regression, SVM, XGBoost; trained on 53,666 pairs): these kept most of the fine-tuned performance. MuRIL peaked at 86.53 F1 with Logistic Regression, and TLMR reached 85.90 F1 with SVM.
  • Efficiency score: TLMR scored highest at 0.9504, with 94.80% Tamil purity and 62% top-5 MLM accuracy. MuRIL scored 0.6727, and DistilmBERT was close to zero.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.15

@inproceedings{r2026under,
  title = {{Under the Surface: Probing Tamil Paraphrase Intelligence}},
  author = {R, Viswadarshan R and Lilian, Dr. J. Felicia and S, Mahalakshmi},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.15},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.15}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sawsan Alqahtani, Vivek Gupta, and Hinrich Schütze. 2026. Tokenization matters: The impact of tokenization on language modeling. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 6423–6438. Association for Computational Linguistics. find
  2. Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394. ↗ our card
  3. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186. find
  6. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402– 12426, Toronto, Canada. Association for Computational Linguistics. ↗ our card
  7. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  8. Fuli Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei-Cheng Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891. find our card
  9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Deberta-v3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗ our card
  10. Simran Khanuja, Sandipan Dandapat, Divyanshu Kakwani, Naveen Arivazhagan, Pushpak Bhattacharyya, Mitesh Khapra, and Irshad Bhat. 2021. Muril: Multilingual representations for indian languages. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 620–633. ↗ our card
  11. Prakash Luitel, Rohan Sharma, and Abhinav Mishra. 2025. Perplexity is not enough: Evaluating language models beyond uncertainty metrics. arXiv preprint arXiv:2502.04561. ↗
  12. Byung-Jun Oh, Sangmin Lee, and Jihwan Choi. 2025. The impact of token granularity on language model performance. arXiv preprint arXiv:2501.01234. ↗
  13. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825–2830. ↗
  14. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Pro-253 10 ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001. find our card
  15. Ricardo Rei, Ana Farinha, Alon Lavie, and André F. T. Martins Silva. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. find our card
  16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (NeurIPS). find
  17. Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. 2019. Natural language understanding with the quora question pairs dataset. ↗
  18. R. Thangarajan, S. R. Balasundaram, and M. Aramud-han. 2016. Tamil paraphrase detection using machine learning approaches. In Working Notes of FIRE 2016 - Forum for Information Retrieval Evaluation. find
  19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 7th International Conference on Learning Representations (ICLR). ↗ our card
  20. Yichong Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1298–1308. 254 11 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/r-2026-under-surface-probing-tamil-paraphrase/