Elamur Research⌕
☰
home / papers / 2024 / From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

Paper · 2024 · Prashant Kodali, Anmol Goel, Likhith Asapu et al.

Summary

Tools that analyse or generate code-mixed text usually assume their training corpora already reflect what bilingual speakers find natural. They do not model acceptability directly. The authors address this gap with Cline, a set of 16,642 English-Hindi (en-hi) code-mixed sentences annotated by humans for acceptability. Some sentences come from synthetic code-mixing tools and the rest from social media posts. The ratings show that acceptability is graded. Popular measures for curating corpora, including CMI, burstiness and the count of switch points, have only a weak relationship with what annotators accept. The authors then benchmark several predictors. An MLP using these metrics as features falls behind fine-tuned multilingual language models. Decoder-only models do best, led by Llama 3.2 3B, and fine-tuned models also beat ChatGPT used zero-shot and few-shot. Checkpoints trained on en-hi transfer zero-shot to English-Telugu better than random. The work gives low-resource code-mixed NLP a resource and baselines for filtering synthetic or scraped data by naturalness.

Key points

  • Dataset: Cline holds 16,642 human-judged en-hi sentences, drawn from synthetic generation and online social media. The authors describe it as the largest dataset of its kind.
  • Metrics vs. humans: Standard code-mixing measures (CMI, Number of Switch Points, Burstiness) show low correlation with human acceptability ratings. This questions their use for filtering code-mixed corpora.
  • Model comparison:
    • Among encoders, XLM-Roberta and Bernice beat IndicBERT.
    • Among encoder-decoders, mBART beats mT5, but neither surpasses encoder-only models.
    • Decoder-only models score highest, with Llama 3.2 3B ahead of similarly sized Qwen and Phi models.
  • ChatGPT and transfer: Fine-tuned models outperform ChatGPT in zero-shot and few-shot settings. Zero-shot transfer from en-hi to en-te beats random baselines.
  • Released resources: The annotations, checkpoints, code-mix corpus and code are public on a Hugging Face collection (prakod) and on GitHub (prashantkodali/CodeMixAcceptability). The paper is published in ACM TALLIP (2025).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓International Institute of Information Technology, HyderabadHyderabad · India38 works · 1,821 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Prashant Kodali, Anmol Goel, Likhith Asapu, Vamshi Krishna Bonagiri, Anirudh Govil, Monojit Choudhury, Ponnurangam Kumaraguru and Manish Shrivastava. 2024. From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences. arXiv:2405.05572.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.05572 · arXiv 2405.05572

@misc{kodali2024human,
  title = {{From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences}},
  author = {Prashant Kodali, Anmol Goel, Likhith Asapu, Vamshi Krishna Bonagiri, Anirudh Govil, Monojit Choudhury, Ponnurangam Kumaraguru and Manish Shrivastava},
  year = {2024},
  doi = {10.48550/arxiv.2405.05572},
  eprint = {2405.05572},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.05572}
}

References

62 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2009. The Cambridge Handbook of Linguistic Code-switching. Cambridge University Press. find
  2. Marah Abdin and Jyoti Aneja et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL] https://arxiv.org/abs/2404.14219 ↗
  3. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1803–1813. https://aclanthology.org/2… ↗
  4. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Boua…
  5. Peter Auer. 1984. Bilingual conversation. (1984). find
  6. Anshul Bawa, Pranav Khadpe, Pratik M. Joshi, Kalika Bali, and Monojit Choudhury. 2020. Do Multilingual Users Prefer Chat-bots that Code-mix? Let’s Nudge and Find Out! Proceedings of the ACM on Human-Computer Interaction 4 (2020), 1 – 23. Manuscript submitted to ACM find
  7. Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. Universal Dependency Parsing for Hindi-English Code-Switching. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pap… find
  8. Özlem Çetinoğlu, Sarah Schulz, and Ngoc Thang Vu. 2016. Challenges of Computational Processing of Code-Switching. In Proceedings of the Second Workshop on Computational Approaches to Code Switching. Association for Computational Linguistics, Austin, Texas, 1–11. doi:10.18653/v1/W16-5801 find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  10. Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. 2022. Bernice: A Multilingual Pre-trained Encoder for Twitter. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi,… find
  11. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (Eds.). Association for Comp…
  12. R. A. Fisher. 1992. Statistical Methods for Research Workers. Springer New York, New York, NY, 66–70. doi:10.1007/978-1-4612-4380-9_6 — [doi:10.1007/978-1-4612-4380-9_6](https://doi.org/10.1007/978-1-4612-4380-9_6) find
  13. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio, et al. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational…
  14. Björn Gambäck and Amitava Das. 2016. Comparing the Level of Code-Switching in Corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). Portorož, Slovenia. find
  15. John J Gumperz. 1977. The sociolinguistic significance of conversational code-switching. RELC journal 8, 2 (1977), 1–34. find
  16. Abhirut Gupta, Aditya Vavre, and Sunita Sarawagi. 2021. Training Data Augmentation for Code-Mixed Translation. In North American Chapter of the Association for Computational Linguistics. find
  17. Gualberto Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for Modeling Code-Switching Across Corpora. In Proc. Interspeech 2017. 67–71. doi:10.21437/Interspeech.2017-1429 — [doi:10.21437/interspeech.2017-1429](https://doi.org/10.21437/interspeech.2017-1429) find
  18. Aravind K. Joshi. 1982. Processing of Sentences With Intra-Sentential Code-Switching. In Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics. https://aclanthology.org/C82-1023 ↗
  19. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. Indic-NLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. In Findings of EMNLP. find
  20. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An Evaluation Benchmark for Code-Switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online,…
  21. Prashant Kodali, Anmol Goel, Monojit Choudhury, Manish Shrivastava, and Ponnurangam Kumaraguru. 2022. SyMCoM - Syntactic Measure of Code Mixing A Study Of English-Hindi Code-Mixing. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dub… find
  22. Terry K Koo and Mae Y Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of chiropractic medicine 15 2 (2016), 155–63. find
  23. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2014. Measuring Gradience in Speakers’ Grammaticality Judgements. Cognitive Science 36 (2014). https://api.semanticscholar.org/CorpusID:11620784 ↗
  24. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2014. Measuring gradience in speakers’ grammaticality judgements. In Proceedings of the annual meeting of the cognitive science society, Vol. 36. find
  25. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive science 41 5 (2017), 1202–1241. find
  26. Amy R Lederberg and Cesáreo Morales. 1985. Code switching by bilinguals: Evidence against a third grammar. Journal of Psycholinguistic Research 14 (1985), 113–136. find
  27. Grandee Lee, Xianghu Yue, and Haizhou Li. 2019. Linguistically Motivated Parallel Data Augmentation for Code-Switch Language Modeling. In Interspeech. find
  28. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics 8 (2020), 726–742. doi:10.1162/tacl_a_00343 find
  29. Edward Ma. 2019. NLP Augmentation. https://github.com/makcedward/nlpaug. ↗
  30. Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Khapra. 2023. Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users. In Findings of the Association for Computational Linguistics: EMNLP 2… find
  31. Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and E. Artemova. 2022. RuCoLA: Russian Corpus of Linguistic Acceptability. In Conference on Empirical Methods in Natural Language Processing. https://api.semanticscholar.org/CorpusID:253097947 ↗
  32. Lesley Milroy and Matthew Gordon. 2008. Sociolinguistics: Method and interpretation. John Wiley & Sons. find
  33. Vandan Mujadia and Dipti Sharma. 2022. The LTRC Hindi-Telugu Parallel Corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 3417–3424. https://aclanthology.org/2022.lrec-1.365 ↗
  34. Pieter Muysken. 2000. Bilingual Speech: A Typology of Code-Mixing. find
  35. C. Myers-Scotton. 1997. Duelling Languages: Grammatical Structure in Codeswitching. Clarendon Press. https://books.google.co.in/books?id= NuYdnTyKkdQC ↗
  36. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How Multilingual is Multilingual BERT?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 4996–5001. doi:10.18653/v1/P19-1493
  37. Shana Poplack. 1980. Sometimes I’ll start a sentence in Spanish Y TERMINO EN ESPAÑOL: toward a typology of code-switching 1. Linguistics 18 (01 1980), 581–618. doi:10.1515/ling.1980.18.7-8.581 — [doi:10.1515/ling.1980.18.7-8.581](https://doi.org/10.1515/ling.1980.18.7-8.581) find
  38. Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data. In Annual Meeting of the Association for Computational Linguistics.
  39. Adithya Pratapa and Monojit Choudhury. 2021. Comparing Grammatical Theories of Code-Mixing. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021). Association for Computational Linguistics, Online, 158–167. doi:10.18653/v1/2021.wnut-1.18 find
  40. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantad… — Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
  41. Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, and Sunayana Sitaram. 2021. GCM: A Toolkit for Generating Synthetic Code-mixed Text. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. Assoc… find
  42. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
  43. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 2699–2712. doi:10.18653/v1/2020.acl-main.240
  44. Carson T. Schütze. 2016. The empirical base of linguistics. Number 2 in Classics in Linguistics. Language Science Press, Berlin. doi:10.17169/langsci. b89.100 — [doi:10.17169/langsci](https://doi.org/10.17169/langsci) find
  45. Anne-Marie Di Sciullo, Pieter Muysken, and Rajendra Singh. 1986. Government and code-mixing. Journal of Linguistics 22 (1986), 1 – 24. find
  46. Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin 86 2 (1979), 420–8. find
  47. Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W Black. 2020. A Survey of Code-switched Speech and Language Processing. arXiv:1904.00784 [cs.CL] — [arXiv:1904.00784](https://arxiv.org/abs/1904.00784) find
  48. Jon Sprouse. 2007. Continuous Acceptability, Categorical Grammaticality, and Experimental Syntax. Biolinguistics (2007). find
  49. Vivek Srivastava and Mayank Singh. 2021. Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text. In Proceedings of the 14th International Conference on Natural Language Generation. Association for Computational Linguistics, Aberdeen, Scotland, UK, 314–319. https://ac… find
  50. Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm.github.io/blog/qwen2.5/ ↗
  51. Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. 2024. Improving Pretraining Data Using Perplexity Correlations. arXiv:2409.05816 [cs.CL] https://arxiv.org/abs/2409.05816 ↗
  52. Daniela Trotta, Raffaele Guarasci, Elisa Leonardelli, and Sara Tonelli. 2021. Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, Dominican… find
  53. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from Disagreement: A Survey. J. Artif. Int. Res. 72 (Jan. 2022), 1385–1470. doi:10.1613/jair.1.12752 — [doi:10.1613/jair.1.12752](https://doi.org/10.1613/jair.1.12752) find
  54. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics 8 (2020), 377–392. doi:10.1162/tacl_a_00321
  55. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics 7 (2018), 625–641. find
  56. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics 7 (2019), 625–641. doi:10.1162/tacl_a_00290 find
  57. Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences. In Conference on Computational Natural Language Learning. Manuscript submitted to ACM find
  58. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  59. Ka Wong and Praveen Paritosh. 2022. k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Dublin, Ireland,… find
  60. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A Benchmark for Chinese Language Model Evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguisti…
  61. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  62. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kodali-2024-human-judgements-predictive-models-unravelling/