Elamur Research⌕
☰
home / papers / 2018 / Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Summary

Code-mixed (CM) text is hard to model because real mixed data is scarce and switch points are rare even inside CM corpora. The authors address this by creating artificial CM sentences under the Equivalence Constraint (EC) theory of Poplack and Sankoff. Their generator takes English–Spanish parallel sentences with word alignments and parses the English side with the Stanford Parser. It then projects that tree onto the Spanish sentence, collapsing subtrees whose structures disagree. Finally, it enumerates mixed variants whose switch points satisfy the EC. The resulting generated CM data (gCM) is combined with monolingual data (Mono) and a small amount of real CM data (rCM) to train an RNN language model. The authors vary both how gCM is sampled, for example to match the real switch-point distribution or by CMI, and the training curriculum. With appropriate sampling and ordering, the synthetic data lowers perplexity about as much as doubling the real CM data. Random mixing gives no benefit. This makes theory-driven augmentation a practical answer to CM data scarcity, and it indirectly supports the linguistic theory behind it.

Key points

  • The language pair is English–Spanish. Spanish parse trees are obtained by projecting Stanford Parser output through word alignments, and a single sentence pair can yield up to thousands of EC-valid variants.
  • A baseline using only Mono and rCM (model 3) needs about twice as much rCM to equal the gCM-augmented model 5(b)-ρ. The advantage from gCM fades only around 50K rCM sentences.
  • In a control experiment, gCM was replaced by randomly spliced monolingual segments matching the rCM switch-point distribution. This produced a perplexity of 1060 on Test-17, worse than the baseline.
  • Switch points remain the hardest part to model. Even the best model, 5(a)-χ, has perplexity about four times higher at switch points than over the whole test set. Short embedded-language runs (length 1) are also especially difficult.
  • The default number of sampled gCM variants per sentence pair was k = 5. Raising k to 10 (tested alongside 1 and 2) does not reliably reduce perplexity further.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1143

@inproceedings{pratapa2018language,
  title = {{Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data}},
  author = {Pratapa, Adithya and Bhat, Gayatri and Choudhury, Monojit and Sitaram, Sunayana and Dandapat, Sandipan and Bali, Kalika},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1143},
  url = {https://doi.org/10.18653/v1/p18-1143}
}

References

38 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Heike Adel, K. Kirchhoff, N. T. Vu, D. Telaar, and T. Schultz. 2014a. Combining recurrent neural networks and factored language models during decoding of code-switching speech. In INTERSPEECH, pages 1415–1419. find
  2. Heike Adel, K Kirchhoff, N T Vu, D Telaar, and T Schultz. 2014b. Comparing approaches to convert recurrent neural networks into backoff language models for efficient decoding. In INTERSPEECH, pages 651–655. find
  3. Heike Adel, N T Vu, and T Schultz. 2013. Combination of recurrent neural networks and factored language models for code-switching language modeling. In ACL (2), pages 206–211. find
  4. Ebru Arisoy, Tara N Sainath, Brian Kingsbury, and Bhuvana Ramabhadran. 2012. Deep neural network language models. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, pages 20–28. Association for Computational Lingui… find
  5. Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. 2017. Curriculum design for code-switching: Experiments with language identification and language modeling with deep neural networks. In Proc. of ICON-2017, Kolkata, India, pages 65–74. find
  6. Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In The 1st Workshop on Computational Approaches to Code Switching, EMNLP 2014. find
  7. Hedi M Belazi, Edward J Rubin, and Almeida Jacqueline Toribio. 1994. Code switching and x-bar theory: The functional head constraint. Linguistic inquiry, pages 221–237. find
  8. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155. find
  9. Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-likelihood augmented discrete generative adversarial networks. arXiv preprint arXiv:1702.07983. — [arXiv:1702.07983](https://arxiv.org/abs/1702.07983) find
  10. A.-M. DiSciullo, Pieter Muysken, and R. Singh. 1986. Government and code-mixing. Journal of Linguistics, 22:1–24. find
  11. Chris Dyer, Victor Chahuneau, and N A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of NAACL-HLT 2013, pages 644–648. Association for Computational Linguistics. find
  12. B. Gamback and A Das. 2014. On measuring the complexity of code-mixing. In Proc. of the 1st Workshop on Language Technologies for Indian Social Media (Social-India). find
  13. B. Gamback and A Das. 2016. Comparing the level of code-switching in corpora. In Proc. of the 10th International Conference on Language Resources and Evaluation (LREC). find
  14. Jan Gebhardt. 2011. Speech recognition on english-mandarin code-switching data using factored language models. find
  15. Yifan He, Yanjun Ma, Andy Way, and Josef Van Genabith. 2010. Integrating n-best smt outputs into a tm system. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 374–382. Association for Computational Linguistics. find
  16. A. K. Joshi. 1985. Processing of Sentences with Intrasentential Code Switching. In D. R. Dowty, L. Karttunen, and A. M. Zwicky, editors, Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives, pages 190–205. Cambridge University Press, Cambridge. find
  17. D Klein and CD Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st annual meeting of the association for computational linguistics. Association of Computational Linguistics. find our card
  18. Ying Li and P Fung. 2013. Improved mixed language speech recognition using asymmetric acoustic model and language model with code-switch inversion constraints. In ICASSP, pages 7368–7372. find
  19. Ying Li and P Fung. 2014. Language modeling with functional head constraint for code switching speech recognition. In EMNLP. find
  20. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. find
  21. Carol Myers-Scotton. 1993. Duelling Languages:Grammatical structure in Code-switching. Clarendon Press, Oxford. find
  22. Carol Myers-Scotton. 1995. A lexically based model of code-switching. In Lesley Milroy and Pieter Muysken, editors, One Speaker, Two Languages: Cross-disciplinary Perspectives on Code-switching, pages 233–256. Cambridge University Press, Cambridge. find
  23. Rana D. Parshad, Suman Bhowmick, Vineeta Chand, Nitu Kumari, and Neha Sinha. 2016. What is India speaking? Exploring the “Hinglish” invasion. Physica A, 449:375–389. find
  24. Shana Poplack. 1980. Sometimes Ill start a sentence in Spanish y termino en espaol. Linguistics, 18:581– 618. find
  25. Ameya Prabhu, Aditya Joshi, Manish Shrivastava, and Vasudeva Varma. 2016. Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2482–2491. find
  26. Shruti Rijhwani, R Sequiera, M Choudhury, K Bali, and C S Maddila. 2017. Estimating code-switching on Twitter with a novel generalized word-level language identification technique. In ACL. find
  27. Ronald Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8):1270–1278. find
  28. Koustav Rudra, S Rijhwani, R Begum, K Bali, M Choudhury, and N Ganguly. 2016. Understanding language preference for expression of opinion and sentiment: What do Hindi-English speakers do on Twitter? In EMNLP, pages 1131–1141. find
  29. David Sankoff. 1998. A formal production-based explanation of the facts of code-switching. Bilingualism: language and cognition, 1(01):39–50. find
  30. A. Sharma, S. Gupta, R. Motlani, P. Bansal, M. Srivastava, R. Mamidi, and D.M Sharma. 2016. Shallow parsing pipeline for hindi-english code-mixed social media text. In Proceedings of NAACL-HLT. find
  31. Sunayana Sitaram, Sai Krishna Rallabandi, Shruti Rijhwani, and Alan W Black. 2016. Experiments with cross-lingual systems for synthesis of code-mixed text. In 9th ISCA Speech Synthesis Workshop. find
  32. Thamar Solorio and Yang Liu. 2008. Part-of-speech tagging for english-spanish code-switched text. In Proc. of EMNLP. find
  33. Thamar Solorio et al. 2014. Overview for the first shared task on language identification in code-switched data. In 1st Workshop on Computational Approaches to Code Switching, EMNLP, pages 62– 72. find our card
  34. Martin Sundermeyer, Hermann Ney, and Ralf Schlüter. 2015. From feedforward to recurrent lstm neural networks for language modeling. IEEE Transactions on Audio, Speech, and Language Processing, 23(3):517–529. find
  35. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association. find
  36. Yogarshi Vyas, S Gella, J Sharma, K Bali, and M Choudhury. 2014. POS Tagging of English-Hindi Code-Mixed Social Media Content. In Proc. EMNLP, pages 974–979. find our card
  37. Robert A Wagner and Michael J Fischer. 1974. The string-to-string correction problem. Journal of the ACM (JACM), 21(1):168–173. find
  38. Fuliang Weng, H Bratt, L Neumeyer, and A Stolcke. 1997. A study of multilingual speech recognition. In EUROSPEECH, volume 1997, pages 359–362. Citeseer. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pratapa-2018-language-modeling-code-mixing-role/