Elamur Research⌕
☰
home / papers / 2019 / Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data

Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data

Summary

The paper studies part-of-speech tagging of text in which speakers mix two languages or dialects inside one sentence (intra-sentential code switching). Annotated CS data is scarce and monolingual taggers fail on it. The authors therefore ask how much pretrained embeddings built from cheap raw text can help. They test four language pairs: MSA-EGY, MSA-LEV, SPA-ENG and HIN-ENG. All four use the Universal POS tag set. They compare two architectures: a BiLSTM-CRF tagger and a multi-task model (MTL-POS Tagger) that predicts POS tags and language IDs jointly. Each is paired with several FastText embedding setups: monolingual, bilingual, a merged bilingual pseudo-CS corpus, and a multilingual "pivot" embedding. Closely related varieties (the Arabic pairs) gain most from multilingual embeddings combined with joint learning. For distant pairs, shared embeddings add noise, and the multi-task POS + language ID model helps more. The systems beat the earlier MSA-EGY state of the art in every configuration and roughly match Soto and Hirschberg (2018) on SPA-ENG. This matters because it shows how to choose embeddings and model design by how close the mixed languages are.

Key points

  • Four CS evaluation sets differ sharply in switch density. CS points make up about 6% of SPA-ENG, 15.17% of HIN-ENG, 30.12% of MSA-LEV and 38.78% of MSA-EGY.

  • FastText was chosen because its character n-gram vectors can represent misspelled or unseen social-media words and suit morphologically rich Arabic.

  • Multilingual embeddings lowered OOV rates:

    • MSA-EGY: 10% → 8%
    • MSA-LEV: 13% → 10%
    • SPA-ENG: 12% → 9%
    • HIN-ENG: 15% → 11%

  • On SPA-ENG, the BiLSTM-CRF and MTL models with Merged Bilingual CFM embeddings scored 95.40% and 95.42%, the second-best results. The authors link this to the corpus's low switch rate.

  • Average CS fragment length:

    • SPA-ENG: 2.16
    • MSA-LEV: 5.1
    • MSA-EGY: 5.8
    • HIN-ENG: 6.1

    Sentences with only one or two inserted foreign words were misclassified by almost every model.

Interesting ideas

  • I1. Subword (character n-gram) embeddings give noisy, misspelled or rare word forms a usable vector instead of an unknown token, which helps on informal text. — For Mova (tokens): the en tagger and lemmatizer could fall back on n-gram-composed vectors for typos and unseen forms in fables or student answers instead of tagging them blindly.
  • I2. Predicting an auxiliary per-token label (language ID) alongside POS in a single model acted as useful extra supervision when the data mixes distinct sources. — For Mova (tokens): the tagger could jointly predict a register or "foreign/quoted word" flag, so that loanwords and names in English text stop confusing POS decisions.
  • I3. Slicing test results by a structural property (switch-point density, length of the mixed fragment) explained accuracy jumps that a single overall score would hide. — For Mova (learn): evaluation could report parser and tagger accuracy bucketed by sentence traits such as length of embedded quotes or rare-construction spans, which exposes where silver data or repair rules fail.
  • I4. A word's sense distribution shifts between corpora (English "bank" in Spanish-dominant text mostly means riverside), so embeddings trained on mismatched text carry the wrong sense priors. — For Mova (world): sense priors used when linking words to WordNet/ontology entries could be conditioned on the text's genre (fables vs. news) rather than taken from one global corpus.
  • I5. OOV rate before and after a resource change served as a cheap, interpretable diagnostic of how much the new resource actually covers. — For Mova (learn): tracking lexicon/OOV coverage per test set when adding treebanks or knowledge sources gives a quick sanity check alongside accuracy numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-1410

@inproceedings{alghamdi2019leveraging,
  title = {{Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data}},
  author = {AlGhamdi, Fahad and Diab, Mona},
  booktitle = {Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2019},
  doi = {10.18653/v1/w19-1410},
  url = {https://doi.org/10.18653/v1/w19-1410}
}

References

48 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 937–947. find
  2. Phil Blunsom and Karl Moritz Hermann. 2014. Multilingual models for compositional distributional semantics. find
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  4. Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2018. Named entity recognition on code-switched data: Overview of the calcs 2018 shared task. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 138– 147. Association for Computational Linguistics. find
  5. Mónica Stella Cárdenas-Claros and Neny Isharyanti. 2009. Code-switching and code-mixing in internet chatting: Betweenyes,’yes’ , ’ya’ , and ’si’ a case study. The Jalt Call Journal, 5(3):67–78. find
  6. Khyathi Chandu, Thomas Manzini, Sumeet Singh, and Alan W. Black. 2018. Language informed modeling of code-switched text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 92–97. Association for Computational Linguistics. find
  7. Mohamed Al-Badrashiny, Heba Elfardy, and Mona T. Diab. 2015. Aida2: A hybrid approach for token and sentence level dialect identification in arabic. In Proceedings of the 19th Conference on Computational Natural Language Learning, CoNLL 2015, Beijing, China, July 30-31, 2015, pages 42– 51. ACL. find
  8. Hongshen Chen, Yue Zhang, and Qun Liu. 2016. Neural network for heterogeneous annotations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 731– 741. find
  9. Fahad AlGhamdi and Mona Diab. 2018. Wasa: A web application for sequence annotation. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association. find
  10. François Chollet et al. 2015. Keras. https://github.com/fchollet/keras. ↗
  11. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. ↗
  12. Fahad AlGhamdi, Giovanni Molina, Mona T. Diab, Thamar Solorio, Abdelati Hawwari, Victor Soto, and Julia Hirschberg. 2016. Part of speech tagging for code switched data. In Proceedings of Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97. Association for Computational Linguistics. find
  13. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  14. Brenda Danet and Susan C Herring. 2007. The multilingual Internet: Language, culture, and communication online. Oxford University Press. find
  15. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015. Machine translation experiments on padic: A parallel Arabic dialect corpus. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, pages 26–34. find
  16. Mona Diab, Mahmoud Ghoneim, Fahad AlGhamdi, Nada AlMarwani, and Mohamed Al-Badrashiny. 2016. Creating a large multi-layered representational repository of linguistic code switched arabic data. In 10th International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia. European Language Resources Association (ELRA). find
  17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. ↗
  18. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. COLABA: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66–74. find
  19. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. ↗
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  21. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
  22. Ramy Eskander, Nizar Habash, Owen Rambow, and Arfath Pasha. 2016. Creating resources for dialectal Arabic from a single annotation: A case study on egyptian and levantine. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3455– 3465. find
  23. The COLING 2016 Organizing Committee. find
  24. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49. Association for Computational Linguistics. find
  25. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462–471. find our card
  26. Pieter Muysken, Carmen Pena Dı́az, Pieter Cornelis Muysken, et al. 2000. Bilingual speech: A typology of code-mixing, volume 11. Cambridge University Press. find
  27. Anupam Jamatia, Björn Gambäck, and Amitava Das. 2015. Part-of-speech tagging for code-mixed English-Hindi Twitter and Facebook chat messages. RECENT ADVANCES IN, page 239. find
  28. Martha Palmer, Rajesh Bhatt, Bhuvana Narasimhan, Owen Rambow, Dipti Misra Sharma, and Fei Xia. 2009. Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In The 7th International Conference on Natural Language Processing, pages 14–17. find
  29. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the Palestinian Arabic dialect. Language Resources and Evaluation, 51(3):745–775. find
  30. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic Gigaword fifth edition LDC2011t11. Philadelphia: Linguistic Data Consortium. find
  31. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn Arabic treebank: Building a large-scale annotated Arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466–467. Cairo. find
  32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  33. Mohamed Maamouri, Ann Bies, Seth Kulick, Dalila Tabessi, and Sondos Krouna. 2012. Egyptian Arabic Treebank pilot. find
  34. Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2011. A Universal Part-of-Speech Tagset. CoRR, abs/1104.2086. (EMNLP), pages 974-979, Doha, Qatar. Association for Computational Linguistics. ↗
  35. Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. 2018. Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067–3072. Association for Computational Linguistics. find
  36. Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014b. Pos tagging of English-Hindi code-mixed social media content. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 974–979. Association for Computational Linguistics. find our card
  37. Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 338–348, Copenhagen, Denmark. find our card
  38. Omar F Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics. find our card
  39. Kalika Bali Royal Sequiera, Monojit Choudhury. 2015. POS tagging of Hindi-English Code Mixed Text from Social Media: Some Machine Learning Experiments. In Proceedings of International Conference on NLP. NLPAI. find
  40. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. arXiv preprint arXiv:1603.06598. ↗
  41. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A Gold Standard Dependency Corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  42. Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393–1398. find our card
  43. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Gohneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview of the first shared task on language identification in code-switched data. In First Workshop on Computational Approaches to Code Switching, EMNLP-2014, Doha, Qatar. Association for Computational Linguistics. find
  44. Thamar Solorio and Yang Liu. 2008. Part-of-speech tagging for English-Spanish code-switched text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1051– 1060. Association for Computational Linguistics. find
  45. Victor Soto and Julia Hirschberg. 2017. Crowdsourcing Universal Part-Of-Speech tags for Code-Switching. In Interspeech 2017, pages 77–81. find
  46. Victor Soto and Julia Hirschberg. 2018. Joint part-of-speech and language id tagging for code-switched data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–10. Association for Computational Linguistics. find
  47. Ivan Vulić and Marie-Francine Moens. 2016. Bilingual distributed word representations from document-aligned comparable data. Journal of Artificial Intelligence Research, 55:953–994. find
  48. Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014a. Pos tagging of English-Hindi code-mixed social media content. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2019-leveraging-pretrained-word-embeddings-part/