Elamur Research⌕
☰
home / papers / 2016 / Overview for the Second Shared Task on Language Identification in Code-Switched Data

Overview for the Second Shared Task on Language Identification in Code-Switched Data

Paper · 2016 · Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim et al.

Summary

This overview describes the second edition of a shared task on token-level language identification in code-switched (CS) social media text. It covers two language pairs collected from Twitter: Modern Standard Arabic–Dialectal Arabic (MSA-DA) and Spanish-English (SPA-ENG). Participants assigned each token one of eight labels: lang1, lang2, ne, other, ambiguous, mixed, plus two new categories split off from "other": fw (a third language) and unk (unintelligible). For SPA-ENG, the organizers reused the 2014 corpora as training and development data, relabeled them, and fixed tokenization. They built a new test set by tracing bilingual users around Spanish-language radio accounts in Miami and New York, then annotated it with SVM pre-annotation, regex rules, CrowdFlower and in-lab checks. Nine teams took part. The top systems reached a weighted Avg-F of 0.973 on SPA-ENG but only 0.876 on MSA-DA. This confirms that closely related varieties are harder to separate. Overall results improved on the first edition, which suggests real progress on a problem that matters for downstream tasks such as tagging, parsing and MT.

Key points

  • Participation: nine teams took part. All of them submitted SPA-ENG systems and four handled MSA-DA. The previous edition had seven teams.
  • Data sizes:
    • MSA-DA: 8,862 train, 1,117 dev and 1,262 test (1,258 used).
    • SPA-ENG: 8,733 train, 1,857 dev and 18,237 test (10,716 used for ranking).
  • Best systems:
    • On SPA-ENG, Shirvani et al. (2016) led with Avg-F 0.973, against a baseline of 0.828.
    • On MSA-DA, Samih et al. (2016) led with Avg-F 0.876, against a baseline of 0.463.
    • The same team won at both the token and tweet level within each language pair.
  • Annotation cost: labeling the new SPA-ENG test data cost about $1,100, roughly $0.02 per tweet. The data came from 61,943 tweets by 21 users.
  • Methods used:
    • Most systems relied on CRF sequence labeling.
    • Deep learning approaches appeared for the first time in this task.
    • The one rule-based entry lagged behind the machine-learning systems.
    • The rare classes (mixed, ambiguous, fw) got essentially zero F-scores from all systems.

Interesting ideas

  • I1. Pre-annotation before crowdsourcing: a model trained on earlier data labels the new data first, then humans check only the most frequent word types and propagate the fixes. This sharply cuts the amount of costly crowd annotation and would suit any new Ukrainian corpus built on top of an older one.
  • I2. Confidence-based re-annotation and selection: tokens below a 0.75 crowd-confidence threshold were sent back for another round. Each tweet was scored by its weakest token, and only the top-ranked tweets were kept for the test set. This is a cheap way to trade corpus size for label reliability.
  • I3. Snowball sampling for rare phenomena: to find CS tweets, the organizers started from bilingual regions and radio station accounts, then expanded through followers and users who interact with them. A similar network walk could locate Ukrainian-Russian surzhyk or mixed-language posts.
  • I4. Consistency audit by word type: the organizers listed word types that carried more than one label and reviewed them by hand. This simple check exposes systematic annotator errors, especially on named entities and homographs.
  • I5. Evaluating switch points: the authors propose scoring whether systems find the positions where the language switches, not just whether a tweet is CS or monolingual. This is a more informative metric for any mixed-language segmentation task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓George Washington UniversityWashington · United States21 works · 7,106 citations🎓University of HoustonHouston · United States7 works · 1,105 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

Cite

Cite as Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab and Thamar Solorio. 2016. Overview for the Second Shared Task on Language Identification in Code-Switched Data. Proceedings of the Second Workshop on Computational Approaches to Code Switching.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-5805

@inproceedings{molina2016overview,
  title = {{Overview for the Second Shared Task on Language Identification in Code-Switched Data}},
  author = {Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab and Thamar Solorio},
  booktitle = {Proceedings of the Second Workshop on Computational Approaches to Code Switching},
  year = {2016},
  doi = {10.18653/v1/w16-5805},
  url = {https://doi.org/10.18653/v1/w16-5805}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Al-Badrashiny and Mona Diab. 2016. A system for the code-switching workshop shared task 2016. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  2. Mohamed Al-Badrashiny, Heba Elfardy, and Mona Diab. 2015. Aida2: A hybrid approach for token and sentence level dialect identification in arabic. find
  3. Rafiya Begum, Kalika Bali, Monojit Choudhury, Koustav Rudra, and Niloy Ganguly. 2016. Functions of code-switching in tweets: An annotation framework and some initial experiments. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard,… find
  4. Arunavha Chanda, Dipankar Das, and Chandan Mazumdar. 2016. Anonymous submission for emnlp 2016 code-switching workshop shared task: System description. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  5. Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Nada AlMarwani, and Mohamed Al-Badrashiny. 2016. Creating a large multi-layered representational repository of linguistic code switched arabic data. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi,… find
  6. Dan Garrette, Hannah Alpert-Abrams, Taylor Berg-Kirkpatrick, and Dan Klein. 2015. Unsupervised code-switching for multilingual historical document transcription. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  7. Fei Huang and Alexander Yates. 2014. Improving word alignment using linguistic code switching data. In EACL, pages 1–9. find
  8. Aaron Jaech, George Mulcaire, Mari Ostendorf, and Noah A. Smith. 2016. A neural model for language identification in code-switched tweets. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. Anupam Jamatia, Björn Gambäck, and Amitava Das. 2015. Part-of-speech taggi… find
  9. Aravind K Joshi. 1982. Processing of sentences with intra-sentential code-switching. In Proceedings of the 9th conference on Computational linguistics-Volume 1, pages 145–150. Academia Praha. Zoë Kosoff. 2014. Code-switching in egyptian arabic: A sociolinguistic analysis of twitter. Al-Arabiyya: Jo… find
  10. Sophia Yat Mei Lee and Zhongqing Wang. 2015. Emotion in code-switching texts: Corpus construction and analysis. find
  11. Dong Nguyen and Leonie Cornips. 2016. Automatic detection of intra-word code-switching. find
  12. ACL 2016, page 82. Özlem Çetinoğlu. 2016. A turkish-german code-switching corpus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proc… find
  13. Younes Samih and Wolfgang Maier. 2016. An arabic-moroccan darija code-switched corpus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Pr… find
  14. Younes Samih, Suraj Maharjan, Mohammed Attia, and Thamar Solorio. 2016. Emnlp code-switching 2016 hhu-uh-g team system. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  15. Royal Sequiera, Monojit Choudhury, and Kalika Bali. 2015. Pos tagging of hindi-english code mixed text from social media: Some machine learning experiments. In ICON 2015 Proceedings. find
  16. Arnav Sharma, Sakshi Gupta, Raveesh Motlani, Piyush Bansal, Manish Shrivastava, Radhika Mamidi, and Dipti M. Sharma. 2016. Shallow parsing pipeline -hindi-english code-mixed social media text. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Li… find
  17. Rouzbeh Shirvani, Mario Piergallini, Gauri Shankam Gautam, and Mohamed Chouikha. 2016. System submission for language identification in spanish-english codeswitching. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  18. Prajwol Shrestha. 2016. Codeswitching detection via lexical features in conditional random fields. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  19. Utpal Sikdar and Bjrn Gambck. 2016. Language identification in code-switched text using conditional random fields and babelnet. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find
  20. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the First Shared Task on Language Identification in Code-Switched Data. In Proceedings of The First Work…
  21. Meng Xuan Xia. 2016. Codeswitching language identification using subword information enriched word vectors. In Proceedings of The Second Workshop on Computational Approaches to Code Switching. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/molina-2016-overview-second-shared-task-language/