Elamur Research⌕
☰
home / papers / 2014 / Overview for the First Shared Task on Language Identification in Code-Switched Data

Overview for the First Shared Task on Language Identification in Code-Switched Data

Summary

This overview describes the first shared task on token-level language identification in code-switched (CS) text, where speakers alternate between languages within a sentence. Organizers built annotated corpora for four language pairs: MSA-DA, MAN-EN, NEP-EN, and SPA-EN. The pairs differ in typology and in how closely related the two languages are. Most data came from Twitter. Each token was assigned a tag from a six-way scheme: lang1 and lang2 for the two languages, plus NE, mixed, ambiguous, and other. To check how well systems hold up when the source of text changes, a "surprise genre" test set was drawn from Facebook, blogs and forums.

Seven teams submitted 42 runs, ranked by weighted averaged F-measure. SPA-EN and NEP-EN were the easiest pairs. The closely related MSA-DA pair was by far the hardest; at the tweet level, a simple lexicon baseline was difficult to outperform. No single system won across all pairs.

The task produced the first multi-pair CS resource with a consistent label set. It also showed that CS language identification remains unsolved. Problems with deleted tweets and noisy crowdsourced labels offer practical lessons for anyone building similar datasets.

Key points

  • Four pairs were covered: Modern Standard Arabic–Dialectal Arabic, Mandarin–English, Nepali–English (Romanized only, no Devanagari) and Spanish–English. Participation was seven teams and 42 system runs.
  • The six token labels separate the two languages from non-words (punctuation, emoticons, numbers), ambiguous forms, and words that mix morphemes from both languages. Examples of mixed words include snapchateando and the MSA-DA form hayqwlwn, which joins a DA future prefix to an MSA stem.
  • SPA-EN had 11,400 training tweets. Its test set was annotated at about 3K tweets, but participants could only crawl 1,626 of them, the worst attrition of any pair. NEP-EN had 9,993 training tweets. MAN-EN was the smallest set, with 1,000 training and 313 test tweets.
  • Annotation was done on CrowdFlower, with an in-lab gold subset used for quality control. Two NEP-EN annotators agreed at 95.34% on an initial sample of 100 tweets (1,739 tokens).
  • A second evaluation operated at the document level: a binary decision on whether a text was monolingual or code-switched, reported for the surprise genre. Organizers suggested CS parsing and socio-pragmatic classification of switch points as possible future tasks.

Interesting ideas

  • I1. Choosing users instead of keywords. Organizers first found CS tweets with keyword searches, then ranked authors by their share of code-switched posts, then collected the full timelines of the heaviest switchers. This yields far more CS content than random sampling and could help in building Ukrainian–Russian or surzhyk corpora.
  • I2. Snowball through followers. The Nepali data was found by starting from a few known code-switchers and following the people who replied to them in mixed language. This works well for communities that are hard to reach with search queries.
  • I3. A separate "mixed" label for words built from morphemes of both languages. Intra-word switching is kept distinct from ambiguity and from named entities, so it can be counted and studied rather than forced into one language. This is relevant for morphologically rich languages.
  • I4. A held-out genre test. Testing on Facebook, blog and forum text, not just Twitter, shows whether a system is learning language cues or platform quirks. It is a cheap robustness check for any classifier.
  • I5. Watch the split strategy and the data platform. Splitting by user produced a heavily skewed MSA-DA test set, so a second, more balanced one was released. Undistributable Twitter data also lost tweets over time. Before committing to a source, check its license and how long the data will stay available, and review class balance after splitting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

146 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3907

@inproceedings{solorio2014overview,
  title = {{Overview for the First Shared Task on Language Identification in Code-Switched Data}},
  author = {Solorio, Thamar and Blair, Elizabeth and Maharjan, Suraj and Bethard, Steven and Diab, Mona and Ghoneim, Mahmoud and Hawwari, Abdelati and AlGhamdi, Fahad and Hirschberg, Julia and Chang, Alison and Fung, Pascale},
  booktitle = {Proceedings of the First Workshop on Computational Approaches to Code Switching},
  year = {2014},
  doi = {10.3115/v1/w14-3907},
  url = {https://doi.org/10.3115/v1/w14-3907}
}

References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kfir Bar and Nachum Dershowitz. 2014. Tel Aviv University system description for the code-switching workshop shared task. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  2. Utsab Barman, Joachim Wagner, Grzegorz Chrupala, and Jennifer Foster. 2014. DCU-UVT: Word-level language classification with code-mixed data. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  3. Gokul Chittaranjan, Yogarshi Vyas, Kalika Bali, and Monojit Choudhury. 2014. A framework to label code-mixed sentences in social media. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  4. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. AIDA: Identifying code switching in informal Arabic text. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  5. Naman Jain and Riyaz Ahmad Bhat. 2014. Language identification in codeswitching scenario. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  6. A. Joshi. 1982. Processing of sentences with intrasentential code-switching. In Ján Horecký, editor, COLING-82, pages 145–150, Prague, July. find
  7. Ben King and Steven Abney. 2013. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1110– 1119, Atlanta,… find
  8. Levi King, Eric Baucom, Tim Gilmanov, Sandra Kübler, Dan Whyatt, Wolfgang Maier, and Paul Rodrigues. 2014. The IUCL+ system: Word-level language identification via extended Markov models. find
  9. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  10. Ying Li, Yue Yu, and Pascale Fung. 2012. A Mandarin-English code-switching corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eighth International Conference on Language… find
  11. Constantine Lignos and Mitch Marcus. 2013. Toward web-scale analysis of codeswitching. In 87th Annual Meeting of the Linguistic Society of America. find
  12. Chu-Cheng Lin, Waleed Ammar, Chris Dyer, and Lori Levin. 2014. The CMU submission for the shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  13. D.C. Lyu, T.P. Tan, E. Chng, and H. Li. 2010. SEAME: a Mandarin-English code-switching speech corpus in South-East Asia. In INTERSPEECH, volume 10, pages 1986–1989. find
  14. Dong Nguyen and A. Seza Doğruöz. 2013. Word level language identification in online multilingual communication. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 857–862, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  15. Prajwol Shrestha. 2014. An incremental approach for language identification in codeswitched text. In Proceedings of the First Workshop on Computational Approaches to Code-Switching, Doha, Qatar, October. ACL. find
  16. Anil Kumar Singh and Jagadeesh Gorla. 2007. Identification of languages and encodings in a multilingual document. In Proceedings of ACL-SIGWAC’s Web As Corpus3, Belgium. find
  17. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: An annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2,… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/solorio-2014-overview-first-shared-task-language/