Elamur Research⌕
☰
home / papers / 2016 / An Arabic-Moroccan Darija Code-Switched Corpus

An Arabic-Moroccan Darija Code-Switched Corpus

Paper · 2016 · Younes Samih and Wolfgang Maier

Summary

Moroccan Darija is a widely spoken Arabic dialect with very few computational resources. Online, its speakers often mix it with Modern Standard Arabic (MSA) and other languages. The authors build a token-level code-switching corpus to support language identification for this setting. They crawled Moroccan blogs and discussion forums, which gave about 15 million tokens. They then removed diacritics, added Buckwalter transliteration and stored each token in a database. Raw web text is dominated by MSA, so they used an iterative keyword bootstrapping step that starts from Darija seed words to pull out text units rich in switching. Three native speakers labelled this subset independently in a custom web tool. They used the EMNLP 2014 shared-task label set plus a new label for third languages. The resulting 223k-token corpus has code-switching in 73.9% of its text units, against roughly 20% in earlier datasets. The authors describe it as the largest manually annotated resource of its kind, useful for code-switching detection and for sociolinguistic study.

Key points

  • The raw crawl has about 15M tokens on topics such as politics, religion, sport and economics. The final annotated corpus has 223k tokens.
  • Extraction started from a seed list of 439 common Darija words and phrases. Each round queried the database with these words and then added all words of the retrieved units to the seed list.
  • The labels are lang1 (MSA), lang2 (Darija), mixed, ne, other and ambiguous, plus a new lang3 for French, English, Spanish or Berber. French may appear in Arabic script, and Berber may appear in Tifinagh.
  • Label distribution:
    • lang1: 109,025 (48.82%)
    • lang2: 76,732 (34.36%)
    • lang3: 1,383 (0.62%)
    • ne: 7.65%
    • other: 8.43%
    • mixed: 86
    • ambiguous: 141
  • Annotation was done by three native speakers working separately, two of them without linguistic training, with no crowd-sourcing. Text was kept as whole posts because no Darija sentence splitter exists.

Interesting ideas

  • I1. Grow a sample around a rare phenomenon by seeding keyword queries and feeding retrieved vocabulary back into the seed list, which raises the share of useful examples far above random sampling.
  • I2. Give annotators explicit "mixed" and "ambiguous" labels instead of forcing a single class, so uncertain or hybrid tokens are recorded honestly rather than mislabelled.
  • I3. Store every token as a pair of original surface form and normalized form, so processing works on a clean representation without losing what was actually written.
  • I4. When sentence splitting is unreliable, keep the natural text unit (the post) intact rather than introducing segmentation errors into the data.
  • I5. Fold rare classes into one catch-all label (lang3) instead of many sparse labels that would have too few examples to learn or evaluate.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data selection could bootstrap from a few trigger words to mine texts dense in rare constructions (e.g. gapping, garden paths), so witness parsers and the LLM teacher spend effort where Mova is weakest.
  • I2 (tokens): The tagger and morphology could emit an explicit "unresolved" or "hybrid" marker for words like foreign stems with English affixes, keeping the uncertainty visible and explainable downstream.
  • I3 (tokens): Keeping raw and normalized spellings side by side in crate en would let later stages and generation refer back to the exact input.
  • I4 (tree): For noisy forum-style English, gparse and the perceptron parser could accept multi-sentence units and leave segmentation as a separate, checkable decision rather than a hidden preprocessing error.
  • I5 (world): Ontology categories with very few instances could be merged under one explicit "other" node until enough data justifies splitting, keeping statistics meaningful.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany33 works · 1,487 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Younes Samih and Wolfgang Maier. 2016. An Arabic-Moroccan Darija Code-Switched Corpus. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{samih2016arabic,
  title = {{An Arabic-Moroccan Darija Code-Switched Corpus}},
  author = {Younes Samih and Wolfgang Maier},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/2j9smzi3aevg},
  url = {https://doi.org/10.63317/2j9smzi3aevg}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Shargi, F. and Rambow, O. (2015). Diwan: A dialectal word annotation tool for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 49–58, Beijing, China. Association for Computational Linguistics.
  2. Bacatan, A. C., Castillo, B. L., Majan, M. J., Palermo, V., and Sagum, R. (2014). Detection of intra-sentential code-switching points using word bigram and unigram frequency count. International Journal of Computer and Communication Engineering, 3(3):184–188. ↗
  3. Bar, K. and Dershowitz, N. (2014). The Tel Aviv University system for the code-switching workshop shared task. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 139–143, Doha, Qatar. Association for Computational Linguistics. find
  4. Benajiba, Y. and Diab, M. (2010). A web application for dialectal Arabic text annotation. In Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects, Valletta, Malta. ELRA. find
  5. Benmamoun, E. (2001). Language identities in Morocco: A historical overview. Studies in the Linguistic Sciences, 31(1):95–106. find
  6. Berk-Seligson, S. (1986). Linguistic Constraints on Intrasentential Code Switching: A Study of Spanish-Hebrew Bilingualism. Cambridge University Press. ↗
  7. Bullock, B. E. and Toribio, A. J. (2009). The Cambridge handbook of linguistic code-switching. Cambridge University Press. find
  8. Chang, J. C. and Lin, C.-C. (2014). Recurrent-neural-network for language detection on twitter code-switching corpus. arXiv preprint arXiv:1412.4314. ↗
  9. Cotterell, R., Renduchintala, A., Saphra, N., and Callison-Burch, C. (2014). An Algerian Arabic-French code-switched corpus. In LREC Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools, Reykjavik, Iceland. find
  10. Dey, A. and Fung, P. (2014). A Hindi-English code-switching corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language Resources Association (ELRA). find
  11. Diab, M., Habash, N., Rambow, O., Altantawy, M., and Benajiba, Y. (2010). COLABA: Arabic dialect annotation and processing. Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects, pages 66–74. find
  12. Mona Diab, et al., editors. (2014). Proceedings of the First Workshop on Computational Approaches to Code Switching. Doha, Qatar. find
  13. Elfardy, H. and Diab, M. (2012). Token level identification of linguistic code switching. In Proceedings of COLING 2012: Posters, pages 287–296, Mumbai, India. find
  14. Elfardy, H., Al-Badrashiny, M., and Diab, M. (2014). AIDA: Identifying code switching in informal Arabic text. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 94–101, Doha, Qatar.
  15. Elinson, A. E. (2013). Dārija and changing writing practices in Morocco. International Journal of Middle East Studies, 45(04):715–730. find
  16. Ennaji, M., Makhoukh, A., Es-saiydy, H., Moubtassime, M., and Slaoui, S. (2004). A Grammar of Moroccan Arabic. Number 25 in Pars Lettres. Faculty of Letters Dhar El Mehraz, Fès, Fès, Morocco. find
  17. Estigarribia, B. (2015). Guaran-Spanish Jopara mixing in a Paraguayan novel – does it reflect a third language, a language variety, or true codeswitching? Journal of Language Contact, 8(2):182–222. find
  18. Ferguson, C. (1959). Diglossia. Word, 15:325–340. find
  19. Habash, N. Y. (2010). Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  20. Richard Harrell et al., editors. (1966). A dictionary of Moroccan Arabic. Georgetown University Press. find
  21. Harrell, R. S. (1962). A Short Reference Grammar of Moroccan Arabic. Georgetown University Press. find
  22. King, B. and Abney, S. (2013). Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1110–1119, Atlanta, Georgia. Association for Computational Linguistics. find
  23. King, L., Baucom, E., Gilmanov, T., Kübler, S., Whyatt, D., Maier, W., and Rodrigues, P. (2014). The IUCL+ system: Word-level language identification via extended Markov models. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 102–106, Doha, Qatar. Association for Computational Linguistics. find
  24. M. Paul Lewis, et al., editors. (2014). Ethnologue: Languages of the World. SIL International, Dallas, Texas, seventeenth edition. Online version: http://www.ethnologue.com. ↗
  25. Maharjan, S., Blair, E., Bethard, S., and Solorio, T. (2015). Developing language-tagged corpora for code-switching tweets. In Proceedings of The 9th Linguistic Annotation Workshop, pages 72–84, Denver, Colorado, USA. find
  26. Muñoa Barredo, I. (2003). Pragmatic functions of code-switching among Basque-Spanish bilinguals. In Comunidades e individuos bilinges: Actas do I Simposio Internacional sobre o Bilingismo. Universidade de Vigo: Servizo de Publicacins. find
  27. Muysken, P. (2000). Bilingual speech: A typology of code-mixing, volume 11. Cambridge University Press. find
  28. Myers-Scotton, C. (1993). Common and uncommon ground: Social and structural factors in codeswitching. Language in Society, 22:475–475. ↗
  29. Myers-Scotton, C. (1997). Duelling Languages: Grammatical Structure in Codeswitching. Clarendon Press. find
  30. Preslav Nakov, et al., editors. (2014). Proceedings of the EMNLP’2014 Workshop on Language Technology for Closely Related Languages and Language Variants. Doha, Qatar. find
  31. Poplack, S. (1980). Sometimes I’ll start a sentence in Spanish y termino en español: Toward a typology of code-switching. Linguistics, 18(7-8):581–618. find
  32. Rodrigues, P. and Kübler, S. (2013). Part of speech tagging bilingual speech transcripts with intrasentential model switching. In AAAI Spring Symposium: Analyzing Microtext. find
  33. Shrestha, P. (2014). Incremental n-gram approach for language identification in code-switched text. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 133–138, Doha, Qatar. Association for Computational Linguistics. find
  34. Solorio, T. and Liu, Y. (2008). Learning to predict code-switching points. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 973–981, Honolulu, Hawaii. Association for Computational Linguistics.
  35. Solorio, T. and Liu, Y. (2008b). Part-of-speech tagging for English-Spanish code-switched text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1051–1060. Association for Computational Linguistics.
  36. Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., and Fung, P. (2014). Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar.
  37. Tratz, S., Briesch, D., Laoudi, J., and Voss, C. (2013). Tweet conversation annotation tool with a focus on an Arabic dialect, Moroccan Darija. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 135–139, Sofia, Bulgaria.
  38. Tratz, S., Briesch, D., Laoudi, J., Voss, C., and Holland, V. M. (2014). Language and dialect identification in social media analysis. Proceedings of SPIE, 9122:91220K–91220K–11. find
  39. Voss, C., Tratz, S., Laoudi, J., and Briesch, D. (2014). Finding romanized Arabic dialect in code-mixed tweets. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
  40. Marcos Zampieri, et al., editors. (2014). Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects. Dublin, Ireland. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samih-2016-arabic-moroccan-darija-code-switched/