Elamur Research⌕
☰
home / papers / 2017 / Toward a Web-based Speech Corpus for Algerian Dialectal Arabic Varieties

Toward a Web-based Speech Corpus for Algerian Dialectal Arabic Varieties

Paper · 2017 · Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari et al.

Summary

Algerian Arabic dialects are under-resourced for speech technology. Recording speakers directly is slow and expensive, and the few existing Arabic dialect speech corpora are mostly paid and usually compare dialects across countries rather than within one. The authors propose a general, reusable procedure for building large speech corpora from online material: YouTube, other social media, and online radio and TV. They apply it to create KALAM'DZ, a corpus of Algerian Arabic sub-dialects. Videos are linked to regions and sub-dialects. The audio is then extracted, converted to a standard format, cleaned of non-speech, and split by speaker through automatic diarization. Utterances receive speaker and sub-dialect labels. The first release covers 8 major sub-dialects, 4881 speakers and about 104.4 hours of speech, cut into utterances of at least 6 seconds. It was assembled in under two months. The corpus is aimed mainly at dialect and speaker identification within a single country. It shows that web harvesting can quickly produce a sizeable resource for low-resource language varieties, although the authors leave a formal evaluation to future work.

Key points

  • KALAM'DZ covers 8 Algerian Arabic sub-dialects: Hilali-Saharan, Hilali-Tellian, High-plains, Ma'qilian, Sulaymite, Algiers Blanks, Sahel-Tell and Pre-Hilālı̄. It contains 4881 speakers and roughly 104.4 hours of speech.
  • About 39% of the speech is judged clean. The rest has background music or outdoor noise but is kept to represent within-language variability.
  • The processing pipeline has three main steps:
    • FFmpeg extracts the audio, and SoX converts it to mono 16 kHz WAV.
    • A VAD step removes non-speech.
    • VoiceID, built on the LIUM Speaker Diarization framework, assigns a speaker ID to each utterance.
  • Coverage is uneven across sub-dialects and topics:
    • Hilali-Saharan is the largest sub-dialect at about 29 h, while Pre-Hilālı̄ has under 1 h.
    • Content is tagged with YouTube-style topic categories such as News & Politics, Education, Entertainment and Humor.
  • The corpus is packaged in the TIMIT layout, and a sample is released at github.com/LIM-MoDos/KalamDZ. The related-work review sorts dialect corpora along two axes: collection method and intra- versus inter-country scope.

Interesting ideas

  • I1. Region-level knowledge about a source (which departments a channel serves) is pushed down as a weak label onto every utterance cut from it, so large data is labeled without per-item annotation.
  • I2. Noisy material is retained rather than discarded, on the view that it captures real variability and can still feed robust models while a quality flag marks it.
  • I3. The authors report per-variety and per-topic hour tables, which openly show where coverage is thin and where a metropolitan web presence skews the data.
  • I4. Releasing the corpus in an established community layout (TIMIT) lowers the cost of adoption by existing tools.
  • I5. Classifying prior resources along two independent axes (how data was gathered, and whether varieties are close or distant) clarifies which gap a new resource fills.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data could inherit coarse labels from document provenance (genre, register, source site), which witness parsers then refine or veto.
  • I2 (tokens): Keep noisy web English (typos, informal spelling) in tagger training with an explicit quality tag, so the tokenizer learns robustness without contaminating clean evaluation sets.
  • I3 (learn): Evaluation reports could break scores down by genre/source cell and flag empty or tiny cells, guarding against conclusions driven by one overrepresented domain.
  • I4 (tree): Converted or silver treebanks should ship as standard CoNLL-U with UD conventions, so external parsers and validators can consume them directly.
  • I5 (tree): When converting UD dialects or picking treebanks, tagging each by annotation method and by closeness to target English variety helps choose training mixes deliberately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Amar Telidji University of LaghouatLaghouat · Algeria8 works · 121 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari and Hadda Cherroun. 2017. Toward a Web-based Speech Corpus for Algerian Dialectal Arabic Varieties. Proceedings of the Third Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1317

@inproceedings{bougrine2017toward,
  title = {{Toward a Web-based Speech Corpus for Algerian Dialectal Arabic Varieties}},
  author = {Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari and Hadda Cherroun},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1317},
  url = {https://doi.org/10.18653/v1/w17-1317}
}

References

25 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacques Leclerc. 30 avril 2012. Algérie dans l’aménagement linguistique dans le monde. find
  2. Mansour Alghamdi, Fayez Alhargan, Mohammed Alkanhal, Ashraf Alkhairy, Munir Eldesouki, and Ammar Alenazi. 2008. Saudi Accented Arabic Voice Bank. Journal of King Saud University-Computer and Information Sciences, 20:45–64. ↗
  3. M. Paul Lewis, F. Simons Gary, and D. Fenning Charles. 2015. Ethnologue: Languages of the World, Eighteenth edition. Web. find
  4. H. Li, B. Ma, and K. A. Lee. 2013. Spoken language recognition: From fundamentals to practice. Proceedings of the IEEE, 101(5):1136–1159, May. ↗
  5. K. Almeman, M. Lee, and A. A. Almiman. 2013. Multi Dialect Arabic Speech Parallel Corpora. In Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6, Feb. find
  6. J. Makhoul, B. Zawaydeh, F. Choi, and D. Stallard. 2005. BBN/AUB DARPA Babylon Levantine Arabic Speech and Transcripts. Linguistic Data Consortium (LDC). LDC Catalog Number LDC2005S08. find
  7. Mansour Alsulaiman, Ghulam Muhammad, Mohamed A Bencherif, Awais Mahmood, and Zulfiqar Ali. 2013. KSU Rich Arabic Speech Database. Journal of Information, 16(6). find
  8. Mohamed Abdelmageed Mansour. 2013. The Absence of Arabic Corpus Linguistics: A Call for Creating an Arabic National Corpus. International Journal of Humanities and Social Science, 3(12):81–90. find
  9. Peter Behnstedt and Manfred Woidich. 2013. Dialectology. find
  10. S. Bougrine, H. Cherroun, D. Ziadi, A. Lakhdari, and A. Chorana. 2016. Toward a Rich Arabic Speech Parallel Corpus for Algerian sub-Dialects. In LREC’16 Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools (OSACT), pages 2–10. find
  11. Philippe Marçais. 1986. Algeria. Leiden: E.J. Brill. find
  12. Sylvain Meignier and Teva Merlin. 2010. Lium spkdiarization: an open source toolkit for diarization. In in CMU SPUD Workshop. find
  13. Vineeth G. Nair. 2014. Getting Started with Beautiful Soup. Packt Publishing. find
  14. Alexandra Canavan and George Zipperlen. 1996. CALLFRIEND Egyptian Arabic LDC96S49. Philadelphia: Linguistic Data Consortium. find
  15. Heikki Palva. 2006. Dialects: classification. Encyclopedia of Arabic Language and Linguistics, 1:604– 613. find
  16. Dominique Caubet. 2000. Questionnaire de dialectologie du Maghreb (d’après les travaux de W. Marçais, M. Cohen, GS Colin, J. Cantineau, D. Cohen, Ph. Marçais, S. Lévy, etc.). Estudios de dialectología norteafricana y andalusí, EDNA, 5(2000-2001):73–90. find
  17. C. Pereira. 2011. Arabic in the North African Region. In S. Weniger, G. Khan, M. P. Streck, and J. C. E. Watson, editors, Semitic Languages. An International Handbook, pages 944–959. Berlin. find
  18. Mourad Djellab, Abderrahmane Amrouche, Ahmed Bouridane, and Noureddine Mehallegue. 2016. Algerian Modern Colloquial Arabic Speech Corpus (AMCASC): regional accents recognition within complex socio-linguistic environments. Language Resources and Evaluation, pages 1–29. ↗
  19. Luis Javier Rodríguez-Fuentes, Mikel Penagarikano, Amparo Varona, Mireia Diez, and Germán Bordel. 2016. Kalaka-3: a database for the assessment of spoken language recognition technology on youtube audios. Language Resources and Evaluation, 50(2):221–243. ↗
  20. Mohamed Embarki. 2008. Les dialectes arabes modernes: état et nouvelles perspectives pour la classification géo-sociologique. Arabica, 55(5):583–604. ↗
  21. Kees Versteegh, Mushira Eid, Alaa Elgibali, Manfred Woidich, and Andrzej Zaborski. 2006. Encyclopedia of Arabic Language and Linguistics. African Studies, 8. find
  22. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. 1993. DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM. find
  23. Samantha Wray and Ahmed Ali, 2015. Crowdsource a little to label a lot: Labeling a speech corpus of dialectal Arabic, volume 2015-January, pages 2824– 2828. International Speech and Communication Association. find
  24. Marwa Graja, Maher Jaoua, and L Hadrich-Belguith. 2010. Lexical Study of A Spoken Dialogue Corpus in Tunisian Dialect. In The International Arab Conference on Information Technology (ACIT), Benghazi, Libya. find
  25. Salima Harrat, Karima Meftouh, Mourad Abbas, Khaled-Walid Hidouci, and Kamel Smaili. 2016. An algerian dialect: Study and resources. International Journal of Advanced Computer Science and Applications-IJACSA, 7(3). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bougrine-2017-web-based-speech-corpus-algerian/