Elamur Research⌕
☰
home / papers / 2019 / A Crowdsourcing-based Approach for Speech Corpus Transcription Case of Arabic Algerian Dialects

A Crowdsourcing-based Approach for Speech Corpus Transcription Case of Arabic Algerian Dialects

Paper · 2019 · Ilyes Zine, Mohamed Cherif Zeghad, Soumia Bougrine et al.

Summary

Algerian Arabic dialects have little annotated speech data. They also have no standard spelling, which makes expert transcription slow and costly. The authors tested whether paid non-expert contributors could produce usable orthographic transcripts for KALAM'DZ, a web-collected speech corpus of Algerian dialects that until then carried only dialect and speaker labels. They took about 10% of the corpus (8.75 hours) and built a platform called SPEECH2TEXT'DZ, following the four-phase crowdsourcing engineering process of Sabou et al. (2014). Contributors received spelling guidelines, special tags and a video tutorial. Quality was controlled in three layers: screening contributors before they start, automatic filtering while work comes in, and final checking by well-trained transcribers (WTT) in the lab. Crowd output was compared with a reference slice transcribed by the WTT. Before expert revision, the crowd's volume-weighted accuracy was about 74.38%. The authors conclude that a crowd with careful quality control is a workable route for transcribing under-resourced dialects. They also offer practical advice for running such projects.

Key points

  • The source corpus is KALAM'DZ: roughly 104 hours, 4881 speakers and eight Algerian Arabic dialects, gathered from YouTube, online radio and TV. The transcribed sample has 5122 segments averaging 6.2 seconds.
  • SPEECH2TEXT'DZ launched in May 2018 and ran for 51 days. It drew more than 208 registered users, each averaging about 25.65 transcriptions. One transcription took about 3 min 21 s on average.
  • Contributors were paid modestly in points that could be redeemed as mobile phone credit.
  • Any submission finished in under 20 seconds was treated as malicious and discarded.
  • The reference set is 5% of the target data, transcribed by WTT. Precision is the mean of per-user scores weighted by each user's utterance count, giving ≈74.38%. Most users scored between 60% and 80%, and only a small group exceeded 80%.

Interesting ideas

  • I1. Annotators are scored against a small expert-made reference slice, and each score is weighted by how much that annotator produced, so prolific low-quality contributors cannot hide behind a simple average.
  • I2. A cheap effort signal, completion time below a threshold, removes junk submissions before any expensive review takes place.
  • I3. Quality control is stacked in layers: screening before work, automatic filtering during work, and expert revision after it, so each layer only handles what the earlier ones let through.
  • I4. For a language with no spelling standard, the project defined a written convention plus explicit tags for special phenomena. Error analysis then showed that many mistakes came from misunderstood guidelines or misused tags rather than from mishearing the audio.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers and LLM teacher tiers could be ranked on a small gold UD slice with a volume-weighted score, so a silver source that contributes many arcs is judged by its real impact.
  • I2 (learn): Silver annotations or teacher outputs could pass through similar inexpensive sanity checks, such as suspiciously short or trivially flat outputs, before reaching costly agreement checks.
  • I3 (learn): The silver-data pipeline could be organised the same way, with rule-based filters first, multi-witness agreement second and human or LLM review last, keeping the expensive steps small.
  • I4 (tree): When converting UD dialects or merging treebanks, disagreements could be sorted into "convention or label misuse" versus "genuine ambiguity", so conversion rules get fixed instead of the parser being retrained.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Science Applications · 131 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksSpeech & spoken language · 519 works

Cite

Cite as Ilyes Zine, Mohamed Cherif Zeghad, Soumia Bougrine and Hadda Cherroun. 2019. A Crowdsourcing-based Approach for Speech Corpus Transcription Case of Arabic Algerian Dialects. Proceedings of the 3rd International Conference on Natural Language and Speech Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zine2019crowdsourcing,
  title = {{A Crowdsourcing-based Approach for Speech Corpus Transcription Case of Arabic Algerian Dialects}},
  author = {Ilyes Zine, Mohamed Cherif Zeghad, Soumia Bougrine and Hadda Cherroun},
  booktitle = {Proceedings of the 3rd International Conference on Natural Language and Speech Processing},
  year = {2019},
  url = {https://aclanthology.org/W19-7411/}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mansour Alghamdi, Fayez Alhargan, Mohammed Alkanhal, Ashraf Alkhairy, Munir Eldesouki, and Ammar Alenazi. 2008. Saudi Accented Arabic Voice Bank. Journal of King Saud University -Computer and Information Sciences, 20:45–64. find
  2. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279–284. IEEE.
  3. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech Recognition Challenge in the Wild: Arabic MGB-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316–322. IEEE.
  4. Mohamed Embarki. 2008. Les dialectes arabes modernes : état et nouvelles perspectives pour la classification géo-sociologique. Arabica, 55(5):583–604. find
  5. Khalid Almeman, Mark Lee, and Ali Abdulrahman Almiman. 2013. Multi Dialect Arabic Speech Parallel Corpora. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. IEEE. find
  6. Hassan Gadalla, Hanaa Kilany, Howaida Arram, Ashraf Yacoub, Alaa El-Habashi, Amr Shalaby, Krisjanis Karins, Everett Rowson, Robert MacIntyre, Paul Kingsbury, David Graff, and Cynthia McLemore. 1997. CALLHOME Egyptian Arabic Transcripts. Linguistic Data Consortium, Philadelphia. find
  7. Claude Barras, Edouard Geoffrois, Zhibiao Wu, and Mark Liberman. 1998. Transcriber: a Free Tool for Segmenting, Labeling and Transcribing Speech. In First international conference on language resources and evaluation (LREC), pages 1373–1376. find
  8. Paul Boersma and Vincent Van Heuven. 2001. Speak and unSpeak with PRAAT. Glot International, 5(9/10):341–347. find
  9. Soumia Bougrine, Hadda Cherroun, and Ahmed Abdelali. 2017a. Altruistic Crowdsourcing for Arabic Speech Corpus Annotation. Procedia Computer Science, 117:137 – 144. find
  10. Soumia Bougrine, Hadda Cherroun, and Djelloul Ziadi. 2017b. Hierarchical Classification for Spoken Arabic Dialect Identification using Prosody: Case of Algerian Dialects. CoRR, abs/1703.10065. ↗
  11. Soumia Bougrine, Hadda Cherroun, Djelloul Ziadi, Abdallah Lakhdari, and Aicha Chorana. 2016. Toward a Rich Arabic Speech Parallel Corpus for Algerian sub-Dialects. In The 2nd Workshop on Arabic Corpora and Processing Tools 2016 Theme: Social Media, pages 2–10. European Language Resources Association (ELRA). find
  12. Hamilton Alexander Rosskeen Gibb, Johannes Hendrik Kramers, Évariste Lévi-Provençal, Bernard Lewis, Charles Pellat, Joseph Schacht, et al. 1986. The Encyclopaedia of Islam, new edition, volume 1, chapter Algeria. E. J. Brill, Leiden. find
  13. Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul, pages 711– 718. European Language Resources Association (ELRA).
  14. Salima Harrat, Karima Meftouh, Mourad Abbas, Khaled-Walid Hidouci, and Kamel Smaili. 2016. An Algerian dialect: Study and Resources. International journal of advanced computer science and applications (IJACSA), 7(3):384–396. find
  15. Sameer Khurana and Ahmed Ali. 2016. QCRI advanced transcription system (QATS) for the Arabic Multi-Dialect Broadcast media recognition: MGB-2 challenge. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 292–298. IEEE. find
  16. Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari, and Hadda Cherroun. 2017c. Toward a Web-based Speech Corpus for Algerian Arabic Dialectal Varieties. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 138–146. Association for Computational Linguistics. find
  17. Mohamed Maamouri, Tim Buckwalter, and Christopher Cieri. 2004. Dialectal Arabic Telephone Speech Corpus: Principles, Tool Design, and Transcription Conventions. In NEMLAR International Conference on Arabic Language Resources and Tools, Cairo, pages 22–23. Linguistic Data Consortium (LDC). find
  18. Dominique Caubet. 2000. Questionnaire de dialectologie du Maghreb (d’après les travaux de W. Marçais, M. Cohen, GS Colin, J. Cantineau, D. Cohen, Ph. Marçais, S. Lévy, etc.). Estudios de Dialectologı́a Norteafricana y Andalusı́ (EDNA), 5:73–92. find
  19. Mohamed Abdelmageed Mansour. 2013. The Absence of Arabic Corpus Linguistics: A Call for Creating an Arabic National Corpus. International Journal of Humanities and Social Science, 3(12):81–90. find
  20. Khalid Choukri, Mahtab Nikkhou, and Niklas Paulsson. 2004. Network of data centres (NetDC): BNSC - an Arabic broadcast news speech corpus. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), pages 889–892. European Language Resources Association (ELRA).
  21. European Language Resources Association (ELRA). 2005. A-SpeechDB ID: ELRA-S0315. http://catalog.elra.info/en-us/ repository/browse/ELRA-S0315/. Accessed: 2018-10-30. ↗
  22. Houda Saadane and Nizar Habash. 2015. A Conventional Orthography for Algerian Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 69–79. Association for Computational Linguistics.
  23. Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 859–866. European Language Resources Association (ELRA). find
  24. Mostafa Lameen Souag. 2006. Explorations in the Syntactic Cartography of Algerian Arabic. Ph.D. thesis, University of London, School of Oriental and African Studies, London. find
  25. James Surowiecki. 2004. The wisdom of crowds, first edition. Anchor Books, New York. find
  26. Samantha Wray, Hamdy Mubarak, and Ahmed Ali. 2015. Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 99–107. Association for Computational Linguistics.
  27. Inès Zribi, Mariem Ellouze, Lamia Hadrich Belguith, and Philippe Blache. 2015. Spoken Tunisian Arabic Corpus ”STAC”: Transcription and Annotation. Research in computing science, 90:123–135. find
  28. Inès Zribi, Marwa Graja, Mariem Ellouze Khmekhem, Maher Jaoua, and Lamia Hadrich Belguith. 2013. Orthographic Transcription for Spoken Tunisian Arabic. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 153–163. Springer-Verlag Berlin Heidelberg. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zine-2019-crowdsourcing-based-approach-speech-corpus/