Elamur Research⌕
☰
home / papers / 2015 / Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription

Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription

Paper · 2015 · Samantha Wray, Hamdy Mubarak and Ahmed Ali

Summary

Dialectal Arabic (DA) has no fixed spelling, so standard crowdsourcing quality checks break down when collecting speech transcripts. Exact agreement between workers or exact match against expert gold answers rejects transcripts that are correct but spelled differently. The authors gathered transcriptions of Al Jazeera debate and news audio on CrowdFlower and compared four quality-control setups: no control, content-blind surface checks on the Arabic text, interleaved gold test questions, and a lenient edit-distance comparison against output from an imperfect MSA/DA ASR system. They ran these setups on Egyptian speech, with both fast and high-quality worker pools, and then repeated them on North African speech. The ASR comparison worked best in every setting. For Egyptian speech from fast workers, poor transcripts fell from 13.2% to 1.0%. For North African speech, they fell from 7.8% to 4.0%. Because the matching is lenient, natural spelling variation survives. The study offers a cheap recipe for building corpora of nonstandard language varieties without forcing their spelling into a norm.

Key points

  • Crowdsourcing on CrowdFlower cost about USD 42 and took about 18 hours per hour of speech, with 5 transcripts per item. A professional transcription house charged USD 300, took 4 days and returned a single transcript.

  • Audio preparation had three stages:

    • Voice activity detection removed non-speech.
    • LIUM SpkDiarization handled speaker segmentation.
    • Crowd workers classified each segment's dialect, with 3–9 annotators per file across MSA/EGY/LEV/NOR/GLF. Only segments with at least 75% agreement on EGY were kept for the main experiment.

  • The checking ASR system had the following setup and performance:

    • It used a DNN acoustic model with fMLLR, trained on about 400 h of Broadcast News data.
    • Decoding used a trigram language model, with 4-gram rescoring.
    • The lexicon held 1.2M words, with a 2.5% OOV rate.
    • WER was 12.35% on report speech, 29.8% on conversational speech and 25.4% combined.

  • Results on 100 EGY segments (share of poor transcripts):

    ConditionNo controlSurface checksGold questionsASR check
    High Speed workers13.2%10.0%7.4%1.0%
    High Quality workers7.6%15.0%4.6%0.4%

    The ASR check counted a transcript as acceptable at just 20% string overlap with the ASR output.

  • On 100 NOR segments with High Speed workers, the rates were 7.8% (no control), 9.4% (surface checks), 13.6% (gold questions) and 4.0% (ASR check).

Interesting ideas

  • I1. A weak automatic system can serve as a gatekeeper rather than a judge: with a deliberately low match threshold, it rejects off-topic or garbage input while tolerating legitimate variation.
  • I2. Checks that only test whether the output looks well-formed, without consulting the input, can backfire. In the High Quality pool, the surface checks raised the error rate from 7.6% to 15%.
  • I3. A single prolific but sloppy contributor can dominate a dataset if no check stops them, so error rates depend on how much each source produces, not just on how many sources are bad.
  • I4. The authors keep spelling variation in the corpus instead of normalizing it away, treating the variants as useful material for later resources such as morphological analyzers.
  • I5. The pipeline first identifies the dialect of each audio segment, then routes it to workers from the matching region, which reduces mismatched expertise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A weak parser could act as a low-threshold sanity witness when screening silver or LLM-teacher trees. It would flag only gross disagreements instead of demanding exact agreement.
  • I2 (learn): Every validator on generated data should be tied to the source sentence. Evaluations should include a negative control showing that format-only checks fail to separate good outputs from bad.
  • I3 (learn): When pooling silver data from several witness parsers or teacher tiers, cap or reweight each source's share so that one noisy, high-volume source cannot skew training.
  • I4 (tokens): Nonstandard or variant English spellings could be kept as surface forms linked to a shared lemma. The tagger would then learn from real variation instead of a pre-normalized text.
  • I5 (tree): Before UD dialect conversion or parsing, a cheap classifier could detect the genre or treebank style of the input and route it to the matching parser or conversion rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Samantha Wray, Hamdy Mubarak and Ahmed Ali. 2015. Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3211

@inproceedings{wray2015best,
  title = {{Best Practices for Crowdsourcing Dialectal Arabic Speech Transcription}},
  author = {Samantha Wray, Hamdy Mubarak and Ahmed Ali},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3211},
  url = {https://doi.org/10.18653/v1/w15-3211}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Hamdy Mubarak, and Stephan Vogel. 2014a. Advances in dialectal arabic speech recognition: A study using twitter to improve egyptian asr. In International Workshop on Spoken Language Translation (IWSLT 2014), pages http– workshop2014.
  2. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Dahak, Stephan Vogel, and Jim Glass. 2014b. A Complete Kaldi Recipe For Building Arabic Speech Recognition Systems. In Spoken Language Technology Workshop (SLT), 2014 IEEE. find
  3. Ahmed Ali, Yifan Zhang, and Stephan Vogel. 2014c. Qcri advanced transcription system (qats). In Spoken Language Technology Workshop (SLT), 2014 IEEE. find
  4. Kartik Audhkhasi, Panayiotis Georgiou, and Shrikanth S Narayanan. 2011. Accurate transcription of broadcast news speech using multiple noisy transcribers and unsupervised reliability metrics. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pages 4980–4983. IEEE. find
  5. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably Effective Arabic Dialect Identification. EMNLP-2014.
  6. Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013. Processing spontaneous orthography. In HLT-NAACL, pages 585–595. find
  7. Keelan Evanini, Derrick Higgins, and Klaus Zechner. 2010. Using Amazon Mechanical Turk for transcription of non-native speech. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 53–56. Association for Computational Linguistics.
  8. Hadrien Gelas, Solomon Teferra Abate, Laurent Besacier, and François Pellegrino. 2011. Quality assessment of crowdsourcing transcriptions for African languages. In Interspeech. find
  9. Nizar Habash, Mona T Diab, and Owen Rambow. 2012. Conventional Orthography for Dialectal Arabic. In LREC, pages 711–718.
  10. Annika Hämäläinen, Fernando Pinto Moreira, Jairo Avelar, Daniela Braga, and Miguel Sales Dias. 2013. Transcribing and annotating speech corpora for speech recognition: A three-step crowdsourcing approach with quality control. In First AAAI Conference on Human Computation and Crowdsourcing. find
  11. Chiaying Lee and James Glass. 2011. A transcription task for crowdsourcing with automatic quality control. In Interspeech, pages 3041–3044. find
  12. Matthew Marge, Satanjeev Banerjee, and Alexander I Rudnicky. 2010a. Using the Amazon Mechanical Turk for transcription of spoken language. In Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pages 5270– 5273. IEEE. find
  13. Matthew Marge, Satanjeev Banerjee, and Alexander I Rudnicky. 2010b. Using the Amazon Mechanical Turk to transcribe and annotate meeting speech for extractive summarization. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 99–107. Association for Computational Linguistics.
  14. Sylvain Meignier and Teva Merlin. 2010. Lium spkdiarization: an open source toolkit for diarization. In CMU SPUD Workshop, volume 2010. find
  15. Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 207–215, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  16. Gabriel Parent and Maxine Eskenazi. 2010. Toward better crowdsourced transcription: Transcription of a year of the let’s go bus information system data. In Spoken Language Technology Workshop (SLT), 2010 IEEE, pages 312–317. IEEE. find
  17. Rachele Sprugnoli, Giovanni Moretti, Matteo Fuoli, Diego Giuliani, Luisa Bentivogli, Emanuele Pianta, Roberto Gretter, and Fabio Brugnara. 2013. Comparing two methods for crowdsourcing speech transcription. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8116–8120. IEEE. find
  18. Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, and Stephanie Strassel. 2013. GALE Phase 2 Arabic Broadcast Conversation Speech. Linguistics Data Consortium. find
  19. Jason D Williams, I Dan Melamed, Tirso Alonso, Barbara Hollister, and Jay Wilpon. 2011. Crowdsourcing for difficult transcription of speech. In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, pages 535–540. IEEE. find
  20. Samantha Wray and Ahmed Ali. 2015. Crowdsource a little to label a lot: Labeling a Speech Corpus of Dialectal Arabic. In Interspeech. (in press). find
  21. Omar F Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1220–1229. Association for Computational Linguistics.
  22. Ines Zribi, Rahma Boujelbane, Abir Masmoudi, Mariem Ellouze, Lamia Belguith, and Nizar Habash. 2014. A Conventional Orthography for Tunisian Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wray-2015-best-practices-crowdsourcing-dialectal-arabic/