Elamur Research⌕
☰
home / papers / 2024 / Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers

Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers

Summary

Automatic feedback on Quranic recitation could help non-Arabic-speaking Muslims who have no access to a teacher. Such speech models, however, need large amounts of labeled learner audio that includes typical errors, and public Quranic speech data is scarce. The authors test whether volunteers will supply both halves of this resource: beginners contributing their own recitations, and skilled reciters judging them. They added a recording API to the existing NamazApp mobile app. There, learners hear a Qari, record a verse, compare the two, and may upload their attempt to get feedback. Uploaded audio is normalized and cleaned before storage. A separate platform, Quran Voice, lets annotators label clips once they pass a training quiz built from expert-labeled gold items. The outcome is about 7000 recitations from 1287 people in more than 11 non-Arabic countries, with 1166 of them labeled into six classes. Crowd accuracy was 0.77 and inter-annotator agreement was 0.63. The label aggregation algorithm matched expert judgments with an agreement of 0.89. The study shows that volunteer crowdsourcing can work for this kind of speech corpus.

Key points

  • Data collection ran through NamazApp, and annotation ran on the purpose-built Quran Voice platform. The process followed the four-stage crowdsourcing engineering plan of Sabou et al.: definition, preparation, execution, and evaluation/aggregation.
  • The audio pipeline used Pydub to convert files to WAV, mono, 16 kHz, 16-bit, and to remove silence. Noisereduce removed background noise.
  • There are six labels: Correct, Has mistakes, and four invalid-audio subtypes (Empty/Not related, Different Verse, Multiple Verses, Incomplete Verse). Tajweed rules are deliberately ignored, and only pronunciation of the diacritics is judged.
  • Experts annotated 62 gold tasks. The training session draws 8 questions from them, covering every label.
  • Results: crowd accuracy 0.77, Krippendorff's alpha 0.63, and algorithm-vs-expert agreement 0.89 (accuracy 0.91).

Interesting ideas

  • I1. Data is gathered as a side effect of a tool that already helps the user (recitation practice with feedback), so contributors get something back for sharing. — For Mova (learn): Self-play or student tools could log user corrections of Mova's answers as optional, consented training signal, rather than relying on separate annotation campaigns.
  • I2. Annotators must pass a short qualification quiz before real work, drawn from a small expert gold set that covers every label at least once. — For Mova (learn): Each LLM teacher tier or witness parser could be scored on a hand-checked mini-set covering all relation types before its votes count toward silver data.
  • I3. The label scheme splits "is this input even the expected item?" from "is it done correctly?". The authors suggest separating these into two annotation stages to raise agreement. — For Mova (answer): Readers could first check that a test item or word problem is well-formed and matches the expected task, and only then grade or answer, with each verdict explained separately.
  • I4. Quality is reported from three angles: accuracy against gold, chance-corrected agreement among annotators (Krippendorff's alpha), and agreement of the aggregation algorithm with experts. — For Mova (learn): Silver treebank evaluation could report witness-vs-gold accuracy, inter-witness alpha, and voted-output-vs-gold agreement, to show whether voting truly adds reliability.
  • I5. Every input is forced into one canonical format and cleaned of noise before it is stored, so all later stages see consistent data. — For Mova (tokens): A single normalization pass (Unicode, quotes, whitespace, stray markup) before tokenization would keep lemmas and tags stable across corpora with different sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.02675 · arXiv 2405.02675

@misc{salameh2024quranic,
  title = {{Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers}},
  author = {Salameh, Raghad and Mdfaa, Mohamad Al and Askarbekuly, Nursultan and Mazzara, Manuel},
  year = {2024},
  doi = {10.48550/arxiv.2405.02675},
  eprint = {2405.02675},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.02675}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Ayyoub, Mahmoud, Nour Alhuda Damer, and Ismail Hmeidi. ”Using deep learning for automatically determining correct application of basic quranic recitation rules.” Int. Arab J. Inf. Technol. 15, no. 3A (2018): 620-625. find
  2. Al-Bakeri, Ahmed AbdulQader, and Abdullah Ahmad Basuhail. ”Asr for tajweed rules: Integrated with self-learning environments.” International Journal of Information Engineering and Electronic Business 9, no. 6 (2017): 1. find
  3. Bettayeb, Nadjla, and Mhania Guerti. ”Speech synthesis system for the holy quran recitation.” Int. Arab J. Inf. Technol. 18, no. 1 (2021): 8-15. find
  4. Estellés-Arolas, Enrique, Raúl Navarro-Giner, and Fernando González-Ladrón-de-Guevara. ”Crowdsourcing fundamentals: definition and typology.” Advances in crowdsourcing (2015): 33-48. find
  5. Chowdhury, Shammur Absar, Hamdy Mubarak, Ahmed Abdelali, Soon-gyo Jung, Bernard J. Jansen, and Joni Salminen. ”A multi-platform Arabic news comment dataset for offensive language detection.” In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6203-6212. 2020. find our card
  6. Farra, Noura, Kathleen McKeown, and Nizar Habash. ”Annotating targets of opinions in Arabic using crowdsourcing.” In Proceedings of the second workshop on Arabic natural language processing, pp. 89-98. 2015. find
  7. Hakouz, Waed, Abdurrahman Ghanem, Samantha Wray, and Ahmed Ali. ”LAHAJET: a Game for Classifiying Dialectal Arabic Speech.” ASRU, 2015. find
  8. Kassem, Lin, Caroline Sabty, Nada Sharaf, Menna Bakry, and Slim Abdennadher. ”tashkeelwap: A game with a purpose for digitizing arabic diacritics.” (2016). find
  9. Bougrine, Soumia, Aicha Chorana, Abdallah Lakhdari, and Hadda Cherroun. ”Toward a Web-based speech corpus for Algerian dialectal Arabic varieties.” In Proceedings of the Third Arabic Natural Language Processing Workshop, pp. 138-146. 2017. find
  10. Bougrine, Soumia, Hadda Cherroun, and Ahmed Abdelali. ”Altruistic crowdsourcing for arabic speech corpus annotation.” Procedia Computer Science 117 (2017): 137-144. find
  11. Zaghouani, Wajdi, and Kais Dukes. ”Can crowdsourcing be used for effective annotation of Arabic?.” In LREC, pp. 224-228. 2014. find
  12. Ardila, Rosana, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. ”Common voice: A massively-multilingual speech corpus.” arXiv preprint arXiv:1912.06670 (2019). ↗ our card
  13. El-Sakhawy, Dayna, Slim Abdennadher, and Injy Hamed. ”Collecting data for automatic speech recognition systems in dialectal arabic using games with a purpose.” In Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction: Second International Workshop, MA3HMI 2014, Held in Conjunction with INTERSPEECH 2014, Singapore, Singapore, September 14, 2014, Revised Selected Papers 2, pp. 99-108. Springer International Publishing, 2015. find
  14. Wray, Samantha, and Ahmed Ali. ”Crowdsource a little to label a lot: Labeling a speech corpus of dialectal Arabic.” In Sixteenth Annual Conference of the International Speech Communication Association. 2015. find
  15. Abubakar Abid, Ali Abid, Ali Abdalla and others. ”Tarteel Initiative White Paper”. SCRIBD, https://www.scribd.com/document/413485204/Tarteel-White-Paper. 2019. ↗
  16. Osman, Hanaa Mohammed, Ban Sharief Mustafa, and Yusra Faisal. ”QDAT: a data set for reciting the Quran.” International Journal on Islamic Applications in Computer Science And Technology 9, no. 1 (2021): 1-9. find
  17. Alobaylani, Ahmed Abdullah, Mohammad Tanvir Parvez, and Suliman A. Alsuhibany. ”A Database for Speech Processing Based Qur’anic Recitation Correction.” In 2020 International Conference on Computing and Information Technology (ICCIT-1441), pp. 1-5. IEEE, 2020. find
  18. Sabou, Marta, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. ”Corpus annotation through crowdsourcing: Towards best practice guidelines.” In LREC, pp. 859-866. 2014. find
  19. Sainburg, Tim, Marvin Thielk, and Timothy Q. Gentner. ”Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires.” PLoS computational biology 16.10 (2020): e1008228. find
  20. Krippendorff, Klaus. ”Computing Krippendorff’s alpha-reliability.” (2011). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salameh-2024-quranic-audio-dataset-crowdsourced-labeled/