Elamur Research⌕
☰
home / papers / 2014 / ArCADE: An Arabic Corpus of Auditory Dictation Errors

ArCADE: An Arabic Corpus of Auditory Dictation Errors

Paper · 2014 · C. Anton Rytting, Paul Rodrigues, Tim Buckwalter et al.

Summary

Learner corpora for Arabic come almost entirely from written essays. Spelling slips that come from mishearing are rare in such data and hard to tell apart from other errors. Generic spell checkers also handle these errors poorly, because they are tuned to native-speaker mistakes. The authors built ArCADE, a dictation-based corpus that targets listening errors directly. A native speaker recorded 261 Modern Standard Arabic words, chosen to be unfamiliar to intermediate learners. Sixty-two native English speakers with at least two semesters of Arabic transcribed them by hand with full diacritics. The words were selected so that every non-glide consonant appears in six positional contexts. Each consonant is labeled as a target (recoverable only from the sound) or a non-target (guessable from morphology). The corpus has an orthographic layer and a phonemic layer that codes each target consonant as a match, substitution, deletion, gemination or degemination. Roughly 28% of target consonant responses are wrong. The resource can support spell correction for dictionary lookup, listening training and research on L2 phonology.

Key points

  • 75 people were recruited from six universities, and 62 remain in the corpus. All are native English speakers who had studied Arabic for a mean of 5.6 semesters (median 4); heritage speakers were excluded.
  • The stimuli are 261 words containing 649 target consonant instances. The contexts are C_V, V_C, V_V, #V, V# and C_#. Textbook vocabulary (Al-Kitaab, Alif-Baa), loanwords, famous place names and dialect-only words were left out.
  • The phonemic layer has 41,121 target consonant records: 29,634 matches (72.1%) and 11,487 errors (27.9%). There are 112 missing responses, and insertions are not annotated in this layer.
  • At the word level, 8,321 of 16,217 words (48.2%) contain at least one target consonant error, and 5,969 (37.1%) are spelled perfectly, ignoring diacritics.
  • The release includes the audio stimuli and about 16K handwritten responses, which could also serve OCR work on L2 script. Known limitations are the lack of a native-listener control group and the reliance on self-reported proficiency only.

Interesting ideas

  • I1. Score only what is unpredictable. The authors mark which units can be recovered only from the signal and compute errors on those, so prior knowledge does not inflate the measured skill.
  • I2. Cover a full grid with few items. The stimulus list was designed so every consonant appears in every legal context, and words carrying several targets at once were preferred to keep the test short.
  • I3. Exclude familiar items on purpose. Familiar and guessable words were removed so that participants could not rely on memorized spellings.
  • I4. Pair a known-intent dataset with a noisy log. The dictation data has a known intended target for every response, which makes it a clean complement to query logs where intent must be guessed.
  • I5. Model errors as separable layers. A phonological error layer is meant to be stacked with a composition error layer in a larger spell checker, rather than collapsing all noise into one model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets could tag the items a model could get right from priors alone, and score parsers or readers separately on the items that truly need the input, as a built-in negative control.
  • I2 (answer): School-test and fable QA suites could be built as a compact grid of constructions × contexts, picking sentences that exercise several frame types at once, so coverage gaps are visible.
  • I3 (tree): Parser evaluation could include a slice of sentences with deliberately rare or unseen lemmas, separating real syntactic generalization from lexical memorization.
  • I4 (learn): Small gold sets with known targets could be used to calibrate the silver data agreed by witness parsers, measuring where agreement fails to match the truth.
  • I5 (tokens): Input normalization could keep separate, explainable noise channels (typing slips, sound-alike confusions, learner morphology) so each correction can be traced to its cause.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🎓University of CincinnatiCincinnati · United States5 works · 38,342 citations

Also on the byline: Independent University, Bangladesh.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as C. Anton Rytting, Paul Rodrigues, Tim Buckwalter, Valerie Novak, Aric Bills, Noah H. Silbert and Mohini Madgavkar. 2014. ArCADE: An Arabic Corpus of Auditory Dictation Errors. Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1813

@inproceedings{rytting2014arcade,
  title = {{ArCADE: An Arabic Corpus of Auditory Dictation Errors}},
  author = {C. Anton Rytting, Paul Rodrigues, Tim Buckwalter, Valerie Novak, Aric Bills, Noah H. Silbert and Mohini Madgavkar},
  booktitle = {Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2014},
  doi = {10.3115/v1/w14-1813},
  url = {https://doi.org/10.3115/v1/w14-1813}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abu al-Rub. 2007. ﺔﯿﺑﺎﺘﻜﻟا ءﺎﻄﺧﻷا ﻞﯿﻠﲢ .ﺎﻫﲑﻐﺑ ﲔﻘﻃﺎﻨﻟا ﺔﯿﺑﺮﻌﻟا ﺔﻐﻠﻟا ﻲﻤﻠﻌﺘﻣ ىلد ءﻼﻣﻹا ىﻮﺘـﺴﻣ ﲆﻋ Tah ̣līl al-akht ā’ ̣ al-kitābīyah ‘ala mustawá al-imlā’ ladá muta‘allimī al-lughah al-‘arabīyah al-nāt iqīna bi-ghayrihā [Analysis of written ̣ spelling errors among non-native speaking learners of Arabic]. ﺔﯿﻧﺎﺴﻧﻹا مﻮﻠﻌﻟا ،تﺎﺳارد .ﺔﯿﻋتماﺟالاو Dirāsāt, al-‘Ulūm al-Insānīyah wa-al-Ijtimā‘īyah [Humanities and Social Sciences], 34(2). http://journals.ju.edu.jo/ ↗
  2. DirasatHum/article/view/1911/1898. find
  3. Ghazi Abuhakema, Anna Feldman, and Eileen Fitzpatrick. 2008. Annotating an Arabic learner corpus for error. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2008). Marrakech, Morocco. find
  4. Ghazi Abuhakema, Anna Feldman, and Eileen Fitzpatrick. 2009. ARIDA: An Arabic interlanguage database and its applications: A pilot study. Journal of the National Council of Less Commonly Taught Languages (NCOLCTL), 7:161–184. find
  5. Abdullah Alfaifi and Eric Atwell. 2013. Potential uses of the Arabic Learner Corpus. In Leeds Language, Linguistics, and Translation PGR Conference 2013. University of Leeds, Leeds, UK. ↗
  6. Yves Bestgen and Sylvaine Granger. 2011. Categorising spelling errors to assess L2 writing. International Journal of Continuing Engineering Education and Life-Long Learning, 21(2/3):235–252. ↗
  7. Ann Bradlow. 2008. Training non-native language sound patterns. In Phonology and Second Language Acquisition, Benjamins, Amsterdam and Philadelphia, pages 287–308. find
  8. Kristin Brustad, Mahmoud Al-Batal, and Abbas Al-Tonsi. 2004a. Al-Kitaab fii Ta‘allum al-‘Arabiyya, volume 1. Georgetown University Press, Washington, DC, 1st edition. find
  9. Kristin Brustad, Mahmoud Al-Batal, and Abbas Al-Tonsi. 2004b. Alif Baa: Introduction to Arabic Letters and Sounds. Georgetown University Press, Washington, DC, 2nd edition. find
  10. Laura Rose Faircloth. 2013. The L2 Perception of Phonemic Distinctions in Arabic by English Speakers. BA Thesis, The College of William and Mary. https://digitalarchive.wm.edu/bitstream/handle/10288/18160/ FairclothLauraRose2013Thesis.pdf? sequence=1. ↗
  11. Faisal Farooq, Liana Lorigo, and Venu Govindaraju. 2006. On the accent in handwriting of individuals. In Tenth International Workshop on Frontiers in Handwriting Recognition. La Baule, France. http://hal.inria.fr/docs/00/11/26/30/PDF/cr103741695994.pdf. ↗
  12. Samira Farwaneh and Mohammed Tamimi. 2012. Arabic learners written corpus: A resource for research and learning. Available from the University of Arizona Center for Educational Resources in Culture, Language, and Literacy web site. http://l2arabiccorpus.cercll.arizona.edu/?q=homepage. ↗
  13. John Field. 2008. Listening in the Language Classroom. Cambridge University Press, Cambridge, UK. ↗
  14. DJ Hovermale. 2011. Erron: A Phrase-Based Machine Traslation Approach to Customized Spelling Correction. Ph.D. thesis, The Ohio State University. find
  15. Khaled Yahya Huthaily. 2008. Second Language Instruction with Phonological Knowledge: Teaching Arabic to Speakers of English. Ph.D. thesis, The University of Montana. Col. find
  16. Stephen A. LaRocca and Rajaa Chouairi. 2002. West Point Arabic speech corpus. Technical report, LDC, Philadelphia. find
  17. Erica B. Michael, Greg Colflesh, Valerie Karuzis, Michael Key, Svetlana Cook, Noah H. Silbert, Christopher Green, Evelyn Browne, C. Anton Rytting, Eric Pelzl, and Michael Bunting. 2013. Perceptual training for second language speech perception: Validation study to assess the efficacy of a new training regimen (TTO 2013). Technical report, University of Maryland Center for Advanced Study of Language, College Park, MD. find
  18. Roger Mitton and Takeshi Okada. 2007. The adaptation of an English spellchecker for Japanese writers. Birbeck ePrints, London. http://eprints.bbk.ac.uk/archive/00000592. ↗
  19. Ryo Nagata, Edward Whittaker, and Vera Sheinman. 2011. Creating a manually error-tagged and shallow-parsed learner corpus. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Portland, OR, pages 1210–1219. find
  20. Nadja Nesselhauf. 2004. Learner corpora and their potential in language teaching. In How to Use Corpora in Language Teaching, Benjamins, Amsterdam and Philadelphia, pages 125–152. find
  21. Takeshi Okada. 2005. Spelling errors made by Japanese EFL writers: with reference to errors occurring at the word-initial and word-final positions. In Vivian Cook and Benedetta Bassetti, editors, Second language writing systems, Multilingual Matters, Clevedon, UK, pages 164– 183. find
  22. Peter Prince. 2012. Writing it down: Issues relating to the use of restitution tasks in listening comprehension. TESOL Journal, 3(1):65–86. ↗
  23. Chetan Ramaiah, Arti Shivram, and Venu Govindaraju. 2013. A Baysian framework for modeling accents in handwriting. In 12th International Conference on Document Analysis and Recognition (ICDAR). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6628752. ↗
  24. C. Anton Rytting, Paul Rodrigues, Tim Buckwalter, David M. Zajic, Bridget Hirsch, Jeff Carnes, Nathanael Lynn, Sarah Wayland, Chris Taylor, Jason White, Charles Blake, Evelyn Browne, Corey Miller, and Tristan Purvis. 2010. Error correction for Arabic dictionary lookup. In Seventh International Conference on Language Resources and Evaluation (LREC 2010). Valletta, Malta.
  25. Abhinav Sethy, Shrikanth Narayanan, Nicolaus Mote, and W. Lewis Johnson. 2005. Modeling and automating detection of errors in Arabic language learner speech. In INTERSPEECH-2005. pages 177–180. Noah H. Silbert, C. Anton Rytting, Paul Rodrigues, Tim Buckwalter, Valerie Novak, Mohini Madgavkar, Katharine Burk, and Aric Bills. in preparation. Similarity and bias in non-native Arabic consonant perception. find
  26. Mark Warschauer and Paige Ware. 2006. Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2):157–180. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rytting-2014-arcade-arabic-corpus-auditory-dictation/