Elamur Research⌕
☰
home / papers / 2014 / De l'arabe standard vers l'arabe dialectal : projection de corpus et ressources linguistiques en vue du traitement automatique de l'oral dans les médias tunisiens [From Modern Standard Arabic to Tunisian dialect: corpus projection and linguistic resources towards the automatic processing of speech in the Tunisian media]

De l'arabe standard vers l'arabe dialectal : projection de corpus et ressources linguistiques en vue du traitement automatique de l'oral dans les médias tunisiens [From Modern Standard Arabic to Tunisian dialect: corpus projection and linguistic resources towards the automatic processing of speech in the Tunisian media]

Paper · 2014 · Rahma Boujelbane, Mariem Ellouze, Frédéric Béchet et al.

Summary

The paper tackles language modelling for speech recognition of Tunisian TV and radio. Speakers there constantly switch between the local Tunisian dialect (TD) and Modern Standard Arabic (MSA). TD has almost no NLP resources, while MSA tools break down on dialect input. The authors treat TD as a variant of MSA and adapt MSA resources in three phases. First, they compare MSA and TD lexical units and build bilingual MSA–TD dictionaries in which TD entries mirror the structure of MSA entries. Second, they automatically convert MSA text into a TD corpus. Third, they judge the generated corpus by the lexical coverage and perplexity of a language model trained on it. The test data are real transcripts of Tunisian broadcasts that mix both varieties. To get those transcripts, they hand-transcribed and annotated 5 h 20 min of political news and debate programmes. They used the CODA orthographic convention and kept full vowelization. The work matters because it gives a path to modelling a low-resource dialect from a high-resource related standard.

Key points

  • Transcribed corpus:
    • TV news: 1 h 42 min, 12,207 tokens, 4,504 types, 21.4 % TD words.
    • Debates: 3 h 40 min, 25,757 tokens, 6,110 types, 37.2 % TD words.
  • Transcription setup: the authors developed CODA (Zribi et al., 2014) for this work and used the Transcriber tool.
  • Four word labels:
    • MSA: a standard word with standard vowels.
    • MSA*: a standard word with dialectal vowels.
    • DT*: a standard word with dialect affixes.
    • DT: a pure dialect word.
  • Target register: the "intellectualized" dialect of educated speakers in broadcasts. The only earlier TD corpus, train-station dialogues, was judged too narrow in vocabulary and topic for media speech.
  • Evaluation: the projected corpus is measured by lexical coverage and language-model perplexity on real broadcast transcripts.

Interesting ideas

  • I1. Create data for a low-resource variety by projecting a large corpus of a related standard variety through bilingual lexicons; this is cheap when parallel data barely exists.
  • I2. Annotate mixed-variety text on a graded scale (pure standard, standard word with dialect affixes, standard word with dialect pronunciation, pure dialect) rather than a binary language tag; this shows exactly where the varieties diverge.
  • I3. Keep in the annotation the information the source makes explicit (here, the vowels heard in speech) instead of discarding it and recovering it later with a diacritizer; this removes ambiguity at its root.
  • I4. Evaluate a synthetic corpus extrinsically, by coverage and perplexity on genuine target-domain text, rather than by inspecting it directly.
  • I5. Give dialect lexicon entries the same internal representation as the standard-language entries, so existing standard tools and processing chains can be reused unchanged.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The same idea could convert existing English UD treebanks into informal or dialectal English variants, giving the parser training trees for non-standard input.
  • I2 (tokens): The tagger could carry a register/variety feature telling the lemmatizer which forms are standard words in non-standard dress.
  • I3 (tokens): When building gold or silver data, record the disambiguating cues the source provides, such as capitalization, punctuation, or explicit markers, as features rather than normalizing them away.
  • I4 (learn): Silver or generated training sets could be checked against held-out real text for vocabulary coverage and model fit before they are trusted for training.
  • I5 (world): Variant or informal lexical entries could be keyed to the same lemma and frame structure as standard entries, so compiled knowledge (verb classes, ontology links) carries over automatically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Rahma Boujelbane, Mariem Ellouze, Frédéric Béchet and Lamia Belguith. 2014. De l'arabe standard vers l'arabe dialectal : projection de corpus et ressources linguistiques en vue du traitement automatique de l'oral dans les médias tunisiens [From Modern Standard Arabic to Tunisian dialect: corpus projection and linguistic resources towards the automatic processing of speech in the Tunisian media]. Traitement Automatique des Langues.

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{boujelbane2014de,
  title = {{De l'arabe standard vers l'arabe dialectal : projection de corpus et ressources linguistiques en vue du traitement automatique de l'oral dans les médias tunisiens [From Modern Standard Arabic to Tunisian dialect: corpus projection and linguistic resources towards the automatic processing of speech in the Tunisian media]}},
  author = {Rahma Boujelbane, Mariem Ellouze, Frédéric Béchet and Lamia Belguith},
  year = {2014},
  url = {https://aclanthology.org/2014.tal-2.4/}
}

References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adda-Decker M., « A corpus-based decompounding algorithm for German lexical modeling in LVCSR. », INTERSPEECH, 2003. find
  2. Baccouche T., « Esquisse d’une étude comparative des schémas des verbes en arabe classique et en arabe tunisien », Les cahiers de Tunisie, vol. 22, p. 87-88, 1974. find
  3. Beesley K. R., « Arabic morphological analysis on the Internet », Proceedings of the 6th International Conference and Exhibition on Multi-lingual Computing, Citeseer, 1998. find
  4. Belguith L. H., Aloulou C., Hamadou A. B., « MASPAR : De la segmentation à l’analyse syntaxique de textes arabes », CÉPADUÈS-Editions, editeur, Revue Information Interaction Intelligence I, vol. 3, p. 9-36, 2007. find
  5. Besacier L., « De l’utilisation d’unités sous-lexicales pour la traduction automatique de la parole », Séminaire ATALA, 2007. find
  6. Boudlal A., Lakhouaja A., Mazroui A., Meziane A., BEBAH M. O. A. O., SHOUL M., « Alkhalil Morpho SYS1 : A Morphosyntactic Analysis System for Arabic Texts », International Arab Conference on Information Technology, 2010. find
  7. Boujelbane R., Khemekhem M. E., Belguith L. H., « Mapping Rules for Building a Tunisian Dialect Lexicon and Generating Corpora », Proceedings of the Sixth International Joint Conference on Natural Language Processing, p. 419-428, 2013. find
  8. Boujelbane R., Mallek M., Ellouze M., Belguith L. H., « Fine-Grained POS Tagging of Spoken Tunisian Dialect Corpora », Natural Language Processing and Information Systems, Springer, p. 59-62, 2014. find
  9. Boukadida N., Connaissances phonologiques et morphologiques dérivationnelles et apprentissage de la lecture en arabe (Etude longitudinale), PhD thesis, Université Rennes 2, 2008. find
  10. Buckwalter T., Buckwalter arabic morphological analyzer version 2.0. LDC catalog number LDC2004L02, Technical report, ISBN 1-58563-324-0, 2004. find
  11. Chiang D., Diab M. T., Habash N., Rambow O., Shareef S., « Parsing Arabic Dialects. », EACL, 2006.
  12. Elshafei M., Al-Muhtaseb H., Alghamdi M., « Statistical methods for automatic diacritization of Arabic text », The Saudi 18th National Computer Conference. Riyadh, vol. 18, p. 301-306, 2006. find
  13. Fishman J. A., « Bilingualism with and without diglossia ; diglossia with and without bilingualism », Journal of social issues, vol. 23, no 2, p. 29-38, 1967. find
  14. Goldsmith J., « Unsupervised learning of the morphology of a natural language », Computational linguistics, vol. 27, no 2, p. 153-198, 2001. find
  15. Graja M., Jaoua M., Belguith L. H., « Building Ontologies to Understand Spoken Tunisian Dialect », arXiv preprint arXiv :1109.0624, 2011. ↗
  16. Graja M., Jaoua M., Hadrich Belguith L., « Lexical Study of A Spoken Dialogue Corpus in Tunisian Dialect », The International Arab Conference on Information Technology (ACIT), Benghazi–Libya, 2010. find
  17. Habash N., Rambow O., « MAGEAD : a morphological analyzer and generator for the Arabic dialects », Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, Association for Computational Linguistics, p. 681-688, 2006. — MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects
  18. Habash N., Rambow O., Roth R., « Mada+ tokan : A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization », Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, p. 102-109, 2009. find
  19. Hamdi A., Boujelbane R., Habash N., Nasr A. et al., « The Effects of Factorizing Root and Pattern Mapping in Bidirectional Tunisian-Standard Arabic Machine Translation », MT Summit 2013, 2013a. — The Effects of Factorizing Root and Pattern Mapping in Bidirectional Tunisian - Standard Arabic Machine Translation
  20. Hamdi A., Boujelbane R., Habash N., Nasr A. et al., « Un système de traduction de verbes entre arabe standard et arabe dialectal par analyse morphologique profonde », Traitement Automatique des Langues Naturellesp. 396-406, 2013b. find
  21. Kirchhoff K., Vergyri D., « Cross-dialectal data sharing for acoustic modeling in Arabic speech recognition », Speech Communication, vol. 46, no 1, p. 37-51, 2005. find
  22. Lamel L., Messaoudi A., Gauvain J.-L., « Investigating morphological decomposition for transcription of Arabic broadcast news and broadcast conversation data », INTERSPEECH, p. 1429-1432, 2008. find
  23. Maamouri M., Bies A., « Developing an Arabic treebank : methods, guidelines, procedures, and tools », Proceedings of the Workshop on Computational Approaches to Arabic Script-based languages, Association for Computational Linguistics, p. 2-9, 2004. — Developing an Arabic Treebank: Methods, Guidelines, Procedures, and Tools
  24. Mangu L., Kuo H.-K., Chu S., Kingsbury B., Saon G., Soltau H., Biadsy F., « The IBM 2011 GALE Arabic speech transcription system », Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, IEEE, p. 272-277, 2011. find
  25. Masmoudi A., Khmekhem M. E., Estève Y., Belguith L. H., Habash N., « A Corpus and Phonetic Dictionary for Tunisian Arabic Speech Recognition », Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), Reykjavik, Iceland, May 26-31, 2014., p. 306-310, 2014.
  26. Roth R., Rambow O., Habash N., Diab M. T., Rudin C., « Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking », ACL (Short Papers), p. 117-120, 2008.
  27. Salloum W., Habash N., « Dialectal Arabic to English Machine Translation : Pivoting through Modern Standard Arabic. », 2013. — Dialectal Arabic to English Machine Translation: Pivoting through Modern Standard Arabic
  28. Scherrer Y., Generating Swiss German sentences from Standard German : a multi-dialectal approach, PhD thesis, University of Geneva, 2012. find
  29. Seng S., Vers une modélisation statistique multi-niveau du langage, application aux langues peu dotées, PhD thesis, Université de Grenoble, 2010. find
  30. Shaalan K., Bakr H. M. A., Ziedan I., « Transferring egyptian colloquial dialect into modern standard arabic », International Conference on Recent Advances in Natural Language Processing (RANLP–2007), Borovets, Bulgaria, p. 525-529, 2007. find
  31. Smrž O., « Elixirfm : implementation of functional arabic morphology », Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages : Common Issues and Resources, Association for Computational Linguistics, p. 1-8, 2007. find
  32. Soltau H., Mangu L., Biadsy F., « From Modern Standard Arabic to Levantine ASR : Leveraging GALE for dialects. », ASRU, p. 266-271, 2011a. find
  33. Soltau H., Mangu L., Biadsy F., « From Modern Standard Arabic to Levantine ASR : Leveraging GALE for dialects. », ASRU, p. 266-271, 2011b. find
  34. De l’arabe standard vers l’arabe dialectal TAL.Volume 55 – n o 2/2014 find
  35. Vergyri D., Kirchhoff K., Duh K., Stolcke A., « Morphology-based language modeling for arabic speech recognition. », INTERSPEECH, vol. 4, p. 2245-2248, 2004. find
  36. Vergyri D., Kirchhoff K., Gadde V. R. R., Stolcke A., Zheng J., « Development of a conversational telephone speech recognizer for Levantine Arabic. », INTERSPEECH, Citeseer, p. 1613-1616, 2005. find
  37. Xiang B., Nguyen K., Nguyen L., Schwartz R., Makhoul J., « Morphological decomposition for Arabic broadcast news transcription », Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on, vol. 1, IEEE, p. I-I, 2006. find
  38. Zbib R., Malchiodi E., Devlin J., Stallard D., Matsoukas S., Schwartz R., Makhoul J., Zaidan O. F., Callison-Burch C., « Machine translation of Arabic dialects », Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Association for Computational Linguistics, p. 49-59, 2012.
  39. Zribi I., Boujelbane R., Masmoudi A., Ellouze M., Belguith L. H., Habash N., « A Conventional Orthography for Tunisian Arabic », LREC, p. 2355-2361, 2014.
  40. Zribi I., Khemakhem M. E., Belguith L. H., « Morphological Analysis of Tunisian Dialect », Proceeding of International Joint Conference on Natural Language Processing (IJCNLP 2013), Nagoya, Japan, 2013. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boujelbane-2014-de-l-arabe-standard-vers/