Elamur Research⌕
☰
home / papers / 2018 / Hybrid approach for transliteration of Algerian arabizi: a primary study

Hybrid approach for transliteration of Algerian arabizi: a primary study

Paper · 2018 · Imane Guellil, Faical Azouaou, Fodil Benali et al.

Summary

Algerian dialect speakers often write online in Arabizi, a spontaneous romanized script that mixes Latin letters, digits and symbols. Most dialect-processing tools expect Arabic script, so this text cannot be processed directly. The authors propose a hybrid rule-plus-statistics transliterator from Algerian Arabizi to Arabic script that does not need a parallel Arabizi–Arabic corpus.

The method has four stages:

  • Normalization. Letter elongations are collapsed, for example "hiaaaaati" becomes "hiati".
  • Letter mapping. A table maps each Latin letter or digit to its possible Arabic letters, and position-dependent rules refine it. Vowels may also map to nothing.
  • Candidate generation. The mappings and rules produce every possible Arabic spelling of the word.
  • Ranking. Candidates are scored against a monolingual Arabic-script corpus of Algerian comments that the authors collected from Facebook. Scoring uses either raw frequency lookup or a language model.

Evaluated on three test corpora, the best setting reached 75.11% precision. The authors call this competitive with general Arabizi transliteration work. The approach matters because it replaces costly parallel data with easy-to-collect monolingual text.

Key points

  • Four problems. Arabizi transliteration has to deal with:
    • vowel handling, where a vowel can map to several Arabic letters or to none;
    • one Latin letter covering several Arabic letters (for example 't' → ت or ط);
    • context-dependent ambiguity (for example 'matar' as "rain" vs "airport");
    • code-switching with French and English.
  • Data.
    • The Arabic-script corpus comes from 133 Algerian Facebook pages, such as Ooredoo, Djezzy and Hamoud Boualem.
    • It is complemented by a dialect dictionary built partly from the PADIC corpus.
  • Candidate generation.
    • The letter table follows a Wikipedia-derived Arabic chat alphabet mapping, similar to van der Wees et al. (2016), extended with Algerian-specific rules.
    • Example sizes: "kraht" yields 32 candidates and "hiati" yields 16.
    • Unlike earlier systems, vowels can be dropped, so the correct form of words like '3afsa' becomes reachable.
  • Result. Best precision was 75.11% across three test corpora.
  • Remaining errors come from two sources:
    • ignoring sentence context, since each word is treated as a unigram;
    • French-origin words that are missing from the Arabic corpus (for example 'lafichage').

Interesting ideas

  • I1. Generate an over-complete candidate set with simple rules, then let a corpus pick the winner; this cleanly separates "what is possible" from "what is likely"
  • I2. Explicitly allow a "null" mapping (deleting a character) during generation, because the right answer often lies outside naive one-to-one substitution
  • I3. Replace expensive parallel data with a large monolingual target-side corpus used as the arbiter of candidate plausibility
  • I4. The error analysis shows that per-word decisions fail exactly where context decides the reading, and the proposed fix is to rank n-gram candidates instead of single words
  • I5. Use the hybrid system to bootstrap a semi-supervised parallel corpus that can later train a purely statistical model

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Noisy or non-standard spellings could be normalized by expanding rule-based variants and ranking them against frequency counts from clean English text.
  • I2 (tokens): Spelling or abbreviation repair in the tokenizer could include deletion and insertion options, so that dropped-vowel forms like "msg" or "pls" can be resolved to lemmas.
  • I3 (learn): Silver data could be filtered by checking candidate analyses against monolingual statistics instead of hand-annotated pairs, which cuts teacher cost.
  • I4 (tree): When the parser picks among ambiguous lexical readings, scoring them jointly with neighbouring words rather than in isolation should cut errors of the "rain vs airport" kind.
  • I5 (learn): Rule-generated, corpus-ranked outputs could become silver training data for a learned component, with a negative-control check on held-out gold data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,484 works

Cite

Cite as Imane Guellil, Faical Azouaou, Fodil Benali, Ala-Eddine Hachani and Houda Saadane. 2018. Hybrid approach for transliteration of Algerian arabizi: a primary study. arXiv:1808.03437 · 25e conférence sur le Traitement Automatique des Langues Naturelles (TALN), May 2018, Rennes, France.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.03437 · arXiv 1808.03437

@misc{guellil2018hybrid,
  title = {{Hybrid approach for transliteration of Algerian arabizi: a primary study}},
  author = {Imane Guellil, Faical Azouaou, Fodil Benali, Ala-Eddine Hachani and Houda Saadane},
  year = {2018},
  doi = {10.48550/arxiv.1808.03437},
  eprint = {1808.03437},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.03437}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Badrashiny M., Eskander R., Habash N., & Rambow, O. (2014). Automatic Transliteration of Romanized Dialectal Arabic. Paper presented at the CoNLL. find
  2. Cotterell R., Renduchintala A., Saphra N., &Callison-Burch, C. (2014).An algerianarabic-french code-switched corpus. Paper presented at the Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme. find
  3. Darwish K. (2013). Arabizi detection and conversion to Arabic.arXiv preprint arXiv:1306.6755. ↗
  4. Guellil I., & Azouaou F. (2017). ASDA: Analyseur Syntaxique du Dialecte Algérien dans un but d'analyse sémantique. arXivpreprint arXiv:1707.08998. ↗
  5. Guellil I., Azouaou F., & Abbas M. (2017). Comparison between Neural and Sta-tistical translation after translitera-tion of Algerian Arabic Dialect. Paper presented at the WiNLP: Women & Underrepresented Minorities in Natural Language Processing (co-located withACL 2017). find
  6. Guellil I., Azouaou F., Abbas M., & Fatiha, S. (2017a). Arabizi transliteration of Algerian Arabic dialect into Modern Standard Arabic. Paper presented at the Social MT 2017/First workshop on Social Media and User Generated Content Machine Translation. find
  7. Guellil I., AZOUAOU F. (2017b). Neural Vs Statistical Translation of Algerian Arabic Dialect written with Arabizi and Arabic letter. The 31st Pacific Asia Conference on Language, Information and Computation PACLIC. find
  8. Guellil., I et AZOUAOU F. (2017). Bilingual Lexicon for Algerian Arabic Dialect Treatment in Social Media. WiNLP: Women & Underrepresented Minorities in Natural Language Processing (co-located with ACL 2017) find
  9. Habash N., Soudi A., &Buckwalter, T. (2007). On arabic transliteration Arabic computational morphology (pp. 15-22): Springer. find
  10. Josan G. S., &Lehal, G. S. (2010).A Punjabi to Hindi Machine Transliteration System.Computational Linguistics and Chinese Language Processing, 15(2), 77-102. find
  11. Kaur K., & Singh P. (2014).Review of Machine Transliteration Techniques.International Journal of Computer Applications, 107(20). find
  12. May J., Benjira Y., &Echihabi, A. (2014). An Arabizi-English social media statistical machine translation system. Paper presented at the Proceedings of the 11th Conference of the Association for Machine Translation in the Americas. find
  13. Meftouh K., Harrat S., Jamoussi S., Abbas, M., &Smaili, K. (2015). Machine translation experiments on PADIC: A parallel Arabic dialect corpus. Paper presented at the The 29th Pacific Asia conference on language, information and computation.
  14. Saâdane H., &Habash, N. (2015).A conventional orthography for Algerian Arabic.In Proceedings of the Second Workshop on Arabic Natural Language Processing (pp. 69-79). find
  15. Saâdane H., & Semmar N. (2012).Utilisation de la transliteration arabe pour l'amélioration de l'alignement de mots à partir de corpus parallèles français-arabe (UsingArabic Transliteration to Improve Word Alignment from French-ArabicParallelCorpora)[in French].Paper presented at the Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 2: TALN. find
  16. Saâdane H., Nouvel,. D., Seffih H., &Fluhr, C. (2017). Une approche linguistique pour la détection des dialectes arabes. Paper presented at the 24e Conférence sur le Traitement Automatique des Langues Naturelles (TALN). find
  17. Rosca M., &Breuel T. (2016). Sequence-to-sequence neural network models for transliteration. arXiv preprint arXiv:1610.09565. ↗
  18. Saâdane H., Guidere M., &Fluhr C. (2013).La reconnaissance automatique des dialectes arabes à l’écrit. Paper presented at the Colloque International Traduction et Champs Connexes, Quelle Place Pour La Langue Arabe Aujourd’hui. van der Wees, M., Bisazza A., &Monz C. (2016). A Simple but Effective Approach to Improve Arabizi-to-English Statistical Machine Translation. WNUT 2016, 43. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guellil-2018-hybrid-approach-transliteration-algerian-arabizi/