Elamur Research⌕
☰
home / papers / 2010 / Mining the Web for the Induction of a Dialectical Arabic Lexicon

Mining the Web for the Induction of a Dialectical Arabic Lexicon

Paper · 2010 · Rania Al-Sabbagh and Roxana Girju

Summary

The paper tackles the shortage of resources for Arabic dialects by inducing a general-purpose lexicon that links Egyptian Cairene Arabic (ECA) words to Modern Standard Arabic (MSA) equivalents. Each ECA entry gets an MSA synonym, a POS tag, gender and number features inherited from that synonym, and top-ranked Web contexts. Through the synonym, the dialect word can reuse the much richer MSA toolset.

The method adapts Rapp's (1999) associationist approach, which finds translations by correlating word co-occurrence patterns across unrelated corpora. Dialect text on the Web is scarce, search results are often duplicated, and engines cap results at 1,000 pages. To cope with this, the authors replace "parallel acquisition" with "circular acquisition": ECA collocates are gathered first, and those collocates are then checked as plausible contexts for candidate MSA words.

The resulting lexicon covers 1,000 ECA entries, and two native-speaker raters judged it manually. It reaches an F-measure of about 70.9%, clearly ahead of an AraMorph baseline. This is a sizeable resource built without parallel corpora or a hand-made seed lexicon.

Key points

  • The first phase covers only ECA-exclusive vocabulary, in three groups: MSA words with "softened" sounds (e.g., /*/→/d/, /q/→glottal), foreign borrowings (English, French, Italian, Turkish, Spanish), and natively coined dialect words. Vocabulary shared with MSA is left for later work.
  • Results come from 300 evaluated entries:
    • Proposed approach: precision 0.72, recall 0.70.
    • AraMorph baseline: precision 0.424, recall 0.32. The baseline is scored only on POS and gender/number, since AraMorph cannot produce synonyms.
  • Inter-rater agreement between the two human judges was measured with the Kappa coefficient.
  • Recall failures break down as follows:
    • 42% of the missed words returned nothing from Web queries.
    • 16% had an association score below the zero threshold.
  • Remaining precision problems have two named causes:
    • Agglutinative ECA verbs that equal a whole MSA phrase, e.g., a single verb meaning "he was arrested".
    • Long-distance dependencies that fall outside narrow co-occurrence windows.

Interesting ideas

  • I1. Instead of collecting contexts separately for each language and matching them afterwards, the method conditions acquisition for the resource-rich side on contexts already found for the sparse side, so sparse data drives the search.
  • I2. A low-resource item inherits grammatical features (gender, number, POS) and tool access from its mapped high-resource synonym, rather than having them annotated directly.
  • I3. Entries store contextual synonyms together with their top contexts, not absolute synonyms, so the lexicon later doubles as training data for sense disambiguation.
  • I4. The error analysis separates "no evidence found" from "evidence below threshold", which shows whether coverage or scoring is the bottleneck.
  • I5. The authors point out that a fixed small co-occurrence window misses long-distance subjects, a trade-off between window size and noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data harvesting for rare constructions could start from contexts seen in our own small sample and only then query large corpora for matching patterns, rather than mining both sides blindly.
  • I2 (tokens): Rare slang or dialectal English words in the lexicon could borrow morphology and tag features from a mapped standard synonym, with the mapping kept explicit so the decision stays explainable.
  • I3 (world): Ontology links from WordNet or Wikidata could keep the evidence contexts that justified each mapping, so later sense choices can cite them.
  • I4 (learn): Evaluation reports could split failures the same way, distinguishing missing knowledge from rejected weak evidence before tuning thresholds.
  • I5 (tree): Using UD dependency neighbors instead of linear windows as co-occurrence contexts would capture distant arguments without adding positional noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Rania Al-Sabbagh and Roxana Girju. 2010. Mining the Web for the Induction of a Dialectical Arabic Lexicon. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alsabbagh2010mining,
  title = {{Mining the Web for the Induction of a Dialectical Arabic Lexicon}},
  author = {Rania Al-Sabbagh and Roxana Girju},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/4af8668ypsiy},
  url = {https://doi.org/10.63317/4af8668ypsiy}
}

References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bakr, H., Shaalan, K. and Zeidan, I. (2008). A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacriticized Arabic. Proceedings of INFO2008, Cairo, Egypt, 2008 find
  2. Buckwalter, T. (2002). Arabic Morphological Analyzer (AraMorph). Version 1.0. Linguistic Data Consortium, catalog number LDC2002L49 and ISBN 1-58563-257-0 find
  3. Chiang, D., Diab, M., Habash, N., Rambow, O. and Shareef, S. (2005). Parsing Arabic Dialects. ↗
  4. Final Report, 2005 JHU Summer Workshop, 2005 find
  5. Dagan, I. (1991). Lexical Disambiguation: Sources of Information and their Statistical Realization. Proceedings of the 29th Annual Meeting of the Association for Computational Linguistics, 1991 find
  6. Duh, K. and Kirchhoff, K. (2005). POS Tagging of Dialectal Arabic: A Minimally Supervised Approach. Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, Ann Arbor, June 2005.
  7. Rambow, O., Chiang, D., Diab, M., Habash, N., Hwa, R., Sima'an, K., Lacey, V., Nichols, C. and Shareef, S. (2005). Parsing Arabic Dialects. Technical Report, the Johns-Hopkins University, 2005 Summer Research Workshop Rapp, R. (1999). Automatic Identification of Word Translations from Unrelated English and German Corpora. Proceedings of the Annual Meeting of the Association for Computational Linguistics, MD, USA, 1999 ↗
  8. Rapp, R. (2009). Automatic Dictionary Expansion Using Non-parallel Corpora. In Advances in Data Analysis, Data Handling and Business Intelligence (pp 317-325). Springer Berlin Heidelberg: USA. ↗
  9. Rapp, R. (2009). The Automatic Generation of Thesauri of Related Words for English, French, German, and Russian, International Journal of Speech Technology, 11, 3-4, 147-156 Appendix A: This appendix gives a snapshot of some entries of our ECA-MSA lexicon, where ENTRY is the ECA target word; POS for the part-of-speech of the ECA word, SEM for the semantic features with +M meaning MALE and –P means SINGLUAR, MSA syn. in the MSA synonym(s), and CNTXT is the context of the target ECA entry. English Translation of the ECA-MSA Contents ECA ENTRY: start POS: V SEM: +M –P MSA Syn.: start; begin CNTXT: My age_strats, time_starts, Islam_strats, The scene_starts Madness_starts ECA ENTRY: fathers POS: N SMF: +M +P MSA Syn.: fathers CNTXT: Fathers_and mothers; Fathers_children Fathers_generation Fathers_daughters ECA ENTRY: immoral POS: ADJ SMF: +M –P MSA Syn.: immoral, spoilt, abusive CNTXT: abusive_clip immoral_movie, Immoral_speech Immoral_topic ECA ENTRY: fair POS: ADJ SEM: +M –P MSA Syn.: fair; blond CNTXT: fair_Turkish, fair_tall, fair_Lebanese, fair_handsome ECA-MSA Lexicon Snapshot ECA ENTRY: ي )9ا POS: V SEM: +M –P MSA Syn.: أ 9 ؛ أ )9ا CNTXT: ى )9ا _ M%>ا ى )9ا _ يG ى )9ا _ RAF>ا ى )9ا _ مP'Qا ى )9ا _ :S>ا ECA ENTRY: ت&S9ا POS: N SEM: +M +P MSA Syn.: ء&9T CNTXT: ل&U7Qا _ ت&S9ا ت&Sاو _ ت&S9ا ت&;>ا _ ت&S9ا :5>ا _ ت&S9ا ECA ENTRY: V9ا POS: ADJ SEM: +M –P MSA Syn.: ؛ '&* ؛ : ; ؛ .& CNTXT: V9ا _ 6آ V9ا _ مPآ V9ا _ H* V9ا _ ع%% ECA ENTRY: 0&X9ا POS: ADJ SEM: +M –P MSA Syn.: F2ا Y9ا CNTXT: 0&X9ا _ 0آ" 0&X9ا _ :/%7 0&X9ا _ 0&;> 0&X9ا _ H'و find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabbagh-2010-mining-web-induction-dialectical-arabic/