📄 Arabic Data Science Toolkit: An API for Arabic Language Feature Extraction

Summary

The paper presents the Arabic Data Science Toolkit (ADST). It is meant for data scientists who need linguistically informed features from Arabic text but know neither Arabic nor NLP. The authors argue that purely statistical representations such as bag of words work poorly for a morphologically rich language like Arabic. ADST therefore focuses on features that need language-specific or psychological knowledge. It is written mainly in Python, served as an HTTP web service through Flask, and exchanges JSON-like data, so any programming language can use it. Individual Python and Perl scripts can also run on their own.

The toolkit targets social-media text in both MSA and Egyptian Arabic. Its first use cases are emotion classification and authorship attribution. It wraps existing resources: MADAMIRA, the NRC Word-Emotion Association Lexicon, and the Attia et al. Arabic word list. On top of these it adds hand-built detectors for laughter, ululation, elongation and misspellings. No classification experiments are reported. The contribution is a single access point to the best Arabic tools, including ones that open-source toolkits like NLTK leave out for licensing reasons.

Key points

• ADST is an HTTP API built on Flask with JSON-like input and output. A user can request all features, a feature set, or a single feature.
• MADAMIRA runs in its Egyptian configuration and segments text under two schemes, ATB and ATB4MT. Its output yields frequency histograms of gender, number, person, aspect, clitics, POS and lemmas, plus a list of OOV words.
• The NRC lexicon covers 8 emotions plus positive and negative sentiment. It was crowdsourced in English and machine-translated into 39 languages, without context, using Google Translate.
• Laughter detectors find single letters repeated three or more times (خ, ح, ه) and multi-letter patterns that also involve ق, ع, ا and ي. They include a dedicated detector for Spanish-style "jaja" written in Arabic script.
• The spelling module checks only OOV words against Arabic Wordlist 1.6 (9,196,215 inflected forms). It ranks candidate intended words with an OpenFST noisy-channel model trained on quickly typed Arabic and separates "minor" from "major" errors.

Interesting ideas

• I1. Laughter variants are kept as distinct categories rather than merged into one class, as TweeboParser does: presence and length signal emotion intensity, while the specific form signals the author.
• I2. Spelling correction runs only on out-of-vocabulary words and returns an n-best list of intended words from a weighted noisy-channel FST, with cost thresholds that sort errors into minor and major.
• I3. Repeated letters (elongation) and laughter length are treated as a measurable degree of emotion, not just a yes/no flag.
• I4. A foreign laugh written in native script ("jaja" in Arabic letters) was identified from contextual evidence: Spanish-language surroundings and nearby laughter emoji.
• I5. The workflow is to extract every available feature first, then use feature selection to keep a smaller fast subset for production.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): the tokenizer could tag interjections with a subtype and a repetition-length attribute rather than a generic INTJ, so later stages still see that signal.
• I2 (tokens): noisy input could be normalized this way before tagging, keeping the original form and the chosen correction with its cost as an explainable trace.
• I3 (frames): an intensity value attached to the speaker could feed speech-act and irony judgments, for example telling enthusiastic agreement apart from flat agreement.
• I4 (world): ambiguous tokens could be resolved by recording surrounding-language and emoji cues as explicit evidence, rather than guessing from the string alone.
• I5 (learn): candidate cues for repair rules or classifiers could be generated broadly, then pruned by measured usefulness against negative controls, keeping only the ones that justify their cost.