📄 A Personalized Markov Clustering and Deep Learning Approach for Arabic Text Categorization

Summary

The paper tackles automatic topic classification of Arabic documents, a language the author describes as underserved by text categorization research. Arabic morphology makes the task hard: words are built by interleaving consonantal roots with patterns, hamza is spelled inconsistently, and diacritics can separate unrelated meanings. The proposed pipeline has three steps. First, function words are removed, each word is reduced to a root with a letter-weighting stemmer, and Tf-Idf weights are computed over roots. Second, words are grouped by root using Fuzzy-C-Means with Mahalanobis distance. Markov clustering then merges redundant groups, and the number of initial clusters is estimated automatically. Third, a separate Deep Belief Network, pre-trained with RBMs, is trained for each cluster. A new document is sent to the model of its closest cluster. On Al-Jazeera and Saudi Press Agency news, the system reaches an F-measure of 91.02% under 10-fold cross-validation, well above the earlier systems it cites. The work suggests that root-aware grouping combined with cluster-specific models can cope with rich, ambiguous morphology.

Key points

• Data: 10,000 Al-Jazeera news documents and 6,000 Saudi Press Agency documents (Al-Harbi et al., 2008). Clustering was run on 12,000 documents sampled from both sources.
• Results: precision 91.2%, recall 90.9%, F-measure 91.02%. In Table 1, prior methods have recall of 57.4%, 66.7% and 80.2%, and none of them is root-based or robust to diacritics.
• Cluster count: the initial number of clusters K is the total number of density modes found when the data is projected onto its eigenvectors. Modes are detected with kernel density estimation, counting each place where the gradient turns from positive to negative. The idea is borrowed from bioinformatics (Pouyan et al., 2015).
• Fuzzy clustering and document assignment: Fuzzy-C-Means uses fuzzifier m = 2 and Mahalanobis distance. A document is assigned to the cluster that shares the most words with its top 40% most frequent words.
• Per-cluster DBN: for example, the network for Cluster 1 has sigmoid hidden layers of 80, 80 and 40 units. It is pre-trained as RBMs with CD-1 and then fine-tuned with backpropagation.
• Errors: most mistakes come from outliers, such as recent words not derived from roots, and from documents that mix several topics.

Interesting ideas

• I1. Instead of fixing the number of clusters by hand, count the density peaks of the data along its principal eigen-directions and use that total as K; this gives an automatic, inspectable estimate of how many groups exist.
• I2. Assign each item to every candidate group with a soft membership, and measure distance with Mahalanobis rather than Euclidean, so that loose groups can absorb more distant members than tight ones.
• I3. Collapse surface variants (spelling, diacritics, inflection) onto a shared underlying unit before any statistics, so that sparse forms pool their evidence.
• I4. First split the input space into coarse clusters, then train a small specialist model per cluster and route each new input to its nearest cluster's model.
• I5. In error analysis, separate inputs that fall outside the method's core assumption (here, recent words not derived from roots) from genuine model mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): when inducing verb classes or sense groups from silver data, this mode-counting test could propose how many classes to create, with the density plots kept as an explanation.
• I2 (world): mapping a lemma onto WordNet/FrameNet senses with graded, spread-aware memberships would expose uncertainty instead of forcing a single early choice.
• I3 (tokens): the lemmatizer could likewise normalize English spelling variants (British/American, hyphenation) to one lemma before tagging statistics are counted.
• I4 (tree): gparse or the perceptron parser could use genre- or construction-specific heads selected by an explicit, loggable routing step, which keeps the specialist choice explainable.
• I5 (learn): evaluations could tag words outside the lexicon or ontology as a separate failure class, acting as a negative control so that they do not hide real parser or frame errors.