📄 Amr&MohamedSabaa at AraGenEval shared task: Arabic Authorship Identification using Term Frequency – Inverse Document Frequency Features with Supervised Machine Learning

Summary

The paper tackles Task 2 of the AraGenEval 2025 shared task: deciding which of 21 Arabic writers produced a given literary passage. The authors deliberately avoid neural networks. Their pipeline has three parts:

• Cleaning: keep only characters from the Arabic Unicode blocks, normalise whitespace, and drop very short snippets.
• Features: each text is represented by two concatenated TF-IDF views, one built on word unigrams and bigrams and one on character 2–4-grams.
• Classifiers: five standard models are compared on these features.

Linear models clearly outperform tree-based ones. A linear SVM scored best on validation, but the team submitted Logistic Regression because it trained more stably and gives usable probability estimates. That submission reached 88.90% accuracy and 82.74% macro F1 on the hidden test set.

An ablation shows that word and character signals add up rather than overlap. Per-author analysis exposes weak spots: rare authors, writers from the same era, and translated texts. The work matters as a cheap, reproducible, interpretable reference point for Arabic stylometry, with code released on GitHub.

Key points

• Data: 35,122 training and 4,157 validation passages from 21 authors, with typical lengths around 1,750–1,850 characters. The 8,413-sample test set is heavily imbalanced, ranging from Hassan Hanafi with 3,735 training samples down to 399 for the smallest author.

• Features:

  • Word TF-IDF: 15,000 dimensions, n-grams 1–2, sublinear TF, max_df 0.9.
  • Character TF-IDF: 5,000 dimensions, n-grams 2–4, min_df 2, max_df 0.8.
  • Combined vector: 20,000 dimensions.

• Validation results:

  Model	Accuracy	Macro F1
  Linear SVM (SGD, hinge)	93.22	87.52
  Logistic Regression	90.54	82.63
  Naive Bayes	79.22	—
  Random Forest	59.32	—
  Decision Tree	32.23	—

  The SVM's stratified 5-fold CV macro F1 was 97.67%.

• Official test (Logistic Regression): 88.90% accuracy, 82.74% macro F1, 84.53% precision, 83.75% recall.

• Ablation (macro F1): characters only 0.8199, words only 0.8508, both combined 0.8752.

• Per-author validation accuracy: Salama Moussa and Gibran Khalil Gibran reach 100%, while Tharwat Abaza drops to 44.44% and Kamel Kilani to 16% (only 25 validation samples).

Interesting ideas

• I1. A cheap lexical baseline (TF-IDF plus a linear classifier) gives any fancier system an honest floor to beat, and its per-class errors show where surface statistics alone fail.
• I2. Combining word-level and character n-gram views helped because sub-word chunks pick up morphology and spelling habits that whole-word features miss.
• I3. The team picked the submitted model for stability and calibrated probabilities rather than top validation score, and test results stayed close to validation.
• I4. Breaking scores down per class and setting them beside class support showed that failures concentrate on rare classes, translated texts and stylistically mixed authors.
• I5. A simple script filter (keep only the target language's Unicode blocks, drop tiny fragments) cleaned the noisy corpus before feature extraction.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Run such a bag-of-words baseline as a negative control on Mova's tests; if the structured text→tree→frames route doesn't beat it, the structure isn't earning its keep.
• I2 (tokens): Character 2–4-gram features could back up the English tagger and lemmatiser on unseen or misspelled words, where whole-word lookup has nothing to go on.
• I3 (learn): When choosing parser or tagger checkpoints, prefer calibrated confidence, so low-confidence sentences can be routed reliably to the LLM teacher tier or witness-parser voting.
• I4 (learn): Report evaluation per relation label, frame type or text genre with support counts, so weak spots surface instead of hiding inside an overall average.
• I5 (tokens): A comparable pre-tokenisation filter could strip stray foreign-script and junk fragments from silver-data sources before they reach the tokenizer.