📄 Sample Size in Arabic Authorship Verification

Summary

The paper looks at authorship verification (AV) for Modern Standard Arabic (MSA): deciding whether one disputed text belongs to a single known author. Earlier Arabic AV work relied on very long documents. Such texts are costly to obtain, because much Arabic material exists only in print and OCR is unreliable. The author built a corpus of five MSA genres: fiction, non-fiction, economics, politics and opinion columns. Each genre has ten authors, mostly Egyptian, for 253 documents of roughly 800–1,160 tokens. The method is an author-profiling approach based on Manhattan Distance (Burrows' Delta), applied to n-grams (n = 1–4) of tokens, stems, roots, diacritics and POS tags produced by MADAMIRA. A first experiment picks the best feature ensemble for each genre. A second experiment shrinks the training set from 10 to 5 documents and evaluates with leave-one-out. With only five training texts, every genre beat the 87.1% Classical Arabic baseline. Accuracy actually dropped as more training documents were added, and the length of the questioned document had no measurable effect. This means small, partly digitized samples can be enough for AV.

Key points

• Corpus: 5 MSA domains × 10 authors, 253 documents in total.
  • Sources include the Hindawi Foundation repository and Egyptian and pan-Arab news sites.
  • Average document length per domain ranges from 802 tokens (columnists) to 1,159 tokens (fiction).
• Best feature ensemble per domain:
  • Columnists: bigrams built from stems
  • Economics: bigrams built from roots
  • Fiction: bigrams built from diacritics
  • Non-fiction: stem unigrams
  • Politics: token unigrams
  • The Classical Arabic baseline (Ahmed 2018) used stem bigrams.
• Accuracy at 5 training documents: 87.84% (columnists), 90.00% (economics), 89.20% (fiction), 89.80% (non-fiction) and 90.59% (politics). The baseline was 87.1%, obtained with 19 much larger Classical Arabic documents.
• Number of training documents vs. accuracy: strongly negative, r = −0.931, p < 0.005.
• Questioned-document length vs. accuracy: no significant link (r = 0.132, p = 0.48).
• Training-set size in tokens vs. accuracy: a moderate positive correlation appeared (0.492, p = 0.003). It vanished once the longer-document fiction and non-fiction genres were removed, so it reflects a genre confound.

Interesting ideas

• I1. The paper re-selects the best feature and n-gram combination for each genre instead of reusing the configuration tuned on Classical Arabic, and the winners differ across domains.
• I2. Verification works from a profile of positive examples alone, with a distance threshold and no imposter or negative set, which avoids the dispute over how to choose negatives.
• I3. A surprising positive correlation was traced to genres clustering at the high end of document length, and re-running the regression without those genres removed it.
• I4. A "diacritics-only" representation keeps the vocalization marks and strips the letters, giving a style fingerprint that is independent of vocabulary and turned out to be the best feature for fiction.
• I5. Accuracy did not depend on the length of the questioned document, so a partial sample of a long text can stand in for the whole.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Settings tuned on one treebank or genre, such as fables or news, should be re-validated per genre rather than assumed to carry over.
• I2 (learn): One-class profiles could flag silver sentences whose features sit far from the agreed-upon bulk, without curating an explicit set of bad parses.
• I3 (learn): When accuracy seems to track data size, repeat the analysis with each genre held out as a negative control before concluding that more data helps.
• I4 (tokens): The analogous trick for English is to reduce tokens to morphology or POS skeleton strings, giving lexicon-free style features for genre or register detection.
• I5 (answer): Cheaper style or genre checks on long input texts could run on a sampled excerpt, and the excerpt's adequacy can be tested empirically in the same way.