📄 Exploring Author Style in Nakba Short Stories: A Comparative Study of Transformer-Based Models

Summary

The authors ask how well transformer encoders can capture shared meaning and individual authorial style in Arabic literary prose. They work with a small corpus of eight Nakba short stories, each by a different author, taken from the Nakba Archive website. They turned the PDFs into a sentence-per-row CSV by stripping diacritics, Latin characters, URLs and other noise, normalizing whitespace and Tatweel with PyArabic, and splitting sentences with NLTK. Each story becomes a set of contextual embeddings from AraBERT, BERT or RoBERTa, and story pairs are scored with cosine similarity, with TF-IDF as the classical baseline. To limit bias, only stories of comparable length are paired. All three transformers give very high similarity scores (roughly 94–99.8%), while TF-IDF stays near 42–58%, and RoBERTa scores highest on every pair. The authors read this as evidence that transformers capture cultural themes and stylistic nuance in Arabic narratives. They also concede that such near-ceiling scores may hide fine stylistic differences. The cleaned dataset is released publicly as a base for later literary NLP work.

Key points

• Corpus: 8 Nakba short stories by 8 different authors, 473 sentences, about 12,288 tokens, sourced from nakba-archive.org and published at github.com/PsArNLP/Nakba.
• Preprocessing tools: PyArabic (diacritic and Tatweel removal, affix stripping), regular expressions for noise, NLTK for sentence splitting and Arabic stopwords, and WordCloud for visualization.
• Only three length-matched pairs are compared: stories 2&6, 7&8 and 3&4.
• Cosine similarity per pair:
  • TF-IDF: 54.40 / 42.16 / 57.97
  • AraBERT: 96.18 / 93.87 / 95.23
  • BERT: 98.23 / 97.89 / 97.77
  • RoBERTa: 99.56 / 99.77 / 99.61
• After stemming and stopword removal, the word cloud surfaces the broken plural TwAbyr ("queues", singular TAbur), which the authors tie to daily hardship in refugee camps.

Interesting ideas

• I1. Pairing texts only with others of similar length keeps document size from distorting a similarity score, which is a cheap control for any pairwise comparison. — For Mova (learn): When comparing outputs from different witness parsers or generated paraphrases, bucket items by length first so agreement scores are not inflated or depressed by size alone.
• I2. A plain TF-IDF baseline sits next to the neural models, so the gain from contextual representations is visible as a gap rather than an isolated number. — For Mova (learn): Every evaluation of a frame or world-building step could report a lexical-overlap baseline alongside it, showing how much the structured pipeline adds beyond bag-of-words matching.
• I3. The authors admit that scores clustered near 99% may fail to separate texts, so a metric that rates everything as similar is a warning sign, not a success. — For Mova (learn): Add negative controls, such as unrelated or shuffled text pairs, to similarity-based checks, and reject a metric that cannot push those controls clearly below genuine matches.
• I4. Stripping affixes and stopwords before counting frequencies exposed a culturally central broken-plural noun that raw counts would have scattered across surface forms. — For Mova (tokens): Aggregating frequencies over lemmas instead of surface forms (irregular plurals, inflections) gives a cleaner signal for spotting a text's key domain words for later stages.
• I5. Converting PDFs into a sentence-per-row table with spare columns for future labels (NER, sentiment) makes later annotation layers easy to attach. — For Mova (world): Store per-sentence records with open metadata slots so entity, state and event annotations from a text's world can be added incrementally and traced back to their source sentence.