📄 Nojoom.AI at AraGenEval Shared Task: Arabic Authorship Style Transfer

Summary

The paper describes Nojoom.AI's entry for Subtask 1 of the AraGenEval2025 shared task, Arabic Authorship Style Transfer. The task is to rewrite a neutral, formal Arabic paragraph so it reads like a named author while keeping its meaning. The team built two systems in sequence.

• Baseline: few-shot prompting of Gemini 2.5 Flash. For each author, the exemplars were picked by K-means clustering over sample embeddings.
• Main system: fine-tuned Arabic-capable seq2seq models (mT5, AraT5, mBART, mLongT5, AraBART). An author tag was prepended to each input, and LoRA was added for extra training on limited GPUs.

Training data came from the official paragraph-level dataset. Its long Arabic inputs made token-length limits a central engineering decision. Beyond the official metrics (BLEU, chrF), the authors also measured meaning preservation with BERTScore and style strength with a separately trained author classifier.

Fine-tuning roughly doubled BLEU over the prompting baseline. The best test result was about 24.5 BLEU and 59.3 chrF, with BERTScore near 92 and style-classifier precision around 86. The compact AraT5-v2-1024 nearly matched far larger multilingual models. This suggests that language-specific pretraining pays off for stylistic generation in morphologically rich languages.

Key points

• Dataset: the official AraGenEval2025 data has 35,122 training, 4,157 validation and 8,143 test paragraphs. Test labels are hidden. Tokenized inputs reach up to 3,361 tokens.
• Token-length caps: with the mBART50 tokenizer, a cap of 750 tokens covers about 90% of samples and a cap of 1024 covers about 99.6%. Caps were chosen against hardware limits, since memory grows roughly quadratically with length.
• Baseline: Gemini 2.5 Flash few-shot prompting with K = 3 K-means-selected exemplars per author. The elbow and silhouette analyses pointed to 2–3 clusters for most authors. Validation scores were 11.66 BLEU and 48.12 chrF.
• Best models:
  • facebook/mbart-large-50 gave the top test scores, 24.46 BLEU and 59.33 chrF.
  • UBC-NLP/AraT5-v2-1024 (367M parameters) was close behind, at 24.07 BLEU on test.
  • Adding LoRA to AraT5-v2-1024 brought only small changes.
• Per-author results (AraT5-v2-1024 vs. baseline):
  • Gains appeared for every author, with an overall uplift of about +11 BLEU and +11 chrF.
  • The largest jump was for Gibran Khalil Gibran: +20.69 BLEU from only 30 validation samples.
  • Nawal El Saadawi and Ahmed Shawqi also improved strongly.

Interesting ideas

• I1. Picking few-shot exemplars by clustering embeddings and taking representatives gives the prompt a varied, typical sample of each class instead of random examples. — For Mova (learn): When prompting LLM teacher tiers, cluster candidate sentences per construction type and feed one representative per cluster, so the teacher sees diverse yet typical cases.
• I2. Sequence-length caps were set from percentile coverage of the actual token-length distribution, weighed against quadratic memory cost, instead of picking a round number. — For Mova (tree): Choose gparse's maximum input length from a coverage table over the treebanks, and route the rare overlong sentences to the perceptron parser or a splitter.
• I3. Evaluation combined three separate axes: surface overlap, semantic similarity, and a trained classifier that checks whether the target style was actually achieved. — For Mova (back): Score generation from the meaning graph on fidelity to the graph and on register/style correctness separately, with a small classifier as the style judge.
• I4. Conditioning a single seq2seq model on a simple prepended tag lets one model serve many target authors. — For Mova (back): A register or audience tag on the graph-to-text input could let one generator produce child-level, school-level or formal versions of the same meaning.
• I5. Results were broken down per author with deltas against the baseline. This exposed how tiny classes (30 or 25 samples) behave differently from the aggregate. — For Mova (learn): Report parser and frame-extraction gains per relation or frame type against the previous version, so improvements on rare categories are not hidden by averages.