📄 Empirical Evaluation of Pre-trained Language Models for Summarizing Moroccan Darija News Articles

Summary

Moroccan Darija (MD) is the main spoken Arabic variety in Morocco, yet it has few NLP resources, particularly for summarization. The authors extend the earlier GOUD.MA news summarization benchmark. They crawled the goud.ma site again and collected 50,517 new articles across 11 categories, released as GOOD.MA_v2. In this corpus the headline, usually written in Darija, serves as the target summary. Article bodies are written either in MSA alone or in MSA combined with Darija.

They compare three groups of models:

• Encoder-only models, warm-started as Seq2Seq systems: AraBERT, DarijaBERT, DziriBERT.
• Native encoder-decoders, fine-tuned directly: AraBART, AraT5, mBART.
• Causal LLMs, prompted zero-shot: GPT-4o mini, Llama 3, Mistral NeMo.

Evaluation uses ROUGE and BERTScore. Training on the enlarged data beats the scores originally reported for GOUD.MA, and mBART performs best overall. Zero-shot LLMs and extractive baselines fall behind the fine-tuned models. The authors also caution that overlap-based metrics may undervalue legitimate paraphrases. The data, models and code are open-sourced, giving a reusable base for dialectal Arabic generation.

Key points

• The new data comes from goud.ma, crawled with Scrapy and Selenium between 2022 and October 2024. The crawl yielded 50,517 articles with date, title, content and category, stored as CSV.
• The merged training split has 189,805 articles, up from 139,288 in GOUD.MA. Validation and test are the original GOUD.MA sets, with 9,497 articles each. Articles average about 254 tokens and titles about 16.4 tokens.
• Encoder-only models were turned into encoder-decoders following Rothe et al. (2020). All fine-tuning ran for 20 epochs with batch size 20 and gradient accumulation of 8.
• Extractive baselines were Lead-2, TextRank and SumRank. Metrics were ROUGE-1, ROUGE-2, ROUGE-L and BERTScore.
• mBART ranked first, with AraBART and AraT5 close behind. Mistral NeMo hallucinated and sometimes answered in French or Spanish, so its outputs were machine-translated with the Google Translate API before scoring.

Interesting ideas

• I1. News headlines can act as free reference summaries, giving a large supervised summarization set with no manual annotation. This is a cheap way to obtain training and evaluation pairs.
• I2. Adding training data while keeping the original validation and test sets fixed isolates the effect of extra data from changes in the benchmark.
• I3. Encoder-only checkpoints can be warm-started into encoder-decoder generators, reusing representations trained for understanding as the basis for generation.
• I4. Zero-shot LLMs may drift into the wrong output language, and repairing that afterwards with machine translation confounds the comparison. Such repairs should be detected and reported explicitly.
• I5. Overlap metrics such as ROUGE can penalize faithful paraphrases. The authors therefore treat their scores as partial and call for human judgment.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): headline–body pairs could serve as a test bed for the text → graph → text compressor, checking whether the generated short text keeps the headline's core frame.
• I2 (learn): when silver data agreed by witness parsers is added, scoring always against the same frozen gold split shows whether the new data actually helps.
• I3 (back): the encoder of the gparse transformer parser could initialize a generator, so the analysis and generation sides share representations.
• I4 (learn): LLM teacher tiers should check output language and format before outputs become silver labels, and log any repair step as a possible confound.
• I5 (back): compressor output could be judged by comparing re-parsed frames against the source meaning graph, rather than by n-gram overlap alone.