Elamur Research⌕
☰
home / papers / 2021 / Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization

Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization

Paper · 2021 · Mram Kahla, Zijian Győző Yang and Attila Novák

Summary

Arabic abstractive summarization lagged far behind English because no suitable training data existed. Earlier Arabic work was mostly extractive. The one prior abstractive attempt trained on KALIMAT, whose summaries were produced automatically by an extractive algorithm, so they were neither human-written nor abstractive. The authors built the first corpus of human-written abstractive Arabic news summaries, with over 21 thousand items. On it they fine-tuned multilingual BERT, AraBERT and multilingual BART-50. Because the corpus is small, they also tried cross-lingual transfer: summarizers already fine-tuned for Hungarian or English (M-BERT-based) or Russian (M-BART-50-based) were trained further on Arabic. Evaluation combined ROUGE with a two-step human study. Native judges first ranked the systems, then scored the best one for adequacy and fluency. Transferred models improved on the baselines, and the English-pretrained M-BERT model came out best. The monolingual AraBERT model was clearly the weakest. The work gives Arabic a usable resource and suggests that the ability to summarize can carry over across languages.

Key points

  • New corpus: over 21k Arabic news articles paired with human-written abstractive summaries. It is presented as the first of its kind.
  • Prior data critique: KALIMAT's 20,291 summaries were generated by the extractive Gen-Summ (AQBTSS) algorithm. EASC has only 153 articles with 765 extractive summaries.
  • Systems compared:
    • M-BERT, AraBERT and M-BART-50 fine-tuned directly on Arabic.
    • M-BERT summarizers first trained for English or Hungarian, then adapted to Arabic.
    • An M-BART-50 summarizer first trained for Russian, then adapted to Arabic.
  • Human evaluation results:
    • m-BERT English ranked best, and its output stays close to the original lead.
    • M-BART-50 and its Russian-transferred variant were nearly as good.
    • m-BERT Hungarian rephrased more freely and produced summaries about 10% longer.
    • AraBERT ranked last. It produced bloated output, often double the length, with meaning reversals and grammar errors.
  • Inter-annotator agreement was measured with Fleiss's kappa, Krippendorff's alpha, Scott's pi and average pairwise Cohen's kappa. Agreement was substantial (about 0.61) for the m-BERT English judgments and much lower for several other settings.

Interesting ideas

  • I1. Chain the fine-tuning by first teaching the task in a data-rich language, then adapting the already task-trained multilingual model to a small target set, so the task skill transfers along with the language knowledge
  • I2. Before building on a dataset, check how its references were actually produced, since KALIMAT's "abstractive" targets turned out to be machine-made extracts
  • I3. Evaluate in two stages: rank all systems cheaply first, then spend detailed 1–5 adequacy/fluency scoring only on the winner
  • I4. Report agreement with several coefficients at once and treat near-zero agreement as a signal about the setting itself, not just as noise
  • I5. Describe failure modes qualitatively beyond the overlap metric, for example length bloat without new content, or summaries that reverse the original meaning

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): train a component on abundant English data first, then adapt that already task-trained model to small specialised sets (a new genre or a scarce annotation type) rather than starting from a raw pretrained model.
  • I2 (learn): tag every silver example with its provenance (which witness parser or teacher tier produced it), so the pipeline never trains a component on output that is secretly its own kind of baseline.
  • I3 (back): when comparing text → graph → text compressor variants, let humans or an LLM teacher rank outputs first and give detailed scores only to the leading variant.
  • I4 (learn): compute Fleiss, Krippendorff and pairwise Cohen over witness parsers' decisions, and flag sentence types where agreement collapses as unreliable silver.
  • I5 (back): add automatic checks to generation for polarity or role reversal against the source graph and for output that grows longer without adding new facts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Pázmány Péter Catholic UniversityBudapest · Hungary2 works · 22 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 worksSummarization · 232 works

Cite

Cite as Mram Kahla, Zijian Győző Yang and Attila Novák. 2021. Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kahla2021cross,
  title = {{Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization}},
  author = {Mram Kahla, Zijian Győző Yang and Attila Novák},
  booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)},
  year = {2021},
  doi = {10.26615/978-954-452-072-4\_074},
  url = {https://doi.org/10.26615/978-954-452-072-4_074}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Molham Al-Maleh and Said Desouki. 2020. Arabic text summarization using deep learning approach. Journal of Big Data, 7:1–17. ↗
  2. Lamees Al Qassem, Di Wang, Hassan Barada, Ahmad Al-Rubaie, and Nawaf Almoosa. 2019. Automatic Arabic text summarization based on fuzzy logic. In Proceedings of the 3rd International Conference on Natural Language and Speech Processing, pages 42– 48.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  4. Aqil M. Azmi and Nouf I. Altmami. 2018. An abstractive arabic text summarizer with user controlled granularity. Information Processing and Management, 54(6):903–921. ↗
  5. J. Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Fouad Soufiane Douzidia and Guy Lapalme. 2004. Lakhas, an Arabic summarization system. Proceedings of DUC2004. find
  8. Mahmoud El-Haj and R. Koulali. 2013. KALIMAT a multipurpose Arabic corpus. In Second Workshop on Arabic Corpus Linguistics (WACL-2), pages 22– 25. find
  9. Mahmoud El-Haj, Udo Kruschwitz, and C. Fox. 2010. Using Mechanical Turk to create a corpus of Arabic summaries. In Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages workshop in conjunction with the 7th International Language Resources and Evaluation Conference (LREC 2010). find
  10. Khalid N. Elmadani, Mukhtar Elgezouli, and Anas Showk. 2020. BERT fine-tuning for Arabic text summarization. ArXiv, abs/2004.14135. ↗
  11. Ilya Gusev. 2020. Dataset for automatic summarization of russian news. AINL 2020. Communications in Computer and Information Science, vol 1292. find
  12. Said Moulay Lakhdar and Mohamed Amine Chéragui. 2019. Building an extractive Arabic text summarization using a hybrid approach. In International Conference on Arabic Language Processing, pages 135– 148. Springer. find
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics. ↗
  16. Mohamed Hedi Maaloul, Iskandar Keskes, Lamia Belguith, and Philippe Blache. 2010. Automatic summarization of Arabic texts based on RST technique. In Proceedings of the 12th International Conference on Enterprise Information Systems - Artificial Intelligence and Decision Support Systems, pages 434– 437. find
  17. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016. Classify or select: Neural architectures for extractive document summarization. arXiv preprint arXiv:1611.04244. ↗
  18. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics.
  19. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. CoRR, abs/1705.04304. ↗
  20. Lamees Al Qassem, Di Wang, Hassan Barada, Ahmad Al-Rubaie, and Nawaf Almoosa. 2019. Automatic Arabic text summarization based on fuzzy logic. In Proceedings of the 3rd International Conference on Natural Language and Speech Processing, pages 42– 48, Trento, Italy. Association for Computational Linguistics.
  21. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. ↗
  22. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computational Linguistics. ↗
  23. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual Translation with Extensible Multilingual Pretraining and Finetuning. arXiv e-prints, page arXiv:2008.00401. ↗
  24. Zijian Győző Yang, Ádám Agócs, Gábor Kusper, and Tamás Váradi. 2021. Abstractive text summarization for Hungarian. In The 1st Conference on Information Technology and Data Science, pages 299– 316. find
  25. Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779–784, Brussels, Belgium. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kahla-2021-cross-lingual-fine-tuning-abstractive/