Elamur Research⌕
☰
home / papers / 2022 / AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization

AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization

Paper · 2022 · Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash et al.

Summary

Arabic summarization research has mostly produced extractive systems. The few abstractive ones were either trained without pretraining or reused BERT, where only the encoder carries pretrained weights and the decoder and cross-attention start from random values. The authors present AraBART, a BART Base–style seq2seq model whose encoder, decoder and the links between them are all pretrained together with BART's denoising objective. Pretraining used the AraBERT corpus with its preprocessing reversed. They fine-tuned it on seven Arabic Gigaword headline subsets, a mixed subset, and the summary and title tasks of XL-Sum. They compared it with CAMeLBERT2CAMeLBERT, mBART25, mT5 base and AraT5 base using ROUGE, BERTScore and human judgments of quality and faithfulness. AraBART scored best almost everywhere, even though most baselines are larger. Its advantage was largest on the most abstractive data. The result shows that a compact monolingual model pretrained end to end can beat big multilingual models on generation for a language with few resources.

Key points

  • The architecture has 6 encoder and 6 decoder layers with 768 hidden dimensions, for 139M parameters in total. To keep FP16 training stable, an additional layer normalization is placed after the final layer of both the encoder and the decoder. The sentencepiece vocabulary has 50K entries, was learned on a 20GB raw sample, and uses 99.99% character coverage.
  • The pretraining corpus is 73GB before tokenization and 96GB after. Noising follows BART: 30% of the text is masked in spans whose lengths come from a Poisson distribution with λ = 3.5, and sentence order is shuffled. Training took about 60h on 128 V100 GPUs, which covered 25 passes over the data.
  • Evaluation data:
    • seven Gigaword sources (AAW, AFP, AHR, HYT, NHR, QDS, XIN), each split 50K/5K/5K;
    • MIX, 60K examples drawn uniformly from those seven sources;
    • XL-Sum summaries (XL-S) and titles (XL-T).
  • Every model was fine-tuned identically: 3 epochs, peak learning rate 5e-5 with linear decay, beam size 3. The baselines range from 275M to 610M parameters.
  • On XL-S, AraBART's ROUGE-L is 2.9 points above mBART25's. On Gigaword the largest gap is 1.4 points, on AAW and HYT.

Interesting ideas

  • I1. The authors rate each dataset's abstractiveness by the share of summary n-grams absent from the source, then check whether the gap between models grows with that share. This shows where a model's advantage actually comes from.
  • I2. Before pretraining a generator, they undid the spacing and segmentation preprocessing of the corpus so that model outputs look like natural text.
  • I3. Pretraining every component jointly, cross-attention included, beat starting some connections from random weights.
  • I4. Results are reported per news source alongside a uniformly mixed pool, which exposes how much performance shifts with house style.
  • I5. All systems shared one fixed fine-tuning recipe, so the comparison rests on equal treatment rather than per-model tuning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Report the novel-n-gram rate for the text → graph → text compressor, so we can confirm it rephrases through meaning and does not just copy surface strings.
  • I2 (tokens): Make the tokenizer and lemma layer exactly reversible back to raw text, so later generation never inherits tokenization artifacts.
  • I3 (tree): When building gparse from pretrained parts, initialize the parser head and its interface layers from a joint pretraining stage instead of from scratch.
  • I4 (learn): Split evaluation sets by text source or genre and add a mixed pool, so a gain on one style cannot hide a loss on another.
  • I5 (learn): When comparing witness parsers or teacher tiers, train them under one shared budget and settings so that differences reflect the method and not tuning effort.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates124 works · 4,709 citations🔬Centre National de la Recherche ScientifiqueParis · France169 works · 62,488 citations🎓Athens University of Economics and BusinessAthens · Greece19 works · 9,652 citations🎓École PolytechniquePalaiseau · France15 works · 6,688 citations🔬Laboratoire d'Informatique de Paris-NordVilletaneuse · France10 works · 510 citations🎓Université Sorbonne Paris NordVilletaneuse · France9 works · 468 citations🔬Laboratoire d'Informatique de l'École PolytechniquePalaiseau · France3 works · 105 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksSummarization · 243 works

Cite

Cite as Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux and Michalis Vazirgiannis. 2022. AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.4

@inproceedings{eddine2022arabart,
  title = {{AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization}},
  author = {Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph {Le Roux} and Michalis Vazirgiannis},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.4},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.4}
}

References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Molham Al-Maleh and Said Desouki. 2020. Arabic text summarization using deep learning approach. Journal of Big Data, 7:1–17. ↗
  2. Abdullah Alshanqiti, Abdallah Namoun, Aeshah Alsughayyir, Aisha Mousa Mashraqi, Abdul Rehman Gilal, and Sami Saad Albouq. 2021. Leveraging distilbert for summarizing Arabic text: An extractive dual-stage approach. IEEE Access, 9:135594–135607. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  4. Alaidine Ben Ayed, Ismaïl Biskri, and Jean-Guy Meunier. 2021. Arabic text summarization via knapsack balancing of effective retention. Procedia Computer Science, 189:312–319. AI in Computational Linguistics. find
  5. Aqil Azmi and Suha Al-thanyyan. 2009. Ikhtasir — a user selected compression ratio Arabic text summarization system. In 2009 International Conference on Natural Language Processing and Knowledge Engineering, pages 1–7. find
  6. Riadh Belkebir and Ahmed Guessoum. 2015. A supervised approach to Arabic text summarization using adaboost. In New Contributions in Information Systems and Technologies, pages 227–236, Cham. Springer International Publishing. ↗
  7. Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive summarization with contrast candidate generation and https://huggingface.co/models ↗
  8. Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox. 2011. Multi-document Arabic text summarisation. 2011 3rd Computer Science and Electronic Engineering Conference (CEEC), pages 40–44. find
  9. Tarek El-Shishtawy and Fatma El-Ghannam. 2012. Keyphrase based Arabic summarizer (kpas). In 2012 8th International Conference on Informatics and Systems (INFOS), pages NLP–7–NLP–14. ↗
  10. Khalid N. Elmadani, Mukhtar Elgezouli, and Anas Showk. 2020. BERT fine-tuning for Arabic text summarization. CoRR, abs/2004.14135. ↗
  11. Ahmad Haboush, Ahmed Momani, Maryam Al-Zoubi, and Motassem Al-Tarazi. 2012. Arabic text summerization model using clustering techniques. World Comput Sci Inf Technol J, 2. Tahmid Hasan, Abhik Bhattacharjee, Md. find
  12. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics. find
  13. Muhammad Helmy, R. M. Vigneshram, Giuseppe Serra, and Carlo Tasso. 2018. Applying deep learning for Arabic keyphrase extraction. In Fourth International Conference On Arabic Computational Linguistics, ACLING 2018, November 17-19, 2018, Dubai, United Arab Emirates, volume 142 of Procedia Computer Science, pages 254–261. Elsevier. find
  14. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. LC-STS: A large scale Chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1967–1972, Lisbon, Portugal. Association for Computational Linguistics. find
  15. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  16. Mram Kahla, Zijian Győző Yang, and Attila Novák. 2021. Cross-lingual fine-tuning for abstractive Arabic text summarization. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 655–663, Held Online. INCOMA Ltd.
  17. Moussa Kamal Eddine, Guokan Shang, Antoine J-P Tixier, and Michalis Vazirgiannis. 2021a. Frugalscore: Learning cheaper, lighter and faster evaluation metricsfor automatic text generation. arXiv preprint arXiv:2110.08559. ↗
  18. Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021b. BARThez: a skilled pretrained French sequence-to-sequence model. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9369–9390, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  19. Ahmed Mostafa Khalil, Y. M. Wazery, Marwa E. Saleh, Abdullah Alharbi, and Abdelmgeid A. Ali. 2022. Abstractive Arabic text summarization based on deep learning. Computational Intelligence and Neuroscience, 2022:1566890. find
  20. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  23. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics. ↗
  24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  26. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. ↗
  27. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 3075–3081. AAAI Press.
  28. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 ↗
  29. Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics. find
  30. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics.
  31. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  32. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic gigaword fifth edition. https://doi.org/10.35111/p02g-rw14. ↗
  33. Aziz Qaroush, Ibrahim Abu Farha, Wasel T. Ghanem, Mahdi Washaha, and Eman Maali. 2021. An efficient single document Arabic text summarization using a combination of statistical and semantic features. J. King Saud Univ. Comput. Inf. Sci., 33:677–692. find
  34. Lamees Al Qassem, Di Wang, Hassan Barada, Ahmad Al-Rubaie, and Nawaf Almoosa. 2019. Automatic Arabic text summarization based on fuzzy logic. In Proceedings of the 3rd International Conference on Natural Language and Speech Processing, pages 42– 48, Trento, Italy. Association for Computational Linguistics.
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  36. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. ↗
  37. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. ↗
  38. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computational Linguistics. ↗
  39. Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation.
  40. Dima Suleiman and Arafat Awajan. 2020. Deep learning based abstractive arabic text summarization using two layers encoder and one layer decoder. Journal of Theoretical and Applied Information Technology, 98:3233. find
  41. Webz.io. 2016. Webz.io’s Arabic news articles. https://webz.io/free-datasets/ arabic-news-articles/. ↗
  42. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗
  44. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654–663, Melbourne, Australia. Association for Computational Linguistics. *Figure 3: The input news articles corresponding to the summaries in Figure 1*

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eddine-2022-arabart-pretrained-arabic-sequence-sequence/