Elamur Research⌕
☰
home / papers / 2023 / Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation

Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation

Paper · 2023 · AbdelRahim Elmadany, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed

Summary

Arabic natural language generation has lacked a single, broad toolkit. Available public tools mostly cover only translation into Arabic. The authors address this with AraT5 v2, a new Arabic text-to-text Transformer, and OCTOPUS, a toolkit that runs eight generation tasks from one model. AraT5 v2 was pretrained on about 250GB of unlabeled text spanning Modern Standard, Classical and dialectal Arabic, with input length raised to 2,048 tokens. It was then adapted with labeled data from eight tasks in the Dolphin benchmark. The study compares three training regimes: span-corruption pretraining alone, single-task and multitask finetuning, and a joint setup where pretraining and finetuning share one objective. According to the authors, the resulting models beat strong baselines such as mT5, mT0, AraBART and AraT5 v1 by wide margins. OCTOPUS packages the model as a Python library, command-line tools, a web demo and a Colab notebook. It gives Arabic NLP a reusable baseline and a practical component for building other systems.

Key points

  • AraT5 v2 was pretrained on roughly 250GB of text (about 25.6B tokens), with no tweets. AraT5 v1 used 248GB, of which 71.77% came from Twitter, and was limited to 512-token sequences.
  • Unlabeled sources:
    • AraNews v2, El-Khair, Gigaword, OSIAN, Arabic Wikipedia, Hindawi Books, OSCAR Egyptian and AraC4;
    • the OpenITI corpus (v1.6), about 11K Islamic books, for Classical Arabic.
  • Labeled data comes from eight of Dolphin's thirteen tasks (Dolphin has 40 datasets in total):
    • diacritization;
    • GEC, using QALB 2014, QALB 2015 and ZAEBUC;
    • news title generation;
    • paraphrasing;
    • question answering, using ARCD, MLQA, XQuAD and TyDiQA;
    • question generation;
    • summarization;
    • transliteration.
  • Machine translation was deliberately left out because the Turjuman toolkit already handles it.
  • OCTOPUS offers several entry points:
    • commands: an interactive command, a general octopus command and per-task commands such as octopus-diacritize;
    • decoding: greedy, beam, top-k and nucleus decoding, with beam size 5 and 2,048 maximum length as defaults;
    • dataframes: integration with Pandas, PySpark and Dask.

Interesting ideas

  • I1. A short textual task tag added to the input tells one shared model which job to perform, so many tasks can live in a single set of weights.
  • I2. Question generation was built by reusing the same passage–answer–question triplets as QA, simply reversing the direction of the mapping.
  • I3. The authors scoped the toolkit by excluding tasks already covered well by a sibling tool (MT via Turjuman), keeping effort on gaps.
  • I4. They shifted pretraining data away from noisy social media toward varied registers, including classical texts, to broaden linguistic coverage.
  • I5. The library returns processing logs alongside generated text and exposes decoding choices with documented defaults.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A gparse-style transformer could be trained on several jointly supervised heads or treebank dialects, each marked with an explicit task or dialect tag instead of being a separate model.
  • I2 (answer): Fable and school-test QA data could be flipped to generate new questions, giving self-play material and negative controls for the readers at no extra annotation cost.
  • I3 (back): Generation work could focus on graph-to-text where Mova is unique, and leave plain surface translation to existing tools rather than duplicating them.
  • I4 (tree): Silver treebank material could deliberately mix older literary English, such as fables and tales, with modern news, so the parser sees the registers that Mova's readers actually face.
  • I5 (back): The text → graph → text compressor could likewise return a trace of decoding decisions next to each output, which supports the system's explainability requirement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksText generation · 202 works

Cite

Cite as AbdelRahim Elmadany, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed. 2023. Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.20

@inproceedings{elmadany2023octopus,
  title = {{Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation}},
  author = {AbdelRahim Elmadany, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.20},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.20}
}

References

52 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2019. AraNet: A Deep Learning Toolkit for Arabic Social Media. arXiv preprint arXiv:1912.13072. ↗
  2. Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building arabic paraphrasing benchmark. In Proceedings of the Second International conference on Data Science E-learning and Information Systems (DATA’ 2019), pages 1–5. find
  3. Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2019. Anetac: Arabic named entity transliteration and classification dataset. arXiv preprint arXiv:1907.03110. ↗
  4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637.
  5. Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong bin Kang, and Rifat Shahriyar. 2021. Crosssum: Beyond english-centric cross-lingual abstractive text summarization for 1500+ language pairs. CoRR, abs/2112.08804. ↗
  6. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. find
  7. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. ↗
  8. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2021. An arabic multi-source news corpus: Experimenting on single-document extractive summarization. Arabian Journal for Science and Engineering, 46:3925–3938. ↗
  9. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  10. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA). https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye 21 https://sites.research.google/trc/about/ ↗
  11. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization.
  12. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019b. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. ↗
  13. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  14. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184. ↗
  15. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritization using deep neural networks.
  16. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. ↗
  17. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198.
  18. Kamel Gaanoun, Abdou Naira, Anass Allak, and Imade Benelallam. 2022. Automatic Text Summarization for Moroccan Arabic Dialect Using an Artificial Intelligence Approach, pages 158–177. find
  19. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  20. Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. ↗
  21. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. ↗
  22. Nisarg Jhaveri, Manish Gupta, and Vasudeva Varma. 2019. clstk: The cross-lingual summarization toolkit. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 766–769. find
  23. Saad Khan, Jesse Hamer, and Tiago Almeida. 2021. Generate: A nlg system for educational content creation. In EDM. find
  24. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. ↗
  25. Ivano Lauriola, Alberto Lavelli, and Fabio Aiolli. 2022. An introduction to deep learning in natural language processing: Models, techniques, and tools. Neurocomputing, 470:443–456. ↗
  26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019a. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, pages 7871–7880. ↗
  27. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60.
  28. Yuval Merhav and Stephen Ash. 2018. Design Challenges in Named Entity Transliteration. In Proceedings of the 27th International Conference on Computational Linguistics, pages 630–640, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  29. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar. Association for Computational Linguistics.
  30. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  31. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. ↗
  32. El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2022a. Jasmine: Arabic gpt models for few-shot learning. arXiv preprint arXiv:2212.10755. ↗
  33. El Moatez Billah Nagoudi, Ahmed El-Shangiti, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2023. Dolphin: A challenging and diverse benchmark for arabic nlg. arXiv preprint arXiv:2305.14989. ↗
  34. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022b. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  35. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022c. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  36. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022d. TURJUMAN: A public toolkit for neural Arabic machine translation. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 1–11, Marseille, France. European Language Resources Association. ↗
  37. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Tariq Alhindi, and Hasan Cavusoglu. 2020. Machine generation and detection of arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  38. Lorenz Nigst, Maxim Romanov, Sarah Bowen Savant, Masoumeh Seydi, and Peter Verkinderen. 2020. Openiti: a machine-readable corpus of islamicate texts. http://doi.org/10.5281/zenodo, 4075046. ↗
  39. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. Camel tools: An open source python toolkit for arabic natural language processing.
  40. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland. European Language Resources Association (ELRA).
  41. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026– 8037. ↗
  42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
  43. Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015. The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 26–35, Beijing, China. Association for Computational Linguistics.
  44. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. ↗
  45. Yves Scherrer. 2020. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6868–6873, Marseille, France. European Language Resources Association (ELRA), European Language Resources Association. find
  46. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  47. Bashar Talafha, Analle Abuammar, and Mahmoud Al-Ayyoub. 2021. Atar: Attention-based lstm for arabizi transliteration. International Journal of Electrical and Computer Engineering, 11:2327–2334. ↗
  48. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  49. Daniel Varab and Natalie Schluter. 2021. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150–10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  50. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280. ↗
  51. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
  52. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elmadany-2023-octopus-multitask-model-toolkit-arabic/