Elamur Research⌕
☰
home / papers / 2021 / AraGPT2: Pre-Trained Transformer for Arabic Language Generation

AraGPT2: Pre-Trained Transformer for Arabic Language Generation

Paper · 2021 · Wissam Antoun, Fady Baly and Hazem Hajj

Summary

Arabic NLP had strong pre-trained models for understanding tasks, mostly masked-LM encoders such as AraBERT, but lacked a capable generative model. Earlier Arabic GPT-2 attempts had only adapted the English GPT-2 checkpoint using Arabic Wikipedia. The authors trained AraGPT2 from scratch on Arabic, a family of causal transformer decoders in four sizes from 135M to 1.46B parameters. The training corpus is about 77GB of filtered text drawn from OSCAR, Arabic Wikipedia, the 1.5B words Arabic Corpus, OSIAN and As-safir news. To fit the largest models on TPUs, they switched to a GROVER-style block and the memory-light Adafactor optimizer.

The largest model, AraGPT2-mega, reaches a perplexity of 29.8 on recent Wikipedia articles held out from training. In a human study, its generated news fooled roughly 60% of readers. Because such convincing output could be misused, the authors also fine-tuned an AraELECTRA-based detector that reliably flags generated text. All models and the detector are publicly released, giving Arabic researchers both a generator and a safeguard.

Key points

  • Model sizes. There are four variants: base (135M), medium (370M), large (792M) and mega (1.46B). All use a 1024-token context.
    • Base and medium use the GPT2 architecture with the LAMB optimizer.
    • Large and mega use the GROVER architecture with Adafactor.
    • Training ran on TPUv3 hardware.
  • Data and tokenization.
    • The corpus is 77GB, about 8.8B words after preprocessing.
    • Documents were discarded if they contained under 3 sentences or if duplicate sentences made up over 20% of their content.
    • Special tokens stood in for user mentions, email addresses and URLs, while diacritics and elongations were removed.
    • A byte-level BPE tokenizer with a 64000-token vocabulary produced about 9.7M training examples of 1024 tokens each.
  • Evaluation. Perplexity was measured on Arabic Wikipedia articles published after August 2020, and mega scored best at 29.8. In zero-shot tests, mega answered 25% of trivia questions correctly but failed at English-to-Arabic translation.
  • Human study. 74 Arabic speakers judged 8 news articles, 4 generated and 4 human-written.
    • About 60% were fooled by the generated texts, and longer passages fooled readers more often.
    • Human-written articles were identified only at chance level, around 50%.
  • Detector. The AraELECTRA-based detector was trained on 1500 human and 1500 generated articles. It reached an F1 of 98.7% on long texts (500 tokens) and 94.9% on short texts (150 tokens).

Interesting ideas

  • I1. Swap the optimizer when memory is the limit. The authors replaced Adam/LAMB with Adafactor, which stores factored second-moment statistics, to make billion-parameter training fit on the hardware. This could help any group training a large model, for example a Ukrainian LM, on limited accelerator memory.
  • I2. Build a time-split validation set. Validating on Wikipedia articles published after the data cutoff gives a cheap held-out set that is very unlikely to overlap with the training data. The same approach works for any language that has a dated Wikipedia dump.
  • I3. Include a human-written control group. The human study asked readers to judge real articles as well as generated ones. This revealed that people misjudge genuine text about half the time, which puts the "fooled" rate in proper context and could improve other human evaluations.
  • I4. Release the detector with the generator. The authors fine-tuned a discriminator pre-trained on the same corpus as the generator and published it alongside the models. This is a practical responsible-release pattern, and it relies on generated text being more predictable than human writing.
  • I5. Report detection by text length. Measuring detection accuracy separately for short and long texts showed that short texts are harder to catch. This kind of split is useful when designing detectors for short formats such as tweets or reviews.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon17 works · 1,847 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 1,484 worksText generation · 145 works

Cite

Cite as Wissam Antoun, Fady Baly and Hazem Hajj. 2021. AraGPT2: Pre-Trained Transformer for Arabic Language Generation. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{antoun2021aragpt2,
  title = {{AraGPT2: Pre-Trained Transformer for Arabic Language Generation}},
  author = {Wissam Antoun, Fady Baly and Hazem Hajj},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.21/}
}

References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, and Lyle Ungar. 2020. Toward micro-dialect identification in diaglossic and code-switched environments. arXiv preprint arXiv:2010.04900. ↗
  2. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Dahak, Stephan Vogel, and James Glass. 2014. A complete kaldi recipe for building arabic speech recognition systems. In 2014 IEEE spoken language technology workshop (SLT), pages 525–529. IEEE. find
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020a. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020b. Araelectra: Pre-training text discriminators for arabic language understanding. ↗
  5. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. ↗
  6. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J. Jansen. 2020. Improving Arabic text categorization using transformer training diversification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 226–236, Barcelona, Spain (Online). Association for Computational Linguistics. find
  7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. ↗
  8. Nick Doiron. 2020. Making a mini gpt-2 with dialect prompts. find
  9. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  10. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77. find
  11. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043. ↗
  12. Injy Hamed, Mohamed Elmahdy, and Slim Abdennadher. 2017. Building a first language model for code-switch arabic-english. Procedia Computer Science, 117:208–216. find
  13. Fouzi Harrag, Maria Dabbah, Kareem Darwish, and Ahmed Abdelali. 2020. Bert transformer model for detecting Arabic GPT2 auto-generated tweets. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 207–214, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  15. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  16. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. ↗
  17. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020b. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  18. Abed Khooli. 2020. gpt2-small-arabic. find
  19. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  20. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  22. Sameer Khurana, Ahmed Ali, and James Glass. 2019. Darts: Dialectal arabic transcription system. arXiv preprint arXiv:1909.12163. ↗
  23. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  25. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  26. Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020. Empathy-driven Arabic conversational chatbot. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 58–68, Barcelona, Spain (Online). Association for Computational Linguistics. find
  27. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics.
  28. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland. European Language Resources Association (ELRA).
  29. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics.
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. find
  32. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  33. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235. ↗
  34. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. ↗
  35. Jörgrg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA). find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998–6008.
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  38. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962. ↗
  39. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in neural information processing systems, pages 9054–9065. ↗
  40. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. ↗
  41. Zhibo Zhang. 2019. Gpt2-ml: Gpt-2 for multiple languages. https://github.com/imcaspar/ gpt2-ml . Generated Samples from A RA GPT2 A ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/