Elamur Research⌕
☰
home / papers / 2021 / ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective

ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective

Paper · 2021 · Sultan Alrowili and K. Vijay-Shanker

Summary

Pre-training large Transformer encoders for Arabic, as done for AraBERT and AraELECTRA, is expensive. That cost puts large models out of reach for many research groups. The authors look for an architecture that grows capacity without raising pre-training cost. They combine Funnel Transformer with the ELECTRA replaced-token-detection objective. Funnel Transformer pools hidden states into progressively shorter sequences across blocks, then upsamples them back to full length.

Two models, B4-4-4 and B6-6-6, were trained on about 45GB of Arabic text. They use a 50K vocabulary built without morphological pre-segmentation. These models were fine-tuned on question answering (ARCD, the Arabic part of TyDi QA) and on sentiment tasks (HARD, AJGT, ArSarcasm). The B6-6-6 model beats or matches stronger baselines on QA, including AraBERT large and AraELECTRA. Its pre-training cost is well under half that of AraELECTRA. The models and code are released publicly. The work shows that architectural efficiency can stand in for raw compute for a mid-resource language.

Key points

  • The pre-training data combines Arabic Wikipedia (1.3GB), the Abu El-Khair corpus (14GB), and unshuffled Arabic OSCAR (30GB). The vocabulary has 50K entries and was built without the Farasa segmenter.
  • B6-6-6 trained in about 2 days 10 hours on a TPUv3-32, with batch size 1024 and learning rate 4e-4. B4-4-4 took about 7.5 days on a TPUv3-8. AraELECTRA needed 24 days on a TPUv3-8, so the relative costs are 0.40x and 0.31x.
  • On TyDi QA, B6-6-6 reaches EM 75.35 and F1 87.21, against 74.91 and 86.68 for AraELECTRA. On ARCD it gets F1 72.70, the highest among the compared models.
  • The smaller B4-4-4 trails noticeably on ARCD (F1 67.70). On a 2.5M-word CCNet sample it also shows a higher loss than B6-6-6 (11.58 vs 11.12).
  • Fine-tuning used a grid search over learning rate, batch size, layer-wise decay, sequence length, and epochs. Each reported score is the best of five runs with fixed, published seeds.

Interesting ideas

  • I1. Funnel-style compression shortens the token sequence in the middle layers, where full resolution is redundant, then upsamples it so each token still gets an output; this frees compute to spend on depth or width.
  • I2. ELECTRA-style training gives every position a binary "original or replaced?" label, so the learning signal is much denser than masking a small fraction of tokens.
  • I3. Training cost is reported as steps times batch size, normalized to a baseline, and is shown alongside wall-clock time and hardware, so efficiency claims can be compared across setups.
  • I4. Fixing and publishing the random seeds for repeated fine-tuning runs makes "best of N" results reproducible by others.
  • I5. The paper's comparison includes AraBERT versions with and without pre-segmentation, and the variant without it did not lose out on QA; this suggests subword vocabularies can absorb rich morphology without a separate segmenter.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could pool token states in its middle encoder layers and upsample before the per-token head and label scorers, cutting GPU cost on long documents.
  • I2 (learn): a small generator could corrupt arcs or labels in silver trees, and a discriminator could learn to flag each altered arc; this would yield per-arc confidence scores to steer repair-rule induction.
  • I3 (learn): parser and teacher-tier experiments could log a normalized compute ratio next to accuracy, making cheaper-versus-better trade-offs explicit in evaluation reports.
  • I4 (learn): perceptron and gparse runs could use a published seed list, and reports could include the spread across seeds, not only the best run, to keep comparisons honest.
  • I5 (tokens): this is a prompt to test whether the en crate's explicit morphological splitting actually helps downstream parsing, or whether it mainly adds error sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of DelawareNewark · United States20 works · 1,056 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Sultan Alrowili and K. Vijay-Shanker. 2021. ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective. Findings of the Association for Computational Linguistics: EMNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.108

@inproceedings{alrowili2021arabictransformer,
  title = {{ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective}},
  author = {Sultan Alrowili and K. Vijay-Shanker},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.108},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.108}
}

References

28 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics.
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  4. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the wanlp 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop.
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454– 470. ↗
  8. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020b. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  10. Abdelghani Dahou, Mohamed Abd Elaziz, Junwei Zhou, Shengwu Xiong, and Rodolfo Zunino. 2019. Arabic sentiment classification using convolutional neural network and differential evolution algorithm. Intell. ↗
  11. Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. In Advances in Neural Information Processing Systems, volume 33, pages 4271–4282. Curran Associates, Inc. ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Ibrahim Abu El-khair. 2016. 1.5 billion words arabic corpus. ↗
  14. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. 2018. Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications, pages 35–52. Springer International Publishing, Cham. find
  15. Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for arabic sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop.
  16. William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better text generation via filling in the . In International Conference on Learning Representations. find
  17. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pages 2672–2680. Curran Associates, Inc. find
  18. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. ↗
  19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  21. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  22. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics.
  23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  24. Ali Safaya. 2020. Arabic-albert. find
  25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc.
  27. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.
  28. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alrowili-2021-arabictransformer-efficient-large-arabic-language/