Elamur Research⌕
☰
home / papers / 2021 / AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding

AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding

Paper · 2021 · Wissam Antoun, Fady Baly and Hazem Hajj

Summary

Earlier Arabic pretrained encoders, including AraBERT, Arabic-BERT and ARBERT, all relied on masked language modeling. This objective uses data inefficiently, because only the roughly 15% of masked positions contribute to the loss. The authors adapt the ELECTRA approach to Arabic. A small BERT-style generator fills in masked tokens, and a full-size discriminator, AraELECTRA, labels every token as original or substituted, so all positions produce a learning signal. To isolate the effect of the objective, they reuse the AraBERTv0.2 pre-training corpus (77GB, 8.8 billion words, mostly news) and its wordpiece vocabulary. The resulting base-size model (136M parameters) is fine-tuned and tested on four benchmarks:

  • question answering: Arabic TyDiQA and ARCD
  • Levantine tweet sentiment: ArSenTD-LEV
  • named-entity recognition: ANERcorp

AraELECTRA outperforms every base-size competitor on all tasks. It also beats the large models on three of the four benchmarks and trails slightly only on ARCD. This suggests that a more sample-efficient objective can partly replace model scale. The model and code are publicly released.

Key points

  • Architecture: The discriminator has 12 layers, 12 heads, hidden size 768 and 512-token inputs. The generator is a small BERT with 12 layers, 4 heads and hidden size 256, and a linear layer projects the 768-dim embeddings down to it.
  • Pre-training corpus: OSCAR, the 1.5B words Arabic Corpus, Arabic Wikipedia (September 2020), OSIAN and As-Safir news articles.
  • Pre-training setup: 2M steps with batch size 256, learning rate 2e-4 and 10,000 warm-up steps, taking 24 days on a TPUv3-8.
  • Results:
    • TyDiQA: 74.91 EM / 86.68 F1, versus 73.72 / 86.03 for AraBERTv0.2-large.
    • ARCD: 37.03 EM / 71.22 F1, slightly behind Arabic-ALBERT-xlarge on EM (37.75) and AraBERTv0.2-large on F1 (71.32).
    • ArSenTD-LEV: 57.20 F1.
    • ANERcorp: 83.95 F1.
  • Fine-tuning: Only three learning rates (2e-5, 3e-5, 5e-5) were tried for every model, and a linear head sits on top of the discriminator.

Interesting ideas

  • I1. Hold the data and tokenizer fixed when comparing objectives. Training on exactly the same corpus and vocabulary as the baseline means any gain can be credited to the pre-training task itself. This is a useful discipline for any comparison of objectives in a new language.
  • I2. Validate on text written after the corpus snapshot. Using Wikipedia articles published after the training dump gives a held-out set that is unlikely to overlap with the pre-training data. The same check against leakage works for any web-scale corpus.
  • I3. Pair a lightweight generator with a full discriminator. The generator is much narrower and connects to the shared embeddings through a projection layer, which keeps replaced-token-detection training affordable. This setup suits low-resource languages where compute is limited.
  • I4. Use a tiny shared hyperparameter grid. Giving every model the same three learning rates keeps a multi-model benchmark cheap and comparable, at the cost of possibly under-tuning some models.
  • I5. Explain benchmark gaps through data quality. The authors attribute weaker ARCD scores to machine-translated training data with foreign scripts, and note HTML artifacts in TyDiQA. Checking datasets for translation noise and markup is worth doing before blaming the model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of BeirutBeirut · Lebanon17 works · 1,847 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksSemitic languages · 1,484 works

Cite

Cite as Wissam Antoun, Fady Baly and Hazem Hajj. 2021. AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{antoun2021araelectra,
  title = {{AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding}},
  author = {Wissam Antoun, Fady Baly and Hazem Hajj},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.20/}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, and Lyle Ungar. 2020b. Toward micro-dialect identification in diaglossic and code-switched environments. arXiv preprint arXiv:2010.04900. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Ramy Baly, Alaa Khaddaj, Hazem Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2018. Arsentd-lev: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 37. ↗
  5. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́Ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  6. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J. Jansen. 2020. Improving Arabic text categorization using transformer training diversification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 226–236, Barcelona, Spain (Online). Association for Computational Linguistics. find
  7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. ↗
  8. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020b. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  11. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  12. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77. find
  13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27:2672–2680. find
  14. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  16. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy. Association for Computational Linguistics.
  17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  18. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics.
  19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗
  20. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  21. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  22. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/