Elamur Research⌕
☰
home / papers / 2023 / On Enhancing Fine-Tuning for Pre-trained Language Models

On Enhancing Fine-Tuning for Pre-trained Language Models

Paper · 2023 · Abir Betka, Zeyd Ferhat, Riyadh Barka et al.

Summary

Fine-tuning a pre-trained transformer for every new task costs a lot of compute time and energy. This paper tests a simple remedy: keep some encoder layers frozen and update only the remaining ones. The authors use four Arabic BERT-base models: AraBERTv0.2, CAMelBERT, QARiB and MARBERT. They first run an exhaustive grid search on the MADAR dialect identification task, which has 26 labels (city dialects plus MSA). The search tries every pairing of a number of frozen bottom layers with a number of frozen top layers and records F1 for each. Training all layers gives the best score, but freezing up to three lower and four upper layers stays very close to it. The chosen setting is then applied to seven Arabic classification and similarity tasks, most of them built on tweets. Fine-tuning time drops by as much as half, while F1 usually falls by only a few points, and in some MARBERT runs it actually improves. The method is a cheap way to speed up experimentation when GPU budget is limited.

Key points

  • Models: AraBERTv0.2, CAMelBERT, QARiB and MARBERT, all BERT-base Arabic encoders pre-trained on MSA, dialectal and (for CAMelBERT) classical Arabic.
  • Tasks:
    • anger detection (Affect in Tweets, SEC, binary),
    • semantic question similarity (0–5 scores),
    • IDAT@FIRE2019 irony detection,
    • MADAR Subtask 1 dialect detection (reduced to Algiers vs Amman downstream),
    • OSACT4 offensive language and hate speech,
    • a 3-way XNLI-style sentence-pair task with 7,500 pairs.
  • The configuration search was a full grid over (n top, m bottom) frozen layers on the 26-class MADAR task. Freezing 3 bottom and 4 top layers came close to full fine-tuning.
  • Fine-tuning time fell by up to 50%. The abstract reports a performance loss under 5%, while the results section says losses rarely exceeded 6%.
  • The paper is inconsistent about how many tasks it covers (it mentions both four ALUE tasks and seven tasks). Results appear only in appendix tables and figures, without variance or significance analysis.

Interesting ideas

  • I1. Freeze both ends of the encoder rather than only the bottom: lower layers hold general linguistic patterns and some upper layers can also stay fixed, which saves backpropagation cost with little quality loss.
  • I2. Tune the freezing configuration once with a grid search on a single large, many-label task, then reuse that setting on smaller tasks instead of searching again for each.
  • I3. Report the speed/quality trade-off as two numbers side by side (runtime saved vs F1 lost) so a cheaper setting can be chosen on purpose.
  • I4. Keeping much of the pre-trained network fixed is argued to limit drift from pre-training knowledge, which the authors suspect may help on out-of-domain text. This remains a hypothesis they leave for future work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when fine-tuning the transformer encoder inside gparse on a new treebank or UD dialect conversion, freezing a few bottom and top layers could cut GPU time per retraining cycle.
  • I2 (learn): run one freeze-pattern sweep on the biggest silver-data parsing set and carry the result over to smaller tagging or frame-labeling fine-tunes.
  • I3 (learn): evaluation reports for each retrained model could log training time next to LAS or accuracy, making it clear when a faster setup is acceptable.
  • I4 (learn): this is worth testing with negative controls on out-of-domain fables and school texts, comparing frozen and fully tuned parsers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar144 works · 14,175 citations🎓University of BiskraBiskra · Algeria5 works · 7 citations🎓University Mohamed Boudiaf of M'silaM'Sila · Algeria2 works · 3 citations

Also on the byline: Virginia Cancer Specialists · Vigilant Cyber Systems.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 works

Cite

Cite as Abir Betka, Zeyd Ferhat, Riyadh Barka, Selma Boutiba, Zineddine Kahhoul, Tiar Lakhdar, Ahmed Abdelali and Habiba Dahmani. 2023. On Enhancing Fine-Tuning for Pre-trained Language Models. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.33

@inproceedings{betka2023enhancing,
  title = {{On Enhancing Fine-Tuning for Pre-trained Language Models}},
  author = {Abir Betka, Zeyd Ferhat, Riyadh Barka, Selma Boutiba, Zineddine Kahhoul, Tiar Lakhdar, Ahmed Abdelali and Habiba Dahmani},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.33},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.33}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the ACL and the 11th IJCNL Processing, Online.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. Houda Bouamor, Sabit Hassan, and Nizar Habash. The MADAR shared task on arabic fine-grained dialect identification. In Proceedings of the 4th Arabic Natural Language Processing Workshop. ↗
  4. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. find
  5. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
  7. Ibrahim Abu El-Khair. 2016. Abu el-khair corpus: A modern standard arabic corpus. International Journal of Recent Trends in Engineering & Research (IJRTER), 2(11):5–13. find
  8. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Annual Meeting of the FIRE, pages 10–13. find
  9. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  10. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net. find
  11. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on OSACT, with a Shared Task on Offensive Language Detection, pages 48–52.
  12. XiPeng Qiu, TianXiang Sun, YiGe Xu, YunFan Shao, Ning Dai, and XuanJing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872– 1897.
  13. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  14. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429. find
  15. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. Alue: Arabic language understanding evaluation. In Proceedings of the Sixth WANLP, pages 173–184.
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  17. Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation. arXiv preprint arXiv:2001.11314. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  19. Elizabeth D Liddy. 2001. Natural language processing. find
  20. Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. 2021. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXiv:2102.01386. ↗
  21. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. Adv. in neural information processing systems, 30.
  22. Chenghao Yang and Xuezhe Ma. 2022. Improving stability of fine-tuning pretrained language models via component-wise gradient norm clipping. In Proceedings of the 2022 Conference on EMNLP. find
  23. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the fourth arabic natural language processing workshop, pages 175–182.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/betka-2023-enhancing-fine-tuning-pre-trained/