Elamur Research⌕
☰
home / papers / 2021 / Arabic Compact Language Modelling for Resource Limited Devices

Arabic Compact Language Modelling for Resource Limited Devices

Paper · 2021 · Zaid Alyafeai and Irfan Ahmad

Summary

Large pretrained transformers are too heavy for phones and embedded hardware, and almost no compression work targets Arabic. The authors build small Arabic BERT-style models in three stages. First, they pretrain a mobileBERT-like teacher, IBERTA. It uses inverted-bottleneck blocks, and its training corpus is about 13 GB of diacritic-free, normalized text from four sources: news, Wikipedia, classical books and tweets. Second, they distill it into two students, MBERTA and MBERTA II. Third, they shrink all models with post-training quantization in tflite. The models are fine-tuned on six sentiment datasets, on PoS tagging with the PADT treebank, and on ARCD question answering, with AraBERT v1 as the baseline. The teacher has roughly half of AraBERT's parameters and saw about half as much pretraining text. It beats AraBERT on most sentence-classification sets but falls behind on token-level tasks (PoS, QA). The students keep classification accuracy close to the baseline but degrade strongly on QA. The work shows that a well-curated, diverse corpus of modest size can produce usable compact models for a morphologically rich, under-resourced language.

Key points

  • Pretraining data:

    • Roughly 10 GB drawn from the 1.5-billion-word news corpus (1.3 B tokens).
    • 1.5 GB of Wikipedia (148 M tokens).
    • 1 GB of Alshamila books (112 M tokens).
    • About 500 MB of Twitter (68 M tokens).
    • Tokenization is WordPiece, with sequence length 512 and 15% masking.

  • Teacher architecture (IBERTA): 24 layers and 4 attention heads, with the inner bottleneck reduced from 1024 to 256. Two variants were trained, with vocabularies of 30K and 50K; the 50K version became the teacher. Training used the lamb optimizer for about 2M steps on a TPUv3-8.

  • Model sizes:

    ModelParametersFine-tuned on diskQuantized on disk
    AraBERT135.2 M1.6 GB516 MB
    IBERTA66.7 M763 MB255 MB
    MBERTA (4 FFN per block)23.7 M272 MB91 MB
    MBERTA II (1 FFN per block)14.2 M163 MB55 MB

  • Distillation procedure: 500K steps of attention transfer plus feature-map transfer, followed by 2M steps of pretraining distillation. The objective mixes the MLM loss and the KD loss through a weight α and adds NSP.

  • Results against AraBERT:

    • IBERTA wins on AJGT (93.3 vs 89.7), TEAD and ASTD.
    • IBERTA loses on PADT F1 (78.8 vs 83.8) and ARCD F1/EM (58.4/26.1 vs 62.2/29.6).
    • The students drop to about 50 F1 on ARCD.

Interesting ideas

  • I1. The student is supervised block by block, copying both the teacher's attention maps and each block's internal features, instead of matching only final outputs, which gives a deep, thin model much richer guidance.
  • I2. The pretraining-distillation loss interpolates between ground-truth targets and teacher predictions with a single weight, so trust in the teacher becomes a tunable knob.
  • I3. To stay fair to the baseline, the authors deliberately skipped their own preprocessing at fine-tuning time, accepting a likely handicap rather than an advantage.
  • I4. The breakdown shows that compression costs little on sentence-level labels but a lot on token-level tasks (PoS, span QA), so an averaged score would hide the damage.
  • I5. Post-training quantization was applied as a cheap last step, cutting on-disk size about threefold with no retraining.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): a CPU-sized parser could be distilled from gparse by matching its per-layer attention and hidden states, not only its arc and label scores.
  • I2 (learn): when training on silver trees from witness parsers or LLM teacher tiers, a similar weight could balance gold treebank loss against the teacher's soft distribution.
  • I3 (learn): comparisons between parsers or taggers should feed every system an identical raw input pipeline, so that gains are not artefacts of normalization.
  • I4 (tokens): any slimmed-down tagger or lemmatizer should be judged on per-token metrics and hard cases, not on aggregate accuracy alone.
  • I5 (tree): gparse weights could be quantized for deployment, with LAS re-measured to confirm the drop is acceptable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Zaid Alyafeai and Irfan Ahmad. 2021. Arabic Compact Language Modelling for Resource Limited Devices. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alyafeai2021arabic,
  title = {{Arabic Compact Language Modelling for Resource Limited Devices}},
  author = {Zaid Alyafeai and Irfan Ahmad},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.6/}
}

References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ElJundi, O., Antoun, W., El Droubi, N., Hajj, H., El-Hajj, W., and Shaban, K. (2019). hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77.
  2. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020). Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  3. Elnagar, A., Khalifa, Y. S., and Einea, A. (2018). Hotel arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer. ↗
  4. Abid, A., Ali, W., Farooq, M. S., Farooq, U., Sabir, N., and Abid, K. (2020). Semi-automatic classification and duplicate detection from human loss news corpus. IEEE Access. ↗
  5. Alkhulaifi, A., Alsahli, F., and Ahmad, I. (2020). Knowledge distillation in deep learning and its applications. arXiv preprint arXiv:2007.09029. ↗
  6. Gerz, D., Vulić, I., Ponti, E., Naradowsky, J., Reichart, R., and Korhonen, A. (2018). Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. Transactions of the Association for Computational Linguistics, 6:451–465. ↗
  7. Alomari, K. M., ElSherif, H. M., and Shaalan, K. (2017). Arabic tweets sentimental analysis using machine learning. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 602–610. Springer. find
  8. Hooker, S., Moorosi, N., Clark, G., Bengio, S., and Denton, E. (2020). Characterising bias in compressed models. arXiv preprint arXiv:2010.03058. ↗
  9. Aly, M. and Atiya, A. (2013). Labr: A large scale arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. ↗
  10. Kudo, T. and Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. ↗
  11. Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  12. Martin, L., Muller, B., Suárez, P. J. O., Dupont, Y., Romary, L., de la Clergerie, É. V., Seddah, D., and Sagot, B. (2019). Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. ↗
  13. Baly, R., Khaddaj, A., Hajj, H., El-Hajj, W., and Bashir Shaban, K. (2018). Arsentd-lev: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. OSACT3. ↗
  14. Mirzadeh, S.-I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. (2019). Improved knowledge distillation via teacher assistant. arXiv preprint arXiv:1902.03393. ↗
  15. Cui, J., Kingsbury, B., Ramabhadran, B., Saon, G., Sercu, T., Audhkhasi, K., Sethy, A., Nussbaum-Thom, M., and Rosenberg, A. (2017). find
  16. Knowledge Mozannar, H., Hajal, K. E., Maamary, E., and Hajj, H. (2019). Neural arabic question answering. arXiv preprint arXiv:1906.05394. ↗
  17. Nabil, M., Aly, M., and Atiya, A. (2015). Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2515–2519. ↗
  18. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  19. Radford, A. et al. (2018). Improving language understanding by generative pre-training. URL understanding paper. pdf. find
  20. Raffel, C. et al. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. preprint, arXiv. shamela (2005). shamela. ↗
  21. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2020). Qbert: Hessian based ultra low precision quantization of bert. In AAAI, pages 8815–8821. find
  22. Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355. ↗
  23. Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. (2020). Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984. ↗
  24. Tian, Y., Krishnan, D., and Isola, P. (2019). Contrastive representation distillation. arXiv preprint arXiv:1910.10699. ↗
  25. Yang, C., Xie, L., Su, C., and Yuille, A. L. (2019). Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2859–2868. find
  26. You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. (2019). Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962. ↗
  27. Zadeh, A. H. and Moshovos, A. (2020). Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. arXiv preprint arXiv:2005.03842. ↗
  28. Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. (2019). Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188. ↗
  29. Zeman, D. (2015). Universal dependencies. https://github.com/UniversalDependencies/UD_ Arabic-PADT . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alyafeai-2021-arabic-compact-language-modelling-resource/