Elamur Research
☰
home / papers / 2024 / LAraBench: Benchmarking Arabic AI with Large Language Models

LAraBench: Benchmarking Arabic AI with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.eacl-long.30.pdf

Summary

Released large language models are seldom tested against strong task-specific systems for individual languages. LAraBench fills this gap for Arabic. It measures how well general-purpose LLMs handle Arabic text and speech, covering both Modern Standard Arabic and dialects, compared with the best existing specialised models. The authors gathered 61 public datasets covering 33 tasks, organised into nine groups. These range from segmentation, POS tagging, parsing and NER to MT, sentiment, harmful-content detection, QA, ASR and TTS. They prompted GPT-3.5-turbo, GPT-4, BLOOMZ and Jais-13b-chat for text, and evaluated Whisper and USM for speech, using zero-shot and few-shot setups. Specialised SOTA systems usually stayed ahead. GPT-4 was the strongest LLM, and giving it three examples narrowed the gap considerably, especially on semantic and QA tasks. Open models lagged behind closed ones, and dialects remained a weak spot. The results show where LLMs are a reasonable choice for Arabic AI. The released resources are integrated into the LLMeBench framework.

Key points

  • Scale: 98 test setups and over 330 experiment runs, using about 296K data points, roughly 46 hours of speech and 30 TTS sentences.
  • Averaged over 21 NLP tasks, the scores as read from Figure 1 are: BLOOMZ 0.429, GPT-3.5 0.535, GPT-4 zero-shot 0.641, GPT-4 3-shot 0.675 and SOTA 0.700.
  • For speech, USM roughly matched SOTA on MSA ASR and coped better with code-switching. Fine-tuning Whisper Large-v2 on just 2 hours of audio greatly reduced its gap to SOTA.
  • Results dropped sharply on dialects, with Moroccan speech among the hardest cases. Gaps to SOTA were smaller on semantic tasks than on syntactic ones.
  • GPT-3.5 sometimes hallucinated extra content, for example when translating the Bible dataset. Contamination checks with GPT-4 on nine datasets, using the guided-instruction method, found no reproduced examples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

56 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 53 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.eacl-long.30

@inproceedings{abdelali2024larabench,
  title = {{LAraBench: Benchmarking Arabic AI with Large Language Models}},
  author = {Abdelali, Ahmed and Mubarak, Hamdy and Chowdhury, Shammur Absar and Hasanain, Maram and Mousi, Basel and Boughorbel, Sabri and Abdaljalil, Samir and Kheir, Yassine El and Izham, Daniel and Dalvi, Fahim and Hawasly, Majd and Nazar, Nizi and Elshahawy, Yousseif and Ali, Ahmed and Durrani, Nadir and Milic-Frayling, Natasa and Alam, Firoj},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.30},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.30}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \# tasks Fine-tuned Models \# datasets GPTAraEval (Khondaker et al., 2023) Seq2Seq (transformer) ARBERT, MARBERT AraBERT, mBERT mBERT, ARBERT, CamelBERT, MARBERT MARBERT, AraT5 LAraBench (Ours) ✗ AraBench (Sajjad et al., 2020) find
  2. ARLUE (Abdul-Mageed et al., 2021) find
  3. ALUE (Seelawi et al., 2021) find
  4. ORCA (Elmadany et al., 2023) Zero-shot GPT-3.5 Few-shot GPT-3.5 Zero-shot GPT-4 Few-shot GPT-4 Zero-shot BLOOMZ Zero-shot Jais-13B-chat SOTA Comp. Modality ✗ ✗ ✗ ✗ ✗ ✗ ✓ T, S ✗ ✗ ✗ ✗ ✗ ✓ find