📄 BanglaLlama: LLaMA for Bangla Language

Summary

Bangla has roughly 240 million native speakers, yet open LLMs handle it poorly because it makes up only a tiny share of their training data. The authors build BanglaLlama, an open family of five Bangla-adapted LLaMA models (from LLaMA 2-7B, 3-8B, 3.1-8B, 3.2-1B and 3.2-3B), each released in base and instruct form.

The base models were further pretrained with LoRA for one epoch on the Bangla portion of CulturaX. Only the LLaMA-2 model got an enlarged vocabulary; the LLaMA-3 models kept their original tokenizer. The instruct versions were then tuned on two new datasets produced by machine-translating English instruction sets into Bangla.

The models were tested on 120 Bangla prompts in eight task categories, with GPT-4o scoring each answer. The strongest variants clearly beat the original Meta models on reasoning, open QA, literature and generation, but do worse on translation. The datasets and models give Bangla NLP research a shared open baseline, including small models that suit communities with limited compute.

Key points

• Instruction data: Bangla-Alpaca (52k pairs, from Stanford Alpaca) and Bangla-Orca (172k, from OpenOrca with code and math items removed), 224k samples in total. Translation used the Google Cloud Translation API at a cost of about $10k, and the authors added hand-written instructions on Bangladeshi culture and events.
• Tokenizer: for LLaMA-2-7B, a SentencePiece tokenizer trained on Bangla Wikipedia added 18k Bangla tokens, growing the vocabulary from 32k to 50k. The LLaMA-3 models kept their default vocabulary of about 128K tokens.
• Evaluation setup:
  • 120 queries covering Translation, Entertainment, Generation, Open_QA, Factual_QA, Reasoning, Ethics and Literature.
  • Each query was sampled 3 times at temperature 0.6.
  • GPT-4o rated every answer on a 1–100 scale.
• Headline scores:
  • BanglaLlama-3-8B vs. Meta LLaMA-3-8B: Reasoning 82.75 vs. 37.50, Open_QA 66.96 vs. 43.40, Generation 66.46 vs. 57.92.
  • BanglaLlama-3.1-8B vs. Meta LLaMA-3.1-8B: Literature 53.37 vs. 24.63.
  • The best model wins 7 of the 8 task categories.
• Limitations and release: translation quality lags behind the Meta models, possibly because bilingual training data was limited. All models are published in the BanglaLLM collection on Hugging Face.