📄 Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM

Summary

Arabic has hundreds of millions of speakers, yet open conversational assistants that can discuss climate change and sustainability in Arabic barely exist. Open LLMs such as Vicuna and Alpaca handle this domain poorly and write weak Arabic. The authors built Arabic Mini-ClimateGPT on Vicuna-7B. They created Clima500-Instruct, an Arabic instruction set of roughly half a million examples, by having ChatGPT do three things in sequence: rewrite plain English QA pairs from CCMRC and ClimaBench as expert dialogue, translate each question and answer together, and then pass the result to native Arabic speakers for cleanup. The fine-tuned model also uses a vector-database retrieval step at inference time. That step adds external documents, including region-specific Arabic material, without any retraining. A ChatGPT judge preferred the model over several open-source competitors about 88–90% of the time. Native-speaker evaluators chose its answers in 81.6% of cases. The model also beat translate-in/translate-out pipelines, which suggests that training directly in Arabic is worth the effort.

Key points

• Data sources. Clima500-Instruct draws on CCMRC (about 21K QA pairs) and the Clima-QA part of ClimaBench (about 485K QA pairs from CDP surveys). The final Arabic set has 506,426 instances; an English set of 512,081 instances was produced along the way.
• Training. Vicuna-7B was fine-tuned on both the English and Arabic versions, following the Vicuna fine-tuning recipe. It ran on 4× A100 80GB with gradient checkpointing and flash attention.
• Retrieval. At inference, the user query is embedded and matched against a vector database. The query is augmented only when the retrieved content matches it closely.
• ChatGPT-judged results. The judge saw two answers plus the ground truth and could also answer "Neither." The model won 88.33% against Vicuna and about 90% against Alpaca, Dolly v2 and Alpaca-ar.
• Human evaluation and translation baselines. Native speakers picked the best of five anonymized answers or none: 81.6% chose this model, and no rival exceeded 8%. Against translation pipelines, the model won 73.98% vs 25.84% over Vicuna_tr and 69.57% vs 30.07% over ClimateGPT_en_tr.

Interesting ideas

• I1. Translate question and answer together. Sending each question and its answer to the translator in a single request keeps terminology and references consistent across the pair, which separate translation calls would not.
• I2. A pairwise judge with a "Neither" option. Letting the judge reject both candidates acts as a cheap control that exposes cases where both systems fail, rather than forcing a winner.
• I3. Translation pipelines as a baseline. Comparing against translate-in/translate-out versions of the same system separates the value of native-language training from the value of domain data.
• I4. Retrieval gated by match strength. Retrieved text is attached to the query only when similarity is high, so weak matches cannot add noise.
• I5. Rule-based checks on generated data. Simple checks remove generated examples with undefined symbols or with untranslated, still-English content before anyone does a manual spot-check.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When an LLM teacher tier labels or rewrites silver examples, give it the whole sentence pair or passage at once, so its frames and coreference decisions stay consistent across the unit.
• I2 (learn): Add an explicit "both wrong" outcome when comparing parser or reader variants against gold, so negative-control runs can surface shared blind spots.
• I3 (back): Compare translation through the meta-world against a plain pivot pipeline on the same items, to show what the meaning-graph route actually adds.
• I4 (world): Pull facts from the knowledge graph or Wikidata only when the entity or frame match clears a confidence threshold, and record that decision so it remains explainable.
• I5 (learn): Run similar sanity filters on teacher-produced silver trees or frames, such as malformed labels or unchanged source tokens, before agreement voting.