📄 UI-Level Evaluation of ALLaM 34B: Measuring an Arabic-Centric LLM via HUMAIN Chat

Summary

The paper examines how well ALLaM 34B, the largest publicly accessible model in SDAIA's Arabic-centric ALLaM family, performs as deployed inside HUMAIN Chat. That service offers no API and no weights, so all testing had to go through the chat interface. The author built 23 prompts covering MSA, five regional dialects (Najdi, Hijazi, Egyptian, Moroccan, Levantine), code-switching and Arabizi, recall of facts, reasoning about arithmetic and time, creative writing tasks, and adversarial safety probes. Each prompt was submitted five times, giving 115 responses. Three LLM judges (GPT-5, Gemini 2.5 Pro, Claude Sonnet-4) scored every response on a 1–5 scale for accuracy, fluency, instruction following, safety, and, where relevant, dialect fidelity. Scores were averaged per judge, then per prompt, then per category, with 95% confidence intervals. Code-switching and generation scored highest, while dialects scored lowest. Levantine and Moroccan were especially weak: the model understands dialectal input but often answers in MSA or even English. The study is useful as a template for evaluating a black-box deployed system and for showing where Arabic dialect coverage falls short.

Key points

• The evaluation set has 23 prompts × 5 runs = 115 outputs, collected only through the HUMAIN Chat web UI. Decoding parameters were not controllable, and observed latency was about 1–3 seconds.
• Category means were:
  • code-switching 4.92 and generation 4.92
  • knowledge 4.77, MSA 4.74, reasoning 4.64
  • safety 4.54, dialect 4.21
  • prompt injection, jailbreak and data exfiltration each exactly 4.20, with zero variance
• The dialect heat map shows a different picture from the category table. Najdi, Hijazi and Egyptian score about 3.8 overall, Moroccan about 3.3, and Levantine 2.73. In the weaker dialects, accuracy drops much more than fluency.
• Typical failure modes in the qualitative examples:
  • Najdi weather questions received structured English summaries.
  • Hijazi questions received long MSA news bulletins.
  • Egyptian small talk received formulaic MSA self-introductions.
• The human check of the LLM judges was limited to a partial review of a subset and is described only qualitatively. Stated limitations include the small prompt pack, dependence on the UI, and reliance on LLM judges.

Interesting ideas

• I1. Each prompt is sent several times and scores come with confidence intervals, so run-to-run instability is visible rather than hidden in a single sample. — For Mova (learn): When an LLM teacher tier labels a sentence, query it several times and accept the label as silver data only if the answers agree, alongside the witness-parser agreement.
• I2. Scores are aggregated in a fixed order: raw score → mean over judges → mean over runs of a prompt → category mean with CI. This keeps the many runs of one prompt from outweighing other prompts. — For Mova (learn): Use the same layered averaging in evaluation reports so that sentences or items with many variants do not dominate the totals for a test category.
• I3. Fluency and accuracy are scored separately, which exposes a model that produces well-formed text with wrong content (fluent but inaccurate dialect answers). — For Mova (back): Score generated text on two separate axes, surface well-formedness and faithfulness to the source meaning graph, so a smooth but unfaithful realisation cannot pass.
• I4. A metric is applied only when it is relevant (dialect fidelity only for dialect prompts), and the overall score averages just the applicable dimensions. — For Mova (answer): Attach metrics to task types, for example numeric correctness only for maths word problems, so irrelevant dimensions do not dilute the scores.
• I5. The paper separates understanding a variety from answering in it: the model grasped dialectal input but switched register or language in its reply. — For Mova (back): Add a check that the output's register and language match those of the input or the requested style, and flag drift to formal or foreign phrasing as a distinct error type.