📄 CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

Summary

European Portuguese (PT-PT) has had no LLM leaderboard of its own. Existing Portuguese efforts mostly target the Brazilian variant, and no Portuguese benchmark measured safety safeguards or cultural alignment. The authors built CLARIN-PT-LDB, an open leaderboard hosted as a HuggingFace Space. Users submit open HuggingFace models, and the PORTULAN CLARIN infrastructure evaluates them on ten benchmarks:

• Tuguesice-PT, a new culture benchmark written from scratch.
• DoNotAnswer-PT, a curated translation of a safeguards dataset.
• Machine-translated reasoning and QA sets: MuSR, MMLU, GPQA Diamond, MMLU Pro and AA-Omniscience-Public.
• CoPA, MRPC and RTE, taken from ExtraGLUE.

All tasks are scored generatively, so models must write their answer as text rather than have options ranked by logits. This works for any model, including API-only ones, and resembles real chatbot use. The pipeline runs on EleutherAI's evaluation harness with per-task YAML configurations, and every benchmark reports accuracy.

Initial results cover Gervásio, Llama and Mistral at 8B, 24B and 70B. Scores rise with model size, and the harder MMLU Pro yields lower scores than MMLU. Gervásio, which was fine-tuned on Portuguese data, outscores its Llama base model on cultural alignment.

Key points

• Tuguesice-PT:
  • It has 327 short-answer question–answer pairs written by nine undergraduate annotators and checked by separate adjudicators.
  • Questions refer to Portugal only implicitly (e.g., asking about "the capital" without naming a country), so the test reveals the model's default cultural assumptions.
  • An answer counts as correct if it contains the gold answer (case-insensitive).
• DoNotAnswer-PT: It has 939 harmful requests across 61 harm categories. A judge model sorts each response into one of six classes, such as refusal, disclaimer or compliance.
• Benchmark sizes: MMLU has 14,042 items, MMLU Pro 12,032, AA-Omniscience-Public 600, GPQA Diamond 198, CoPA 500, MRPC 1,730 and RTE 3,000.
• Cultural alignment result: On Tuguesice-PT, Gervásio 70B scores 39.76 against 25.69 for Llama 70B. At 8B, Gervásio scores 11.31 and Llama 9.79.
• Infrastructure: The frontend is adapted from HuggingFace's Open LLM Leaderboard code. The backend runs on two NVIDIA L40S nodes (ten GPUs each) and two RTX Pro 6000 nodes (one 96 GB GPU each).