📄 Schützen: Evaluating LLM Safety in Bulgarian and German Contexts

Summary

Most LLM safety benchmarks are built for English or Chinese and usually test other languages one at a time. That leaves open whether safety behaviour holds when languages share the same legal and cultural setting but differ in available resources. The authors present Schützen, a bilingual safety benchmark for Bulgarian (low-resource) and German (high-resource), two languages governed by common European norms. They took prompts from the Do-Not-Answer family, rewrote them with local people, events and policies instead of simply translating them, and added prompts on topics that are sensitive specifically in Bulgaria or Germany. Each item is a direct attack, an indirect attack, or a benign question that contains sensitive words. Responses from 15 LLMs were labelled safe or unsafe and assigned a fine-grained response pattern, using a pipeline in which humans and GPT-5 annotate together. Every model is safe on more than 95% of prompts overall, but performance falls on region-specific content and is lower for Bulgarian than for German. The authors conclude that localized evaluation is needed.

Key points

• Size and structure: about 8K prompts, 16 harm types grouped into six risk areas, and three prompt types (Ori = direct, FN = indirect attack, FP = over-sensitivity check). More than 95K responses were collected from 15 LLMs.
• Models tested include GPT-4o, Claude-3.7, Llama-4, EuroLLM-9B, BgGPT-27B/2.6B, LLaMmlein-7B-chat and Leo-hessianai-13B-chat.
• Best models by language:
  • Claude-3.7 is safest for German and BgGPT-27B for Bulgarian.
  • BgGPT-2.6B ranks lowest for Bulgarian and LLaMmlein-7B-chat lowest for German.
  • GPT-4o scored below most open-source systems.
• On region-specific prompts:
  • bgGPT leads for Bulgaria and EuroLLM-9B for Germany.
  • Overall, models do better in German than in Bulgarian.
• Fine-grained labels cover six safe patterns and four unsafe types.
  • The most common safe pattern is generic harmless information.
  • Llama-4-Maverick refuses most often.
  • The dominant unsafe pattern differs by language: Bulgarian is mostly confidently stated wrong information, while German is mostly misinformation that leads to false beliefs.

Interesting ideas

• I1. The benchmark adds benign questions that contain alarming words as a control alongside the real attacks, so it measures both missed harm and over-refusal.
• I2. Raw unsafe counts favoured direct attacks, but only because there were about twice as many of them; after normalizing per prompt type, indirect attacks were the more effective bypass.
• I3. During annotation, an LLM supplies a label and a rationale, and a human resolves only the cases where the LLM and the human annotators disagree. The model's alternative readings helped counter human tunnel vision, and the human checked them against evidence.
• I4. Prompts were culturally localized, with local figures, events and norms substituted in, rather than translated literally, so that what is tested is grounded in the target context.
• I5. Annotation separates the safe/unsafe verdict from a finer label for how the model responded, such as refusing, hedging or giving generic information.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation sets for Mova could pair each hard case with a superficially similar but harmless twin, to check that a repair rule is not just firing on a keyword.
• I2 (answer): Error reports on school tests or word problems should be normalized by question type before deciding which reader component is weakest.
• I3 (learn): The LLM teacher tier could annotate the silver data that witness parsers disagree on, with humans reviewing only the conflicts rather than everything.
• I4 (world): When fables or QA items are adapted from other sources, the entities and background facts should be replaced with ones present in Mova's ontology and knowledge graph, so that comprehension is tested rather than unknown names.
• I5 (frames): Speech-act analysis could tag a response's strategy (refusal, disclaimer, evasion) separately from its truth value, which would make answer behaviour easier to explain.