Elamur Research⌕
☰
home / papers / 2026 / From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

Paper · 2026 · Camilla Dalerci, Thilo Michael, Robin Schaefer et al.

Summary

Public bodies choosing an LLM get little help from standard leaderboards. Those leaderboards are mostly English-centric and US-oriented, and they rarely look beyond task scores. The authors present first results from MÖVE, a living evaluation framework built for German public administration with input from stakeholders. This paper rates 39 models from 13 providers on three governance dimensions:

  • Inference energy: estimated with EcoLogits, using the output-token counts each model actually produced on nine German-language tasks.
  • Provider transparency: a manually checked 21-question matrix derived from the EU General-Purpose AI Code of Practice.
  • Political knowledge: whether a model can predict official party stances recorded in the Wahl-O-Mat.

No model wins on every axis. Energy use spans roughly a 63-fold range, and parameter count only partly explains it. Disclosure is consistently thin on compute, energy and training data. The best party-position accuracy is only 0.671, and European models show no home advantage. The takeaway for procurement is that model selection should follow the requirements of the deployment context rather than a single universal ranking.

Key points

  • Energy estimates range from 0.647 to 40.6 Wh per query (median 1.788 Wh). Summarization costs the most because it produces longer outputs, and reasoning models are especially expensive.
  • Transparency is scored 0/1/2 per question, with the Code of Practice signature counted as a binary item, for a maximum of 42.
    • Models score between 18 and 38, across 819 model-question assessments in total.
    • Compute and energy is the weakest domain at 11.5%. Of the models, 84.6% report no training energy at all.
  • An automated scoring agent initially disagreed with the human ratings on 27.4% of items. After the authors wrote explicit scoring notes, disagreement fell to 16.2%.
  • The political-knowledge set contains 4,788 stances from 64 parties across the 2013, 2017, 2021 and 2025 federal elections. Labels are 47.5% agree, 39.7% disagree and 12.8% neutral.
  • DeepSeek R1 and GPT-4o Mini tie at the top with 0.671 accuracy. The majority baseline is 0.475.

Interesting ideas

  • I1. Disagreements between the human and automated raters were not simply averaged away; recurring conflicts were written down as explicit scoring rules, and the remaining disagreement was reported as a consistency measure, not as accuracy.
  • I2. Political stance is treated as a factual question checked against an external record ("what did party X say?"), which keeps the model's knowledge separate from its own bias.
  • I3. Cost is estimated from the real output lengths each model produced on each task, not from a fixed assumed length, so the cost of verbose tasks shows up honestly.
  • I4. Each graded score (0/1/2) is published together with its justification and the sources behind it, so anyone can audit it.
  • I5. A majority-class baseline sits next to every model score, so modest accuracies are read against what blind guessing already achieves.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When witness parsers disagree on silver data, recurring conflict patterns could be codified as explicit repair or adjudication rules. Inter-witness agreement should then be reported as consistency, never as correctness.
  • I2 (world): Beliefs and positions of entities in a text could be stored as attributed facts with a holder and a source. This keeps "who claims what" separate from the system's own assertions.
  • I3 (learn): Calls to the LLM teacher tiers could be budgeted per task from logged token usage. This would show which tasks justify an expensive teacher.
  • I4 (world): Facts compiled into the knowledge graph from Wikidata or WordNet could carry a graded evidence level and a source link. Every downstream answer could then cite why it is believed.
  • I5 (answer): School-test and fable-QA results could always be reported beside a majority or answer-prior baseline. This acts as a negative control that exposes guessing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BundesdruckereiBerlin · Germany2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Camilla Dalerci, Thilo Michael, Robin Schaefer and Daniel Weinland. 2026. From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector. arXiv:2608.17827.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.17827 · arXiv 2608.17827

@misc{dalerci2026global,
  title = {{From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector}},
  author = {Camilla Dalerci, Thilo Michael, Robin Schaefer and Daniel Weinland},
  year = {2026},
  doi = {10.48550/arxiv.2608.17827},
  eprint = {2608.17827},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.17827}
}

References

19 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42–50, Punta Cana, Dominican Republic. Association for Computational Linguistics. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico. Association for Computational Linguistics.
  2. Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, and Joel Niklaus. 2023. SCALE: Scaling up the Complexity for Advanced Language Model Evaluation. Preprint, arXiv:2306.09237. ↗
  3. Luca Rettenberger, Markus Reischl, and Mark Schutera. 2025. Assessing political bias in large language models. Journal of Computational Social Science, 8(2):42. ↗
  4. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. 2024. BetterBench: assessing AI benchmarks, uncovering issues, and establishing best practices. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  5. Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022. Eur-lex-sum: A multi-and cross-lingual dataset for long-form summarization in the legal domain. arXiv preprint arXiv:2210.13448. ↗
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol., 15(3). ↗
  7. Samuel Rincé and Adrien Banse. 2025. EcoLogits: Evaluating the Environmental Impacts of Generative AI. Journal of Open Source Software, 10(111):7471. ↗
  8. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15295–15311, Bangkok, Thailand. Association for Computational Linguistics. ↗
  9. Camilla Dalerci, Thilo Michael, Robin Schaefer, and Daniel Weinland. 2026. MÖVE: A Holistic LLM Benchmark for the German Public Sector. Preprint, arXiv:2606.13111. ↗
  10. Chadi Helwe, Oana Balalau, and Davide Ceolin. 2025. Navigating the Political Compass: Evaluating Multilingual LLMs across Languages and Nationalities. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17179–17204, Vienna, Austria. Association for Computational Linguistics. find
  11. Shivalika Singh and 1 others. 2025. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv preprint. ArXiv:2009.03300 [cs]. ↗
  13. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  14. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 31 others. 2023. Holistic Evaluation of Language Models. Preprint, arXiv:2211.09110. ↗
  15. Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, and William Chandra Tjhi. 2025. SEA-HELM: Southeast Asian Holistic Evaluation of Language Models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12308–12336, Vienna, Austria. Association for Computational Linguistics. ↗
  16. Neil Majithia, Rajat Shinde, Zo Chapman, Prajun Trital, Jordan Decker, Manil Maskey, Elena Simperl, and Nigel Shadbolt. 2026. The CitizenQuery Benchmark: A Novel Dataset and Evaluation Pipeline for Measuring LLM Performance in Citizen Query Tasks. Preprint, arXiv:2602.04064. ↗
  17. Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval. ↗
  18. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: a stickier benchmark for general-purpose language understanding systems. Curran Associates Inc., Red Hook, NY, USA.
  19. Yanran Wu, Inez Hua, and Yi Ding. 2025. Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10560–10576, Vienna, Austria. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dalerci-2026-global-benchmarks-local-evaluations-benchmarking/