Elamur Research⌕
☰
home / papers / 2025 / Benchmarking Vision Language Models on German Factual Data

Benchmarking Vision Language Models on German Factual Data

Paper · 2025 · Rene Peinl and Vincent Tischler

Summary

Vision language models (VLMs) are trained mostly on English and Chinese material, so even a well-resourced language like German may be served poorly. The authors ask where the gap comes from: the language skills of the LLM backbone, or the English-centred image/text pairs used later in VLM training. They built FactSightVQA, a benchmark of simple perceptual facts such as a person's name, a car's make and model, or a species name. It covers six categories (animals, plants, celebrities, sights, cars, products), and each category has a German-content subset and an international or US subset. Open-weight VLMs were asked about every image in both German and English, and a jury of judge models scored the answers. The results split by category. Celebrities and sights fail because the models do not recognise German content visually. Animals and plants are often recognised under scientific or English names, but the German name is missing, and the authors trace this to the LLM backbone. Cars and products show no gap. The work argues that benchmarks should include country-specific images and compare across languages.

Key points

  • The dataset is published on Hugging Face as rpeinl/FactSightVQA. It has roughly 700 images per side (German vs international), not counting Celeb1k. Most images were self-collected from public websites while avoiding Wikipedia. Some parts reuse Stanford cars and oxford-flowers102.
  • Image sizes range from 400 to 1300px, averaging about 750px. As a contamination control, the authors added a filtered subset of the Hugging Face celeb-1000 set (211 people, two images each, 256px).
  • The study tested 13 open-weight VLMs between December 2024 and January 2025:
    • eight medium models (7–13B parameters),
    • two small models (2–4B),
    • three large models (25–38B). Aria was the only Mixture-of-Experts model.
  • Qwen 2.5 VL was the best model overall. Llama 3.2 Vision did very well here but has only 57.3 on the Open VLM leaderboard. InternVL 2.5 38B and Ovis 1.6 Gemma 2 27B lead public benchmarks yet were mediocre on this one. Molmo was the weakest on German images.
  • Products could be answered through OCR alone, so they did not test image understanding. Cars showed no gap, which the authors attribute to the global car market.

Interesting ideas

  • I1. Crossing two factors, prompt language (de/en) and image origin (German/international), separates language failures from perception failures in a single grid.
  • I2. When the full model misses a German name, the authors query the bare LLM backbone with the scientific or English name to see which component lacks the knowledge.
  • I3. A copy of a public, probably-seen dataset (Celeb1k) sits next to freshly collected data of the same kind, so inflated scores from memorisation become visible.
  • I4. The scoring rule adjusts how specific an answer must be to the size of the family. A family name counts when the family holds hundreds of species (mayfly), but a specific name is required where species are few, and words like "fish" are always too coarse.
  • I5. Products turned out solvable by reading their labels, which shows how one category can quietly test a different skill than intended.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could cross text domain with the UD dialect of the training treebank, so that a drop in LAS can be blamed on the right cause instead of on "the parser".
  • I2 (world): When an answer is wrong, the same question can be put to the knowledge graph or ontology alone (e.g., the WordNet lemma lookup), showing whether the fault lies in compiled knowledge or in the upstream tree/frames.
  • I3 (learn): Test suites could pair well-known public fables with newly written ones of the same type, acting as a negative control for memorised answers from LLM teachers.
  • I4 (answer): Answer matching could use WordNet hypernym depth and sibling counts to decide whether a more general noun still counts as correct.
  • I5 (answer): School-test and word-problem sets could be checked for questions answerable by surface string matching, and those flagged so that scores reflect real tree→frame→world reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hof University of Applied SciencesHof · Germany2 works · 5 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Rene Peinl and Vincent Tischler. 2025. Benchmarking Vision Language Models on German Factual Data. arXiv:2504.11108.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.11108 · arXiv 2504.11108

@misc{peinl2025benchmarking,
  title = {{Benchmarking Vision Language Models on German Factual Data}},
  author = {Rene Peinl and Vincent Tischler},
  year = {2025},
  doi = {10.48550/arxiv.2504.11108},
  eprint = {2504.11108},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.11108}
}

References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Wu, W. Gan, Z. Chen, S. Wan, and P. S. Yu, “Multimodal Large Language Models: A Survey,” In 2023 IEEE Int. Conf. on Big Data (pp. 2247-2256). IEEE.
  2. J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 2024 ↗
  3. J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning, in BLIP. PMLR, 2022, pp. 12888–12900. ↗
  4. J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” In International conference on machine learning (pp. 19730-19742). PMLR.
  5. W. Dai et al., “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” Jun. 15, 2023, arXiv: arXiv:2305.06500. ↗
  6. Y. Qiao et al., “Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs,” Jun. 20, 2024, arXiv: arXiv:2406.14544. ↗
  7. “Open VLM Leaderboard - a Hugging Face Space by opencompass.” Accessed: Feb. 13, 2025. [Online]. Available: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard ↗
  8. Z. Chen et al., “Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling,” Jan. 13, 2025, arXiv: arXiv:2412.05271. ↗
  9. Q. Team, “Qwen2.5 VL! Qwen2.5 VL! Qwen2.5 VL!,” Qwen. Accessed: Feb. 12, 2025. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5-vl/ ↗
  10. A. I. Meta, “Llama 3.2: Revolutionizing edge AI and vision with open, customizable models,” Meta AI Blog Retrieved Dec., vol. 20, p. 2024, 2024. find
  11. M. Deitke et al., “Molmo and PixMo: Open Weights and Open Data for State-ofthe-Art Vision-Language Models,” Dec. 05, 2024, arXiv: arXiv:2409.17146. ↗
  12. P. Agrawal et al., “Pixtral 12B,” Oct. 10, 2024, arXiv: arXiv:2410.07073. ↗
  13. R. Peinl and J. Wirth, “Evaluation of Medium-Sized Language Models in German and English Language,” Int. J. Nat. Lang. Comput., vol. 13, no. 1, Feb. 2024. ↗
  14. D. Romero et al., “CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark,” Nov. 04, 2024, arXiv: arXiv:2406.05967. ↗
  15. H. Wang et al., “M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models,” May 24, 2024, arXiv: arXiv:2405.15638. ↗
  16. Z. Li, X. Wu, H. Du, H. Nghiem, and G. Shi, “Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A Survey,” Jan. 29, 2025, arXiv: arXiv:2501.02189. ↗
  17. S. Antol et al., “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425–2433. find
  18. K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in Proc. of the IEEE/cvf conference on computer vision and pattern recognition, 2019, pp. 3195–3204. find
  19. A. Singh et al., “Towards vqa models that can read,” in Proc. of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8317–8326. find
  20. A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning,” Mar. 19, 2022, arXiv: arXiv:2203.10244. ↗
  21. P. Lu et al., “MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,” 12th Int. Conf. on Learning Representations ICLR2024.
  22. W. Yu et al., “MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities,” 41st Int. Conf. on Machine Learning, PMLR 235:57730-57754, 2024. ↗
  23. X. Yue et al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9556–9567.
  24. Y. Wang et al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,” Nov. 06, 2024, arXiv: arXiv:2406.01574. ↗
  25. T. Guan et al., “HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14375–14385.
  26. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating Object Hallucination in Large Vision-Language Models,” In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing (pp. 292-305).
  27. B. Li et al., “SEED-Bench-2: Benchmarking Multimodal Large Language Models,” Nov. 28, 2023, arXiv: arXiv:2311.17092. ↗
  28. B. Li, Y. Ge, Y. Chen, Y. Ge, R. Zhang, and Y. Shan, “SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension,” Apr. 25, 2024, arXiv: arXiv:2404.16790. ↗
  29. Y. Liu et al., “MMBench: Is Your Multi-modal Model an All-Around Player?,” in Computer Vision – ECCV 2024, vol. 15064, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, Eds., in Lecture Notes in Computer Science, vol. 15064. Cham: Springer Nature Switzerland, 2025, pp. 216–233.
  30. J. Tang et al., “MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering,” Nov. 19, 2024, arXiv: arXiv:2405.11985. ↗
  31. R. J. Das, S. E. Hristov, H. Li, D. I. Dimitrov, I. Koychev, and P. Nakov, “EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models,” Mar. 15, 2024, arXiv: arXiv:2403.10378. ↗
  32. H.-L. Sun et al., “Parrot: Multilingual Visual Instruction Tuning,” Aug. 11, 2024, arXiv: arXiv:2406.02539. ↗
  33. L. Li et al., “M 3 IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning,” ArXiv Prepr. ArXiv230604387, 2023. ↗
  34. J. Pfeiffer et al., “xGQA: Cross-Lingual Visual Question Answering,” Mar. 17, 2022, arXiv: arXiv:2109.06082. ↗
  35. C. Park et al., “Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering,” Jun. 04, 2024, arXiv: arXiv:2406.02331. ↗
  36. S. Nayak et al., “Benchmarking Vision Language Models for Cultural Understanding,” Oct. 14, 2024, arXiv: arXiv:2407.10920. ↗
  37. G. I. Winata et al., “WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines,” Feb. 08, 2025, arXiv: arXiv:2410.12705. ↗
  38. A. Vayani et al., “All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages,” Nov. 26, 2024, arXiv: arXiv:2411.16508. ↗
  39. N. Du et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in Int. Conf. on Machine Learning (pp. 5547-5569) 2022. ↗
  40. D. Li et al., “Aria: An Open Multimodal Native Mixture-of-Experts Model,” Jan. 10, 2025, arXiv: arXiv:2410.05993. ↗
  41. P. Verga et al., “Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models,” May 01, 2024, arXiv: arXiv:2404.18796. ↗
  42. H. Huang et al., “An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4,” Nov. 05, 2024, arXiv: arXiv:2403.02839. ↗
  43. J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” Proceedings of Machine Learning and Systems, 6, 87-100. 2024 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peinl-2025-benchmarking-vision-language-models-german/