Elamur Research⌕
☰
home / papers / 2025 / VLM@school — Evaluation of AI image understanding on German middle school knowledge

VLM@school — Evaluation of AI image understanding on German middle school knowledge

Paper · 2025 · Rene Peinl and Vincent Tischler

Summary

Peinl and Tischler argue that common VLM benchmarks have several weaknesses. They are mostly in English, they rely heavily on multiple choice, and they often contain items a model can solve from the text alone. As a test of everyday competence, they build a German benchmark from middle-school material (grades 8–10, Gymnasium and Realschule). It has 2038 open-ended questions over 486 images in nine school subjects. German native speakers wrote or adapted the questions; nothing was machine-translated. Following MMMU-Pro, the authors removed questions that text-only LLMs could answer, printed some questions inside the image, and added adversarial items that ask about things the picture does not show. They tested thirteen open-weight VLMs. None reached 45% overall, and music, mathematics and the adversarial items were the weakest areas. Leaderboard standing did not predict results: the InternVL3 family leads MMMU and the Open VLM leaderboard, but smaller Mistral, Qwen and Gemma models beat it here. The authors conclude that ordinary school tasks are still a demanding and underused test bed for multimodal reasoning, especially outside English.

Key points

  • The dataset has 486 images and 2038 free-text questions in nine domains, including mathematics, history, biology, geography, music, religion, art and sports. Images are capped at 1300px per side, and the data is published at iisys-hof.github.io/vlms-at-school.
  • Thirteen open-weight VLMs were evaluated, and the best one stays below 45% accuracy. InternVL3 78B scores about 39.7%.
  • Mistral 3.1, Qwen 2.5 VL 32B and Gemma 3 27B outperform InternVL3. On the adversarial questions, even InternVL3 78B gets only about a third of the best competitors' score.
  • In a text-only check, LLMs answered 8.5% (best model) and 5% (worst model) of questions without the image.
  • Some planned work was dropped or left out:
    • Chemistry was abandoned because questions kept leaking enough hints to be answerable from text.
    • Commercial API models were not tested.
    • No human baseline was collected.

Interesting ideas

  • I1. Filter out every question that a model can answer without the image, so the benchmark truly needs the visual input.
  • I2. Add adversarial questions about objects that sound plausible but are absent from the image; models rarely respond that the thing is not there.
  • I3. Label image parts with numbers above 5 or with letters from the second half of the alphabet, which lowers the chance of a lucky guess.
  • I4. In math items, ask for symbolic results (e.g. keep Pi as a variable) so the score measures whether the model set up the right term, not its arithmetic.
  • I5. To rule out memorization of famous paintings, the authors reran the religion items on freshly generated images and got similar accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run the readers with the source text withheld as a negative control, and discard test items that are still solved from the question wording alone.
  • I2 (answer): Seed school-course and fable QA sets with questions about entities missing from the text. The reader should answer "not stated" after checking the world of the text, rather than inventing an answer.
  • I3 (answer): Apply the same labeling choice to generated multiple-choice and labeling tasks, so a correct answer reflects understanding rather than positional bias toward A–D.
  • I4 (answer): Grade maths word problems on the extracted equation as well as the number, which makes errors traceable to the frame/world step.
  • I5 (learn): Evaluate on newly composed fables or story variants that cannot be in any training data. This separates real comprehension from recall of well-known texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hof University of Applied SciencesHof · Germany2 works · 5 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Rene Peinl and Vincent Tischler. 2025. VLM@school — Evaluation of AI image understanding on German middle school knowledge. arXiv:2506.11604.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.11604 · arXiv 2506.11604

@misc{peinl2025vlm,
  title = {{VLM@school — Evaluation of AI image understanding on German middle school knowledge}},
  author = {Rene Peinl and Vincent Tischler},
  year = {2025},
  doi = {10.48550/arxiv.2506.11604},
  eprint = {2506.11604},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.11604}
}

References

6 references, 6 of them with a card of ours. ↗ leads to the work.

  1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13.14.15.16.17.18. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015). Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an asr corpus based on public domain audio books. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5206–5210.
  2. IEEE (2015). Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z.: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. Adv. Neural Inf. Process. Syst. 37, 95266–95290 (2025). Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., Fernandez-Llorca, D.: Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation, https://arxiv.org/abs/2502.06559v1, last accessed 2025/05/26. Raji, I.D., Bender, E.M., Paullada, A., Denton, E., Hanna, A.: AI and the Everything in the Whole Wide World Benchmark, http://arxiv.org/abs/2111.15366, (2021). Weij, T. van der, Hofstätter, F., Jaffe, O., Brown, S.F., Ward, F.R.: AI Sandbagging: Language Models can Strategically Underperform on Evaluations, http://arxiv.org/abs/2406.07358, (2025). https://doi.org/10.48550/arXiv.2406.07358. Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Re, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: Proceedings of the ACM Conference on Health, Inference, and Learning. pp. 151–159. ↗
  3. ACM, Toronto Ontario Canada (2020). https://doi.org/10.1145/3368555.3384468. Ethayarajh, K., Jurafsky, D.: Utility is in the Eye of the User: A Critique of NLP Leaderboards, http://arxiv.org/abs/2009.13888, (2021). Reuel-Lamparth, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., Kochenderfer, M.J.: BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. Adv. Neural Inf. Process. Syst. 37, 21763–21813 (2024). Dehghani, M., Tay, Y., Gritsenko, A.A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., Vinyals, O.: The Benchmark Lottery, http://arxiv.org/abs/2107.07002, (2021). Kang, E.B.: Ground truth tracings (GTT): On the epistemic limits of machine learning. Big Data Soc. 10, 20539517221146122 (2023). Peinl, R., Wirth, J.: Evaluation of Medium-Sized Language Models in German and English Language. Int. J. Nat. Lang. Comput. 13, 01–18 (2024). Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., Stanovsky, G.: State of what art? A call for multi-prompt LLM evaluation. Trans. Assoc. Comput. Linguist. 12, 933–949 (2024). Rauh, M., Marchal, N., Manzini, A., Hendricks, L.A., Comanescu, R., Akbulut, C., Stepleton, T., Mateos-Garcia, J., Bergman, S., Kay, J.: Gaps in the Safety Evaluation of Generative AI. In: AAAI/ACM Conf. on AI, Ethics, and Society. pp. 1200–1217 (2024). Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y.: MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9556–9567 (2024). Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., Chen, K.: VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models. In: 32nd ACM International Conference on Multimedia. pp. 11198–11201. ↗
  4. ACM, Melbourne VIC Australia (2024). Announcing ARC-AGI-2 and ARC Prize 2025, https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025, last accessed 2025/05/26. Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C.B.C., Shaaban, M., Ling, J., Shi, S.: Humanity’s last exam. ArXiv Prepr. ArXiv250114249. (2025). ↗
  5. René Peinl and Vincent Tischler 19. Peinl, R., Tischler, V.: Benchmarking Vision Language Models on German Factual Data. Presented at the 21st Int. Conf. on Artificial Intelligence Applications and Innovations , Limassol, Cyprus June 26 (2025). 20. Gemini 2.5: Our most intelligent AI model, https://blog.google/technology/googledeepmind/gemini-model-thinking-updates-march-2025/, last accessed 2025/05/26. 21. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, https://ai.meta.com/blog/llama-4-multimodal-intelligence/, last accessed 2025/05/26. 22. Introducing Claude 4, https://www.anthropic.com/news/claude-4, last accessed 2025/05/26. 23. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., Su, Y., Chen, W., Neubig, G.: MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, http://arxiv.org/abs/2409.02813, (2025). 24. Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning, http://arxiv.org/abs/2203.10244, (2022). https://doi.org/10.48550/arXiv.2203.10244. 25. Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A Diagram is Worth a Dozen Images. In: Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 235–251. ↗
  6. Springer, Cham (2016). 26. Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities, http://arxiv.org/abs/2308.02490, (2024). https://doi.org/10.48550/arXiv.2308.02490. 27. Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., Heng, P.-A.: SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems, http://arxiv.org/abs/2503.10627, (2025). 28. Mathew, M., Karatzas, D., Jawahar, C.V.: DocVQA: A dataset for VQA on document images. In: IEEE/CVF Winter Conf. on Applications of Computer Vision. pp. 2200–2209 (2021). 29. Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., Bai, X.: OCRBench: on the hidden mystery of OCR in large multimodal models. Sci. China Inf. Sci. 67, 220102 (2024). https://doi.org/10.1007/s11432-024-4235-6. 30. Xing, S., Sun, Z., Xie, S., Chen, K., Huang, Y., Wang, Y., Li, J., Song, D., Tu, Z.: Can Large Vision Language Models Read Maps Like a Human?, http://arxiv.org/abs/2503.14607, (2025). https://doi.org/10.48550/arXiv.2503.14607. 31. Das, R.J., Hristov, S.E., Li, H., Dimitrov, D.I., Koychev, I., Nakov, P.: EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models, http://arxiv.org/abs/2403.10378, (2024). 32. Sun, H.-L., Zhou, D.-W., Li, Y., Lu, S., Yi, C., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., Zhan, D.-C., Ye, H.-J.: Parrot: Multilingual Visual Instruction Tuning, http://arxiv.org/abs/2406.02539, (2024). https://doi.org/10.48550/arXiv.2406.02539. 33. Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X.: M 3 IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. ArXiv Prepr. ArXiv230604387. (2023). 34. Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M.O., Roth, S., Vulić, I., Gurevych, I.: xGQA: Cross-Lingual Visual Question Answering, http://arxiv.org/abs/2109.06082, (2022). 35. Park, C., Lee, K., Lim, H., Kim, J., Park, J., Heo, Y.-J., Chang, D.-S., Choo, J.: Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering, http://arxiv.org/abs/2406.02331, (2024). 36. Deng, Y., Zhao, Y., Li, M., Ng, S.-K., Chua, T.-S.: Don’t Just Say “I don’t know”! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations, http://arxiv.org/abs/2402.15062, (2024). 37. Brahman, F., Kumar, S., Balachandran, V., Dasigi, P., Pyatkin, V., Ravichander, A., Wiegreffe, S., Dziri, N., Chandu, K., Hessel, J.: The art of saying no: Contextual Evaluation of AI image understanding on German knowledge 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. noncompliance in language models. Adv. Neural Inf. Process. Syst. 37, 49706–49748 (2024). Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Huang, J., Zhang, J., Ning, M., Yuan, L.: MoE-LLaVA: Mixture of Experts for Large Vision-Language Models, (2024). Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S.: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, (2023). Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., Lu, J.: Q-VLM: Post-training Quantization for Large Vision-Language Models. In: 38 Annual Conference on Neural Information Processing Systems (2024). Zhu, L., Wang, X., Wang, X.: JudgeLM: Fine-tuned Large Language Models are Scalable Judges, (2023). Huang, H., Qu, Y., Bu, X., Zhou, H., Liu, J., Yang, M., Xu, B., Zhao, T.: An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4, http://arxiv.org/abs/2403.02839, (2024). Raina, V., Liusie, A., Gales, M.: Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment, (2024). Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P.: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, http://arxiv.org/abs/2404.18796, (2024). Shi, L., Ma, C., Liang, W., Ma, W., Vosoughi, S.: Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. ArXiv Prepr. ArXiv240607791. (2024). Alibaba: Qwen3: Think Deeper, Act Faster, https://qwenlm.github.io/blog/qwen3/, last accessed 2025/05/27. Xu, G., Jin, P., Li, H., Song, Y., Sun, L., Yuan, L.: LLaVA-CoT: Let Vision Language Models Reason Step-by-Step, http://arxiv.org/abs/2411.10440, (2025). Team Kimi: Kimi-VL Technical Report, http://arxiv.org/abs/2504.07491, (2025). https://doi.org/10.48550/arXiv.2504.07491. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peinl-2025-vlm-school-evaluation-ai-image/