📄 Dunhuang-Bench: How Well Do MLLMs Understand Cultural Heritage?

Summary

Multimodal large language models (MLLMs) do well on general vision–language benchmarks, but their grasp of Dunhuang art had not been tested. This art, from the Mogao Caves and other Silk Road grottoes, requires both close visual inspection and specialist cultural knowledge. The authors built Dunhuang-Bench from two expert reference works, the Dunhuang Art Dictionary and the Dunhuang Studies Dictionary. They generated questions with templates and validated them in two stages: first with GPT-5, then by humans.

The benchmark is organized around art-historical theory. Panofsky's iconology underpins two tasks: visual perception (free-text QA) and knowledge reasoning (multi-turn dialogue). The formal analytic tradition underpins a third task on artistic appreciation (multiple choice). Scoring uses an LLM judge for the dialogues and a Normalized Rank Score for the appreciation task.

The authors tested 20 open and closed MLLMs. Scores fall consistently from perception and appreciation down to reasoning, and even the strongest model averages only around half. Chain-of-Thought and few-shot prompting bring little gain and sometimes hurt. The resource exposes clear gaps in how MLLMs understand cultural heritage.

Key points

• Scale: 486 images and 22,970 QA pairs, about twice the size of CVQA. They split into 11,688 perception questions (Task 1), 2,020 dialogues containing 8,080 QA pairs (Task 2), and 3,202 appreciation MCQs (Task 3).
• Data pipeline: 5,780 stylistic terms were taken from the Dunhuang Studies Dictionary. Text was extracted with Dolphin OCR and corrected by Dunhuang specialists. Task generation used Doubao.
• Models evaluated include GPT-5.1, Gemini-2.5, and the InternVL and Qwen-VL families. The best performer, InternVL3.5-241B-A28B, averages roughly 50.9%.
• Chain-of-Thought often lowers scores. For example, Qwen2.5-VL-7B drops from 12.55% to 9.85% on Task 2 and from 36.15% to 35.47% on Task 1.
• Few-shot prompting mainly helps perception: Qwen2.5-VL-7B rises from 36.15% to 39.40% on Task 1 with 4 shots. It gives no reliable benefit on reasoning or appreciation.