大语言模型的中文多轮指令遵循能力评测研究

Summary

Benchmarks of instruction following for Chinese LLMs are often translated from English. They tend to test a single turn and carry over checks that suit only Latin script, such as letter case. This paper presents CMIF (Chinese Multiturn Instruction Following Benchmark), built for Chinese multi-turn evaluation. Each item pairs a real user question, drawn from AlignBench-style categories, with three atomic instructions that are given one per turn over a three-turn dialogue. The atomic instructions include Chinese-specific constraints, such as pinyin, tonal patterns, parallel sentences and classical-Chinese style. A two-stage filter removes contradictory instruction combinations: simple rules catch obvious clashes, and a strong LLM catches subtler ones. Scoring is rule-based rather than LLM-judged, which keeps evaluation cheap and objective. Fourteen open and closed models were tested. All of them handle single turns reasonably well, but accuracy falls steadily as turns accumulate. Chinese-specific instructions are much harder than generic formatting ones. The benchmark therefore exposes concrete weaknesses in phonological and grammatical control that general benchmarks miss.

Key points

• Instruction categories: CMIF defines 25 instruction types grouped into five top-level classes: keywords, length limits, special symbols, extra generation, and Chinese-specific features.
• Metrics: Four accuracies are computed, each at instruction level and dialogue level, under strict and loose criteria. Their mean is reported as a composite score.
• Overall results: Claude-3.5-Sonnet ranks best among closed models with a composite of 0.538. Qwen2.5-72B-Instruct ranks best among open models with 0.490. Claude's instruction-level accuracy drops from 73.8% to 40.0% in multi-turn settings.
• Chinese-specific tasks: The best composite on these tasks is only 51.0%, achieved by GPT4o. The other four classes average about 79.0%. Tonal patterns (仄起平收), pinyin checks, keyword frequency and sentence-mood tasks all score below 30% on average.
• Model size and context length: Larger Qwen2.5 models lose less accuracy across turns. Doubao-pro-128k underperforms Doubao-pro-32k, suggesting a longer context window does not help here.