MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.acl-long.401.pdf

Summary

Existing LLM benchmarks mostly test single-turn instructions. Multi-turn ones such as MT-Bench cover only two turns and broad skill categories, so they miss much of what happens in real chatbot conversations. MT-Bench-101 fills this gap with a fine-grained multi-turn benchmark. The authors studied real user–chatbot logs from ShareGPT and RealChat alongside teaching taxonomies from educational psychology. From this, they built a three-level ability hierarchy:

• three top-level abilities: perceptivity, adaptability and interactivity;
• seven mid-level abilities;
• 13 concrete tasks.

They used GPT-4 with task-specific prompts to generate the dialogues. For scoring, GPT-4 acts as the judge, applying per-task guidelines. Models answer with golden context as dialogue history, and each dialogue receives its weakest turn's score. With this setup, 21 LLMs were evaluated. GPT-4 performed best overall. Adaptability and interactivity emerged as the main weaknesses across models. Larger models scored higher, but alignment methods such as RLHF and chat-oriented designs gave no clear multi-turn gains. The benchmark thus offers a validated, diagnostic tool for analysing conversational LLM behaviour.

Key points

• The dataset contains 4208 turns in 1388 multi-turn dialogues across 13 tasks. Examples include anaphora resolution, topic shift, self-correction, self-affirmation and proactive interaction.
• 21 models were tested: 2 closed-source and 19 open-source.
• GPT-4's judgments matched human experts 87% of the time, higher than the 80% agreement among the experts themselves. Removing the scoring guidelines lowered agreement to 77%. Averaging turn scores instead of taking the minimum lowered it to 75%.
• Feeding models their own earlier answers as history let errors pile up over turns, while golden context raised scores. This motivated the choice of golden context in the evaluation protocol.
• Data and code are released at github.com/mtbench101/mt-bench-101.