大语言模型评测综述

Summary

This survey from Peking University examines how large language models (LLMs) should be evaluated. As LLMs become more general-purpose, the older approach of scoring one model on one task with one metric no longer works well. The authors identify three problems with it. Benchmarks are too narrow to separate strong models. Test data often comes from limited domains or populations. Properties beyond raw accuracy, such as reasoning, robustness, fairness and harmful output, are largely ignored.

The paper is a literature review and introduces no new dataset. It sorts benchmarks into two groups. The "classical" paradigm compares model outputs against reference answers for NLU and NLG tasks. The "new" paradigm combines many datasets and also probes alignment with human preferences, knowledge, reasoning and safety. The authors then cover evaluation metrics and methods, point out weaknesses such as unreasonable task design and results that are hard to interpret, and suggest future research directions. Chinese-language benchmarks get particular attention, which makes the survey a useful map for anyone building evaluation suites for languages other than English.

Key points

• Classical benchmarks surveyed:
  • GLUE has 9 NLU tasks, and SuperGLUE adds five harder ones.
  • XTREME spans 40 languages and 9 tasks; XGLUE has 11 tasks across 19 languages.
  • CLUE (2020) was the first large-scale Chinese NLU benchmark.
  • BLEU, METEOR and ROUGE are discussed as generation metrics.
• Integrated and LLM-scale benchmarks:
  • CUGE organizes 21 datasets into 7 language functions and 18 NLP tasks.
  • EleutherAI LM Harness covers 200+ datasets.
  • BIG-bench (Google, 2022) took 442 researchers two years and covers 200+ datasets.
• Preference-alignment benchmarks:
  • MT-Bench has 80 multi-turn questions across 8 scenarios.
  • Chatbot Arena collected about 30,000 crowdsourced pairwise judgments in its first month.
  • SuperCLUE-Open has 1,200 Chinese multi-turn questions.
• C-EVAL contains 13,948 multiple-choice questions spanning 52 subjects and four difficulty levels. To reduce contamination, its questions come mainly from mock and local exams. A harder subset, C-EVAL HARD, is also provided.
• SAFETYPROMPTS tests the safety of Chinese LLMs using 8 safety scenarios (e.g., insult, privacy, mental health) and 6 instruction-attack types (e.g., goal hijacking, prompt leaking).