从人类偏好到自主对齐: 大语言模型对齐方法综述

Summary

This Fudan University survey reviews how large language models are aligned with human values and with the capabilities humans expect from them. Earlier surveys sorted methods by algorithm, signal source, or outer versus inner alignment. The authors instead group methods by learning paradigm, with three families:

• Imitation from human preferences: supervised instruction tuning, including distillation from aligned teacher models, and direct preference learning such as DPO and its variants.
• Inductive learning from feedback: signals from rules, reward models, environments like code sandboxes, or combinations of these.
• Autonomous alignment: self-reflection, self-critique, self-play, and interaction among multiple models.

For each family the paper explains how it works and weighs its strengths and limitations. It also catalogues the main training datasets and evaluation methods. Unlike prior reviews, it covers long-reasoning models in the style of DeepSeek-R1, both their capability alignment and their safety challenges. The survey ends with open problems and recommendations, making it a structured entry point to a fast-moving field.

Key points

• The paper lists common alignment training sets with their sizes:
  • Value alignment: HH-RLHF (170K, Anthropic), SafeRLHF (83.4K, Peking University), UltraFeedback (256K) and the Chinese CValues-Comparison (145K).
  • Math capability: GSM8K (8K) and MATH (12.5K).
  • Code capability: mbpp (974 problems) and The Stack v2 (358 programming languages).
  • Tool use and agents: MSAgent-Bench (600K, Chinese) and ToolBench (120K).
• The direct preference learning methods compared are DPO, IPO, KTO, CPO and ORPO:
  • IPO adds regularization against DPO overfitting.
  • KTO needs only a binary desirable/undesirable signal, drawing on prospect theory.
  • CPO targets the "alignment tax" in multi-objective alignment.
  • ORPO combines SFT and preference optimization in one objective.
• On distillation, Huang et al. found that a small model tuned on tens of thousands of o1-distilled long reasoning chains surpassed o1-preview on AIME.
• The data-synthesis methods covered include FLAN (more than 170 mixed tasks), Self-Instruct (175 seed instructions), Evol-Instruct and Databricks-dolly-15K.
• The evaluation benchmarks discussed include SafetyBench, ALERT, AlignBench, Flames, Length-controlled AlpacaEval, LongBench and L-Eval.