📄 Self-Taught Evaluators

Summary

Training model-based evaluators usually depends on large sets of human preference judgments. These are expensive to collect, especially for expert domains such as coding or math, and they lose value as newer models produce better responses. The Meta FAIR authors propose the Self-Taught Evaluator, which trains an LLM-as-a-Judge using only synthetic data. The method starts from unlabeled user instructions taken from WildChat and filters them by category with an LLM. For each instruction, the model writes a normal answer. It then writes a slightly altered version of the instruction and answers that instead, which gives a plausible but inferior response to the original question. Because the better response is known by construction, the current judge can sample several reasoning traces, and only those reaching the correct verdict are kept for fine-tuning. Repeating this loop produces a progressively stronger judge. Starting from Llama-3-70B-Instruct, RewardBench accuracy rises substantially and matches reward models trained on human labels. This suggests evaluators can be improved and adapted to new tasks without fresh annotation.

Key points

• RewardBench accuracy improves from 75.4 (seed Llama-3-70B-Instruct) to 88.3, or 88.7 with majority voting, with zero human-labeled preferences. This beats GPT-4 used as a judge.
• In the same LLM-as-a-Judge setup, training on the human-annotated HelpSteer2 set (about 10k labels) reaches 85.6. Four iterations on HelpSteer2 peak at 87.0, still below the synthetic pipeline.
• Asking the LLM directly for a bad answer works less well (80.7) than the altered-instruction strategy (83.8) on the same reasoning-focused instructions.
• Judgments are filtered with rejection sampling, and answer order is randomized to counter position bias. The set of examples with correct judgments grows as the model improves, which acts as an automatic curriculum.
• Mixing synthetic and HelpSteer2 preferences in various ratios keeps performance strong and yields small gains, with overall accuracy up to about 85.8 after the first iteration.