📄 Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge

Summary

LLM-as-a-Judge systems often produce chain-of-thought (CoT) judgments that focus on only a few aspects of the responses, so their verdicts can be shallow or wrong. Majority voting and criteria expansion only partly fix this. The former is costly and passive, and the latter ignores what is actually in each response. The authors propose Crowd-based Comparative Evaluation (CCE). Several extra "crowd" responses are generated for the same instruction, and each candidate is judged against each crowd response. The resulting crowd judgments are then filtered and fed back as context for the final pairwise CoT judgment. The filter keeps judgments where the candidate loses and strips their final verdicts. Across five judge benchmarks, CCE raises accuracy by 6.7% on average. Its richer CoTs also help in two other settings: distilling smaller judge models and selecting SFT data through "crowd rejection sampling". This matters because CCE offers a response-aware, test-time-scalable way to make automatic evaluation more thorough and less biased.

Key points

• Benchmarks: RewardBench, HelpSteer2, MTBench Human, JudgeBench and EvalBias, with an average accuracy gain of 6.7% over the vanilla judge.
• Crowd responses: These come from models of different sizes, such as Qwen2.5-0.5B-Instruct and Mistral-Nemo-Instruct-2407, sampled at varied temperatures.
• Selection strategy: "Criticizing Selection" with outcome removal performs best. With 16 crowd judgments it averages 80.3 accuracy versus 78.6 for random selection. Accuracy and CoT length grow as crowd judgments increase from 0 to 16.
• Distillation and data selection:
  • Distilling CCE judgments into Qwen 2.5-7B yields gains of 4.5–5.6%, with notably better bias robustness.
  • Whether 1K or 10K samples are drawn, crowd rejection sampling beats both random selection and the plain rejection-sampling baseline when evaluated on MTBench and AlpacaEval-v2.
• CoT analysis and code: GPT-4o key-point counting and Bart-base cross-attention coverage both show that CCE's CoTs are more comprehensive. Code is available at github.com/Don-Joey/CCE.