📄 Rhombus: Incentivizing Coordination in Parallel Thinking through Reinforcement Learning

Summary

Parallel reasoning lets an LLM explore several solution attempts at once and then merge them. In practice, the explorers and the merger are usually built separately. The explorers do not know what the aggregator needs, so useful intermediate evidence gets lost at the hand-off. Rhombus treats this hand-off as a communication problem and learns it with reinforcement learning.

A single shared backbone works in two prompted roles:

• Proposers: several run in parallel and each writes a short, truncated cue that ends with a tentative answer.
• Synthesizer: it sees only these cues, never the full traces, and reasons over them to produce the final answer.

Both roles are trained jointly with a GRPO-based objective and a shared task reward. Proposer rewards mix their own correctness with the success of the whole system. A role-level advantage normalization keeps training stable.

On hard math benchmarks, with the token budget matched to long chain-of-thought baselines, Rhombus is more accurate and noticeably faster. This suggests that the gains of parallel test-time compute depend on learned coordination, not just on adding more branches.

Key points

• Headline result: at a matched token budget, accuracy is 6.0% higher than LongCoT baselines. Wall-clock latency is 39.4% lower, with macro-average time falling from 129.17s to 78.26s on one H20 GPU.
• Scaling with more Proposers: the model was trained with N = 3 Proposers, yet it keeps improving up to N = 8 without saturating.
  • At N = 0, the comparison LongCoT checkpoint is about 2.7% better.
  • At N = 8, Rhombus leads it by 4.1%.
  • Rhombus also beats Majority Voting.
• Reward weighting ablation: the reward weight c is set to 1/3.
  • Rewarding only individual correctness (c = 1) causes "Communication Collapse," where proposals become very short and uninformative.
  • Rewarding only system success (c = 0) gives noisy credit, and Proposer accuracy stops improving.
• Coordination gap: the gap is measured as Proposer Pass@3 minus system Pass@1.
  • It shrinks from 4.6% to 0.7% compared with LongCoT-trained Proposers paired with the same Synthesizer.
  • In that baseline, Pass@3 is 72.1% but realized accuracy is only 67.5%.
• Source of the speedup: several short sequences avoid the quadratic attention cost of one long chain, and the Proposers can run concurrently.