Elamur Research⌕
☰
home / papers / 2026 / PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation

PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation

Paper · 2026 · Yongfu Xue

Summary

PIRA targets two weaknesses of discriminative reward models used in RLHF. First, they need large amounts of labeled data because they feed a plain question–answer concatenation and never tell the LLM that the task is scoring. Second, policies tend to exploit them (reward overoptimization). PIRA addresses both with three strategies:

  • Instruction framing. Each input is wrapped in a preference-evaluation instruction drawn from a set that an LLM generated and humans reviewed.
  • Instruction averaging. At inference, rewards are averaged across several such instructions.
  • Value-head dropout averaging. Many dropout passes, at varying rates, are run through the lightweight value head only, on top of a single deterministic backbone pass.

Training uses the Bradley–Terry loss with LoRA and separate learning rates for the backbone and the head. Across four backbones and six public preference datasets, PIRA mostly beats a plain baseline, Thomas and WARM. In PPO experiments, it keeps KL divergence bounded and lets the gold reward rise steadily instead of collapsing. The approach therefore gives cheaper, more stable reward models that are harder to hack.

Key points

  • Setup: Backbones were Mistral-7B-v0.1, LLaMA3-8B, Qwen2.5-1.5B and Qwen2.5-7B. Datasets were HH, HH-cleaned, SHP, Alpaca-farm, Oasst and UltraFeedback. Inference used K=6 instructions and M=12 dropout passes, with rates sampled from Uniform(0.1, 0.4).
  • Average accuracy gains over the baseline:
    • Mistral-7B: 65.7 → 70.1
    • LLaMA3-8B: 65.0 → 69.8
    • Qwen2.5-7B: 63.5 → 69.8
    • Qwen2.5-1.5B: 62.1 → 66.1, where WARM* is marginally higher at 66.2
  • Ablation (LLaMA3-8B, HH-cleaned): Instruction framing alone lifts accuracy from 64.3 to 73.0. Adding both averaging steps reaches 75.5 and cuts the standard deviation from 2.1 to 0.7.
  • Cost: Value-head averaging adds only about 7% latency at M=12, while K=6 instructions multiply inference cost roughly sixfold.
  • Reward-hacking test: PPO used Qwen2.5-1.5B as policy and reward model, with Qwen2.5-7B as the gold judge, on Alpaca-farm. PIRA suppressed reward inflation compared with the baseline and Thomas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yongfu Xue. 2026. PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.117

@inproceedings{xue2026pira,
  title = {{PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation}},
  author = {Yongfu Xue},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.117},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.117}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791. +0 +2 +3 +5 +7 *Table 4: Impact of Value-Head Stochastic Averaging.* find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xue-2026-pira-preference-oriented-instruction-tuned/