📄 Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings

Summary

Most RL training for neural MT scores a translation with one number per sentence. This sparse signal does not say where errors occur, so learning is slow and unstable, especially for long inputs. The authors instead derive a reward for each token from XCOMET, a quality estimation model that marks error spans and labels them minor, major or critical. They propose a custom severity map that converts these labels into numeric token rewards, replacing standard MQM weights. The token rewards feed into REINFORCE and PPO.

They test the approach with NLLB (encoder-decoder) and TOWER (LLM-based) systems on EN→DE, EN→FR, DE→EN and FR→EN, using both small and large datasets. Token-level RL (tRL) beats sentence-level RL (sRL) on most automatic metrics. It also wins in human Direct Assessment, with larger gains for longer sentences, and mean rewards rise more steadily during training. The work shows that span-level quality metrics can serve as dense, human-aligned training signals for MT.

Key points

• Reward model and mapping: XCOMET supplies the token-level rewards. All words inside an error span receive the same severity value, set by the new severity map.
• PPO vs. REINFORCE: Both algorithms work, but PPO gives the best overall scores. On IWSLT2017 EN→FR, for example, tPPO with XCOMET-MQM reaches 87.17 and 92.17 on two of the reported metrics.
• Cost of the reward model: As a tRL reward, XCOMET yields COMET22 of 74.66 versus 72.22 for BLEU (TOWER, WMT24 EN→DE). The trade-off is much higher latency and lower throughput.
• Human evaluation on WMT24 EN→DE: Two professional translators rated outputs, with 50 samples per length bin. tRL scored an average DA of 82.6, which is 2.3 points above sRL. Inter-annotator agreement was Pearson r = 0.59 and Spearman ρ = 0.57.
• Hybrid variant: A model using sRL for short inputs and tRL for long ones beats sRL but still falls behind pure tRL (Appendix B).