Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
This paper proposes a new method for advantage shaping in reinforcement learning called Contrastive Policy Optimization (CPO), which uses contrastive disagreement between reference-guided and vanilla generation distributions to indicate correctness. Practitioners might care because it can improve the effectiveness of reinforcement learning methods in generating correct responses.