RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

Yuxin Xiong, Xunyi Jiang, Rohan Surana, Xintong Li, Sheldon Yu, Nikki Lijing Kuang, Ryan A. Rossi, Jingbo Shang, Tong Yu, Julian McAuley, Junda Wu
July, 2026
PDF
Type
Preprint
Publication
arXiv preprint, 2026

RRPO extends group-relative reinforcement learning to tasks without a direct correctness verifier. It compares candidate rollouts against positive and negative reference sets to construct training advantages for reasoning and open-ended generation.

Reinforcement Learning Large Language Models
Xunyi Jiang
Xunyi Jiang
PhD in CS @ 🇺🇸NYU
MS in CS @ 🇺🇸UCSD
BS in DS @ 🇨🇳SUSTech

I am interested in LLM Evaluation, Generative Engine Optimization, Human-AI collaboration, AI for creativity🎻.