Related Works

Why Does Self-Distillation (Sometimes) Degrade Reasoning?

Analysis revealing that self-distillation can suppress reflective tokens essential for robust reasoning — a key motivation for RLRT.

COLM 2026

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Jeonghye Kim1,2*♦, Jiwon Jeon2*, Dongsheng Li1, Yuqing Yang1†
1Microsoft Research    2KAIST
NeurIPS 2026
*Equal contribution    †Corresponding author
♦Work done during an internship at Microsoft Research
Paper Code

RLRT is an LLM post-training method that teaches models how to think independently.

Encouraging the model to think differently from the teacher's guidance
while still getting the right answer leads to better reasoning.

The teacher tells the student to stop exploring; the student tries other methods and succeeds

In the end, for humans and AI alike, what really matters is learning how to think, beyond just memorizing known solutions. When facing a new problem, it’s how well you think that makes the difference.

Reversing the Teacher Signal

In self-distillation, a single model plays both roles: the student sees only the question, while the teacher also sees a privileged context — the ground-truth solution or a successful rollout. A teacher that already knows the answer has no reason to hesitate, so a student trained toward it drops words like wait and maybe and stops exploring. We flip the signal: on correct answers, we reward self-driven tokens, the choices the student made against the teacher's prediction that still led to the right answer.

Training score when upweighting teacher-predicted vs. self-driven tokens

Upweighting self-driven tokens during GRPO significantly improves training efficiency, while upweighting teacher-predicted tokens hurts it.

Explore-leaning vs. exploit-leaning reasoning markers

Self-driven tokens open new paths (wait, maybe, another);
teacher-predicted tokens close them (conclude, final, yield).

RLRT: RLVR with Reversed Teacher

RLRT is a simple modification to GRPO: it changes only how credit is assigned to tokens. The setup is the usual self-distillation one — the teacher sees a successful rollout that the student does not — but instead of pulling the student toward the teacher, we invert the ratio. On correct rollouts each token is weighted by \(w_t = P_S(y_t) / P_T(y_t)\), so tokens the student preferred over the teacher are amplified, where \(P_S\) and \(P_T\) are the student and teacher token probabilities.

RLRT training pipeline
\[ A_t^{\mathrm{RLRT}} = \begin{cases} A \cdot \bigl[(1-\lambda) + \lambda\cdot\mathrm{clip}(w_t,\, 1-\varepsilon_w,\, 1+\varepsilon_w)\bigr] & \text{if } r = 1, \\ A & \text{if } r = 0 \end{cases} \]

\(A\): GRPO advantage; \(r\): correctness; \(\lambda\) sets the weighting strength (\(\lambda = 0\) is GRPO); \(\varepsilon_w\) bounds the clipped weight.

Correct rollouts: amplify self-driven tokens (\(P_S > P_T\)). Incorrect rollouts: keep GRPO.

So if the student thought differently from the teacher and still got it right, it found a new path on its own — and we reinforce that even more. This acts as an important exploration signal in RLVR.

Training and Benchmark Performance

We train on DAPO-Math-17k with four Qwen3 backbones and evaluate on six math benchmarks (AIME24/25/26, HMMT26, AMC23, MATH500). RLRT trains faster and outperforms GRPO and self-distillation baselines on all four backbones, by up to 18%.

Training score curves on four Qwen3 backbones Mean avg@16 across six math benchmarks

The gap is largest on the base models: they have more room to explore, and that is where RLRT’s exploration signal helps most.

RLRT Expands Reasoning Beyond the Base Policy

On hard prompts, we measure how far each fine-tuned policy’s next-token distribution moves from that of the base policy, and label a token position as high-divergence when the Jensen–Shannon divergence exceeds 0.1. At these positions, GRPO and RLSD mostly sharpen tokens the base model already prefers, whereas RLRT reorganizes the set of top candidate tokens itself.

JS divergence CCDF and base probability of new top-1 tokens

(a) More positions change. Higher curve = more changed positions. RLRT changes the next-token distribution at more positions than GRPO or RLSD.

(b) Rare tokens become top-1. Tokens with base probability <0.001 become the most likely next token over 10× as often with RLRT as with GRPO or RLSD.

Comparison with Other Exploration Methods

We compare RLRT against two representative exploration methods: GRPO with an entropy bonus (GRPO+EB) for token-level entropy regulation, and DIVER for sequence-level diversity, using pass@k for k ∈ {1, 2, …, 256} on Qwen3-8B-Base.

pass@k comparison with exploration baselines on AIME24 and AIME26

Outperforms exploration baselines. On Qwen3-8B-Base, RLRT leads from pass@1 to pass@256 on AIME24 and AIME26.

BibTeX

@inproceedings{kim2026rebellious,
  title     = {Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR},
  author    = {Jeonghye Kim and Jiwon Jeon and Dongsheng Li and Yuqing Yang},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.10781}
}