💡 The paper addresses the issue of prefix failure in on-policy distillation, where the student model commits to a wrong reasoning direction and subsequent generations build on this deviation, producing misguided continuations. The authors identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction. They propose Relay On-Policy Distillation, which converts this asymmetry into a label-free handoff trigger.

During training, Relay On-Policy Distillation constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy.

The method is evaluated on eight mathematical reasoning benchmarks with a Qwen 3-4B-Instruction-2507 teacher and Qwen 3-0.6B/1.7B-Non-Thinking students. The results show that Relay On-Policy Distillation achieves the best or second-best results on every benchmark, outperforming standard on-policy distillation by 5.73 percent and the strongest baseline Fast On-Policy Distillation by 1.49 percent on average for 1.7B, with consistent gains at 0.6B. Additionally, the training trajectory length is reduced by over 50 percent.

Overall, the paper proposes a novel method to address prefix failure in on-policy distillation, achieving significant improvements in performance and efficiency.