APSD: Correctness-Dependent Asymmetric Target Distributions

Abstract

Self-distillation policy optimization provides dense token-level supervision for LLM posttraining through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6× fewer training steps on average across Qwen3 model scales.

APSD adapts the self-distillation target to rollout correctness: preserve successful behavior and push more strongly away from failed behavior.

APSD interpolates between student and teacher on correct rollouts and extrapolates beyond the teacher on incorrect rollouts.

APSD interpolates to preserve correct behavior and extrapolates to strengthen correction on incorrect behavior.

Why Does One Target Fail?

Self-distillation turns feedback into token-level supervision by asking the current policy to learn from a feedback-conditioned version of itself. But using that self-teacher as the same target for every rollout overlooks a crucial distinction: whether the student's behavior succeeded.

Teacher-student discrepancy in SDPO on correct and incorrect rollouts from LiveCodeBench v6.

Teacher-student discrepancy in SDPO. Direct matching can perturb a correct preference, while the teacher can retain the student's incorrect preference.

On correct rollouts, the self-teacher can shift probability away from the student's already-correct choice, so matching it directly risks disrupting successful behavior. On incorrect rollouts, the self-teacher may retain the same wrong preference as the student, providing too little correction toward a better alternative.

Key insight: preserve behavior on successful rollouts, but apply stronger correction to failed ones. The target should depend on rollout correctness.

How APSD Works

APSD overview: rollout sampling, correctness-dependent target construction, token-level self-distillation, and analysis.

Overview of APSD: sample rollouts, construct an asymmetric target based on correctness, and optimize token-level self-distillation.

Let π_θ be the student and π its feedback-conditioned self-teacher. APSD reweights the student distribution using the teacher-to-student probability ratio:

$$\pi_{\theta,\alpha}^{\mathrm{APSD}}(v) \propto \pi_{\theta}(v)^{1-\alpha}\,\pi(v)^{\alpha}$$

For correct rollouts, 0 < α < 1 yields interpolation. For incorrect rollouts, α > 1 yields extrapolation beyond the self-teacher. The paper characterizes this objective through three propositions.

Proposition 1: α balances reverse-KL and Rényi divergences

The APSD loss decomposes into the SDPO reverse-KL term and an additional Rényi-divergence term:

$$\mathcal{L}_{\mathrm{APSD}} = \alpha D_{\mathrm{KL}}(\pi_{\theta}\|\pi) - (1-\alpha)D_{\alpha}(\pi\|\pi_{\theta})$$

This second term changes sign with α around 1. For correct rollouts, its contribution counteracts excessive movement toward the teacher; for incorrect rollouts, it emphasizes alternatives favored by the teacher.

Proposition 2: α determines the target's position in reverse-KL geometry

The APSD target is a generalized reverse-KL barycenter of the student and self-teacher:

$$\pi_{\theta,\alpha}^{\mathrm{APSD}} = \underset{r\in\Delta(\mathcal{V})}{\operatorname{argmin}}\; (1-\alpha)D_{\mathrm{KL}}(r\|\pi_{\theta}) + \alpha D_{\mathrm{KL}}(r\|\pi)$$

When the policies differ, the target's KL distance from the student grows monotonically with α. Thus 0 < α < 1 keeps the target between student and teacher, whereas α > 1 moves it beyond the teacher.

Proposition 3: detaching the target collapses APSD to a scaled SDPO update

If the entire student-dependent target is detached during differentiation, the resulting gradient is only a rescaled SDPO gradient:

$$\nabla_{\theta}\mathcal{L}_{\mathrm{APSD}}^{\mathrm{sg}} = \alpha\,\nabla_{\theta}\mathcal{L}_{\mathrm{SDPO}}$$

Full target detachment discards the additional coupling term. APSD therefore retains gradient flow through the target so that α can shape the update direction, not merely its magnitude.

Experiments

We evaluate APSD with scalar verifiable rewards (RLVR) and rich textual feedback (RLRF). The paper reports RLVR results on scientific reasoning and tool-use benchmarks, and RLRF results on LiveCodeBench v6 across three Qwen3 model scales.

RLVR performance across five benchmarks.

RLVR performance on five benchmarks; the final column reports average accuracy.

RLRF performance on LiveCodeBench v6 across Qwen3 model scales.

RLRF performance on LiveCodeBench v6. APSD achieves the strongest overall results across the three Qwen3 scales.

Training Dynamics and Gradient Flow

Across Qwen3 model scales, APSD reaches comparable performance in fewer training steps than SDPO. Preserving gradient flow through the student-dependent target is also important: detaching the target weakens performance.

Training accuracy over steps for APSD, SDPO, and GRPO across Qwen3 model scales.

Training accuracy across model scales; shaded regions show variation across three random seeds.

Target-gradient ablation comparing SDPO, APSD with a detached target, and full APSD.

Correctness-Dependent Target Control

On LiveCodeBench v6, interpolation improves successful rollouts, while extrapolation provides stronger recovery on unsuccessful ones.

Execution accuracy for student, interpolation, self-teacher, and extrapolation sampling distributions on correct and incorrect rollouts.

Execution accuracy on regenerated suffixes. Change is measured in percentage points relative to the student distribution.

Ablation of Correctness-Specific Components

Interpolation and extrapolation each contribute to APSD, and combining both yields the strongest results across all three model scales.

Ablation results comparing GRPO, SDPO, interpolation-only APSD-C, extrapolation-only APSD-I, and full APSD across three Qwen3 scales.

Ablation results on LiveCodeBench v6 (Avg@4). APSD-C applies interpolation only to correct rollouts; APSD-I applies extrapolation only to incorrect rollouts.

Conclusion

We present APSD, a self-distillation method that constructs asymmetric targets based on rollout correctness. APSD interpolates between student and self-teacher distributions to preserve successful behaviors and extrapolates beyond the self-teacher to strengthen correction of unsuccessful ones. Our analyses characterize APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient flow through student-dependent targets. Experiments across RLVR and RLRF show strong overall performance against reinforcement-learning and self-distillation baselines, supporting the need for correctness-dependent student–self-teacher distribution mixing.

BibTeX

@misc{yoon2026correctnessdependentasymmetrictargetdistributions,
  title={Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization},
  author={Suwan Yoon and Seongryong Jung and Changhee Lee},
  year={2026},
  eprint={2610.25973},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.25973}
}