Can On-Policy Self-Distillation Transform AI Post-Training?
A novel reinforcement learning technique known as on-policy self-distillation aims to solve the challenge of teaching foundation models new capabilities without degrading their existing performance. This approach could significantly streamline how developers refine large language models, offering a more stable pathway for post-training advancements in artificial intelligence development.
Source: Towards Data Science