💡 The paper addresses the limited generalization of Supervised Fine-Tuning SFT for Large Language Models LLMs compared to reinforcement learning. The authors analyze the standard SFT gradients and find that they implicitly encode a problematic reward structure that restricts the model's generalization capabilities. To overcome this, they propose Dynamic Fine-Tuning DFT, a simple yet effective method that stabilizes gradient updates for each token by dynamically rescaling the objective function with the probability of the token. This approach significantly outperforms standard SFT across multiple challenging benchmarks and base models, demonstrating greatly improved generalization. Additionally, DFT shows competitive results in offline reinforcement learning settings, providing a simpler alternative to existing methods. The authors provide a theoretical motivation for their approach and demonstrate its effectiveness through experiments, substantially advancing the performance of SFT. The proposed method is a single-line code change, making it easy to implement, and the code will be made available for further use. Overall, the paper contributes to the improvement of SFT for LLMs, bridging the gap between theoretical insights and practical solutions.