💡 The paper proposes a new framework called Latent-to-Pixel transfer paradigm, or L2P, which allows for efficient creation of pixel-space models using pre-trained latent diffusion models. The problem addressed is that training advanced pixel-space models from scratch requires significant computational and data resources. To solve this, L2P harnesses the knowledge of pre-trained latent diffusion models to build powerful pixel-space models with minimal training overhead.

The method involves discarding the Variational Autoencoder in favor of large-patch tokenization and freezing the intermediate layers of the pre-trained latent diffusion model. Only the shallow layers are trained to learn the latent-to-pixel transformation, using synthetic images generated by the pre-trained model as the training data. This approach enables rapid convergence without the need for real data collection.

The results show that L2P achieves negligible training overhead while performing on par with the source latent diffusion model. The framework is able to migrate massive latent priors to the pixel space using only 8 GPUs, and it unlocks native 4K ultra-high resolution generation by eliminating the Variational Autoencoder memory bottleneck. Extensive experiments demonstrate that L2P reaches 93 percent performance on GenEval and performs well on DPG-Bench, making it a promising approach for efficient pixel-space model creation.