#Stable Diffusion: Latent Space

In 2021, the release of Latent Diffusion Models, later known as Stable Diffusion, solved the problem of computational cost in generative AI. While earlier diffusion models worked directly on image pixels, they were incredibly slow and resource-heavy. Researchers at LMU Munich and Runway proposed that generation should instead happen in a 'latent space'—a compressed mathematical representation of an image. It was a push to make high-quality generation accessible on consumer hardware by separating the act of creation from the act of rendering.

#Diffusion in Latent Space

Latent Diffusion architecture showing the interaction between the autoencoder, the U-Net, and the conditioning mechanism.

The technical shift was the use of an autoencoder to compress images into a smaller, more manageable space before applying the diffusion process. By removing imperceptible details and focusing only on the semantic essence of an image, the model can operate much more efficiently. As the authors stated, 'By decomposing the image formation process into a sequence of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results... we show that they can also be applied in the latent space of pre-trained autoencoders.' This allows for high-resolution synthesis without the need for massive computing clusters.