A short summary of must-know papers in image generation
This is a short list of papers on image generation that I compiled mostly for myself, but figured I might as well put it into a public place.
There are loads of papers being published every day, so this list is very much non-exhaustive. I try to pick the papers with the biggest practical use for people who try experiment with image generation models themselves.
CLIP (Learning Transferable Visual Models From Natural Language Supervision)
The authors scrape 400 million <text, image> pairs from the internet. They then train a text encoder and image encoder to maximise the cosine similarity of each actual <text, image> pair while minimising the cosine similarity between <text, image> pairs that don’t belong together. They use a transformer as text encoder and experiment with both a ResNet and a Vision Transformer as image encoder.
The resulting model can be used for image retrieval given text or images, zero-shot or few-shot image classification or as a guidance for text-to-image models (see DALL-E, Stable Diffusion).
DALL-E 2 (Hierarchical Text-Conditional Image Generation with CLIP Latents)
The authors use a dataset of <text, image> pairs to create a text-to-image model, by encoding the text using the CLIP text encoder, then using a “prior” model to generate a CLIP image embedding from the text embedding and then finally feed the image embedding together with the text into a decoder model that generates the final image. For the prior model, the authors experiment with both an autoregressive transformer model and a diffusion model. For the transformer model, they dimensionality-reduce and discretise the image embeddings to get discrete tokens. The decoder is a diffusion model conditioned on the image embedding and the text caption.
DALL-E 2 can generate variations of images by encoding an image using the CLIP image encoder and then feeding that image through the DALL-E 2 image decoder.
GitHub: https://github.com/lucidrains/DALLE2-pytorch (unofficial implementation)
Stable Diffusion (High-Resolution Image Synthesis with Latent Diffusion Models)
The central idea in Stable Diffusion is to perform the diffusion process in a compressed latent space instead of the more expensive pixel space. To do that, the authors train a denoising autoencoder to encode images into a latent space with two spatial dimensions. A UNet is trained to denoise images in that latent space. The UNet can be conditioned using cross attention. To condition the model on text, the authors use a CLIP text encoder to generate text embeddings to feed into the cross attention. The model is trained on the LAION-400m dataset, i.e. 400 million <image, text> pairs.
Stable Diffusion was the first open source image generation model that was practically useful while running on consumer hardware.
Textual Inversion (An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion)
Textual inversion adds a new token to represent a specific concept (e.g. a person) or style (e.g. Picasso painting). The embedding of that token is trained through a frozen stable diffusion model to on a small number of images of that concept, using prompts like “A photo of S” where S is the new token.
Conditions a Stable Diffusion model on additional inputs (e.g. edge maps, segmentation maps) by freezing the SD model, adding a trainable copy of the SD model and adding feature maps from the trainable model to the feature maps from the frozen model. Augmenting the frozen model with an additional trainable model ensures the resulting model doesn’t “forget” concepts when trained on smaller datasets.
An ablation study (https://github.com/lllyasviel/ControlNet/discussions/188) shows that the deep encoder (aka. trainable copy of the SD model) can be replaced by a lighter encoder model or even an MLP trained from scratch without losing much performance on text-to-image generation. The deep encoder model does significantly outperform both alternatives when no text prompt is given (i.e. the image is only conditioned on e.g. edge maps).