How Vision-Language Models Work: Encoders, Decoders, and CLIP's Shared Embedding Space
~13 min read
A VLM needs to turn pixels into something a language model can reason over. CLIP's key idea: train an image encoder and a text encoder together so matching image/text pairs land at the SAME point in a shared embedding space.
How Vision-Language Models Work: Encoders, Decoders, and CLIP's Shared Embedding Space is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •A VLM needs an image encoder (converting pixels to embeddings, e.g. a Vision Transformer) and a text decoder (an LLM generating output conditioned on those embeddings plus text)
- •CLIP trains an image encoder and text encoder TOGETHER on massive (image, caption) pairs, using a contrastive objective
- •The contrastive objective pushes matching image/caption pairs to have high cosine similarity, and non-matching pairs to have low similarity, within each training batch
- •This produces a SHARED embedding space where images and text are directly comparable — a dog photo and the text 'a dog' land close together as vectors
- •This shared space enables zero-shot image classification (comparing an image to candidate text labels) and is the foundation later VLM architectures build on