How Vision-Language Models Work: Encoders, Decoders, and CLIP's Shared Embedding Space

An LLM (covered throughout this curriculum) only understands sequences of token embeddings — it has no native way to process a grid of pixel values. A vision-language model (VLM) needs some mechanism to bridge that gap: convert an image into a representation the language side of the model can reason over, alongside ordinary text tokens.

The general architecture has two main pieces. An image encoder (typically a Vision Transformer, or ViT — the same Transformer architecture from this curriculum's neural-networks material, but applied to image patches instead of text tokens) converts an input image into a set of embedding vectors, the same way a text encoder converts tokens into embeddings (recalling the embeddings-basics prerequisite topic). A text decoder (an ordinary LLM, essentially) then generates text output, conditioned on both the image embeddings and any text prompt — the same next-token-prediction process covered in probability-basics, just with image embeddings included in the context the model attends to.

CLIP (Contrastive Language-Image Pre-training), from OpenAI's 2021 paper, established the foundational technique for ALIGNING these two modalities — making an image embedding and a text embedding directly comparable, the way two text embeddings are comparable via cosine similarity (from vector-search-basics). CLIP trains an image encoder and a text encoder SIMULTANEOUSLY, on a massive dataset of (image, caption) pairs scraped from the internet, using a contrastive objective: for a batch of matched image/caption pairs, the training pushes each image's embedding to be CLOSE (high cosine similarity) to its correct matching caption's embedding, and FAR from every other caption's embedding in the same batch. Over enough training on enough pairs, the two encoders learn to place semantically related images and text into the SAME shared embedding space — a photo of a golden retriever and the text 'a golden retriever' end up genuinely close together as vectors, even though one started as pixels and the other as tokens.

This shared embedding space is what makes VLMs practically useful beyond just generating captions: it enables zero-shot image classification (compare an image's embedding against several candidate text descriptions' embeddings and pick the closest match, without ever training on that specific classification task), and it's the foundational building block that later architectures (covered in the next subtopic) build on to let a full LLM reason jointly over both images and text in a single conversation.

# Illustrating CLIP's contrastive alignment idea: push MATCHING
# image/text embedding pairs together, push non-matching pairs apart.
import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na, nb = math.sqrt(sum(x**2 for x in a)), math.sqrt(sum(y**2 for y in b))
    return dot / (na * nb) if na and nb else 0.0

# Toy embeddings AFTER CLIP-style training -- notice matching
# image/caption pairs land close together in the SAME space,
# even though one came from pixels and the other from text.
image_embeddings = {
    "photo_of_dog.jpg":  [0.9, 0.1, 0.2],
    "photo_of_cat.jpg":  [0.1, 0.9, 0.3],
}
text_embeddings = {
    "a photo of a dog": [0.88, 0.12, 0.25],   # close to photo_of_dog.jpg
    "a photo of a cat": [0.12, 0.87, 0.28],   # close to photo_of_cat.jpg
}

def zero_shot_classify(image_name: str, candidate_labels: list[str]) -> str:
    """Zero-shot classification: pick whichever text label's embedding
    is closest to the image's embedding -- no task-specific training needed."""
    img_vec = image_embeddings[image_name]
    scores = {label: cosine_similarity(img_vec, text_embeddings[label])
              for label in candidate_labels}
    return max(scores, key=scores.get)

labels = ["a photo of a dog", "a photo of a cat"]
print("photo_of_dog.jpg classified as:", zero_shot_classify("photo_of_dog.jpg", labels))
print("photo_of_cat.jpg classified as:", zero_shot_classify("photo_of_cat.jpg", labels))
print("\nSimilarity(dog photo, 'a photo of a dog'):",
      round(cosine_similarity(image_embeddings["photo_of_dog.jpg"], text_embeddings["a photo of a dog"]), 3))
print("Similarity(dog photo, 'a photo of a cat'):",
      round(cosine_similarity(image_embeddings["photo_of_dog.jpg"], text_embeddings["a photo of a cat"]), 3))