Key Architectures: CLIP, LLaVA, and GPT-4V/Gemini Vision

The previous subtopic's CLIP established how to align image and text embeddings — but CLIP itself doesn't GENERATE text; it only measures similarity between existing images and text. This subtopic covers how different architectures build FROM that foundation toward models that can actually converse ABOUT images.

CLIP, recapped in this context: two separate encoders (image, text) trained to share an embedding space. Its practical uses are things like zero-shot classification and image/text retrieval (finding images that match a text query, or vice versa) — genuinely useful, but it can't hold a conversation about an image or answer an open-ended question like 'what's unusual about this photo?'

LLaVA (Large Language and Vision Assistant) shows the more common pattern for building a conversational VLM WITHOUT training a giant multimodal model completely from scratch: take a pretrained, frozen CLIP-style image encoder, take a separate pretrained, frozen open LLM (like LLaMA or Vicuna), and train a small, lightweight 'projection' layer whose only job is translating the image encoder's embeddings into the same representational space the LLM's own text embeddings live in. Critically, the two big pretrained pieces (the vision encoder and the LLM) are largely kept frozen or only lightly fine-tuned — only that small connecting projection layer (plus some instruction-tuning on multimodal conversation data) needs real training. This makes LLaVA-style architectures dramatically cheaper to build than training a full multimodal model from zero, since it reuses two already-expensive, already-trained components and just teaches them to 'speak the same language.'

GPT-4V (and its successors) and Gemini Vision represent a different design point: NATIVELY multimodal frontier models, where vision capability is built into the model's core training from a much earlier and deeper stage, rather than bridging two separately-pretrained components after the fact. The publicly documented tradeoffs reflect this: these models generally handle a wider range of visual reasoning tasks more robustly (complex charts, multi-image reasoning, fine-grained document understanding) than the LLaVA-style bridged approach typically achieves, at the cost of being proprietary, far more expensive to train from scratch, and only accessible via API rather than as an open, self-hostable model.

The practical spectrum this creates: CLIP alone for embedding-based retrieval/classification tasks that don't need open-ended generation; LLaVA-style bridged architectures for cost-effective, open, self-hostable conversational VLMs built by connecting existing components; and GPT-4V/Gemini Vision-class natively multimodal models when maximum visual reasoning capability matters more than cost or self-hosting control — a decision that mirrors the model-selection tradeoffs covered elsewhere in this curriculum's llm-deployment material, just for the vision-language dimension specifically.

# Modeling the STRUCTURAL difference between LLaVA's bridged
# architecture (frozen encoder + frozen LLM + small trainable
# projection layer) and a natively multimodal model.

class LlavaStyleVLM:
    """Two big FROZEN pretrained pieces, connected by one small
    TRAINABLE projection layer -- the only genuinely new training
    needed is that thin bridge."""
    def __init__(self):
        self.vision_encoder_params = 300_000_000   # frozen, e.g. a CLIP-style ViT
        self.llm_params = 7_000_000_000            # frozen, e.g. LLaMA/Vicuna
        self.projection_layer_params = 5_000_000    # the ONLY newly-trained piece

    def trainable_fraction(self) -> float:
        total = self.vision_encoder_params + self.llm_params + self.projection_layer_params
        return self.projection_layer_params / total

class NativelyMultimodalVLM:
    """Vision capability is trained into the model from a much earlier,
    deeper stage -- most/all parameters are involved in multimodal training."""
    def __init__(self):
        self.total_params = 200_000_000_000   # hypothetical frontier-scale model
        self.multimodal_trained_params = self.total_params   # ~all of it

    def trainable_fraction(self) -> float:
        return self.multimodal_trained_params / self.total_params

llava = LlavaStyleVLM()
native = NativelyMultimodalVLM()

print(f"LLaVA-style: only {llava.trainable_fraction():.2%} of total params "
      f"need new training (the projection layer)")
print(f"Natively multimodal: {native.trainable_fraction():.0%} of params "
      f"are involved in multimodal training from the start")
print("\n-> This is exactly why LLaVA-style bridging is dramatically")
print("   cheaper to build than training a frontier multimodal model")