Key Architectures: CLIP, LLaVA, and GPT-4V/Gemini Vision
~13 min read
Three architectures at three different points on the same spectrum: CLIP aligns embeddings but doesn't generate text, LLaVA connects a CLIP-style encoder to an open LLM via a small trainable bridge, and GPT-4V/Gemini Vision are natively multimodal frontier models.
Key Architectures: CLIP, LLaVA, and GPT-4V/Gemini Vision is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •CLIP aligns image/text embeddings but doesn't generate text — useful for retrieval and zero-shot classification, not open-ended conversation about images
- •LLaVA connects a frozen pretrained vision encoder and a frozen pretrained LLM via one small trainable projection layer — dramatically cheaper than training a multimodal model from scratch
- •GPT-4V and Gemini Vision are natively multimodal, with vision built into training from a much deeper stage, rather than bridging separately-pretrained pieces
- •Natively multimodal models generally handle harder visual reasoning (complex charts, multi-image, document understanding) more robustly, at higher training cost and no self-hosting option
- •The practical spectrum: CLIP for embedding tasks, LLaVA-style for cost-effective open conversational VLMs, GPT-4V/Gemini Vision when maximum capability matters most