Vision-Language Models (API Usage)
How vision-language models work — just enough to use them in RAG pipelines and multimodal apps
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Explain how CLIP aligns image and text embeddings via contrastive learning
- •Describe how images are tokenized into patches for ViT-based models
- •Call GPT-4V/Claude/Gemini with image inputs via API
- •Build a simple visual Q&A pipeline
- •Understand the difference between image embeddings and text embeddings
What is it?
Vision-language models (VLMs) are models that can process both images and text together — enabling visual question answering, image captioning, and multimodal RAG. Scope here: API usage and embedding concepts only.
Why it exists
Many real-world documents contain images, charts, and diagrams that text-only RAG systems miss. VLMs enable reasoning over visual content alongside text.
Problem it solves
RAG pipelines that skip images in PDFs, losing critical information. Applications needing to describe screenshots or reason about visual content in documents.
Intuition
Think of CLIP as teaching a model to associate a photo of a dog with the word "dog" by showing it millions of image-caption pairs and training it to push matching pairs closer in vector space.
Analogy
Text embeddings map words to a coordinate in meaning-space. Image embeddings map visual content to the SAME coordinate space — so "a cat on a mat" (text) and [photo of cat on mat] (image) land near each other.
Technical explanation
CLIP (Contrastive Language-Image Pre-training) trains an image encoder (ViT) and text encoder (Transformer) jointly. Contrastive loss maximizes similarity between matching image-text pairs and minimizes it for non-matching pairs in the same batch. ViT (Vision Transformer) splits images into fixed-size patches (e.g., 16×16 pixels), flattens each patch into a vector, and processes the sequence of patch vectors through a Transformer — treating patches like tokens. VLM APIs (GPT-4V, Claude claude-3-sonnet, Gemini Vision) accept images as base64-encoded strings alongside text prompts. The image is processed by a vision encoder, its embeddings are projected into the text embedding space, and the LLM reasons over both.
Architecture
CLIP: Image → ViT encoder → image embedding (512/768d). Text → Transformer encoder → text embedding (same dim). Similarity = cosine(image_emb, text_emb). VLM: [image_patch_tokens + text_tokens] → cross-attention or concatenation → LLM → response.
Workflow
- For similarity search: encode image with CLIP, store in vector DB, query with text.
- For VQA: encode image as base64, send with question to VLM API.
- For multimodal RAG: index document screenshots with CLIP, retrieve relevant images, send to VLM for answer generation.
Example
import anthropic, base64 client = anthropic.Anthropic() with open("diagram.png", "rb") as f: img_b64 = base64.b64encode(f.read()).decode() message = client.messages.create( model="claude-opus-4-8", messages=[{"role": "user", "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img_b64}}, {"type": "text", "text": "What architecture does this diagram show?"}]}] )
Real-world usage
Document understanding (PDF with charts), visual QA for e-commerce (product images), diagram interpretation in engineering docs, screenshot analysis for UI testing.
Trade-offs
VLM API cost vs local CLIP: API is simpler but expensive at scale. Local CLIP embeddings are cheap for indexing but lack reasoning capability.
Visual explanation
CLIP: [Image Encoder] → image_embedding [Text Encoder] → text_embedding Contrastive loss pulls matching pairs together, pushes non-matching apart.
VLM API call: send base64 image + text prompt → text response
Advantages
- —
Unlocks visual content in document pipelines
- —
CLIP embeddings work zero-shot across domains
- —
VLM APIs require no local GPU
Disadvantages
- —
Image tokens are expensive (1 image ≈ 1000 tokens in GPT-4V)
- —
CLIP embeddings are weaker than specialized vision models for specific tasks
- —
Base64 encoding adds latency and payload size
Common mistakes
- —
Sending full-resolution images to VLM APIs (resize to 1024px first)
- —
Using CLIP for visual reasoning tasks — CLIP only produces embeddings, not answers
- —
Ignoring image preprocessing (normalization, format) for CLIP
🎤 Interview questions
What is CLIP and how does contrastive learning work?
How do you add image understanding to a RAG pipeline?
What is the difference between image embeddings and text embeddings?
📂 Subtopics
How Vision-Language Models Work: Encoders, Decoders, and CLIP's Shared Embedding Space
A VLM needs to turn pixels into something a language model can reason over. CLIP's key idea: train an image encoder and a text encoder together so matching image/text pairs land at the SAME point in a shared embedding space.
~13 min
Key Architectures: CLIP, LLaVA, and GPT-4V/Gemini Vision
Three architectures at three different points on the same spectrum: CLIP aligns embeddings but doesn't generate text, LLaVA connects a CLIP-style encoder to an open LLM via a small trainable bridge, and GPT-4V/Gemini Vision are natively multimodal frontier models.
~13 min
Common Tasks: Image Captioning, Visual QA, Document Understanding, and OCR
Four task categories cover most real VLM use cases — describing an image, answering questions about it, understanding structured documents, and reading text within an image — each with different difficulty and reliability profiles.
~12 min
Using VLMs in Practice: API Usage and Prompt Design for Images
Calling a vision-capable API follows the same request/response shape from api-basics, with images sent as base64 or URLs alongside text — and prompt design for images has its own specific best practices.
~13 min