Vector Embeddings Explained
Learn how text tokens are mapped to dense vector arrays, representing semantic meanings in multi-dimensional space.
Embedding Inputs:
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Describe vector representation of words as lists of numbers
- •Explain how similarity corresponds to proximity in high-dimensional space
- •Understand how models compute dense embedding vectors
What is it?
A vector embedding is a dense numerical representation of semantic meaning — a list of floating-point numbers (typically 768 to 3072 values) that encodes the meaning of text into a point in high-dimensional space. Similar meanings occupy nearby points; unrelated meanings are far apart. Embeddings are the foundation of semantic search, RAG, and recommendation systems.
Why it exists
Computers cannot calculate conceptual similarity between text strings — "car" and "automobile" share no characters but have identical meaning. Embeddings solve this by mapping text into a geometric space where semantic similarity equals proximity. This enables retrieval by concept rather than exact keyword match.
Problem it solves
Keyword search fails when users do not use the exact words in the document — a user asking "how do I cancel my order?" should find a document about "order cancellation" even without exact word overlap. Embeddings power semantic search that understands intent and meaning, not just character strings.
Intuition
Imagine a giant coordinate system where every possible piece of text has a location. Words and sentences with similar meanings cluster together in this space. "Cat" and "kitten" are neighbors. "Cat" and "automobile" are across the map from each other. An embedding is just the coordinates of a piece of text in this space.
If you come from Java/Spring Boot: an embedding is like a hash code that encodes semantic meaning instead of exact value. Two strings that are semantically equivalent will have similar (nearby) embeddings, unlike hashcodes that are completely different for semantically identical strings. The vector database is like a spatial index (R-tree) that can find "nearby" hash codes efficiently.
If you come from React/Frontend: an embedding is like a serialized semantic object. Instead of JSON {"meaning": "fast four-wheeled vehicle"}, you get a compact 1536-number array that encodes the same information in a format computers can compare mathematically. The cosine similarity between two arrays is equivalent to checking how semantically similar two objects are.
Analogy
Embeddings are like GPS coordinates for meaning. Just as latitude/longitude uniquely locates any place on Earth, an embedding uniquely locates any piece of text in semantic space. "Paris, France" and "City of Light" have different GPS coordinates but map to nearby semantic coordinates. A search for "European capitals" in semantic space automatically finds both.
Technical explanation
Embedding models are transformer encoders trained with contrastive objectives: similar sentence pairs are pulled together in vector space, dissimilar pairs are pushed apart.
Architecture: text → tokenize → transformer encoder (12–24 layers) → [CLS] token hidden state OR mean pooling of all token states → L2-normalize → embedding vector.
Popular models and their specs: text-embedding-3-small (OpenAI): 1536-dim, $0.02/1M tokens, ~62% MTEB score text-embedding-3-large (OpenAI): 3072-dim, $0.13/1M tokens, ~64% MTEB score BAAI/bge-m3 (open source): 1024-dim, free, multilingual, ~67% MTEB score all-MiniLM-L6-v2 (sentence-transformers): 384-dim, tiny/fast, good for dev/testing
Context limits: text-embedding-3-small handles up to 8191 tokens. Content longer than the limit is truncated — this is why chunking is critical for RAG.
Computing similarity: cosine similarity = dot_product(a, b) / (|a| × |b|). For normalized vectors (L2 norm = 1), cosine similarity = dot product. Values range from -1 (opposite) to 1 (identical). In practice, RAG retrieval uses a threshold of 0.7–0.8 to filter relevant chunks.
Architecture
Embedding pipeline for RAG: Input: "How does RAG work?" → Tokenizer: [101, 2129, 2515, ...] (token IDs) → Transformer encoder: 12 layers of self-attention → Pooling: average of all token hidden states → L2 normalization: vector with unit length → Output: [0.023, -0.156, 0.891, ...] (1536 floats)
Workflow
- Choose embedding model: OpenAI text-embedding-3-small for best quality/cost, bge-m3 for local/free
- Chunk documents: split long text into 256–512 token pieces (embedding models have context limits)
- Embed each chunk: call embedding API or run model locally → get 1536-dim vector per chunk
- Store in vector DB: upsert vector + original text + metadata (source, page, timestamp)
- At query time: embed the user question with the SAME model
- Search: find top-K chunks by cosine similarity to query embedding
- Return: the original text of matching chunks (not the vectors themselves)
Example
import anthropic import numpy as np
Use Anthropic for generation, but need embeddings separately
Example with sentence-transformers (free, local)
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, fast
Embed documents
docs = ["RAG retrieves relevant documents", "Fine-tuning updates model weights"] embeddings = model.encode(docs) # shape: (2, 384)
Embed query and find similar
query = "How does retrieval augmented generation work?" q_emb = model.encode([query]) # shape: (1, 384)
Cosine similarity
from sklearn.metrics.pairwise import cosine_similarity scores = cosine_similarity(q_emb, embeddings) # [[0.89, 0.23]] print(f"Most similar: {docs[np.argmax(scores)]}") # RAG doc
Real-world usage
Every RAG system uses embeddings: Notion AI, GitHub Copilot, Cursor IDE, Perplexity, and every enterprise knowledge base tool. OpenAI processes billions of embedding requests daily. The MTEB benchmark (Massive Text Embedding Benchmark) is the standard for comparing embedding model quality across 56 tasks.
Trade-offs
Embedding dimension vs performance: higher dimensions (3072 vs 384) capture more nuance but use 8× more storage and compute for similarity calculations. For most production RAG systems, 1536 dimensions (text-embedding-3-small) provides an excellent quality/cost balance.
Local vs API embeddings: running bge-m3 locally costs only hardware; OpenAI API costs $0.02/1M tokens but is easier to deploy. For 100M tokens/month, local saves ~$2K/month at the cost of serving infrastructure.
Visual explanation
Embedding Dimension Mapping: Word "Puppy" ──(Embedding Model)──> Vector [0.23, -0.45, 0.89, ...] (Coordinates in 1536-D space)
Semantic clustering: [king, queen, prince, princess] → cluster in one region [cat, dog, puppy, kitten] → cluster in another region Distance between clusters >> distance within clusters
Advantages
- —
Enables semantic matching
- —
Reduces text representations from huge sparse vectors to small dense arrays
Disadvantages
- —
Embedding models have fixed context limits
- —
Vectors cannot capture context drift easily without complex architectures
Common mistakes
- —
Thinking that embedding models can generate text. Embedding models are encoders — they produce a vector from text. They cannot produce new text. Text generation requires decoder models (GPT, Claude, Llama). Common confusion: "I'll use text-embedding-3-small to answer questions" — no, you use it to find relevant context, then pass that context to a generative model.
- —
Comparing embeddings from different models. A vector from text-embedding-3-small and a vector from bge-m3 live in completely different coordinate systems. Cosine similarity between them is meaningless. Always use the exact same model for all embeddings in a single index.
- —
Not normalizing vectors before dot-product similarity search. If your vectors are not L2-normalized (unit length), dot product does not equal cosine similarity — longer documents would dominate search results regardless of relevance. Most embedding APIs return normalized vectors, but verify this for any model you use.
- —
Embedding entire documents without chunking. A 10-page PDF has one embedding that averages over all 10 pages — the embedding for "Chapter 3 on RAG" is diluted by 9 other chapters. Chunk first, embed each chunk, retrieve at chunk level.
- —
Using the wrong embedding model for multilingual content. text-embedding-3-small works well for English. For mixed-language content, use multilingual models like BAAI/bge-m3 or OpenAI's multilingual embeddings — otherwise non-English content clusters poorly.
🎤 Interview questions
Why do we use dense embedding vectors instead of traditional sparse Bag-of-Words (one-hot) vectors?
📂 Subtopics
What Are Embeddings: From Words to Vectors
An embedding is a list of numbers that represents the MEANING of a piece of text — placed so that similar meanings end up close together in space, which is what makes semantic search possible.
~12 min
How Embeddings Are Created: From Word2Vec to Modern Sentence Transformers
Word2Vec pioneered learning embeddings by predicting context words around a target word. Modern sentence transformers extend that same core intuition — 'you shall know a word by the company it keeps' — to whole sentences.
~14 min
Embedding Dimensions: What 768 vs 1536 Means and the Tradeoffs
The dimension count of an embedding is how many numbers describe each piece of text — more dimensions can capture finer distinctions in meaning, at the cost of more storage and slower search.
~12 min
Using Embeddings in Practice: Generating Them and Computing Cosine Similarity
Turning the theory into code: call an embedding API or a local Hugging Face model to get vectors, then use cosine similarity — the standard metric — to measure how close two pieces of text are in meaning.
~13 min