Vector Search & Similarity Mathematics
Learn Cosine Similarity, Dot Product, Euclidean distance, and Nearest Neighbors indexing used in RAG systems.
Vector Metric Rules:
Click anywhere inside the coordinate vector grid to position the Query vector. The top-3 nearest document nodes are highlighted in emerald.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Calculate Dot Product between two vectors
- •Explain Cosine Similarity and why it ignores vector magnitude
- •Understand Nearest Neighbors search logic
What is it?
Vector search (also called similarity search or nearest neighbor search) finds the K most similar items in a database of vectors by computing mathematical distance or similarity between a query vector and stored vectors. It is the retrieval engine inside every RAG system — when you ask a question, your question is embedded into a vector and the vector database finds the document chunks with the most similar vectors.
Why it exists
Traditional database indexes (B-trees, hash indexes) find exact matches. They cannot answer "find me documents semantically similar to this query." Vector search fills this gap — it is the only way to efficiently search a corpus of millions of documents for semantic relevance rather than exact keyword overlap.
Problem it solves
Finding relevant information when you do not know the exact words. A customer support bot needs to find relevant help articles even when the customer's phrasing does not match the article headings. Vector search enables this "fuzzy" semantic matching at production scale — millions of vectors in milliseconds.
Intuition
If you stand at a point on a GPS map, vector search is finding the 3 closest coffee shops — regardless of what their names are or whether they share any letters with your search query. The only thing that matters is the geometric distance between your position (query vector) and theirs (document vectors).
If you come from Java/Spring Boot: vector search is like a geospatial query with spatial indexes. Just as PostGIS finds restaurants near a GPS coordinate, a vector database finds documents "near" a semantic coordinate. HNSW is to vector search what an R-tree index is to geospatial queries — both enable fast nearest-neighbor lookups without scanning every record.
If you come from React/Frontend: vector search is like fuzzy search in a UI (Fuse.js) but for semantic meaning. Fuse.js finds strings that are lexically similar (typos, partial matches). Vector search finds content that is conceptually similar — completely different words but same idea.
Analogy
Vector search is like finding your neighborhood in a foreign city by showing someone a photo of your house. You do not speak the language, you do not have an address — but you have a visual representation of what you're looking for, and they can match it to similar-looking areas on a map. The photo is your query embedding; the map is the vector database.
Technical explanation
Similarity metrics:
-
COSINE SIMILARITY: measures the angle between two vectors, ignoring magnitude. cos(θ) = (A·B) / (|A| × |B|) = Σ(aᵢ × bᵢ) / (√Σaᵢ² × √Σbᵢ²) Range: -1 (opposite) to 1 (identical). For normalized vectors: just the dot product. Use case: most text embedding similarity. Magnitude-independent — a long document and a short one about the same topic get similar scores.
-
DOT PRODUCT: A·B = Σ(aᵢ × bᵢ) Faster than cosine (no normalization). Only equivalent to cosine if vectors are L2-normalized. Use case: when you control normalization (most production embedding APIs normalize by default).
-
EUCLIDEAN DISTANCE (L2): √Σ(aᵢ - bᵢ)² Measures actual geometric distance. Sensitive to vector magnitude. Use case: image embeddings, when magnitude carries meaning.
Approximate Nearest Neighbor (ANN) algorithms: Exact search: compute similarity to every vector — O(n×d) time. Correct but too slow for 1M+ vectors. HNSW (Hierarchical Navigable Small World): graph-based index. O(log n) search. Widely used, excellent recall. IVF (Inverted File): partition vectors into k clusters, search only nearby clusters. Fast but lower recall. Product Quantization (PQ): compress vectors to reduce memory. Enables billion-scale search.
Production vector databases: Pinecone (managed), Weaviate (open source), Chroma (local dev), pgvector (Postgres extension), Qdrant (open source, Rust-based).
Architecture
Vector search index structure (HNSW): Layer 2 (sparse): [v1] ──── [v7] ──── [v15] Layer 1 (medium): [v1]─[v3]─[v5]─[v7]─[v10]─[v15] Layer 0 (dense): all vectors with full neighbor connections
Search algorithm:
- Enter at top layer, find approximate nearest neighbor
- Descend to next layer, refine search around that area
- At bottom layer, return exact top-K neighbors → O(log n) comparisons instead of O(n)
Workflow
- Convert query to vector: embed query text with same model used for indexing
- Choose similarity metric: cosine for text, L2 for image/audio
- Query vector DB: top-K search (K=5 typical for RAG)
- Set similarity threshold: filter results below 0.70 to remove irrelevant chunks
- Return top matches: original document chunks, not the vectors themselves
- Optional rerank: cross-encoder reranker rescore top-20 candidates, return top-5
Example
import numpy as np a = np.array([0.1, 0.9]) b = np.array([0.15, 0.88]) sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) print(sim) # Output: ~0.999 (highly similar)
Production: Chroma vector DB with similarity threshold
import chromadb client = chromadb.Client() col = client.get_or_create_collection("docs")
Query with threshold
results = col.query( query_texts=["How does RAG work?"], n_results=5, where_document={"$contains": "retrieval"} # optional filter ) for doc, score in zip(results["documents"][0], results["distances"][0]): if score < 0.3: # chroma uses L2 distance; 0 = identical print(f"Relevant ({score:.2f}): {doc[:100]}")
Real-world usage
Retrieving context blocks from a Milvus vector database during a user question session.
Trade-offs
Approximate search (like HNSW) is fast but can miss the absolute nearest neighbors (compromises recall).
Visual explanation
Vector Space search: Query Vector ──(Calculate Cosine Angle)──> Identifies document vectors with smallest angles (highest similarity)
Advantages
- —
Captures semantic matchings
- —
Highly scalable using Approximate Nearest Neighbor (ANN) algorithms
Disadvantages
- —
Computationally heavy compared to standard database index lookups
- —
Vector database updates require index rebuild overhead
Common mistakes
- —
Using Dot Product search on vectors that are not normalized (magnitudes will bias search results)
- —
Using different similarity metrics during indexing and retrieval (e.g. indexing with Cosine but searching with L2)
🎤 Interview questions
Derive the equation for Cosine Similarity. Why does normalizing vectors simplify search calculation?
📂 Subtopics
The Search Problem: Why Keyword Search Fails for Semantic Queries
Keyword search only matches literal words, so it misses queries that mean the same thing but use different words. Vector search fixes this by matching MEANING, not spelling.
~11 min
Similarity Metrics: Cosine Similarity, Dot Product and Euclidean Distance
Three ways to measure how 'close' two vectors are — cosine similarity (angle), dot product (angle + magnitude), and Euclidean distance (straight-line distance) — each suited to different situations.
~13 min
Approximate Nearest Neighbor (ANN) Search and HNSW Intuition
Checking every stored vector against a query doesn't scale past a few hundred thousand documents. ANN algorithms like HNSW trade a tiny bit of accuracy for massive speed by building a searchable shortcut structure instead.
~14 min
Vector Databases: FAISS vs Pinecone vs Weaviate vs ChromaDB
A quick, practical comparison of the four vector databases you'll encounter most — a local library, a managed cloud service, and two flexible open-source servers — and when to reach for each.
~13 min