What Are Embeddings: From Words to Vectors

Computers can't directly compare the MEANING of two pieces of text — 'meaning' isn't something you can subtract or measure a distance between, at least not in its raw text form. Embeddings solve this by converting text into a list of numbers (a vector) in such a way that texts with SIMILAR meaning end up with SIMILAR numbers. Once meaning becomes numbers, you can finally do math on it — and 'how similar are these two meanings' becomes a plain distance calculation.

Picture a big empty room where every possible word or sentence gets a specific spot to stand in. Words with related meanings stand near each other; unrelated words stand far apart. This course that introduces embeddings in its RAG chapter puts it simply: in a good embedding space, the embeddings of fruits are found close to each other, forming one cluster, while cities form another cluster elsewhere in the room. Nobody manually decided where 'apple' or 'Paris' should stand — the positions EMERGE from training a model on huge amounts of text, where words that tend to appear in similar contexts naturally drift toward similar positions.

Each embedding is a vector — literally just a fixed-length list of floating-point numbers, like [0.021, -0.384, 0.157, ...]. The room from the analogy isn't 2D or 3D like physical space; it typically has hundreds or thousands of dimensions (the next subtopic covers exactly what that dimension count means). You can't visualize a 768-dimensional room directly, but the same idea holds: 'distance' and 'direction' still make mathematical sense in that many dimensions, even though humans can't picture it.

Why does this matter so much for AI systems? Because it turns 'find text with similar MEANING' — a fuzzy, human, hard-to-program task — into 'find vectors that are close together' — a precise, fast, well-understood math problem. This is the foundation underneath semantic search, recommendation systems, clustering, and Retrieval-Augmented Generation (RAG): a user's query gets embedded into the same space as a document collection, and whichever documents landed nearest the query's position are treated as the most relevant ones. Every later subtopic in this unit — and the vector-search unit that follows it — builds directly on this one idea: meaning as position in space.

# A tiny, hand-built 2D 'embedding space' to make the analogy concrete
# -- real embeddings have hundreds of dimensions, but the distance
# math works identically regardless of how many dimensions there are.
import math

# Pretend embeddings (in reality these come from a trained model)
toy_embeddings = {
    "apple":  (0.9, 0.1),   # fruits cluster near (0.9, 0.1)
    "banana": (0.8, 0.2),
    "mango":  (0.95, 0.05),
    "paris":  (0.1, 0.9),   # cities cluster near (0.1, 0.9)
    "tokyo":  (0.15, 0.85),
}

def euclidean_distance(a, b):
    return math.sqrt(sum((ai - bi) ** 2 for ai, bi in zip(a, b)))

def nearest(word, embeddings):
    """Find which OTHER word's embedding is closest -- 'similar meaning'
    is just 'small distance' once text becomes vectors."""
    target = embeddings[word]
    others = {w: v for w, v in embeddings.items() if w != word}
    return min(others, key=lambda w: euclidean_distance(target, others[w]))

for word in toy_embeddings:
    print(f"nearest to {word!r}: {nearest(word, toy_embeddings)!r}")
# 'apple' and 'mango' land near each other (both fruits);
# 'paris' and 'tokyo' land near each other (both cities) --
# exactly the clustering-by-meaning the book describes