How Embeddings Are Created: From Word2Vec to Modern Sentence Transformers

The previous subtopic described WHAT an embedding is; this one covers HOW a model actually learns to place words and sentences at sensible positions, rather than random ones.

Word2Vec (2013) was one of the first widely-used methods, and its core trick is beautifully simple: train a small neural network to predict a word's SURROUNDING words (its 'context'), and along the way, that network builds an internal numeric representation for every word that captures what kind of contexts it tends to appear in. This follows linguist J.R. Firth's famous idea: 'you shall know a word by the company it keeps.' The specific training setup Word2Vec popularized is called skip-gram: given a target word (say 'coffee'), the model is trained to predict which words are likely to appear NEARBY it in real sentences ('cup', 'morning', 'drink', 'hot'). Words that tend to share similar neighboring words — like 'coffee' and 'tea' — end up needing similar internal representations to succeed at this prediction task, and THAT shared internal representation IS the embedding. Nobody ever tells the model 'coffee and tea are similar' directly — the similarity emerges purely from both words appearing near similar company across millions of sentences.

Word2Vec has one major limitation: it produces exactly ONE fixed embedding per word, regardless of context — 'bank' gets the same vector whether you mean a riverbank or a financial bank. Modern sentence transformers (built on the Transformer architecture, the same family LLMs come from) fix this by computing embeddings that depend on the FULL surrounding sentence — the same word gets a different embedding in different contexts, because the model reads the whole sentence at once (using the self-attention mechanism) before deciding on a representation. Just as importantly, they embed WHOLE sentences or paragraphs directly into one single vector, not just individual words — exactly what's needed for RAG, where you embed whole document chunks and whole queries, not word by word (matching this course's distinction, from the RAG chapter, between 'word embedding models' and the 'context embedding models' that power modern retrieval).

In practice today, you almost never train an embedding model from scratch. You download a pretrained one (from Hugging Face's sentence-transformers library, or call an API like OpenAI's) and use it directly — the heavy lifting of learning 'what makes two things semantically similar' from billions of words has already been done for you.

# Illustrating the skip-gram TRAINING SIGNAL Word2Vec uses --
# extracting (target_word, context_word) pairs is literally the
# training data; the neural network then learns embeddings that
# make these pairs predictable.

def skip_gram_pairs(sentence: list[str], window: int = 2):
    """For each word, pair it with every word within `window`
    positions -- the exact training signal skip-gram uses."""
    pairs = []
    for i, target in enumerate(sentence):
        start, end = max(0, i - window), min(len(sentence), i + window + 1)
        for j in range(start, end):
            if j != i:
                pairs.append((target, sentence[j]))
    return pairs

sentence = ["i", "drink", "hot", "coffee", "every", "morning"]
pairs = skip_gram_pairs(sentence, window=2)
print("skip-gram (target, context) training pairs:")
for target, context in pairs:
    print(f"  ({target!r}, {context!r})")
# A model trained on millions of sentences like this learns that
# 'coffee' and 'tea' need similar embeddings, since both tend to
# co-occur with words like 'hot', 'drink', 'morning'

# In practice, you use a pretrained model instead of training one:
#   from sentence_transformers import SentenceTransformer
#   model = SentenceTransformer('all-MiniLM-L6-v2')
#   vec = model.encode("I drink hot coffee every morning")  # one vector, whole sentence