How Embeddings Are Created: From Word2Vec to Modern Sentence Transformers
~14 min read
Word2Vec pioneered learning embeddings by predicting context words around a target word. Modern sentence transformers extend that same core intuition — 'you shall know a word by the company it keeps' — to whole sentences.
How Embeddings Are Created: From Word2Vec to Modern Sentence Transformers is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Word2Vec learns embeddings by training a network to predict a word's surrounding context words — 'you shall know a word by the company it keeps'
- •Skip-gram (Word2Vec's core training setup) creates (target, nearby-context) word pairs from real text; words appearing in similar contexts end up with similar embeddings
- •Word2Vec gives each word exactly ONE fixed embedding regardless of context ('bank' the riverbank = 'bank' the financial institution), which is a real limitation
- •Modern sentence transformers use the full surrounding sentence (via self-attention) so the same word gets different embeddings in different contexts, and embed whole sentences as one vector
- •In practice you almost always use a pretrained embedding model (Hugging Face sentence-transformers, OpenAI's API) rather than training one from scratch