Embedding Model Fine-Tuning
Building and fine-tuning custom embedding models for domain-specific retrieval: contrastive learning with triplet loss, the SBERT architecture, MultipleNegativesRankingLoss for efficient training, hard negative mining, and Matryoshka Representation Learning for multi-size embeddings.
Embedding Inputs:
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Explain why general-purpose embedding models underperform on domain-specific retrieval tasks
- •Implement contrastive learning with triplet loss (anchor, positive, negative) to train an embedding model
- •Use MultipleNegativesRankingLoss for efficient fine-tuning without explicit negative mining
- •Apply hard negative mining to select the most informative training negatives
- •Describe Matryoshka Representation Learning and when dimension truncation is useful
What is it?
Embedding model fine-tuning adapts a pre-trained embedding model to produce better-quality embeddings for a specific domain or task. The goal: reduce the retrieval gap between 'semantically similar in general English' and 'semantically similar for YOUR use case.'
Pre-trained models like text-embedding-3-small are trained on general web text. For a medical records retrieval system, 'hypertension' and 'high blood pressure' should be very close in embedding space — but a general model trained on news articles might not represent this equivalence well. Fine-tuning adjusts the model's embedding space so that domain-relevant synonyms, technical jargon, and query-document relationships are correctly captured.
Jay Alammar (Hands-On LLMs, Ch.10/11) introduces Sentence BERT (SBERT) as the architecture foundation: a siamese network that produces fixed-length sentence embeddings optimized for comparison via cosine similarity, trained with contrastive learning on (anchor, positive, negative) triplets.
Why it exists
General-purpose embedding models optimize for a broad objective: encode the semantic meaning of text in a way that works across thousands of topics and tasks. This breadth comes at the cost of depth: they don't know that 'myocardial infarction' and 'heart attack' are synonyms in clinical text, that 'API endpoint' and 'REST route' are equivalent in engineering documentation, or that in legal text, 'discovery' refers to a pre-trial process, not finding something.
When your RAG system uses a general embedding model for domain-specific retrieval, Recall@K suffers: relevant documents are retrieved less often because the model doesn't understand the domain's vocabulary and synonym structure. Fine-tuning the embedding model on domain-specific (query, relevant_document) pairs directly improves Recall@K, which improves RAG output quality end-to-end.
Empirically, domain-adapted embedding models typically improve Recall@10 by 10-30 percentage points on specialized retrieval tasks compared to general models of the same parameter size.
Problem it solves
- My RAG system misses relevant documents in retrieval — the embedding model doesn't understand domain vocabulary.
- A query for 'MI treatment' doesn't retrieve documents about 'myocardial infarction care' because the general model doesn't know they're related.
- I want to reduce embedding dimensionality to save memory and speed up vector search without losing too much quality.
- I have query-document pairs but no explicit negatives — how do I train without expensive negative mining?
- My domain has technical jargon that general embedding models represent poorly — how do I adapt an existing model without training from scratch?
Intuition
Embedding model fine-tuning is like calibrating a translation dictionary for a specialized profession.
A general English dictionary defines 'discovery' as 'the act of finding something.' A legal professional's dictionary defines 'discovery' as 'the pre-trial phase in which parties exchange evidence.' A medical professional's dictionary defines 'culture' as 'growing microorganisms in a controlled medium,' not as 'social customs and arts.'
Fine-tuning an embedding model is like updating that dictionary for your domain: you show the model thousands of examples where 'MI' and 'myocardial infarction' appear as synonyms in context, and the model learns to place them close together in its internal space — not because we explicitly told it they're synonyms, but because they consistently appear as positive pairs in our training data.
The contrastive learning framework is the mechanism: pull similar pairs together in embedding space, push dissimilar pairs apart. After enough training steps, the model's space reflects YOUR domain's notion of similarity.
Analogy
Contrastive learning is like teaching a dog to sort objects by category through a reward-and-correction system.
You show the dog a red ball (anchor), a red cube (positive — same color, different shape), and a blue ball (negative — same shape, different color). You reward the dog for grouping the anchor with the positive, and correct it when it groups with the negative. After thousands of trials, the dog has learned 'same color goes together' as the relevant criterion.
In contrastive learning: anchor = query, positive = relevant document, negative = irrelevant document. The loss function rewards the model for making (query, relevant_doc) vectors close and (query, irrelevant_doc) vectors far apart. After training, the model's embedding space encodes YOUR notion of relevance (which may be very domain-specific) rather than general-purpose semantic similarity.
Hard negatives are like giving the dog a red ball and a red cube as positive and a slightly different red ball as negative — the hardest case to distinguish, which forces the most learning.
Technical explanation
SENTENCE BERT (SBERT) — Jay Alammar Ch.10: BERT's original architecture was designed for pairwise classification (classify whether sentence A and B are related). For retrieval, this requires encoding every (query, document) pair together — O(n²) inference for n documents. SBERT (Reimers & Gurevych, 2019) instead trains a siamese network: two BERT encoders sharing weights encode query and document independently to fixed-length vectors. Retrieval becomes: pre-compute doc embeddings offline, embed query at inference, compute cosine similarity — O(n) not O(n²).
CONTRASTIVE LEARNING LOSSES:
-
Triplet Loss: L = max(0, ||f(a) - f(p)||² - ||f(a) - f(n)||² + margin)
- anchor a, positive p, negative n
- Margin (0.5 typical): positive must be AT LEAST margin closer than negative
- Problem: requires explicit negative mining
-
MultipleNegativesRankingLoss (MNRL, Henderson et al., 2017):
- Input: only (anchor, positive) pairs — no negatives needed
- In-batch negatives: in a batch of N pairs, the positives of OTHER pairs serve as negatives for each anchor
- Loss: cross-entropy over softmax(similarity(anchor_i, all_docs_in_batch))
- Efficient: larger batch = more negatives = better training signal
- Default choice for fine-tuning with (query, relevant_doc) pairs
-
CoSENT / AnglE Loss: more recent alternatives with better gradient properties
HARD NEGATIVE MINING: Random negatives are easy — the model quickly learns to distinguish 'heart attack' from 'stock market crash'. Hard negatives are semantically close but wrong: 'myocardial infarction' vs. 'myocardial contusion' — both are cardiac terms but the document about contusions is NOT relevant to a query about infarctions. Methods:
- BM25 top-K that aren't positive: fast, retrieves lexically similar non-matches
- Cross-encoder reranking: embed, retrieve top-100, run a cross-encoder to identify false positives (high similarity but not relevant) as hard negatives
- ANN retrieval: find the current model's nearest neighbors for each query that aren't in the positive set
MATRYOSHKA REPRESENTATION LEARNING (MRL, Kusupati et al., 2022): Standard embedding models produce fixed-dimension vectors. MRL trains the model so the FIRST d dimensions are maximally informative at every scale d. Training: compute the contrastive loss at multiple dimensions (64, 128, 256, 512, 1024, 1536) and sum them. Result: you can truncate the embedding to any target dimension and it remains useful — 256-dim MRL embedding often matches quality of 1536-dim standard. Practical use: store 256-dim for ANN search (fast, small), keep 1536-dim for reranking. OpenAI text-embedding-3-* models use MRL.
DATA REQUIREMENTS:
- Minimum: 1K-10K (query, relevant_document) pairs
- Sweet spot: 10K-100K pairs
- Can generate with an LLM: for each doc, prompt LLM to generate 3 queries that could be answered by this document (synthetic query generation)
WHEN TO FINE-TUNE vs. USE GENERAL MODEL:
- Domain has specialized vocabulary not in general web text (medical, legal, code)
- Your Recall@K is < 0.7 on a held-out retrieval eval set
- You have (or can generate) > 1K query-document pairs
- ROI check: cost of fine-tuning + serving vs. improved retrieval quality
Architecture
End-to-End Embedding Fine-Tuning Pipeline:
┌─────────────────────────────────────────────────────────────────┐ │ DATA PREPARATION │ │ │ │ Source: (query, relevant_doc) pairs from: │ │ • User click-through logs (user clicked → implicitly relevant)│ │ • Expert annotation (human labels relevant pairs) │ │ • Synthetic generation (LLM generates queries for each doc) │ │ │ │ Optional: add hard negatives via BM25 or ANN retrieval │ │ Split: 80% train / 10% dev / 10% test │ └──────────────────────────────┬──────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FINE-TUNING (sentence-transformers library) │ │ │ │ Base model: all-MiniLM-L6-v2 (or text-embedding-3-small │ │ via distillation for API models) │ │ │ │ Loss: MultipleNegativesRankingLoss (MNRL) │ │ Batch size: 32-256 (larger = more in-batch negatives) │ │ LR: 2e-5, warmup 10%, AdamW │ │ Epochs: 1-5 (stop when dev Recall@K plateaus) │ └──────────────────────────────┬──────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ EVALUATION (held-out retrieval benchmark) │ │ │ │ Metric: Recall@K (K=5, 10), NDCG@10, MRR │ │ Compare: fine-tuned vs. base model on SAME eval set │ │ Expect: 10-30% Recall@10 improvement on domain-specific set │ └──────────────────────────────┬──────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ DEPLOYMENT │ │ │ │ Re-embed entire document corpus with fine-tuned model │ │ Update vector database (FAISS/Weaviate/Pinecone) │ │ A/B test: fine-tuned retrieval vs. original on live traffic │ └─────────────────────────────────────────────────────────────────┘
Workflow
-
BASELINE RETRIEVAL EVAL:
- Build a retrieval benchmark: 100-500 (query, relevant_docs) pairs
- Measure Recall@10 with your current general embedding model
- If Recall@10 > 0.85, fine-tuning may not be worth the effort
- If Recall@10 < 0.7, fine-tuning is likely high ROI
-
PREPARE TRAINING DATA: a. Collect (query, relevant_doc) pairs from click logs or annotation b. If no logs: generate synthetic queries with LLM per document c. (Optional) mine hard negatives: BM25 top-100 minus known positives d. Minimum: 1K pairs; target 10K+
-
FINE-TUNE:
- Use sentence-transformers library
- Loss: MultipleNegativesRankingLoss for (query, doc) pairs
- Or TripletLoss if you have explicit negatives
- Monitor dev Recall@K per epoch; stop when it plateaus
-
EVALUATE:
- Compare Recall@K, NDCG@10, MRR on held-out benchmark
- Must improve over baseline; if not, check data quality
-
DEPLOY:
- Re-embed all documents with fine-tuned model
- Update vector index
- A/B test on production traffic (live traffic split)
Example
# Fine-tuning an embedding model with sentence-transformers # pip install sentence-transformers datasets from sentence_transformers import SentenceTransformer, InputExample, losses from sentence_transformers.evaluation import InformationRetrievalEvaluator from torch.utils.data import DataLoader # 1. Load base model model = SentenceTransformer('all-MiniLM-L6-v2') # 2. Prepare training pairs (query, relevant_document) # In production, these come from click logs or human annotation. # Here: synthetic examples for illustration. train_examples = [ InputExample(texts=[ 'What is the treatment for heart attack?', 'Myocardial infarction is treated with aspirin, thrombolytics, and PCI.' ]), InputExample(texts=[ 'How do statins work?', 'HMG-CoA reductase inhibitors reduce LDL cholesterol by blocking liver synthesis.' ]), # ... many more pairs ] # 3. DataLoader — larger batch = more in-batch negatives train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=64) # 4. MultipleNegativesRankingLoss: positives are the pairs; # in-batch non-matching docs become the negatives automatically train_loss = losses.MultipleNegativesRankingLoss(model) # 5. Evaluator: Recall@K on held-out retrieval set # queries: {id: query_text}, corpus: {id: doc_text}, relevant_docs: {query_id: {doc_ids}} evaluator = InformationRetrievalEvaluator( queries={'q1': 'heart attack symptoms'}, corpus={'d1': 'myocardial infarction presents with chest pain...'}, relevant_docs={'q1': {'d1'}}, ) # 6. Fine-tune model.fit( train_objectives=[(train_dataloader, train_loss)], epochs=3, warmup_steps=100, evaluator=evaluator, evaluation_steps=500, output_path='./medical-embedding-model', save_best_model=True, ) # 7. Use fine-tuned model for retrieval fine_tuned = SentenceTransformer('./medical-embedding-model') query_emb = fine_tuned.encode('MI treatment options') # Now 'MI' embeds close to 'myocardial infarction' documents # Synthetic query generation for training data from anthropic import Anthropic client = Anthropic() def generate_queries(document: str, n: int = 3) -> list[str]: '''Generate n queries that this document could answer (Self-Instruct for retrieval).''' resp = client.messages.create( model='claude-haiku-4-5-20251001', max_tokens=200, messages=[{ 'role': 'user', 'content': ( f'Generate {n} diverse search queries that this document could answer.\n' f'Output only the queries, one per line.\n\n' f'Document: {document[:500]}' ) }] ) return [q.strip() for q in resp.content[0].text.strip().split('\n') if q.strip()]
Real-world usage
-
Cohere Embed v3 (2023): fine-tunable embedding model sold as an API. The model card documents 15-25% improvement in domain-specific retrieval after fine-tuning on customer's (query, document) pairs vs. the general model.
-
Hugging Face MTEB Leaderboard: the Massive Text Embedding Benchmark evaluates embedding models on 56 datasets across 8 tasks. Models fine-tuned on domain-specific data consistently outperform general models on domain tasks but may degrade on out-of-domain benchmarks (catastrophic forgetting).
-
Jay Alammar (Hands-On LLMs, Ch.11): 'When RAG retrieval quality is below your bar, the first thing to try before scaling up the retrieval infrastructure is fine-tuning the embedding model. It's often cheaper and more impactful than switching from FAISS to a managed vector database.'
-
OpenAI text-embedding-3-small/large: both use Matryoshka Representation Learning, enabling users to truncate embedding dimensions (e.g., 1536→256) for storage/speed savings with minimal quality loss.
-
Sentence Transformers library (Reimers & Gurevych, 2019 → ongoing): the de-facto standard library for embedding model fine-tuning, used in production by thousands of companies for domain adaptation.
Trade-offs
General model vs. fine-tuned: general models are available immediately, require no training infrastructure, and generalize across topics. Fine-tuned models give better domain performance but need training data, GPU time, and maintenance when the domain evolves. Decision: if Recall@K > 0.8 with a general model, don't fine-tune. If < 0.7, fine-tuning is likely high ROI.
Full fine-tuning vs. adapter-based: full fine-tune updates all model weights and gets maximum performance but risks forgetting general language understanding. Adapter-based (LoRA on encoder) is faster and preserves more of the base model but has a lower performance ceiling. For embedding models, full fine-tune is typically used (the model is smaller than LLMs, e.g., 22M-340M params).
MNRL vs. triplet loss: MNRL requires only positive pairs (much more common data) and is more efficient per pair. Triplet loss with hard negatives can be more precise but requires expensive negative mining. Start with MNRL; add hard negatives in a second-stage fine-tune if MNRL quality is insufficient.
Embedding dimension: large dimensions (1536) have higher quality ceiling but cost more to store and search. MRL-trained models let you choose the right tradeoff at serving time — a 256-dim MRL embedding is fast and small; the full 1536-dim MRL embedding is used only for reranking the top-K candidates.
Visual explanation
SBERT Siamese Network (Training):
Anchor text Positive text
'heart attack' 'myocardial infarction'
│ │
[BERT encoder] [BERT encoder] ← same weights
(shared params) (shared params)
│ │
[Pool] [Pool]
│ │
emb_anchor emb_positive
└─────────┬─────────────┘
│
cosine_similarity → 0.93
│
[Contrastive Loss] ← wants similarity → 1.0
pulls anchor+positive closer in embedding space
Triplet Loss (with explicit negative):
anchor ──────────────────► embedding_a positive ─────────────────► embedding_p negative ─────────────────► embedding_n
Loss = max(0, d(a,p) - d(a,n) + margin) Goal: d(a,p) < d(a,n) - margin (positive must be 'margin' closer than negative)
MultipleNegativesRankingLoss (efficient, no explicit negatives):
Batch of (query_i, relevant_doc_i) pairs: [(q1, d1), (q2, d2), (q3, d3)]
Similarity matrix: d1 d2 d3 q1 [0.91 0.42 0.38] ← q1 should be closest to d1 q2 [0.35 0.88 0.41] ← q2 should be closest to d2 q3 [0.39 0.37 0.85] ← q3 should be closest to d3
Cross-entropy loss on each row: d_j (j≠i) are in-batch negatives! Efficient: N pairs → N positives + (N-1) negatives each, no mining needed.
Matryoshka Representation Learning:
Full embedding: [d1, d2, d3, ..., d1536] Half truncated: [d1, d2, d3, ..., d768] Quarter truncated: [d1, d2, d3, ..., d384]
MRL trains the FIRST dimensions to be maximally informative, so truncated embeddings remain useful for retrieval.
Advantages
- —
Domain-adapted embedding models improve Recall@K by 10-30% on specialized retrieval tasks — directly improves RAG output quality
- —
MultipleNegativesRankingLoss requires only (query, document) pairs — no explicit negative mining needed, dramatically lowering the data preparation cost
- —
Synthetic query generation (LLM writes queries for each document) enables fine-tuning with no human annotation, at the cost of LLM API calls
- —
MRL embeddings (OpenAI text-embedding-3-*) can be truncated to any target dimension — 256-dim MRL often matches 1536-dim standard quality at 6x lower storage cost
- —
Fine-tuned models are smaller than proprietary API models and can be self-hosted, reducing per-query embedding cost at scale
Disadvantages
- —
Fine-tuned models can overfit to training domain and degrade on out-of-domain retrieval (catastrophic forgetting) — always evaluate on a broad benchmark alongside the domain benchmark
- —
Requires GPU for training — sentence-transformers fine-tuning with 10K pairs takes 1-4 hours on a single A100
- —
Synthetic queries generated by LLMs have biases and blind spots — they tend to be well-formed and specific, missing the short, typo-prone, ambiguous queries real users send
- —
Hard negative mining requires an initial retrieval pass, which means you need a baseline embedding model before you can mine good negatives for training
- —
Re-embedding the entire document corpus after fine-tuning is expensive for large corpora (millions of documents) and requires downtime or dual-index serving during migration
Common mistakes
- —
Not running a baseline retrieval eval before fine-tuning. If your current embedding model achieves Recall@10 = 0.88, fine-tuning adds complexity for marginal gain. Always measure first.
- —
Training without validation. Running fine-tuning for a fixed number of epochs without monitoring Recall@K on a dev set leads to overfitting. The best dev-set checkpoint may be at epoch 2 of 5. Use InformationRetrievalEvaluator with evaluation_steps to checkpoint the best model.
- —
Using only synthetic queries for training. LLM-generated queries are well-formed and specific. Real user queries are short, ambiguous, and typo-prone. A model trained only on synthetic queries can degrade on real user traffic. Mix synthetic and real queries; or augment synthetic queries with EDA (typos, truncations).
- —
Forgetting to re-embed the document corpus. Fine-tuning the embedding model changes the representation space. If you update the query encoder but keep old document embeddings in your vector database, query and document live in different spaces — retrieval quality collapses. Always re-embed all documents after fine-tuning before deploying.
- —
Not evaluating catastrophic forgetting. Fine-tuning on domain data can reduce performance on general retrieval tasks. Evaluate the fine-tuned model on a general benchmark (MTEB BEIR tasks) alongside your domain benchmark. If general performance drops > 5%, consider adding general-domain data to the training mix.
🎤 Interview questions
Explain MultipleNegativesRankingLoss. How does it create negatives from a dataset of only (query, document) pairs, and why does batch size matter?
What is Matryoshka Representation Learning? Why do OpenAI's text-embedding-3 models use it, and when would you use a truncated dimension at serving time?
📂 Subtopics
Contrastive Learning and SBERT Architecture
How SBERT enables efficient sentence comparison with shared-weight siamese encoders, and the contrastive losses used to train it.
~45 min
Domain Adaptation and Hard Negative Mining
Fine-tuning strategies for domain-specific retrieval: data generation, hard negative mining, and Matryoshka embeddings.
~40 min