Domain Adaptation and Hard Negative Mining

Domain adaptation fine-tunes a general embedding model to understand your specific domain's vocabulary and relevance relationships.

Step 1: Decide whether to fine-tune Build a retrieval benchmark (100-500 (query, relevant_doc) pairs from your domain).

• Recall@10 ≥ 0.85: general model is good enough, don't fine-tune
• Recall@10 < 0.70: fine-tuning is high ROI
• Between: consider fine-tuning if you have easy access to training pairs

Step 2: Training data generation

Option A — Click logs (best quality): User searched 'MI treatment' → clicked document about 'myocardial infarction care' → implicit (query, relevant_doc) pair

Option B — Expert annotation: Annotators label which documents are relevant to each query Cost: $15-100/hour; quality: highest

Option C — Synthetic query generation (no annotation needed): For each document, prompt LLM: 'Generate 3 queries this document answers' Cost: ~$0.001/document; quality: good but biased toward well-formed queries

def generate_queries(doc: str) -> list[str]:
    resp = client.messages.create(
        model='claude-haiku-4-5-20251001',
        max_tokens=200,
        messages=[{'role': 'user', 'content':
            f'Write 3 diverse search queries that this document answers. '
            f'One per line, no numbering.\n\nDocument: {doc[:400]}'}]
    )
    return resp.content[0].text.strip().split('\n')

Step 3: Hard negative mining Random negatives are too easy — any random document is clearly not relevant. Hard negatives force the model to learn fine distinctions:

• BM25 hard negatives: retrieve top-20 by BM25 lexical match → exclude known positives → use rest as negatives (lexically similar but not relevant)
• ANN hard negatives: embed, retrieve nearest neighbors not in positive set
• Cross-encoder filtering: retrieve top-100 → run a cross-encoder → use high-scoring non-positives as hard negatives

Matryoshka Representation Learning (MRL) Standard training: loss computed only on full-dimension embeddings. MRL training: loss computed at multiple dimensions simultaneously: L_total = L(dim=64) + L(dim=128) + L(dim=256) + L(dim=512) + L(dim=1536) Result: first 64 dimensions are most informative, next 64 add more detail, etc. Use case: ANN search on 256-dim (fast, small), reranking on 1536-dim (precise). Models: OpenAI text-embedding-3-*, E5-mistral-7b-instruct (MRL-trained)

Catastrophic forgetting mitigation: Mix general-domain pairs (~20% of batch) with domain-specific pairs during fine-tuning. Evaluate on MTEB BEIR tasks alongside domain benchmark to monitor regression.