Contrastive Learning and SBERT Architecture

SBERT (Sentence-BERT, Reimers & Gurevych 2019) solved a fundamental problem with BERT for retrieval.

The BERT retrieval problem: Original BERT classifies if two texts are related by feeding BOTH texts through the encoder together. For a corpus of 10,000 documents:

• Naive: embed every (query, doc) pair at query time → 10,000 BERT calls → too slow
• SBERT solution: encode query and docs INDEPENDENTLY with shared encoder weights → pre-compute doc embeddings offline, one query embed at inference → fast

SBERT architecture:

• Siamese network: two encoder copies with IDENTICAL weights
• Each encoder takes one sentence → produces a token sequence
• Pooling: mean of token embeddings (or [CLS] token) → fixed-size vector
• Training: minimize contrastive loss on the two pooled vectors

Key contrastive losses:

Cosine Similarity Loss (for similarity regression):

• Input: (text_a, text_b, label) where label ∈ [0, 1]
• Loss: MSE between cosine_sim(emb_a, emb_b) and label
• Requires labeled similarity scores (harder to collect)

Triplet Loss (with explicit negatives):

• Input: (anchor, positive, negative)
• Loss: max(0, ||f(a)-f(p)||₂ - ||f(a)-f(n)||₂ + margin)
• Margin (default 0.5): positive must be at least this much closer than negative
• Requires labeled triplets — harder to get than pairs

MultipleNegativesRankingLoss (most practical):

• Input: only (anchor, positive) pairs — NO explicit negatives
• In a batch of size N: each anchor's positive is correct; all OTHER batch positives are treated as negatives
• Loss: cross-entropy over similarity(anchor, all_batch_docs)
• Scales with batch: batch=64 → 63 negatives per anchor
• Works directly from click logs or annotated (query, relevant_doc) pairs

Why MNRL is the default choice: Collecting (query, relevant_doc) pairs is easy — click logs, annotation tools. Collecting explicit negatives (which document is NOT relevant to this query?) is harder and less reliable. MNRL turns any pair dataset into a contrastive training set automatically.