Contrastive Search & SLED: Newer Strategies and When to Use Them
~15 min read
Contrastive search penalizes tokens too similar to what's already been generated to fight repetition while keeping coherence. SLED goes further, using EVERY transformer layer's predictions — not just the final one — for more factually grounded generation.
Contrastive Search & SLED: Newer Strategies and When to Use Them is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Contrastive search penalizes candidates too similar to recently-generated content, balancing probability against a diversity penalty
- •This directly fights the repetition/looping problem, especially valuable for longer generations like stories
- •SLED targets factual grounding, not repetition — it uses ALL transformer layers' predictions, not just the final layer's
- •SLED nudges the final logits toward cross-layer consensus, requiring no retraining or extra data
- •Neither is a universal default like temperature+top-p — reach for contrastive search when fighting repetition, SLED when factual accuracy matters most