Contrastive Search & SLED: Newer Strategies and When to Use Them

Beyond the four foundational strategies (greedy, multinomial, beam search) sit two newer techniques that each target a specific remaining weakness.

Contrastive search balances fluency with diversity by explicitly penalizing repetitive continuations. At each generation step, it considers a set of candidate next tokens, and for each one, checks how similar that candidate would be to what's already been generated (typically via embedding similarity to recent context). It then applies a penalty proportional to that similarity, and selects the token that best balances raw probability against this diversity penalty — high-probability-but-too-similar-to-recent-context tokens get down-weighted, favoring instead a token that's still reasonably probable but meaningfully different from what's already on the page. This directly targets the 'stuck in a loop' failure mode that greedy decoding is prone to, while keeping overall coherence high — unlike pure multinomial sampling, which can wander into genuinely incoherent territory purely by chance since it has no similarity-aware penalty at all. Contrastive search is especially effective for longer generations — stories, long-form articles — where repetition has more room to creep in over the course of the generation.

SLED (Self-Logits Evolution Decoding) targets a different problem entirely: factual grounding, not repetition. Every decoding strategy covered so far relies on the logits produced by the model's FINAL layer — that's how Transformers normally generate text. The issue: factual signals that were present in earlier layers can fade as information passes deeper through the network, leading the final layer to sometimes favor fluent-sounding but occasionally inaccurate output. SLED changes this by looking at how logits evolve across ALL layers, not just the last one — each layer contributes its own prediction, SLED measures how closely these layer-wise predictions agree with each other, and then nudges the final logits toward this cross-layer consensus before a token actually gets selected. This requires no retraining, no extra training data, and no significant additional compute — by leveraging the model's full internal knowledge rather than only its very last processing step, SLED tends to produce more grounded, more factual generations from the exact same underlying model weights.

Neither of these is a universal default the way temperature+top-p sampling is — contrastive search is worth reaching for specifically when repetition is the observed problem in long-form generation, and SLED is worth reaching for specifically when factual accuracy matters more than raw fluency and you have access to intermediate layer outputs (which not every inference setup exposes).

import numpy as np

def cosine_sim(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))

def contrastive_search_step(
    probs: np.ndarray, candidate_embeddings: np.ndarray,
    generated_embedding: np.ndarray, alpha: float = 0.6, top_k: int = 5,
) -> int:
    """Balance raw probability against similarity to what's already generated —
    lower alpha favors probability, higher alpha favors avoiding repetition."""
    top_k_idx = np.argsort(probs)[-top_k:]
    best_idx, best_score = None, -np.inf
    for idx in top_k_idx:
        similarity_penalty = cosine_sim(candidate_embeddings[idx], generated_embedding)
        score = (1 - alpha) * probs[idx] - alpha * similarity_penalty
        if score > best_score:
            best_idx, best_score = idx, score
    return int(best_idx)

def sled_adjust_logits(layer_logits: list[np.ndarray], final_logits: np.ndarray, strength: float = 0.3) -> np.ndarray:
    """Nudge the final layer's logits toward the cross-layer consensus,
    instead of trusting only the final layer's raw prediction."""
    consensus = np.mean(layer_logits, axis=0)  # what ALL layers collectively predict
    return (1 - strength) * final_logits + strength * consensus