Setting Up the Evaluation: Dataset and the LevenshteinRatio Metric

Automated prompt optimization can't work without something concrete to optimize AGAINST — this is exactly why the Opik workflow starts with the evaluation setup, not the optimizer itself.

The first ingredient is the evaluation dataset: a basic test dataset with input-output pairs (this course refers to a minimal example dataset as tiny_test). Each pair represents a case where you know what a GOOD output looks like for a given input — this is what grounds the whole optimization process in something measurable rather than a human's subjective sense of 'looks good.'

The second ingredient is the metric: LevenshteinRatio, the metric used to evaluate the prompt's effectiveness in generating a precise output for the given input. Levenshtein distance (also known as edit distance) measures how many single-character edits — insertions, deletions, substitutions — are needed to turn one string into another; the ratio form normalizes this into a similarity score, typically between 0 (completely different) and 1 (identical). This gives the optimizer a concrete number to maximize: how close is the LLM's actual output, character-by-character, to the expected target output for each dataset example?

With both pieces in hand, the workflow proceeds by defining the evaluation dataset, then configuring the evaluation metric, which tells the optimizer how to score the LLM's outputs against the given label. This scoring configuration is what the next subtopic's MetaPromptOptimizer actually consumes: for every candidate prompt it tries, it runs the prompt against the dataset, scores each result with LevenshteinRatio, and uses those scores to judge whether the candidate prompt is an improvement over the previous one.

It's worth noting that LevenshteinRatio is a strict, character-level metric — well-suited to tasks with a precise expected output format, less suited to open-ended generation tasks where many different phrasings could all be equally 'correct.' This course's example specifically uses a task with a well-defined target output, which is exactly the kind of task this metric fits best.

# Conceptual setup — an evaluation dataset and a Levenshtein-ratio-
# style scoring function, the two ingredients the optimizer needs.

def levenshtein_ratio(output: str, target: str) -> float:
    """Simplified Levenshtein-ratio-style similarity score between
    0 (completely different) and 1 (identical) — the same idea
    Opik's LevenshteinRatio metric provides."""
    if not output and not target:
        return 1.0
    # dynamic-programming edit distance
    m, n = len(output), len(target)
    dp = [[0] * (n + 1) for _ in range(m + 1)]
    for i in range(m + 1):
        dp[i][0] = i
    for j in range(n + 1):
        dp[0][j] = j
    for i in range(1, m + 1):
        for j in range(1, n + 1):
            cost = 0 if output[i - 1] == target[j - 1] else 1
            dp[i][j] = min(dp[i - 1][j] + 1, dp[i][j - 1] + 1, dp[i - 1][j - 1] + cost)
    edit_distance = dp[m][n]
    return 1 - edit_distance / max(m, n, 1)

# The evaluation dataset — input/output pairs the optimizer scores against
tiny_test = [
    {"input": "Summarize: The cat sat on the mat.", "expected_output": "A cat sat on a mat."},
    {"input": "Summarize: The dog ran in the park.", "expected_output": "A dog ran in a park."},
]

for example in tiny_test:
    simulated_output = example["expected_output"]  # stand-in for an actual LLM call
    score = levenshtein_ratio(simulated_output, example["expected_output"])
    print(f"Score: {score:.2f} for input: {example['input']!r}")