Setting Up the Evaluation: Dataset and the LevenshteinRatio Metric
~12 min read
Before any prompt can be automatically improved, you need an evaluation dataset of input-output pairs and a metric like LevenshteinRatio to score how close a generated output is to the target.
Setting Up the Evaluation: Dataset and the LevenshteinRatio Metric is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Automated optimization needs two ingredients first: an evaluation dataset of input-output pairs, and a metric to score outputs
- •The evaluation dataset (e.g. `tiny_test`) provides known-good target outputs for a set of inputs
- •LevenshteinRatio scores how close a generated output is to the target via normalized character-level edit distance (0 to 1)
- •The metric configuration tells the optimizer how to score outputs against labels — this is what later drives the optimization decisions
- •LevenshteinRatio suits precise, well-defined target outputs — less suited to open-ended tasks with many equally valid phrasings