Automated Agent/Prompt Optimization with Opik
Using the open-source Opik Agent Optimizer to automatically improve a prompt via an LLM-critique-and-refine loop, instead of manually iterating by hand.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain why manual prompt iteration doesn't scale across models and tasks
- •Set up an Opik evaluation dataset and a scoring metric (e.g., LevenshteinRatio)
- •Run MetaPromptOptimizer to automatically improve a base prompt against a dataset
- •Interpret optimization results via result.display() and the Opik dashboard
What is it?
Opik Agent Optimizer is an open-source toolkit that automates prompt optimization — instead of a developer manually tweaking a prompt, running it, eyeballing the output, and repeating (a process that doesn't scale and whose results can degrade when you switch models), Opik lets an LLM itself iteratively critique and refine your prompt against a real evaluation dataset, automatically converging on a stronger version of your original instruction.
Why it exists
Developers manually iterating through prompts to find an optimal one is slow, subjective, and fragile — a prompt hand-tuned against one model's quirks can silently degrade when the underlying model is upgraded or swapped, and there's no systematic record of what was tried or why one version won. Opik Agent Optimizer exists to replace this ad hoc process with a measurable, repeatable one: define what 'good' means with an evaluation dataset and metric, then let an optimization algorithm (MetaPromptOptimizer) search the space of prompt variations automatically, tracking results so you can see exactly why the winning prompt won.
Problem it solves
It solves the 'prompt engineering doesn't scale' problem — for any single task, you might reasonably try 5-10 prompt variations by hand, but doing this systematically across dozens of tasks, or re-validating existing prompts every time you consider a model upgrade, is infeasible manually. It also solves the 'no ground truth for why a prompt is better' problem — ad hoc manual tuning is subjective ('this looks better to me'), while Opik's dataset+metric approach gives you an objective, reproducible score comparing any two prompt candidates.
Intuition
Manual prompt tuning is like editing an essay by reading it over and over, tweaking a sentence, and just trusting your gut that it's better now. Opik's automated optimization is like having a professional editor who scores every draft against a specific, agreed-upon rubric, tries dozens of alternative phrasings systematically, and hands you back the version that objectively scored highest against that rubric — with the full history of what was tried and why.
Analogy
It's like the difference between a chef randomly adjusting a recipe based on taste alone (manual prompt tuning) versus A/B testing dozens of recipe variations against a panel of blind taste-testers who score each one on a consistent scale, then keeping whichever version scored best (Opik's dataset+metric-driven optimization) — the second approach is slower to set up but gives you an objective, defensible answer instead of a hunch.
Technical explanation
The workflow uses several specific Opik components. LevenshteinRatio is an example metric that evaluates a prompt's effectiveness by measuring string-similarity between the LLM's actual output and a given expected label — useful when you need outputs to closely match a target string, though other metrics can be swapped in for different tasks. MetaPromptOptimizer is the optimization algorithm: it uses a reasoning model to critique and iteratively refine your initial instruction prompt, generating new candidate prompts, evaluating each against your dataset, and converging toward the highest-scoring version. A basic dataset of input-output pairs defines your evaluation set — this is what 'good' is measured against.
The process is: configure the evaluation metric (telling the optimizer how to score outputs against labels), define your base prompt (the initial instruction the optimizer will try to improve), instantiate MetaPromptOptimizer (specifying which model drives the optimization/reasoning process), then call optimizer.optimize_prompt(...) with the dataset, metric configuration, and base prompt. The optimizer first evaluates the initial prompt to establish a baseline score, then iterates through several AI-generated prompt variations, evaluating each, and ultimately returns the most optimal prompt found. result.display() shows a summary including the best prompt and its score, and the same results are available in the Opik dashboard for deeper visual analysis.
Architecture
An Opik optimization run has four components: (1) an Evaluation Dataset — input/output example pairs defining ground truth; (2) a Metric — a scoring function (e.g., LevenshteinRatio) comparing model output against the expected label; (3) a Base Prompt — the starting instruction to be improved; (4) the MetaPromptOptimizer — driven by a reasoning model, which generates, evaluates, and iterates on prompt candidates. Results (baseline score, all candidates tried, and the final best prompt) are logged and viewable both programmatically (result.display()) and visually in the Opik dashboard.
Workflow
- Install opik and the opik_optimizer package, and configure your Opik account/local instance.
- Import LevenshteinRatio (or an appropriate metric for your task), MetaPromptOptimizer, and your dataset utilities.
- Define your evaluation dataset — a set of representative input/output pairs covering the range of cases your prompt needs to handle well.
- Configure the evaluation metric, telling the optimizer how to score the LLM's outputs against your dataset's labels.
- Define your base prompt — the initial instruction you want improved.
- Instantiate a MetaPromptOptimizer, specifying which model should drive the critique-and-refine process.
- Call optimizer.optimize_prompt(dataset=..., metric=..., prompt=...) to start the optimization run.
- Review results via result.display() (best prompt + score summary) and the Opik dashboard (full visualization of the optimization trace).
Example
import opik from opik.evaluation.metrics import LevenshteinRatio from opik_optimizer import MetaPromptOptimizer from opik_optimizer.datasets import tiny_test
opik.configure() # connect to Opik Cloud or a local instance
1. Evaluation dataset — input/output pairs defining ground truth
dataset = tiny_test()
2. Metric — how to score a candidate prompt's outputs against the labels
metric = LevenshteinRatio()
3. Base prompt — the starting instruction to be improved
base_prompt = 'Answer the question based on the provided context.'
4. Optimizer — a reasoning model that critiques and refines the prompt
optimizer = MetaPromptOptimizer(model='gpt-4o')
5. Run the optimization
result = optimizer.optimize_prompt( dataset=dataset, metric=metric, prompt=base_prompt, )
result.display() # prints baseline score, best prompt found, and its score
Full trace of every candidate tried is also viewable in the Opik dashboard
Note: this runs 100% locally if you swap gpt-4o for any local model —
Opik itself is fully open-source.
Real-world usage
Teams running the same core prompt across multiple LLM providers (e.g., supporting both GPT-4o and a local Llama model) use automated optimizers like this to re-validate and re-tune prompts per model without manual re-engineering each time — a prompt hand-tuned for GPT-4o's quirks often underperforms on a different model until re-optimized. LLM application teams running continuous evaluation pipelines integrate prompt optimization as a step that runs whenever the evaluation dataset grows or the underlying model changes, keeping prompts continuously tuned rather than 'set once and forget.' Because Opik is fully open-source and can run 100% locally with any LLM (not just GPT-4o), teams with strict data-residency or cost constraints can run the entire optimization loop without sending data to a third-party optimization service.
Trade-offs
Automated optimization trades upfront setup cost (building a representative evaluation dataset and picking a meaningful metric) for a systematic, reproducible, and scalable improvement process compared to manual tuning. The quality of the optimized prompt is only as good as the evaluation dataset and metric you provide — a small or unrepresentative dataset, or a metric that doesn't actually capture what 'good' means for your task (e.g., using string-similarity for a task where semantic correctness matters more than exact phrasing), can lead the optimizer toward a prompt that scores well on your metric but doesn't actually perform better in practice. The optimization run itself costs additional LLM calls (both the candidate-generating reasoning model and the evaluation calls), so it has a real, measurable cost compared to free-form manual tuning.
Visual explanation
A loop diagram: [Base Prompt] + [Evaluation Dataset (input/output pairs)] + [Metric (e.g., LevenshteinRatio)] → feed into [MetaPromptOptimizer] → the optimizer evaluates the base prompt first to set a [Baseline Score] → then generates and evaluates [Candidate Prompt 1], [Candidate Prompt 2], ... [Candidate Prompt N] (each scored against the same dataset+metric) → the highest-scoring candidate becomes the [Optimized Prompt] → results (including the full trace of candidates tried) are visualized in the [Opik Dashboard].
Advantages
- —
Replaces subjective, ad hoc manual prompt tuning with a systematic, measurable, reproducible process
- —
Automatically re-validates and re-tunes prompts when switching or upgrading the underlying LLM, instead of manual re-engineering each time
- —
Fully open-source and runs 100% locally with any LLM, avoiding vendor lock-in or third-party data-sharing requirements
- —
Produces a full trace of candidates tried and their scores, making the optimization process auditable rather than a black box
Disadvantages
- —
Requires upfront investment in building a representative evaluation dataset and choosing a meaningful metric before any optimization can begin
- —
Optimization quality is only as good as the dataset/metric — a flawed metric (e.g., pure string similarity for a semantic task) can steer toward the wrong 'improvement'
- —
Adds real LLM call costs during the optimization run itself, on top of whatever the final optimized prompt costs to run
- —
Doesn't replace human judgment entirely — results should still be spot-checked, especially for subjective quality dimensions the metric may not capture
Common mistakes
- —
Using a metric that doesn't actually reflect task quality — e.g., optimizing for exact string match (LevenshteinRatio) on a task where semantically correct but differently-phrased answers should count as equally good
- —
Building too small or unrepresentative an evaluation dataset, causing the optimizer to overfit to a narrow set of cases that don't generalize
- —
Treating the optimizer's output as final without spot-checking it against cases outside the evaluation dataset
- —
Re-running optimization from scratch every time instead of tracking and comparing against previous optimization runs in the dashboard
- —
Forgetting to re-run optimization after a significant model upgrade or provider switch, silently running a stale, sub-optimal prompt
📂 Subtopics
Why Manual Prompt Iteration Doesn't Scale
The problem Opik Agent Optimizer solves: manually tweaking a prompt, running it, eyeballing the output, and repeating doesn't scale — and results can degrade when you switch models.
~10 min
Setting Up the Evaluation: Dataset and the LevenshteinRatio Metric
Before any prompt can be automatically improved, you need an evaluation dataset of input-output pairs and a metric like LevenshteinRatio to score how close a generated output is to the target.
~12 min
Configuring and Running the MetaPromptOptimizer
MetaPromptOptimizer uses a reasoning model to critique and iteratively refine your base prompt — install, configure, define a base prompt, and call optimize_prompt() to start.
~15 min
Reading the Results: Baseline vs. Optimized, and the Opik Dashboard
After optimization completes, result.display() shows the best prompt found and its score against the baseline — with full results also available in the Opik dashboard for deeper analysis.
~10 min