Configuring and Running the MetaPromptOptimizer

With the evaluation dataset and metric in place (previous subtopic), the actual optimization step is what this course walks through next.

The engine doing the work is the MetaPromptOptimizer: an algorithm that uses a reasoning model to critique and iteratively refine your initial instruction prompt. 'Meta' here is literal — it's a prompt (run through a reasoning model) whose job is to look at ANOTHER prompt (your base prompt) and propose improvements to it, informed by how that base prompt actually performed against your evaluation dataset.

The setup sequence this course walks through: first, install Opik and its optimizer package, and configure Opik (this connects the library to wherever your Opik project/dashboard lives). Then import the required classes and functions from opik and opik_optimizer — specifically LevenshteinRatio (the metric from the previous subtopic), MetaPromptOptimizer (the optimization algorithm itself), and tiny_test (the example evaluation dataset).

Next, define your base prompt — this is the initial instruction that the MetaPromptOptimizer will try to enhance. This is exactly the prompt you'd otherwise be manually tweaking by hand (the problem covered in this topic's first subtopic) — the starting point, not the final answer.

Then, instantiate a MetaPromptOptimizer, specifying the model to use in the optimization process — this is the reasoning model that will do the critiquing and rewriting work (this course's example uses GPT-4o, though this course notes this can be swapped for any other LLM since Opik is fully open-source).

Finally, the optimizer.optimize_prompt(...) method is invoked with the dataset, metric configuration, and prompt to start the optimization process. Internally, it starts by evaluating the initial prompt, which sets the baseline — establishing a reference score before any changes are made — and then iterates through several different prompts (written by the reasoning model itself), evaluating each against the dataset and metric, converging toward a version that scores better than the baseline.

# Conceptual walkthrough of the book's setup sequence for the
# Opik Agent Optimizer — install/configure, define dataset + metric,
# define base prompt, instantiate optimizer, run optimize_prompt().

# pip install opik opik-optimizer

from opik_optimizer import MetaPromptOptimizer
from opik_optimizer.metrics import LevenshteinRatio

tiny_test = [
    {"input": "Summarize: The cat sat on the mat.", "expected_output": "A cat sat on a mat."},
    {"input": "Summarize: The dog ran in the park.", "expected_output": "A dog ran in a park."},
]

# The initial instruction the MetaPromptOptimizer will try to enhance —
# exactly what a human would otherwise be manually tweaking
base_prompt = "Summarize the following sentence briefly."

optimizer = MetaPromptOptimizer(
    model="gpt-4o",   # the reasoning model doing the critique + rewrite
)

result = optimizer.optimize_prompt(
    prompt=base_prompt,
    dataset=tiny_test,
    metric=LevenshteinRatio(),
)

# Internally: evaluates base_prompt first (the baseline), then
# iterates through several LLM-written candidate prompts, scoring
# each against tiny_test using LevenshteinRatio
print(result.best_prompt)
print(result.best_score)