HyDE: Hypothetical Document Embeddings

A subtle but important problem with naive RAG: questions are not semantically similar to their answers. 'What year did the company IPO?' and 'The company went public in 2019' don't necessarily embed close together, even though one is exactly the answer to the other — questions and their answers often use different vocabulary and phrasing. As a result, several irrelevant contexts can get retrieved during the retrieval step, purely because they happen to have higher cosine similarity to the literal question than the actual answer-containing document does.

HyDE (Hypothetical Document Embeddings) solves this with a clever reframing. Instead of embedding the question directly, HyDE works in four steps: (1) use an LLM to generate a hypothetical answer H to the query Q — this answer does not need to be entirely correct; (2) embed that hypothetical answer using a contriever-style embedding model to get an embedding E; (3) use E to query the vector database and fetch the relevant real context C; (4) pass the hypothetical answer H, the retrieved context C, and the original query Q together to the LLM to produce the final answer.

The key insight: even though the generated hypothetical answer will likely contain hallucinated details, this doesn't severely hurt retrieval performance, because of how the contriever embedding model works. That model is trained via contrastive learning and effectively acts as a near-lossless compressor whose job is to filter out the hallucinated specifics of the fake document while preserving its overall semantic shape. The resulting embedding ends up being more similar to real documents' embeddings than the raw question's embedding ever was — because a hypothetical ANSWER, even a wrong one, is written in answer-shaped language, which is exactly the vocabulary and phrasing style real documents containing the actual answer also use.

Multiple studies have shown HyDE genuinely improves retrieval performance over directly embedding the question with a traditional embedding model. The cost: increased latency (an extra LLM call before retrieval even starts) and more LLM usage overall, since you're now generating a hypothetical document on every single query before doing the actual work.

from openai import OpenAI

client = OpenAI()

def embed(text: str) -> list[float]:
    return client.embeddings.create(model="text-embedding-3-small", input=text).data[0].embedding

def generate_hypothetical_answer(query: str) -> str:
    # H does not need to be factually correct — it just needs to be
    # written in the same "shape" as a real answer would be
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": f"Write a plausible-sounding answer to: {query}"}],
    )
    return resp.choices[0].message.content

def hyde_rag(query: str, vector_store, top_k: int = 3) -> str:
    hypothetical = generate_hypothetical_answer(query)      # step 1
    hyde_embedding = embed(hypothetical)                    # step 2
    results = vector_store.similarity_search(hyde_embedding, k=top_k)  # step 3
    real_context = "\n\n".join(r.text for r in results)

    resp = client.chat.completions.create(                  # step 4
        model="gpt-4.1",
        messages=[{"role": "user", "content":
            f"Hypothetical answer: {hypothetical}\n\n"
            f"Real retrieved context:\n{real_context}\n\n"
            f"Question: {query}\n\n"
            f"Answer using the REAL context, not the hypothetical one."}],
    )
    return resp.choices[0].message.content