Query Decomposition: Breaking Complex Questions into Sub-Queries

A single retrieval pass works well when a query maps cleanly onto one piece of information. But many real questions are actually several questions bundled into one: 'How does our Q3 revenue growth compare to our top competitor's, and what drove the difference?' isn't answerable by one similarity search — it genuinely needs (at least) our Q3 revenue, the competitor's Q3 revenue, and some explanation of drivers behind the gap.

Query decomposition is exactly the technique for this: instead of embedding and retrieving for the complex question as a single unit, an agent first breaks it into smaller, independently-answerable sub-queries — in the example above, something like 'What was our Q3 revenue growth?', 'What was [competitor]'s Q3 revenue growth?', and 'What factors contributed to differences in revenue growth between the two?' Each sub-query gets its own retrieval pass (potentially against different sources), producing focused, relevant context for that specific piece of the original question rather than one noisy retrieval trying to cover everything at once.

Once each sub-query has been answered (or has its supporting context retrieved), a final synthesis step combines the individual results back into one coherent answer to the ORIGINAL complex question — this synthesis step matters as much as the decomposition itself, since simply concatenating three separate answers isn't the same as actually answering the combined question the user asked.

This complements the source-selection and relevance-checking agents already part of the core agentic RAG workflow: decomposition typically happens early, right after (or as part of) the query-rewriting stage, before source selection and retrieval even begin for each individual sub-query. It's especially valuable for analytical, comparative, or multi-hop questions — anything where the answer genuinely depends on combining multiple distinct pieces of information — and adds little value (just extra LLM calls) for simple, single-fact questions that a plain agentic RAG pass already handles well.

import json
from openai import OpenAI

client = OpenAI()

def decompose_query(query: str) -> list[str]:
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content":
            f"Break this question into independently-answerable sub-questions. "
            f"Return a JSON list of strings. If it's already simple, return a "
            f"single-item list.\n\nQuestion: {query}"}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content).get("sub_questions", [query])

def answer_sub_query(sub_query: str, vector_store) -> str:
    results = vector_store.similarity_search(sub_query, k=3)
    context = "\n\n".join(r.text for r in results)
    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {sub_query}"}
    ])
    return resp.choices[0].message.content

def decomposed_rag(query: str, vector_store) -> str:
    sub_queries = decompose_query(query)
    sub_answers = [answer_sub_query(sq, vector_store) for sq in sub_queries]

    synthesis_prompt = (
        f"Original question: {query}\n\n"
        + "\n\n".join(f"Sub-question: {sq}\nAnswer: {a}" for sq, a in zip(sub_queries, sub_answers))
        + "\n\nSynthesize a single coherent answer to the original question."
    )
    return client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content": synthesis_prompt}
    ]).choices[0].message.content