Building the Retrieval Step: Tensorlake, Milvus Indexing, Firecrawl and arXiv

The previous subtopic gave the workflow's shape; this subtopic covers how this course actually implements the RETRIEVAL side — gathering raw context from all 4 sources before any filtering happens — following this course's own numbered steps #2 through #6.

Step #2, Prepare data for RAG, uses Tensorlake to convert the document into RAG-ready markdown chunks for each section. The extracted data can be directly embedded and stored in a vector DB without further processing. This matters because raw documents (PDFs, complex layouts, tables) aren't naturally in a form an embedding model handles well — Tensorlake's job is specifically bridging that gap, producing clean markdown chunks ready for the next step, connecting directly to the rag-chunking-strategies topic elsewhere in this curriculum.

Step #3, Indexing and retrieval, takes those RAG-ready chunks (along with their metadata) and stores them in a self-hosted Milvus vector database. We retrieve the top-k most similar chunks to our query — the exact ANN-based similarity search covered in the vector-search-basics prerequisite topic, applied here concretely with a named, specific vector database rather than in the abstract.

Step #4, Build memory layer, uses Zep as the core memory layer of the workflow. It creates temporal knowledge graphs to organize and retrieve context for each interaction. We use it to store and retrieve context from chat history and user data. The word 'temporal' is worth noting specifically — rather than just storing flat facts, a temporal knowledge graph tracks HOW information changes over time (a user's preference last month versus their preference today), which is a meaningfully richer approach to Memory (Context Type 4 from the companion topic) than a simple key-value store.

Step #5, Firecrawl web search, fetches the latest news and developments related to the user query. Firecrawl's v2 endpoint provides 10x faster scraping, semantic crawling, and image search, turning any website into LLM-ready data — this is the source that gives the research assistant access to information newer than its training cutoff, directly addressing the same staleness problem that motivates RAG generally.

Step #6, arXiv API search, further supports research queries by using the arXiv API to retrieve relevant results from their data repository based on the user query — giving the assistant access to recent academic papers specifically, complementing Firecrawl's broader web coverage with a source specialized for research literature.

# Implementing each retrieval-side step (#2-#6) as a distinct function,
# mirroring the book's own step-by-step structure.

def tensorlake_prepare_docs(raw_document: str) -> list[dict]:
    """Step #2: convert a raw document into RAG-ready markdown chunks,
    ready to embed directly with no further processing."""
    sections = raw_document.split("\n\n")
    return [{"chunk": s.strip(), "metadata": {"section": i}}
            for i, s in enumerate(sections) if s.strip()]

def milvus_index_and_retrieve(chunks: list[dict], query: str, top_k: int = 2) -> list[dict]:
    """Step #3: store chunks in a vector DB, retrieve top-k most similar
    (a toy word-overlap stand-in for real embedding + ANN search)."""
    def overlap_score(chunk_text, query):
        return len(set(chunk_text.lower().split()) & set(query.lower().split()))
    ranked = sorted(chunks, key=lambda c: overlap_score(c["chunk"], query), reverse=True)
    return ranked[:top_k]

def zep_memory_layer(user_id: str, memory_store: dict, query: str) -> str:
    """Step #4: temporal knowledge graph -- stores/retrieves chat history
    and user data per interaction."""
    history = memory_store.get(user_id, [])
    memory_store.setdefault(user_id, []).append(query)   # record this interaction
    return f"prior interactions for {user_id}: {history}"

def firecrawl_web_search(query: str) -> str:
    """Step #5: fetch latest news/developments -- addresses training staleness."""
    return f"[live web result for {query!r}]"

def arxiv_api_search(query: str) -> str:
    """Step #6: retrieve relevant results from arXiv's data repository."""
    return f"[arXiv paper matching {query!r}]"

doc = "Transformers use self-attention.\n\nAttention computes weighted sums of values."
chunks = tensorlake_prepare_docs(doc)
top_chunks = milvus_index_and_retrieve(chunks, "attention computation")
print("Retrieved doc chunks:", top_chunks)
print(zep_memory_layer("user_1", {}, "how does attention work?"))
print(firecrawl_web_search("latest transformer research"))
print(arxiv_api_search("attention mechanism"))