The Complete Working Example: Kicking Off the Workflow and the Streamlit App

The previous three subtopics covered the pipeline's pieces — 4 retrieval sources, then filtering. This final subtopic covers how this course actually RUNS the complete assembled workflow, and how it's packaged for a real user.

Step #8, Kick off the workflow, per this course: finally, we kick off our context engineering workflow with a query. Based on the query, we notice that the RAG tool, powered by Tensorlake, was the most relevant source for the LLM to generate a response. This observation is worth pausing on: even though the workflow always queries all 4 sources (documents, memory, web, arXiv) on every run, the FILTERING step from the previous subtopic means the actual response can end up grounded predominantly in whichever source turned out most relevant for THAT specific query — for a question well-covered by the indexed documents, the RAG/Tensorlake source dominates; a different, more current-events-flavored query might instead lean on Firecrawl's web results. The pipeline doesn't hard-code which source matters — relevance is decided dynamically, per query, by the filtering agent.

This course also translates this entire workflow into a Streamlit app — giving it an actual user-facing interface rather than leaving it as backend code only. The app specifically provides citations with links and metadata, and provides insights into relevant sources. This is a meaningful design choice beyond just 'making it pretty': surfacing WHICH sources contributed to an answer (and linking back to them) directly addresses a core weakness of RAG systems generally — a user who can see 'this answer drew on these 2 specific document chunks and this 1 arXiv paper' can verify the answer's grounding themselves, rather than trusting the LLM's output as an unverifiable black box.

This course's final, important closing caveat, worth carrying forward from this whole topic: the workflow explained above is one of the many blueprints. Your implementation can vary. This isn't a throwaway disclaimer — it's the practical takeaway of the entire hands-on build. The SPECIFIC tools (Tensorlake, Zep, Firecrawl, Milvus, CrewAI) and the SPECIFIC 8 steps are one working, concrete instantiation of the 6-types/4-stages conceptual framework from the companion topic — genuinely useful as a template to adapt, but not a one-size-fits-all architecture. A different use case might need different sources (a legal research assistant might swap arXiv for a case-law database), a different memory system, or a different orchestration framework than CrewAI — the PATTERN (multi-source retrieval, agent-based filtering, cited generation, persisted memory) is what's meant to transfer, not the exact tool list.

# The complete, assembled workflow (steps #1-#8) as one runnable
# pipeline, plus a citation-tracking layer standing in for the
# book's Streamlit app (which surfaces sources + links to the user).

def kick_off_workflow(query: str, memory_store: dict, user_id: str = "user_1") -> dict:
    """Step #8: run the full pipeline end to end for a real query."""
    sources = {
        "documents (Tensorlake+Milvus)": [f"[doc match for: {query}]"],
        "memory (Zep)": [f"[past context for: {query}]"] if user_id in memory_store else [],
        "web (Firecrawl)": [f"[web result for: {query}]"],
        "arxiv (arXiv API)": [f"[arxiv paper for: {query}]"],
    }

    # Filtering agent (previous subtopic) decides what's relevant PER SOURCE --
    # notice the pipeline doesn't hard-code which source "wins"
    relevance_scores = {"documents (Tensorlake+Milvus)": 0.9, "memory (Zep)": 0.1,
                        "web (Firecrawl)": 0.3, "arxiv (arXiv API)": 0.2}
    kept_sources = {name: chunks for name, chunks in sources.items()
                    if relevance_scores[name] > 0.25 and chunks}

    dominant_source = max(kept_sources, key=lambda s: relevance_scores[s], default=None)
    response_text = f"Answer to {query!r}, primarily grounded in: {dominant_source}"

    memory_store[user_id] = memory_store.get(user_id, []) + [response_text]  # save to memory

    return {
        "response": response_text,
        "citations": list(kept_sources.keys()),   # what a Streamlit UI would render as links
        "dominant_source": dominant_source,
    }

memory = {}
result = kick_off_workflow("how does self-attention work in transformers?", memory)
print("Response: ", result["response"])
print("Citations:", result["citations"])
print("Dominant source (query-dependent, not hard-coded):", result["dominant_source"])