Build a Multi-Source Context Engineering Workflow [Hands-On]
A full hands-on build of a multi-agent research assistant gathering context from documents, memory, web search, and arXiv — using Tensorlake, Zep, Firecrawl, Milvus, and CrewAI, deployed as a Streamlit app with citations.
Semantic paragraph chunking
PDF textbooks are parsed and split into chunks with 100-character overlaps to keep semantic continuity.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Design a context-engineering pipeline that aggregates context from 4 distinct sources (documents, memory, web, academic papers)
- •Use Tensorlake to convert complex documents into RAG-ready chunks
- •Use Zep's temporal knowledge graphs as a memory layer for chat history and user data
- •Build a context-filtering agent and a synthesizer agent using CrewAI, and deploy the result with citations
What is it?
This is a complete, hands-on build of a multi-agent research assistant that puts the '6 types of context' and the 'Write/Read/Compress/Isolate' framework into practice with a real tech stack. The agent gathers context from 4 sources — Documents, Memory, Web search, and arXiv — then uses one agent to filter that aggregated context down to what's relevant, and a second agent to synthesize a final response, all deployed as a Streamlit app with citations and source insights.
Why it exists
The abstract 6-type/4-stage context engineering framework needs a concrete reference implementation to be genuinely learnable — this workflow exists to show exactly what 'gathering the right context from multiple sources, filtering it, and synthesizing a response' looks like in real, runnable code using genuinely useful open-source and commercial tools, rather than staying purely conceptual. It's explicitly presented as one of many possible blueprints — the specific tools and pipeline shape will vary by use case, but the pattern (aggregate → filter → synthesize → persist) generalizes broadly.
Problem it solves
It solves the 'how do I actually wire this up' problem that pure conceptual frameworks leave open — specifically for multi-source research and knowledge-work agents that need to combine private documents, prior conversation memory, live web information, and academic literature into one coherent answer. Naive single-source RAG (just embed some docs and retrieve) cannot handle this — a query needing document context, remembered user preferences, current news, and recent papers simultaneously requires genuine multi-source orchestration, which is exactly what this workflow demonstrates end to end.
Intuition
Think of this workflow as building a research assistant with four different information-gathering 'senses': reading your private document library (Documents via Tensorlake+Milvus), remembering what you've discussed before (Memory via Zep), checking the current news (Web via Firecrawl), and consulting the academic literature (arXiv API) — then having one internal reviewer (the filtering agent) throw out anything irrelevant before a final writer (the synthesizer agent) drafts the actual answer with proper citations.
Analogy
It's like assembling a research team where one member specializes in your internal filing cabinet (documents), one remembers every past conversation with the client (memory), one reads the day's news (web search), and one keeps up with academic journals (arXiv) — they each report back to an editor (filter agent) who discards anything not relevant to the specific question, and then a writer (synthesizer agent) turns the vetted material into a coherent, cited answer.
Technical explanation
The workflow has 8 concrete steps. (1) Crew flow — a top-down CrewAI orchestration defining the overall pipeline shape (this specific blueprint is one of many valid ones; the pattern varies by use case). (2) Prepare data for RAG — Tensorlake converts source documents into RAG-ready markdown chunks per section, producing extracted data that can be directly embedded and stored without further processing. (3) Indexing and retrieval — the RAG-ready chunks plus metadata are stored in a self-hosted Milvus vector database, from which the top-k most similar chunks to a query are retrieved. (4) Build memory layer — Zep acts as the workflow's core memory layer, creating temporal knowledge graphs to organize and retrieve context for each interaction, used to store and retrieve context from both chat history and user data. (5) Firecrawl web search — fetches the latest news and developments related to the user query; Firecrawl's v2 endpoint provides fast scraping, semantic crawling, and image search, turning any website into LLM-ready data. (6) ArXiv API search — retrieves relevant academic results from the arXiv data repository based on the user query, supporting research-oriented questions specifically. (7) Filter context — the combined context from all 4 sources is passed to a context-evaluation agent that filters out irrelevant material before it reaches generation. (8) Kick off the workflow — the filtered context is passed to a synthesizer agent that generates the final response; depending on the query, different sources end up mattering more (e.g., for a document-heavy query, the Tensorlake-powered RAG tool is typically the most relevant source). The final implementation is wrapped in a Streamlit app that provides citations with links and metadata, plus insights into which sources were actually relevant.
Architecture
Four parallel context-gathering components (Tensorlake+Milvus for documents, Zep for memory, Firecrawl for web, arXiv API for academic papers) feed into a Context Filtering Agent, which feeds a Synthesizer Agent, which produces the final response — this response is both displayed (via Streamlit, with citations) and written back to Zep's memory layer for future interactions. CrewAI orchestrates the agent coordination across the filter and synthesize steps.
Workflow
- Set up the 4 context-source integrations: Tensorlake for document processing, Milvus for vector storage/retrieval, Zep for the memory layer, and Firecrawl + the arXiv API for external/live sources.
- Process your source documents through Tensorlake to get RAG-ready markdown chunks, then index them in Milvus.
- Configure Zep to store and retrieve chat history and user-specific data as a temporal knowledge graph.
- Wire up Firecrawl web search and arXiv API search as additional context-gathering tools available to the workflow.
- Build a CrewAI crew with (at minimum) a context-filtering agent that receives all 4 sources' aggregated output and strips irrelevant material, and a synthesizer agent that generates the final cited response from the filtered context.
- Kick off the workflow with a real multi-faceted query and inspect which source(s) actually contributed to the final answer.
- Wrap the pipeline in a Streamlit (or similar) UI that displays the response alongside citations with links/metadata and source-relevance insights.
- Write the final response back into the Zep memory layer so future interactions can build on this one.
Example
from crewai import Agent, Task, Crew
── 4 parallel context sources, each wrapped as a callable ──
def get_document_context(query: str) -> list[str]: # Tensorlake-processed chunks, retrieved from Milvus return milvus_client.search(embed(query), top_k=5)
def get_memory_context(query: str, user_id: str) -> list[str]: return zep_client.memory.search(user_id, query) # temporal knowledge graph
def get_web_context(query: str) -> list[str]: return firecrawl_client.search(query) # latest news/developments
def get_arxiv_context(query: str) -> list[str]: return arxiv_client.search(query) # academic papers
── Filter agent: strips irrelevant material from the aggregated context ──
filter_agent = Agent( role='Context Evaluator', goal='Filter aggregated context down to only what is relevant to the query', )
── Synthesizer agent: generates the final cited response ──
synthesizer_agent = Agent( role='Research Synthesizer', goal='Generate a cited answer from filtered context', )
def run_workflow(query: str, user_id: str) -> dict: aggregated = ( get_document_context(query) + get_memory_context(query, user_id) + get_web_context(query) + get_arxiv_context(query) ) crew = Crew( agents=[filter_agent, synthesizer_agent], tasks=[ Task(description=f'Filter this context for relevance: {aggregated}', agent=filter_agent), Task(description='Synthesize a cited response from the filtered context', agent=synthesizer_agent), ], ) result = crew.kickoff() zep_client.memory.add(user_id, query, result) # write back for future turns return result
Real-world usage
This exact pattern — multi-source context aggregation, filter, synthesize, cite — mirrors how production research-assistant products (Perplexity, Elicit, Consensus) are architected internally, combining live web search with structured academic/document retrieval and a citation-generating synthesis step. Enterprise knowledge-work copilots (internal research tools at consulting firms, legal research assistants) commonly need exactly this 4-source pattern (internal documents, conversation memory, current news, and domain literature) since no single source alone answers real research questions. Companies building 'chat with your data' products that also need current external information (not just static internal docs) adopt this exact filter-then-synthesize two-agent pattern to avoid either ignoring relevant internal context or drowning the response in irrelevant external noise.
Trade-offs
This multi-source, multi-agent pipeline is significantly more infrastructure than a single-source RAG setup — 4 separate integrations (Tensorlake, Milvus, Zep, Firecrawl/arXiv) plus a 2-agent CrewAI orchestration layer, versus one vector store and one generation call. It's worth this complexity specifically when queries genuinely need multiple, heterogeneous context sources simultaneously — for single-source use cases, this full pipeline is significant over-engineering. The filter-then-synthesize two-agent split adds latency (2 sequential LLM calls minimum, on top of 4 parallel retrieval calls) compared to a single-shot RAG call, in exchange for meaningfully better relevance and reduced noise in the final answer.
Visual explanation
A pipeline diagram: [User Query] → fans out to 4 parallel context sources: [Tensorlake-processed Documents in Milvus (top-k retrieval)], [Zep Memory (chat history + user data via temporal knowledge graph)], [Firecrawl Web Search (latest news/developments)], [arXiv API (academic papers)] → all 4 outputs converge into → [Context Filtering Agent] (strips irrelevant context) → filtered context passed to → [Synthesizer Agent] (generates the final response with citations) → output rendered in a [Streamlit App] showing the response, citations with links/metadata, and relevant-source insights → final response also written back to → [Zep Memory] for future turns.
Advantages
- —
Provides a complete, concrete, runnable reference implementation of the abstract 6-type/4-stage context engineering framework
- —
Combines 4 genuinely different context source types (structured documents, conversational memory, live web, academic literature) that no single-source RAG setup could handle
- —
The filter-then-synthesize agent split measurably improves relevance by removing noise before generation, rather than dumping all 4 sources' raw output into one prompt
- —
Ships with citations and source-relevance insights out of the box, addressing a common trust/verifiability gap in LLM-generated research answers
Disadvantages
- —
Requires setting up and maintaining 4+ separate service integrations (Tensorlake, Milvus, Zep, Firecrawl, arXiv API) plus a CrewAI orchestration layer
- —
Adds real latency: 4 parallel retrieval calls plus 2 sequential agent calls (filter, then synthesize) versus a single-shot RAG response
- —
The specific tool choices (Tensorlake, Zep, Milvus, Firecrawl) are opinionated — teams may need to substitute different tools for their existing infrastructure
- —
Over-engineered for single-source use cases where a simple RAG pipeline would suffice
Common mistakes
- —
Building this full 4-source pipeline for a task that only ever needs one source (e.g., static internal documents), adding unnecessary integration and orchestration overhead
- —
Skipping the context-filtering agent step and passing all 4 sources' raw output directly to the synthesizer, reintroducing the noise problem this pattern exists to solve
- —
Not writing the final response back to the memory layer (Zep), losing the continuity benefit that makes the memory source useful in future turns
- —
Treating the specific tool stack (Tensorlake/Zep/Milvus/Firecrawl/CrewAI) as mandatory rather than as one of many valid blueprints — the pattern matters more than the exact tools
- —
Omitting citations/source-attribution in the final UI, undermining user trust in a research-assistant product where verifiability matters most
📂 Subtopics
Workflow Overview: A Multi-Agent Research Assistant Over 4 Context Sources
The book's hands-on build: a multi-agent research assistant gathering context from Documents, Memory, Web search, and arXiv, filtered and synthesized through a CrewAI pipeline.
~12 min
Building the Retrieval Step: Tensorlake, Milvus Indexing, Firecrawl and arXiv
The book's steps #2-#6: turning raw documents into RAG-ready chunks with Tensorlake, indexing them in Milvus, and pulling live context from Firecrawl web search and the arXiv API.
~14 min
Compression and Selection: Filtering Aggregated Context Before Generation
The book's step #7, filter context, uses a dedicated context-evaluation agent to trim the combined output of 4 retrieval sources before it ever reaches the response-generating agent.
~12 min
The Complete Working Example: Kicking Off the Workflow and the Streamlit App
The book's step #8 ties every prior piece together into a single running query, wrapped in a Streamlit app that surfaces citations and source insights to the end user.
~13 min