Building the Retrieval Step: Tensorlake, Milvus Indexing, Firecrawl and arXiv
~14 min read
The book's steps #2-#6: turning raw documents into RAG-ready chunks with Tensorlake, indexing them in Milvus, and pulling live context from Firecrawl web search and the arXiv API.
Building the Retrieval Step: Tensorlake, Milvus Indexing, Firecrawl and arXiv is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Step #2 (Tensorlake) converts raw documents into RAG-ready markdown chunks that can be embedded directly with no further processing
- •Step #3 (Milvus) indexes those chunks in a self-hosted vector DB and retrieves the top-k most similar chunks to a query — the vector-search-basics ANN concept applied concretely
- •Step #4 (Zep) is the memory layer, using temporal knowledge graphs (tracking how information changes over time) rather than a flat key-value store
- •Step #5 (Firecrawl) fetches live web content via a fast v2 endpoint, giving the assistant information newer than any model's training cutoff
- •Step #6 (arXiv API) adds a source specialized for recent academic research papers, complementing Firecrawl's broader general web coverage