What Are Vector Databases & Why RAG Needs Them

A vector database stores unstructured data (text, images, audio, video, etc.) in the form of vector embeddings. Each data point — a word, a document, an image, or any other entity — is transformed into a numerical vector using ML techniques, and the embedding model is trained so that these vectors capture the essential features and characteristics of the underlying data. Considering word embeddings specifically, you'd discover that in the embedding space, the embeddings of fruits cluster close to each other, cities form another cluster, and so on — this shows embeddings can learn the semantic characteristics of the entities they represent. Once stored in a vector database, similar objects can be retrieved for a given query, enabling similarity search, clustering, and classification over unstructured data — operations that are difficult with traditional databases. When an e-commerce site recommends similar items or searches for a product based on a query, you're usually interacting with a vector database behind the scenes.

So where do vector databases fit into RAG specifically? An LLM is deployed after learning from a static version of the corpus it was trained on — if a model was trained on data up to some cutoff date, it has no idea what happened afterward. Repeatedly retraining a new model every day on fresh data is impractical and cost-ineffective (LLMs can take weeks to train). And even beyond staleness, if you open-source an LLM and someone wants to use it on their own privately-held dataset, the model will have no idea about that data either, since it wasn't shown during training. But is it really the objective to train an LLM to know absolutely everything in the world? Not at all — training is really about teaching the model the overall structure of language: how to understand and generate it well. So the real goal becomes: can we let LLMs look up new information they weren't trained on, and use it in text generation, without retraining the model at all?

One option would be to just stuff that information directly into the prompt. But LLMs have a limited context window (the number of tokens they can accept), and additional information can easily exceed that limit. Vector databases solve this problem: since they store information as vectors capturing semantic meaning, you can maintain a much larger pool of available information encoded into vectors via an embedding model, and only pull out the specific, relevant pieces needed for a given query — via an approximate similarity search between the query's vector and the stored vectors — rather than trying to fit everything into the prompt at once. The retrieved content (gathered from the payload stored alongside each vector at indexing time) is then combined with the user's actual prompt and given to the LLM, letting it easily incorporate this new information while generating a response, because the relevant details are now sitting right there in the prompt.