Every RAG pipeline needs a way to find "the most relevant chunks of text" out of potentially millions of candidates, in milliseconds. That's the specific job a vector database does — and understanding it properly changes how you design retrieval, not just which product you pick.
What is a vector database?
A vector database stores embeddings — numerical representations of meaning, typically hundreds or thousands of numbers per piece of text — and is purpose-built to answer one question extremely fast: "given this query vector, which stored vectors are closest to it?" Popular options include Chroma, Pinecone, Weaviate, Qdrant, and pgvector (a Postgres extension that adds vector search to a database you may already run).
Why a regular database won't do this
A traditional database (SQL or NoSQL) is built to retrieve exact matches or ranges efficiently — "find rows where user_id = 42." It has no native concept of "semantic closeness" between two pieces of text. You could compute similarity between a query and every row manually, but that's a full scan over your entire dataset for every single query — it doesn't scale past a small dataset.
Vector databases exist specifically to make "find the closest points in high-dimensional space" fast at scale, using specialized indexing structures instead of brute-force comparison.
How similarity search works
Once text is converted into an embedding (a point in high-dimensional space), "similar meaning" becomes a geometric question: how close are two points? The most common distance measures are:
- Cosine similarity — measures the angle between two vectors, ignoring magnitude. The most common choice for text embeddings.
- Euclidean distance — straight-line distance between two points.
- Dot product — related to cosine similarity, often used when embeddings are already normalized.
Whichever measure is used, the database's job is to return the top-K closest vectors to your query vector — these become the chunks your RAG pipeline hands to the LLM as context.
Approximate nearest neighbour indexes
Finding the exact closest vectors out of millions, for every query, is computationally expensive. In practice, almost every vector database uses an Approximate Nearest Neighbour (ANN) index — a data structure (commonly HNSW, a navigable graph structure) that trades a small amount of accuracy for a very large speed gain. This is a deliberate, well-studied trade-off: near-perfect retrieval accuracy at a fraction of the computational cost, which is why it's the industry default rather than a shortcut.
You don't need to implement HNSW yourself to use a vector database well — but knowing it exists explains why retrieval occasionally misses a result a human would consider obviously relevant. It's approximate by design.
Choosing a vector database
| Option | Good for |
|---|---|
| Chroma | Learning, prototyping, and small-to-medium production apps — simple to run locally, which is why our course teaches it first. |
| Pinecone | Managed, scalable production use without operating your own infrastructure. |
| Weaviate / Qdrant | Self-hosted or managed options with strong metadata filtering and hybrid search support. |
| pgvector | Teams already running Postgres who want vector search without adding a new system to operate. |
In the Generative AI course, we deliberately teach one stack end to end (Chroma) rather than a shallow tour of five tools — the concepts transfer directly once you understand one implementation properly. Frameworks like LangChain and LlamaIndex are covered as "alternatives that exist," so you know the ecosystem without learning it before you understand the fundamentals underneath it.
Metadata filtering matters more than people think
Pure semantic search isn't always enough. If a user asks "what changed in our refund policy this year," you don't just want semantically similar chunks — you want chunks similar in meaning and filtered to documents from this year, tagged as "policy." Most production RAG systems combine vector similarity with structured metadata filters (date, source, document type, access permissions) — this is often the difference between a retrieval system that feels magical and one that returns technically-relevant-but-practically-useless results.
Where vector databases fit in your learning path
Vector databases are one piece of a larger RAG pipeline — see what RAG is for the full picture, or the hands-on RAG Guide if you're ready to build one. In the course, this is covered hands-on in the RAG & Knowledge Systems module, immediately followed by the Chat With Your Documents project, so the concept gets used within the same week it's taught.
Keep learning: Put this into practice with the RAG Guide, or see the full course curriculum.