What Is RAG (Retrieval-Augmented Generation)?

Ask a base LLM a question about your company's internal policy document, and it will either say it doesn't know — or worse, confidently make something up. Retrieval-Augmented Generation (RAG) is the standard fix, and it's the single most requested skill in applied GenAI job postings right now. Here's what it actually is, without the framework-specific jargon.

What is RAG?

Retrieval-Augmented Generation is a technique that gives an LLM access to information it wasn't trained on — your documents, your database, your knowledge base — by retrieving the most relevant pieces of that information and inserting them directly into the prompt before the model generates an answer.

The anchor analogy we use in the course: a base LLM is a closed-book exam. It can only answer from what it memorized during training. RAG turns it into an open-book exam — the model is handed exactly the right page of the textbook right before it has to answer, so it can quote from it directly instead of guessing from memory.

Why RAG exists

Three problems push almost every real GenAI product toward RAG:

• Knowledge cutoff — a model only knows what existed in its training data. RAG lets it answer questions about documents created yesterday.
• Private data — your company's contracts, codebase, or support tickets were never in any model's training set, and shouldn't be (you'd have to retrain the model, which is neither practical nor secure).
• Hallucination reduction — when a model answers from retrieved source text instead of memory, you can show the source, and the model is far less likely to invent facts wholesale.

This is also why RAG shows up so early in most production AI agent systems — an agent that can look things up is dramatically more reliable than one working purely from memory.

How a RAG pipeline works

Strip away the framework of your choice, and every RAG pipeline does the same four things:

1. Ingestion & chunking

Documents — PDFs, Notion pages, support tickets, whatever your knowledge base is — are broken into smaller chunks before anything else happens. A 300–800 token chunk with a small overlap (10–20%) is a reasonable starting point; embedding an entire 200-page PDF as one block loses the specificity that makes retrieval useful. The right chunk size depends on the content — dense technical text usually needs smaller chunks than narrative prose.

2. Embedding

Each chunk is converted into a vector — a list of numbers that represents its meaning — using an embedding model. These vectors, along with the original text and metadata (source, date, section), are stored in a vector database. The embedding model has to stay identical for documents and queries, or similarity search breaks entirely.

3. Retrieval

When a user asks a question, that question is embedded with the same model, and the pipeline searches the vector database for the chunks whose vectors are closest by similarity — closest in meaning, not matching keywords. Most pipelines retrieve the top 3–6 chunks (often called top-K); too few risks missing the answer, too many dilutes the LLM's context with irrelevant text.

4. Generation

The retrieved chunks are inserted into the prompt as context, and the LLM is explicitly instructed to answer only from that context. This single instruction does the most to reduce hallucination — the model quotes and paraphrases the retrieved text instead of blending it with whatever it memorized during training. Most production pipelines also show which source chunks were used, so a wrong answer is debuggable instead of a mystery.

In our RAG & Knowledge Systems module, students build exactly this stack end to end using Chroma as the vector database and one embedding model — deliberately one stack, done properly, rather than a shallow tour of five different tools.

A RAG pipeline in code

Here's the shape of it end to end, using Chroma as the vector database — the same four stages above as a minimal working script:

from chromadb.utils import embedding_functions
import chromadb

# 1. Ingestion & chunking (see the RAG Guide for a chunk_text() implementation)
chunks = chunk_text(document_text, chunk_size=500, overlap=75)

# 2. Embedding — convert to vectors, store alongside metadata
embed_fn = embedding_functions.DefaultEmbeddingFunction()
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")
collection.add(
    documents=chunks,
    embeddings=embed_fn(chunks),
    ids=[f"chunk-{i}" for i in range(len(chunks))],
)

# 3. Retrieval — embed the question, fetch the closest chunks
results = collection.query(query_texts=[user_question], n_results=4)
context = "\n\n".join(results["documents"][0])

# 4. Generation — answer grounded only in the retrieved context
prompt = f"""Answer using ONLY the context below. If the answer isn't there, say you don't know.

Context:
{context}

Question: {user_question}"""

answer = llm_client.generate(prompt)

Treat this as a reference for the shape of a pipeline, not production code — there's no chunking function shown here on purpose. The RAG Guide walks through all six steps individually, including chunking, evaluation, and the reasoning behind each choice.

Embeddings, explained simply

An embedding is a way of turning meaning into geometry. Every piece of text gets converted into a point in high-dimensional space, positioned so that texts with similar meaning end up close together — think of it as GPS coordinates for meaning. "How do I reset my password" and "I forgot my login" end up near each other in that space even though they share almost no words, which is exactly why semantic search beats simple keyword search for this use case.

RAG vs fine-tuning

Both are ways to make a model "know" more, but they solve different problems:

• RAG adds knowledge at query time, without retraining the model. It's fast to update (add a new document, it's immediately queryable), transparent (you can show sources), and cheap relative to training.
• Fine-tuning changes the model's weights to shift its behaviour or style — better for teaching a model a tone, format, or specialized skill, not for injecting frequently-changing factual knowledge.

Most production systems use RAG for knowledge and reserve fine-tuning (if they use it at all) for behaviour. If your data changes weekly, RAG is almost always the right first move.

What it takes to build one

Conceptually simple, RAG has real engineering depth once you go past a demo: chunk size and overlap affect retrieval quality significantly, metadata filtering (by date, source, document type) is often what separates a good RAG system from a mediocre one, and evaluating whether retrieval is actually working requires its own methodology — see our LLM interview questions resource for how this gets tested in practice.

If you want to build a full RAG pipeline yourself — chunking, embeddings, vector search, and generation, in one working project — the RAG Guide is our deeper, hands-on companion to this article, and Chat With Your Documents is the exact project our students ship in the Generative AI course.

Frequently asked questions