What Is RAG (Retrieval-Augmented Generation)?

Ask a base LLM a question about your company's internal policy document, and it will either say it doesn't know — or worse, confidently make something up. Retrieval-Augmented Generation (RAG) is the standard fix, and it's the single most requested skill in applied GenAI job postings right now. Here's what it actually is, without the framework-specific jargon.

What is RAG?

Retrieval-Augmented Generation is a technique that gives an LLM access to information it wasn't trained on — your documents, your database, your knowledge base — by retrieving the most relevant pieces of that information and inserting them directly into the prompt before the model generates an answer.

The anchor analogy we use in the course: a base LLM is a closed-book exam. It can only answer from what it memorized during training. RAG turns it into an open-book exam — the model is handed exactly the right page of the textbook right before it has to answer, so it can quote from it directly instead of guessing from memory.

Why RAG exists

Three problems push almost every real GenAI product toward RAG:

• Knowledge cutoff — a model only knows what existed in its training data. RAG lets it answer questions about documents created yesterday.
• Private data — your company's contracts, codebase, or support tickets were never in any model's training set, and shouldn't be (you'd have to retrain the model, which is neither practical nor secure).
• Hallucination reduction — when a model answers from retrieved source text instead of memory, you can show the source, and the model is far less likely to invent facts wholesale.

This is also why RAG shows up so early in most production AI agent systems — an agent that can look things up is dramatically more reliable than one working purely from memory.

How a RAG pipeline works

Strip away the framework of your choice, and every RAG system does the same four things:

1. Ingestion & chunking — documents (PDFs, docs, web pages) are split into smaller chunks, because embedding and retrieving a 200-page PDF as one block doesn't work well.
2. Embedding — each chunk is converted into a vector (a list of numbers) that represents its meaning. See vector databases explained for the next layer of this.
3. Retrieval — when a user asks a question, the question is embedded too, and the system finds the chunks whose vectors are closest in meaning (not just keyword-matched).
4. Generation — the retrieved chunks are inserted into the prompt as context, and the LLM generates an answer grounded in that context, usually with a citation back to the source.

In our RAG & Knowledge Systems module, students build exactly this stack end to end using Chroma as the vector database and one embedding model — deliberately one stack, done properly, rather than a shallow tour of five different tools.

Embeddings, explained simply

An embedding is a way of turning meaning into geometry. Every piece of text gets converted into a point in high-dimensional space, positioned so that texts with similar meaning end up close together — think of it as GPS coordinates for meaning. "How do I reset my password" and "I forgot my login" end up near each other in that space even though they share almost no words, which is exactly why semantic search beats simple keyword search for this use case.

RAG vs fine-tuning

Both are ways to make a model "know" more, but they solve different problems:

• RAG adds knowledge at query time, without retraining the model. It's fast to update (add a new document, it's immediately queryable), transparent (you can show sources), and cheap relative to training.
• Fine-tuning changes the model's weights to shift its behaviour or style — better for teaching a model a tone, format, or specialized skill, not for injecting frequently-changing factual knowledge.

Most production systems use RAG for knowledge and reserve fine-tuning (if they use it at all) for behaviour. If your data changes weekly, RAG is almost always the right first move.

What it takes to build one

Conceptually simple, RAG has real engineering depth once you go past a demo: chunk size and overlap affect retrieval quality significantly, metadata filtering (by date, source, document type) is often what separates a good RAG system from a mediocre one, and evaluating whether retrieval is actually working requires its own methodology — see our LLM interview questions resource for how this gets tested in practice.

If you want to build a full RAG pipeline yourself — chunking, embeddings, vector search, and generation, in one working project — the RAG Guide is our deeper, hands-on companion to this article, and Chat With Your Documents is the exact project our students ship in the Generative AI course.