What Is RAG?

Every LLM has the same three limitations when answering factual questions:

RAG addresses all three by the same mechanism: at inference time, relevant source text is retrieved from a corpus you control and placed into the model's context window. The model shifts from "recall a fact from weights" to "read provided text and reason over it" — a task LLMs are far more reliable at.

A RAG system has two phases — indexing (offline, run once then refreshed) and querying (runtime):

INDEXING  Documents -> Chunk -> Embed -> Vector Store
QUERYING  Query -> Embed -> ANN search -> Top-k chunks -> LLM -> Answer

Indexing: ingest documents, split into chunks (L4), encode each chunk as a dense vector (L2), store vectors + metadata in a vector database (L3). Query: embed the question with the same encoder, run approximate-nearest-neighbour (ANN) search, inject the top-k chunks into the prompt, and generate.

The example uses a fictional internal policy — the kind of text absent from any public training corpus.

Source document: "P0 incidents must be acknowledged within 15 minutes. If no acknowledgement in 15 minutes, escalation to VP of Engineering triggers."

Without RAG: The model admits ignorance and returns a generic answer — "generally 15–30 minutes." Useless.

With RAG: The retrieved chunk is injected into the prompt. The LLM responds: "P0 incidents must be acknowledged within 15 minutes. After that window, escalation to the VP of Engineering triggers automatically. (Source: Incident Response Policy v3.2, §4.1)"

Specific, accurate, cited — because the model is reading the document, not guessing.

Every essential RAG component in one file. The course replaces each part with production-grade alternatives and measures quality at each step.

import os
from openai import OpenAI
import chromadb
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")
collection = chromadb.Client().create_collection("demo")

# Index
docs = [
    "P0 incidents must be acknowledged within 15 minutes.",
    "P1 incidents must be acknowledged within 60 minutes.",
]
collection.add(documents=docs,
               embeddings=encoder.encode(docs).tolist(),
               ids=["d0", "d1"])

# Retrieve
query = "What is the SLA for a P0 incident?"
results = collection.query(
    query_embeddings=encoder.encode([query]).tolist(), n_results=1)
chunk = results["documents"][0][0]

# Generate
llm = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
resp = llm.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content":
               f"Context: {chunk}\n\nQuestion: {query}\nAnswer in one sentence."}],
)
print(resp.choices[0].message.content)

export OPENAI_API_KEY="sk-..."
python rag_demo.py
# A P0 incident must be acknowledged within 15 minutes, per the context.

Three approaches compete when you need an LLM grounded in specific knowledge:

Fine-tuning is the most common misapplication. The model learns language patterns, not lookup tables — fine-tuned models still hallucinate specific policy clauses and version numbers. Fine-tune for style; use RAG for facts.

RAG sits inside a three-layer knowledge architecture:

• Application layer — chat UI, API endpoint, team-chat bot
• RAG layer — query, retrieve, augment, generate (where this course lives)
• Knowledge layer — vector store, metadata DB, source documents

Knowledge layer hygiene matters most. Stale documents and inconsistent ingestion are the leading causes of poor RAG output — no retrieval tuning compensates for a corrupt index.

Skip RAG when: the answer requires live data (use a direct API call), the question needs multi-hop reasoning over structured data (Lesson 8 covers graph + RAG), or the full corpus fits in one context window at low volume (prompt stuffing is simpler).