A large language model is a text predictor. Your application sends it a list of messages, and it returns the words that best fit those messages. It has no memory of your last request, no login to your database, and no idea who your customer is.
That is the whole problem. A support assistant built on a raw model can write well about returns policies in general, but it has never read yours. Ask it whether a laptop can go back after 20 days and it will answer anyway, fluently, from whatever it absorbed during training.
Retrieval-augmented generation, shortened to RAG, is the standard fix. This guide explains what RAG is, why it exists, and how every piece works, assuming you have never used the term. One example runs throughout: a store with 4,000 help-centre articles and one returns policy.
TL;DR
- RAG means searching your own documents first, then putting what you found into the prompt so the model answers from it.
- Chunking splits documents into pieces of 500 to 1,000 tokens, because a prompt has no room for a 40-page policy.
- Embeddings turn each chunk into a list of numbers that encodes meaning, so related text is found without shared keywords.
- Vector search returns the chunks nearest the question.
pgvectorin Postgres is enough for most applications. - The prompt carries those chunks and the question, with an instruction to answer only from the chunks.
- Hybrid search and reranking fix most retrieval failures, and retrieval failures are what make RAG answers wrong.
- The index is a cache of your documents, so it has to be updated whenever a document changes.
What Is RAG (Retrieval-Augmented Generation)?
Retrieval-augmented generation is a pattern with three steps, and its name lists all three. Your code retrieves text related to the user's question from a source you control, augments the prompt by pasting that text into it, and the model generates an answer from what it was handed.
The closest everyday comparison is an open-book exam. A model without RAG is a student answering from memory. A model with RAG is the same student with the right page already open. The student has not become smarter, and the book has not become fluent.
The gap in that diagram is the reason RAG exists. The question arrives, the model answers from training data, and the correct policy sits in your database with no path into the conversation. The model cannot reach out to anything, so your code has to do the reaching.
Trust is the second reason. Because the answer is built from passages you chose, the interface can show the source beside it, and a doubtful reader can open the original in one click.
Why RAG Instead of Fine-Tuning or a Bigger Prompt
Two alternatives come up every time, and both lose for the same reason.
Fine-tuning continues the model's training on your own examples. It is the right tool for teaching a style, a format or a vocabulary, and the wrong tool for facts. Your returns policy changes on a Tuesday afternoon, and retraining a model to absorb one new number is slow, costly, and impossible to audit afterwards.
A bigger prompt means skipping the search and sending everything. Models accept very long inputs now, so why not paste all 4,000 articles into each request? Because you pay per token on every call, latency grows with input length, and accuracy drops when the one useful line is buried in a very long prompt.
RAG wins because facts that change belong in a database rather than in model weights. Editing a document and re-indexing takes seconds. Many systems use both: RAG for the facts, a little fine-tuning for the house tone.
How a RAG Pipeline Works
A RAG system has two halves that run at different times, and confusing them is the most common beginner mistake.
Indexing runs ahead of time. Documents are split into chunks, each chunk goes to an embedding model, and the resulting vectors are stored next to the chunk text and its metadata. For 4,000 articles this is a batch job of a few minutes, and afterwards it only repeats for documents that change.
Querying runs on every request and has to be fast. The question is embedded with the same model, the nearest chunks come back, the prompt is assembled, and the model writes the answer. A typical split is 30 ms to embed, 20 ms to search, and one to three seconds for the model.
Only two steps in that picture leave your stack. The rest is application code.
Chunking Documents for Retrieval
Nothing in the pipeline searches a whole document. Search runs over pieces of documents, called chunks, because a prompt has room for a few thousand words of context and not for a 40-page policy.
A chunk of 500 to 1,000 tokens, roughly 400 to 750 English words, suits prose. Split on structure first, so a heading stays with the text beneath it and a table is not cut in half. Then overlap each chunk with the next by 10 to 15 percent, repeating a sentence or two across the boundary.
The overlap exists because a boundary eventually lands in a bad place. If one chunk ends with "a laptop may be returned within" and the next starts with "30 days of delivery", neither piece answers the question alone, and nothing later in the pipeline puts them back together.
Keep the heading path inside the chunk text. A chunk reading "within 30 days" is useless by itself, while the same chunk prefixed with "Returns > Laptops and Tablets" is specific, and the extra words give the embedding model more to work with.
Embeddings and Vector Search Explained
Text cannot be compared to a question directly, because computers compare numbers and not meaning. An embedding model solves that. Hand it a piece of text and it returns a vector: a fixed-length list of decimal numbers, commonly 1,536 of them, that encodes what the text means.
Treat each vector as a coordinate in a space with 1,536 axes rather than two. Text about return windows lands in one region, text about delivery times in another. Two passages about returns sit close together even when they share no words, which is where a keyword search gives up.
Vector search is then geometry. The question is embedded with the same model, and the store returns the chunks whose vectors sit nearest to it, usually by cosine distance, which compares the direction of two vectors rather than their length.
One rule here has no exceptions. The question must be embedded with the model that embedded the chunks, because two models produce two unrelated coordinate systems, and comparing across them returns confident nonsense.
Storing the Vectors: pgvector or a Vector Database
The vectors need a home that searches them quickly. A vector database is any store built for that job, such as Pinecone, Qdrant or Weaviate.
Most full-stack applications do not need one. The pgvector extension adds a vector column and distance operators to the Postgres database you already run, and the query is ordinary SQL:
select chunk_text, document_id
from chunks
where tenant_id = $1
order by embedding <=> $2
limit 5;
<=> is the cosine distance operator, so rows arrive ordered from nearest meaning to furthest, and limit 5 keeps the best five. The tenant_id filter is not an optimisation. It is the line that stops one customer's documents reaching another customer's prompt, and it belongs inside the query, not in code that filters the rows afterwards.
One database also means one backup and one migration path. Move to a dedicated store when scale forces it.
Building the RAG Prompt
What comes back is text, and from here nothing is specific to AI. Your code builds a string:
const context = chunks.map((c, i) => `[${i + 1}] ${c.chunk_text}`).join('\n\n')
const messages = [
{
role: 'system',
content:
'Answer only from the passages below. If they do not contain the answer, say you do not know. Cite the passage number you used.',
},
{ role: 'user', content: `Passages:\n${context}\n\nQuestion: ${question}` },
]
Three instructions in that system message do real work. Answering only from the passages is what turns a general chat model into something that speaks for your documents. Permission to say "I do not know" is what stops a confident guess when retrieval comes back empty. Numbering the passages buys citations almost for free, since the model returns a number and your code knows which document it came from.
Hybrid Search and Reranking
A RAG feature fails in two places, and the two look nothing alike. A generation failure is loud, because the answer contradicts the passage above it. A retrieval failure is quiet: the model answers politely from five chunks that were the wrong five, while the right paragraph sat in the index all along.
Pure vector search causes most of them. It matches meaning, so it misses an exact order number, a SKU or an error code that a keyword index finds instantly. Hybrid search runs both, a keyword search such as BM25 and a vector search, then merges the two ranked lists.
Reranking handles the rest. Fetch 50 candidates instead of five, score each against the question with a cross-encoder model that reads the question and the chunk together, and keep the best five. It costs a few hundred milliseconds and buys the largest single jump in answer quality available.
Keeping the Vector Index in Sync
The index is a copy of your documents, and like every copy it goes stale the moment someone edits the original. A policy updated in the admin panel does not change its vectors, so the assistant keeps quoting the old return window and nothing in the logs says why.
Treat it as the cache it is and invalidate on write. Store a content hash with each chunk, re-embed only the chunks whose hash changed when a document is saved, and delete the chunks whose source is gone.
Wrapping Up
RAG has a reputation as an AI technique, and almost none of it is. The two model calls are single HTTP requests. Everything between them is work full-stack developers already do: splitting text, writing a filtered query, ranking results, building a string, and invalidating a cache when the source changes.
That is also where the effort goes. A team unhappy with a RAG feature is usually unhappy with its retrieval, and the fix lives in the chunking and the ranking, not the prompt.
Build the simplest version first: pgvector, fixed-size chunks, plain vector search. Measure it against real questions whose answers you know, then add hybrid search and reranking where the numbers ask for them. For how this comes up in hiring rounds, the Top 50 AI Interview Questions for Full Stack Developers covers RAG alongside embeddings, vector search, tool calling and agents.
Want to learn more about AI Engineering? Check out 20 Important Concepts of AI Engineering.
Frequently Asked Questions
What is RAG in simple terms?
RAG means searching your own data for text related to a question, then pasting that text into the prompt so the model answers from it. The model supplies the language and your search supplies the facts.
Does RAG stop hallucination?
It reduces hallucination without removing it. The model can still misread a passage or fill a gap when retrieval returns nothing useful, which is why the system prompt must allow "I do not know" and why answers should cite their passages.
Do I need a vector database for RAG?
No. pgvector in Postgres handles millions of chunks and keeps everything in one store. Move to a dedicated vector database when scale, filtering or index tuning actually demands it.
What is the difference between RAG and fine-tuning?
RAG supplies facts at question time and is updated by editing a document. Fine-tuning changes the model's weights and suits style and format. Facts that change belong in RAG.
What chunk size should I use for RAG?
Start at 500 to 1,000 tokens with a 10 to 15 percent overlap and split on headings. Smaller chunks retrieve precisely but lose context, and larger chunks carry context but dilute the match.
Does RAG work with any LLM?
Yes. RAG is prompt construction, so it works with any model that accepts text, including open-weight models you host yourself.
