A general-purpose AI model knows a lot about the world and nothing about your business. It has never read your price list, your refund policy or your internal handbook. Retrieval-augmented generation (RAG) fixes that without retraining anything. We compared Amazon Web Services' explanation with engineering guides from Zilliz, StackAI and an independent chunking comparison, and distilled what they agree on.
Key takeaways
- RAG = find the most relevant passages first, then let the model answer using them.
- It keeps answers current, cheaper to maintain than retraining and able to cite sources.
- Quality depends mostly on retrieval: clean documents, sensible chunking, hybrid search and reranking.
- Enforce permissions during retrieval and measure everything on your own questions.
Why RAG Exists
AWS explains the problem well. Language models can state false information confidently, give outdated or generic answers and draw on non-authoritative sources. RAG adapts a general model to a specific domain or an organisation's own knowledge by retrieving information from an authoritative source outside the model's training data. It gives you more control over output, and lets users see where an answer came from.
How RAG Works
- Prepare. Documents from files, databases or APIs are cleaned, split into chunks and converted into numerical vectors ("embeddings") that capture meaning.
- Store. Embeddings go into a vector database or search index.
- Retrieve. The user's question is converted the same way and the system finds the closest chunks.
- Augment. The retrieved passages are added to the prompt.
- Generate. The model writes an answer, ideally with citations.
- Refresh. Documents and embeddings are updated on a schedule or in real time.
RAG vs Fine-Tuning
Retraining a foundation model is expensive in compute and money, and new facts only arrive with the next training run. With RAG, new information is added at retrieval time, so answers reflect recent documents, and you can restrict access by permission level and trace incorrect references. Use fine-tuning when you need a particular style or output format, not as your main way of teaching facts.
Chunking: The Step Most Teams Underestimate
Chunking determines what the retriever can find and what the model sees. The sources describe the same trade-off: small chunks make it easier to pull the exact supporting sentence but can surface near-matches that do not really answer the question, while large chunks keep context together but dilute relevance and inflate prompts. Common approaches:
- Fixed-size splitting: simple and cheap, but it ignores natural boundaries and can cut a definition in half. Overlap between chunks reduces boundary losses at the cost of duplicate text.
- Recursive splitting by separators (headings, paragraphs, sentences): a practical default that prefers natural units.
- Semantic chunking: splits where the topic changes; better for mixed content, with extra processing cost.
- Parent/child ("small-to-big"): match on small chunks, but pass the larger parent section to the model. Zilliz's experiments found a setup using small chunks of about 175 tokens, larger ones of 512 and a small overlap improved answer quality.
- Structure-aware chunking for PDFs, tables and code, keeping headings, captions and function signatures intact.
Include the document title and heading path in each chunk so a passage still makes sense out of context. In Zilliz's tests, around 512 tokens gave the most relevant answers, with 256 close behind and faster, but treat that as a starting point and test on your own documents.
Retrieval: Hybrid Search and Reranking
StackAI recommends running keyword search (BM25) alongside vector search. Keywords catch exact identifiers like product codes, error numbers and version strings; vectors catch meaning. Merge the two lists with a rank-fusion method. Then rerank: pull a broad candidate set (tens to a couple of hundred passages), score each question-passage pair with a cross-encoder, and keep only the best handful for the model. Zilliz's study similarly found hybrid search and reranking improved results, while exotic query-rewriting tricks added noticeable latency for modest gains.
Two more practical tips: filter by metadata (product, region, version, effective date) before ranking, which often helps more than swapping embedding models, and place the most relevant passages closest to the question in the prompt.
Permissions and Security
Enforce access rules during retrieval, not after the answer is written. Attach user or group labels to chunks, filter on them at query time, and separate tenants where data boundaries require it. Otherwise your assistant may quote a document the user should never see. Log which chunks were retrieved and cited so answers can be audited.
Where RAG Projects Go Wrong
- Messy source content. Duplicated, outdated or contradictory documents produce confusing answers. Clean first.
- Weak retrieval. If the right passage is not found, the best model cannot rescue the answer.
- No "I don't know". Instruct the model to answer only from the sources and to say when they do not contain the answer.
- No evaluation, so nobody knows whether a change helped.
How to Measure Quality
Build a golden set of 50 to 200 real user questions with the correct source documents. Measure retrieval directly (is the right passage in the top results?) and measure answers for faithfulness (grounded in the retrieved text, no invention) and relevancy (does it actually answer the question). Measure stages separately (keyword, vector, fused, reranked) so you can see where a regression came from, then roll out changes gradually. For the live-monitoring side, see our AI observability guide.
Where RAG Fits
- Customer support assistants grounded in your help centre, a natural upgrade to any website chatbot.
- Internal "ask the handbook" tools for HR and operations.
- Sales enablement: product specs and case studies on demand.
- Knowledge for AI agents that must act on company policy.
Getting Started
Begin with one well-maintained knowledge source, a simple pipeline and 30 to 50 test questions. Improve retrieval before you touch the model. Node-devs builds RAG assistants for websites and internal tools, from data preparation to a polished chat interface.
References and further reading
This guide was compiled from the sources below. We compared them, kept the points they agree on and wrote the article in our own words. Check the originals for more detail.
