What is RAG (retrieval-augmented generation)?
Knowledge, data and integration
RAG, short for retrieval-augmented generation, is a way of using artificial intelligence that lets a general AI model answer questions using your own trusted, current information rather than only what it learnt during training. Before the model writes an answer, the system first finds the most relevant pieces of information from a source you control, such as your policies, manuals or help articles, and hands them to the model to work from. The result is an answer grounded in your material, with far less guesswork, and one you can usually trace back to its source.
Reviewed by Jackie, Head of Learning & Development, Levellers · Last reviewed 8 June 2026
What this means
A large language model, the kind of AI behind tools such as ChatGPT, Claude and Gemini, is trained once on an enormous quantity of text. After that training is finished, its knowledge is essentially fixed. It knows nothing about your business, it has never seen your contracts or your staff handbook, and it has no awareness of anything that happened after its training was completed. When asked a question, it produces an answer from memory, and because it is designed to sound fluent and confident, it will often give a plausible but wrong answer rather than admit it does not know.
Retrieval-augmented generation changes that arrangement. The simplest way to picture it is the difference between a closed-book exam and an open-book exam. Without RAG, the model is sitting a closed-book exam, answering entirely from memory. With RAG, it is sitting an open-book exam, and the book is your own information. When a question comes in, the system first searches a collection of trusted material you have provided, pulls out the passages that look most relevant, and places them in front of the model along with the question. The model then writes its answer based on what is in front of it.
This matters because it separates two things that people often confuse: the model's general language ability and the specific facts an answer needs. The model supplies the fluency, the reasoning and the ability to phrase a clear reply. Your information supplies the facts. You keep control of the facts, you can update them whenever you like, and you can usually see which passages an answer was based on.
The pattern has three plain steps that give it its name. Retrieve: find the relevant information. Augment: add that information to the question. Generate: have the model write the answer. The whole point is grounding, which simply means tying the answer to real, identifiable source material rather than letting the model invent it.
RAG has become one of the most widely used practical AI patterns in business for a straightforward reason. Most organisations do not need an AI that knows everything about the world. They need an AI that knows everything about their world, their documents, their policies, their products, and can answer questions about them reliably. RAG is the most direct and affordable way to get that, which is why it is built into the major business AI platforms and why so many of the AI tools organisations already use rely on it underneath.
Why it matters
For a senior decision-maker, the case for understanding RAG rests on four practical benefits, each of which addresses a real and well-documented weakness of general AI models.
The first is grounding answers in trusted, current and private information. A general model can only draw on what it was trained on, which is a snapshot of largely public text frozen at a point in time. It has no access to your internal knowledge and no knowledge of recent events. RAG connects the model to the specific, authoritative information you choose, so a question about your refund policy is answered from your refund policy, not from a vague impression of how refunds tend to work across the internet.
The second is reducing made-up answers. The tendency of AI models to state false information confidently, often called hallucination, is the single biggest barrier to trusting them in a business setting. By forcing the model to work from retrieved source material, RAG measurably reduces this behaviour. It does not abolish it, a point this article returns to, but grounding answers in real evidence is the most effective practical lever most organisations have to make AI answers more dependable.
The third is keeping information current without retraining the model. Retraining or heavily customising a large model is expensive, slow and requires specialist skill. With RAG, when your information changes, you update the documents the system retrieves from. The model itself is untouched. A policy that changed yesterday can be reflected in answers today simply by replacing the relevant document. This makes RAG far more practical for the ordinary reality of business, where information is always moving.
The fourth is traceability and trust. Because a RAG answer is built from identifiable passages, the system can usually show its sources, so a person can check the answer against the original. This is the difference between an AI that asserts things and an AI that can show its working. For any use where being wrong carries a cost, that ability to verify is what makes the difference between an interesting demonstration and something you can actually rely on.
The business value shows up across a familiar set of uses: internal knowledge assistants that let staff ask questions of company documents instead of hunting through shared drives, customer support assistants that answer from your help content, and research or policy assistants that work across a defined set of documents. These are not exotic applications. They are the everyday information problems of most organisations, which is why RAG has moved so quickly from research idea to standard practice. Adoption is now broad: 88 per cent of organisations report regularly using AI in at least one business function, up from 78 per cent a year earlier. But the same evidence carries a sober warning that bears directly on RAG. Turning that usage into measurable financial value remains the harder task, and it is rare: of nearly 2,000 organisations recently surveyed, only about 5.5 per cent said that more than a twentieth of their operating profit could be attributed to AI. The lesson is that value comes far more from disciplined implementation, clean information and redesigned ways of working than from the technology itself.
How it works
This section explains the idea of RAG in enough depth for a leader to reason about it, while staying away from the engineering. Where the mechanics matter for building a system, the technical companion to this article covers them properly.
The core idea: retrieve, then generate
At its heart RAG is two systems working in sequence. The first is a search system. The second is a language model. Neither is new on its own. What the original research did was combine them so that the search results inform what the model writes.
Picture the flow from the user's point of view. Someone types a question. Before the model sees it, the question is sent to a search step that looks through your collection of information and returns the handful of passages most likely to contain the answer. Those passages are then bundled together with the original question into a single instruction, and that instruction is what the model actually responds to. The model reads the question and the supplied passages, and composes an answer from them.
The plain consequence is that the model is no longer answering from memory. It is answering from the material just handed to it. If the right passage is retrieved, the model has what it needs to answer well. If the search step fails to find the right passage, the model is back to guessing. This is why the quality of the retrieval step matters so much, and it is the single most important thing to understand about how RAG can succeed or fail.
Why models need this in the first place
It helps to be clear about exactly what RAG is fixing, because each weakness it addresses is a genuine and structural feature of how these models work, not a temporary flaw that will be patched away.
The first weakness is the training cut-off. A model's knowledge ends on the date its training data was gathered. It cannot know about a price change, a new regulation or a document written last week. Its world is frozen.
The second is the absence of your private information. These models are trained overwhelmingly on public text. Your internal documents, by definition, were never part of that. So a general model simply has no way to know anything specific about your organisation.
The third is hallucination. Because the models are built to produce fluent, probable-sounding text, they will fill a gap with an invented answer rather than leave it blank. The original RAG research framed this directly, noting that purely memory-based models cannot easily expand or revise what they know, struggle to show why they produced a given answer, and may produce what it called hallucinations. RAG was proposed specifically to give models access to knowledge they could point to, update and be held accountable for.
What RAG needs to work
Three ingredients are required, and thinking about them in plain terms is the best way for a leader to judge whether a use is realistic.
First, a trusted source of information. This is your raw material: the documents, records or content the system will draw on. Everything else depends on this being accurate, reasonably organised and genuinely the authority on the questions you want answered. RAG cannot turn poor source material into good answers. If the underlying information is wrong, out of date or contradictory, the answers will faithfully reflect those problems.
Second, a way to find the relevant pieces. When a question arrives, the system has to locate the right passages quickly and accurately from what may be a large body of material. Modern RAG usually does this with what is called semantic search, which finds passages by meaning rather than by exact keyword. This relies on a technique called embeddings, which is a way of turning text into numbers that capture its meaning so that passages about the same topic sit close together and can be matched even when they use different words. Embeddings and the databases that store them, often called vector databases, are covered properly in the technical companion. For a leader, the point to hold onto is simply that the system matches questions to information by meaning, and that getting this matching right is most of the work.
Third, the model to write the answer. This is the general language model. Its job is to read the question and the retrieved passages and produce a clear, accurate reply grounded in them. Importantly, the same general model can serve almost any subject. The intelligence about your particular domain comes from the information you retrieve, not from the model.
RAG compared with the alternatives
A leader choosing how to make AI useful with their own information has a few options, and the honest answer is that they are not mutually exclusive. Understanding the genuine differences prevents both wasted money and disappointed expectations.
Compared with a general chatbot, the difference is grounding. A general chatbot is fluent but unmoored, answering from training memory with no access to your facts and no way to show its sources. RAG keeps the fluency but anchors it to your material. For any question whose answer lives in your documents, a general chatbot is the wrong tool and RAG is the right one.
Compared with fine-tuning, the difference is what you are changing. Fine-tuning means further training the model itself on examples so that it adapts its behaviour, style or grasp of a specialist area. It changes how the model behaves. RAG changes what information the model can see at the moment of answering. The widely used analogy is that fine-tuning teaches the model how to think and respond in a particular way, while RAG gives it the right book to read. They solve different problems. Fine-tuning is well suited to teaching a consistent style, format or specialist manner of reasoning. RAG is well suited to supplying current facts that must be correct and traceable. Many serious systems use both: fine-tuning for how the model communicates, RAG for the facts it communicates. For most organisations starting out, RAG is the more natural first step, because it is cheaper, faster to set up, easier to keep current and far easier to verify. Fine-tuning generally demands a large, high-quality set of training examples and specialist effort, and the resulting model still needs updating as the world moves on.
Compared with long-context models, the difference is more subtle and is the subject of a genuine, ongoing debate. Newer models can accept enormous amounts of text in a single question, sometimes around 500 pages or more at once. This raises a fair question: if you can simply paste your entire knowledge base into the model each time, why bother retrieving selectively? For a small, stable body of information, that simpler approach can indeed be the better choice. One leading model provider advises that if your knowledge base is smaller than about 500 pages of material you can just include all of it in the prompt, with no need for RAG at all. But the case for RAG remains strong for most real situations. Pasting everything in for every question is costly when repeated at scale, it slows responses, and models genuinely struggle with very long inputs: facts placed at the very start or end of a long input are retrieved more reliably than those buried in the middle, and even at the largest context sizes roughly one query in ten can still get the wrong answer. RAG stays efficient as your information grows beyond what fits comfortably, it lets you update knowledge instantly by changing the underlying documents, and it allows fine control over who can see what, which matters greatly for private information. The emerging consensus is not that one replaces the other, but that they are increasingly used together, with longer model capacity letting RAG systems include more retrieved material rather than removing the need to retrieve. The claim that long-context models have made RAG obsolete is not supported by current practice; enterprise demand for dedicated retrieval has, if anything, continued to grow.
Where RAG fits in a business
RAG is the right pattern whenever the core need is answering questions from a defined body of information that you trust and control. That covers a large share of the practical value organisations are seeking from AI today.
It is a particularly good fit where the information changes regularly, where answers must be checkable against a source, where the material is private and cannot be sent off to train somebody else's model, and where the same questions are asked repeatedly and currently consume expensive human time. It is a weaker fit where the task is not really about retrieving facts at all, for instance where you mainly need the model to adopt a particular writing style, or where a question can be answered perfectly well by a capable model on its own without any private information. A useful discipline before building anything is to ask whether the problem is genuinely about facts the model lacks, in which case RAG fits, or about behaviour the model does not exhibit, in which case it may not.
Examples
The following examples are deliberately ordinary. They reflect the uses where RAG is delivering value now, rather than speculative possibilities.
An internal knowledge assistant over company documents. A mid-sized organisation has its working knowledge scattered across a staff handbook, dozens of policy documents, process guides and a sprawl of shared drives. Staff waste time hunting for answers, and newer employees ask the same questions repeatedly. A RAG-based assistant is connected to this material so that an employee can ask, in plain language, how much annual leave they have accrued or what the procedure is for approving a supplier, and receive an answer drawn from the current documents, with a link to the source. The model handles the phrasing; the handbook supplies the facts. When a policy changes, the document is updated and the assistant reflects the change immediately. This pattern is in active use for human resources and policy questions, IT help and knowledge search, and operational queries against manuals and logs, and it is exactly how the AI assistants now built into major workplace software ground their answers in an organisation's own files while respecting each user's existing access permissions.
A customer support assistant grounded in help content. A company with a large volume of repetitive customer questions about orders, returns and common problems connects a support assistant to its help centre, product documentation and policy pages. Routine questions are answered instantly, at any hour, with answers grounded in the company's own published guidance and citing the source article so the customer can read further. Human agents are freed to handle the genuinely complex or sensitive cases. Reported results in this area are encouraging, with well-implemented assistants deflecting roughly 40 to 50 per cent of routine queries, though honest accounts also note that answer-only assistants tend to plateau, with the share of queries handled without a human flattening at around 30 to 40 per cent, and that careful design and a clean knowledge base are what separate good results from poor ones. A cautionary case underlines the stakes: in February 2024 the British Columbia Civil Resolution Tribunal ordered Air Canada to pay a customer just over 800 Canadian dollars after its website chatbot gave him wrong information about bereavement fares, ruling that the airline had not taken reasonable care to ensure the chatbot was accurate and dismissing as a remarkable submission its argument that the chatbot was a separate legal entity. The principle is plain: an organisation remains responsible for what its AI tells customers, regardless of how the answer was produced.
A research or policy assistant over a document set. A team that must work across a defined and often dense body of material, such as legal clauses, regulatory guidance, financial filings or a research corpus, uses RAG to ask questions of the whole set at once. Rather than reading every document, a researcher can ask for the relevant clause, the obligations in a contract or the passage that addresses a specific point, and receive an answer with citations back to the exact source. This is valuable precisely because the answers are checkable, which is essential where being wrong has consequences and a human expert must be able to verify before relying on anything.
A small organisation using a simple RAG-based tool. RAG is not only for large enterprises. A small business can achieve a great deal with off-the-shelf tools that build the retrieval step in for them. Uploading a set of documents to a tool that creates a custom assistant, or using a research tool that answers questions from a defined set of sources you provide, is RAG in practice, without any engineering. A small team can stand up a useful internal assistant or a customer-facing helper in a short time and at modest cost, starting with a handful of trusted documents and expanding as confidence grows. The principle is the same at every size: the value comes from the quality of the information you point it at.
Common misunderstandings
A handful of misreadings come up repeatedly, and clearing them up early saves a great deal of disappointment later.
RAG eliminates hallucination. It does not. It reduces it, often substantially, by grounding answers in retrieved evidence, but it does not remove it. A RAG system can still go wrong: it can retrieve the wrong passage, retrieve a passage that is irrelevant, or correctly retrieve good material and then misread it. If the answer is not present in the information at all, a poorly configured system may still invent one rather than say it does not know. Grounding makes answers more reliable; it does not make them guaranteed.
RAG is the same as fine-tuning. These are different and are often confused. Fine-tuning further trains the model to change how it behaves. RAG leaves the model alone and changes what information it is shown when answering. One adjusts the model; the other adjusts its reading material. They can be combined, but they are not interchangeable, and reaching for fine-tuning when the real need is current, traceable facts is a common and expensive mistake.
RAG means the AI permanently learns your data. It does not. This is one of the most important points for a leader to grasp, and it is reassuring. In a RAG system the model does not absorb or memorise your documents. Your information sits in a separate store that the system searches at the moment a question is asked. The relevant passages are shown to the model just for that one answer and are not baked into it. This is why you can update or remove information instantly, and it is central to keeping control of private material.
Long-context models make RAG obsolete. They do not, though they change the calculation for small, stable bodies of information. As discussed earlier, the practical evidence points to the two approaches being used together rather than one replacing the other, and dedicated retrieval remains the standard approach for information that is large, changing, private or needs tight access control.
RAG is simple to get right. It is simple to describe and simple to demonstrate, which is precisely the trap. A basic version can be assembled quickly and will look impressive on a few easy questions. Making it reliable across the messy reality of real questions and real documents is considerably harder, and the difficulty lives almost entirely in the retrieval step and the quality of the source information. The phrase practitioners use is that RAG is easy to start and hard to master. A leader should expect the build to be quick and the tuning to accuracy to be the real work.
Risks and boundaries
RAG is genuinely useful, and a clear-eyed view of its limits is what allows you to use it well. The risks below are not reasons to avoid it. They are the things to manage.
What RAG does not do. It does not make an AI infallible, and it does not turn a model into an expert that understands your business. It supplies relevant information at the moment of answering and leaves the model to compose a reply. It is an information-access pattern, not a guarantee of correctness, and it should be treated as a way to make AI answers far more dependable rather than perfectly so.
Its dependence on source quality. This is the single most important boundary. A RAG system is only as good as the information it draws on. The well-worn phrase in the field is garbage in, garbage out, and it applies precisely here. If your documents are out of date, contradictory, incomplete or simply wrong, the system will serve those flaws back with the same confident fluency as it serves good answers. Cleaning, organising and maintaining the source information is not a preliminary chore; it is the main determinant of whether the system works. The clean, current knowledge base is the foundation, and there is no clever configuration that compensates for a poor one.
The residual risk of wrong or outdated answers. Even with good source material, retrieval can miss, and an answer can be wrong or based on a document that has since changed. This is why answers should be traceable to their sources, why higher-risk uses need a person reviewing before anything is acted upon, and why no organisation should put an unmonitored RAG system in front of customers or staff for consequential decisions. Keeping the information current is an ongoing responsibility, not a one-off task, because answers are only ever as current as the documents behind them.
Security and access control for private information. This deserves particular attention from any leader handling sensitive or personal data. Connecting an AI to your internal information creates a real risk that it could surface material to someone who should not see it. The crucial principle is that access controls must be enforced at the point of retrieval, so the system can only ever retrieve documents the particular user is permitted to see, rather than relying on the model to be discreet. The major business platforms are explicit that their retrieval respects each user's existing permissions for exactly this reason. Sensitive information disclosure is recognised as a leading risk for this kind of application, and it must be designed in from the start. For UK organisations there is a further dimension: where the information includes personal data, data protection law applies in full, the UK regulator expects a risk-based approach including data protection impact assessments for higher-risk uses, and questions of where data is stored and processed, often called data residency, need deliberate decisions rather than defaults.
This is not technical implementation advice. This article is written to help a leader understand and reason about RAG, not to specify how to build it. The choices that determine whether a system is accurate, secure and compliant, how information is prepared, how retrieval is tuned, how the system is evaluated and monitored, are engineering and governance matters. The technical companion addresses the build; this piece is about whether, when and why to pursue it.
What to do next
The path below is deliberately sequenced. Each step is cheap to do well and expensive to skip.
Start by identifying a question-answering use grounded in your own information. Look for a place where staff or customers repeatedly ask questions whose answers already exist in your documents, where those answers are checkable, and where getting them faster or more consistently would clearly save time or improve service. Internal knowledge assistance and customer support are the usual starting points because the value is obvious and the source material already exists. Resist the urge to pick the most complex or highest-stakes problem first.
Get the source information in order. Before anything is built, look hard at the documents the system would draw on. Are they current? Accurate? Free of contradictions? Reasonably organised? This is where most of the eventual quality is won or lost, and it is work you can begin immediately and benefit from regardless of any AI project. If the information is in poor shape, fixing it is the first task, not an afterthought.
Start with a small, trusted document set. Do not begin by pointing the system at everything you have. Choose a contained, high-trust, well-maintained set of documents covering a clear area, and build the first version against that. A narrow, reliable starting point produces a system you can actually judge, and it keeps the early risk low.
Pilot and check accuracy honestly. Run the system with a limited group of real users on real questions, and measure how often it is right, how often it is wrong, and what happens when the answer is not in the documents. Keep a person in the loop for anything consequential. The discipline that distinguishes the organisations getting value from AI is precisely this: defined processes for when human review is required, and honest tracking of quality rather than enthusiasm. Decide in advance what level of accuracy would justify going further and what level would mean stopping to fix the source material or the retrieval.
Scale with attention to access control and maintenance. Only once a contained pilot is genuinely reliable should you widen the document set, the user base or the range of questions. As you scale, two things become non-negotiable: enforcing access controls at the point of retrieval so the system never surfaces information a user should not see, and committing to ongoing maintenance of the source information so answers stay current. Treat the knowledge base as a living asset with a named owner, not a project that finishes.
The thresholds that should change your plan are straightforward. If accuracy in the pilot is poor, the problem is almost always the source information or the retrieval, and the response is to fix those rather than to abandon the idea or to reach prematurely for fine-tuning. If the information is small, stable and not sensitive, consider whether a simpler approach using a long-context model is enough before investing in dedicated retrieval. If the use involves personal data or sensitive material, bring in data protection and security considerations before you pilot, not after. And if you cannot keep the source information current, reconsider the use, because a RAG system feeding on stale documents will quietly become a liability.
Have a question or a suggestion, or want to understand how we research and review these guides? Read about our editorial standards and how to reach us.
FAQs
What is RAG in simple terms?
RAG, or retrieval-augmented generation, is a way of using AI where the system first finds relevant information from a trusted source you control, then has an AI model write an answer based on that information. It is like giving the AI an open book of your own material to consult, rather than asking it to answer from memory. This grounds answers in your real, current facts instead of leaving the model to guess.
Why do organisations use RAG?
Because a general AI model knows nothing about your organisation and nothing about recent events, and it tends to produce confident but wrong answers when it does not know something. RAG connects the model to your own trusted, current information so it can answer questions about your policies, products and documents reliably, keep those answers up to date without retraining anything, and show the sources behind them.
Does RAG stop AI from hallucinating?
It reduces hallucination significantly but does not eliminate it. By making the model answer from retrieved evidence rather than memory, RAG produces more factual and more traceable answers. But it can still go wrong if the search step retrieves the wrong passage, if the information is not in the source at all, or if the model misreads what it is given. Grounding makes answers more dependable, not guaranteed, which is why higher-risk uses still need human review.
What is the difference between RAG and fine-tuning?
Fine-tuning further trains the model itself to change how it behaves, such as adopting a particular style or grasping a specialist area. RAG leaves the model unchanged and instead supplies relevant information for it to read at the moment of answering. A useful way to put it: fine-tuning teaches the model how to think and respond, while RAG gives it the right book to read. They address different needs and are often used together, but for supplying current, checkable facts, RAG is usually the better and cheaper first step.
Do long-context models replace RAG?
Not in general. Newer models can take in very large amounts of text at once, around 500 pages or more, and for a small, stable body of information that can be simpler than building a retrieval system. But pasting everything in for every question becomes costly and slow at scale, and models can lose track of details buried in very long inputs. RAG stays efficient as information grows, updates instantly when documents change, and allows fine control over who can see what. In practice the two are increasingly combined rather than one replacing the other.
What information can RAG use?
Almost any text-based information you control: policy documents, staff handbooks, product manuals, help articles, contracts, regulatory guidance, research material, support tickets and records held in databases. The essential requirements are that the information is accurate, reasonably organised and genuinely authoritative for the questions you want answered. The quality of this source material is the single biggest factor in whether the system works well.
How accurate is RAG?
It depends almost entirely on the quality of your source information and how well the search step finds the right passages. With clean, current, well-organised material and careful tuning, RAG can be very reliable and, importantly, its answers can be traced back to their sources for checking. With poor or out-of-date material it will produce poor answers with the same confidence. Accuracy is something you must measure with real questions during a pilot, not something to assume.
Can small companies use RAG?
Yes. RAG is not only for large enterprises. Off-the-shelf tools now build the retrieval step in for you, so a small business can upload a set of documents and have a working assistant in a short time, at modest cost and with no engineering. The same principle applies at every size: the value comes from the quality of the information you point it at, so a small organisation with well-maintained documents can get excellent results.
What does it cost to run RAG?
Costs come from a few parts: a usually one-off cost to process your documents into a searchable form, an ongoing cost to store that searchable information, and a per-question cost each time the AI model generates an answer. For a small organisation using off-the-shelf tools, this can be very modest. At larger scale the running costs rise with the volume of information and the number of questions, and there is also the real cost of preparing and maintaining the source information, which is often underestimated and matters more than the technology bill.
Sources
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv / NeurIPS (Lewis et al.)). PRIMARY. Origin and definition of RAG; the term coined here; the stated limitations of memory-only models including that they cannot easily expand or revise their memory, cannot readily explain their predictions, and may produce hallucinations; first submission 22 May 2020.
The state of AI in 2025: Agents, innovation, and transformation (McKinsey & Company). SECONDARY (analyst). 88 per cent of organisations report regular AI use in at least one function (up from 78 per cent); only about 5.5 per cent report more than 5 per cent of operating profit attributable to AI.
Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal (reported by Pinsent Masons)). SECONDARY (legal reporting of a primary ruling, 14 February 2024). Air Canada ordered to pay the customer; the airline did not take reasonable care to ensure its chatbot was accurate; organisations remain responsible for what their AI tells customers.
Guidance on AI and data protection (Information Commissioner's Office (UK)). PRIMARY (regulator). UK data protection law applies to AI using personal data; risk-based approach; data protection impact assessments for higher-risk processing.
Seven Failure Points When Engineering a RAG System (arXiv (Barnett et al.)). SECONDARY. Garbage in, garbage out dependence on source quality; failure modes including missing context producing plausible but wrong answers.