RAG, or what does your AI do when it searches
You ask an AI about the current state of a topic, and it answers with sources. Did it look something up online? Or answer from memory? Or search your own documents? All three happen, and from the outside they look alike.
A language model initially knows only what it read during training, and that state becomes outdated at some point. For anything current or your own, the application has to hand the model additional information. It gets it from the internet (web search) or from a chosen collection of documents such as your manual. Experts call the principle behind this RAG (retrieval-augmented generation, meaning “answering after a lookup”). In the narrower sense, RAG usually means searching your own documents. Web search belongs to it in the wider sense, but is often called something else. In both cases the model does not run the search itself: the application searches and puts the passages it found in front of the model, and the model answers from them.
Why this matters to you: depending on the route, the answer comes from different sources, and you have to check it differently. This article shows you how to tell the routes apart and which one suits which task.
What this means for you
Start with the simplest solution and move up only one rung when the previous one is not enough.
- 01Instruction
Shorten or rewrite a text
Give the AI the text and tell it exactly what to do.
Clear prompt - 02Context
A question about a short document
Upload the document straight into the chat.
Document in the chat - 03Own documents
Repeated questions about many documents
Software first looks up the matching passages and passes them to the AI.
RAG - 04Internet
Current information from the web
Use an AI with web search and look at the sources.
Web search - 05Your software
Current data, such as an order status
Connect the AI to the system. At first it may only read.
Tool, read-only - 06Fixed process
A process that is always the same
Have a programme built that works through the steps in order.
Workflow - 07Own decision
The AI should decide for itself what comes next
Test it against the fixed process first. Does it really bring anything?
Limited agent
Your next step: pick a task from your everyday work. Write down what a normal request looks like, what can go wrong and what the result should be. Have the AI build and demonstrate the smallest suitable app. That way you see whether the extra technology really brings anything.
Prompting: the brief to the AI
To shorten a text or rephrase an answer, a clear instruction to the AI, the so-called prompt, is often enough. Describe who the text is for, what the AI should do, which information it has and what the result should look like. Then check the draft yourself.
What speaks for it: You need no additional software. A short manual that you query once can be given to the AI directly, as long as length and data protection allow.
What speaks against it: With many or changing documents, that is no longer enough.
RAG: finding the right information before answering
RAG stands for retrieval-augmented generation. The app looks up matching passages in your documents and passes them to the AI together with your question. The AI model does not need to be retrained for this. AWS · RAG basics
In the support example, the app finds the policy on delivery times, and the AI drafts an answer from it and names the passage.
What speaks for it: It can make sense for many recurring questions about a larger collection of documents. The source reference makes checking easier.
What speaks against it: A source reference does not guarantee a correct answer. You have to check whether the passage is current and fits the question. The quality depends on how well the documents are prepared and kept up to date, and on whether each person sees only the documents they are allowed to see.
Web search: what the AI actually does on the internet
With a web search, the model decides from your question whether it needs to look something up, and it words a search query. The search itself is run by the provider. The model then answers on the basis of the hits and names the sources. According to Anthropic’s documentation, Claude searches for information that is current or changing, such as news or prices, and answers directly when the knowledge is stable. Anthropic · Web search tool
The difference to RAG in the narrower sense: with RAG you decide which documents are searched. With web search, the search engine decides what comes up as a hit. Whoever builds an app can limit the search: how often it searches and which websites are allowed or blocked.
What speaks for it:
- The AI can use information that came into being after its training cut-off.
- The answer contains sources with links that you can read up on.
What speaks against it:
- The quality of the hits varies, because websites differ in how reliable they are.
- A source reference does not guarantee a correct answer; you still have to check it.
- Parts of your question go out as a search query. So do not put confidential details into questions that can trigger a web search.
- At Anthropic, web search costs 10 US dollars per 1,000 searches on top of the usage costs of the AI model (according to the documentation, as of 25 October 2026).
Tools: when the AI accesses data or functions
“How long does standard delivery take?” is answered by a policy. “Where is my order?” instead needs the current data on that very order. For that, the AI has to be connected to your order system. In the AI world, such a connection is called a “tool”. What the AI may do with it is set up in the app and in the connected system. An instruction alone gives the AI neither access nor rights.
What speaks for it: The answer rests on the current data rather than on a document.
What speaks against it: Every connection brings rights and test cases with it. To learn, start with made-up data and with a connection that may only read.
Workflow or agent: who decides the next step?
In a fixed process (workflow), the order of the steps is set in advance. An agent decides for itself which step to take next and which tools to use for it. The documentation of LangGraph, a software package for such processes, draws the same distinction. LangGraph · Workflows and agents
What speaks for a fixed process: For “look up a policy, draft an answer, show the source”, a fixed order is often enough.
What speaks for a limited agent: It becomes interesting when the app is meant to choose between approved tools depending on the request and to ask follow-up questions.
What speaks against an agent: The more it decides for itself, the more cases you have to test.
The coding agent has a different role: it helps you build the app and its tests. The agent inside the app later handles the requests within the limits you have set for it.
Local or external: where does each part run?
Where a part runs is a decision of its own. You can use RAG with an AI model at a provider or with a model you run yourself, and fixed processes and agents are not tied to one place either.
Draw the search, AI model, tools and log separately and note where each part runs. If only the user interface runs on your computer and text passages go to a provider’s AI model, the processing of that content is not local. Whoever runs the model themselves has to clarify technology, licence terms, maintenance and responsibilities.
The provider Lyzr describes its Sovereign offering as a combination of a platform for agents and the technology for the AI model. That is an example of how these decisions have to be made separately. What the specific offering really contains, you have to check there. Lyzr · Sovereign AI
Sources and context
Checked on 23 October 2026, the Anthropic documentation on 25 October 2026. The support examples and the decision aid are our own teaching derivations. Lyzr is a vendor source; the basics are additionally placed in context with official documentation.