Agentic RAG
The agent decides when and what to retrieve, rewrites queries and checks relevance.
Problem
Classic retrieval-augmented generation runs one search with the user’s exact words and stuffs the top results into the prompt. That fails when the question is vague, needs facts from several sources, uses different wording than the documents, or doesn’t need retrieval at all. The model then answers from whatever came back, relevant or not.
- User questions rarely match how documents are written.
- Complex questions need several searches, often in different sources.
- Irrelevant retrieved text misleads the model and wastes context.
- Each extra search adds latency and cost.
Solution
Make retrieval a set of tools the agent controls instead of a fixed first step. The agent decides whether it needs to retrieve, which source to search (documents, tickets, a database), and how to phrase the query, splitting complex questions into sub-queries. It grades what comes back for relevance, discards weak results, and rewrites the query or tries another source when results are poor. It answers only from passages it kept, with a citation for each claim, and says so when it can’t find support.
Diagram
Retrieval is a loop the agent controls. Grading results before answering is what separates this from one-shot RAG.
When to use
Use it when
- Questions vary widely in wording and complexity
- Answers often need facts from more than one source
- Users need citations they can check
- Bad answers from irrelevant context are costly
Avoid it when
- Simple lookups where one search reliably works
- Strict latency budgets that can't absorb extra searches
- The whole corpus fits in the context window; just include it
Trade-offs
Gains
- Handles vague and multi-part questions far better than one-shot search
- Citations make answers verifiable
- Can say "I couldn't find this" instead of guessing
Costs
- Slower than single-pass retrieval
- Grading quality limits answer quality
- Retrieval loops can wander without a step limit
Worked example
An internal assistant answers employee questions from HR policies, benefits documents and past HR tickets.
- Employee“Can I use my learning budget for a conference in Europe, and does the travel count against it?”
- AgentSplits this into two sub-queries: learning budget eligible expenses, and conference travel policy. It searches the policy documents first.
- GraderKeeps 2 of 6 passages. Drops a 2023 version of the policy, which has a newer replacement, and an unrelated passage on team offsites.
- AgentThe travel question isn’t answered by policy text. It rewrites the query and searches HR tickets, finding 3 resolved tickets about the same case.
- AnswerYes for the conference fee, up to the $2,000 annual budget. Travel is covered separately by the travel policy and needs manager approval. Each sentence cites a policy section or ticket.
Failure modes
| Failure | Symptom | Mitigation |
|---|---|---|
| Retrieval when not needed | Simple questions slowed down by unnecessary searches | Let the agent answer directly when no retrieval is needed; measure it |
| Stale or conflicting sources | Answer mixes an old policy with a new one | Store version and date metadata; prefer the latest; flag conflicts |
| Weak grading | Irrelevant passages kept and quoted | Grade each passage against the specific sub-question |
| Unsupported claims | Answer goes beyond what retrieved text says | Require a citation per claim; check with Draft–Verify |