Haystack Agentic RAG: How deepset's Pipelines Work

Haystack is deepset's open-source Python framework for building LLM applications, including agentic RAG. Version 3.0 shipped on July 20, 2026, and the current release is 3.4.1. Haystack gives you two ways to make retrieval agentic. One is a pipeline graph with routing and loops. The other is an Agent component that calls retrieval as a tool. Below are both, built from the components in the docs. After them come the Haystack 3 changes and what an agent keeps after a run ends.

What is agentic RAG in Haystack, in one sentence?

Agentic RAG in Haystack is retrieval-augmented generation where a router or an LLM decides at run time whether to search, where to search and when to stop.

Plain RAG runs one fixed path: retrieve documents, build a prompt, generate. Haystack's pipeline docs describe pipelines as directed multigraphs of components. That means a pipeline can branch and loop. The framework is Apache-2.0 licensed and installs from PyPI as haystack-ai.

How do routed pipelines differ from the Agent component?

Both are agentic. They put the decision in different places.

Sources: Haystack's pipeline docs and Agent docs for version 3.4.

The two also combine. An Agent is a component, so it can sit inside a pipeline. Haystack 3.4 added telemetry that reports an Agent's chat generator and tools when it runs inside a pipeline.

How does a routed agentic RAG pipeline work?

The reference example is Haystack's fallback tutorial, last updated June 12, 2026. It answers from a local document store first and goes to the web only when it has to.

1. A BM25 retriever pulls documents from an in-memory document store.
2. A prompt tells the LLM to answer from those documents, or to reply "no_answer" if they do not contain it.
3. A ConditionalRouter reads the reply. A normal answer goes to the output and the run ends.
4. On "no_answer", the router sends the original query to a web search component (SerperDevWebSearch, from an integration package). A second prompt and LLM answer from the web results.

The routing logic is a string check, so you can unit-test each branch. The cost is rigidity. A half-answered question still takes the local branch, because only a "no_answer" reply triggers the web search.

Pipelines also support loops, such as a generator paired with a validator that sends bad output back for another try.

How does the Haystack Agent handle retrieval?

The Agent component is a loop around a chat model that supports tool calling. You give it a chat generator and a list of tools, plus a system prompt if you want one. It keeps calling the model and running tools until an exit condition fires. The default exit is a plain text reply with no tool call. You can also name a tool that ends the run after it executes. The step cap, max_agent_steps, defaults to 100.

Retrieval becomes a tool. Haystack's experimental Agent Pack shows the pattern in its Advanced RAG Agent. You pass a document store and a retriever. The retriever becomes a search_documents tool. Three metadata tools let the agent inspect the store's fields before it builds filters. Agent Pack requires Haystack 3.1 or later, and the tutorial warns that its APIs may change.

Within a run, the Agent keeps a State object. You declare its keys in a state schema. Tools then read from and write to those keys. In the Advanced RAG Agent, retrieved documents collect in State under a documents key. They are deduplicated by ID and come back with the answer.

What changed in Haystack 3?

The maintainers slimmed the core and reworked the Agent in 3.0, per the release notes:

• Legacy generators are gone. Chat generators replace them.
• About 30 components moved into separate integration packages.
• AsyncPipeline merged into Pipeline, and the Agent now runs tools itself.
• The Agent gained hooks at six points, from before_run to after_run. Human-in-the-loop confirmation is now a before_tool hook.
• A SkillToolset loads skills on demand, so the model sees short names and descriptions until it needs more.
• Tools must map State values explicitly with inputs_from_state.

The haystack-experimental package is no longer installed automatically. Version 3.4.0, released on October 15, 2026, added an experimental compactor for long runs. It moves older tool results out of context into a store and leaves a reference the agent can read back.

Does a Haystack agent remember anything between runs?

Not by default. The Agent docs describe State and message history within one run. They do not describe any persistence between separate run calls.

The older chat-history components are no longer maintained. Haystack's conversational RAG tutorial uses chat message components from haystack-experimental. The tutorial says that package is now archived and no longer maintained, and the components were not adopted into core. Its store is also in-memory. For persistence, the tutorial points to a memory store such as Mem0, or to history kept in your own application state.

The docs list a Mem0 integration backed by the Mem0 cloud API. You scope memories with IDs, such as one per user.

The bottom line

Start with a routed pipeline if your fallback logic fits in a few branches, because you can test every path. Use the Agent when questions are open-ended or span several stores, and cap its steps. On Haystack 2.x, plan the 3.0 migration around the removed generators and the explicit State mapping. Pick a persistence layer early if users expect the agent to remember them.

How do you keep what your coding agent learns about a Haystack project?

Building a Haystack app produces a second kind of memory problem, on the developer's side. Over weeks you learn which retriever wins on your evaluation set and which 3.0 breaking change broke your build. A coding agent such as Claude Code or Codex starts each session without that history. If you switch between them, each keeps its own notes.

memU keeps that developer-side record for the coding agent, separate from your app's user-facing chat history. A scheduled background task slices new session logs into jobs. The agent distills each job into memory and skill Markdown, and memU commits it. memU makes no LLM or chat calls itself. It stores, embeds and retrieves the Markdown the agent prepared. A standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before answering. All hosts share one backend, so a lesson from a Codex session reaches Claude Code.

The hosted version is free and syncs across devices. Get an API key from memu.so and send your agent one message:

Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU, API Key is YOUR_MEMU_API_KEY.

The self-hosted option and the host support matrix are in the memU GitHub repository.

FAQ

Is Haystack free to use?

Yes. Haystack is open source under the Apache-2.0 license and installs from PyPI as haystack-ai. Many model and store integrations ship as separate packages.

What is the latest version of Haystack?

Haystack 3.4.1, released on October 21, 2026, is the latest on PyPI. It is a security patch on top of 3.4.0, which shipped on October 15, 2026.

Does Haystack have built-in agent memory?

Not as a core feature. The Agent keeps State only within a run. The archived haystack-experimental package had chat message stores, and the docs now point to integrations such as Mem0 for persistent memory.

What is the difference between a Haystack pipeline and an Agent?

A pipeline is a graph of components you wire, with branches and loops. An Agent is a component that loops between an LLM and tools until an exit condition is met. An Agent can run inside a pipeline.

How do I add web search fallback to Haystack RAG?

Prompt the LLM to reply "no_answer" when the documents lack the answer. Then use a ConditionalRouter to send those queries to a web search component such as SerperDevWebSearch. A second prompt and LLM answer from the results.