Crawl4AI: The Open-Source Web Crawler for LLMs

Crawl4AI is an open-source Python crawler that turns web pages into Markdown a language model can read. It runs a real browser, so it handles JavaScript-heavy sites. It can return structured JSON through CSS selectors or an LLM. Its GitHub repository had about 86,500 stars and release 0.9.5 out in October 2026. Below you will see when a hosted scraping API is the better buy, and what your agent has to remember that the crawler forgets.

What is Crawl4AI, in one sentence?

Crawl4AI is an Apache-2.0 web crawler and scraper that converts websites into LLM-ready Markdown for RAG pipelines and AI agents.

That wording follows the project's GitHub README. The README listed release 0.9.5, dated October 14, 2026, when we checked. The documentation says it is open source "with no forced API keys." You install it with pip, then run a setup command that installs a Playwright browser.

How does Crawl4AI differ from a hosted scraping API?

The main alternative is a paid API such as Firecrawl or Tavily that runs the browsers for you. Crawl4AI puts the browser on your machine or your server.

Crawl4AI details are from the README and the docs.

Self-hosting gives you control over data and cost. The trade-off is operations: you keep browsers patched and deal with sites that block automation.

How does Crawl4AI work?

You open an async crawler and pass it a URL with a run config. Crawl4AI loads the page in a headless browser and runs any hooks or scripts you set. It then turns the result into Markdown.

The raw Markdown keeps everything on the page, including menus and footers. "Fit Markdown" filters strip that boilerplate before the text reaches your model. The README lists three content filters. Two are cheap: one prunes by page structure, and one ranks blocks against a query with BM25. The third asks an LLM.

For structured data you pick an extraction strategy. The CSS and XPath strategies take a JSON schema of selectors and return typed records without any model call. This is the cheap and repeatable option for sites with stable markup. The LLM strategy sends page content to any provider that LiteLLM supports and asks for JSON that matches your schema. It copes with messy pages but costs tokens on every run.

Deep crawls follow links from a start page using breadth-first, depth-first or best-first order. The README says a resume state lets a crashed deep crawl pick up where it stopped.

What is adaptive crawling?

Adaptive crawling answers a common agent problem: how many pages are enough? Per the adaptive crawling docs, the AdaptiveCrawler scores what it has gathered against your query. It measures coverage of the query terms and whether pages agree. It also checks saturation, meaning new pages stop adding information. Crawling stops when confidence passes 0.7 (the default) or after 20 pages.

The default statistical strategy is term-based and needs no API calls or model. An embedding strategy handles vaguer queries, but it needs an embedding model plus a chat model to expand the query. Progress saves to a state file for later resumption. The collected pages export as a JSONL knowledge base that another session can import.

What does Crawl4AI remember between runs?

Crawl4AI keeps three kinds of state.

The cache stores fetched pages. The cache modes page lists five settings, from normal read and write to bypassing the cache for one call. The old boolean flags were removed in version 0.5.

Sessions reuse one browser tab across sequential steps, such as logging in and then paging through results. The session docs say they are designed for sequential workflows. You close a session yourself when you are done.

The third kind is the adaptive crawl state and its exported knowledge base. All of it describes pages and crawl progress. Notes on which selector schema finally worked, or which site needed stealth mode, end up nowhere in it.

How do agents use Crawl4AI?

The README describes a self-hosted Docker server with a REST API. Its endpoints return Markdown and crawl results, and they can also render screenshots or PDFs. It requires an API token you set. The same server exposes MCP, so an agent like Claude Code can call the crawler as a tool.

What are Crawl4AI's limits?

The star count and feature list are the project's own. We found no independent benchmark that compares its extraction accuracy with hosted services. Running browsers at scale is your job, and so is respecting each site's terms and robots rules. The LLM strategy also sends page content to whichever provider you configure, so check what that provider retains.

The bottom line

Use Crawl4AI when you want crawling under your control and can run a browser fleet. Start with Fit Markdown and a CSS schema for stable sites. Reach for the LLM strategy only where markup is unreliable. For research-style tasks, try the adaptive crawler before writing a fixed link-following crawl.

How do you keep what an agent learns while building crawlers?

Crawl4AI caches pages and saves crawl progress, and the know-how stays with whoever built the crawl. An agent that spent an hour finding the right selector schema for a retailer's product pages starts from zero in the next session. The same applies to the proxy setting that got past a rate limit.

memU records that experience as a shared wiki of readable Markdown. A scheduled background task slices new session logs into jobs, which the agent distills into memory and skill Markdown. A working extraction recipe for one site can become a reusable skill. A standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before it answers. memU itself makes no LLM calls. It stores and retrieves the Markdown the agent prepared, with embeddings for search.

All hosts share one backend, so a recipe Codex worked out is available to Claude Code. Host support is listed in the memU GitHub repository. The hosted version is free and syncs across devices. Get an API key from memu.so and send your agent one message:

Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU, API Key is YOUR_MEMU_API_KEY.

FAQ

Is Crawl4AI free?

Yes. Crawl4AI is open source under the Apache-2.0 license and needs no third-party API key. You pay only for your own compute and for any LLM provider you use for extraction.

How do I install Crawl4AI?

Run pip install -U crawl4ai, then crawl4ai-setup to install the browser. If setup fails, the README suggests installing Chromium through Playwright directly.

Can Crawl4AI extract data without an LLM?

Yes. The CSS and XPath extraction strategies use a JSON schema of selectors and return structured records without calling any model.

Does Crawl4AI work with Claude Code?

Yes, through its self-hosted Docker server, which exposes an MCP endpoint that agents such as Claude Code can call.

What is Fit Markdown in Crawl4AI?

Fit Markdown is page Markdown with boilerplate such as navigation and footers removed by a content filter. Filters can prune by structure, rank with BM25 against a query, or use an LLM.