What Is LiteLLM? The Open-Source LLM Gateway Explained
LiteLLM is an open-source gateway that lets an application or agent call models from many providers through one OpenAI-compatible API. The docs name OpenAI and Anthropic among them, along with Bedrock and Vertex AI. Teams use it so an agent can switch models with a config change and so platform owners can hand out keys with budgets. Before you install it from PyPI, read the section on its two 2026 supply chain incidents.
What is LiteLLM, in one sentence?
LiteLLM is a Python SDK and a self-hosted proxy server that translate one OpenAI-style request format into calls to more than 100 LLM providers.
The LiteLLM docs describe the proxy as "a self-hosted OpenAI-compatible gateway." Every response follows the OpenAI Chat Completions format, whichever provider served it. The code is on GitHub, where BerriAI now calls it an AI gateway with a Rust core and a Python SDK. Most of the repository is MIT licensed. The enterprise directory has its own commercial license.
How does the LiteLLM SDK differ from the LiteLLM proxy?
The SDK is a library you import. The proxy is a service your whole team calls. Start with the SDK and move to the proxy once several apps share provider keys.
| Provider SDKs directly | LiteLLM Python SDK | LiteLLM proxy (gateway) | |
|---|---|---|---|
| How you call it | One client per provider | One completion function for all providers | Any OpenAI client pointed at your proxy URL |
| Where provider keys live | In each app | In each app | In the proxy only |
| Routing and fallbacks | You write them | Router class in code | Same router, set in the proxy config |
| Budgets and virtual keys | None | None | Per key, team and user |
| Logging and guardrails | Per app | Per app | Central |
| What you operate | Nothing | Nothing | A server, plus Redis to share state across instances |
Sources: the LiteLLM docs and the routing docs.
How does LiteLLM routing work?
You define a model group, such as "gpt-4o", and list several deployments behind it: the same model on OpenAI and Azure, or two regions of one provider. The router picks a deployment for each request. The strategies are:
- simple-shuffle (the default, recommended for production): picks by RPM or TPM limits or by weight
- latency-based-routing: the lowest response time
- least-busy: the fewest calls in flight
- cost-based-routing: the lowest cost
- usage-based-routing: the lowest TPM use that minute. The docs advise against it in production.
When a deployment fails, the router takes it out of rotation for 5 seconds by default. A 429 triggers that cooldown, as does a 401, 404 or 408 error or a minute in which more than half its calls fail. Rate-limit errors are retried with exponential backoff, and fallbacks send a failed request on to another model group you list. The docs use Redis in production to share cooldown and usage state across proxy instances.
What does the LiteLLM proxy control?
Per the docs, the proxy issues virtual keys with budgets and rate limits for each team or user. It tracks spend across providers in one place. Requests also pass through its central logging and caching. Its guardrails can filter content and mask PII.
For agents, that means one place to cap what an autonomous loop can spend. A coding agent that retries a failing tool fifty times stops when its key's budget runs out.
How fast is LiteLLM?
The GitHub README claims "8ms P95 latency at 1k RPS." That is the vendor's own benchmark. In our view that overhead is small next to agent model calls that take seconds each. Measure it on your own hardware if you sit LiteLLM in a latency-sensitive path.
What happened in the 2026 LiteLLM supply chain incidents?
Three LiteLLM releases on PyPI carried malware in 2026, in two separate incidents.
On March 24, 2026, versions 1.82.7 and 1.82.8 were live on PyPI from 10:39 UTC for about 40 minutes. Both carried a credential stealer. LiteLLM's security update traced the likely cause to the Trivy dependency in its CI security scanning. Users of the official proxy Docker image were not affected. A clean 1.83.0 shipped on March 30 through a rebuilt release pipeline.
On October 21, 2026, version 1.107.0 was compromised through the release pipeline. Per LiteLLM's incident post, it was on PyPI for about 1 hour 43 minutes. Several Docker tags, including latest, also pointed to bad images for a while. The post says it contained malware that steals credentials. It recommends treating any host that ran it with internet access as compromised. It also says to rotate every key the host could reach and reinstall pinned to 1.106.2. Enterprise customers on the private registry were not affected.
A gateway holds every provider key you have. Pin exact versions and verify image signatures with cosign, which the README documents for GHCR images. The October attack moved the latest Docker tag, so do not deploy from it.
Read the fine print
The provider count ("100+") and the latency figure come from LiteLLM. The security tag of its blog also lists several 2026 CVEs in the proxy, each fixed in a later release. Running the proxy means following those advisories. LiteLLM announced on October 27, 2026 that fixes for the most severe issues will reach enterprise customers through a private registry before public release.
The bottom line
Use the LiteLLM SDK if one app needs to swap providers. Run the proxy when several apps or agents share keys and you need budgets in one place. Pin exact versions in either case, because a compromised gateway exposes every provider key at once.
How do you keep agent memory when the gateway switches models?
A gateway makes model switching cheap. A fallback can move an agent from one provider to another mid-task, and a config change can swap the default model overnight. None of that carries what the agent learned. Context lives in the session, and project knowledge lives wherever the host keeps it. When the model or the host changes, that knowledge has to come from somewhere outside both.
memU keeps it in readable Markdown files outside any model or host. A scheduled background task slices new session logs into jobs. The agent distills each job into memory and skill files, which memU commits. memU's core makes no LLM calls. It indexes those files and serves them back when a standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before answering. Hosts such as Claude Code and Codex share one backend, so what one learned is there for the others whatever model they call. More on the design is in why memU gives memory back to the agent.
To try it with an API key from memu.so, send your agent one message:
Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU
Self-hosting instructions are in the memU GitHub repository.
FAQ
Is LiteLLM free?
The core SDK and proxy are open source under the MIT license. Code in the repository's enterprise directory is under a separate commercial license. LiteLLM also sells enterprise features and a private registry.
What is the difference between LiteLLM and OpenRouter?
LiteLLM is software you run yourself, with your own provider keys. OpenRouter is a hosted service that gives access to hundreds of models through a single API endpoint.
Which LiteLLM versions were compromised?
Versions 1.82.7 and 1.82.8 on PyPI on March 24, 2026, and version 1.107.0 on PyPI plus several Docker tags on October 21, 2026. LiteLLM recommends rotating credentials on any host that ran them.
Does LiteLLM need Redis?
Not for a single instance. The routing docs use Redis in production to share cooldowns and rate-limit usage across proxy instances.
Is LiteLLM safe to use after the supply chain attacks?
The compromised releases were pulled, and LiteLLM moved to a rebuilt release pipeline after March. Pin a known-good version and verify signed images. Then follow the security advisories on its blog.