Modal Serverless GPUs: How Modal Works for AI Agents
Modal runs Python functions and containers on cloud GPUs and CPUs, and you manage no servers. Agent builders use it two ways. One is to serve models and batch jobs on GPUs that scale to zero. The other is Sandboxes, isolated containers for code an LLM wrote. Both are ephemeral by design, and that shapes how an agent on Modal keeps state.
What is Modal, in one sentence?
Modal is a serverless cloud that runs containers and GPUs you define in Python code, billed by the second.
Modal's guide calls it an AI infrastructure platform. You pick a GPU type in code and Modal builds the container image. It adds containers as traffic grows. It pools capacity across several clouds. The homepage says it spans 20+ clouds and can burst to thousands of GPUs at peak. Those are Modal's own claims, with no published method.
How does Modal differ from renting a GPU server?
The main differences are billing granularity and who handles scaling.
| Modal | A reserved GPU server | |
|---|---|---|
| Billing | Per second of use, scales to zero | Per hour or month, idle or not |
| H100 price | $0.001097 per second, about $3.95 per hour | Varies by provider and contract |
| Scaling | Automatic, more containers as traffic grows | You add machines yourself |
| Startup | Cold start on each new container | Always on |
| Environment | Container defined in Python code | You manage the OS and drivers |
| State on disk | Gone when the container stops, unless written to a Volume | Stays on the disk |
Sources: prices from Modal pricing (the hourly figure is our math from the per-second rate), behavior from Modal's guide.
Modal fits bursty work. Picture an agent that calls a model a few hundred times an hour, then goes quiet. In our view, a GPU that stays busy around the clock is often cheaper as a reserved machine.
What does Modal cost?
The pricing page lists three plans. Starter is $0 a month plus compute, with $30 of free credits each month. Team is $250 a month plus compute, with $100 of credits. Enterprise is custom.
Selected GPU prices per second, with our hourly conversion:
- B200: $0.001736 (about $6.25 per hour)
- H100: $0.001097 (about $3.95 per hour)
- A100 80 GB: $0.000694 (about $2.50 per hour)
- L4: $0.000222 (about $0.80 per hour)
CPU is $0.0000131 per physical core per second, and one physical core equals two vCPUs. Memory is $0.00000222 per GiB per second. Sandboxes pay three times the standard CPU and memory rates, because they are non-preemptible. Their GPU prices match the standard ones.
How does Modal make cold starts fast?
Every new container has to import its libraries and load its model before it can serve. Modal's main tool for this is the memory snapshot. Modal saves a container's memory after setup. New containers restore that saved state and skip the repeated work. The docs note that importing torch alone involves over 20,000 file operations.
Modal says initialization-heavy functions often start 3 to 10 times faster from a snapshot. In its sentence-transformers example, startup fell from about 6 seconds to about 2. CPU snapshots work only for deployed apps. GPU snapshots, which also save GPU state, are an alpha feature. They do not speed up loading model weights from storage, and they generally do not work with multi-GPU code.
For Sandboxes, the homepage claims sub-second starts on large custom images. That is a vendor claim. Time your own image.
How do agents use Modal Sandboxes?
A Sandbox is a secure container for running untrusted code, such as code an LLM generated. A coding agent can write a script, run it in a Sandbox and read the output. Your own machine never runs the code. Modal's homepage lists Cognition among its customers. Cognition's CEO is quoted there as running millions of sandboxes on Modal.
Sandboxes are short-lived by default. The default maximum lifetime is 5 minutes, and you can raise the timeout to 24 hours. An idle timeout can end one sooner. For longer work, Modal recommends taking a filesystem snapshot and restoring it in a new Sandbox.
What survives when a Modal container stops?
Only what you write somewhere durable. Modal gives you three places:
- Volumes are a distributed file system built for data you write once and read many times, such as model weights. Writes stay private to a container until they are committed. Modal commits in the background every few seconds and once more on shutdown. Other containers see the changes only after they reload the Volume.
- Dicts are a distributed key-value store that survives redeploys. Each entry expires after 7 days with no reads or writes, and objects can be up to 100 MiB.
- Filesystem snapshots save a Sandbox's disk so a later Sandbox can pick up where it stopped.
All three store bytes. Choosing what an agent should remember, in a form the next run can use, is your code's job. Say an agent finds that one image tag breaks a CUDA build. That fact stays in the run's logs unless your code writes it down.
The bottom line
Use Modal when your GPU load is bursty or when an agent needs a safe place to run code it wrote. Turn on memory snapshots for heavy imports, and measure cold starts on your own image before you trust a latency number. Put anything an agent must keep in a Volume, and design what goes there on purpose.
How do you keep what an agent learned after its container is gone?
On Modal, a container's state is lost when it stops unless you save it. A Sandbox lives 5 minutes unless you extend it, and its disk is gone unless you commit or snapshot it. Volumes and Dicts keep whatever your code writes, so your code has to choose what that is. The same gap shows up on the developer's side. The coding agent that wrote your Modal app learns which GPU fit and which image settings failed. Next session, it may pick the same failing image tag again.
memU keeps that kind of knowledge as readable Markdown for coding agents such as Claude Code and Codex. A scheduled background task slices new session logs into jobs. The agent distills each job into memory and skill Markdown, and memU commits it. memU makes no LLM or chat calls itself. It stores, embeds and retrieves the Markdown the agent prepared. A standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before answering. All hosts share one backend, so what one host learned, another can retrieve. memU does not plug into Modal containers directly. Check the support matrix in the memU repository for where it runs.
The hosted version is free and syncs across devices. Get an API key from memu.so and send your agent one message:
Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU, API Key is
YOUR_MEMU_API_KEY.
FAQ
Is Modal free?
The Starter plan costs $0 a month and includes $30 of free compute credits each month. Beyond that you pay per second for the resources you use.
How much does an H100 cost on Modal?
Modal lists the H100 at $0.001097 per second, which works out to about $3.95 per hour. You pay only while a container runs.
How long can a Modal Sandbox run?
A Sandbox runs for up to 5 minutes by default. You can raise the timeout to 24 hours. For longer jobs, Modal recommends restoring a filesystem snapshot into a new Sandbox.
How do I persist data on Modal?
Write it to a Volume, which persists files across containers once committed, or to a Dict for key-value data. Dict entries expire after 7 days without reads or writes.
How fast are Modal cold starts?
It depends on your image and imports. Modal says memory snapshots often make initialization-heavy functions start 3 to 10 times faster. Its own example dropped from about 6 seconds to about 2.