Groq LPU: How It Works, Speed, and Pros and Cons
The Groq LPU is a chip built only for running language models, and GroqCloud sells access to it as an API. Its pitch is speed: GroqCloud lists gpt-oss-20b at 1,000 tokens per second. That speed comes from a design that keeps model weights in on-chip SRAM and plans every step at compile time. Below we explain how the LPU works and what GroqCloud charges today, then compare it with Cerebras. Then come the trade-offs for enterprise teams and what changed after NVIDIA licensed Groq's technology in December 2025.
What is the Groq LPU, in one sentence?
The Groq LPU (Language Processing Unit) is an inference chip that keeps weights in on-chip SRAM and lets a compiler schedule all work in advance.
Groq's own explainer of the LPU names four design principles. The two that explain the speed are deterministic compute and on-chip memory. The company says it did not start chip design until the compiler's architecture was settled. The compiler is model-independent, so one compiler handles each new model without hand-written kernels.
How the Groq LPU differs from a GPU
| Groq LPU (first generation) | Typical inference GPU | |
|---|---|---|
| Main memory for weights | On-chip SRAM, 230 MB per chip | Off-chip HBM |
| Memory bandwidth (Groq's figures) | Up to 80 TB/s on die | About 8 TB/s for HBM |
| Scheduling | Static, fixed by the compiler | Dynamic, decided at run time |
| Chip-to-chip links | Direct, no routers or controllers | Not specified by Groq |
| Speed (Groq's claim) | Up to 10x faster | Baseline |
The figures in the LPU column come from Groq's GroqChip spec sheet and its explainer. The HBM bandwidth figure is Groq's general comparison and names no specific GPU.
The catch is memory size. 230 MB of SRAM is very fast but very small, so a large model must be split across many chips. NVIDIA's Vera Rubin engineering blog says cross-chip exchange is unavoidable in any SRAM-based architecture that cannot hold the model on one chip. Groq's answer is fast direct links and a compiler that plans every transfer.
How fast is Groq, in tokens per second?
GroqCloud's models page lists these speeds and prices for its self-serve models in October 2026:
| Model | Speed (Groq's figure) | Input / output per 1M tokens |
|---|---|---|
| gpt-oss-20b | 1,000 tokens/s | $0.075 / $0.30 |
| gpt-oss-120b | 500 tokens/s | $0.15 / $0.60 |
| qwen3.8-27b (preview) | 450 tokens/s | $0.80 / $4.00 |
Both gpt-oss models have a 131,072-token context window.
Those are vendor numbers. Artificial Analysis tests providers with a 10,000-token input workload. It reports a median of 470 tokens per second for gpt-oss-120b on Groq, with a time to first chunk of 0.66 seconds.
Llama 3.1 8B (560 tokens per second) and Llama 3.3 70B (280) are still listed for enterprise contracts only. The deprecations page shows they left the free and developer tiers on August 16, 2026.
How does Groq compare with Cerebras?
Cerebras goes further with the same idea. Its wafer-scale CS-3 system puts 44 GB of SRAM on a single wafer, with 21 PB/s of memory bandwidth. Cerebras's model docs list gpt-oss-120b at about 3,000 tokens per second. It costs $0.35 per million input tokens and $0.75 per million output tokens.
In the Artificial Analysis comparison on gpt-oss-120b, Cerebras measured 1,767 tokens per second to Groq's 470, and its first chunk arrived in 0.38 seconds versus 0.66. Groq is cheaper on that model, at $0.15 input and $0.60 output per million tokens. Pick Cerebras if speed on that model matters most, and Groq if price does. Both vendors serve only a short list of self-serve models. Cerebras says more model families are available through dedicated deployments.
What are the pros and cons of Groq for enterprise inference?
The pros are speed you can predict and low prices on open-weight models. Deterministic scheduling means latency varies little from request to request, which matters for voice agents and long tool-calling loops. GroqCloud also offers a batch API at a 50% discount. Prompt caching on the gpt-oss models gives a 50% discount on cached input, and cached tokens do not count toward rate limits. The two discounts do not stack.
The cons start with the catalog. Self-serve customers get four text models: the three in the table plus gpt-oss-safeguard-20b. None are closed frontier models. Groq says preview models can be discontinued at short notice. In 2026 it removed several models, from Kimi K2 in April to Qwen3-32B in July. Compound, Groq's built-in agentic system, was retired on September 21, 2026 with no replacement listed. The gpt-oss models do not support parallel tool calls on GroqCloud, per the tool use docs.
Our view: Groq fits latency-sensitive workloads that run well on open-weight models. Teams that need a frontier closed model, or a long-lived model choice, should plan a second provider.
What happened between Groq and NVIDIA?
On December 24, 2025, Groq announced a non-exclusive license of its inference technology to NVIDIA. Founder Jonathan Ross and president Sunny Madra joined NVIDIA. Groq said GroqCloud would continue to operate without interruption. Neither company published a deal value.
NVIDIA then built the technology into its own product. On August 24, 2026 it said the NVIDIA Groq 3 LPX rack was in full production. Per NVIDIA's LPX page, each next-generation LPU has 500 MB of SRAM and 150 TB/s of SRAM bandwidth, and a rack holds 256 of them. In that design, Rubin GPUs handle attention over the KV cache and the LPUs run the feed-forward layers. Groq raised $650 million in June 2026 under CEO Adam Winter. In August it joined the NVIDIA Cloud Partner program, and it says it will be among the first to deploy LPX. Its own platform page still describes LPX racks as arriving when released, so LPX is not yet part of GroqCloud.
Read the fine print
Most speed numbers here are vendor claims. Groq's 500 tokens per second on gpt-oss-120b holds up in Artificial Analysis's test (470). Cerebras's 3,000 does not fully hold up (1,767 in the same test), possibly because of the 10,000-token input workload. NVIDIA's widely quoted 3,400 tokens per second is for Gemma 4 31B on Vera Rubin plus LPX, a system GroqCloud does not run. Its claim of being 4x faster than the nearest alternative does not name that alternative.
Rate limits deserve a check before you commit. GroqCloud's models page and its rate-limits page list different developer-tier limits for gpt-oss-120b. Confirm the numbers on your own account.
The bottom line
The Groq LPU is fast because it trades memory capacity for bandwidth and moves scheduling into the compiler. GroqCloud is a good fit for latency-bound agents on gpt-oss or Qwen at low prices. Test it with your real prompt sizes, and keep a fallback provider for models Groq does not serve or later retires.
Does faster inference help an agent remember?
No. A fast model is still stateless. Every GroqCloud request carries its full context, and the prompt cache only reuses an identical prefix. Cached prefixes expire after two hours without use. A fast agent loop still forgets the project facts it worked out yesterday. Something has to write them down and feed them back.
memU handles that part. The agent decides what is worth keeping. It writes that into a shared wiki of readable Markdown files, as memory or as reusable skills. memU makes no LLM calls of its own. It stores and embeds what the agent wrote, then returns it when asked. A standing instruction in the host's instruction file, such as CLAUDE.md or AGENTS.md, tells the agent to retrieve before answering. The model provider underneath can change without losing what the agent learned. The reasoning behind this design is in why memU leaves the thinking to the agent.
To try the hosted version, get an API key from memu.so and send your agent one message:
Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU, API Key is
YOUR_MEMU_API_KEY.
Supported agents and the self-hosted setup are listed in the memU GitHub repository.
FAQ
What does LPU stand for in Groq?
Language Processing Unit. It is Groq's inference chip. It keeps weights in on-chip SRAM, and a compiler schedules all work ahead of time.
How many tokens per second does Groq generate?
GroqCloud lists 1,000 tokens per second for gpt-oss-20b and 500 for gpt-oss-120b. Artificial Analysis measured 470 tokens per second for gpt-oss-120b on Groq with a 10,000-token input.
How much SRAM does a Groq chip have?
The first-generation GroqChip has 230 MB of on-chip SRAM with up to 80 TB/s of bandwidth. NVIDIA lists 500 MB per chip for the next-generation LPU in its Groq 3 LPX rack.
Is Groq faster than Cerebras?
Not on gpt-oss-120b. Artificial Analysis measured Cerebras at 1,767 tokens per second and Groq at 470. Groq charges less for that model, $0.15 against $0.35 per million input tokens.
Did NVIDIA buy Groq?
No. In December 2025 Groq licensed its inference technology to NVIDIA on a non-exclusive basis, and some of its leaders joined NVIDIA. Groq still runs GroqCloud as a separate company.