On-Device AI Explained: Local Models and Memory

On-device AI means the model runs on the phone or laptop in front of you, with no server call for each prompt. Apple and Google ship it in their phone platforms, and the open-source Ollama project brings it to laptops. The pitch is privacy and offline use, with no per-token bill. The catch is size: Apple documents a 4,096-token context window per session for its on-device model, a few pages of text. Below we compare the main options and show how to give a small model memory that outlasts one session.

What is on-device AI, in one sentence?

On-device AI is running a language model directly on a user's hardware, so prompts and outputs are processed locally without a network round trip.

Google's Gemini Nano page puts the benefits plainly. On-device inference "executes prompts locally, eliminating server calls," keeps sensitive data on the device and works offline. The same page warns about token limits and the boundaries of device hardware.

How do on-device models differ from cloud models?

The sharpest difference is the context window. It sets how much of a conversation fits in one call.

Sources: Apple's 2025 foundation models report and context window technote. The 2026 figures come from WWDC26 session 241. The rest is from Google's Gemini Nano page and the Ollama FAQ.

How does on-device AI work on Apple devices?

Apple's on-device model is about 3 billion parameters. Its 2025 technical report explains how it fits on a phone. Decoder weights are compressed to 2 bits per weight with quantization-aware training. The model also shares key-value caches between two blocks, which Apple says cuts KV cache memory by 37.5%. Apple describes the model as built for tasks like summarization and entity extraction, and says it is not meant as a general-knowledge chatbot.

Developers reach it through the Foundation Models framework in Swift. Mark a Swift type with a macro and the output matches it. The model can also call tools you define. Apple calls the inference "free of cost" to app developers.

At WWDC26, Apple said the on-device model was "rebuilt from the ground up," with better logic and tool calling. Session 241 also said iOS 26.4 added APIs to read the model's context size and count tokens. Apple tells developers to use them "to adapt your app to the hardware it's running on." The session's code sample printed a context size of 8,192, so the number now depends on the device. Code against the API, and assume the smaller window from the technote on older hardware.

For heavier work, the same session introduced developer access to the Private Cloud Compute model. It has a 32,000-token context window and reasoning. It needs no API keys and has no cloud API cost for developers under 2 million first-time downloads.

How does on-device AI work on Android and laptops?

On Android, Gemini Nano runs inside AICore, a system service that keeps the model updated. Apps call it through ML Kit's GenAI APIs. Those cover summarization, proofreading, rewriting, image description and speech recognition, plus a general prompt API. Google says AICore keeps no record of inputs or outputs after processing. For unsupported devices or harder queries, Google suggests a hybrid setup that falls back to Gemini models in the cloud.

On a laptop, Ollama is the common way to run open-weight models. Its FAQ says Ollama does not see your prompts when you run locally. The default context is 4,096 tokens, and you can raise it with the num_ctx option or an environment variable. A larger window costs memory, so the practical ceiling is your RAM or VRAM. Ollama also offers cloud-hosted models. Those send prompts to Ollama's service, so check which kind you are calling.

Why do small context windows need memory?

Every token in a session counts against the window. Apple's technote lists what goes in: instructions, every prompt, tool schemas and their outputs, and every model response. In a multi-turn chat, a 4,096-token window fills fast.

Apple's advice is to split big tasks, run each step in a new session and pass forward only a summary. In effect, the app keeps the memory: it decides what to carry into the next session and how few tokens to spend on it.

The same applies on a laptop. A local model that forgets your project at the end of each session is fine for one-off questions. For anything ongoing, you need a store outside the model and a way to retrieve only the relevant pieces. Small windows make retrieval quality matter more, because there is no room to paste everything.

What should you read in the fine print?

Most performance claims here come from the vendors. Apple's report grades the model on Apple's own evaluations. Google publishes no context size or device list on its page. We found no independent benchmark that puts these on-device options through the same tasks.

Privacy claims cover the model call only. If your app sends the output to a server or an analytics tool, that data leaves the device. Hybrid setups route some prompts to the cloud by design. Log which requests leave the device before you call the app private.

The bottom line

Use on-device models for short, private tasks such as summaries, where offline use and zero inference cost matter. Plan for a 4,096-token window unless the device reports more, and keep long-term state outside the model, retrieved per request. Fall back to a cloud model for long-context or reasoning-heavy work.

Can agent memory stay local too?

Small local models make the memory problem obvious. Nothing survives the session unless something outside the model stores it. A few thousand tokens leave no room to replay history. Coding agents on your laptop hit the same wall. Each session starts cold unless project knowledge is written down where the agent can search it.

memU stores that knowledge as a shared wiki of readable Markdown. A scheduled background task slices new session logs into jobs. The agent decides what is worth keeping and writes it as memory or skill Markdown. memU's memory service makes no LLM or chat calls. It stores and embeds the Markdown the agent prepared, then retrieves it.

A standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before answering. memU works with coding agents such as Claude Code and Codex. Apps built on Apple's or Google's on-device models are outside its scope. There are two ways to run it. The self-hosted version is private and single-device, and it needs an embedding key. If you want a fully offline setup, check where your embedding provider runs. The hosted version on memu.so is free and syncs across devices. The memU GitHub repository lists supported agents. Install the self-hosted version with one message to your agent:

Read https://raw.githubusercontent.com/NevaMind-AI/MemU/main/SKILL.md and follow it to install memU.

FAQ

What is the context window of Apple's on-device model?

Apple's technote gives 4,096 tokens per session. Since iOS 26.4 apps can read the actual size from the API. Apple's WWDC26 sample showed 8,192 on newer hardware.

Is on-device AI free?

There is no per-token bill. Apple calls the inference free to developers, and Ollama runs on your own hardware. You pay in battery life and device requirements.

Does Gemini Nano work offline?

Yes. Google says on-device inference with Gemini Nano runs prompts locally and enables offline use. Hybrid setups that fall back to cloud Gemini models need a connection.

What is the default context length in Ollama?

Ollama's FAQ gives a default of 4,096 tokens. You can change it with the num_ctx option per request or the OLLAMA_CONTEXT_LENGTH environment variable for the server.

How do local models remember past conversations?

They do not, on their own. Your app or agent has to store what matters outside the model and retrieve the relevant parts into each new session.