Offline AI is a product requirement

Almost every AI product I use assumes four things at once: a fast connection, cheap data, a recent device, and a user willing to wait a few seconds for a cloud response. Break any one of those and the interface usually does the same thing, which is spin and then fail.

That is the starting point for AANI, the research project where I keep testing this. The question I care about is not whether a model can answer. It is whether the system still helps when the network drops halfway through the answer.

What “offline” actually has to cover

Offline is not one state. When I started writing down the cases, there were at least five, and each needs a different product answer.

1. Fast and cheap. The normal case every product is built for. Use the cloud model.
2. Slow but present. The request completes, eventually. Streaming matters more than model quality here, because a partial answer arriving in two seconds beats a better answer arriving in twenty.
3. Intermittent. The connection comes and goes mid-session. This one breaks the most code, because a request can succeed, then the follow-up fails, and the app has no idea which half of the conversation the server kept.
4. Metered. The connection works and the user does not want to spend on it. A product that silently burns data on a large context window is making a spending decision for someone else.
5. Absent. No network. The device has to answer from what it already holds, or say clearly that it cannot.

Most products collapse all five into “online” and “error.”

The models that make this possible now

This stopped being theoretical around the time the small open models got genuinely usable. The ones I keep coming back to:

• Gemma 3 1B and 4B from Google. The 1B is the one I reach for first on a mid-range Android phone, since quantised it lands in the few-hundred-megabyte range and still handles extraction and classification.
• Llama 3.2 1B and 3B. Meta built these specifically for on-device summarisation and instruction following, and the 3B is the point where output stops feeling like a toy.
• Qwen2.5 and Qwen3 at 0.5B to 3B. The 0.5B is small enough to be interesting for pure structured output, and the family is unusually good at following a JSON schema for its size.
• Phi-3.5-mini at 3.8B. Trained heavily on reasoning-style data, so it punches above its parameter count on anything that looks like a short logic step.
• SmolLM2 at 135M to 1.7B. Genuinely tiny. Not a chat model in any satisfying sense, but fine for routing and tagging.
• Whisper tiny and base, plus faster-whisper, for speech. Voice input matters more than text input for a lot of the users I have in mind, and this runs locally.

The runtimes matter as much as the weights. llama.cpp with GGUF quantisation is the workhorse, Ollama wraps it for desktop, MLC and ExecuTorch handle mobile, and transformers.js with WebGPU puts a small model inside the browser with no install at all. Quantisation is what makes the arithmetic work: a 3B model at 4-bit sits around 2 GB, which is the difference between shipping and not.

Where small models earn their place

A small model running on the device is not a worse version of a frontier model. It is a different component with a different job. It handles structure well: pulling fields out of text, classifying a request, drafting something short, answering from a local document set. Those are the tasks where a few billion parameters is enough and where the latency win is large, because there is no round trip at all.

Concretely, the jobs I hand to a local model are extracting a date and an amount from a message, deciding whether a question needs the cloud at all, tagging a note, and answering from a small set of documents already on the device. The jobs I do not hand to it are long open-ended writing, anything needing current information, and multi-step reasoning across a large context.

The tasks it does not handle well are the open-ended ones, and that is fine, because those are the ones worth spending a network request on. The design work is deciding which is which before the user hits send, not after.

The pattern I keep coming back to

Route by capability, not by connectivity. Decide what class of task the request is, then pick the cheapest place that can do it. Local first when local is sufficient, cloud when the task needs it and the network allows it, queued when the network cannot.

Queuing is the part I underestimated. If a request is deferred, the user needs to know it was deferred, needs to keep using the app, and needs the answer to arrive somewhere they will find it later. That is closer to how email works than how chat works, and it changes the interface more than the model choice does.

SMS sits at the far end of this. It is slow, it is text only, and it works on hardware that will never run a local model. Designing a flow that fits in a few messages forces a level of clarity that helps the rich version too.

Why this is a requirement and not a nice to have

If offline is treated as a fallback, it gets built last, tested least, and shipped broken. It ends up as an error screen with a retry button.

If it is treated as a requirement, it changes the architecture on day one: what gets stored locally, how state syncs, what a request costs, what the app is allowed to promise. Those are not decisions you retrofit.

The test I use now is simple. Turn the network off in the middle of a session and see what the product does. If the answer is nothing useful, the product only works in the conditions it was demoed in.