Gemma 4 Now Runs on a Phone: Google's QAT Release Makes Local AI Actually Usable

Gemma 4 Now Runs on a Phone: Google's QAT Release Makes Local AI Actually Usable

6 min readJune 8, 2026

Quick verdict

Google released quantization-aware training checkpoints for the whole Gemma 4 family, and this is the version that matters if you want to stop paying a monthly AI bill. The point of QAT is that the model is trained to expect low precision, so the 4-bit version keeps most of the quality instead of falling apart the way naive post-training quantization usually does. Google says the smallest checkpoint fits in roughly 1GB, the E4B runs on a 6GB laptop, and there is a dedicated mobile format. Ollama and vLLM both supported it the same day.

What actually shipped

Google published official q4_0 GGUF checkpoints across the lineup: E2B, E4B, 12B, 26B-A4B, and 31B, plus 2-bit and 4-bit variants and a separate mobile target. The idea behind quantization-aware training is simple to state. Instead of training at full precision and then squashing the weights down afterward, the model sees the quantization during training and adapts to it. The result is a 4-bit model that behaves much closer to the full bf16 version than a standard squeeze-down would.

  • The E2B checkpoint is claimed to run in about 1GB of memory, which puts a usable model on hardware most people already own.
  • The E4B QAT release is small enough to load on a 6GB VRAM laptop, the kind of machine that could not touch the full model a year ago.
  • There is a mobile quantization format aimed at on-device inference, so phone deployment is an explicit target, not a hack.
  • Ollama and vLLM shipped support immediately, so you can run it through tooling you may already have installed.

One real caveat surfaced fast. Daniel Han of Unsloth pointed out that converting the QAT weights straight into llama.cpp's q4_0 lattice loses accuracy, because the lattice does not line up with how the model was trained. Unsloth's dynamic GGUF builds recover most of that loss. So the checkpoint you pick matters: a careless conversion throws away the exact thing QAT was supposed to buy you.

The other honest gap is benchmarks. Google's release blog does not publish a direct q4 versus bf16 quality comparison, and that was the first thing the local-model community asked for. The claim that QAT preserves capability is reasonable and matches how the technique works, but right now you are taking some of it on trust until independent numbers land.

Why it matters

For a long time, "run AI locally" meant accepting a real quality drop or buying a workstation GPU. QAT changes the math on that. A model that runs in 1GB to 6GB and keeps most of its quality is the difference between local AI being a hobby and being a replacement for a paid tier on everyday tasks. If a 6GB laptop can run a capable assistant offline, the case for a second or third monthly subscription gets weaker for anyone whose work is mostly drafting, summarizing, and quick coding help.

This does not retire the frontier models. A 4-bit Gemma 4 on a laptop is not GPT-5.5 or Opus 4.8, and the reliability research this week was a reminder that even the frontier labs are not as dependable as the marketing suggests. The win is a different one: a free, private, offline floor that is now high enough to handle a lot of real work, which is exactly the calculation worth making before you renew anything. If you are weighing what to keep paying for, our breakdown of free versus paid AI and the wider open-source model roundup for 2026 are the place to start.

Video: running Gemma 4 QAT locally

A walkthrough of loading the QAT checkpoints and running them on local hardware, including the tooling setup.

FAQ

What is quantization-aware training, in plain terms?

The model is trained while already seeing the low-precision format it will run in, so it adapts to 4-bit instead of being crushed into it afterward. That is why the QAT q4 version holds quality better than a normal post-training squeeze.

Can I really run Gemma 4 on my phone or an old laptop?

The smallest checkpoint is claimed to fit in about 1GB and the E4B targets a 6GB laptop, with a dedicated mobile format on top. Bigger sizes like the 12B and 31B still want a real GPU, so pick the checkpoint that matches your hardware.

Should I just use this instead of paying for ChatGPT or Claude?

For drafting, summarizing, and light coding, a local model can cover a surprising amount and costs nothing per month. For frontier reasoning you will still want a paid model some of the time. The smarter move is usually one shared layer over many models rather than stacked subscriptions, which we cover in how to save money on AI subscriptions.

Sources

Further reading

Try all the models mentioned in this article

Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Free to start.

Start free on Admix

Related articles