DiffusionGemma Generates Text 4x Faster by Refusing to Write One Word at a Time

DiffusionGemma Generates Text 4x Faster by Refusing to Write One Word at a Time

6 min readJune 13, 2026

Quick verdict

Google released DiffusionGemma, an open-weight text model that throws out the usual one-word-at-a-time approach. Instead of predicting the next token and then the next, it refines whole blocks of 256 tokens at once, the way an image model denoises a picture. The payoff is raw speed: around 1,000 tokens per second on an H100 and 700-plus on a consumer RTX 5090, roughly four times faster than a comparable Gemma 4. The catch is quality. It is not as sharp as the standard autoregressive Gemma, so it lands as a fast tool for the right jobs rather than a default for everything.

What actually shipped

DiffusionGemma is a 26B mixture-of-experts model with about 3.8B active parameters, built on Gemma 4 and the earlier Gemini Diffusion research. Google put it out under Apache 2.0 with open weights, so you can run it yourself. It takes text, image, and video input, has a 256K context window, a reasoning mode, and JSON and function calling.

The architecture is the story. A normal language model writes left to right, committing to each token before it predicts the next. DiffusionGemma starts with a noisy block of 256 tokens and refines the whole block in parallel over several denoising steps, then commits that block and moves to the next one with block-autoregressive KV-cache. Because the work is parallel compute rather than a long sequential chain, it maps better onto GPUs that have plenty of math throughput but limited memory bandwidth, which describes most consumer cards.

The speed numbers held up across the people who tested it:

SetupReported throughput
Single H1001,000+ tokens/sec
RTX 5090 (consumer)700+ tokens/sec
vLLM, batch size 1, H200 FP81,200+ tokens/sec
vs. comparable Gemma 4~4x faster

Tooling showed up fast. The vLLM team said DiffusionGemma is the first diffusion language model with native vLLM support. Unsloth shipped GGUFs that run the model locally on roughly 18GB-class hardware, and Daniel Han showed it running through llama.cpp. NVIDIA published an NVFP4 4-bit quantized build for Hopper and Blackwell cards. Demis Hassabis flagged the 4x speedup himself, and Omar Sanseviero joked that the demos had to be slowed down so viewers could follow them.

The trade you are actually making

Every report came with the same asterisk: output quality sits below standard Gemma 4. One commenter put it plainly, that ultra-fast is no good if the model is not smart enough to trust. So the useful question is not whether DiffusionGemma beats a frontier model on a reasoning benchmark, because it does not. It is where 1,000 tokens per second changes what you can build.

A few spots stand out. Latency-sensitive work like quick local search, draft generation, and context compression benefits from speed more than from the last few points of accuracy. Code infilling is a natural fit because the model attends to text on both sides of a gap, so filling in the middle does not need special tokens. And agent loops can use a fast diffusion model to spit out candidate actions while a slower, more careful model verifies them, all inside one latency budget. That candidate-and-check pattern is where a lot of people expect diffusion text models to earn their keep.

Why it matters

Two things make this more than a one-day speed demo. First, Google shipped fundamental architecture work with open weights instead of hiding it behind an API. After a stretch where most labs productized rather than published, a major lab releasing a non-autoregressive text model under Apache 2.0 is a real signal for anyone building on open models. We track that wider shift in the best open-source AI models in 2026.

Second, the hardware math is friendly to people who do not own datacenter GPUs. Local autoregressive inference is usually held back by memory bandwidth, since the weights get streamed for every single token. Shifting that work to parallel refinement uses the compute that consumer cards already have, which is the same reason quantized local models keep getting more practical. If you are weighing local against hosted, our take sits in running Gemma 4 locally with QAT and the best free AI coding agents.

For most people, the practical move is not to replace your main model with DiffusionGemma. It is to treat speed as one more axis you can route on, fast model here, careful model there, rather than paying a premium for one model to do every job. That is the same argument we make in one subscription for all AI models.

Video: how text diffusion models work

A short visual explainer on why refining blocks in parallel can be so much faster than decoding one token at a time.

FAQ

What makes DiffusionGemma different from regular Gemma?

Regular Gemma is autoregressive, generating one token at a time from left to right. DiffusionGemma refines a 256-token block in parallel through several denoising steps, then moves to the next block. That parallel approach is what gets it to 1,000-plus tokens per second, at the cost of some output quality.

Can I run DiffusionGemma on my own machine?

Yes. It is Apache 2.0 with open weights, and Unsloth has GGUF builds that run on roughly 18GB-class hardware. It also has native support in vLLM and an NVFP4 quantized version from NVIDIA for newer cards. Note that llama.cpp needs the DiffusionGemma-specific branch because of the block-diffusion architecture.

Is it good enough to replace my main coding model?

Not yet. Quality trails standard Gemma 4 and frontier models, so it works best for latency-sensitive tasks, code infilling, and as a fast candidate generator paired with a stronger verifier. For everyday coding, see the best AI models for coding in 2026.

Sources

Further reading

Try all the models mentioned in this article

Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Free to start.

Start free on Admix

Related articles