DeepSeek Open-Sourced Its Inference Engine and Showed the Math: $38 per Chip-Hour

DeepSeek Open-Sourced Its Inference Engine and Showed the Math: $38 per Chip-Hour

6 min readOctober 22, 2026

Quick verdict

DeepSeek open-sourced DeepInfer, the inference engine that serves its V4.1 Flash model, under Apache 2.0. The code matters less than the engineering report that came with it: one cluster handling 155 trillion tokens a day, a 96% input-cache hit rate, and roughly $38 in revenue per accelerator-hour at off-peak prices. People ran the arithmetic and landed on inference margins somewhere north of 80%. That number is real but soft, because almost everything expensive about running a production service sits outside it, and because the cache is carrying far more of the load than the headline suggests.

What actually shipped

DeepSeek put out two repositories. DeepInfer is the high-performance inference framework behind V4.1 Flash, with speculative decoding, mixture-of-experts expert parallelism, SWA bounded replay, structured output, and async scheduling. It targets NVIDIA SM100 and Huawei Ascend 950 hardware. The second, deepseek-recipe, is a Rust library that handles API protocol conversion across Chat Completions, Responses, and Messages, plus the token encoding that production serving needs.

  • 155 trillion tokens per day from a single cluster
  • 96% input-cache hit rate, 400ms average time to first token
  • 230 tokens per second generation, rising to 300 for coding workloads
  • $38 revenue per accelerator-hour at off-peak pricing

These are first-party production claims, not independently audited measurements, and DeepSeek admits the release is not the whole service. The KV-cache management and load-balancing layers, the parts most tangled up with internal infrastructure, are not fully out. So "fully open source" describes the engine, not a push-button copy of DeepSeek's serving stack.

Why the $38 number set off a profitability fight

Revenue per accelerator-hour is the kind of figure the industry almost never discloses, so once it landed, people started estimating margins out loud. One widely shared model put inference margins at 84.2% to 96.1% after its author corrected an early overstatement. Using a higher $7 hardware cost against the same $38 revenue, Yuchen Jin got about 81.6%, while noting DeepSeek never said which chip the revenue figure was tied to. At $5 to $10 an hour, the accelerator-only math spans roughly 74% to 87%.

Notice what every one of those estimates leaves out. They count the accelerator and nothing else: not the 100-petabyte SSD cache fleet, not the CPUs and networking, not idle and reserve capacity, not failures, not engineering or support. Training spend and future compute commitments are a separate line again. A business can post gorgeous contribution margins on inference and still lose money at the company level, which is exactly the shape the frontier labs are in right now. For the bigger picture on where token prices are heading, our read on the future of AI pricing walks through the forces pulling in both directions.

The cache is doing most of the work

That 96% input-cache hit rate is not a footnote, it is the story. When 96% of input tokens are served from cache, "155 trillion tokens a day" stops meaning 155 trillion freshly computed tokens. Cache hits and newly generated output cost wildly different amounts, so the throughput headline and the revenue figure both lean hard on a token mix that happens to be very cache-friendly. DeepSeek keeps persistent prefixes in a separate storage cluster, three days for agent traffic and seven for chat, which is why agents that resend a growing pile of context keep hitting the cache.

The timing is not a coincidence either. In the same window, OpenAI announced 12-hour API prompt caching, up from a 30-minute default, aimed at the same thing: persistent agents that reuse context across long gaps. Two different companies, same bet, that the cheapest token is the one you already computed. If you are weighing where the savings actually show up for your own usage, our breakdown of what AI subscriptions really cost is a useful companion.

vLLM already caught up to within 6%

The claim that DeepSeek has some untouchable serving moat did not survive the week. The vLLM team reported decode throughput within 6% of DeepInfer's published numbers on a matched GB200 configuration, published reproduction recipes, and pointed out that DeepInfer was built on vLLM in the first place and released under Apache 2.0 with credit. That is decode-stage parity, not a clone of the entire service, and it is still self-reported by a team with its own engine to promote. But it reframes the release as production know-how flowing back into an existing open ecosystem rather than a secret weapon.

Tim Dettmers judged DeepSeek's inference stack comparable to OpenAI's and ahead of Anthropic's at this scale, though without a matched cross-lab benchmark to back it. The honest read is that the efficiency is impressive and mostly reproducible, not magic. For how the three big labs stack up on strategy and price, see our OpenAI vs Anthropic vs Google comparison.

Video: the economics of enterprise AI

A plain-language walkthrough of why inference, not training, is where the ongoing cost lives, and how companies actually model it.

Why it matters

If serving a frontier-class open model really does clear 80% contribution margins, two things follow for anyone paying for AI. Token prices have more room to fall than the "AI is a money pit" story suggests, because a provider with an openly deployable stack can undercut rivals and still make money on each request. And the competitive pressure moves to who has the cheapest cache-heavy traffic and the best hardware deal, not who guards the fanciest serving code. That pressure is already showing up in falling per-token prices and the on-again-off-again talk of a compute glut, which we covered in the GPU bubble and what it does to AI pricing. For readers, the practical move is the same as ever: do not lock into one provider while the economics are this unsettled.

FAQ

Does DeepInfer prove AI inference is profitable?

It is strong evidence that the per-request math can be very favorable, with accelerator-only margin estimates mostly between 74% and 96%. It does not prove company-level profitability, because those estimates ignore storage, networking, idle capacity, support, and training. High inference margins and large company losses can coexist.

What is V4.1 Flash and how does this connect to it?

V4.1 Flash is DeepSeek's cheap, fast open-weight model, and DeepInfer is the engine that serves it in production. We covered the model itself in our DeepSeek V4.1 Flash launch breakdown.

Can I run DeepInfer myself?

Partly. The engine is open under Apache 2.0, but the KV-cache management and load-balancing components that make the full service work are not fully released, so reproducing DeepSeek's exact numbers takes infrastructure you still have to build.

Sources

Further reading

Try all the models mentioned in this article

Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Try it for 7 days for $1.99.

Try Admix for $1.99

Related articles