DeepSeek open-sources DeepInfer, the engine that serves its V4.1 Flash model

What DeepSeek released

DeepInfer is the software DeepSeek uses in production to run its AI models for users, now published on GitHub under the Apache 2.0 licence.¹² It supports one model for now, DeepSeek-V4.1-Flash, and DeepSeek says future models will be added.¹² The project is built mainly on vLLM, with ideas from SGLang and Nvidia’s TensorRT-LLM.¹²

What is inside

• Speculative decoding: a small draft model proposes several tokens at once and the main model checks them in one pass.¹²
• Faster prompt processing: a technique that, DeepSeek says, nearly halves prefill computation.¹
• Splitting work across chips, plus features for consistent outputs and fixed output formats.²
• deepseek-recipe: a companion library that converts text to tokens and back, matching DeepSeek’s API.¹²

What was left out

DeepSeek did not release its cache management, separation of prompt processing from answer generation, or load-balancing code, which it says is tightly tied to internal systems.¹²

Hardware

It runs on Nvidia SM100 (Blackwell) data-centre GPUs and Huawei Ascend 950 chips.¹² Older Nvidia architectures such as Hopper are not listed.¹

Performance

DeepSeek reports a 4% to 7% throughput gain at production load from its decoding changes.¹ SiliconANGLE reports DeepSeek disclosed revenue of $38 per GPU per hour at off-peak prices.¹ These are DeepSeek’s own figures; no independent tests had been published.² By the afternoon of release, the repository had more than 8,000 GitHub stars, TNW reports.²

What it means

Most open-source AI releases share model weights. Sharing the serving engine is less common and very practical: it shows how a lab that sells cheap API access keeps its costs down. Cloud providers and companies running DeepSeek models on their own servers can now use the same methods. The vLLM project said it would apply ideas from DeepInfer.²

Support for Huawei’s Ascend chips is notable too, giving teams a tested option outside Nvidia hardware. For Indian companies hosting open models on their own GPUs, the parts that matter most for large-scale serving, caching and load balancing, still need other tools.

Separately, the US recently ordered federal civilian agencies to remove DeepSeek products; that order does not apply to private companies.

What to watch

• Independent benchmarks of DeepInfer against vLLM and SGLang.
• Whether DeepSeek adds support for more models or Nvidia Hopper GPUs.

Related: US Homeland Security orders DeepSeek removed from federal civilian systems, Google’s TorchTPU and DeepSeek’s funding round.