DeepSeek DeepInfer: a production inference engine goes open source

On October 20, DeepSeek announced on X that it is open-sourcing DeepInfer, its in-house inference framework. It’s the engine that runs the production DeepSeek-V4.1-Flash service. The code is on GitHub under the Apache 2.0 license, and the thread about it hit number one on Hacker News.

What is DeepSeek DeepInfer?

It’s the engine that takes model weights and turns them into a working service: it splits the work across accelerators, groups requests into batches and generates tokens as fast and as cheaply as possible. The README calls it “the high-performance inference framework that powers DeepSeek’s large-scale production services”, and the released version offers offline inference with the key features needed to serve at scale.

Inside, among other things:

1. DeepSeek’s own kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, libraries the company has already published, plus kernels written in TileLang.
2. Parallelism for an MoE (mixture-of-experts) model: data parallelism for attention and expert parallelism with expert load balancing (EPLB).
3. DSpark, speculative decoding with draft tokens, where the model judges how many of them are worth verifying.
4. SWA bounded replay, which limits prefill recomputation to a short trailing chunk.
5. Deterministic sampling (same input, same output; NVIDIA only for now) and structured outputs with regex, JSON Schema or an EBNF grammar.

Interestingly, the acknowledgements say DeepInfer was “originally built primarily on vLLM”, with inspiration from SGLang and TensorRT-LLM.

Why is it making waves?

First, it’s a 180-degree turn. In spring 2025, DeepSeek explained in its open-infra-index repo that it had considered releasing the whole engine but decided against it: the code was a heavily modified fork of vLLM, tied to internal infrastructure, and the team didn’t have the bandwidth to maintain a large project. Instead, it promised to spin off individual libraries. Now we get the whole engine.

Second, scale. According to a Zhihu Frontier summary of DeepSeek’s technical post on Zhihu, a single cluster serves 155 trillion tokens a day, with a 96% input cache hit rate, an average time to first token of about 400 ms and generation at 230 tokens per second (up to 300 for coding). These are production figures reported by the company itself, so let’s take them with a grain of salt.

Third, hardware flexibility. Two platforms are supported: NVIDIA GPUs with the SM100 (Blackwell) architecture and Huawei Ascend 950. A Chinese accelerator inside the production engine of one of the most talked-about labs is a story in itself, as the HN comments show (from “a nuclear bomb” to questions about motives).

On top of that, DeepSeek published a library called deepseek-recipe which, according to Zhihu Frontier, supports the Chat Completions, Responses and Messages APIs, plus tokenization that matches the official API.

Is there a catch?

A few, and the README lists them openly:

1. Pieces of a full service are missing: a KV cache system, prefill/decode disaggregation onto separate nodes (though most of the logic is already in the engine) and load balancing across replicas.
2. One model: for now, only DeepSeek-V4.1-Flash is supported.
3. Top-shelf hardware: the README benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode. We won’t run this on a home graphics card (and that’s not the point ;)).

So it’s more of a recipe for companies and clouds hosting DeepSeek models than a tool for an enthusiast with one GPU. But now that the production optimizations are public, other projects can adopt them: the LightSeek team wrote on X that their TokenSpeed engine “will learn from and adopt these optimizations”.

How to try DeepInfer?

If we have access to SM100 GPUs or an Ascend 950, all we need is Linux, Python 3.12, GCC 13+ and CUDA 13 (or CANN 9.2 for Ascend). We build the project with ./build.sh, then run python examples/offline_inference.py. On the first run, the script downloads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face.

Without that kind of hardware, the easiest option is to try the model itself, for example DeepSeek V4.1 Flash for free in OpenCode. And if other open models are more our thing, the post on Reflection Beam is worth a look.

FAQ

Is DeepInfer free?

Yes. The code is under the Apache 2.0 license, so it can be used commercially too.

Does DeepInfer work with models other than DeepSeek’s?

Not yet. The README lists only DeepSeek-V4.1-Flash.

Does DeepInfer replace vLLM?

Not quite. It grew out of vLLM, but it’s specialized for DeepSeek models and two hardware platforms, and it still lacks pieces of a full service, such as a KV cache system.