DeepSeek Open-Sources DeepInfer: Production Inference Stack Launches on NVIDIA and Ascend Alike

On October 20, 2026, DeepSeek announced on its official X account — in a four-post thread — the open-sourcing of DeepInfer, the inference framework behind its large-scale production services. The README sets the tone in its very first sentence:

"DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale."

The deepseek-ai/DeepInfer repository is released under the Apache-2.0 license: anyone can use it, modify it, and sell products built on it. The announcement shot straight to #1 on Hacker News (356 points, per our scout's same-day snapshot). DeepSeek's own announcement post reads:

"🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable."

But what makes this worth a full article isn't "yet another open-source framework" — it's the reversal. Eighteen months ago, DeepSeek said on the record: the full engine stays closed. And now the README's opening line is "powers DeepSeek's large-scale production services" — this isn't "we built a decent framework," this is "we published the blueprints of the money printer." Three pieces still separate the blueprints from the money printer itself (more on that below), but the blueprints alone were enough to send the infra world cloning overnight.

The "no" of 18 months ago, and today's open source

In April 2025, DeepSeek published a note titled The Path to Open-Sourcing the DeepSeek Inference Engine. It was remarkably candid: the team had considered releasing its full inference engine but decided against it. Three reasons: the code was a heavily customized early fork of vLLM; it was "tightly coupled" with internal cluster tooling, and disentangling it would be painful; and the small team simply lacked the bandwidth to maintain a large open-source project. The promise at the time: no full engine, but design improvements and selected components contributed upstream as standalone libraries.

Eighteen months later, the full engine is on GitHub. The repository was created on October 16, 2026, and by announcement day it had 9,740 stars and 845 forks (same-day snapshots) — a phenomenal star velocity for an infra project; many mature inference frameworks take years to reach those numbers. The README's acknowledgments section keeps the old honesty:

"DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM."

My take: this reversal isn't a change of heart — it's arithmetic. Open-sourcing the inference framework doesn't touch DeepSeek's real moat (the serving-side KV cache systems, traffic scheduling, and scale effects — precisely the things not open-sourced, as we'll see), but it buys three things: ecosystem mindshare (making the world's deployments orbit DeepSeek's models), talent gravity (the best infra engineers want to hack on the most hardcore open-source projects), and geopolitical leverage — making Ascend a first-class citizen of production-grade inference, which deserves its own section below. That is analysis, not a DeepSeek official statement.

What this release actually contains: the feature list in plain English

DeepInfer currently supports exactly one model: DeepSeek-V4.1-Flash (we covered its weights release in cycle80 — that was a model release, this is an inference framework; two different things, and here it only appears as "the sole launch-day model"). On first run it downloads the October 17 checkpoint from Hugging Face automatically. The feature list is the set of tricks DeepSeek actually uses in production, translated into human language:

• Hand-tuned kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels. This is the deep reason DeepSeek's inference is cheap — the hot paths (matrix multiplication, attention, MoE communication) are hand-optimized to the extreme.
• Parallelism strategy: data parallelism / balanced context parallelism for attention, expert parallelism with expert-parallel load balancing (EPLB) for the MoE layers. In plain English: split a giant model across many chips without letting some sit idle while others choke.
• DSpark speculative decoding: a small draft model guesses several tokens ahead, then confidence estimates decide when to verify them. This is the production engineering of the speculative-sampling family — one of the core decoding accelerators.
• SWA bounded replay: the decoder replays only a bounded suffix during prefill, cutting redundant computation.
• Asynchronous scheduling: CPU batch scheduling overlaps with device execution; on by default; billed as a "zero-overhead batch scheduler."
• Batch-invariant inference: your output stays identical no matter who else is packed into the batch. For anyone doing evals and regression testing, this is a reproducibility essential.
• Deterministic sampling: identical input tokens + identical sampling parameters + a fixed seed = bit-identical output. A lifesaver when debugging "why did the agent produce something different this time."
• Structured outputs: force generations to conform to a regex, JSON schema, or EBNF grammar. Anyone building agent tool calls knows the difference: not "roughly JSON-shaped," but "guaranteed JSON."

Note the "(NVIDIA only)" suffix on two features: batch-invariant inference and deterministic sampling are currently NVIDIA-side only — the Ascend side hasn't caught up. Dual-chip launch does not mean feature parity; the README is honest about that.

NVIDIA SM100 and Huawei Ascend 950, on the same line

The README's "Supported hardware" section is exactly two lines:

"Supported hardware:
NVIDIA SM100 GPUs
Huawei Ascend 950 NPUs"

SM100 is the compute-capability designation for NVIDIA's Blackwell-generation data-center GPUs; the Ascend 950 is Huawei's high-end NPU. Build requirements split into two lanes as well: CUDA 13 on the NVIDIA side, CANN 9.2 on the Ascend side, plus Linux, GCC ≥ 13.0, and Python 3.12.

The weight of that line: this is the first production-grade inference framework to treat the Ascend 950 as a launch-day first-class citizen. What the Ascend ecosystem has lacked was never paper FLOPS — it was software that can serve production traffic. DeepSeek released Ascend versions of its core libraries last month (September 2026), and its founder has publicly pushed training the largest models on Chinese chips; DeepInfer is the third piece of that puzzle. For teams doing on-prem deployments in China, it means "domestic chips running an open-source inference stack" finally has a production-validated option — not a demo-grade port.

The other side of the geopolitics: just one day before the DeepInfer announcement, US authorities added DeepSeek to a federal civilian-agency ban list, covering "third-party products and services developed using DeepSeek implementations" — without naming specific products or clarifying whether open-source code counts. We covered the ban in cycle80; here it's background only: a federal ban on one side, a full Apache-2.0 open-sourcing on the other — "open technology" vs. "deployment ban" playing out in the same week is 2026 AI geopolitics in miniature. See the cycle80 piece for ban details.

Why the timing is worth a second thought: the ban targets deployment; the open-sourcing targets technology diffusion. A government can keep federal agencies off DeepSeek's services, but it can't stop engineers worldwide from git-cloning an Apache-2.0 repository. Open-source licensing and export-control jurisdiction are two parallel systems here. DeepSeek has clearly done this math — when your technology is destined to be "de-risked" out of certain markets, turning it into a public good everyone can use is the strongest counter-move. Again, analysis, not an official statement.

The three things not open-sourced: the real moat

DeepSeek is admirably upfront in the README about "what a complete inference service needs that is not included in this release":

• a KV cache system;
• prefill/decode disaggregation ("most required engine functionality is already implemented");
• load balancing across data-parallel replicas (both prefill and decode).

My take: these three are precisely what turns "a fast engine" into "a cheap public API." KV caching determines long-context cost, PD disaggregation determines how far the throughput/latency trade-off can be pushed, and load balancing determines cluster utilization. In other words, DeepSeek open-sourced the "being fast" part and kept the "selling cheap" part. Anyone hoping to replicate DeepSeek's pricing will have to build these three wheels themselves or borrow them from vLLM/SGLang. That's not "open-sourcing nothing" — the engine-level kernels and scheduling are genuinely valuable — but be clear-eyed: the recipe is open; the restaurant is not.

What happened between April 2025 and October 2026

Zoom out and the reversal stops looking abrupt. In February 2025, DeepSeek ran an "Open Source Week," releasing FlashMLA, DeepEP, DeepGEMM and other standalone components — exactly the "contribute selected components upstream" promise from the April note, kept. Each was serious on its own: FlashMLA is an efficient implementation of MLA attention, DeepEP a communication library for MoE expert parallelism, DeepGEMM an FP8 general matrix-multiplication library. In other words, DeepSeek spent eighteen months open-sourcing in slices — components first, the whole machine now.

The chip thread runs in parallel. Last month DeepSeek shipped Ascend versions of its core libraries; the founder has publicly pushed for training the largest models on domestic chips. Listing the Ascend 950 as launch-day supported hardware is the natural continuation: training-side adaptation first, inference-side framework now, the Ascend software stack being filled in piece by piece.

Then there's HN sentiment. DeepInfer hit #1 on Hacker News on announcement day with 356 points — a number that says less about "DeepSeek fandom" and more about genuine hunger in the infra community for a production-grade open-source inference stack. vLLM and SGLang are excellent, but they're general frameworks; DeepInfer is the actual stack of the company with the lowest inference costs in the industry. In the infra world, "production-validated" is hard currency.

The "fourth pole" after vLLM, SGLang, and TensorRT-LLM?

The inference-framework race is a three-way contest: vLLM (Berkeley-born, broadest ecosystem), SGLang (Berkeley LMSYS lineage, strong in structured generation and scheduling), and TensorRT-LLM (NVIDIA's own, fastest to adopt new Blackwell features). DeepInfer's README names all three in its acknowledgments, striking a humble pose: "originally built primarily on vLLM."

Humble, but the differentiation is real. First, it's born for MoE: serving MoE models like DeepSeek-V4.1-Flash bottlenecks on expert-parallel communication (DeepEP) and load balancing (EPLB) — areas where general frameworks lag far behind DeepSeek's hand-written kernels. Second, it's born for cheap: DSpark speculative decoding, SWA bounded replay, async scheduling — every item aims at cost per token, not at pretty peak-throughput numbers.

My take: DeepInfer won't displace vLLM's ecosystem position any time soon — one model, two chips, two orders of magnitude less generality. But it changes one thing: from now on, any inference framework talking about "cost optimization" has to answer "did you learn DeepInfer's tricks." It turned DeepSeek's production know-how into open course material; the vLLM and SGLang maintainers will very likely absorb its scheduling ideas in reverse. Everyone who deploys models wins in the end.

One telling detail: DeepInfer's benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode — "each within one scale-up domain." The test setup itself sends a message: DeepSeek's production deployment splits into a large decode cluster and a small prefill cluster — the classic shape of PD disaggregation, which is precisely the part not open-sourced. The README's line that "most required engine functionality is already implemented" is effectively an official admission: the engine capabilities for disaggregation are given; the cluster orchestration is not. If you want to copy the homework, understand the network topology first.

Jargon translator: what those terms actually mean

DeepInfer's feature list has a few terms that make non-infra people's eyes glaze over. Translated into human language — because understanding them is the key to understanding "why is my agent so expensive/slow":

Speculative decoding (DSpark): LLMs emit tokens one at a time, each pass running through tens of billions of parameters — that's where the slowness lives. The idea: let a small, fast draft model guess the next 5 tokens in one shot (DEEPINFER_USE_NEXTN_DECODE defaults to 5), then have the big model verify all 5 at once — keep the correct guesses, roll back the wrong ones. The higher the hit rate, the bigger the speedup. In plain English: let the intern write the first draft; the expert only reviews — reviewing is far faster than writing from scratch. If your agent generates multi-thousand-token responses, speculative decoding directly determines perceived latency.

Batch-invariant inference: to save money, serving systems pack many users' requests into one batch and compute them together. But batching changes floating-point operation ordering, so "the same request, run alone vs. run batched, can differ subtly." DeepInfer guarantees your output is bit-identical regardless of who else is in the batch. Why should vibe coders care? Because when you're regression-testing an agent, if the underlying inference itself jitters, you can't tell whether your prompt change broke something or the infra did. Reproducibility is the foundation of evals — DeepSeek shipping it as a flag (DEEPINFER_BATCH_INVARIANT=1) tells you they've been burned by this in production.

MoE parallelism (EP + EPLB): MoE models have hundreds of "expert" sub-networks; each token activates only a few. At inference time experts are spread across chips (expert parallelism), and the headache is that some experts are wildly popular — traffic floods them and their chip becomes the bottleneck. EPLB (expert-parallel load balancing) dynamically replicates/migrates hot experts to spread the load. In plain English: a cafeteria that opens more counters at whichever window has the longest line. It's one of the unsung heroes behind DeepSeek-V4.1-Flash's low price.

KV cache (the part not open-sourced): when generating token N, the attention computations for the previous N−1 tokens can be cached and reused — that's the KV cache. In long-conversation, long-context agent scenarios, KV cache hit rates directly determine cost. DeepSeek kept it closed — reasonably; it's the last mile of "being cheap" and every inference provider's real secret sauce. vLLM's PagedAttention became famous precisely by managing KV cache well; there will always be new stories in this space.

Want to try it? Check this list first

If you're tempted to run it yourself, the README's requirements are refreshingly concrete:

• OS: Linux (no mention of macOS/Windows — don't fight it);
• Toolchain: GCC ≥ 13.0, Python 3.12;
• Accelerator, pick one: CUDA 13 on the NVIDIA side, CANN 9.2 on the Ascend side, selected via the DEEPINFER_ACCELERATOR environment variable (nvidia or ascend) before building;
• Build: ./build.sh from source;
• Run: python examples/offline_inference.py; first run auto-downloads the DeepSeek-V4.1-Flash October 17 checkpoint from Hugging Face (or point DEEPINFER_HF_MODEL_DIR at local weights).

Three practical tips for vibe coders: first, start in deterministic mode (DEEPINFER_BATCH_INVARIANT=1 plus a fixed seq_seed) — make "reproducible" your baseline before turning on performance optimizations; tuning a non-reproducible system is misery. Second, play with structured outputs first (pass a JSON schema via so_args) — the highest-ROI feature for agent scenarios, verifiable in half an hour. Third, don't rush to production: remember the three missing pieces — KV cache, PD disaggregation, load balancing. Real serving means doing that homework yourself or waiting for the community ecosystem to catch up.

One more note: the repo ships an eval/ directory and a Chinese-language Q&A document (per secondary reports, the Q&A contains DeepSeek's own production figures, e.g. characterizations of per-cluster daily token volumes). That material is worth a careful read — studying how a top-tier team does evals teaches more than staring at benchmark numbers. But note: no figure I haven't personally verified appears in this article; go dig through the repo yourself if you're curious.

Why this matters to vibe coders: inference cost is agent cost

This piece isn't written for infra engineers — it's for people who tune agents every day. Why is an inference framework's open-sourcing worth ten minutes of your time?

First, inference cost = agent cost. Every extra tool-call round your agent runs, every self-reflection loop, burns inference-side tokens and latency. Understanding concepts like speculative decoding, batch scheduling, and KV caching won't make you write CUDA — but when you're choosing models, setting temperature, planning on-prem deployments, or negotiating API discounts, you'll know which layer the money goes to.

Second, determinism and reproducibility are prerequisites for agent engineering. DeepInfer making batch-invariance and deterministic sampling first-class features tells you DeepSeek has been tortured in production by "the same request producing different results." The flaky tests you hit when eval-ing agents locally often have their roots in the sampling layer, not in a badly written prompt.

Third, Apache-2.0 means directly usable commercially. If you're doing on-prem deployments for clients, or your SaaS wants to crush inference costs, DeepInfer is an option you can legally modify and sell — provided your chips are SM100 (Blackwell) or Ascend 950. The on-ramp is Linux + Python 3.12 + building from source (./build.sh), and the README even ships the benchmark reproduction script (run_sweep.sh). The official benchmarks ran prefill on 4 cards and decode on 64 (within one scale-up domain) — a scale no individual can reproduce; read it as a trend, and discount vendor self-benchmarks as usual.

One honest closing note: DeepInfer supports exactly one model today, DeepSeek-V4.1-Flash. Want to run other open models on it? Not currently possible. But the direction is clear — once the "cheapest inference stack" is open, the "cheapest API price" will sooner or later be undercut by someone wielding it. For anyone building agent applications, a cost curve bending downward is always good news.

Looking back at these eighteen months: the "no" of April 2025 was because the code was tangled with internal tooling and the team was small; the "open source" of October 2026 is because the components have been released for a year and a half, the Ascend track is laid, and the ecosystem position is worth contesting. DeepSeek's arithmetic is sharp — what went open is "being fast"; what stayed closed is "selling cheap." The recipe is public; the restaurant is not. That doesn't stop DeepInfer from being one of the year's highest-signal infra open-source releases: while the world talks about models getting smarter, DeepSeek handed everyone the playbook for making smart cheap. For vibe coders wincing at their agent bills every day, that's far more tangible than yet another model release.

FAQ: five questions you might ask

Q1: What's DeepInfer's relationship to vLLM? Will it replace vLLM?
A: The README is explicit — "originally built primarily on vLLM." It's a heavily customized fork, not a rewrite. It won't replace vLLM any time soon: vLLM supports dozens of models across many hardware targets; DeepInfer supports one model and two chips. But long-term, DeepInfer's scheduling ideas will likely be absorbed back by vLLM/SGLang. Ecosystem races are usually won by generality, not by single-event champions.

Q2: I don't have SM100 or Ascend 950. Can I use it?
A: Not currently. The README's hardware list is two lines: no consumer GPUs, no AMD, no older A100/H100. This is a production framework, not a toy — its audience is people with data centers. Individual developers are better off reading the code and docs to learn the ideas than trying to deploy it.

Q3: Apache-2.0 — can I use it commercially, modify it, sell closed-source products built on it?
A: Yes. Apache-2.0 permits commercial use, modification, and distribution including closed-source commercialization, as long as copyright notices and the license text are preserved. Caveat: the license covers the code, not the model weights or compliance — V4.1-Flash weights follow their own model terms, and US federal scenarios carry the DHS ban compliance question (see below).

Q4: Will it support more models?
A: The team hasn't said. This article won't speculate. Signals worth watching: whether the deepinfer/ directory structure and doc/ docs develop multi-model abstractions in later commits. If DeepSeek wants DeepInfer as an ecosystem on-ramp, multi-model support is the necessary path; if it only serves in-house models, it's a reference implementation. Both scripts are plausible — watch the commits.

Q5: Does the DHS ban conflict with this open-sourcing? Can US companies use DeepInfer?
A: The facts first: the DHS ban (covered in our cycle80 piece) targets federal civilian-agency deployments and covers "third-party products and services developed using DeepSeek implementations," without naming products or clarifying whether open-source code counts. Apache-2.0 itself restricts no one. So "can you use it" depends on who you are and who you sell to: ordinary commercial self-use is fine license-wise; selling to the US government — talk to a lawyer first. This is not legal advice, only public information.

Transparency notes

• Repository stars (9,740), forks (845), and the HN front-page ranking/points (#1 / 356) are all same-day snapshots from October 20, 2026, and will change over time.
• DeepSeek's official X announcement thread: X requires login, which I could not complete, so I did not open the posts directly. The announcement quotes are cited via madrobot.blog's October 20, 2026 coverage, which also cross-verified the April 2025 note.
• The April 2025 note The Path to Open-Sourcing the DeepSeek Inference Engine: I did not open the original text. Its existence and core content ("considered but decided against releasing the full engine; would contribute components upstream instead") are cross-confirmed by dig.watch's April 16, 2025 report and madrobot.blog's October 20, 2026 coverage.
• Paragraphs marked "My take" are the author's analysis based on public information, not DeepSeek official statements.
• Performance benchmark charts are DeepSeek's own self-measurements (original README figures); no independent third-party reproduction has been seen — keep that in mind when citing them.