Inference, the Stack Layer by Layer

An attributed record of what physically happens when an AI model answers a request, traced through the production system DeepSeek open-sourced on 20 October 2026, from tokens and scheduling to caches, experts across GPUs, kernels and the price of each token.

The headlines

DeepSeek has released the engine behind one cluster that serves 155 trillion tokens a day, on NVIDIA or Huawei chips.
That cluster earns $38 per GPU-hour at off-peak prices. Its costs are undisclosed, but at Google Cloud's three-year price for a B200, $7.09–8.51, the margin would be 78%–81% (derived).
Speed for one user costs total output. A GPU serving 8 requests gives each about 325 tokens a second; at 256 requests, each gets about 103, but the GPU writes about 10 times as much (step 5).
Prompts are read on ordinary 8-GPU machines and answers are written on 64–128-GPU supernodes, where each token's work is split across 6 of 384 experts on different GPUs.
Conversations' working memory is kept on SSD for 3–7 days, so a returning prompt costs 50 times less than a new one.

The record behind this page holds 309 values, each with its source: 202 reported, 15 read off published charts, 74 derived, 10 assumed and 8 estimated.

How does inference work?

Follow one question, “Why is the sky blue?”, through DeepSeek's system.

Inference, step by step

Drawn after Compute's serve loop. The first three steps happen once; the last three repeat for every token. The answer is example text.

Step 1

A message becomes numbers

A model never sees words. DeepSeek's deepseek-recipe splits your text into pieces called tokens and swaps each for its number in a vocabulary of 129,280.

“Why is the sky blue?” becomes 6 numbers. DeepSeek's template adds 30 more, including a line telling the model how hard to think: 36 in all.

The model receives 36 numbers. Your question is 6 of them.

Each box is one token: the piece of text above, the number the model sees below. The rest is DeepSeek's template: start and role markers, and a line setting how hard the model should think. IDs from our run of DeepSeek's V4.1 tokenizer. Source: deepseek-recipe ↗.

Sources: DeepSeek-V4.1-Flash model card and config ↗ (reported).

Step 2

The model lives in GPU memory

DeepSeek-V4.1-Flash is 552 billion numbers, called weights, arranged in 40 layers. To use them, a GPU must hold them in its high-bandwidth memory (HBM), the fast memory packed beside the chip.

Its published files come to 510 GB, and one GPU holds about 180 GB, so the model is split across GPUs. Spread over 8, each holds an eighth and keeps the rest of its memory for work. How DeepSeek actually splits it comes back in step 7.

The weights don't fit in one GPU, so each of eight holds an eighth.

Weights are DeepSeek's published checkpoint files. DeepSeek doesn't name its GPUs; 180 GB per GPU is NVIDIA's DGX B200, used as a reference. The even split is our arithmetic and an upper bound: it ignores runtime buffers. Source: DeepSeek-V4.1-Flash checkpoint files ↗; NVIDIA DGX B200 ↗.

Sources: DeepSeek-V4.1-Flash model card and config ↗ (reported); DeepSeek-V4.1-Flash checkpoint file listing ↗ (reported); NVIDIA DGX B200 specifications ↗ (reported).

Step 3

Reading the prompt: prefill

First the GPUs read all 36 tokens at once, pushing them through every layer together. This is called prefill.

For each token it leaves behind a small working memory, the KV cache, so later tokens can look back without redoing the work. DeepSeek-V4.1-Flash keeps 890 bytes per token.

Reading bigger batches gets more from each GPU, but every step takes longer.

Read off DeepSeek's chart (±2%). Throughput rises by half while the time per step grows about elevenfold: past a point, a bigger batch mostly makes the next user wait. Source: DeepSeek, DeepInfer prefill benchmark ↗.

Sources: DeepSeek-V4.1-Flash technical report ↗ (reported).

Step 4

Writing the answer: decode

Then it writes the answer one token at a time. This is decode. Each step runs the model again for the newest token, adds that token's memory, and repeats.

Every step must read the active weights, about 16 billion numbers, and every token's memory from HBM. The arithmetic is small; the reading is what takes the time. That one fact shapes everything that follows.

One pass, one token16 billion weights read

Then again, and againone pass per token

Memory sets the pacereading, not arithmetic, takes the time

To write one token, the chip reads every active weight out of the memory beside it, does a moment of arithmetic, and one token comes out.

16 billion weights read

The new token joins the conversation and the whole pass repeats. Each token needs its own full read of the weights.

one pass per token

The chip could do far more arithmetic than decode asks of it. What it waits on is the stream of weights from memory. That is why the next steps are about reading the weights less often per token.

reading, not arithmetic, takes the time

Slowed down: a real pass takes about 10–32 ms. The answer is example text.

Sources: DeepSeek-V4.1-Flash model card and config ↗ (reported).

Step 5

Serving many people at once

Since every step reads the weights anyway, a GPU writes the next token for many requests in the same pass. More requests make each step slower but far more productive.

In DeepSeek's benchmark, a GPU serving 8 requests gives each about 325 tokens a second and writes 2,600 in total. At 256, each gets about 103, but the GPU writes 26,300: ten times as much.

Every request also needs its own working memory. At 890 bytes a token, one 8-GPU machine holds about 7,968 long conversations, against 144 for DeepSeek's 2025 model (our arithmetic).

More people per GPU means a slower answer for each, and far more output in total.

The same 64 GPUs at six loads. Read off DeepSeek's chart (±2%), assuming 3.25 tokens per step, a setting in DeepSeek's plotting script. Source: DeepSeek, DeepInfer decode benchmark ↗.

Sources: DeepInfer README and benchmark charts ↗ (chart-read); DeepSeek-V4.1-Flash technical report ↗ (reported, derived).

Step 6

Guessing ahead

Decode leaves the GPU's arithmetic mostly idle, so DeepSeek puts it to work. A small draft model, DSpark, guesses the next 5 tokens, and the main model checks them all in one pass.

It keeps the guesses up to the first one it disagrees with and writes its own token there. On average 3.2–3.3 tokens come out of each step instead of one.

Checking fewer, likelier guesses: 1.5% slower at 16 conversations per GPU, 11% faster at 96.

DeepSeek's estimates, computed from its 64-GPU benchmark; it runs production at about 60 conversations per GPU. Checking fewer guesses barely changes how many are kept (3.29 against 3.2–3.3 per step). Source: DeepSeek, DeepInfer deployment Q&A ↗.

Sources: DeepSeek-V4.1 technical report, §2.4.3 DSpark ↗ (reported); DeepInfer inference deployment Q&A ↗ (reported).

Step 7

The model is spread across 64 GPUs

Remember the model that didn't fit? Each layer is split into 384 small expert networks, and each token uses only 6, plus one shared expert. DeepSeek spreads the experts over 64 GPUs, 7 slots each.

At every layer, each token travels to the GPUs holding its experts and back. That traffic is why decode runs on a supernode: 64 to 128 GPUs joined by one fast network.

Spreading the experts over 64 GPUs instead of 8 makes each user's answer faster.

With 64 GPUs each holds 7 expert slots, so each step reads less from each GPU's memory. Read off DeepSeek's chart (±2%), assuming 3.25 tokens per step. Source: DeepSeek, DeepInfer deployment Q&A ↗.

Sources: DeepSeek-V4.1-Flash model card and config ↗ (reported); DeepInfer README and benchmark charts ↗ (reported); DeepInfer inference deployment Q&A ↗ (reported).

Step 8

Reading and writing get separate machines

Reading a prompt keeps a GPU's arithmetic busy; writing an answer keeps its memory busy. On shared GPUs, one long prompt would stall everyone's next token.

So DeepSeek runs them apart: prefill on ordinary 8-GPU machines, decode on supernodes, about one prefill GPU per decode GPU. The prompt's working memory then crosses over. At DeepSeek's budget of 400 Gbps per 8 GPUs, a 131,072-token conversation's memory crosses in about 2.3 ms (our arithmetic).

Sources: DeepInfer inference deployment Q&A ↗ (reported, derived); DeepInfer README and benchmark charts ↗ (reported).

Step 9

Remembering the conversation

Chat apps and agents resend the whole conversation on every turn. Rather than read it again, DeepSeek saves each prompt's working memory to about 113 petabytes of SSD on its 3FS file system, for 3 days for agents and 7 for chat.

When a conversation returns, the saved memory loads straight in and only the new message is read. That's why DeepSeek charges $0.003 per million cached tokens against $0.15 for new ones. It hasn't published how often that happens for this model.

Memory per token fell 437-fold in three years, so one machine holds 7,968 long conversations instead of 16.

Bytes per token are DeepSeek's. Counts are our arithmetic and an upper bound: an NVIDIA DGX B200's memory, less an even eighth of the weights per GPU, divided by each model's memory per conversation. Source: DeepSeek, V4.1 technical report, Figure 1(b) ↗.

Sources: DeepInfer inference deployment Q&A ↗ (reported); Models and pricing ↗ (reported).

Step 10

The whole system

Put together, that's DeepSeek's production system, as drawn in its DeepInfer release. Every box is a step you've just seen: the tokenizer, the prefill and decode machines with their expert balancers, the memory handoff and the shared cache.

DeepSeek open-sourced the engine inside the boxes. The services that connect them, it kept; the table below shows which is which.

Part III

The business

What a token costs, what DeepSeek earns, and what it gave away.

01What does a token cost, and what does DeepSeek earn?

DeepSeek's prices are public and its revenue rate is reported. Its cost is not, so any margin rests on an assumption.

The price list mirrors the machine (Figure 1). A cached prompt costs a fiftieth of a new one because the work is already done (step 9). The answer costs four times a new prompt because writing is the slow phase (step 4). Prices double for 7 hours of each weekday, in the Chinese working day.

Figure 1

US dollars per million tokens, off-peak

Prompt, already cached

$0.003

Prompt, new

$0.15

Answer, written by the model

$0.60

Figure 1: What DeepSeek charges per million tokens for deepseek-flash, now DeepSeek-V4.1-Flash. A cached prompt costs a fiftieth of a new one; the answer costs four times a new prompt. Peak-hour prices are double. Source: DeepSeek API docs, Models and pricing ↗, retrieved 20 October 2026.

With DeepInfer, DeepSeek gave two business numbers: one cluster serves 155 trillion tokens a day, about 1.8 billion a second, and takes in $38 per GPU-hour at off-peak prices.

Margin is one minus cost over revenue, so it moves with the assumed cost of a GPU-hour (Figure 2). One analyst, assuming $3–6, estimated 84.2–96.1%. At Google Cloud's three-year price for a B200, $7.09–8.51, it would be about 78–81%. Replies objected that cached tokens still cost storage, GPUs never run full, and the analyst's assumed 97% cached share drives the result.

Figure 2

Figure 2: DeepSeek's reported $38 of off-peak revenue per GPU-hour leaves a margin that depends entirely on what a GPU-hour is assumed to cost, which DeepSeek does not disclose. Margins are our arithmetic; Google Cloud's price includes power and buildings. Source: DeepSeek, DeepInfer deployment Q&A ↗; @zephyr_z9 ↗; MTS Compute, Google Cloud Billing Catalog ↗.

DeepSeek's last claim, in February 2025, was a "cost profit margin" of 545%. That was a markup on cost; as a gross margin it is 84.5%, on theoretical revenue at an assumed $2 per GPU-hour.

More detail and what DeepSeek doesn't publish

Full off-peak list: cached $0.003, new $0.15, output $0.60 per million tokens; peak is double. Peak hours are 01:00–04:00 and 06:00–10:00 UTC on weekdays.
For comparison, DeepSeek-V4-Pro off-peak costs $0.02, $0.66 and $1.98. In February 2025, R1 output cost $2.19, 3.7 times today's off-peak Flash price.
Reading a full 131,072-token context costs about $0.02 fresh and $0.000 from cache.
The analyst assumed a mix of 97% cached, 2% new and 1% output tokens, and 80,000 and 16,000 tokens a second per GPU for reading and writing. His first range was 87.76–96.94%, before using DeepSeek's $38 and its ~80% production efficiency.
Other Google Cloud B200 prices per GPU-hour: one-year commitment $11.12–13.34, on-demand $8.05–11.28, spot $1.81–8.84.
Not published: DeepSeek's cost per GPU-hour, cluster size, GPU model, token mix and cache-hit rate.

02What can you run yourself?

The engine. DeepInfer is the code that runs DeepSeek-V4.1-Flash in production, released under the Apache 2.0 licence for NVIDIA SM100 GPUs (B200 and GB200 class) and Huawei Ascend 950. It runs the model on one group of GPUs.

What DeepSeek kept is the system around it: the parts that turn many copies of the engine into a service. The released kernels are close to DeepSeek's own. Its public attention kernel is 5–10% slower at decoding, and the end-to-end gap is at most 5%.

Part	Released	What it is
Inference engine	Yes	DeepInfer: scheduler, expert parallelism and balancing, DSpark, SWA Bounded Replay, structured outputs. Repository ↗
Tokenizer	Yes	deepseek-recipe, the same encoding the DeepSeek API uses. Repository ↗
Kernels	Yes	FlashMLA (attention), DeepGEMM (matrix maths, including Mega-MoE), DeepEP (expert traffic), TileKernels, DeepSelect.
Benchmarks and notes	Yes	The sweep scripts behind Figure 1 and a long deployment Q&A. Q&A ↗
KV cache service	No	The system that stores and finds cached prompts on 3FS (step 9).
Prefill/decode split	No	Running the two phases as separate services and moving memory between them (step 8).
Load balancers	No	Routing requests across many copies of the engine.

That split shaped this page. Where code exists, the numbers come from it; where it doesn't, from DeepSeek's own description, labelled as such. DeepSeek's benchmarks are its own and have not been reproduced independently.

About this record

Every value on this page is labelled by how we know it. Reported values are stated by their source. Chart-read values are read off a published chart, to within about 2%. Derived values are our arithmetic on reported ones. Assumed values are settings in a source's own calculation, not measurements. Estimated values come from a third party's calculation, with its assumptions named. No one has independently reproduced DeepSeek's benchmarks. DeepInfer is pinned at commit 774ce24. Retrieved 20 October 2026.