How to serve 134 GB of model weights with 24 GB VRAM / 64 GB RAM
How oom-inference serves roughly 134 GB of text-model weights across a 24 GB 4090, 64 GB of host RAM and NVMe.
1. Why does capacity matter
Memory is used to store weights and KV cache. Having the weights in VRAM means they can take advantage of your graphics card's memory bandwidth and processing power. RAM is similar, just less efficient, because CPUs are less suited to this workload and bandwidth is more constrained. Disk is terrible - it's not feasible to include disk reads on a hot path.
Part
Size
Where the engine keeps it
Decoder experts
67.95 GB
2.7 MB records: hot in VRAM, warm in pinned RAM, all on NVMe
PLE tables (FP8)
51.20 GB
On NVMe; buffered reads gather the rows a prompt needs
Dense
10.08 GB
VRAM
MTP experts
5.03 GB
Own tier: one VRAM chunk, records in pinned RAM
Total
134.26 GB
Against 24 GB VRAM + 64 GB RAM = 88 GB
2. How I got NERDsniped
I started reading about the work that FlashML were doing with FreeToken to enable running larger models on consumer hardware by tiering experts between GPU & CPU. This got me thinking about extending that rationale to experts on disk.
oom-inference is my attempt at making that work for Qwen 3.8 Flash Next.
3. How does the tiering work?
For every layer in the MoE our router picks 10/512 experts to use for a token. That means that most experts are idle & depending on the workload you may only trigger a subset of experts for a session.
One layer: 10 of 512 routed experts run for a token
Key
Part
1
The router scores the token and picks 10 of the layer's 512 routed experts.
2
The expert pool. Heavy squares are the experts selected for this token; faint ones sit idle. Their positions are illustrative.
3
The shared expert runs for every token whatever the route, and its output joins the routed experts'. Attention and the other dense layers also run for every token.
To enable this we split the weights into individual records per expert (~2.7 MB).
Where the expert is
Cost for one 2.7 MB record
In VRAM
No transfer
Host hit, run on CPU (one-token decode)
About 70 µs; a 10 KB row moves
Host hit, copied
About 110 µs over PCIe
NVMe miss
About 0.45 ms (illustrative), then the copy
For experts we let the engine manage O_DIRECT reads through io_uring so that we have complete control over the caching behaviour. This lets us optimize placement over many streams & make data driven decisions about when to move experts around.
The ngram lookup is different, we know in advance what a token needs from the input text so the engine can just retrieve them ahead of time and rely on normal page cache behaviour.
More context == Slower Experts
In this model KV Cache has to live somewhere - whether it's in VRAM or RAM it's displacing experts. We dynamically size this so that we get maximum throughput depending on how much of the 256k context window is in-use.
Context (tokens)
KV cache
Expert records it displaces
2,048
0.04 GB
15
32,768
0.68 GB
246
131,072
2.73 GB
985
262,144
5.45 GB
1,969
4. Parsing prompts by layer
When we throw a large prompt at a model normally in VRAM the efficient thing to do is to have it complete a pass all at once. For us that's actually super slow since we end up with a lot of idle GPU time while our copy engine moves experts around.
Small batches made this much more efficient - we run each layer across all chunks, get the union of their experts once and then move on to the next batch. Our next layer's batch can be loaded while the current batch computes.
Stage-ahead submitted
This layer's copies wait
Warm 32k prefill
Warm 95k prefill
All at once: 450 records, 1.2 GB, about 45 ms
Up to 45 ms
10.30 s
28.1 s
8 records at a time, about 1 ms each
At most one piece
9.40 s
27.4 s
5. Guessing ahead
Every decode step reads the dense weights and fetches experts anyway, so checking 8 guessed tokens costs far less than producing 8 one at a time. Guesses come from the model's own MTP head, or from prompt lookup: if the last few tokens appeared earlier, guess that whatever followed them comes next.
Draft tokens kept of 7
8-token step, 110–125 ms
One token per step
3
32–36 tok/s
40 tok/s
4
40–45 tok/s
40 tok/s
5
48–55 tok/s
40 tok/s
7
64–73 tok/s
40 tok/s
Lookup shines on file edits, which repeat big chunks of their input: output rose 22–40% in our experiments. On prose it costs a few percent.
When lookup supplies the guesses, the MTP head misses those steps and falls behind, so we now catch it up first. That took the demo from 43.4 to 45.6 tok/s.
6. Checking the maths
A broken kernel can still write fluent text, so every layer is tested against Hugging Face's reference outputs. That also tells us exactly how much accuracy each faster setting costs. The demos use all four:
Setting
Default
Faster
Memory
Drift from exact reference
k8v6 KV cache
Yes
Decode +19% at long context
About 3x less KV memory; about 1,400 more expert slots at 95k
Inside the model's own rounding noise
--dense fp8
No
Verify step 35–37 ms against 43–50 ms
About 1,660 more expert slots
KL 0.10–0.13, 86–88% top-1; about 3x the rounding floor
--expert-precision bf16
No
Warm 32k prefill 16.2 s to 14.0 s
KL 0.058, 92.1% top-1; about at the rounding floor
--attention-precision bf16
No
Prefill attention 1.51 s to 1.27 s
KL 0.051, 92.6% top-1; inside the rounding floor
7. Reusing work between requests
Agents tend to re-send the same long document, so we save the model's state partway through a prompt and pick up from there.
Follow-up on a 32k-token document
First token
Full prefill, nothing reused
13.1 s
Stored 1.1 GB entry restored
0.95 s
Live checkpoint
0.64 s
With continuous batching, requests running together share the dense-weight reads and expert fetches, so total throughput climbs up to about four at once.
Requests at once
Total tok/s, 8 streams
Decode tok/s per request
First token p50 / max
Total tok/s, 1 stream
First token p50 / max, 1 stream
1
43.6
53.4
0.55 / 0.55 s
43.4
0.55 / 0.55 s
2
48.0
31.3
0.64 / 1.3 s
42.6
0.64 / 3.6 s
4
54.9
16.9
2.1 / 2.8 s
41.5
6.7 / 9.9 s
8
47.3
7.1
3.7 / 5.9 s
39.9
13.4 / 23.1 s
The server speaks the OpenAI and Anthropic APIs, including streaming and tool calls.
8. What didn't work
Idea
Result
Draft width growing from one to three tokens while recent drafts were kept
Tied the demo (46.4 against 46.3 tok/s) and lost 3 to 7% on file edits. Fixed two- and three-token drafts ran the demo at 45.6 and 42.8 tok/s
--kv-placement host
Identical output and 9% faster decode at 95k, but 4% slower at 32k and slower at every length once --dense fp8 frees the VRAM anyway
A larger host tier
Cut disk reads 19% but didn't fix the prefill stall; the copy-engine queue did
Grouping adjacent prefill tokens to share key tiles
A key in a 4-token group's union is seen by about 1.2 of them, so shared tiles mostly compute masked-out scores
Smaller residual rings (32 or 64 rows)
Same speed as 256 on edit workloads
9. How does it compare?
Published runs on a 96 GB RTX PRO 6000, where every expert fits in VRAM, hit 60–190 tok/s. The 4090 does 45.
Speed only counts if the answers hold up. On the full GSM8K set, run with the settings RadixArk published for this checkpoint, the 4090 scores the same as their unquantised and SGLang runs:
--dense fp8 --expert-precision bf16 scores the same as the defaults on the same questions: 97.00 against 97.33 on 300 GSM8K problems, 77.5 against 77.0 on 200 MMLU-Pro questions and 98.1 against 98.5 on 95k-token retrieval, all inside the noise.
GPQA-Diamond (graduate-level science, thinking on) is partway through: 88.1 on the first 118 of 198 questions (95% CI ±5.8), against Qwen's 91.7 for the unquantised model. That interval still covers Qwen's number, and the first 118 rows of the file may not be a fair sample, so it settles nothing yet. AIME26 hasn't run.
Should you buy a 4090 for this? Probably not, API inference is cheap. But if you already have one, a 125B model at home is now practical.