The question we get most often about NVIDIA's DGX Spark is "what models can I run on it?" The short answer: almost anything up to roughly 200 billion parameters will load if you quantize to 4 bits. Far fewer models will run at a speed people tolerate. The models that work well are small dense models, mid-size dense models at 4-bit for a handful of users, and sparse mixture-of-experts (MoE) models with a small active parameter count, which suit this machine better than anything else.
The reason is one pair of numbers on the spec sheet: 128 GB of memory, and 273 GB/s of memory bandwidth. The first decides what fits. The second decides how fast it decodes. Most disappointment with the Spark comes from sizing on the first number and ignoring the second.
This post walks through the arithmetic so you can size your own workload rather than rely on anyone's benchmark, including ours.
The hardware in one paragraph
A DGX Spark is built around the GB10 Grace Blackwell chip: a 20-core Arm CPU and a Blackwell-generation GPU sharing 128 GB of LPDDR5x unified memory with up to 273 GB/s of bandwidth. The GPU supports FP4 natively, which is where NVIDIA's "up to 200 billion parameters" claim comes from. It has ConnectX-7 networking at up to 200 Gbps, and two units can be linked to work on models NVIDIA describes as up to 405 billion parameters. It draws power from a 240 W external supply and sits on a desk. At the time of writing, NVIDIA's list price for the 4 TB unit is $4,699 (it launched in October 2025 at $3,999); partner versions from the major OEMs are priced differently.
One practical note that catches teams out: the CPU is Arm, so every container image and Python wheel you depend on needs an aarch64 build. The mainstream serving stacks have them. Niche dependencies sometimes do not.
Capacity: what fits
The memory budget has three parts:
total memory needed = model weights + KV cache + runtime overheadWeights are parameter count times bytes per parameter:
| Precision | Bytes per parameter | 8B model | 32B model | 70B model | 120B model |
|---|---|---|---|---|---|
| BF16 | 2 | 16 GB | 64 GB | 140 GB | 240 GB |
| FP8 / INT8 | 1 | 8 GB | 32 GB | 70 GB | 120 GB |
| 4-bit (FP4, AWQ, GGUF Q4) | about 0.5 to 0.6 | 4 to 5 GB | 16 to 19 GB | 35 to 42 GB | 60 to 72 GB |
The 4-bit row is a range because real 4-bit formats carry scaling factors and usually keep some layers at higher precision.
KV cache is the part people forget. Every token in every active conversation holds memory for the lifetime of the request:
KV bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_value
Llama 3.1 70B (80 layers, 8 KV heads, head_dim 128, FP16 cache):
2 x 80 x 8 x 128 x 2 = 327,680 bytes (about 0.31 MiB per token)
one 32k-token context = about 10 GiB
eight 32k-token contexts = about 80 GiB
Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128, FP16 cache):
2 x 32 x 8 x 128 x 2 = 131,072 bytes (0.125 MiB per token)
one 32k-token context = about 4 GiBTwo things follow. Context length times concurrency, not model size, is often what exhausts memory. And quantizing the KV cache to FP8 halves these numbers, usually with little quality cost, but test it on your own evals.
Runtime overhead covers the serving engine, CUDA context, activation buffers, and, because memory is unified, the operating system and everything else running on the box. As a planning assumption we budget 100 to 115 GB of the 128 GB for weights plus KV cache and treat the rest as not ours.
Put together: a 70B model at 4-bit takes about 40 GB and leaves room for a healthy KV cache. A 120B-class model at 4-bit takes 60 to 70 GB and still leaves 30 to 40 GB for cache. A 200B model at FP4 takes about 100 GB and leaves almost nothing, which means short contexts and one user. It fits; it is not a service.
Bandwidth: how fast it decodes
Generating text has two phases with different bottlenecks.
Prefill processes the whole prompt in parallel. It is compute-bound: the GPU's arithmetic throughput sets the pace. Prefill determines time to first token.
Decode produces one token at a time. For each token, the processor must read every active weight from memory once. At low batch sizes this is memory-bandwidth-bound, not compute-bound. That gives a hard ceiling:
decode tokens/sec (single stream, upper bound)
= memory bandwidth / bytes of active weights read per tokenApply it to the Spark's 273 GB/s:
| Model | Active weights per token | Upper bound, single stream |
|---|---|---|
| 8B dense, 4-bit | about 5 GB | about 55 tok/s |
| 8B dense, BF16 | 16 GB | about 17 tok/s |
| 32B dense, 4-bit | about 18 GB | about 15 tok/s |
| 70B dense, 4-bit | about 40 GB | about 7 tok/s |
| 200B dense, FP4 | about 100 GB | under 3 tok/s |
For reference, comfortable reading speed is somewhere around 5 to 10 tokens per second. A 70B dense model on a Spark is therefore at the edge of usable for an interactive chat with one person, and below it once anything else is happening. For comparison, a datacenter GPU with HBM memory has anywhere from several times to more than ten times the bandwidth, which is most of why the same model feels so different there.
Why MoE models are the sweet spot
A mixture-of-experts model has a large total parameter count but routes each token through only a few experts, so the active parameter count per token is small. Capacity cost follows the total; decode speed follows the active.
That is exactly the Spark's shape: lots of memory, modest bandwidth.
gpt-oss-120b: about 117B total parameters, about 5.1B active per token
weights at its native 4-bit format: roughly 60 to 65 GB -> fits comfortably
active bytes per token: roughly 3 GB -> bound near 90 tok/s
70B dense at 4-bit:
weights: about 40 GB -> fits comfortably
active bytes per token: about 40 GB -> bound near 7 tok/sBoth of those are upper bounds, and real MoE throughput falls further short of its bound than dense does, because expert routing scatters memory access and the attention layers and KV cache are not sparse. But the order-of-magnitude gap is real and it survives measurement. The same logic applies to smaller MoE models such as Qwen3-30B-A3B (about 30B total, about 3B active).
If you are choosing a model for a Spark, start with the MoE options in your quality range, and only fall back to a dense model if your evals demand it.
Concurrency: how many people can it serve
Batching changes the arithmetic in your favor, up to a point. When the engine decodes several requests in the same step, it reads the weights once and produces a token for every request in the batch. Aggregate tokens per second rises with batch size while per-user speed degrades only gradually, until one of two limits arrives:
- KV-cache memory. Every concurrent request needs its context held in memory. Use the formula above with your real context lengths.
- Compute. As the batch grows, decode stops being bandwidth-bound and becomes compute-bound, and the GB10 has far less compute than a datacenter GPU. Prefill competes for the same compute, so a user pasting in a long document slows everyone's generation unless chunked prefill is tuned.
In practice we frame the Spark's serving capacity like this:
| Workload | Fit |
|---|---|
| One developer prototyping, fine-tuning small models, running evals | Excellent. This is what it was designed for. |
| A small team (a handful of simultaneous users) on an 8B to 30B-class or MoE model with moderate context | Good, with a batching engine such as vLLM and sensible context limits. |
| Overnight batch jobs: document classification, extraction, summarization | Good. Latency does not matter, batching is efficient, and the box costs nothing per token. |
| Long-context RAG (tens of thousands of prompt tokens per request) for multiple users | Marginal. Prefill time dominates and users wait on each other. Prefix caching helps if prompts share a prefix. |
| A customer-facing product with unpredictable traffic | Not a fit. No redundancy, no headroom, and tail latency will be poor under bursts. |
| A 70B dense model as an interactive assistant for a department | Not a fit. The decode ceiling is too low before you add a second user. |
Two units linked together
Linking two Sparks over ConnectX-7 doubles memory to 256 GB, which is how a 405B-parameter model at 4-bit (roughly 200 GB and up) becomes loadable. Run the ceiling formula before getting excited: with the model split across both units, each reads about half the weights per token, so the bound is on the order of 2 to 3 tokens per second, before interconnect overhead. It is a legitimate way to test or evaluate a very large model privately. It is not a way to serve one.
Where two units do help in production is running two different models, or two replicas of a smaller one, side by side.
Signs you have outgrown it
- Requests queue at peak hours. Time to first token climbs during the working day and recovers at night. You are compute-bound on prefill or out of batch slots.
- The engine reports KV-cache preemption. vLLM surfaces preemptions in its logs and metrics when it has to evict a request and recompute it later. Occasional is fine; regular means you lack memory for your concurrency times context.
- You have shortened context or dropped to a smaller model to keep it responsive, and quality suffered on your evals.
- Someone needs an uptime commitment. One box is a single point of failure. A second Spark as a replica is a reasonable answer for small internal workloads; beyond that you want server hardware.
- Your quality bar requires a large dense model, interactively. That is a bandwidth problem, and no amount of tuning adds bandwidth.
The next steps up are usually a workstation or server with one or more discrete GPUs (much higher bandwidth, less memory per dollar), or a multi-GPU server when you need both. The right choice depends on the same three inputs: model, context length, and concurrency. Our self-hosted vs API cost comparison covers the financial side of that decision.
A sizing procedure you can run today
- Pick the candidate model and precision. Compute weights in GB.
- Compute KV bytes per token from the model's config file (
num_hidden_layers,num_key_value_heads,head_dim), multiply by your realistic context length and peak concurrent requests. - Check weights plus KV cache against a 100 to 115 GB budget.
- Compute the single-stream decode ceiling from active bytes per token. If the ceiling is below what your users will accept, stop; tuning will not rescue it.
- Only then benchmark, with your own prompts, at your own concurrency, and look at p95 latency rather than the average.
The Blueprint planner automates steps 1 through 4 for a catalog of current models and hardware, including the Spark, so you can compare it against discrete-GPU options before buying anything.
The DGX Spark is a very good machine for what it is: a private, quiet, affordable box with enough memory to load serious models. Size it on bandwidth as well as capacity and it will do what you expect. If you would like a second opinion on a sizing, get in touch.
