CPU-Only Model Deployments: Capabilities, Limitations, and Risks

What CPU Inference Actually Delivers

The throughput numbers are unambiguous. A single AMD EPYC 9654 socket carries 96 Zen 4 cores and a theoretical 460.8 GB/s of DDR5 memory bandwidth across twelve channels [3]; with every channel populated it generates roughly 4 tokens per second on a 34B Q4_0 model [4]. A tuned single-socket EPYC 9554 running DDR5-5600 across all twelve channels reaches about 7 tokens per second on a 70B Q4_K_M model, measured with full llama-bench output and disclosed thread settings [5]. Push to full F16 precision on a dual-socket Turin platform and single-socket generation on Llama-3.1-70B falls to a few tokens per second; the second socket buys almost nothing, because token generation scales poorly once cross-NUMA memory traffic dominates [6]. A Georgia Tech characterization found that spreading inference across two sockets (96 cores) ran slower than a single 48-core socket, and Puget Systems saw decode fall roughly 40% when a dual-EPYC box was pushed to use both sockets, because the inter-socket interconnect throttles the memory bandwidth decode depends on [7], [8]. Smaller models run faster: server hardware with AVX2/AVX512 instruction sets and fully populated memory channels reaches roughly 5–15 tokens per second on 7B INT4 and 1–7 tokens per second on 70B INT4, the spread driven by memory speed, bandwidth, and thread tuning. For CPU inference, memory bandwidth is the biggest limiting factor [9], [10], [11].

The GPU comparison varies by tier and quantization, but the gap holds. A single RTX 5090 GPU with 32 GB of GDDR7 memory at 1,792 GB/s generates roughly 145–186 tokens per second on 8B models at Q4_K_M [12], [13], about 1.5–1.8× the RTX 4090 on the same bandwidth-bound decode, since the 5090’s VRAM bandwidth is close to 1.8× the 4090’s bandwidth. Absolute figures swing hard with the harness: LocalScore’s methodology-controlled run reports 66.3 generated tokens per second on Llama 3.1 8B and 45.5 tok/s on Qwen2.5 14B [14], while less-constrained community runs report rates two to three times higher [13]. Red Hat’s emerging-tech group makes the same caution the basis for a standardized CPU benchmarking framework, noting that vendors publish isolated best-case throughput without reproducible methodology, which leaves infrastructure teams unable to compare claims [15]. Single-user throughput compares cleanly only within one methodology, not across suites.

The RTX PRO 6000 Blackwell Server Edition carries 96 GB of GDDR7 memory at 1.6 TB/s [16]. Its single-user rate on small models lands within roughly 10% of the RTX 5090’s. The two cards differ by about that much in bandwidth on decode operations where bandwidth governs. The 96 GB memory capacity is the operationally meaningful difference. A 70B Q4_K_M model occupies about 40–44 GB of memory [5], leaving roughly 50 GB for KV cache and runtime overhead on a single card [17]. Production-serving throughput matters more than single-user speed and is where the two architectures diverge most. Continuous batching on a GPU lifts aggregate throughput into the thousands of tokens per second across concurrent requests by amortizing each weight read over many in-flight tokens [18]. GPU-rental operators report a single PRO 6000 in the high thousands of tokens per second on a 30B AWQ model under high-concurrency vLLM serving; read those as vendor-reported rather than independent [19]. The order of magnitude is the point: GPU serving sustains thousands of tokens per second per card under load, against single digits on a CPU socket.

The CPU-to-GPU ratio depends on quantization and model size. At 7B Q4_K_M the single-user gap runs roughly 4–13×; at 7B Q8_0, where more bytes move per token, it widens further. The 10–30× range commonly cited for GPUs holds as a representative middle when the model fits in GPU VRAM, provided the quantization level travels with the number rather than being treated as fixed. The mechanism splits by phase. Prefill, which processes the whole prompt at once, is compute-bound, and recent server CPUs narrow the gap here with dedicated matrix units (Intel’s Advanced Matrix Extensions, or AMX, on Sapphire Rapids and later Xeons; IBM’s Matrix Multiply Assist; Arm’s Matrix Extension) that the Georgia Tech study measured at 6.3–9.1× the prefill throughput of the prior CPU generation [7]. Decode, which generates one token at a time, is memory-bandwidth-bound, because each token requires reading the full set of weights and a single-request batch leaves almost no arithmetic to overlap that read [20].Decode is where CPUs lose. The EPYC 9654’s 460 GB/s of theoretical DDR5 bandwidth is between a quarter and a third of what the RTX 5090 (1,792 GB/s) or the PRO 6000 Server Edition (1,600 GB/s) moves from GDDR7 [3], [12], [16], [7].

The ratio inverts once a model no longer fits in GPU VRAM. When the GPU must offload weights to system RAM and pull them back across PCIe for every token, the same study measured an AMX-equipped Xeon beating an A100 by 12.7× in throughput on a 30B model and an H100 by 5× on a 66B model, because the GPUs spent 59–95% of their time moving weights over the bus rather than computing [7]. That finding is the argument for sizing VRAM to the model rather than against it. The Phase 1 choice of 96 GB on the PRO 6000 exists precisely to keep the target 70B-class models resident in VRAM, on the fast side of this crossover, instead of in the offload regime where a CPU would win by default.

The Multi-User Problem

The standard framing says three concurrent users each get a third of single-user throughput. That undersells the problem, because the mechanism that enables multi-user concurrency on GPUs is the one a CPU cannot run.

Out of the box, the llama.cpp inference engine runs a single slot, an isolated execution context and KV cache allocation, and processes requests one at a time. A user who submits behind two others waits for both to complete before seeing a first token. At 5 tokens per second on a 500-token response, the third user in a queue waits roughly 200 seconds for that first token; ten deep means nine full request cycles of waiting. Enabling parallel slots and continuous batching removes the full-queue wait. It does not remove the ceiling. On a GPU, continuous batching raises aggregate throughput because batching many requests lifts arithmetic intensity into the regime where the GPU’s idle parallel compute finally gets used; vLLM’s PagedAttention and continuous batching deliver 2–4× the throughput of earlier optimized serving systems at equal latency, and over an order of magnitude more than a naive serving loop, by managing the KV cache so the batch can grow [18]. A CPU holds no comparable reserve of parallel compute to activate. The same memory-bandwidth limit that caps single-user decode also caps the batch, so added users mostly divide a fixed throughput while tail latency climbs and the KV cache splits across slots, shrinking each request’s usable context window.

The user-experience thresholds show how thin the margin was to begin with. Interactive reading is equivalent to 5 tokens per second where anything below that rate reads as unusably slow. 10–30 tok/s keeps up with the average reading pace and anything above 30 tok/s outruns the average. A single user at 4 tokens per second on a 70B model sits at the edge of tolerable when nothing else runs. Add one concurrent request and it falls off the edge.

Memory Footprint and the Server That Cannot Spare It

A 70B INT4 model in GGUF format needs roughly 40–44 GB of RAM for holding the model parameters (weights), reflecting the format’s roughly 4.8 effective bits per parameter after block-metadata overhead [21], [22]. The common “40 GB” shorthand sits within rounding error; the weights alone run 40–44 GB, with another 2–4 GB per request of KV cache at 4K context and 2–4 GB of operating-system overhead on top. KV cache is the part that scales with load. It grows linearly with both sequence length and the number of concurrent requests, and at long context with a full batch it can exceed the weights themselves [7]. Red Hat’s CPU benchmarking work budgets it as the per-request footprint × planned concurrency × a 1.25 safety margin, which is the discipline that keeps a shared server off an out-of-memory kill during a traffic spike [15]. Smaller models scale down proportionally. A 7B INT4 model needs about 4.5–5 GB and a 13B INT4 model about 7–8 GB; INT8 doubles those figures and FP16 quadruples them [23].

A modern dual-socket server with 256–512 GB of DDR5 RAM holds even the largest GGUF model without strain. Where issues start to present themselves is when AI workloads are added to CPU-based servers on top of their existing workloads. A general-purpose application server with 64–128 GB of RAM cannot serve a 70B INT4 model alongside its existing database queries, web services, and build jobs without risking out-of-memory conditions or swap thrashing under concurrent load. That is operational risk, not theoretical risk, and it is the most common way CPU inference fails in production. Not because inference is slow, but because inference destabilizes whatever else the server was supposed to run.

Quality Degradation: A More Precise Account

INT4 models do show measurable degradation on complex reasoning, math, and coding. The conventional framing is wrong about why, and the why drives the deployment decision.

On GPUs, the most comprehensive study to date evaluated FP8, INT8, and INT4 across the Llama-3.1 family over more than 500,000 evaluations and found FP8 effectively lossless at every model scale, well-tuned INT8 within 1–3% of the FP16 baseline, and INT4 weight-only competitive with INT8 [24]. Four-bit weights, calibrated well, are not the problem. The specific quantization scheme, and where it runs, is. Degradation concentrates by task rather than spreading evenly. An independent run of Qwen3-32B at GPTQ-INT4 quantization lost under two points on MMLU-Pro yet dropped about eight points on HumanEval code generation [25], and a systematic study of quantized reasoning models found that bit-widths below 8 introduce real accuracy risk concentrated on mathematical and multi-step reasoning, with model size and task difficulty as the deciding factors [26]. The average retention figure hides the part that matters.

CPU inference under llama.cpp uses the GGUF format, which applies simpler per-block quantization (unless the model’s build supplies an importance matrix) and lack the activation-aware calibration that GPTQ and AWQ quantization techniques on GPUs apply by default. Many distributed GGUF quants are uncalibrated, and their losses run higher and more task-sensitive than GPU-side GPTQ or AWQ. Benchmarks on aggressive GGUF quantization show degradation surfacing earlier on mathematical reasoning (GSM8K) than on general language tasks [27]. Compounding matters more than per-token loss. A 2–3% error rate per reasoning step reads as tolerable in isolation; chained across a 10–20 step agentic workflow, it becomes the dominant cause of task failure. For knowledge-base question answering over short context, where one retrieval-grounded answer settles the question, GGUF INT4 on CPU is genuinely fine. For agentic planning, tool-use chains, and code generation with multi-step verification, it is not, and the failure mode is not a slightly worse answer but a chain that breaks halfway through while the agent loops on a malformed tool call.

A second myth cuts the other way and deserves the same correction. Lower-bit quantization is widely assumed to mean faster inference everywhere, on the logic that fewer bytes per weight means less bandwidth per token. That holds on bandwidth-bound CPU decode. It does not hold universally: a University of Virginia characterization across Apple Silicon and NVIDIA GPUs found that the cost of dequantizing packed weights back to compute precision can offset the bandwidth saving, so a 4-bit model is not reliably faster than an 8-bit one on every platform [28]. Quantization is a quality-and-speed trade whose sign depends on the hardware, not a free speedup.

The Migration Trap

Sculley and colleagues named the dynamic that governs what happens next [29]. In machine-learning systems, components entangle with their surroundings until changing one forces a re-audit of everything connected to it. Their “changing anything changes everything” principle was about input data and features, and the same logic applies to serving infrastructure. CPU-tuned workflows encode the infrastructure’s limits as design assumptions: long timeouts because generation is slow, single-request queues because batching buys little, no token streaming because the latency profile makes it pointless, output-length caps because longer responses mean unacceptable waits, and prompt designs squeezed to fit the model classes that fit in available RAM. None of these choices is wrong for the environment that produced it. All of them turn wrong the moment the serving stack changes.

Rebuilding those workflows on GPU-accelerated inference means re-architecting for streaming, redesigning queue behavior, retuning prompts for the larger models the GPU now permits, and replacing the request-handling layer to use continuous batching. That work consistently costs more than buying GPU hardware first and designing against its capabilities. The teams that pay it are the ones that treated the proof-of-concept as a production foundation.

Where CPU Inference Belongs

None of this means CPU inference has no place. Its place is specific, and the specificity is the point.

Air-gapped and edge deployments live within the single-user, low-throughput envelope where CPU performance is enough. At the smallest model sizes, CPUs can be the faster choice outright. A 2025 study running a 1-billion-parameter model on an iPhone 15 Pro measured the CPU at 17 tokens per second against 12.8 on the device’s GPU, because for a model that small the overhead of staging data to the GPU outweighs its compute advantage [30]. A technician’s lightweight assistant on an air-gapped resource-constrained machine and an analyst’s local reference model on a workstation inside a SCIF (Sensitive Compartmented Information Facility) both sit in this envelope, and both are often where GPU procurement is genuinely infeasible for reasons of procurement, supply, regulation, or facility power and cooling. Embedded inference on industrial hardware without the power, thermal, and procurement budget for a GPU has no alternative; the CPU is the only processor in the box. Secondary classification and routing models that read short inputs and feed a larger pipeline run comfortably at 5–15 tokens per second, because they classify rather than generate, and classification does not need interactive-chat speed.

A second niche sits at the opposite extreme: models too large for any affordable GPU. A Q4 quantization of DeepSeek-R1 or Llama-3.1-405B can run through hybrid CPU/GPU offload or on a large unified-memory system, trading speed for the ability to run at all. On a dual-EPYC workstation with the KTransformers hybrid-offload library, Puget Systems clocked DeepSeek-R1 at roughly 10–14 tokens per second in decode, with a long prompt taking minutes to answer [8]; Apple’s unified-memory M-series reaches similar territory for ultra-large models at lower cost than a multi-GPU NVIDIA build [28]. That is useful for batch and non-interactive work and unusable for anything a person waits on live.

CPU compute also earns a place as a component of a GPU deployment rather than a substitute for one. An agentic compute server uses CPUs as the processing layer for tool execution, document parsing, retrieval orchestration, and the dozens of non-inference operations that surround an agentic workflow. That is the correct architectural use of CPU in modern AI infrastructure: handling everything that is not matrix math, while the GPUs handle what is.

What remains is not whether to put GPUs in the rack but which ones, how many, and in what configuration. That decision turns on two points the constraints above make concrete: the model classes the organization needs to serve, and the concurrency it must sustain before throughput collapses. Both are sizing questions, and both rest on procurement limits.