NVIDIA vs. AMD: Structured Comparison and Hardware Recommendation

The Six Dimensions

Software stack maturity. The largest operational gap is TensorRT-LLM, NVIDIA’s compiled-engine inference runtime, which runs only on NVIDIA GPUs [5]. Argonne National Laboratory’s LLM-Inference-Bench evaluation found TensorRT-LLM both faster and more power-efficient than vLLM across 7B- and 70B-class models on NVIDIA hardware, though the study also recorded model-shape cases where vLLM led, so the advantage is real but workload-dependent rather than uniform [7]. On applicable workloads, TensorRT-LLM’s speculative decoding adds up to 3.6× faster generation [8]. AMD’s nearest answer is AITER, the AI Tensor Engine for ROCm, which supplies AMD-specific fused kernels for mixture-of-experts, attention, and normalization operations in vLLM behind the VLLM_ROCM_USE_AITER flag and ships in vLLM V1 as a first-party feature [9], [10]. AITER closes part of the gap on data center Instinct GPUs; it does not replicate a compiled-engine runtime. FlashAttention 3, an optimized algorithm that accelerates the transformer’s attention mechanism, is likewise unavailable on ROCm because it depends on NVIDIA Hopper- and Blackwell-specific tensor hardware; ROCm falls back to FlashAttention-2-class kernels through Composable Kernel and Triton backends [6]. NVIDIA released CUDA publicly in 2007 and AMD released ROCm in 2016; a nine-year head start. The operational head start runs longer because CUDA reached production-grade AI-framework support years before ROCm did. This dimension goes to NVIDIA.

Model compatibility out of the box. ROCm in 2026 supports every model class this paper targets, a materially better position than its reputation from earlier generations suggests. vLLM lists ROCm as a first-party target with ROCm-specific Python wheels that retire the prior Docker-only friction, named ROCm bug fixes in each release, and expert parallelism, data parallelism, and AITER-optimized multi-head latent attention kernels in V1 [10]. Ollama and llama.cpp run on AMD Instinct GPUs through the HIP/ROCm backend, and SGLang ships first-class ROCm support. The Phase 1 set of Gemma 4 31B, a 70B-class model at FP8, and gpt-oss-120b at MXFP4 has a documented AMD path, as do the larger Phase 2 classes including Llama 4 Maverick, for which AMD publishes a dedicated MI300X deployment guide [11]. MXFP4 serving on the MI355X, CDNA 4’s first FP4-capable generation, is production-demonstrated: AMD’s MLPerf 6.0 submission ran gpt-oss-120b, an MXFP4-native model, at 111% of B200 Offline and 115% of B200 Server single-node performance [1]; the residual step is validating the specific target model at procurement time. Packaged inference microservices now exist on both sides. NVIDIA NIM has shipped since 2024; AMD answered in November 2025 with AMD Inference Microservices (AIM) inside the open-source AMD Enterprise AI Suite, prebuilt vLLM-based containers with OpenAI-compatible APIs whose catalog reached the Llama, Qwen, DeepSeek, and Mistral families with MI350X and MI355X optimization in the suite’s 1.8 release of March 2026 [12]. AIM is a young analog with a narrower catalog and a shorter production history, not a missing capability. The remaining gap is timing: new architectures get tested, benchmarked, and documented on NVIDIA first, and optimization guides arrive on NVIDIA first. NVIDIA wins this dimension narrowly, and the margin shrinks each release cycle.

Community and tooling support. Staffing decides the weight of this dimension, which is where the band below 500 employees is thinnest: the operating assumption throughout this paper is one-to-few engineers, or a multi-person team, running the deployment alongside other duties. The mechanism is mundane. When a novel error surfaces during a short turnaround window, the engineer searches the error; on the CUDA side that search returns nearly two decades of accumulated answers, developer-blog posts, and resolved GitHub issues; the same search on ROCm returns far less. The independent MI300X analysis by Ambati and Diep, an arXiv preprint from October 2025, traces AMD’s residual inference deficit to software maturity, naming collective-communication libraries, compiler optimization for LLM kernels, and stack stability rather than the chip [13]. A community corpus is a function of production-years, and AMD’s documentation investment, real and visible in the per-model optimization guides shipped with ROCm 7.2.3, cannot manufacture that history. For an engineer with no prior ROCm operational experience, novel debugging on AMD takes longer, and that cost compounds across every incident.

Cost per token and memory economics. The answer is workload-conditional, so the blanket “NVIDIA wins” and “AMD wins” framings in vendor marketing both misread the evidence. For the single-GPU serving profile Phase 1 runs, CloudRift’s November 2025 test measured a single RTX PRO 6000 at 3,140 tokens per second against 2,987 for an H100 SXM on GLM-4.5-Air at 4-bit AWQ under vLLM, which works out to roughly 28% lower cost per token ($0.18 versus $0.25 per million tokens) at the study’s rental-rate assumptions and about one third of the H100’s roughly $30,000 hardware cost; the card tested was the Workstation Edition, which shares the GB202 silicon and 96 GB memory of the Server Edition this paper specifies [14]. Akamai’s NIM-profile test on Llama-3.3-Nemotron-Super-49B put the card at 1.63× the H100 NVL 96GB’s FP8 throughput when running FP4 at 100 concurrent requests [15]. Both results carry a boundary: CloudRift’s January 2026 long-context follow-up, at 8K-token input and 8K-token output, measured the B200 at up to 4.9× the RTX PRO 6000’s throughput and found the B200 the cost-per-token leader across every tested model, so the RTX PRO 6000’s cost advantage is a short-to-moderate-context, single-GPU result, the Phase 1 profile, and not a general law [16]. No directly comparable third-party cost-per-token benchmark exists for the MI355X at this model class; MLPerf 6.0 covers Llama 2 70B Server, not 30B-class AWQ workloads. The math reverses at the opposite end. Llama 3.1 405B at BF16 needs roughly 810 GB of GPU memory for weights; a single 8× MI300X node (1.5 TB HBM3) runs it without cross-node tensor parallelism while an 8× H100 SXM5 node (640 GB) cannot. dstack’s benchmark put the 8× MI300X node ahead of the 8× H100 node on both throughput and cost once prompt and batch sizes saturated the H100’s memory [17]. Llama 4 Maverick, at 400 billion total parameters with 17 billion active, follows the same logic at BF16 [18]. AMD wins on system cost for production-precision models that exceed 80 GB per GPU; NVIDIA wins on cost per token for everything that fits inside 80–96 GB.

Memory capacity and bandwidth. AMD’s per-GPU memory lead over the NVIDIA Hopper generation is real, and it becomes decision-relevant only for workloads whose weights and KV cache exceed the NVIDIA part’s capacity at target precision. The MI300X (192 GB HBM3) carries 2.4× the H100 SXM5’s 80 GB, the MI325X (256 GB HBM3E) carries 1.8× the H200’s 141 GB [4], and the MI350X and MI355X (288 GB HBM3E at 8.0 TB/s) match NVIDIA’s B300, which also ships 288 GB of HBM3e at 8.0 TB/s [19]. At the Blackwell Ultra generation, AMD’s flagship single-GPU VRAM advantage is neutralized. Phase 1–2 NVIDIA hardware (RTX PRO 6000 at 96 GB, H100 at 80 GB, H200 at 141 GB) sits below AMD’s current generation on memory; Phase 2–3 NVIDIA hardware (B200 at 180 GB, B300 at 288 GB) closes the gap. The Phase 3+ comparison resets at the Vera Rubin generation (288 GB HBM4, second half of 2026) [20], where AMD’s announced MI455X (432 GB HBM4) would reopen a 50% capacity lead that Phase 2–3 procurement should re-test against then-current NVIDIA specifications [21]. The dimension goes to AMD at the H100/H200 generation, neutral at B300, and back to AMD at the MI455X-versus-Vera-Rubin comparison if the announced MI455X specifications reach production.

OEM availability through Dell and Supermicro. The Dell relationship is asymmetric. Dell’s featured AI server line is primarily NVIDIA-based (XE9780, XE7745, XE9712), with AMD configurations available through the same catalog but not the featured marketing focus [22]. Dell currently offers the PowerEdge XE9680 with 8× MI300X and the PowerEdge XE9785 with 8× MI355X [23], [24]. Supermicro treats both vendors as first-class: its AMD GPU servers include the 10U air-cooled MI355X chassis, the 4U liquid-cooled MI355X system, the 8U air-cooled MI350X system, and the prior-generation AS-8125GS-TNMR2 with 8× MI300X, all through the same channel that serves NVIDIA HGX B200 and B300 systems [25], [26], [27]. Supply favors AMD at the margin for Phase 1 procurement windows. NVIDIA’s November 2025 results confirmed demand running ahead of supply, with CEO Jensen Huang stating that “Blackwell sales are off the charts, and cloud GPUs are sold out” and the company citing half a trillion dollars of Blackwell and Rubin revenue visibility through calendar 2026 [28], so MI355X configurations may carry shorter lead times. Treat that as directional and verify lead times in writing at quotation. The dimension goes to NVIDIA at Dell, neutral at Supermicro, with a slight AMD edge on Phase 1 lead times.

The Comparison Matrix

Across the six dimensions, NVIDIA takes software stack maturity, model compatibility, and community support outright, and takes Phase 1 cost per token for the model classes a Phase 1 deployment actually runs. AMD takes large-model memory economics at 405B-and-above BF16. The remaining dimensions split by generation and channel: memory capacity favors AMD at the H100/H200 generation and reaches parity at B300, and OEM availability favors NVIDIA at Dell while staying neutral at Supermicro. The verdict does not rest on a vote count. It rests on the two dimensions that govern how a single practitioner or small engineering team absorbs an incident: community and tooling support, which sets how long a novel issue takes to resolve, and software stack maturity, where the absent TensorRT-LLM runtime is the most concrete gap in the comparison.

The Phase 1 Recommendation

The Phase 1 procurement is one or two NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (96 GB GDDR7) in a Dell PowerEdge XE7745 or a Supermicro 4U air-cooled GPU SuperServer. A single card sustains 8,425 tokens per second on Qwen3-Coder-30B AWQ under vLLM [29], roughly 18× the 450 tokens per second a fifteen-user load needs at thirty tokens per second each, and runs 70B models at FP8 inside the 96 GB envelope (about 70 GB for weights plus roughly 26 GB of KV-cache headroom). The same card runs gpt-oss-120b at its native MXFP4 format on one GPU, so the largest Phase 1 model class needs no separate hardware path. Against the H100 NVL 96GB it delivers 1.63× the FP8 throughput when running FP4 at 100 concurrent requests [15], and against the H100 SXM it delivers roughly 28% lower cost per token at single-GPU scale [14].

Headcount does not determine the size of this build; peak concurrent generation does. Dividing the measured 8,425 tokens per second by a thirty-tokens-per-second user rate gives roughly 280 concurrent sessions, which places a single card far above the fifteen concurrent users design point and has the same configuration through the 100 to 499 employee band this paper calls medium. The recommendation continues to hold up to 999 employees on the same arithmetic, subject to two bounds. CloudRift measured aggregate throughput at its own batch settings rather than at 280 concurrent sessions, so that figure is an unvalidated ceiling and not a tested capacity [29]. KV-cache capacity inside the 96 GB envelope, not compute, is what gives way first as context lengths and session counts rise, so an organization near the top of the medium band should measure peak concurrency at its own context lengths and budget the second card against that measurement. Both the Dell XE7745 and the Supermicro 4U GPU SuperServer accept incremental GPU additions up to eight cards in the same chassis, which preserves the Phase 1-to-Phase 2 expansion path without a server swap. The eight-card maximum configuration draws about 4.8 kW GPU-only, which fits inside the 15–20 kW per-rack envelope that standard Computer Room Air Conditioning and Air Handling (CRAC/CRAH) equipment supports with no facility retrofit.

NVIDIA’s community depth and CUDA software maturity are the deciding factors for a first build operated by one-to-few engineers. Those two dimensions drive the recommendation independent of any pre-existing vendor relationship; existing Dell or Supermicro procurement relationships are confirming factors, not the basis of selection. AMD hardware is not recommended for Phase 1. Its throughput exceeds the requirement by orders of magnitude; the disqualifiers are the entry unit and the software stack. The smallest current-generation Instinct deployment is an 8-GPU node [25], [26], a unit that overshoots both the Phase 1 workload and any capital budget scaled to an organization below 500 employees, and the ROCm stack plus the thin community corpus load disproportionate operational risk onto a team with no slack to absorb it.

The Phase 2 Recommendation

Phase 2 stays on NVIDIA for both the inference scale-out and the training server. The path is to add RTX PRO 6000 cards to the Phase 1 server, or stand up a second identical server, for inference capacity, then procure a dedicated 8× H200 or 8× B200 SXM system for post-training and fine-tuning. Two considerations decide the training-server side. The fine-tuning toolchain (PyTorch FSDP, DeepSpeed, Unsloth) develops and validates on CUDA first with differing interconnect ceilings: AMD’s Infinity Fabric supports 8-way tensor parallelism inside a node at 896 GB/s of aggregate peer-to-peer bandwidth per GPU but caps the scale-up domain at eight GPUs through the MI355X generation [25], while NVLink/NVSwitch runs at 1,800 GB/s per GPU and extends to a 72-GPU rack-scale domain in the GB200 and GB300 NVL72 [30]. CloudRift’s long-context benchmark reinforces the B200 as the Phase 2 inference target: at 8K-token input and output, the B200 led every tested model on cost per token and reached up to 4.9× the RTX PRO 6000’s throughput [16]. Phase 2 inference targets including Llama 4 Maverick run on multi-GPU NVIDIA configurations: an 8× H200 node (1,128 GB) holds Maverick at BF16, and an 8× H100 node (640 GB) runs it at FP8, the deployment Meta itself describes on a single H100 DGX host [18]. The RTX PRO 6000 path does not extend to Maverick: the model’s 400 billion total parameters need roughly 200 GB of weights at INT4, above the 192 GB two cards supply, so serving it lands on the SXM node regardless of interconnect [18].

One Phase 2 workload profile warrants serious AMD evaluation: a hard requirement to serve Llama 4 Maverick or Llama 3.1 405B at BF16 with no quantization compromise. INT4 introduces measurable quality degradation that some production agentic workflows cannot accept. If that requirement becomes non-negotiable, a single 8× MI355X node (2.3 TB HBM3E) eliminates the multi-node tensor-parallelism complexity the equivalent NVIDIA configuration would carry, and system cost may favor AMD at that specification. Re-run the decision against then-current NVIDIA Blackwell Ultra (B300, 288 GB) and Vera Rubin (288 GB HBM4) parts, because the VRAM gap that distinguishes AMD at the H100/H200 generation shrinks to parity at B300 and reopens only at the MI455X-versus-Vera-Rubin comparison in the second half of 2026.

When to Revisit This Decision

Phase 2 procurement, twelve to twenty-four months after the Phase 1 pilot, is the first meaningful re-evaluation window. Five conditions should trigger a formal AMD reassessment then, each because it changes a specific term in the analysis above.

1. The ROCm Core SDK (TheRock) build system reaches production release. The technology-preview stream (7.9 series and later) carries a new build system and ManyLinux-compliant Python-wheel packaging, with a mid-2026 production target [31]. Production release would cut ROCm installation and deployment friction, which is a precondition for taking AMD seriously because deployment friction is part of the operational tax that ruled AMD out at Phase 1.

2. AMD ships a compiled-engine runtime, or AITER reaches comparable optimization depth on MI455X hardware. This is the single largest software gap in the mid-2026 comparison, and closing it rewrites the first dimension. Verify against independent benchmark data, MLPerf Inference v7.0 or equivalent, rather than vendor claims, because the gap is exactly the kind a vendor blog overstates.

3. The MI400 series enters production with verified third-party benchmarks. The MI455X (432 GB HBM4, 19.6 TB/s, announced for the second half of 2026) targets Vera Rubin directly [21], [20]. Independent benchmarks against Vera Rubin NVL72 on production workloads at comparable price points are the evidence a Phase 2–3 AMD decision requires, and none exist while both parts are in pre-production or early production.

4. The VRAM advantage persists against then-current NVIDIA hardware. If the MI455X holds 432 GB against Vera Rubin’s 288 GB, AMD keeps a 50% flagship-generation lead that matters for the largest production-precision models. A higher-VRAM Rubin variant would erase that argument, which is why the capacity case has to be re-checked at procurement rather than assumed forward.

5. Cost per token is verified on the actual Phase 2 model classes. An independent benchmark comparing AMD accelerators against NVIDIA on the models destined for Phase 2 production must precede any AMD commitment. Vendor benchmarks and MLPerf submissions both inform the picture; neither substitutes for cost analysis on the models actually going into production.

The recommendation is conditional in form and unambiguous in substance for the decision in front of the organization. The order to issue is an NVIDIA RTX PRO 6000 build in a Dell PowerEdge XE7745 or its Supermicro equivalent. The AMD re-evaluation is a dated procurement-planning item one to two years out, carrying the five verification criteria above, to be run against whatever NVIDIA and AMD actually ship by then rather than against what either announced at CES.