Phased Hardware Deployment Roadmap

Phase 1 (Months 0 to 12): The Pilot

Phase 1 delivers production value to five to fifteen concurrent users from the day it is racked. It is the first increment of the production system rather than a throwaway proof-of-concept, and the inference-server procurement that anchors it sets the trajectory for everything after. Two configurations resolve the choice.

Option A: RTX Pro 6000 (cost-optimal, recommended for most organizations). A Dell PowerEdge XE7745 chassis with one or two NVIDIA RTX Pro 6000 Blackwell Server Edition GPUs (96 GB GDDR7 each), or the Supermicro SYS-422GL-NR 4U air-cooled equivalent. A single RTX Pro 6000 sustains 8,425 tokens per second on a Qwen3-Coder-30B AWQ (activation-aware weight quantization) workload [2], roughly 18× the 450 tokens-per-second floor for fifteen concurrent users reading at thirty tokens per second each. The 96 GB envelope holds a 70B model at FP8 with key–value (KV) cache headroom, and gpt-oss-120B fits in its native MXFP4 format on one card [3]. Akamai’s controlled test on a 49B reasoning model measured the RTX Pro 6000 at FP4 delivering 1.63× the H100 NVL 96 GB’s FP8 throughput at 100 concurrent requests, narrowing toward parity by 200 [4], with independent benchmarking placing it about 28% lower in cost per token than the H100 at single-GPU scale [5]. Total Phase 1 capital expenditure (CapEx) is approximately $60K–$150K+ depending on GPU count and server configuration. Of that, the storage subsystem and 100 GbE switching account for roughly $25K–$50K.

Option B: H200 NVL PCIe (more memory and bandwidth per card, higher cost). Two to four NVIDIA H200 NVL 141 GB PCIe cards (141 GB HBM3e, 4.8 TB/s, 600 W, air-cooled) in the same XE7745 chassis, which Dell qualifies for both parts, or a Supermicro PCIe GPU server [6], [7]. The H200’s 141 GB exceeds the RTX Pro 6000’s 96 GB, and its 4.8 TB/s HBM3e bandwidth is roughly 2.7× the Pro 6000’s 1.79 TB/s GDDR7, which raises decode throughput on memory-bound serving and extends the single-card context and KV-cache ceiling. H200 NVL cards also pair over NVLink bridges, two- or four-way at 900 GB/s per GPU [6], so a model too large for one card shards across the bridged set without the PCIe bottleneck the bridgeless RTX Pro 6000 carries for tensor parallelism. The trade-off is the precision floor: the H200 is Hopper-generation and lacks native FP4 tensor cores, so where MXFP4 or FP4 throughput is the priority (including gpt-oss-120B at its native format), the Blackwell RTX Pro 6000 of Option A is the better-matched part. Choose Option B when per-card memory capacity, NVLink-bridged sharding, and bandwidth for FP8 or FP16 serving of 70B-class-and-larger models outweigh FP4 compute. Total Phase 1 CapEx is approximately $150K–$300K+ depending on GPU count and server configuration.

NVIDIA B200 and B300 systems are Phase 2 training-server sizing, not Phase 1 inference. A single 8× B200 server delivers a peak system output of 92,909 tokens per second on gpt-oss-120B in Artificial Analysis’s continuously updated load test, a point-in-time figure [8], roughly 200× the Phase 1 concurrency threshold; standing one up to serve fifteen users spends capital on capacity the pilot cannot reach.

Phase 1 storage is a high-capacity NVMe pool for model weights and hot retrieval-augmented-generation (RAG) data, a spinning-disk network-attached storage (NAS) tier for cold data, and a separate backup NAS tier covering model weights, vector-database snapshots, and dataset versioning. The backup tier is a named Phase 1 deliverable, not a Phase 3 add-on. The fine-tuned weights, curated datasets, and vector indices are the assets that justify the on-premises investment, and a hardware failure without backups deletes the compounding-value rationale the roadmap rests on. The full server taxonomy specifies the components: SeaweedFS for S3-compatible object storage, which is the maintained open-source path now that MinIO’s community edition has reached end-of-life; Qdrant or pgvector for vectors; and NFS or SMB for model-weight access. Phase 1 networking is 100 GbE switching among the inference server, storage, and management. A single-server Phase 1 deployment neither needs nor benefits from NVIDIA’s InfiniBand.

Phase 1 rack power is the constraint that makes the RTX Pro 6000 recommendation more than a cost argument. Option A draws about 1 kW of total system power with one card (600 W GPU plus roughly 250–500 W of system overhead) and about 4.8 kW GPU-only at the eight-card maximum, inside the air-cooling envelope the hardware analysis established for a standard 15–20 kW rack. Option B’s two-to-four-card H200 NVL configuration draws 1.2–2.4 kW GPU-only, roughly 2–3.2 kW at the system level, within the same envelope. Neither requires a facility cooling retrofit.

What does require attention is the facility cooling assessment, itself a named Phase 1 deliverable that must complete before any hardware order. It records the target rack’s current power capacity in kilowatts, the current cooling method and its maximum rack-kW rating, chilled-water availability and loop temperature should Phase 2 or Phase 3 hardware need direct liquid cooling, and the physical space for a coolant distribution unit (CDU) if liquid cooling enters the plan. Phase 1 hardware needs none of this. The assessment matters because its answer determines whether Phase 2’s training server can be air-cooled in the same facility or whether liquid-cooling infrastructure must be planned and capitalized eighteen months ahead of the hardware that depends on it. The governing standard is ASHRAE TC 9.9’s Thermal Guidelines for Data Processing Environments, 5th Edition, published March 2021 [9]; it predates B200- and B300-generation densities, and no later edition had superseded it as of mid-2026.

Five software capabilities are Phase 1 deliverables, not Phase 2 add-ons. Containerized vLLM inference serving runs under k3s or microk8s with the NVIDIA GPU Operator, supplying the health checks, automated restart, and rolling model updates that raw vLLM lacks and that multi-user production serving requires [10], [11]. OpenTelemetry trace logging captures every request and response with timestamp, user ID, model version, token counts, and latency; it is the operational foundation of the audit control the governance and security sections commit to, and without it that governance claim is aspirational. The RAG ingestion pipeline runs Apache Airflow on a PostgreSQL backend with LlamaIndex or Unstructured.io for document parsing and scheduled re-indexing, because a knowledge-base chatbot whose retrieval corpus is hand-populated goes stale within a month. AI-asset backup schedules weight backups to the separate NAS tier, vector-database snapshots aligned to re-indexing cadence, and dataset versioning replicated to secondary storage, with quarterly restore verification. The Phase 1 security baseline implements the subset of the security architecture that production traffic needs from day one: VLAN segmentation for AI infrastructure traffic, API-key access control on the inference endpoint, and the data-classification policy that governs which inputs the system may process.

Phase 1 capability: model serving for five to fifteen concurrent users and their agents on 30B–120B-class models at FP8 or MXFP4 quality, a RAG-backed internal knowledge chatbot, developer code assistance, and document summarization. The pilot serves real users with real workloads. The operational fluency it builds under production load is what makes the Phase 2 procurement decision evidence-driven rather than speculative. In a small organization the evidence may point to stopping here, since five to fifteen concurrent users and their agents can cover an entire engineering or analyst department in a firm below 100 employees. Two findings justify the Phase 2 order: measured saturation of the Phase 1 server, or a fine-tuning workload the Phase 1 hardware cannot run. Elapsed months are not a third.

Phase 2 (Months 12 to 24+): Expansion

Phase 2 turns the pilot into a production system for 25–50 concurrent users and their agentic workflows, adds custom fine-tuning on proprietary data, and installs the MLOps platform that makes those fine-tunes safely promotable. That user band is the medium organization’s production tier, and it is also where the training server first has a population large enough to keep it busy. Below roughly 100 employees the fine-tuning demand is usually intermittent, and a $400K–$800K training line then buys silicon that idles between runs; where no named workload justifies it, build the agentic compute, embeddings, storage, and MLOps additions and defer the training server. The training-server choice is the largest single capital allocation in the five-year arc.

8× H200 SXM5 (lower-risk, lower-cost). A Supermicro SYS-821GE-TNHR or Dell PowerEdge XE9680 with eight H200 GPUs at 141 GB HBM3e each: 1,128 GB aggregate, well-proven for fine-tuning, about 5.6 kW GPU-only and roughly 10–12 kW total depending on configuration, inside the air-cooling envelope of most enterprise facilities. Training-server CapEx is roughly $400K–$800K. The full Phase 2 build on this path, including the agentic compute, embeddings, and storage additions below, lands near $500K–$950K. This is the recommended path where the Phase 1 cooling assessment confirmed air-cooling capacity but did not provision liquid cooling.

8× HGX B300 (higher-capability, higher-cost, usually liquid-cooled). Eight Blackwell Ultra GPUs at 288 GB HBM3e and 15 PFLOPS of dense NVFP4 each, for 2,304 GB aggregate [12]. At full specification each GPU draws up to 1,400 W and requires liquid cooling. Dell’s air-cooled XE9780 de-rates the HGX B300 to 270 GB and 1,100 W per GPU to fit a roughly 10U air-cooled chassis [13], while Supermicro’s DLC-2 4U runs the full 288 GB, 1,400 W part on direct liquid cooling [14]. NVIDIA does not publish DGX or HGX B300 list pricing; mid-2026 third-party aggregations place an 8-GPU system in the low-to-mid six figures, and a custom Supermicro or Dell HGX build typically runs below a turnkey NVIDIA DGX system. Total Phase 2 on the B300 path is on the order of $1.3M–$1.8M, contingent on quotation. Between the two paths sits the 8× B200 (192 GB per GPU, 1,440 GB aggregate): Supermicro’s 4U DLC-2 HGX B200, shipping since August 2025, is the lower-cost liquid-cooled Blackwell entry [14], and the air-cooled DGX B200 draws about 14.3 kW at maximum [15], near the practical edge of air-cooling margin. It fits organizations that want Blackwell FP4 training economics without the B300’s aggregate capacity.

The decision rule is workload-conditional. The current anchor for the large-model end is NVIDIA’s Nemotron 3 Ultra, a 550B-parameter mixture-of-experts model with 55B active parameters, pretrained in NVFP4 and positioned by NVIDIA for long-running agentic workflows [16]. Its weights occupy roughly 275 GB at the native 4-bit format and about 550 GB at FP8, so an 8× H200 node’s 1,128 GB holds it at either precision with KV-cache headroom, and the same node holds Llama 4 Maverick at BF16. The H200 path therefore covers every model class the multi-user scaling analysis targets for Phase 2 production. The B300 path earns its premium only when the workload demands models past roughly 500B parameters at BF16, or trillion-parameter classes at FP8, without quantization compromise; those exceed the H200 node’s headroom, and only the B300’s 2,304 GB carries them with KV-cache margin. The H200 path is the default. The B300 path warrants justification against a specific workload requirement, not a hedge against future needs.

In May 2026 Dell and AMD added a Phase 2 option that changes the AMD calculus. Dell announced support for the AMD Instinct MI350P (144 GB HBM3e, up to 4,600 peak teraflops at MXFP4, PCIe, air-cooled) in the PowerEdge XE7745 and R7725, with availability starting July 2026 [17], [18]. The MI350P drops into the same XE7745 chassis a Phase 1 Option A deployment already runs: no new server, no liquid-cooling redesign, no separate procurement channel. Dell markets its 144 GB as the highest capacity available in a PCIe accelerator, a claim that checks out against the H200 NVL’s 141 GB [18], [6]. For Phase 2 inference expansion (not the training server), the MI350P in the existing chassis is the lowest-friction AMD adoption path on the table. The re-evaluation criteria from the NVIDIA-versus-AMD comparison still apply: confirm ROCm 7.x stability, obtain independent MI350P-versus-H200 benchmark data, which did not yet exist as of mid-2026, and run a workload-specific cost-per-token comparison before committing. Treat the evaluation as a calendar item for the twelve-month mark.

The dedicated agentic compute server is the Phase 2 addition most often missing from infrastructure plans, and its absence is what converts a working chat deployment into an agentic one that quietly underperforms. Agent tool execution, orchestration, and document parsing are CPU-bound work that starves the inference GPU when it shares the same host. The recommended configuration is an AMD EPYC 9965 (192 cores / 384 threads, Zen 5 Turin, 12-channel DDR5 up to 6400 MT/s, 614 GB/s per socket) or the cost-optimized EPYC 9755 (128 cores), with 512 GB to 1 TB of DDR5-6400 ECC RAM, four to eight NVMe SSDs, and a 25 GbE link to the inference server [19]. CapEx is about $25K–$50K+ fully configured. This server also gives a permanent home to the MLOps platform, data pipeline, and observability backend that ran on temporary infrastructure in Phase 1. For a small organization that defers the training server, this is the Phase 2 purchase that still matters, because agentic workload growth arrives well before fine-tuning demand does.

The embeddings server is the smallest Phase 2 addition: an NVIDIA L40S (48 GB GDDR6, PCIe, 350 W) in a 4U chassis or added to existing inference hardware [20]. The Phase 2 embedding workload, full knowledge-base indexing and periodic re-embedding, does not justify H100 or H200 silicon. CapEx approximately $20K–$30K.

Phase 2 power and cooling follow from the training-server choice. The 8× H200 server fits most enterprise air-cooled racks. The air-cooled de-rated B300 (Dell XE9780, roughly 10U, 13–15 kW maximum total) sits at the threshold of standard air cooling and warrants a dedicated rack with supplemental in-row cooling. The full-spec liquid-cooled B300 (Supermicro DLC-2 4U) requires the CDU and chilled-water loop whose feasibility the Phase 1 assessment established. If that assessment found liquid cooling cannot be provided by Phase 2 procurement, the H200 path is the only viable training server, a constraint to validate against the assessment’s findings rather than discover when hardware arrives.

Phase 2 software adds three named capabilities to the Phase 1 baseline. The MLOps platform is MLflow 3: experiment tracking, a model registry with LoggedModel lineage, and a CI/CD promotion pipeline, self-hosted on the agentic compute server with a PostgreSQL backend and the on-premises object store for artifacts, with no commercial license and no cloud dependency [21]. Without it, fine-tuning produces artifacts of unknown provenance with no audit trail and no rollback; the training hardware exists to produce custom models, and MLflow is the discipline that keeps them from becoming orphan files. Orchestration graduates from k3s to full Kubernetes with the NVIDIA GPU Operator and llm-d for KV-cache-aware routing [22]. Structured quality evaluation adds drift detection and scoring on top of Phase 1’s OpenTelemetry traces; Langfuse is the recommended self-hostable layer, with Arize Phoenix the alternative where stronger statistical drift detection matters.

Once Phase 2 agentic workflows reach production, evaluate prefill/decode disaggregated serving against head-of-line blocking under mixed load. The concurrency-architecture analysis specifies the mechanism; the trigger is specific and observable, namely 95th-percentile (P95) time-to-first-token (TTFT) degrading under mixed agentic load while average GPU utilization stays moderate. That signature is head-of-line blocking, and it is the signal that disaggregation will pay off. Deploying it before the signal appears buys complexity with no return.

Phase 2 capability: custom fine-tuning of domain models on internal data, production agentic workflows for 25–50 users and their agents, a full RAG knowledge base over indexed institutional documentation, and versioned models with CI/CD promotion. The fine-tuned models Phase 2 produces are the assets that justify Phase 3’s redundancy investment; they exist only because Phase 2 hardware was bought, and they need the availability guarantees Phase 3 provides.

Phase 3 (Months 24+ to 60+): Full Deployment

Phase 3 builds the production-grade availability and capacity headroom that separate a working deployment from a resilient one: redundancy, multi-node training fabric, expanded storage, and observability that scales to 50–100+ concurrent users with multi-model serving. Its hardware additions are smaller than Phase 2’s because the foundational decisions were already made. That concurrency band puts Phase 3 at the upper end of the 100 to 499 employee range this paper defines as medium, and the same additions continue to hold for organizations of 500 to 999 employees, a tier IDC still counts inside its small and midsize business category [23]. The second inference server is what carries that extension: it establishes horizontal scale-out, so capacity past it arrives by adding nodes to an existing fabric under an unchanged software stack. Above 999 employees the architecture still functions, but the sizing evidence assembled here does not cover it, and the figures below should be treated as a floor rather than a plan.

The second inference server is the headline Phase 3 addition. Whatever ships at Phase 3 procurement in 2028–2030 will have superseded today’s Vera Rubin generation (288 GB HBM4, 50 PFLOP NVFP4, partner availability from the second half of 2026) [24], but the architectural role holds across generations: horizontal scaling and redundancy against Phase 1’s single point of failure. Phase 1’s server keeps serving inference, Phase 2’s handles training and fine-tuning, and the Phase 3 second inference server enables A/B model testing, zero-downtime model updates, and failover. CapEx approximately $215K–$510K conservative or $350K–$700K mid-range, depending on GPU generation and configuration.

The expanded training cluster adds InfiniBand NDR (400 Gbps) between the Phase 2 training server and additional GPU nodes for multi-node all-reduce. Phase 1 and Phase 2 used 100 GbE because a single training server does not need InfiniBand; a multi-node cluster does. NVIDIA Quantum-2 InfiniBand switches with SHARP run collective operations in the switch fabric, cutting all-reduce latency [25], [26]. Meta’s two 24,576-GPU Llama 3 clusters, one on RoCE Ethernet and one on Quantum-2 InfiniBand, trained large models including Llama 3 itself on both without hitting network bottlenecks [27], [28], which is why Ethernet stays defensible through single-node scale. The InfiniBand premium is real but earns its place only once multi-node training collectives dominate. Most organizations below 500 employees never reach that point, and this line is the first Phase 3 item to cut when they do not. CapEx for the InfiniBand expansion: $50K–$100K.

The storage expansion adds NVMe over Fabrics (NVMe-oF) with GPUDirect Storage for direct GPU-to-storage data paths during training [29], grows the NAS and object-storage tiers to 500 TB to 1 PB for accumulated datasets and checkpoints, and scales out the observability backend that Phase 1’s traces and Phase 2’s quality evaluation generate at production volume. CapEx $50K–$150K+ depending on capacity.

Phase 3 capability: multi-model inference serving several models at once, redundant infrastructure with no single point of failure, multi-agent orchestration at production scale, a continuous fine-tuning pipeline that retrains production models on incremental data, and observability with drift alerting and per-user cost attribution. At the end of Phase 3 the organization holds the on-premises AI capability the strategy argued for: operational, resilient, and scaling against its own demand rather than a vendor’s allocation queue.

The Phase Summary Table

The total row spans the Phase 1 option and the Phase 2/3 configuration choices; the Phase 3 InfiniBand fabric and storage expansion ($100K–$250K combined) are incremental for organizations that reach multi-node training, and the lean floor assumes an organization that stops at the conservative Phase 3 second server. An organization below 100 employees that stops at Phase 2 carries the Phase 1 and Phase 2 lines only, and the total row does not describe its commitment. The Phase 1 and Phase 2 software deliverables sit in the master taxonomy and are not duplicated here. Every figure needs a vendor-quotation refresh at procurement. The HGX Blackwell supply environment in mid-2026 remained allocation-constrained, with NVIDIA-confirmed sell-through into mid-year and lead times that varied from weeks to quarters depending on OEM relationship, while H200 and RTX Pro 6000 hardware remained readily available; ordering Phase 1 on immediately available silicon preserves the lead time to bring Phase 2 Blackwell hardware in during the Phase 2 window. Confirm lead times in writing at quotation.

The live decision at the close of this analysis is the Phase 1 order: a Dell PowerEdge XE7745 or Supermicro SYS-422GL-NR with one or two RTX Pro 6000 Blackwell Server Edition cards (Option A), or two to four H200 NVL PCIe cards (Option B), validated against the hardware performance matrix and the facility cooling assessment, with the five Phase 1 software deliverables in place from day one. That order commits the organization to nothing it cannot expand, since every later phase drops into the chassis, rack, and stack it establishes. What remains before installation is the deployment architecture that isolates it: the network segmentation and, where the data classification demands it, the air gap that turn the procurement order into a production system rather than a server in a closet.