Open-Weight Models Ready for Production

Google Gemma 4

Gemma 4, released April 2, 2026, ships under Apache 2.0, a break from prior Gemma versions, which carried Google’s custom Gemma Terms of Use [1], [11]. The family spans five tiers. The E2B and E4B variants (roughly 2.3B and 4.5B effective parameters via Per-Layer Embeddings) target edge and mobile hardware. The 12B variant is a unified, encoder-free multimodal model with multi-step reasoning and agentic workflow performance nearing the 26B model running locally with just 16GB of VRAM or unified memory. [12]. The 26B A4B variant is a mixture-of-experts (MoE) design (26B total, about 4B active per token) that runs on a single-GPU server inference and fits within a 24 GB card at INT4 quantization. The 31B dense variant is the maximum-quality server model, fitting within 24–32 GB at INT4 and roughly 62 GB at FP16. Modality and context length split by tier rather than running uniformly across the family. Every variant accepts image input; E2B, E4B, and the 12B Unified model also accept audio, and the 31B dense model does not. Context length runs to 128K tokens on E2B and E4B and to 256K on the 12B, 26B A4B, and 31B, carried by a hybrid attention scheme that interleaves local sliding-window layers with full global attention and always closes on a global layer [11].

Gemma 4 31B scores about 85% on MMLU-Pro and 89% on AIME 2026, and posts a Codeforces ELO of 2150, up from Gemma 3’s 110 [11]. For a Phase 1 deployment optimizing hardware efficiency at production quality on knowledge work and coding, Gemma 4 31B at FP8 on a single RTX PRO 6000 Blackwell is the closest match between capability and silicon currently available.

Mistral

Mistral AI, a French AI company founded in 2023 and headquartered in Paris, offers a catalog of mostly Apache 2.0 licensed AI models. That licensing posture is the operational headline. Mistral Large 3 (December 2025) is a 675B-total, 41B-active MoE under Apache 2.0, trained from scratch on 3,000 NVIDIA H200 GPUs, with a 256K context window, native image input via a 2.5B vision encoder, 73.11% on MMLU-Pro, and 93.60% on MATH-500 [2]. Mistral Small 4 (March 2026, Apache 2.0) consolidates instruct, reasoning, multimodal, and agentic coding into a single MoE model with 119B total and 6.5B active parameters and a configurable reasoning-effort parameter [13]. The Ministral 3 family (3B, 8B, 14B dense; Apache 2.0) covers edge and local deployment [2].

Two exceptions sit outside Apache 2.0. Mistral Medium 3.5 (April 2026) is a 128B dense multimodal flagship that merges instruction-following, reasoning, and agentic coding into one set of weights, with a 256K context window and per-request reasoning-effort control; it scores 77.6% on SWE-Bench Verified and ships under a modified MIT license that gates high-revenue commercial use through a paid channel [14]. Voxtral TTS, Mistral’s open-weights text-to-speech model, ships under CC BY-NC 4.0 and needs a separate agreement for commercial use. For an organization with EU exposure that wants to avoid the Llama 4 multimodal restriction discussed below, the Apache-licensed core of Mistral is the strongest substitute at the same capability tier, with no monthly-active-user caps, no geographic carve-outs, and no acceptable-use restrictions beyond standard Apache terms.

NVIDIA Nemotron 3

Nemotron 3 is a hybrid Mamba-Transformer MoE line in which Mamba-2 layers provide linear-time sequence processing, which is what makes a 1-million-token context window practical rather than theoretical [15], [16]. Three tiers shipped across the first half of 2026.

Nemotron 3 Nano (31.6B total, 3.2B active) is the cost-efficient inference tier; NVIDIA reports 3.3× the throughput of Qwen3-30B-A3B (a popular Chinese open-weight model variant) and 2.2× that of gpt-oss-20b on a single H200 GPU at 8K input / 16K output [16]. Nemotron 3 Super (120B total, 12B active), released March 11, 2026, targets multi-agent systems and high-throughput production. Independent evaluation by Artificial Analysis placed Super at 36 on its Intelligence Index at release, ahead of gpt-oss-120b at 33 but behind Qwen3.5 122B at 42; NVIDIA reports roughly 2.2× the inference throughput of gpt-oss-120b at that intelligence tier [17], [18]. Nemotron 3 Ultra (550B total, 55B active) was open-sourced June 4, 2026 with full post-training (supervised fine-tuning, reinforcement learning, and multi-teacher distillation); Artificial Analysis scored it 48 at release, the highest-rated US-developed open-weight model on that index at the time [19]. Both positions have since moved. Under the index revision current in July 2026 Artificial Analysis lists Ultra at 38 and gpt-oss-120b at 24 with Thinking Machines Inkling passing Ultra on July 15 [6].

The license is the NVIDIA Nemotron Open Model License: not Apache 2.0 and not approved by the Open Source Initiative (OSI), but unusually transparent for a custom license, since NVIDIA releases open weights, training datasets, and recipes alongside each model [20]. Nemotron’s advantage is tight coupling to NVIDIA hardware and the NVIDIA Inference Microservices (NIM) path with built-in TensorRT-LLM optimization. Where NIM is already in the production stack, Nemotron is the natural model choice.

OpenAI gpt-oss

OpenAI’s August 2025 release of gpt-oss-120b and gpt-oss-20b is the company’s first open-weight release since GPT-2 in 2019, and its first under a fully permissive license [3]. Both models ship under Apache 2.0. The 120B variant carries 116.8B total parameters and activates 5.1B per token; its MXFP4 quantization fits the entire model in roughly 61 GB, so it runs on a single 80 GB GPU (an H100 or AMD MI300X) and comfortably on the 96 GB RTX PRO 6000. It supports a 128K context window and exposes configurable reasoning effort with full chain-of-thought output. Near-parity with o4-mini on core reasoning is the independently corroborated result [3], [4]; OpenAI’s own evaluation also reports wins over o4-mini on HealthBench and AIME and over o3-mini on Codeforces and tool-calling, which should be read as vendor-reported until reproduced.

For an organization with provenance concerns about Chinese-origin models (addressed below) and EU constraints on Llama 4 multimodal, gpt-oss-120b is a US-origin, Apache-2.0 alternative at the 100B-plus class, and its single-GPU footprint is a direct Phase 1 sizing advantage.

Meta Llama 4

Meta released Llama 4 on April 5, 2025 as a natively multimodal MoE family [21]. Scout (17B active, 16 experts, 109B total) fits a single H100 at INT4 and ships with a 10-million-token context window. Maverick (17B active, 128 experts, 400B total) targets a single H100 DGX host. Behemoth (2T total, 288B active) remains unreleased; Meta used it as a teacher model for the released variants through codistillation and has given no release window. On NVIDIA Blackwell B200, NVIDIA’s optimized stack reports Scout above 40,000 output tokens per second, and above 12,000 on H200 [21], [22].

The license is the consequential detail. Llama 4 ships under the Llama 4 Community License Agreement, a custom Meta license that is neither Apache 2.0 nor OSI-approved, which permits royalty-free commercial use below 700 million monthly active users, requires “Built with Llama” attribution on distributed derivatives, and permits training on Llama 4 outputs with attribution [23]. The material restriction is geographic: the multimodal capabilities cannot be licensed to individuals domiciled in, or companies with principal place of business in, the European Union; text-only paths remain unrestricted in the EU [24]. Any organization with EU subsidiaries or EU-based staff using the multimodal capabilities needs legal counsel to clear this restriction before standardizing on Llama 4 multimodal variants.

Cohere

Cohere, a Toronto-based enterprise AI company founded in 2019 and merged with Germany’s Aleph Alpha in 2026, changed its open-model posture in mid-2026 by releasing two models under Apache 2.0. That is a departure from the CC-BY-NC research licensing that still governs its earlier Command research weights and its Aya and Tiny Aya multilingual families, all of which remain non-commercial and therefore out of scope for production. Both Apache-licensed releases target the sovereign, on-premises deployment this paper argues for, and both carry clean, allied-Western provenance that simplify the supply-chain review.

Command A+ (command-a-plus-05-2026), released May 20, 2026, is a 218B-total, 25B-active sparse MoE with vision input, native citation grounding, agentic tool use, and a 128K context window, published on Hugging Face in BF16, FP8, and W4A4 quantizations across 48 languages. It is the first frontier-scale model a major enterprise vendor has released under a fully permissive license. At W4A4 it runs on a single B200 or two H100s/200s, which places it above the single-RTX PRO 6000 Phase 1 target but inside the enterprise tier; Cohere applies NVFP4 to the MoE experts while holding the attention path at full precision and reports the result as near lossless against BF16 [25], [26]. For an organization that wants a general-purpose, multimodal, agentic open model without the licensing or provenance overhead of the Chinese flagships, Command A+ is a strong clean-provenance option at its scale.

North Mini Code (June 9, 2026), the first model in Cohere’s North code-agent family, is a 30B-total, 3B-active MoE built for agentic software engineering, with a 256K context window and a 64K maximum generation length, also under Apache 2.0 [27]. Its small active footprint runs comfortably on a single RTX PRO 6000 or H100 at FP8, and independent evaluation by Artificial Analysis placed it at 33.4 on its Coding Index, ahead of several larger open models though slightly behind Qwen3.6-35B-A3B [28]. The documented tradeoff is verbosity: in independent testing it generated roughly three times the output tokens of comparable models, a cost that compounds in high-volume serving [29]. For a Phase 1 coding-assistant workload that values permissive licensing and single-GPU deployment, it is a Western-origin counterpart to Qwen3.6-35B-A3B and DeepSeek V4-Flash (non-reasoning).

Thinking Machines Inkling

Inkling, released July 15, 2026, is the first production model from Thinking Machines Lab and the highest-scoring open-weight model published by a US lab [6], [30]. It is a 975B-total, 41B-active MoE under Apache 2.0, pretrained on 45 trillion tokens, with a 1-million-token context window on the downloadable checkpoint, native text, image, and audio input, text-only output, and a thinking-effort control the caller sets per request across a 0.2 to 0.99 range [30]. Weights shipped to Hugging Face on release day rather than on a promised future date, and vLLM, SGLang, llama.cpp, and Transformers support arrived with the launch.

Artificial Analysis scored Inkling at 41 on the Intelligence Index, ahead of Nemotron 3 Ultra at 38, Gemma 4 31B at 29, and gpt-oss-120b at 24, and behind the Chinese open-weight leaders [6]. Two results matter more than the headline rank. Inkling is markedly token-efficient, averaging 25K output tokens per index task against 43K for GLM-5.2, 38K for Kimi K2.6, and 37K for DeepSeek V4-Pro, which changes serving economics on a fixed GPU budget [6]. Against that, it scores +2 on AA-Omniscience with 40% accuracy and a 63% hallucination rate, so it guesses rather than abstains more often than the open-weight leaders, and it should not carry accuracy-critical retrieval work without external grounding and citation enforcement [6].

The deployment footprint places Inkling above Phase 1. The BF16 checkpoint requires more than 2 TB of aggregate VRAM; the NVFP4 checkpoint reduces that to roughly 600 GB, which is an 8-GPU H200 or B200 node [31]. Thinking Machines has previewed Inkling-Small at 276B total and 12B active on a similar training recipe, with weights pending, and that variant is the one to watch for mid-tier hardware. Inkling’s strategic value is the combination of Apache 2.0 terms, US provenance, and an explicit design intent as a customization base rather than a leaderboard entry. For an organization that needs a permissively licensed, clean-provenance foundation for domain fine-tuning at frontier-adjacent scale, it is the strongest candidate currently available.

The Chinese-Origin Models

Five Chinese open-weight families now compete at the frontier of capability: Qwen (Alibaba Cloud), DeepSeek (High-Flyer), GLM (Z.ai), Kimi (Moonshot AI), and MiniMax (Shanghai). Most ship under permissive Apache or MIT licenses and post benchmarks within striking distance of the leading closed models on coding and agentic tasks; MiniMax is the exception, releasing its M3 flagship under a more restrictive custom license. The procurement question is whether that capability justifies the supply-chain review the origin demands.

Qwen’s open ceiling is Qwen3.5-397B-A17B (February 2026, Apache 2.0), a 397B sparse MoE that pairs Gated DeltaNet linear attention with standard gated attention and supports 262K native context [32]. The smaller Qwen3.6-35B-A3B (April 2026, Apache 2.0) carries 35B total and 3B active, fits a single 80 GB GPU, and is the better default for most deployments; Artificial Analysis rates it among the leading models for its size while noting it’s more expensive per token and verbose [33], [34]. The Qwen3.7-Max flagship, announced May 2026, is API-only, with no released weights [35]. DeepSeek shipped V4 on April 24, 2026 in two MIT-licensed tiers: V4-Flash at 284B total with 13B active and a 1-million-token context window, and V4-Pro at 1.6T total with 49B active; V4 is approximately tied with the strongest closed models on SWE-Bench Verified near 80% and leads on LiveCodeBench [36]. The prior generation remains widely deployed, including V3.1-Terminus and the January 2025 R1 reasoning model that first matched OpenAI o1 among open weights [37]. GLM-5 (Z.ai, February 2026) is a 744B MoE with 40B active, a 200K context window, MIT licensing, and end-to-end training on Huawei Ascend neural processing units (NPUs), the first frontier-scale model trained with no NVIDIA hardware in the pipeline [38]; Z.ai has since shipped GLM-5.1 (754B, MIT) and, on June 16, 2026, the MIT-licensed GLM-5.2, which keeps the GLM-5.1 footprint (roughly 753B total, 40B active), extends the context window to 1 million tokens, and reached 51 on the Artificial Analysis Intelligence Index to become the leading open-weight model on that benchmark; GLM-5.2 is text-only, with Z.ai’s vision capability shipping separately as the non-open GLM-5V line [39], [5]. Kimi K2.6 (Moonshot AI, April 2026) is a 1-trillion-parameter MoE activating 32B per token, natively multimodal, under a modified MIT license that adds one attribution requirement above 100M monthly active users or $20M monthly revenue and behaves as standard MIT below those thresholds [40]; it edges out the strongest closed coding models on SWE-Bench Pro (about 58.6%) at a fraction of their per-token cost. Moonshot refreshed the line on June 12, 2026 with Kimi K2.7-Code, a coding-specialized build on the same trillion-parameter, 256K-context architecture that cuts reasoning-token usage by roughly 30% versus K2.6 under the same modified MIT terms; its launch benchmarks are vendor-reported, with independent results still pending [41]. MiniMax (Shanghai) joined the frontier tier on June 1, 2026 with M3, a natively multimodal MoE carrying about 428B total and 23B active parameters, with open weights reaching Hugging Face on June 7 and the architecture’s technical report on June 11 [42], [43]. M3’s MiniMax Sparse Attention (MSA), a block-sparse mechanism layered on grouped-query attention, is what makes its 1-million-token context window economical, cutting per-token compute at full context to about one-twentieth of the prior M2 generation with roughly 9× faster prefill and 15× faster decode [42], [43]. M3 posts a vendor-reported 59.0% on SWE-Bench Pro and an independently measured 44 on the Artificial Analysis Intelligence Index, tied with DeepSeek V4-Pro and behind GLM-5.2’s 51 [5]. Its licensing is the procurement caveat: unlike the Apache- and MIT-licensed Chinese frontier models, M3 ships under MiniMax’s custom “minimax-community” license, which conditions commercial use and drew criticism at launch, so it warrants the same license review as Llama 4 or Voxtral before any standardization [42].

Kimi K3 reset the open ceiling in July 2026. Moonshot announced it on July 16 with the API live at launch and the full weights available in July 27 alongside a technical report. Its architecture has 2.8 trillion total parameters in a highly sparse MoE routing 16 of 896 experts per token, Kimi Delta Attention carrying a 1-million-token context window, text and image input, and reasoning effort fixed at maximum with lower settings promised later [7], [44]. Artificial Analysis scored it 57 on the Intelligence Index, fourth of 189 models tracked, level with Claude Opus 4.8 at 56 and above every open-weight peer, with GLM-5.2 at 51 and DeepSeek V4-Pro at 44 [8], [9]. Cost per completed task, rather than price per token, is the number that governs a total-cost comparison: at $3.00 per million input tokens and $15.00 per million output tokens with cached input discounted 90% to $0.30, Artificial Analysis measured $0.94 per Intelligence Index task against $1.04 for GPT-5.6 Sol and $1.80 for Opus 4.8, so K3 undercuts the closed frontier at comparable intelligence while costing roughly three times GLM-5.2 and more than twenty times DeepSeek V4-Pro on the same basis, and it serves at 62 output tokens per second against a 72 t/s median for its price tier [8], [45], [46]. At 2.8 trillion parameters, Kimi K3 is quite a large model. Quantized to 4 bits, it requires roughly 1.4 TB of high-bandwidth memory before factoring in key-value cache and runtime overhead, which puts self-hosting into multi-node territory and outside any single-node build this paper recommends.

The capability is real. The provenance review is mandatory. By February 2025, the U.S. House of Representatives, the Pentagon, NASA, the U.S. Navy, and multiple state governments had restricted DeepSeek’s cloud product on government devices [47]. Cisco’s Robust Intelligence team, with University of Pennsylvania researchers and the standard HarmBench jailbreak suite, recorded a 100% attack success rate against DeepSeek R1, since every one of 50 test prompts elicited a harmful response, against 26% for OpenAI’s o1-preview on the same battery [48]. Wiz Research separately found a publicly accessible DeepSeek database exposing chat history, API tokens, and system logs, and DeepSeek’s terms state that user data is stored on Chinese servers under Chinese law [49]. Representatives Gottheimer and LaHood introduced the No DeepSeek on Government Devices Act (HR 1121) on February 7, 2025 [50], and an April 2025 House Select Committee report documented ties between DeepSeek and a state-backed research institute with military-research adjacency [51].

These concerns attach to the cloud product, not automatically to the self-hosted weights. A DeepSeek V4, GLM-5.2, or MiniMax M3 deployment running on-premises, behind the firewall, on hardware the organization controls, transmits nothing to the originating vendor. MIT-licensed weights do not become a backdoor by virtue of origin; as of mid-2026 no peer-reviewed publication documents a cryptographic backdoor in any of these weights. The risk that travels with the weights is behavioral: observable content censorship aligned with Chinese government policy, training-data-encoded bias, and the institutional affiliations of the originating lab. The Cisco jailbreak finding applies whether the model runs on the vendor’s servers or yours, and it matters for any deployment where untrusted users can craft prompts.

A supply-chain review for a Chinese-origin model documents model origin and corporate structure; any government or military research affiliations; published behavior research on the specific model; an observable behavior audit covering content restrictions and geopolitical framing; and legal counsel’s read on applicable procurement regulations. Until that review is signed off, initial production use belongs in non-sensitive, air-gap-isolated contexts. For organizations whose posture does not require this scrutiny, Qwen3.6-35B-A3B, DeepSeek V4, GLM-5.2, Kimi K2.7-Code, and MiniMax M3 cover different points on the size-versus-capability curve. For organizations with U.S. government contracts, defense adjacency, or sensitive data classifications, the U.S.-origin pair of Gemma 4 31B and gpt-oss-120b covers the same capability range without the procurement overhead, with Inkling available above them for organizations that can host it.

Allen Institute OLMo

Allen Institute for AI (Ai2), a Seattle based non-profit AI research institute founded in 2014 by the late Paul Allen, released Olmo 2 and Olmo 3 under Apache 2.0. These models are the most fully open frontier line from a US institution, releasing weights, the Dolma training dataset, training code, the evaluation framework, and intermediate checkpoints. Olmo 2 32B (March 2025) was the first fully open model to outperform GPT-3.5-Turbo and GPT-4o-mini on a multi-skill suite, trained to 6T tokens at one-third the compute of Qwen 2.5 32B [52], [53]; Olmo 3 (November 2025) adds OlmoTrace, which links model outputs to specific training-data decisions and supports behavior provenance at a level no other model in this roster documents [54]. For regulated industries and contractors that must trace behavior to training data for compliance, Olmo is the right choice. Its value is auditability, not raw capability; on general reasoning, Gemma 4, Mistral Large 3, Nemotron 3 Super, gpt-oss, and Llama 4 outperform it at comparable scale.

Defining the Class

Belcak et al. at NVIDIA Research define a small language model (SLM) by deployability rather than by parameter count: a model that fits on a common consumer device and processes one user’s requests with latency low enough to be practical [55]. They anchor that to a rule of thumb of roughly 10 billion parameters as of 2025 and note the threshold moves with hardware [55]. That definition is useful for procurement because it tracks what a model costs to serve rather than what its parameter count signals in marketing. Vendor naming does not follow it. Nemotron 3 Nano carries 31.6 billion total parameters and activates 3.2 billion [16], so it sits well outside the class by total footprint even though its active-parameter count looks small.

The Belcak paper is a position piece and should be read as one. Its authors argue that SLMs are already powerful enough for the language work agents actually perform, that they suit agentic systems better operationally, and that they cost less by virtue of size [55]. The supporting evidence is assembled from prior published results rather than generated by a controlled study. The appendix estimates that 40% to 70% of language-model calls in the MetaGPT, Open Operator, and Cradle agents could move to specialized SLMs; those figures are the authors’ own assessments rather than measurements [55]. The authors also grant that the counter-argument from centralized inference economies of scale is valid and unsettled [55]. What the paper contributes is a well-specified hypothesis with named mechanisms. The measurements below are what test it.

Capability Compression

Google’s Gemma 4 model card supplies the cleanest test available, because it holds the vendor, the evaluation harness, and the benchmark suite constant across a single generation step. Gemma 4 E4B carries 4.5 billion effective parameters. Gemma 3 27B carries roughly six times that. On MMLU-Pro, Gemma 4 E4B scores 69.4% against Gemma 3 27B’s 67.6%. The margin widens on the benchmarks that predict agentic behavior: 42.2% against 16.2% on Tau2 tool use, and 52.0% against 29.1% on LiveCodeBench v6 [11]. Even E2B at 2.3 billion effective parameters beats the prior-generation 27B on Tau2, 24.5% against 16.2%, while losing on MMLU-Pro at 60.0% against 67.6% [11].

One caveat worth noting with those numbers is that the model card scores Gemma 3 27B in its non-thinking configuration while the Gemma 4 entries use the family’s configurable thinking mode [11], so part of the margin is test-time compute rather than pretrained capability. The direction of the result survives that caveat while its magnitude does not. A team sizing hardware on these numbers should re-measure at its own inference settings.

IBM’s Granite 4.0 line shows the same compression expressed as an architecture decision. The family runs from a 350M dense model through a 1.5B hybrid, a 3B hybrid, a 7B mixture of experts activating 1B parameters, and a 32B mixture of experts activating 9B [56]. The hybrid variants interleave Mamba-2 state-space layers with transformer attention with IBM reporting more than 70% lower memory use and twice the inference speed against comparable models, concentrated in multi-session and long-context serving [56]. Read that as a vendor measurement against an unnamed comparison set. The mechanism is sound, since state-space layers avoid both the quadratic attention cost and the growing key-value cache that dominate memory at long context, but the specific multiple is IBM’s and has no independent replication in the published record. Granite 4.1 then returned to dense architectures at 3B, 8B, and 30B, in base and instruction-tuned variants with optional FP8 checkpoints [57]. That reversal is the more instructive fact: hybrid state-space designs win on memory while the tooling maturity of dense transformers still wins often enough that IBM ships both and tells users to pick the dense variant wherever Mamba-2 support is immature [56].

Specialization and the Fine-Tuning Return

The small tier’s strongest case rests on fine-tuning rather than on out-of-the-box capability. Google’s FunctionGemma is a 270-million-parameter model built on the Gemma 3 270M backbone, trained only for function calling, with a 32K context window and an explicit statement in its model card that it is not intended for use as a dialogue model [58]. Out of the box it is weak where deployment would need it: 36.2% on Berkeley Function Calling Leaderboard (BFCL) Live Simple and 20.8% on Live Parallel Multiple [58]. Fine-tuned on Google’s published Mobile Actions dataset, accuracy on that task moves from 58% to 85% [58]. At dynamic INT8 the artifact occupies 288 MB, peaks at 551 MB resident, and decodes 125.9 tokens per second on the CPU of a Samsung S25 Ultra with a 0.3-second time to first token [58]. Nothing in a Phase 1 rack strains against those numbers. The transferable finding is the 27-point gap between base model and fine-tune, which is the return on a task-specific dataset rather than on parameters.

The economics of collecting that return follow from size. Belcak et al. report that parameter-efficient methods including Low-Rank Adaptation (LoRA) and its quantized variant QLoRA, along with full-parameter fine-tuning at small scale, need only a few GPU-hours, which moves specialization from a quarterly project to an overnight one, and that 10,000 to 100,000 curated examples suffice for a small model [55]. They also report that serving a 7B model costs 10 to 30 times less than a 70B to 175B model in latency, energy, and floating-point operations, a figure the paper carries from secondary industry sources rather than measures [55]. Treat the order of magnitude as sound and the specific multiple as unverified until an organization measures it on its own traffic.

Where the Small Tier Fails

Two important limits worth mentioning that affect production deployments.

The first is long-horizon execution, which is not a small-model problem alone. A published synthesis of agent failure studies reports METR’s time-horizon finding that frontier models succeed on close to 100% of tasks a skilled human completes in under roughly four minutes and on under 10% of tasks taking a human more than roughly four hours, with the task length completable at 50% reliability doubling about every seven months [59]. The HORIZON diagnostic benchmark, which collected more than 3,100 trajectories from current GPT-5 and Claude agents across four domains, attributes the collapse to agents losing the original objective and repeating failed actions rather than to retrieval failure [60]. Both are preprints and neither is SLM-specific, which is precisely why they bind here: if the frontier degrades this way, the orchestrating role belongs to the largest model a deployment can afford, and the small tier takes the bounded steps that orchestrator dispatches.

The second limit is breadth; stated plainly in the model cards. Microsoft trained Phi-4-Reasoning-Vision-15B primarily on English and says it is not intended to support multilingual use [61]. IBM trained and tested Granite Guardian 4.1 only on English, and warns that the model must be used strictly in its prescribed scoring mode, that deviation may produce unsafe output, and that the model is susceptible to adversarial attack [62]. Google’s FunctionGemma safety evaluations covered English prompts only [58]. An organization with multilingual users cannot staff a small tier from these models without measuring the degradation on its own languages first.

What the Small Tier Changes in Phase 1

Nothing in this tier displaces the 31B-to-120B model on the RTX PRO 6000. It changes what else runs on that card and the concurrency arithmetic. Weight footprints at these sizes are arithmetic rather than measurement: a 4B model at FP8 holds roughly 4 GB of weights, a 270M task model under half a gigabyte, and a 2B speech model about 4 GB at BF16, all before key-value cache.

Co-location is not free. Every gigabyte a specialist holds is a gigabyte the primary model’s key-value cache does not get, which sets maximum concurrency. A 96 GB card running a 70B model at FP8 with roughly 26 GB of cache headroom has no room for a specialist tier at all. The same card running Gemma 4 26B A4B at INT4 has tens of gigabytes left over. That trade, fewer concurrent sessions on the largest model against a specialist tier that absorbs the calls which never needed the largest model, is the real Phase 1 decision; settled with measured traffic rather than assumed ratios.

Measured traffic has a deadline. Belcak et al. describe a migration path that begins with logging every non-interface model call, filtering personally identifiable information out of the captured data, clustering the logged prompts into recurring task types, and fine-tuning one specialist per cluster [55]. The logging step is the one that cannot be deferred. An organization that does not capture its agentic traffic from the first day of Phase 1 has no dataset to specialize on in Phase 2; no amount of future hardware buys that data back.