Generative Artificial Intelligence for Small to Medium Size Organizations

A Comprehensive Strategy for On-Premises Deployment

Emilio Borges

Version 1.0,

Revision historyPlanned topicsHow to cite

Contents

  1. Introduction

    Who the paper is for, what it covers, how its sources are checked, why it was written, and how it defines small and medium size organizations.

    Subsections
    1. Preface
    2. Organization Size Definition
  2. Executive Summary

    Leadership summary of the case for owning core AI capability on premises, the phased build, its five-year cost, and the Phase 1 decision.

    Subsections
    1. Why Adopt AI at All
    2. What Cloud AI Covers and Where It Stops
    3. Control of the Data and of the Record
    4. The Financial Case
    5. A Phased Build
    6. Isolated and Critical Infrastructure Environments
    7. Procurement Timing and Supply Risk
  3. The Rapid Evolution of AI and Why Its Acceleration Is the Strategy

    How shrinking intervals between AI capability shifts, from AlexNet to agents, raise the cost of waiting and favor building capability early.

    Subsections
    1. The Long Runway: 1958 to 2012
    2. 2012: AlexNet and the GPU Inflection
    3. 2017: The Transformer
    4. 2020: GPT-3 and Few-Shot Capability
    5. November 2022: ChatGPT and the Board-Level Inflection
    6. 2023–2024: Open Weights and the Self-Hosting Precondition
    7. 2024–2026: AI as Agent
    8. 2026 Onward: Smaller Models, New Architectures, and the Move to Local Compute
    9. The Acceleration Argument
  4. Owning the Stack: Geopolitical and Economic Arguments for On-Premises AI

    Supply chain, tariff, hyperscaler debt, resource, labor, and EU AI Act pressures that turn owned AI infrastructure into a one-time cost instead of a recurring one.

    Subsections
    1. The geopolitical fracture in AI hardware supply
    2. Tariffs and the cost-pass-through asymmetry
    3. The financing structure behind cloud AI pricing
    4. Energy, water, and the resource footprint of AI at scale
    5. Resource allocation under mispriced infrastructure
    6. Labor market disruption and the talent recruiting signal
    7. On-premises AI as resilience infrastructure
    8. Research and technology for the benefit of all
  5. Cloud AI Providers: Capabilities, Plans, and Strategic Risks

    Capabilities, pricing, customization limits, and availability risks of the Anthropic, OpenAI, Microsoft, Google, AWS, and Cohere AI offerings as of mid-2026.

    Subsections
    1. Anthropic Claude
    2. OpenAI and ChatGPT Enterprise
    3. Microsoft: M365 Copilot, GitHub Copilot, Azure OpenAI, Copilot Studio
    4. Google Gemini, Workspace AI, and the Gemini Enterprise Agent Platform
    5. Amazon Bedrock and Amazon Q
    6. Cohere
    7. Risk Analysis: Where Cloud AI Becomes Untenable
    8. Where This Leads
  6. Rebuttal: “We Already Have [Vendor X]’s AI”

    Why existing vendor AI subscriptions cover commodity work yet cannot deliver owned weights, proprietary training, persistent agents, or full inference audit.

    Subsections
    1. The Workload Framework Decides Where Each Tool Belongs
    2. What Vendor-Bundled AI Does Well
    3. The Customization Ceiling in 2026
    4. What Vendor AI Cannot Do
    5. The Investment Trajectory Argument
    6. Where the Gap Becomes Competitive Advantage
  7. The AI Productivity Paradox: Why Most Deployments Underdeliver

    Why most AI deployments show no measurable productivity gain, and the task and workflow conditions under which controlled studies find one.

    Subsections
    1. The Macro Evidence
    2. The Micro Evidence: The METR Study and Its Update
    3. Why Deployments Underdeliver
    4. The Affirmative Case
    5. Implications for AI Deployments Within Technical Teams
  8. Generative AI Fundamentals for Decision Makers

    Core AI concepts for decision makers, from training and inference costs to GPU memory, quantization, inference engines, and open-weight licensing.

    Subsections
    1. Neural Networks
    2. Large Language Models
    3. Transformers
    4. Graphics Processing Units
    5. Tokens
    6. Context Windows and the KV Cache
    7. Prompts
    8. Context Engineering
    9. Inference vs. Training
    10. Prefill and Decode
    11. Inference Engines
    12. Speculative Decoding
    13. Transformer Architecture Optimizations
    14. Fine-Tuning
    15. Foundation Models
    16. Multimodal Models
    17. Mixture of Experts
    18. Small Language Models
    19. Reasoning Models and Adjacent Architectures
    20. Embeddings
    21. Retrieval-Augmented Generation
    22. Hallucination
    23. Quantization
    24. Model File Formats
    25. Agents, Tools, Harnesses, and Orchestration
    26. Harness Engineering
    27. Loop Engineering
    28. Open Source, Open Weight, Closed Source, and Proprietary Fine-Tuned
  9. Open-Weight Models Ready for Production

    Which open-weight models are ready for on-premises production in 2026, how they compare with closed models, and how license, provenance, and memory shape selection.

    Subsections
    1. The Models That Matter
    2. The Small-Model Tier
    3. Adjacent Model Classes
    4. Hardware Tier Performance Matrix
    5. Quantization and the Quality Floor
    6. What This Means for Phase 1
  10. CPU-Only Model Deployments: Capabilities, Limitations, and Risks

    Why CPU-only inference suits proofs of concept and constrained edge cases but cannot carry a multi-user production AI service.

    Subsections
    1. What CPU Inference Actually Delivers
    2. The Multi-User Problem
    3. Memory Footprint and the Server That Cannot Spare It
    4. Quality Degradation: A More Precise Account
    5. The Migration Trap
    6. Where CPU Inference Belongs
  11. Hardware Deep-Dive: NVIDIA GPU Ecosystem

    NVIDIA GPUs, servers, and power requirements for each deployment phase, with the RTX PRO 6000 Blackwell Server Edition as the Phase 1 recommendation.

    Subsections
    1. Why CUDA Wins the Tooling Argument
    2. The Workstation Tier: DGX Spark and DGX Station
    3. The 8-GPU Rack Tier: DGX B200, B300, and OEM Equivalents
    4. The Mid-Range PCIe Tier and the Phase 1 Inflection Point
    5. The Multi-Node Tier: DGX SuperPOD and Vera Rubin
    6. System Comparison Across Tiers
    7. Power and Cooling as Hardware Selection Gates
    8. The Phase 1 Procurement Position
  12. Hardware Deep-Dive: AMD GPU Ecosystem

    AMD Instinct hardware, ROCm maturity, and benchmark results, and why AMD is a Phase 2 re-evaluation candidate rather than a Phase 1 choice.

    Subsections
    1. The Current Generation: MI350X and MI355X
    2. Prior Generations: MI300X and MI325X in Context
    3. The MI400 Series and the H2 2026 Roadmap
    4. OEM Platforms: Dell, Supermicro, and HPE
    5. ROCm: Where It Works and Where It Doesn’t
    6. Structured Specification Reference
    7. What This Means for the AMD Decision
  13. NVIDIA vs. AMD: Structured Comparison and Hardware Recommendation

    A structured comparison of NVIDIA and AMD that selects NVIDIA for Phase 1 on operational risk and sets dated criteria for an AMD re-evaluation.

    Subsections
    1. The Six Dimensions
    2. The Comparison Matrix
    3. The Phase 1 Recommendation
    4. The Phase 2 Recommendation
    5. When to Revisit This Decision
  14. Full Server Taxonomy: All Systems Required

    The six hardware and four software roles a production on-premises AI deployment needs, with specifications, costs, and facility requirements for each.

    Subsections
    1. Inference Server
    2. GPU Partitioning and Virtualization
    3. Training and Fine-Tuning Server
    4. Embeddings Server
    5. Agentic Compute Server
    6. Data Storage Server
    7. Networking Infrastructure
    8. Out-of-Band Management
    9. Cluster Management, Provisioning, and Monitoring
    10. MLOps Platform
    11. Data Pipeline Infrastructure
    12. LLM Observability Infrastructure
    13. Facility Space, Power, and Cooling
    14. The Master Taxonomy Table
  15. AI Agents and Agentic Infrastructure

    What AI agents require, how they multiply inference demand, which frameworks and protocols to adopt, and the security controls that must accompany tool access.

    Subsections
    1. What an Agent Is, Precisely
    2. Agent Harnesses and Frameworks
    3. Model Context Protocol
    4. Agent Types Relevant to Technical Organizations
    5. Infrastructure Requirements Specific to Agentic Workloads
    6. Security Constraints for Agentic Systems
  16. Multi-User and Agentic Scaling: Concurrency Architecture

    How to size and schedule GPU inference for concurrent users and agents, from vLLM batching and priority scheduling to routing, disaggregation, and scaling triggers.

    Subsections
    1. PagedAttention and Continuous Batching
    2. The Request Scheduler
    3. The Agentic Workload Multiplier
    4. Vertical and Horizontal Scaling
    5. Storage and Networking Under Agentic Load
    6. Monitoring and Capacity Planning
    7. Prefill/Decode Disaggregation
    8. The Phase 2 Scaling Architecture
  17. Phased Hardware Deployment Roadmap

    A three-phase hardware and software roadmap for on-premises AI over five-plus years, with costs, power envelopes, and the triggers for each expansion.

    Subsections
    1. Phase 1 (Months 0 to 12): The Pilot
    2. Phase 2 (Months 12 to 24+): Expansion
    3. Phase 3 (Months 24+ to 60+): Full Deployment
    4. The Phase Summary Table
  18. Expert Voices: External Validation

    What economists, researchers, and practitioners with differing views agree on about AI value, its limits, and the deployment conditions that decide returns.

    Subsections
    1. Karpathy: Verifiability as the Selection Rule
    2. LeCun: The Strongest Architectural Skeptic
    3. Sutton: The Limit the Deployment Has to Design Around
    4. Amodei: The Capability Trajectory
    5. Meijer and Kim: Verification Is the Scarce Skill
    6. Böckeler and Fowler: Field Reports on Non-Deterministic Tools
    7. Torvalds: A Skeptic Who Changed Position
    8. Hightower: The Location Decision Is an Economics Decision
    9. Willison: The Failure Mode That Bounds Agent Design
    10. Nadella: The Test a Business Leader Set for Himself
    11. Acemoglu and Brynjolfsson: An Economics Disagreement
    12. Topol and the Clinical Trial Record
    13. What the Convergence Establishes
  19. Where AI Delivers Real Value: Domain-by-Domain Assessment

    Where AI returns measurable value by domain, from software development and IT operations to engineering design, research, and knowledge continuity.

    Subsections
    1. A New Class of Institutional Asset
    2. Software Development
    3. Software Testing and QA
    4. Technical Documentation
    5. Project Management and Executive Decision Support
    6. Troubleshooting: Software and Hardware Systems
    7. Engineering Design
    8. Scientific and Technical Research Assistance
    9. Lessons Learned and Institutional Knowledge Capture
    10. Realistic Expectations
  20. The ROI Case: Time and Cost Savings

    A five-year return on investment model built on avoided cloud AI spend, with productivity gains layered as conditional upside on the H200 and B300 paths.

    Subsections
    1. The Primary Argument: Avoided Cloud Costs
    2. Layered Upside: Productivity Gains on Favorable Task Types
    3. Compounding Model Value
    4. Data Sovereignty Risk: Qualitative, Not a Line Item
    5. Opportunity Cost for Organizations Not Yet Building
    6. Five-Year Total Cost of Ownership
    7. What This Says About the Investment Decision
  21. Effective AI Use: Practical Guidance for Teams

    Practical guidance for teams on prompting, context management, interface choice, output validation, and the workflow redesign that turns AI use into value.

    Subsections
    1. Prompting: The Skill That Sets the Ceiling
    2. Choosing the Interface
    3. The Limitations, Stated Honestly
    4. Validation Workflows
    5. Building Adoption That Works
    6. Winning Skeptics
  22. Skills and Organizational Readiness

    The skills, learning sequence, certifications, and contractor support that one to a few practitioners need to build and run on-premises AI safely.

    Subsections
    1. The Capability Roadmap, Sequenced
    2. Security Competencies That Belong From Day One
    3. When Consultants and Managed Services Earn Their Place
    4. The Key-Person Dependency Problem
    5. What This Roadmap Buys
  23. Security Architecture for On-Premises AI

    The nine security controls for on-premises AI, from network isolation and access tiers to retrieval, agent, and model supply-chain protections.

    Subsections
    1. Network Segmentation
    2. Access Controls for Model APIs
    3. Data Classification and AI Interaction Policy
    4. Shadow AI Policy
    5. Acceptable Use
    6. Model Output Auditing
    7. Retrieval Corpus and Vector Store
    8. Prompt Injection in Agentic Systems
    9. Supply Chain Security for Open-Weight Models
    10. The Architecture as a Whole
  24. Air-Gapped and Critical Infrastructure AI Deployment

    How to deploy and operate open-weight AI with no external connectivity, covering model transfer, provenance verification, model updates, and agent tool control.

    Subsections
    1. Where Air-Gapped AI Earns Its Place
    2. What Changes When the Network Disappears
    3. Model Selection Inside the Air Gap
    4. Agents Without the Open Internet
    5. Operational Model Management as Release Engineering
    6. What the Regulators Have Said and Have Not
    7. The Honest Frame
  25. Governance as Infrastructure: Building Accountable AI Operations from Day One

    How to build AI governance alongside the infrastructure, with output review tiers, named approvers, logged records, and alignment to NIST, ISO/IEC 42001, and the EU AI Act.

    Subsections
    1. Output Quality Assurance: Trust Calibrated to Use Case
    2. Human Oversight: Accountability, Not Capability
    3. Bias and Hallucination: Permanent Operational Constraints
    4. Internal Policies as Enabling Infrastructure
    5. Why This Is Operational, Not Bureaucratic
  26. Conclusion and Strategic Recommendations

    Final recommendations and the ordered Phase 1 actions, from the facility cooling assessment to a governed, instrumented first deployment.

    Subsections
    1. The Case for Building Now
    2. Recommendations by Audience
    3. Build or Rent

Contents