Governance as Infrastructure: Building Accountable AI Operations from Day One

Output Quality Assurance: Trust Calibrated to Use Case

Trust in AI output is not binary. Quality thresholds must be defined by what the output is for, not by what the model is capable of. Three tiers cover the working set of internal use cases.

Internal research and ideation is at the lowest threshold: early-stage drafts, summaries the requester will read and revise, and exploratory analysis. The human reads the result before it goes anywhere and acts as the quality gate.

Output that enters customer-facing documents, external communications, or work products colleagues will rely on requires review against the source material. The reviewer checks whether the model fabricated facts, misrepresented source documents, or imported claims the organization cannot stand behind. Roig’s 172-billion-token evaluation of 35 open-weight models on document question answering measured a best case of 1.19% fabrication at 32K context, with top-tier models running 5% to 7%; fabrication nearly triples at 128K and exceeds 10% for every model tested at 200K [8]. That work is a preprint that has not been peer-reviewed as of this writing, and its author writes from a commercial AI infrastructure vendor, but the scoring is deterministic against ground truth and the method is reported in full, which is more than most hallucination benchmarks offer. The design consequence is a retrieval budget; not a context budget. A retrieval pipeline that fills a 200K context window because the model is capable of processing a context window that size will more than likely result in the model fabricating information from the unnecessary extra tokens fed into its context. Cap retrieved context at the smallest window that answers the question and put a reviewer at the tier where the output moves beyond the organization’s control.

Decisions carrying financial, legal, or personnel consequence require human sign-off regardless of how confident the model sounds. Model confidence and model correctness are separate variables. NIST AI 600-1 names the failure directly under Human-AI Configuration, where over-reliance and automation bias are the documented risk rather than model error alone [4]. Treating fluency as evidence produces the worst incidents.

The prompt-and-response records specified in the security architecture section is what makes these thresholds enforceable after the fact. A tier commitment with no stored record of which tier a given interaction fell into is a statement of intent, which is why the policy, logging, and audits are designed together.

Human Oversight: Accountability, Not Capability

For consequential decisions, such as personnel actions, financial commitments, legal filings, public communications, and production deployments, it is perfectly acceptable to use AI models for assistance, but these models should never be the ones to make the final decisions. This is not a claim about capability. Models will continue to improve, and some of these decisions will eventually fall within their technical competence. Human oversight assigns people within the organization the responsibility of making sure these systems operate securely, within specification, and behave as expected, which includes holding someone accountable when something goes wrong that could have been prevented. This accountability cannot be assigned to a system that has no stake in the outcome of a decision it made.

The European Commission’s High-Level Expert Group on AI defined the three oversight governance mechanisms still in use. Human-in-the-loop (HITL) is the capability for human intervention in every decision cycle. Human-on-the-loop (HOTL) is the capability to intervene during the design cycle and to monitor operation. Human-in-command (HIC) is oversight of the system’s overall activity, including its broader economic, social, legal, and ethical effects, together with the authority to decide whether to use it at all and to override its output [9]. The decision categories above belong in HITL or HIC configurations. They do not belong on the loop, where the human reviews aggregate behavior rather than approving specific actions. Fraud detection alerting can run on the loop. Approving an employee’s termination cannot.

Article 14 of the EU AI Act places the design-side oversight obligation on providers of high-risk systems; Article 26(2) places the operational oversight obligation on deployers, requiring that they assign human oversight to natural persons with “the necessary competence, training and authority, as well as the necessary support” [5]. Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on July 27, 2026 and deferred the Annex III high-risk obligations from August 2, 2026 to December 2, 2027, with Annex I embedded systems moving to August 2, 2028; the Article 50 transparency obligations were not deferred [10]. A sixteen-month deferral is a timing change, not a repeal, and it is shorter than the horizon of a phased on-premises build. Organizations outside the Act’s scope should read Article 26(2) as the operationally correct standard anyway: a named person, competent in the specific system, with authority to override it.

Bias and Hallucination: Permanent Operational Constraints

Two technical limitations of current large language models (LLMs) require a continuous governance response.

Bias enters these models through training data and gets amplified through reinforcement learning from human feedback (RLHF) and fine-tuning passes. Mehrabi et al. established the taxonomy of sources, from sampling and measurement bias in data through algorithmic and user-interaction bias [11]. Ferrara’s survey extends it to generative systems, where models reproduce and amplify societal stereotypes carried in their pretraining corpora [12]. Wyllie, Shumailov, and Papernot demonstrated at FAccT 2024 that iterative training on model-generated data produces fairness feedback loops that converge toward majority-group representation [13]. Their result concerns synthetic data specifically, and extending it to fine-tuning on internal organizational corpora is an analogy the paper does not test. The analogy is worth acting on regardless, because internal communications, performance reviews, hiring records, and technical documentation are curated subsets of human output that encodes the organization’s own history. Governance policy for Phase 2 fine-tuning should require a demographic and representational audit of training data before any run, with the findings retained beside the resulting artifact in the MLflow registry specified in the server taxonomy section, where dataset versions are already recorded per run.

Hallucination is the second limitation, which current research suggests is nowhere near solved. Dang, Tran, and Nguyen found that hallucination attributes to both prompt design and intrinsic model behavior, with vague prompts producing a 38.3% hallucination rate against 18.1% under chain-of-thought prompting across the models they tested, and no single mitigation eliminates it [14]. Their evaluation covers five open models at or below 67B parameters at 100 examples per model and dataset, so the absolute rates are dated; the attribution result is the durable finding. Magesh et al. measured 17% to 33% hallucination in commercial legal research tools that advertised retrieval-augmented generation (RAG) as a fix, against 43% for GPT-4 on the same queries, testing the May 2024 product versions [15]. Retrieval reduced the rate substantially, but it did not eliminate it. At the frontier, the AA-Omniscience benchmark reported in the AI Index puts hallucination rates across 26 leading models between 22% and 94% on open-ended knowledge questions [1]. Organizations have noticed: inaccuracy is now the risk most often rated relevant, at 74% in 2025, up fourteen points in a year, with 71% reporting active mitigation [1].

The governance response is not to wait for a fix. It is validation requirements tied to the use-case tiers above, interface warnings that confident output may be wrong, and review workflows that catch fabrications before they propagate into downstream documents. NIST AI 600-1 anticipates exactly this architecture under its Confabulation and Harmful Bias and Homogenization categories [4].

Internal Policies as Enabling Infrastructure

Acceptable use policies, attribution requirements, and prohibited use categories give workers the clarity to use AI confidently. Framing them as restrictions gets the causation backwards. People will use AI whether a policy exists or not; the policy decides whether they do it with guidance or by guessing. The AI Index survey supports the reading: knowledge and training gaps are the top-cited obstacle to implementing responsible AI at 59%, ahead of budget constraints at 48% and regulatory uncertainty at 41% [1].

A working policy set covers four areas. The acceptable use policy specifies what AI may be used for, what needs manager or governance approval, and what is prohibited; this is the document the typical worker actually reads. Attribution requirements address when AI-assisted output used externally must be disclosed, a question with no settled industry answer, which is precisely why the organization should resolve it internally rather than leave it to individual judgment. Prohibited use categories cover discriminatory applications, such as employment, credit, or housing decisions made in ways that disparately affect protected classes; substitution for licensed professional advice in matters requiring legal, medical, or financial judgment; and surveillance applications used on the organization’s own workforce or anyone else for that matter. Sector-specific extensions apply on top for regulated industries.

The most mature sector regime shows why those extensions solve less than expected. On April 17, 2026, the Federal Reserve, the Office of the Comptroller of the Currency (OCC), and the Federal Deposit Insurance Corporation (FDIC) jointly issued SR 26-2, revised supervisory guidance on model risk management superseding the SR 11-7 framework that US banks had built programs around for fifteen years [16]. Two features of it matter here. The agencies expect it to be most relevant to banking organizations above $30 billion in total assets, which excludes nearly every organization this paper addresses, and a scope footnote places generative and agentic AI models “not within the scope of this guidance” on the grounds that they are novel and rapidly evolving [16]. The principles continue to apply to traditional statistical models and to non-generative AI, and for anything generative the agencies hand the question back: the organization’s own risk management practices determine the controls. Healthcare deployments face the US Food and Drug Administration’s lifecycle guidance for AI-enabled device software functions, which was still in draft as of mid-2026 [17]. Critical infrastructure operators have the joint principles the Cybersecurity and Infrastructure Security Agency (CISA) issued with eight partner agencies in December 2025 [18]. The pattern repeats across all three: the sector regulator either does not reach a mid-sized organization or does not yet cover generative systems.

The policy document is half the work. The other half is making the compliant choice the easy one. An approved internal chatbot that is slower or less capable than the online consumer alternative an employee already uses will not redirect behavior, whatever the policy says. That is the same argument the security architecture section makes about shadow AI from the other direction.

Why This Is Operational, Not Bureaucratic

The argument that AI governance is overhead reverses cause and effect. A deployment without it produces outputs whose reliability is unknown at the point of use and decisions nobody can reconstruct under scrutiny. It runs fast until the first incident and then absorbs cost in proportion to what the record cannot explain. Organizations report the same relationship when asked what blocks them from scaling agentic systems: security and risk concerns lead at 62%, well ahead of technical limitations and regulatory uncertainty at 38% each [1]. Governance is the mechanism that enables safe and secure scaling, whereas its absence is what halts it.

Sequencing follows from that. A policy can be written retroactively; the record cannot. Every quarter that a deployment runs without defined output tiers, named approvers, and captured prompt-and-response pairs accumulates decisions the organization can describe later but never produce as evidence. Define the tiers, assign an owner to each control, and set the review cadence before the first production query, or accept the consequences of an organization that cannot produce detailed documentation of what its models did in production.