Governing AI at Scale: How We Built a Trustworthy AI Compliance System from the Ground Up
Governing AI at Scale: How We Built a Trustworthy AI Compliance System from the Ground Up
From identifying the gap in how our AI agents were being governed, to building a working compliance framework, forensic assessment tooling, and an operational portal — this is a practitioner’s account of what it actually takes to make enterprise AI trustworthy by design.
The Governance Gap in Enterprise AI
As AI adoption accelerates across enterprise environments, a pattern is emerging that should concern anyone responsible for delivery risk, regulatory exposure, or client trust. Organisations are deploying AI agents into meaningful business processes — finance, HR, customer service, contract review — without a systematic framework for evaluating whether those agents are safe to deploy in the first place.
This is not a criticism of the people making those decisions. The tooling, frameworks, and internal capability to assess AI agents rigorously simply hasn’t kept pace with the speed of deployment. Teams move fast, and governance frameworks tend to arrive after the fact — if they arrive at all.
The question we kept returning to was straightforward: who is actually verifying that our AI agents are legally compliant, ethically sound, operationally robust, and commercially justified — and how are they doing it?
Fusion5 is a Microsoft partner operating across New Zealand and Australia, working with clients across retail, finance, healthcare, and professional services. We have been building AI agents into real business processes at pace. Each of those agents carries exposure: regulatory, reputational, and contractual. A finance agent making automated decisions touches the NZ Privacy Act, the Australian Privacy Act, potentially GDPR if the client has EU operations, and the EU AI Act depending on risk classification. An HR agent involved in screening or evaluation is high-risk under virtually every major framework currently in force or in development.
The honest assessment of where we were: smart people making good individual judgements, some documentation, good intentions — but no repeatable framework and no systematic process. Every new agent deployment relied on the knowledge and diligence of whoever happened to be on that project. That is sustainable at small scale. It is not sustainable at the scale we are moving toward.
Building the Trustworthy AI Archetype
The starting point was definitional. Before you can assess whether an AI solution is trustworthy, you need a precise, shared understanding of what “trustworthy” means in practice — not as an aspiration, but as a set of specific, testable requirements.
We developed the Trustworthy AI Archetype: a structured compliance framework built around four dimensions, each covering a distinct aspect of what responsible AI deployment requires. The archetype currently sits at version 1.4.0, with over 180 individual requirements. Each requirement is classified as MANDATORY, RECOMMENDED, or ASPIRATIONAL, and jurisdiction-specific requirements activate conditionally based on a solution’s declared profile.
The four dimensions are:
LAWFUL
Regulatory compliance across applicable jurisdictions (NZ Privacy Act, AU Privacy Act, GDPR, EU AI Act), contractual obligations to vendors and clients, IP ownership, anti-discrimination, and audit defensibility. These are the legal minimums — non-negotiable regardless of solution maturity.
ETHICAL
Human autonomy, harm prevention, fairness and bias controls, transparency and explainability, privacy by design, and accountability mechanisms. Covers how the solution treats the people it affects — including those who may not be directly interacting with it.
ROBUST
Reliability, safety controls, security posture, observability and monitoring, failure handling, and continuous improvement mechanisms. Covers whether the solution behaves predictably under normal and adverse conditions, and whether problems can be detected and contained.
COMMERCIAL
Value alignment, cost transparency, IP protection, vendor risk and dependency management, and commercial fairness. Ensures deployment decisions are economically sound and that commercial arrangements are properly understood and documented.
Why four separate dimensions? Because AI risks are orthogonal, not additive. A solution can be technically robust but ethically problematic. It can be legally compliant but commercially unjustifiable. Strong performance in one dimension does not compensate for gaps in another — each must meet a minimum threshold independently before deployment is approved.
The Archetype Portal: Making Compliance Operational
A framework on paper has limited value. The objective was to make compliance assessment fast enough and accessible enough that teams would actually use it — not as an end-of-project checkbox, but as a routine part of how we bring AI solutions to production.
We built the Archetype Portal: a Flask-based web application that allows any AI solution to be registered, assessed, scored, and monitored against the archetype. The assessment process follows four steps:
1. Solution Profile. Declare the solution’s jurisdictions, decision type, data subjects, EU AI Act risk tier, and deployment scope. This profile determines which of the 180+ requirements are applicable — a minimal-risk internal tool has a materially different compliance surface than a client-facing agent making consequential decisions in a regulated domain.
2. Evidence upload. Upload whatever documentation exists: design documents, privacy impact assessments, data flow diagrams, testing artefacts, architecture notes. The portal accepts and indexes all of it.
3. AI-powered assessment. Claude analyses the uploaded evidence against every applicable requirement, surfaces gaps, confirms coverage, and identifies what needs attention before the solution can progress.
4. Maturity scoring. Each dimension receives a score from 0 to 5. The portal provides a detailed breakdown of what is evidenced, what is partial, and what is missing — along with recommended next steps.
The Registration Skill: Forensic Compliance Assessment in a Single Command
The most significant capability we built is a Claude Code skill called /register-f5-agent. A developer runs it inside any AI project from their terminal. What the skill does in the background is where the value lies.
Rather than relying on a developer to manually identify and upload relevant files, the skill actively searches the codebase — dimension by dimension, requirement by requirement — and produces structured compliance evidence with exact file paths and line numbers before submitting anything to the portal. It is the difference between providing an auditor with a folder of documents and providing them with a fully annotated evidence map that references specific controls at specific lines of code.
/register-f5-agent doesJurisdiction, risk classification, decision type, data subjects — the fields that determine which requirements apply. The developer confirms or corrects; the skill writes
SOLUTION_PROFILE.yaml to the repository.Searches for Privacy Impact Assessments, breach notification procedures, audit logging implementations, data retention policies, IP ownership documentation, and vendor agreement references. Every finding is cited with a file path and line number. Absences are documented explicitly — a missing PIA is recorded as a gap, not ignored.
Searches for human override mechanisms, bias and fairness testing, explainability patterns, consent flows, and user autonomy controls. The skill distinguishes between documented intent and actual implementation.
Kill switches, circuit breakers, observability hooks, structured error handling, cost tracking implementation, vendor dependency analysis, and fallback patterns. Verifies that controls exist in the code, not just in the documentation.
COMPLIANCE_EVIDENCE.yaml and self-registers.Structured evidence mapped to specific archetype requirements, with file:line citations throughout. Authenticates to the portal via Azure AD and submits directly. The portal’s AI assessment then operates on structured evidence rather than raw files — improving assessment accuracy from approximately 60% to ≥90%.
The entire process runs in approximately five minutes on a well-structured codebase. The output is a registered compliance instance in the portal, a maturity score across all four dimensions, and two YAML artefacts committed to the repository that serve as the living compliance record for that agent going forward.
For context: a traditional compliance review for an AI system of this type — questionnaires, manual evidence collection, consultant review, report production — typically takes two to four weeks and produces a document that is partially out of date by the time it is delivered. This process takes five minutes and can be re-run every time the codebase changes.
We used Claude to assess the compliance of the portal we built to run compliance assessments. It surfaced gaps we had not documented. We addressed them. It is, we acknowledge, a somewhat recursive situation — but the result speaks for itself.
The System in Production
The portal is live and in active use. Below are representative views from the production system.
Dashboard showing registered AI solutions, per-dimension maturity scores, and portfolio compliance health.
Per-instance compliance detail — requirement status (met / partial / missing) with evidence citations for each finding.
Live assessment run — Claude processing structured evidence and mapping findings to archetype requirements in real time.
AOC platform — real-time monitoring of agent fleet health, compliance status, cost telemetry, and escalation management.
From Concept to Production: The Journey
The system was built iteratively over approximately four weeks. The sequence matters because it reflects a deliberate prioritisation: framework before tooling, framework before portal, portal before automation.
/register-f5-agent/register-f5-agent Claude Code skill shipped — forensic codebase analysis against 180+ criteria, self-registration via Azure AD, structured evidence YAML produced and committed to the repository. Archetype updated to v1.4.0.The Maturity Model
Every dimension is scored on a 0–5 scale. The minimum threshold for production deployment is a score of 3 across all four dimensions — independently, not on average. A strong LAWFUL score does not compensate for an inadequate ROBUST score.
| Score | Level | What it means |
|---|---|---|
| 0 | Non-Compliant | No evidence of controls or consideration in this dimension. Deployment not permitted. |
| 1 | Initial | Requirements have been considered but controls are not yet documented or implemented. |
| 2 | Developing | Mandatory requirements are being addressed but material gaps remain. Not production-ready. |
| 3 | Defined | All mandatory requirements met with evidence. Recommended controls in progress. Minimum bar for production deployment. |
| 4 | Managed | Mandatory and recommended requirements met. Strong evidence base. Target posture for any client-facing or consequential solution. |
| 5 | Optimising | Aspirational requirements actively pursued. Continuous improvement loop operating. The solution is measurably improving its own compliance posture over time. |
Where This Is Heading
Agent Operations Centre. A real-time fleet monitoring platform that surfaces the compliance and operational health of every deployed agent in the portfolio. Operators can see at a glance which agents are healthy, which have active compliance gaps, which are drifting from their assessed baseline, and where cost is accumulating relative to business value delivered. Escalation management and containment protocols are built in.
Self-improving framework. Assessment findings are beginning to feed back into the archetype itself. When a real-world assessment surfaces a gap or a pattern not adequately covered by the current requirements, the Guardian agents can propose additions or revisions. The Archetype Coordinator ratifies all changes before they are version-controlled. The framework improves from exposure to the systems it assesses.
Compliance by design. The longer-term objective is to shift compliance from a post-build assessment into an input to the design process. Archetype requirements should be visible to developers at the point they are making architectural decisions — not surfaced for the first time when they run an assessment at the end of a project. The /register-f5-agent skill is a step in this direction; the next step is integrating compliance signals into the development workflow continuously.
Runtime compliance monitoring. At maturity, agents should be monitoring their own compliance posture at runtime — detecting drift from assessed behaviour and triggering alerts or safe-mode responses before a human notices a problem. A ROBUST score of 5 should ultimately mean the agent is capable of identifying its own failure modes before they manifest.
Building AI Responsibly
If your organisation is deploying AI agents into consequential business processes, the governance question is not optional — it is a question of when you address it, not whether. The regulatory environment across Australia, New Zealand, and internationally is tightening. Client scrutiny of AI governance is increasing. And a reactive response to a compliance failure is considerably more expensive than a proactive one.
Fusion5 works with organisations across Australia and New Zealand to design, build, and govern AI solutions responsibly. If you want to understand what a trustworthy AI framework looks like in practice for your context, we are happy to talk.
Talk to Fusion5 →
Comments
Post a Comment