AI Governance Without AI Understanding
A committee that has ratified a charter believes, not unreasonably, that it has acted — and belief in control is precisely what stops the next, harder question from being asked.
Executive Summary
Across the enterprise, the machinery of AI governance has been assembled at remarkable speed: principles, charters, ethics councils, review gates. Almost none of it is built or led by people who understand the systems it is meant to govern. This essay examines a fault line that has widened quietly over the past several years, and that the arrival of generative models this summer has thrown into sharp relief — the distance between the altitude at which oversight is exercised and the altitude at which the risks actually live.
The argument is uncomfortable. Governance conducted without sufficient technical understanding does not merely underperform; it manufactures a convincing simulation of control that is, in one respect, worse than visible neglect, because it reassures. Yet the essay refuses the easy conclusion that boards should become rooms full of data scientists. It holds two truths in tension — that no board can govern what it cannot interrogate, and that no board can or should master the mathematics — and resolves them not by demanding universal fluency but by insisting on understanding at the right altitude: enough to ask the question the mechanism can fail, not merely the question the charter can answer.
The gap persists because it is comfortable, because principles are cheaper than mechanisms, and because the abstractions our suppliers sell us are engineered to make the model disappear. What it reveals about transformation is a pattern we should recognise by now: we are fluent in the vocabulary of change, and far less fluent in its substance.
The charter and the model
Picture the scene, because I have sat through a version of it more than once. A board risk committee receives a paper recommending adoption of an enterprise AI ethics charter. Seven principles: fairness, transparency, accountability, human oversight, privacy, robustness, and a seventh that varies with house style. The paper is well written. It nods to the direction of travel in Brussels, observes that peer institutions have published similar statements, and proposes a standing AI ethics council to steward the commitments. The committee approves it with the quiet satisfaction of a body that has discharged a duty. The charter goes up on the intranet.
Three floors down, a credit-decisioning model has been in production for fourteen months. It was described to that same committee, once, in a single phrase on a single slide, as “92% accurate.” Nobody asked accurate at what, and measured on whom. Had they asked, they would have found that the headline was computed across the entire applicant population, and that within one customer segment the model’s false-decline rate ran at roughly three times the book average — that a cluster of postcodes, standing in silently for something the model had never been told and was never meant to learn, was being quietly and consistently refused. The freshly ratified charter had precisely nothing to say about this, because the charter spoke in principles and the failure lived in a proxy variable.
This is not a story about a bad model, or a negligent committee. The model was ordinary; the committee was conscientious. It is a story about a mismatch of altitude. The governance was exercised at the level of values — do we believe in fairness? — while the risk was sitting at the level of mechanism: which features does this thing consume, what is it really correlating, and what happens to those correlations when next year’s applicants no longer resemble last year’s training data? Between the value and the mechanism there was a floor no one had built, and the charter, for all its seriousness, could not span the gap.
We should be honest that the reassurance is the point of failure. A committee that had done nothing would at least know it had done nothing. A committee that has ratified a charter believes, not unreasonably, that it has acted — and belief in control is precisely what stops the next, harder question from being asked.
What we are really asking for when we ask for understanding
The reflexive objection arrives immediately: you cannot expect a board to understand gradient descent, and you are right, you cannot. But “understanding” is not one thing, and the word has been allowed to do too much work. It is worth separating the altitudes at which one might understand a machine-learning system, because governance fails when it aims at the wrong one.
- The mathematics — the optimisation, the architecture, the loss function. This is the specialist’s floor. No board needs it, and demanding it is a category error that serves mainly to excuse the board from the floors it does need.
- The mechanism — what the system takes in, what it emits, what it is actually optimising for, and how those relate to the decision it influences. What features does the credit model see? What is the objective it maximises, and is that objective the same as the outcome we care about? This floor is not mathematical. It is interrogable in plain language, and it is exactly the floor that most governance skips.
- The failure modes — the specific, recurring ways these systems go wrong: proxy variables that smuggle in the attribute you excluded; distribution shift, when the world stops resembling the training set; feedback loops, when the model’s own decisions reshape the data it later learns from; the brittle, confident wrongness at the edges of the input space.
- The operating envelope — the conditions under which the system was validated, and therefore the conditions outside which its behaviour is simply unknown.
Understanding, for a governor, means fluency at floors two, three, and four — and it is precisely the disclaiming of floor one that has been used to wave away floors two through four along with it. “We’re not technical people” becomes a licence to remain ignorant of the mechanism, when the mechanism is not technical in the sense that matters. To ask a credit model which features it consumes is no more technical than to ask an underwriter which factors they weigh. We have simply allowed the mathematics to cast a shadow large enough to hide everything behind it.
The specialist owns the mathematics. The governor owns the mechanism, the failure modes, and the envelope. The moment a board accepts that these are the same floor — and that not knowing the first excuses not knowing the rest — governance has already failed.
Why the gap is so stable
If the problem were only ignorance, it would be self-correcting; people learn. That it persists, and persists in sophisticated organisations that are not short of talent, tells us it is held in place by structure. Several forces sustain it.
- Principles are cheaper than mechanisms. A statement of principles can be drafted in a fortnight, benchmarked against peers, and displayed. Understanding a portfolio of models — what each consumes, where each was validated, how each fails — is slow, unglamorous, and never finished. Faced with a choice between a deliverable that signals control and a discipline that produces it, organisations under time pressure reliably choose the signal.
- The abstraction is sold deliberately. Much of the AI now embedded in the enterprise arrives through vendors, and a vendor’s commercial interest is to present the model as an appliance: reliable, finished, and above all not your concern. The interface is designed so the mechanism disappears. This is convenient for procurement and corrosive for governance, because you cannot interrogate what has been packaged specifically so that you need not.
- Governance has been organised as a profession apart. Risk, compliance, and audit evolved to oversee processes and controls that could be understood by reading them. They are staffed accordingly. When the object of oversight became a statistical artefact whose behaviour is emergent rather than legible, the function did not re-tool; it reached for the instrument it already had — the policy, the attestation, the sign-off — and applied it to an object that does not yield to attestation.
- The delivery side is complicit in the comfort. Those who build the models are not always eager to be truly governed. A single headline accuracy figure travels well through a steering committee; a candid account of subgroup performance and distribution risk invites questions and delay. It is easier, on both sides of the table, to trade in the reassuring abstraction than in the awkward mechanism.
- Nobody is rewarded for the question that finds the problem. The incentive gradient runs entirely towards shipping. The person who asks, at the gate, what the false-decline rate looks like in the segment nobody mentioned is a source of friction in a process optimised to reduce it. Governance that depends on individual courage against the grain of the incentives will lose, most days, to the grain.
Notice that none of these forces is a failure of intelligence or good faith. They are the ordinary physics of organisations. Which is why exhortation — be more rigorous, care more about ethics — does so little. The gap is not a values deficit to be closed with a stronger charter. It is a structural equilibrium, and only structural change disturbs it.
The aviation objection
There is a serious counter-argument, and it deserves its strongest form rather than a convenient caricature. It runs like this: we govern complex technical systems we do not personally understand all the time, and we govern them well. A board member need not understand the thermodynamics of a jet engine to oversee an airline safely. Aviation is astonishingly safe precisely because governance operates at the level of outcomes, risk appetite, and assurance — not mechanism. Demanding that governors understand the machine is not only unrealistic; it misunderstands what governance is for. Set the tolerances, require the evidence, hold someone accountable, and let the specialists own the internals. Why should machine learning be any different?
It is the best objection there is, and meeting it honestly sharpens the whole argument rather than dissolving it.
Aviation governance works not because governors are ignorant of mechanism but because the system around them has, over decades, digested the mechanism into forms a governor can rely upon. There is a mature body of standardised engineering knowledge; there are certification regimes that test against known failure modes; there is a no-blame incident-reporting culture that surfaces near-misses; there are components with characterised tolerances and histories. A governor can defer to the abstraction because the abstraction has been earned — the understanding exists, densely, in the layer immediately beneath them, and it is independent, contestable, and honest.
None of that substrate yet exists for enterprise machine learning, and certainly not in 2022. The failure modes are not standardised; they recur, but each organisation rediscovers them privately and often too late. There is no incident-reporting culture — models fail silently, and the failures are commercially embarrassing, so they are buried rather than shared. Validation is bespoke and frequently marked by the same team that built the model. The “component” is a statistical object whose behaviour drifts with the data. In aviation, the governor sits atop a mountain of characterised understanding and may safely look only at the summit. In AI, there is no mountain yet — and a governor who adopts the aviation posture of trusting the abstraction is trusting an abstraction that nothing underneath has validated.
So the objection does not overturn the argument; it locates it. The aim is not for governors to understand the mathematics. It is to build, quickly, the layer of earned understanding that would one day let them not have to — the standardised failure taxonomy, the independent validation, the honest incident-sharing. Until that layer exists, deferring to the abstraction is not mature governance. It is faith.
Governing the mechanism, not the charter
What, then, would it look like to govern at the right altitude? Less exotic than it sounds, and mostly a matter of changing the questions the gate asks. A governance process reveals what it truly values by what it makes impossible to skip. Today most AI gates make it impossible to skip the documents: is there an ethics review, is there a DPIA, has the charter been acknowledged. They leave the mechanism entirely optional. The correction is to make the mechanism questions the ones that cannot be waved through.
| The question the charter asks | The question the mechanism asks |
|---|---|
| Has an ethics review been completed? | Which features does the model consume, and which of them could stand in for something we are not allowed to use? |
| Is there human oversight? | On what basis does the human overrule the model, and do they in practice ever do so? |
| Is the model accurate? | Accurate for whom — what does performance look like in the worst-served segment, not the average? |
| Is the system robust? | Under what data conditions was it validated, and how would we know if the world had drifted outside them? |
| Are we accountable? | Who is named against this model’s failure, and what number would tell them it was failing? |
The shift this table describes is not a matter of adding controls. It is a matter of altitude. Every question on the right can be asked and answered in plain language; none requires the mathematics; and each one lands exactly where the charter’s principle floats uselessly above the risk. A board does not need to compute a confusion matrix. It needs to have learned to ask for subgroup performance rather than accepting a single headline figure — and to treat the absence of that breakdown not as a technicality but as the answer itself.
This has organisational consequences. It means governance cannot be wholly centralised in a council that meets monthly and reads papers; the mechanism questions have to be embedded at the point of delivery, asked by people close enough to the model to understand the answers. The central body’s proper role is not to review every model but to own the standard — to insist that no model reaches production without its subgroup performance characterised, its operating envelope documented, and its failure signal defined in advance. And it means the governor’s own literacy has to rise, not to the mathematics, but to floors two through four, until asking the mechanism question is reflex rather than heroics.
A governance regime should be judged not by the quality of its principles but by the sharpness of the questions it refuses to let pass. If the hardest question your AI gate can force is “was an ethics review done?”, you do not have AI governance. You have a filing system with a conscience.
The dawn and the discipline
All of this was already true when the only models in the building were the quiet, embedded, predictive ones — the credit scorers, the fraud filters, the churn predictors, the screening tools that had been making consequential decisions for years while governance looked the other way. What has changed this year is the visibility of the problem, not its nature. Over a single summer, image-generation systems have moved from research curiosity to something a marketing team can use before lunch; code assistants now sit inside developers’ editors; the generative capability that was a specialist’s toy is suddenly, vividly, in front of executives who had never given a model a second thought. The reaction in boardrooms has been a mixture of exhilaration and unease, and the unease is instructive, because it is the first time many governors have felt the gap they have been living with all along.
The temptation now will be to reach, once more, for the charter — to convene the council, publish the principles, and mistake the statement for the control, exactly as before, only faster and under brighter lights. The draft regulation moving through Brussels will add urgency to that reflex, because a published framework is a convenient thing to show a regulator. But the lesson of the embedded models is that the reflex is the trap. Capability is now outrunning comprehension more openly than at any point I can recall, and a governance apparatus that has never been able to interrogate a fourteen-month-old credit model is not suddenly ready for systems that generate rather than merely predict.
This is where the theme returns to something larger than AI. The pattern beneath all of it is the transformation profession’s oldest failing: we are fluent in the language of change and far less fluent in its substance. We can produce the charter, the operating model, the target picture, the principles — the artefacts of understanding — with great facility, and we reach for them precisely because they let us act without the slower, harder work of understanding the thing itself. AI has merely made the failing legible, by placing beneath our governance an object that punishes the gap between the vocabulary and the substance more quickly and more visibly than most.
The dawn does not call for new principles. We have enough principles. It calls for the discipline to govern the mechanism — to build, at last, the earned understanding on the floor beneath the board, and to ask at every gate the question the charter was designed, however unconsciously, to let us avoid. Until we do, we will keep ratifying our fairness while the proxy variable does its quiet work three floors down, and we will keep calling the reassurance control.