AI governance is the set of decisions an organisation makes once, in advance, about which data may reach a model, who approves a new one, what gets logged, who reviews the output, and what happens when it is wrong. Written after a pilot, it records what already happened. Written before, it is what keeps a mistake internal.

The organisations that have ended up in the news over the past five years were not reckless. They were large, competent and well advised, and in most cases the failure was not the model. It was that nobody had agreed in advance who owned the answer.

What is AI governance, and what does it actually cover?

AI governance covers five things: which data may be sent to which model, who authorises a model or tool for use, how outputs are evaluated before and after release, who is accountable for a decision the system informs, and what is recorded so any of it can be reconstructed later.

It is distinct from AI policy. A policy states a position. Governance is the set of controls that make the position true whether or not anyone is watching. Most organisations have written the first and assume it delivers the second.

The distinction shows up in a specific way. A policy says staff must not put confidential information into public AI tools. A control is the technical measure that prevents it, or the sanctioned alternative that removes the reason to try.

Do we need an AI policy before we build anything?

You need both a policy and the controls that enforce it, and the policy on its own will not hold. A policy that has no control behind it becomes a document produced during an audit rather than something that changes what happens on a Tuesday afternoon.

The practical sequence is to classify the data first, decide which categories may reach an external service, then write the policy to match what the controls actually enforce. Writing the policy first produces commitments nobody can keep.

Should we ask suppliers for ISO 42001?

Not as a mandatory requirement yet, in most procurements. ISO/IEC 42001:2023 certifies how an organisation governs AI rather than whether any individual system works, and UKAS granted the first UK accreditation for certifying against it on 15 January 2026, so a buyer who gates on it today produces a very short bidder list.

Score it rather than gating on it, and ask every bidder to describe their own release gate instead: what has to be true before an AI system goes live, and who is accountable for saying so. What the standard covers, how to check a certificate and when to start requiring it are set out in ISO 42001: what an AI management system certifies.

We hold ISO 27001, ISO 27017, ISO 27018 and ISO 20000-1, certified through BSI, plus ISO 9001, ISO 14001 and Cyber Essentials Plus. We do not hold ISO 42001, so a buyer should ask us how AI is governed inside the management systems those certificates already cover.

The failure modes are published, and worth reading

The Open Worldwide Application Security Project maintains a Top 10 for LLM Applications, currently the 2025 edition. The list runs: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption.

Two of those deserve attention from people who are not security specialists.

Excessive agency describes a system given more permission than its reliability justifies. It is the risk that grows fastest as organisations move from assistants that suggest to agents that act, and the control for it is boring and effective: give the system the narrowest set of permissions that lets it do its job, and require a person to authorise anything irreversible.

Misinformation is the failure most teams think they have covered and have not, because the control they have applied is a warning in the interface rather than a check on the output.

Unbounded consumption, the tenth, is the one that arrives as an invoice rather than as an incident, and it is covered separately in our piece on what bot traffic costs you.

Who is accountable when an AI system gets it wrong?

The organisation that deployed it. This has been tested and the answer was not close.

In Moffatt v Air Canada, decided by British Columbia's Civil Resolution Tribunal in February 2024, a passenger relied on the airline's website chatbot, which told him he could apply for a bereavement fare retroactively. He could not. Air Canada argued, in the tribunal's summary, that the chatbot "is a separate legal entity that is responsible for its own actions". The tribunal rejected that, found the airline had not taken reasonable care to ensure the chatbot was accurate, and awarded $650.88 in damages for negligent misrepresentation.

The sum is trivial and the finding is not. An organisation owns what its AI tells the public, in the same way it owns what its website says. Any governance model that assumes otherwise has a gap in it.

What the documented failures actually have in common

Three more cases, all at large and capable organisations, and none of them is a story about a bad model.

In October 2025, Deloitte agreed to refund part of a contract with Australia's Department of Employment and Workplace Relations, after a report on welfare compliance was found to contain fabricated references and an invented quotation attributed to a Federal Court judgment. The department published a corrected version in October 2025 and a further corrected version on 3 February 2026. The control that was missing was verification of the output before delivery, not a prohibition on the tool.

New York City launched its MyCity chatbot in October 2023. In March 2024 an investigation by The Markup found it telling business owners that they could refuse tenants with housing vouchers and take workers' tips, both unlawful. The chatbot remained available after the findings were published, and was withdrawn in February 2026. The interesting failure is not the wrong answers. It is that nobody had agreed in advance what evidence would trigger a withdrawal.

In December 2021 the Dutch data protection authority fined the country's tax administration €2.75 million for unlawful and discriminatory data processing in the childcare benefits system. Among the findings, the authority determined that nationality had been used as an indicator in a system that automatically designated certain applications as risky. Tens of thousands of families were wrongly pursued for repayment over a period of years, and the government resigned in January 2021.

It is the most serious case on this list and the oldest. The control that was missing is the one still absent in most places: somebody independent checking what the variables in a risk model actually do.

The common thread is that in every case a person could have caught it, and no process required one to.

What the UK expects, and how well it is being met

The UK has no cross-sector AI statute. What it has for the public sector is the Algorithmic Transparency Recording Standard, which is mandatory for all government departments and for arm's-length bodies that deliver public or frontline services or deal directly with the public.

Take-up has been slow. The Public Accounts Committee reported on 26 March 2025 that only 33 records had been published as of January 2025, and recommended that DSIT write to the committee within six months on departmental compliance with the standard.

Separately, the Information Commissioner is now under a statutory duty, in force from 12 May 2026, to prepare a code of practice on AI and automated decision-making. That code has not yet been published, and the ICO consulted on draft non-statutory guidance on automated decision-making and profiling, with responses closing in May 2026. An organisation waiting for the code before establishing controls will be waiting a while, and the safeguards in data protection law apply in the meantime.

What should we ask an AI vendor before signing?

Four things. Where the data is processed and under whose jurisdiction. What testing has been done and whether you can see it. What the supplier does when the model underneath them is updated or retired. And where liability for a wrong output actually sits.

The last one is the one to press. Most AI agreements as drafted place that risk with the customer, and after Air Canada the customer is also the party the public will hold responsible.

The full set, with model answers and a way to score them, is in AI due diligence: what to ask a supplier before you sign.

The eight controls to agree before integration

  • Data classification, and which categories may reach which class of model
  • A named approver for any new model, tool or provider
  • An evaluation set built from your own data, run before release and after every model change
  • Defined thresholds for where a human must review, and where a human must decide
  • Logging of inputs, outputs and the model version, with a stated retention period
  • Spend limits per user and per service, with alerting
  • An incident route, including who can withdraw a system and on what evidence
  • A decommissioning plan, covering data, logs and the notice given to users

You will have some of these already. The two most often absent are the withdrawal trigger and the decommissioning plan, which is why systems that are known to be wrong stay live.

Before your next AI project

Take whatever is furthest along in your organisation and check it against the eight controls above. Write down which ones exist as documents, which exist as working controls, and which do not exist. The gap between the first two columns is usually the finding.

If you want to talk through what you find, get in touch.

Frequently asked questions

Is AI governance the same as AI policy? No. A policy states a position. Governance is the set of controls that make the position true in practice. An organisation can have a well-written AI policy and no governance, and it usually finds out at the point something goes wrong.

Does ISO 42001 certify that an AI system is safe? No. ISO/IEC 42001:2023 certifies an organisation's management system for AI, meaning its process for deciding what to build, how to test it and how to withdraw it. It makes no claim about any individual model or product, which is why buyers should ask about both.

Do we need an AI governance framework if we only use off-the-shelf tools? Yes, and arguably more urgently. With a bought tool you control less of the processing chain and still carry the accountability for what it produces, as the Air Canada decision established. The controls change shape but they do not go away.