AI Incident Governance Needs a Memory, Not Another Set of Principles

AI governance has enough principles to wallpaper a ministry.

Trustworthy AI. Responsible AI. Human-centred AI. Safe AI. Fair AI. Ethical AI. Transparent AI. Accountable AI. The words matter, but the repetition has become suspicious. If principles were enough, we would not need so many of them.

The more urgent problem is memory.

When AI systems fail, who records the failure? Who decides how severe it was? Who verifies what happened? Who learns from it? Who tells other deployers before the same pattern repeats? Who has the authority to demand disclosure? Who protects affected people from becoming case studies without remedy?

Other safety-critical domains understand incident memory. Aviation investigates crashes and near misses. Cybersecurity tracks vulnerabilities, exploits, and incident patterns. Public health tracks outbreaks. These systems are imperfect, but they share a premise: failure must be remembered institutionally, not privately buried.

AI still lacks that culture.

The AI Incident Database was created to index reports of harms or near harms involving AI systems, drawing inspiration from aviation and computer security approaches to learning from failure. Partnership on AI describes the database as a way to help anticipate risks by collecting more than 1,200 reports of intelligent systems causing safety, fairness, or other real-world problems.

This kind of infrastructure is essential because AI harms are often diffuse. Some incidents are spectacular: deepfake scams, wrongful arrests, autonomous vehicle failures, discriminatory classifiers. Others are slow and administrative: welfare errors, biased ranking systems, opaque moderation, automated denial of services, mental health harms from chatbots, unsafe advice, or amplification effects.

Without incident memory, governance becomes reactive. Each failure looks new because the system has no institutional way of remembering old ones.

A serious AI incident governance regime needs five parts.

First, common incident definitions. Not every bad output is an incident, but the threshold cannot be left entirely to corporate discretion. Definitions should include actual harm, near harm, systemic vulnerability, repeated failure patterns, rights impacts, safety risks, and misuse enabled by foreseeable design choices.

Second, severity and confidence scales. Severity should distinguish between inconvenience, rights restriction, financial loss, physical harm, democratic harm, psychological harm, privacy exposure, and discrimination. Confidence should be separate from severity: an allegation may be severe but still under investigation.

Third, mandatory reporting for high-risk deployments. The EU AI Act introduces reporting duties for serious incidents in relation to high-risk AI systems. That is a start, but incident governance should not stop at legal minimums. Organisations deploying AI in sensitive domains should maintain internal incident logs, external disclosure channels, and post-incident reviews.

Fourth, public-interest access to incident data. Researchers, regulators, journalists, and civil society need access to aggregated incident patterns. If incident data stays inside companies, the public learns only what companies choose to admit.

Fifth, remedy and learning loops. Incident reporting should not become reputation management. Affected people need correction, appeal, compensation where appropriate, and guarantees that failure patterns are addressed.

The hardest part is uncertainty. AI incidents are often messy. It may be unclear whether the model caused harm, contributed to it, amplified it, failed to prevent it, or merely appeared in the chain. A 2024 paper on AI incident editing noted that incident reporting must deal with uncertainty around cause, extent of harm, severity, and technical details.

This should not be a reason to avoid incident reporting. It is a reason to design incident reporting honestly.

A good AI incident record should include what is known, what is unknown, what is disputed, what evidence exists, what confidence applies, what mitigation occurred, and what questions remain. Uncertainty should be documented, not hidden.

This is also where AI governance can learn from digital rights work. Communities affected by platform harms have long documented incidents through screenshots, case notes, timelines, and evidence packets because official systems failed them. The challenge is to turn that practice into recognised governance infrastructure without extracting from affected communities or exposing them to further risk.

AI governance does not need another declaration that systems should be safe. It needs institutions that remember when they were not.

A principle says: we value accountability. An incident system asks: accountability for what, when, to whom, and with what evidence?

That is the difference between ethics language and governance infrastructure.