AI Governance and Compliance
The documentation and process obligations around shipping AI. What holds regardless of jurisdiction, and what to write down.
On this page
Specific regulations differ by jurisdiction and change faster than any article can track. What is stable is the underlying set of obligations, which converge on similar requirements regardless of which framework applies.
This covers those. For anything binding, get advice from someone qualified in your jurisdiction — this is not legal guidance.
Risk scales with consequence
The common thread across frameworks: obligations scale with what happens when the system is wrong.
Low consequence. Drafting, summarizing, internal search. A wrong output wastes time. Minimal formal obligation.
Meaningful consequence. Recommendations a person relies on, customer-facing communication, content prioritization. Disclosure and quality expectations.
High consequence. Decisions materially affecting someone — employment, credit, housing, education, medical, legal. Substantial obligations, and in several jurisdictions specific legal requirements.
Prohibited. Some applications are simply not permitted in some places.
Locate your system on that scale before deciding what process it needs. Most internal tooling is at the bottom; anything affecting an individual’s access to something important is not.
What tends to be required
Disclosure that AI is involved. Increasingly a baseline expectation, and reasonable regardless.
Human review for consequential decisions. Automated decisions about individuals typically require a meaningful human role — meaningful being the operative word. A person clicking approve on a hundred cases an hour is not review, and rubber-stamping does not satisfy the requirement it appears to satisfy.
Explainability. Some ability to say why an output occurred. Note that a model’s stated reasoning is not a reliable audit trail — for RAG systems, retrieved sources are far better evidence than generated explanation.
Bias assessment for systems affecting people. Measured across relevant groups, not asserted.
Data handling. Lawful basis, minimization, retention limits, deletion on request — reaching memories, logs, and indexes, not just your primary database.
Records. What the system does, what data it uses, how it was tested, what its limitations are.
What to write down
The documentation burden is smaller than it sounds if you write it while building rather than reconstructing it later.
Purpose and scope. What it does, what it is not for, who uses it.
Data. What goes in, where it comes from, where it goes, retention, whether a third-party provider processes it.
Model and configuration. Provider, model, version, and the fact that hosted models change under stable names.
Testing. Your eval set, results, and known failure modes. Documented limitations are an asset — they demonstrate you looked.
Human oversight. Which actions are gated, who reviews, and the measured approval rate.
Incident history. What went wrong and what changed.
Most of this is engineering documentation you should have anyway. The compliance framing mainly adds the requirement to keep it current.
Recurring hard problems
Vendor terms shift. Provider data handling changes. Whoever tracks your dependency updates should track this too.
Silent model updates. Behaviour changes without a version change, which means your documented testing describes something that no longer exists. Periodic re-evaluation is the only answer.
Deletion across derived stores. An erasure request must reach embeddings, stored memories, and logs. Retrofitting this is genuinely painful — design for it before you need it.
Bias measurement needs group data you may not collect, and may not be permitted to collect. There is no clean resolution; document the approach you took.
Cross-border transfer. Where the model runs may determine whether you can use it.
Proportionality
An internal summarization tool does not need a formal risk assessment. A system deciding loan applications does.
Over-processing low-risk work has a cost: it consumes attention that should go to the systems that matter, and it trains people to treat governance as paperwork. Match the process to the consequence.
What to remember
- Obligations scale with consequence, not with how sophisticated the system is.
- Common requirements: disclosure, meaningful human review, explainability, bias assessment, data handling, records.
- Document purpose, data flows, model version, testing, oversight, and incidents — mostly engineering docs you need anyway.
- Retrieved sources are better evidence than generated explanations.
- Recurring problems: shifting vendor terms, silent model updates, deletion across derived stores, and bias data you cannot collect.
- Match process to consequence; over-processing low-risk systems wastes the attention high-risk ones need.