Designing checks around AI systems
A practical extension of HyperSanity: evidence, independent checks and the cost of mistakes in an AI workflow.
In HyperSanity, I argue that a judgement needs checks whose failures differ from its own. That question has a direct application in AI architecture: what would detect an error that the generator itself cannot recognise?
Asking a model to reconsider its answer can be useful. Agreement after reconsideration still leaves a question about the evidence. A second answer may inherit the same mistaken premise, retrieved document or framing as the first.
Match the check to the claim
Suppose a system drafts a customer proposal. There are several distinct things to verify. Are the calculations right? Does the proposed service exist? Is the quoted price current? Has someone authorised the commitment?
Those checks should not all be delegated to a general impression that the proposal looks reasonable. Recompute arithmetic. Compare product identifiers and prices against the authoritative catalogue. Ask the responsible person to approve the commercial commitment. A readable draft is useful, but readability does not settle any of those questions.
A check earns trust for a particular reason. A catalogue can be authoritative about current product records while being unable to judge whether the proposed solution fits the customer. Make its scope explicit.
Separate permission from interpretation
A model can interpret a request and propose an action. The surrounding application should enforce who is permitted to perform that action and on which records.
For example, an assistant might propose an account update. The application still needs to validate the caller, the account, the fields and any required approval before committing it. A persuasive explanation is not an access credential.
This separation also makes review easier. The person assessing the proposed change can see the actual fields and values, rather than approve an ambiguous instruction to “take care of the account”.
Keep evidence that can answer a later question
When an output is challenged, can the team reconstruct the relevant input, source versions, proposed action, checks and final decision? Store enough to investigate while limiting access and avoiding unnecessary personal data.
A trace is valuable when it helps answer something specific: which source supplied this number, which check passed it, and what was actually written? An unreadable pile of logs is not the same as an explanation.
Production readiness also extends beyond the model. Google’s ML Test Score offers a testing and monitoring rubric for machine learning systems. Its broader lesson is a useful one: evaluate the surrounding data and infrastructure as well as model behaviour.
Spend review effort where mistakes matter
An internal draft and an irreversible external action should not have identical review requirements. For each step, consider the consequence of an error, how quickly it will be noticed and whether it can be undone.
A team might allow automatic tagging of low-risk internal notes, require review of customer-facing claims, and require explicit approval for a financial or contractual commitment. Those boundaries need to reflect the actual operation, rather than a generic enthusiasm for autonomy.
Review also has a cost. If every harmless step requires attention, people can become less attentive to the consequential ones. Record false alarms and review time as well as missed errors, then adjust the system using the evidence.
Keep the checks under examination
A test can pass because the example is too easy. A reviewer can share the author’s blind spot. An authoritative record can be outdated. No individual safeguard ends the work of judgement.
The practical response is to inspect failures, add examples that expose them, and revisit the assumptions behind the checks. When a system earns more freedom, it should be because its record supports that decision. When its environment changes, that record may need to be rebuilt.
Dependability is maintained through this work. The model is one part of it; the surrounding architecture determines how errors are caught, limited and corrected.