Human-in-the-Loop Design

Where to put a person in an automated system. Gate by reversibility and confidence, not by task importance.

Full automation fails on the cases that matter. Full manual review discards the value. The design question is where exactly a person belongs.

The answer is not “on important tasks.” It is on irreversible actions and low-confidence outputs — two properties you can actually evaluate.

Gate by reversibility

Sort actions by what a mistake costs.

Reversible and cheap — reading data, running a query, generating a draft, editing a version-controlled file. Let the agent run freely. Review is optional, and gating here destroys the value of automation for no safety benefit.

Reversible but visible — posting an internal comment, creating a ticket, updating a non-critical record. Notify rather than gate. Let it proceed and make it easy to undo.

Irreversible or externally visible — sending an email, issuing a payment, deleting data, deploying, messaging a customer. Confirm first. Always, initially.

This is the primary axis, and it is more useful than importance because it is observable. An important read-only query needs no gate; a trivial-seeming outbound email does.

A design move worth making before adding gates: convert irreversible actions into reversible ones. Draft instead of send. Soft delete instead of delete. Stage instead of deploy. A draft the user approves is a gate that costs nothing extra and often produces a better experience than a confirmation dialog.

Gate by confidence

The second axis. Route uncertain cases to a person and let confident ones through.

The difficulty is that a model’s stated confidence is not calibrated — asking “are you sure” produces a confidence-shaped sentence with no more grounding than the original answer.

Signals that actually work:

Validation failure. Output failing a schema check, a range check, or an arithmetic check. Cheap and reliable.

Disagreement across runs. Generate several times at nonzero temperature; inconsistency indicates genuine uncertainty. Costs several calls, and it is the most honest signal available.

Retrieval quality. For RAG, low similarity scores or a failed retrieval predicts a bad answer better than anything the model says about itself.

Explicit abstention. Give the model a way to say “not confident” or “not in the documents,” and reward using it. It works because it counteracts the trained bias toward answering.

Out-of-distribution input. Unusually long, unusually short, wrong language, or unlike anything in your eval set.

Designing the review itself

A gate that people ignore is worse than no gate, because it creates the appearance of oversight without the substance.

Show what changed, not what was said. A diff, the exact outbound message, the specific record being modified. Not the model’s summary of its intent — that is generated text, not an audit trail.

Include what it was based on. For a RAG answer, the retrieved passages. For an agent action, the reasoning trace. A reviewer cannot judge without inputs.

Make approval cheap and rejection informative. One click to approve; a reason on reject that becomes an eval case.

Batch where it fits. Twenty items in one screen beats twenty interruptions.

Watch for rubber-stamping. If approval rate is near 100%, the gate is theatre. Either the model is reliable enough to remove the gate, or the review is not real — measure it rather than assuming.

Loosening over time

Gates should be the starting position, not the permanent one.

Start with review on everything consequential. Measure the approval rate by category. Where it is consistently high, remove the gate for that category and keep sampling. Where it is low, the gate stays and the underlying problem needs work.

This requires actually tracking approvals — which makes the review queue a source of eval data as well as a safety mechanism. Rejected cases are exactly the examples worth having in a test set.

What to remember

• Gate on reversibility and confidence, not importance — both are observable.
• Free rein on reversible actions; notify on visible ones; confirm on irreversible ones.
• Better than gating: make actions reversible — draft instead of send, soft delete instead of delete.
• Model self-reported confidence is not calibrated; use validation failures, cross-run disagreement, retrieval scores, and explicit abstention.
• Show diffs and inputs, make approval cheap, and treat a 100% approval rate as a broken gate.
• Loosen gates category by category as measured reliability justifies it.

Next: Evaluating Agents