AI mistakes don’t just cause chaos—they can leak data, corrupt systems, and crush a business.
A safeguard?
Human-in-the-loop (HITL).
Used wisely, it protects data confidentiality, integrity, and availability. But do it incorrectly, and you can slow innovation or even cause more damage than otherwise.
Here are three concrete ways to apply HITL:
1) Default deny
Nothing happens until a human approves. This is for life-or-death or heavily-regulated decisions like:
Surgery
Hiring decisions.
Drone strikes (!).
Important note: laws like Colorado’s SB-205 and NYC Local Law 144 may still apply, even with default deny.
2) Default allow (with intervention window)
Actions run unless a human interrupts within a set time. Balancing speed with oversight, in this mode the system:
Alerts someone before execution.
Gives time to stop harmful steps.
But proceeds by default.
This approach fits medium-risk use cases like:
Database writes.
Business emails.
Public social media posts .
This is also generally how autonomous vehicles operate (if there is still a driver in the car). A human is observing the AI’s decisions but has the ability to override them.
3) Post-hoc review
Humans validate the system’s performance after the fact. By auditing the AI’s outputs against rules or manual benchmarks, humans can determine its performance.
This is good for:
Fraud alerts.
Marketing tests.
Performance tuning.
This approach offers the lowest control but fastest throughput.
Post-hoc review can take the form of iterative human feedback to an AI system at conditions-based or scheduled points during its operation. In the most advanced case, the system would identify to its human operator the most ambiguous cases and ask for guidance to improve its performance over time.
Hybrid approaches
Confidence-sensitivity
An AI system could automatically choose which of the 3 approaches to use based on its confidence threshold. For example, in the cybersecurity context a zero trust policy engine could take the following actions based on how sure it was about the legitimacy of a given login attempt:
<80% confidence: default deny (out-of-band notification to user).
80-90% confidence: default allow (with intervention from security team).
>90% confidence: automatically approved (with post-hoc review).
Context-sensitivity
Similarly, a set of qualitative criteria could determine which approach is required. For example, in healthcare an AI might recommend medications using the below rule:
Prescription: default deny (no recommendation made without human doctor).
Over-the-counter: default allow (with intervention notification to human doctor).
Human-in-the-loop (HITL) can come in three forms
1. Default Deny
2. Default Allow
3. Post-Hoc Review
Importantly, the method you choose is itself a decision. And not making a call can have the same impacts as the wrong one.
Need help choosing the right tool for the job?
StackAware helps AI-powered companies build ISO 42001-compliant governance programs to:
Manage risk.
Build customer trust.
Avoid costly fines and regulatory enforcement actions.
Ready to learn more?
Thanks to Martin Koder for his comment recommending the paper “Humans in the Loop” by Rebecca Crootof, Margot E. Kaminski, and W. Nicholson Price II. Although I think the 3 approaches in my article entirely cover all forms of HITL, the paper did provide inspiration for some of the hybrid methods I described.