In nineteen years as a Licensed Marriage and Family Therapist, I noticed a pattern that took me a long time to name: the patients who trusted me unconditionally were the ones not making progress. They were outsourcing their judgment to me instead of building their own. The therapeutic goal was never maximum trust — it was calibrated trust, and that same distinction turns out to be the central design problem in every human-AI system I’ve built since.
The Same Dynamic in Human-AI Systems
Over-trust and under-trust are both failures, and they fail in opposite directions. Over-trust — what the research calls automation bias — leads users to accept AI outputs they should have scrutinized, because the system seems authoritative and questioning it takes effort. Under-trust — algorithm aversion — leads users to ignore correct AI recommendations after a single bad experience, falling back on worse human judgment. A well-designed system has to fight both at once, which means the target is not somewhere on a line between trust and distrust. The target is trust that tracks reality.
Defining Calibrated Trust
Calibrated trust is trust that scales with the AI’s actual reliability in a specific domain and a specific task type. It is not a global setting. A model can be excellent at classifying tax scenarios and useless at predicting human behavior, and a calibrated user trusts it accordingly — high confidence in the first domain, near-zero in the second. The work of design is making that task-specific, domain-specific reliability visible, so the user’s trust has something accurate to attach to instead of a vague sense that the machine is smart.
What the Automation Bias Research Shows
Mosier and Skitka’s automation bias studies, adapted from aviation, found something counterintuitive — users in automated environments made more errors not by actively choosing wrong, but by passively accepting incorrect system outputs. The automation seemed authoritative, so the human stopped checking. The error was an error of omission, a failure to verify, which is exactly the dynamic I watched in the therapy room with patients who stopped examining their own thinking because they assumed I had already done it for them.
The Other Failure: Algorithm Aversion
Dietvorst, Logg, and colleagues documented the mirror image. After watching an algorithm fail just once, users preferred worse human judgment over better algorithmic judgment — even when the algorithm’s overall accuracy was demonstrably higher. One visible mistake erased a track record of being right. That asymmetry matters for design, because it means a single uncaught error doesn’t just cost you that decision — it can collapse a user’s willingness to use a tool that is, on balance, helping them.
Design Principles for Calibrated Trust
Three principles have held up across my builds. Show confidence scores anchored to historical calibration — not a bare percentage, but a statement like “this model is correct 87% of the time on cases like this one,” which gives the number meaning. Surface the cases where the model tends to underperform, so users learn its blind spots before they get burned by one. And require human sign-off on high-stakes decisions regardless of confidence, because some decisions need an accountable person no matter how good the model is. I explored the behavioral-science foundation for this approach in my post on why a therapist builds AI systems.
The Lockheed Example
On a major defense aircraft program, the variance detection model I helped build required an analyst sign-off on any flag above $500,000. That threshold had nothing to do with the model being untrustworthy — its accuracy was strong. It existed because the accountability structure demanded a human decision-maker in the loop for material amounts. The model could surface the flag, explain its reasoning, and rank it by confidence, but a person owned the decision. That is calibrated trust built into process: automation for the volume, human judgment for the consequence.
What Transfers
Calibrated trust is the design goal — not maximum trust, not zero trust, but trust that matches demonstrated reliability. Whether the relationship is between a patient and a therapist or a user and a model, the healthy state is the same: confidence that rises and falls with actual track record, and a structure that keeps the human’s judgment engaged rather than dormant. Build systems that earn trust where they deserve it and invite scrutiny where they don’t.
→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.
Leave a Reply