The Behavioral Science Argument for Explainable AI

As a Licensed Marriage and Family Therapist who spent 19 years building AI systems inside a defense contractor, I can tell you that the most dangerous assumption in AI explainability design is the belief that showing people more information makes them understand better. It doesn’t — and the cognitive science has been clear on this for decades. Humans don’t form beliefs by evaluating evidence and then reaching conclusions; they reach conclusions first, usually within seconds, and then work backward to construct a rationale that justifies what they already decided. Post-hoc rationalization isn’t a flaw in a few people — it’s the default operating mode of the human mind.

Why “Just Show the Model Output” Fails

The standard approach to AI transparency is to surface the output with some kind of confidence score or feature attribution attached. The implicit theory is: if users can see why the model decided what it decided, they’ll calibrate their trust appropriately. The problem is that people don’t use explanations to calibrate trust — they use them to confirm the conclusion they already formed. If the model output looks plausible, they rationalize the explanation as supporting it. If the output looks wrong, they rationalize the explanation as evidence of model failure. The explanation becomes post-hoc justification for whatever the user already believed, which means it’s doing exactly zero calibration work. This is not a theory — it’s what happens in practice, and it’s what I watched happen in clinical settings long before I was building AI systems. Patients who received detailed diagnoses with extensive supporting evidence were not more compliant than patients who received clear, direct recommendations in their own language. In fact, they were sometimes less compliant, because more information gave them more material to selectively interpret.

The Two Failure Modes from I/O Psychology

Industrial/Organizational psychology names two failure modes that show up in every AI deployment I’ve worked on. The first is automation bias — users trust the AI output even when it is clearly wrong, because the system has established a track record of being right and the user’s vigilance erodes over time. The second is algorithm aversion — users reject AI output even when it is clearly right, because a visible failure early in the deployment destroyed their confidence and they never rebuilt it. Both failure modes are trust calibration problems, and neither one is solved by adding more explanation text to the interface. Automation bias gets worse when the explanations sound authoritative, because authoritative-sounding text reinforces rather than disrupts overconfidence. Algorithm aversion gets worse when explanations expose model uncertainty — if a user who already distrusts the system sees “confidence: 61%” attached to a prediction, that number confirms their suspicion that the model doesn’t really know what it’s doing.

What Actually Works: Domain Language, Not Model Language

The principle that transfers from clinical practice to AI explainability is this: speak the patient’s language, not the diagnostic manual’s language. A therapist who explains a diagnosis using DSM-5 criteria loses the patient in the first 30 seconds. A therapist who says “this looks like the same pattern we saw last spring, when work stress was highest and sleep dropped below six hours” keeps the patient engaged because the explanation is constructed from their own experience. The same principle applies to AI explanations — but almost no one applies it. At Lockheed, the early version of our variance detection system surfaced explanations using model terminology: anomaly scores, deviation coefficients, feature weights. The finance analysts looked at those explanations, nodded, and then completely ignored the model outputs within three weeks. After we rebuilt the explanation layer to use their domain language — “this cost line exceeded the 3-year program average by $142,000, which is outside the normal range for this phase of production” — adoption recovered and stayed above 80% for the rest of the program. Nothing in the model changed. Only the explanation framing changed. That’s the behavioral science argument for explainable AI: the explanation is a communication design problem, not a technical transparency problem. You can read more about how that kind of thinking applies across my enterprise AI work in my post on deploying AI in regulated environments.

What Transfers

Clinical communication training teaches one thing above everything else: you cannot inform someone into a different belief — you have to meet them inside the belief system they already have and build from there. That’s not manipulation; it’s the actual mechanism by which understanding happens. An AI explanation that says “SHAP value = 0.73 for feature: cost_variance” is not communicating anything to a finance analyst. An explanation that says “this decision looks different from your last 12 similar decisions” is communicating something real, in a frame the user can act on. The lesson extends beyond AI — any system that asks humans to change their behavior based on algorithmic output needs to be designed around how trust is actually built and maintained in human cognition, not around how transparency is theoretically supposed to work. If you’re building AI systems that touch consequential decisions and you haven’t read the automation bias literature, that’s the first gap worth closing. If you want to understand more about why I approach AI from a behavioral science foundation, that’s exactly what I wrote about in this post on why a therapist builds AI systems.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.