Tag: behavioral-science

As a licensed Marriage and Family Therapist who spent years doing couples and family work before pivoting into AI systems, I can tell you that reflective listening is not a soft skill — it’s a precision technique with documented, measurable outcomes, and almost nobody building conversational AI has been trained in it.

The research on therapeutic alliance — the quality of the relationship between therapist and client — consistently shows it accounts for roughly 30% of therapy outcomes, separate from the specific intervention being used. The mechanism behind that effect is largely communication design: how the therapist structures questions, how they signal understanding, when they respond versus when they wait. These are learnable patterns. And they translate directly into how conversational AI should be designed.

Reflective Listening Is Not Just Paraphrasing

Reflective listening has a specific structure: you paraphrase what you heard, then check for accuracy. “So what I’m hearing is X — is that right?” That check is not optional. It signals to the other person that you’re trying to understand them, not perform understanding. The distinction matters clinically and it matters in AI design.

Most conversational AI systems skip this entirely. They receive input, generate a response, and proceed — sometimes asking a clarifying question at the end as a kind of afterthought. The therapeutic sequence is inverted: demonstrate understanding first, then respond. The response is only useful if it’s calibrated to what the person actually meant, and you can only calibrate it if you’ve confirmed your interpretation before acting on it.

There’s a related concept that AI designers routinely confuse: validation versus agreement. “That makes sense given what you described” is not agreement. It’s not endorsing the person’s conclusion. It’s acknowledging that their experience is coherent — that a reasonable person in their situation might feel or think what they’re describing. In therapy, this distinction is fundamental. Agreeing with a client’s distorted belief reinforces the distortion. Validating their experience of having that belief helps them examine it. Conversational AI that conflates the two either capitulates to whatever the user believes or sounds dismissive when it doesn’t — there’s no middle path unless the design explicitly encodes it.

The Interrogation Problem in Chatbot Design

The default mode for most chatbot intake flows is closed questioning — yes/no, multiple choice, structured menus. This is procedurally efficient and psychologically counterproductive. Closed questions put users in a reactive posture. They produce minimal, guarded responses. They signal that the system already knows what categories the answer will fit into, which causes users to fit their answer into those categories even when their actual situation doesn’t.

Good therapeutic intake starts with open questions and follows the client’s lead. “What brings you here today?” is not a soft opener — it’s a deliberate technique that gives the client control of the initial framing. What they choose to say first, and how they say it, is diagnostic information that a closed-question flow destroys before you ever get it. The AI equivalent is letting the user describe their problem in their own terms before offering any categorization or structure.

Three design patterns from clinical practice apply directly here. Open before closed: start with an expansive prompt and narrow only after the user has established their own frame. Name before normalizing: reflect the specific language the user chose before offering your framing — if they said “overwhelmed,” use “overwhelmed,” not “stressed” or “busy.” Check understanding before advising: never make a recommendation before demonstrating, explicitly, that you understood the problem. Each of these patterns is a deviation from how most AI conversation flows are currently built, and each one corresponds to a documented failure mode in therapeutic communication that produces worse outcomes.

Emotional Flooding and Degraded Inputs

John Gottman’s research on emotional flooding — what happens physiologically and cognitively when someone is overwhelmed — is directly applicable to AI system design. When clients are flooded, their communication degrades: they become less precise, more repetitive, and their inputs contain less usable signal. Pushing forward with more questions in that state produces worse data. The clinical intervention is to slow down, regulate, and return to the content when the person can engage with it.

AI systems have analogous signals available: response latency patterns, session abandonment rates, unusual repetition in user inputs, dramatically shorter responses after longer ones. These are detectable. A system designed with therapeutic principles would recognize them as flooding indicators and shift its conversational mode — shorter prompts, more validation, less cognitive load — rather than continuing to push the standard flow. The behavioral science case for this approach is well established; the implementation is an engineering choice that most teams haven’t made yet.

What Transfers

Therapeutic listening as a conversational design framework applies to any AI system taking natural language input from humans in non-trivial situations — medical intake, financial advising, legal question-answering, HR support tools. The stakes in these contexts mean that users who feel processed rather than understood will withhold information, distort their responses to fit perceived categories, or disengage entirely. The cost of that outcome isn’t just a poor user experience; it’s systematically degraded input data flowing into whatever downstream system the AI is feeding.

Designing for genuine understanding — not the performance of understanding — requires knowing what genuine understanding looks like in practice. That’s what nineteen years of combined clinical training and enterprise systems work taught me: the human side of the interface is not a UX problem. It’s a behavioral science problem, and it has behavioral science solutions.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

Family systems therapy taught me not to diagnose individuals in isolation — a child’s behavior is only legible in the context of the family system that produces and reinforces it. I’ve come to think that this is precisely the conceptual frame that most AI bias discussions are missing, and missing it produces interventions that move the problem rather than resolve it.

The Category Error at the Center of the Bias Conversation

The dominant framing of AI bias treats it as a model property — something present in the weights, detectable with fairness metrics, and removable through debiasing techniques applied to the model. This framing is wrong in a precise way: it mistakes a system-level phenomenon for a component-level defect. The bias you can measure in a model’s outputs is not located in the model. It’s produced by a system — the training data, the feedback mechanisms, the deployment context, and the humans who interact with all of it.

This matters because component-level interventions applied to system-level problems produce predictable results: the problem moves. You apply adversarial debiasing and the model’s aggregate fairness metric improves on your evaluation set while the bias redistributes to subgroups you weren’t measuring. You reweight the training data and you reduce one source of bias while introducing a new imbalance in a different direction. Neither outcome means the technique was wrong — it means the framing that led you to apply only that technique was incomplete.

Four Systemic Bias Sources That Precede the Model

The first source is historical data encoding past human decisions. If a hiring model is trained on ten years of hiring decisions made by a team with a documented preference for candidates from specific universities, the model learns that preference precisely. There is no malfunction. The model is performing exactly as trained. The bias is in the decision record, not in the algorithm.

The second source is feedback loops. A model’s outputs shape the next round of training data. A content recommendation system that surfaces more of what users engage with will amplify engagement patterns — including the ones that reflect anxiety, outrage, or compulsion rather than genuine preference. The model is not broken. The feedback loop is producing exactly what feedback loops produce.

The third source is deployment context mismatch. A model trained primarily on data from Group A deployed in a context where Group B is the primary user population will generalize poorly to Group B — not because it was built with malicious intent, but because it was built with insufficient attention to the difference between its training distribution and its deployment distribution. This is a systems design failure that shows up as a model failure.

The fourth source is user behavior adaptation. Users who interact with AI systems over time learn the system’s tendencies and adjust their behavior accordingly — which changes the data the system sees, which changes what the system learns in the next training cycle. The model and its users are a coupled system, and treating the model as a static artifact misses the dynamic entirely.

The Clinical Parallel: Confirmation Bias Is Not a Bug in Clinicians

Confirmation bias in clinical practice is well-documented. Licensed clinicians with years of training consistently show a tendency to confirm their initial diagnostic hypotheses — to weight evidence that supports their first impression more heavily than evidence that challenges it. This is not a character flaw. It is a feature of how human cognition works under uncertainty. Structured clinical interviews and standardized diagnostic protocols exist precisely to force against this tendency — to build a system-level constraint that counteracts a predictable individual-level pattern.

The lesson is not “clinicians are biased, therefore don’t trust them.” The lesson is “predictable cognitive patterns require structural countermeasures, not just individual awareness.” The same logic applies to AI systems. Knowing that your training data encodes historical human decisions is not sufficient — you need structural interventions in the data pipeline, the feedback mechanism, and the deployment monitoring that function as countermeasures, not as one-time fixes.

At Lockheed, auditing an AI system for systematic patterns required going back to the raw training data, not just examining model outputs. That process took six months for one system. The output-level audit had found nothing alarming. The data-level audit found several years of historical decisions with a consistent pattern that the model had faithfully reproduced. The model was working as designed. The design was the problem.

What Transfers

The shift in question that systems thinking requires is from “is this model biased?” to “what are the feedback loops in this system, and what do they produce?” The first question has a tractable-sounding answer. The second question is harder and more useful. It asks you to map the full system — data generation, model training, deployment context, user behavior, retraining inputs — and identify where bias enters, where it’s amplified, and where your interventions are actually redirecting it rather than eliminating it.

This is the contribution that behavioral and clinical training makes to responsible AI work — not a set of soft skills layered on top of the engineering, but a different set of questions to ask before the engineering begins. I’ve written about the broader case for this perspective in why a therapist builds AI systems. The short version: systems that produce harmful outcomes usually aren’t built by people who intended harm. They’re built by people who were asking the wrong unit of analysis.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

As a Licensed Marriage and Family Therapist who spent 19 years building AI systems inside a defense contractor, I can tell you that the most dangerous assumption in AI explainability design is the belief that showing people more information makes them understand better. It doesn’t — and the cognitive science has been clear on this for decades. Humans don’t form beliefs by evaluating evidence and then reaching conclusions; they reach conclusions first, usually within seconds, and then work backward to construct a rationale that justifies what they already decided. Post-hoc rationalization isn’t a flaw in a few people — it’s the default operating mode of the human mind.

Why “Just Show the Model Output” Fails

The standard approach to AI transparency is to surface the output with some kind of confidence score or feature attribution attached. The implicit theory is: if users can see why the model decided what it decided, they’ll calibrate their trust appropriately. The problem is that people don’t use explanations to calibrate trust — they use them to confirm the conclusion they already formed. If the model output looks plausible, they rationalize the explanation as supporting it. If the output looks wrong, they rationalize the explanation as evidence of model failure. The explanation becomes post-hoc justification for whatever the user already believed, which means it’s doing exactly zero calibration work. This is not a theory — it’s what happens in practice, and it’s what I watched happen in clinical settings long before I was building AI systems. Patients who received detailed diagnoses with extensive supporting evidence were not more compliant than patients who received clear, direct recommendations in their own language. In fact, they were sometimes less compliant, because more information gave them more material to selectively interpret.

The Two Failure Modes from I/O Psychology

Industrial/Organizational psychology names two failure modes that show up in every AI deployment I’ve worked on. The first is automation bias — users trust the AI output even when it is clearly wrong, because the system has established a track record of being right and the user’s vigilance erodes over time. The second is algorithm aversion — users reject AI output even when it is clearly right, because a visible failure early in the deployment destroyed their confidence and they never rebuilt it. Both failure modes are trust calibration problems, and neither one is solved by adding more explanation text to the interface. Automation bias gets worse when the explanations sound authoritative, because authoritative-sounding text reinforces rather than disrupts overconfidence. Algorithm aversion gets worse when explanations expose model uncertainty — if a user who already distrusts the system sees “confidence: 61%” attached to a prediction, that number confirms their suspicion that the model doesn’t really know what it’s doing.

What Actually Works: Domain Language, Not Model Language

The principle that transfers from clinical practice to AI explainability is this: speak the patient’s language, not the diagnostic manual’s language. A therapist who explains a diagnosis using DSM-5 criteria loses the patient in the first 30 seconds. A therapist who says “this looks like the same pattern we saw last spring, when work stress was highest and sleep dropped below six hours” keeps the patient engaged because the explanation is constructed from their own experience. The same principle applies to AI explanations — but almost no one applies it. At Lockheed, the early version of our variance detection system surfaced explanations using model terminology: anomaly scores, deviation coefficients, feature weights. The finance analysts looked at those explanations, nodded, and then completely ignored the model outputs within three weeks. After we rebuilt the explanation layer to use their domain language — “this cost line exceeded the 3-year program average by $142,000, which is outside the normal range for this phase of production” — adoption recovered and stayed above 80% for the rest of the program. Nothing in the model changed. Only the explanation framing changed. That’s the behavioral science argument for explainable AI: the explanation is a communication design problem, not a technical transparency problem. You can read more about how that kind of thinking applies across my enterprise AI work in my post on deploying AI in regulated environments.

What Transfers

Clinical communication training teaches one thing above everything else: you cannot inform someone into a different belief — you have to meet them inside the belief system they already have and build from there. That’s not manipulation; it’s the actual mechanism by which understanding happens. An AI explanation that says “SHAP value = 0.73 for feature: cost_variance” is not communicating anything to a finance analyst. An explanation that says “this decision looks different from your last 12 similar decisions” is communicating something real, in a frame the user can act on. The lesson extends beyond AI — any system that asks humans to change their behavior based on algorithmic output needs to be designed around how trust is actually built and maintained in human cognition, not around how transparency is theoretically supposed to work. If you’re building AI systems that touch consequential decisions and you haven’t read the automation bias literature, that’s the first gap worth closing. If you want to understand more about why I approach AI from a behavioral science foundation, that’s exactly what I wrote about in this post on why a therapist builds AI systems.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.