Grounded Intelligences

Blog

  • LLM-as-Judge: When to Trust Your AI Evaluator (And When to Override It)

    LLM-as-judge is seductive because it solves an expensive problem cheaply: instead of paying human annotators to evaluate model outputs, you have a larger or more capable model do the grading. The cost savings are real. The risks are also real, and teams that adopt LLM-as-judge without understanding the failure modes end up with evaluation infrastructure that gives them confident-sounding metrics while quietly measuring the wrong thing.

    When It Works

    LLM-as-judge works well when the evaluation criteria are explicit and the judge model can be given precise instructions that human annotators would also receive. Factual accuracy, format compliance, completeness against a checklist, tone matching against a defined style guide — these are tasks where a capable judge model can achieve inter-rater reliability with human annotation that makes it a reasonable substitute. The test for whether your LLM-as-judge setup is working is to run it in parallel with human annotation on a sample and calculate Cohen’s kappa between the two. If kappa is above 0.7, your automated evaluation is reliable enough to trust at scale. Below that threshold, you’re measuring something that doesn’t sufficiently correlate with human judgment to be useful.

    When It Fails

    LLM-as-judge fails predictably on tasks that require domain expertise the judge model doesn’t have, tasks where the rubric is genuinely ambiguous, and tasks where the judge model has systematic biases that don’t match the target population. On domain expertise: a general-purpose model grading the accuracy of specialized tax scenario classification outputs is grading against its own training data, not against ground truth. The grade it gives correlates with how confident the output sounds, not whether it’s correct. On rubric ambiguity: if the evaluation criteria aren’t precise enough for two human annotators to agree consistently, the judge model will also be inconsistent — but it will be inconsistent in a way that looks consistent because it applies the same internal model repeatedly. On bias: LLMs have documented preferences for certain output styles, lengths, and formats that don’t always correlate with quality. A judge model that prefers longer, more hedged responses will systematically grade those outputs higher regardless of their actual correctness.

    Calibration Against Human Annotation

    The calibration protocol that worked in the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours — applied the same methodology to LLM-as-judge validation. Before deploying automated evaluation at scale, we sampled 200 outputs and had both human annotators and the judge model evaluate them independently. The kappa between human annotators came in at 0.81 on unambiguous cases and 0.54 on boundary cases. The kappa between human annotation and LLM-as-judge came in at 0.78 on unambiguous cases and 0.41 on boundary cases. That calibration told us exactly where to trust the automated evaluation and where to require human review: for clear-cut cases at scale, the judge model was reliable; for boundary cases, it was not, and routing those to human annotators was the correct design.

    The Consistency Trap

    The most dangerous property of LLM-as-judge is that it can be consistently wrong. A human annotation process with poor inter-rater reliability at least has the property that its errors are somewhat random — with enough annotators, the noise averages out. A judge model applies the same systematic bias repeatedly, which means the errors don’t average out; they compound. If your judge model systematically misclassifies a category of outputs, every data point in that category gets the wrong grade, and your optimization process confidently drives the model you’re evaluating in the wrong direction. The consistency that makes LLM-as-judge seem reliable is also what makes its systematic errors harder to detect.

    Practical Implementation Guidelines

    Three guidelines have held up across my use of LLM-as-judge in production systems. First, calibrate before deploying: run the judge in parallel with human annotation on a representative sample and calculate kappa before trusting it at scale. Second, stratify by case type: most evaluation tasks have a subset of cases where automated grading is reliable and a subset where it isn’t — treat those subsets differently rather than applying a single threshold. Third, re-calibrate periodically: the judge model’s behavior can change across versions, and a calibration established on one model version doesn’t automatically transfer to the next.

    What Transfers

    The clinical assessment parallel holds here too. Structured clinical interviews have a validated set of questions and scoring criteria specifically because clinician judgment varies in ways that affect diagnostic reliability — the same symptoms get different interpretations from different practitioners without standardized tools. LLM-as-judge is automated judgment, and it has the same reliability problem as human judgment: it needs to be calibrated, validated, and monitored rather than trusted by default. The tools for doing that — inter-rater reliability measurement, calibration protocols, stratified validation — come from the same field that figured this out for human judgment decades ago.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Therapeutic Listening as a UX Method: What Couples Therapy Teaches AI Designers

    As a licensed Marriage and Family Therapist who spent years doing couples and family work before pivoting into AI systems, I can tell you that reflective listening is not a soft skill — it’s a precision technique with documented, measurable outcomes, and almost nobody building conversational AI has been trained in it.

    The research on therapeutic alliance — the quality of the relationship between therapist and client — consistently shows it accounts for roughly 30% of therapy outcomes, separate from the specific intervention being used. The mechanism behind that effect is largely communication design: how the therapist structures questions, how they signal understanding, when they respond versus when they wait. These are learnable patterns. And they translate directly into how conversational AI should be designed.

    Reflective Listening Is Not Just Paraphrasing

    Reflective listening has a specific structure: you paraphrase what you heard, then check for accuracy. “So what I’m hearing is X — is that right?” That check is not optional. It signals to the other person that you’re trying to understand them, not perform understanding. The distinction matters clinically and it matters in AI design.

    Most conversational AI systems skip this entirely. They receive input, generate a response, and proceed — sometimes asking a clarifying question at the end as a kind of afterthought. The therapeutic sequence is inverted: demonstrate understanding first, then respond. The response is only useful if it’s calibrated to what the person actually meant, and you can only calibrate it if you’ve confirmed your interpretation before acting on it.

    There’s a related concept that AI designers routinely confuse: validation versus agreement. “That makes sense given what you described” is not agreement. It’s not endorsing the person’s conclusion. It’s acknowledging that their experience is coherent — that a reasonable person in their situation might feel or think what they’re describing. In therapy, this distinction is fundamental. Agreeing with a client’s distorted belief reinforces the distortion. Validating their experience of having that belief helps them examine it. Conversational AI that conflates the two either capitulates to whatever the user believes or sounds dismissive when it doesn’t — there’s no middle path unless the design explicitly encodes it.

    The Interrogation Problem in Chatbot Design

    The default mode for most chatbot intake flows is closed questioning — yes/no, multiple choice, structured menus. This is procedurally efficient and psychologically counterproductive. Closed questions put users in a reactive posture. They produce minimal, guarded responses. They signal that the system already knows what categories the answer will fit into, which causes users to fit their answer into those categories even when their actual situation doesn’t.

    Good therapeutic intake starts with open questions and follows the client’s lead. “What brings you here today?” is not a soft opener — it’s a deliberate technique that gives the client control of the initial framing. What they choose to say first, and how they say it, is diagnostic information that a closed-question flow destroys before you ever get it. The AI equivalent is letting the user describe their problem in their own terms before offering any categorization or structure.

    Three design patterns from clinical practice apply directly here. Open before closed: start with an expansive prompt and narrow only after the user has established their own frame. Name before normalizing: reflect the specific language the user chose before offering your framing — if they said “overwhelmed,” use “overwhelmed,” not “stressed” or “busy.” Check understanding before advising: never make a recommendation before demonstrating, explicitly, that you understood the problem. Each of these patterns is a deviation from how most AI conversation flows are currently built, and each one corresponds to a documented failure mode in therapeutic communication that produces worse outcomes.

    Emotional Flooding and Degraded Inputs

    John Gottman’s research on emotional flooding — what happens physiologically and cognitively when someone is overwhelmed — is directly applicable to AI system design. When clients are flooded, their communication degrades: they become less precise, more repetitive, and their inputs contain less usable signal. Pushing forward with more questions in that state produces worse data. The clinical intervention is to slow down, regulate, and return to the content when the person can engage with it.

    AI systems have analogous signals available: response latency patterns, session abandonment rates, unusual repetition in user inputs, dramatically shorter responses after longer ones. These are detectable. A system designed with therapeutic principles would recognize them as flooding indicators and shift its conversational mode — shorter prompts, more validation, less cognitive load — rather than continuing to push the standard flow. The behavioral science case for this approach is well established; the implementation is an engineering choice that most teams haven’t made yet.

    What Transfers

    Therapeutic listening as a conversational design framework applies to any AI system taking natural language input from humans in non-trivial situations — medical intake, financial advising, legal question-answering, HR support tools. The stakes in these contexts mean that users who feel processed rather than understood will withhold information, distort their responses to fit perceived categories, or disengage entirely. The cost of that outcome isn’t just a poor user experience; it’s systematically degraded input data flowing into whatever downstream system the AI is feeding.

    Designing for genuine understanding — not the performance of understanding — requires knowing what genuine understanding looks like in practice. That’s what nineteen years of combined clinical training and enterprise systems work taught me: the human side of the interface is not a UX problem. It’s a behavioral science problem, and it has behavioral science solutions.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • ContentShield Pro: Building a Forensic Watermarking SaaS on Cloudflare Workers

    ContentShield Pro is a forensic watermarking SaaS I built entirely on the Cloudflare stack — no servers, no containers, no infrastructure bills — designed to answer a single question when content leaks: whose copy was it?

    That question sounds simple. The engineering behind answering it reliably is not. Most people think of watermarking as the visible stamp a stock photo agency burns into a preview image. Forensic watermarking is the opposite — the mark is invisible, unique per recipient, and designed to survive reasonable transformations like screenshots, reformatting, or re-encoding. When a document surfaces somewhere it shouldn’t, you don’t need anyone to confess. You detect the embedded fingerprint and trace it back to the specific copy you issued.

    The Architecture: Workers, D1, R2

    The entire system runs on three Cloudflare primitives. Workers handle the API layer — three endpoints do all the work. POST /watermark accepts a content payload and a recipient list, generates N uniquely watermarked versions, and returns them with their fingerprint identifiers. GET /verify accepts any copy of the content — clean or suspected leak — and extracts the embedded fingerprint. GET /trace resolves that fingerprint against the recipient mapping stored in D1 and returns the full record: who received it, when, and via which distribution channel.

    D1 is the relational backbone — it stores the watermark-to-recipient table, the distribution logs, and the webhook configurations for each client account. R2 holds the watermarked content files themselves, which can run large for image-heavy assets. The combination means I’m storing structured query data in D1 where I need joins and lookups, and binary content in R2 where I need object storage. Workers AI handles image processing at inference time. Total infrastructure cost for a system serving multiple clients: $0 per month on the Cloudflare free tier, with costs entering only at serious enterprise scale.

    Two Techniques: Text and Image

    Text watermarking in ContentShield Pro uses Unicode zero-width character sequences. Zero-width joiners, zero-width non-joiners, and zero-width spaces are invisible in rendered text but detectable programmatically. Combined with selective homoglyph substitution — replacing certain Latin characters with visually identical Unicode equivalents — each copy gets a unique binary fingerprint encoded into the character stream. A 2,000-word document can carry a 32-bit fingerprint with no visible change to any reader. The sequence is deterministic given the recipient ID, so verification doesn’t require database lookup on every character — the Worker checks the pattern against the expected encoding for any known recipient, then queries D1 only to resolve the match to a human record.

    Image watermarking uses LSB steganography in the alpha channel. The least significant bit of each pixel’s alpha value carries one bit of the fingerprint — across a 1920×1080 image, that’s 2,073,600 available carrier bits for a mark that needs fewer than 64. Workers AI handles the pixel-level processing. The mark is robust against JPEG recompression at quality settings above 85 and survives standard screenshot crops as long as the marked region is preserved. Below that threshold, the mark degrades — which is a known limitation I document in the client-facing technical spec.

    The Webhook-Based Leak Alert System

    ContentShield Pro isn’t purely reactive. When a client submits a suspected leak to /verify, the Worker extracts the fingerprint, queries D1 to resolve the recipient, and immediately fires a webhook to the client’s configured endpoint — with the full trace report as the payload. That report includes the recipient record, the distribution timestamp, the fingerprint confidence score, and a forensic hash of the submitted content for chain-of-custody documentation. The client’s incident response workflow receives a structured JSON payload the moment the trace completes, typically within 400 milliseconds of the /verify request.

    The SaaS economics here are worth stating plainly. When your infrastructure cost is near-zero — because Cloudflare Workers runs at the edge with no server provisioning — the gross margin on usage revenue is extremely high. The variable costs are D1 writes per watermarked copy generated and Workers AI inference per image processed. Text watermarking at scale costs fractions of a cent per document. That margin structure is only possible because the entire product runs on Cloudflare’s edge infrastructure — the same stack I’ve used to run 60+ Workers at $0/month across my personal and professional projects.

    What Transfers

    The forensic audit trail design principle in ContentShield Pro — every operation produces a traceable artifact — is not specific to content protection. It applies directly to AI systems in any regulated environment. Every inference a model makes should be logged: the input, the output, the model version, the timestamp, and the confidence score. Every process action an automation takes should produce a record that can be examined after the fact. The 17+ AI and automation projects I ran at Lockheed Martin all had this in common: the ones that survived audit scrutiny were the ones where every step of the pipeline was traceable. The ones that didn’t were the ones where someone assumed the output was trustworthy without capturing the evidence that would prove it.

    Forensic design isn’t about distrust — it’s about building systems that can prove their own correctness after the fact, to an audience that wasn’t in the room when you built them.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • The Site Is the Portfolio: What Happened When I Instrumented My Own Job Search

    While stress-testing this site the way a recruiter would, I asked my own career assistant a question I had not prepared it for: whether I was open to remote work. The assistant did exactly what I had built it to do — it declined, because the answer was not in its evidence base. Correct behavior, wrong outcome. It is a question every recruiter asks, and the honest answer was a refusal.

    Here is the part that matters: I knew about it within the day, because the assistant logs every question it cannot answer. The gap went into the evidence base, the assistant now answers that question with my actual availability, and a weekly test asserts it never regresses. That loop — measure, change, verify — is the most honest demonstration of how I work that I know how to give. So I built this entire site around it.

    A portfolio that runs instead of describes

    Twenty years of my applied work sits inside environments that cannot be shown. That is the standing problem for anyone who has built serious systems inside a defense enterprise: the better the work, the less of it you can put on a website. Most people solve this with adjectives. I decided to solve it with running systems.

    The status page is the live monitoring surface of my real infrastructure — service health over service bindings, backup freshness with a 26-hour dead-man’s switch, self-healing activity counts. The demos run production methods on synthetic data in your browser: streaming anomaly detection, fuzzy entity resolution, a whole-life financial system with a Monte Carlo retirement engine. The assistant answers questions about my record from a structured evidence base, with guardrails, and is itself the strongest example of the retrieval-grounded systems I build. None of it is staged. If a backup goes stale, the status page says so, and I get paged.

    The psychology is load-bearing

    I am a licensed marriage and family therapist alongside the data science career, and this site is where the two halves stop being a novelty pairing and start being one discipline. Every interactive page has a small 🧠 toggle that annotates the design decisions in place: why the demos let you break things yourself (self-generated evidence beats presented evidence), why the sample data ships with flaws (a trend that only goes up reads as marketing), why the assistant is allowed to say “I don’t have that on record” (a system that never declines teaches people to over-trust it).

    The newest demo makes that last point measurable. The Trust Calibration Lab asks you to review ten pieces of an intentionally imperfect AI’s work — first with bare, confident-sounding answers, then with calibrated confidence and the right to decline — and scores your own over-trust and under-trust in each condition. It is the guardrail philosophy from my production RAG systems, compressed into ninety seconds you perform on yourself. Your anonymous score joins the aggregate, which means the demo’s evidence gets stronger with every visitor.

    Measuring in public

    The status page now includes a section called “this site studies itself”: how many questions the assistant took this week, how many it refused, how many résumés were downloaded. Aggregate counts only — there is no individual tracking anywhere on this site, which is itself a design position. The interesting column is not the counts. It is what changed because of them. The remote-work gap above. A hedging habit in the assistant’s voice that is now banned in its instructions and tested weekly. A tendency to improvise answers about weaknesses, replaced with a written honest-limits statement.

    There is also an A/B experiment running on the résumé page’s download button right now. I will publish the result either way, including if the boring variant wins, because an experiment you only report when it flatters you is not an experiment.

    What stays private, and why that is part of the design

    The autonomous system behind my job search — discovery, AI fit-scoring, tailored document generation — has a live console I do not publish, because it holds real pipeline data about real companies. The boundary is deliberate and disclosed. Twenty years in regulated environments taught me that what a system refuses to expose tells you as much about its builder as what it shows. This site tries to demonstrate both.

    If you are evaluating me — as a recruiter, a hiring leader, an engineer, or someone’s screening agent — the invitation is the same: do not take my word for any of this. Measure yourself in the lab, interview the assistant, check the live numbers, and take the résumé with you. The site is the portfolio.

  • Explainability for Executives: Translating Model Outputs Into Decisions

    The meeting that clarified the explainability problem for me happened when a CFO looked at a dashboard I’d helped deploy, saw a field labeled “anomaly score: 0.87,” and asked — with complete seriousness, not as a challenge — “What does 0.87 mean? Is that good?” The room went quiet. The team had spent months on the model. They had spent approximately zero time on what the model’s outputs should mean to the person who was supposed to act on them.

    Three Levels of Explainability That Actually Matter

    Enterprise AI systems serve at least three distinct audiences, and each audience needs a different form of explanation. Conflating them — building one explanation interface and pointing everyone at it — produces a tool that serves none of them well.

    The operational level is for the analyst who runs the system daily. She needs to understand what the model did on this specific transaction or record — which inputs drove the output, whether the result is consistent with similar cases she’s reviewed, and what she should check if she wants to verify or override it. Technical detail is appropriate here. She uses the system 40 hours a week and can develop fluency with model-specific language.

    The managerial level is for the supervisor or team lead who reviews AI-flagged items and decides how to allocate analyst time. He needs to know what action to take — which flags require immediate investigation, which can wait, and how to triage a backlog of 200 alerts that all show scores above 0.7. He does not need to know how the model produces a score. He needs to know what the score implies about urgency and risk in terms he already uses to manage his team.

    The executive level is for the CFO, program director, or board member who needs to understand what the AI system means for the business — not what it does technically, but what decisions it supports, what it can’t tell you, and what it would take for you to rely on it for a consequential call. Most AI teams never build this level of explanation at all. They present the model’s performance metrics — accuracy, F1 score, precision-recall curve — to audiences who have no frame for evaluating what those numbers mean in operational terms.

    Why SHAP Values Don’t Belong in a Leadership Meeting

    SHAP values are genuinely useful for the operational level — an analyst who wants to understand why the model flagged a particular record can look at feature attributions and build intuition about the system’s behavior over time. That is a legitimate use of the tool.

    Presenting SHAP values to an executive is a category error. It’s not that executives are unsophisticated — it’s that SHAP values answer a question they’re not asking. They’re asking whether to trust the system’s recommendation on a $2M procurement decision. A bar chart showing that “vendor tenure” contributed 0.34 to the anomaly score does not answer that question. It transfers the cognitive work of interpretation to someone who shouldn’t have to do it.

    What works at the executive level is natural language that maps model outputs to business concepts they already use. “This purchase order is $142K above the 3-year average for this vendor category” is an explanation. “Anomaly score: 0.87” is an output that requires explanation. Counterfactuals work well too — “If the forecast had used Q3 actuals instead of Q2 estimates, the variance flag would not have fired” gives an executive the conditional reasoning she needs to evaluate whether the flag is meaningful or an artifact of a data lag.

    The Dashboard Redesign That Moved Adoption by 44 Points

    At Lockheed, a finance AI dashboard that had been running at 34% active adoption for three months went through an explainability redesign. The change was not to the model — the underlying system was unchanged. The change was entirely in the output layer: probability scores were replaced with natural language summaries written in the vocabulary of the finance analysts who used it. “Probability: 0.73” became “This cost pattern occurs in fewer than 8% of similar transactions — the last time this pattern appeared in this cost center, the variance was caused by a billing timing mismatch that took 11 days to resolve.”

    Two months after the redesign, active adoption was at 78%. The model hadn’t changed. The explainability layer had. Analysts who had been ignoring the dashboard because they couldn’t tell whether 0.73 warranted action were now using it because the output spoke in terms of their existing workflow and experience.

    What Transfers

    In clinical training, one of the first things you learn is to speak the client’s language rather than the DSM’s. A client describing their experience as “I get really wound up and can’t settle down” is telling you something useful. Responding with “that sounds like hyperarousal consistent with an anxiety spectrum presentation” communicates nothing to them — it’s accurate and useless. The skill is translation: taking precise clinical language and rendering it in terms the person in front of you can act on.

    AI explainability design is the same skill applied to a different context. The one-sentence test I apply before any model goes to a leadership meeting: can I explain this system’s decision in one sentence to a non-technical executive who will use that explanation to make a consequential call? If the answer is no, the model isn’t ready for that meeting — not because the model is bad, but because the explanation layer isn’t built yet. Technical completeness and operational readiness are not the same thing, and confusing them is how teams ship AI systems that sit unused at 34% adoption when they had the capacity to reach 78%.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • How to Quantify AI ROI Without Misleading Your Stakeholders

    The $3M in savings slide is almost always wrong — not because the savings don’t exist, but because the methodology didn’t hold up to scrutiny when anyone actually looked.

    I’ve sat in enough enterprise finance reviews — 19 years of them at Lockheed Martin, many directly tied to AI and automation projects I led — to recognize the pattern. A team builds something that genuinely improves a process, they attach a number to the improvement, and then that number gets challenged in a budget review and can’t be defended. The project survives or doesn’t based on whether the finance organization decides to extend credibility rather than because the ROI case was actually sound.

    Two Failure Modes Before You Calculate Anything

    Underselling is the failure mode that gets less attention. ROI exists — real hours are recovered, real error rates drop — but the team has no rigorous way to measure it, so the project gets framed as qualitative improvement. Qualitative improvements don’t survive budget compression. They get cut when finance needs to find savings and the project has nothing auditable to defend itself with.

    Overselling is more visible and more damaging to the broader AI program. A team attributes all observed efficiency gains to the AI system when the gains were actually produced by a combination of the tool, a concurrent process redesign, an analyst who changed how she organized her work, and a quarter with lower transaction volume than the prior year. If you claim credit for all of it and a skeptical CFO asks how you isolated the AI contribution, you don’t have an answer. That moment — one moment — kills credibility for the next five AI proposals your organization brings forward.

    The Costs That Disappear From ROI Slides

    Every ROI methodology I’ve seen that didn’t survive scrutiny had the same structural problem: it counted only the benefits of the AI and the initial build cost, then stopped. The cost categories that routinely disappear from the slide are annotation and labeling cost for supervised systems, the retraining cadence — a model that needs quarterly retraining has a recurring engineering cost that should appear in the denominator, human review of AI outputs in any high-stakes workflow, and the cost of errors the system makes rather than only the errors it prevents.

    That last one is particularly common in automation ROI. A system that processes 10,000 transactions per month with a 0.5% error rate introduces 50 errors per month that someone has to catch and correct. If those errors are in finance or compliance workflows, the correction cost — loaded labor rate × correction time, plus any downstream rework — can meaningfully offset the efficiency gains. An honest ROI methodology puts that number in the model. A credibility-optimized ROI methodology hopes no one asks about it.

    How a Defense Finance Automation ROI Number Was Built

    The $360K annual cost avoidance figure from a major defense aircraft program withholds automation project held up to finance leadership scrutiny because it was constructed to be auditable, not to be impressive. The methodology: loaded labor rate for the analyst population multiplied by 2,132 recovered hours per year — hours that were timed, not estimated, against a documented pre-automation baseline — minus annual system maintenance cost, minus periodic retraining and validation cost for the Alteryx and Python workflows involved, minus an allocation for edge cases that still required manual intervention.

    The methodology was presented visibly, not buried in appendices. Finance leadership could see exactly which assumptions the number depended on and could stress-test any of them. When the program office asked what the number looked like if loaded labor rate was adjusted down by 15%, we had that sensitivity analysis ready. The figure that survived scrutiny was lower than the first draft — and more credible precisely because of that.

    Framing ROI by Audience

    A single ROI figure presented identically to every audience is a communication failure before it’s a methodology failure. CFOs want payback period and net present value — they’re evaluating capital allocation and need to compare this project against others competing for the same budget. Operational leaders want hours recovered per analyst per month because they’re managing capacity and need to understand what changes for their team on Monday. CTOs want the technical debt comparison — build versus buy, and what the maintenance cost trajectory looks like in years two and three when the team that built the system has moved on.

    Each of those framings uses the same underlying data. The work is disaggregating the ROI model into components that answer the specific question each audience is actually asking. Presenting a single number to all three audiences and hoping it resonates is why most AI ROI conversations stall in the room rather than producing decisions.

    What Transfers

    Financial rigor from defense FP&A is not defense-specific. The discipline of building auditable cost and benefit models — with documented baselines, explicit attribution logic, visible cost categories, and audience-specific framings — applies to any AI project that needs to survive budget scrutiny. The aerospace context made the stakes higher and the review process more formal, but the underlying methodology is the same one any enterprise AI team should be using. The first AI project that can’t defend its ROI doesn’t just lose its own budget. It makes the case against the next five.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Bias Is a System Property, Not a Model Property: A Clinician’s Perspective

    Family systems therapy taught me not to diagnose individuals in isolation — a child’s behavior is only legible in the context of the family system that produces and reinforces it. I’ve come to think that this is precisely the conceptual frame that most AI bias discussions are missing, and missing it produces interventions that move the problem rather than resolve it.

    The Category Error at the Center of the Bias Conversation

    The dominant framing of AI bias treats it as a model property — something present in the weights, detectable with fairness metrics, and removable through debiasing techniques applied to the model. This framing is wrong in a precise way: it mistakes a system-level phenomenon for a component-level defect. The bias you can measure in a model’s outputs is not located in the model. It’s produced by a system — the training data, the feedback mechanisms, the deployment context, and the humans who interact with all of it.

    This matters because component-level interventions applied to system-level problems produce predictable results: the problem moves. You apply adversarial debiasing and the model’s aggregate fairness metric improves on your evaluation set while the bias redistributes to subgroups you weren’t measuring. You reweight the training data and you reduce one source of bias while introducing a new imbalance in a different direction. Neither outcome means the technique was wrong — it means the framing that led you to apply only that technique was incomplete.

    Four Systemic Bias Sources That Precede the Model

    The first source is historical data encoding past human decisions. If a hiring model is trained on ten years of hiring decisions made by a team with a documented preference for candidates from specific universities, the model learns that preference precisely. There is no malfunction. The model is performing exactly as trained. The bias is in the decision record, not in the algorithm.

    The second source is feedback loops. A model’s outputs shape the next round of training data. A content recommendation system that surfaces more of what users engage with will amplify engagement patterns — including the ones that reflect anxiety, outrage, or compulsion rather than genuine preference. The model is not broken. The feedback loop is producing exactly what feedback loops produce.

    The third source is deployment context mismatch. A model trained primarily on data from Group A deployed in a context where Group B is the primary user population will generalize poorly to Group B — not because it was built with malicious intent, but because it was built with insufficient attention to the difference between its training distribution and its deployment distribution. This is a systems design failure that shows up as a model failure.

    The fourth source is user behavior adaptation. Users who interact with AI systems over time learn the system’s tendencies and adjust their behavior accordingly — which changes the data the system sees, which changes what the system learns in the next training cycle. The model and its users are a coupled system, and treating the model as a static artifact misses the dynamic entirely.

    The Clinical Parallel: Confirmation Bias Is Not a Bug in Clinicians

    Confirmation bias in clinical practice is well-documented. Licensed clinicians with years of training consistently show a tendency to confirm their initial diagnostic hypotheses — to weight evidence that supports their first impression more heavily than evidence that challenges it. This is not a character flaw. It is a feature of how human cognition works under uncertainty. Structured clinical interviews and standardized diagnostic protocols exist precisely to force against this tendency — to build a system-level constraint that counteracts a predictable individual-level pattern.

    The lesson is not “clinicians are biased, therefore don’t trust them.” The lesson is “predictable cognitive patterns require structural countermeasures, not just individual awareness.” The same logic applies to AI systems. Knowing that your training data encodes historical human decisions is not sufficient — you need structural interventions in the data pipeline, the feedback mechanism, and the deployment monitoring that function as countermeasures, not as one-time fixes.

    At Lockheed, auditing an AI system for systematic patterns required going back to the raw training data, not just examining model outputs. That process took six months for one system. The output-level audit had found nothing alarming. The data-level audit found several years of historical decisions with a consistent pattern that the model had faithfully reproduced. The model was working as designed. The design was the problem.

    What Transfers

    The shift in question that systems thinking requires is from “is this model biased?” to “what are the feedback loops in this system, and what do they produce?” The first question has a tractable-sounding answer. The second question is harder and more useful. It asks you to map the full system — data generation, model training, deployment context, user behavior, retraining inputs — and identify where bias enters, where it’s amplified, and where your interventions are actually redirecting it rather than eliminating it.

    This is the contribution that behavioral and clinical training makes to responsible AI work — not a set of soft skills layered on top of the engineering, but a different set of questions to ask before the engineering begins. I’ve written about the broader case for this perspective in why a therapist builds AI systems. The short version: systems that produce harmful outcomes usually aren’t built by people who intended harm. They’re built by people who were asking the wrong unit of analysis.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • OSINT at the Edge: Building a Threat Monitoring System with Cloudflare Workers

    Running meaningful threat intelligence monitoring at any scale — certificate transparency logs, domain registration alerts, threat feed normalization — costs either a commercial subscription in the thousands of dollars per month or custom infrastructure that requires active maintenance, unless you build it on Cloudflare Workers, where the entire stack runs at $0.

    I’ve covered the Digital Sentinel system overview in a previous post. This one goes deeper — into the specific data sources, the edge architecture that connects them, and the numbers that came out of running it in production against real client scope.

    Certificate Transparency: Every Certificate, Indexed in Public

    Every SSL/TLS certificate issued by a trusted Certificate Authority gets logged to public Certificate Transparency logs — crt.sh, Google Aviator, Cloudflare Nimbus, and others. This is a regulatory requirement, and it means that if someone registers a typosquatting domain and immediately obtains an HTTPS certificate for it, that certificate appears in the CT logs within minutes of issuance. Most organizations don’t have a system watching those logs for their own name patterns. They find out about impersonation domains when a customer calls.

    A Cron Trigger Worker polls the crt.sh JSON API on an hourly schedule. The worker queries for certificates issued to domains matching configurable pattern sets — brand name variants, common typosquats, executive name combinations — and filters the response against a D1 database of previously seen certificate IDs. New matches are written to D1 with full certificate metadata: issuing CA, subject alternative names, issuance timestamp, and the registrant domain. The total runtime per execution is under 800 milliseconds for a scope of approximately 40 pattern variants. Cloudflare’s free-tier Cron Triggers handle this without any cost.

    Threat Feeds: Normalization Is the Actual Work

    AlienVault OTX and AbuseIPDB both offer free API tiers with meaningful data volumes. The operational problem isn’t access — it’s normalization. OTX returns indicators in a pulse-based format with nested JSON. AbuseIPDB returns scored IP addresses with abuse category codes. Neither format matches the other, and neither maps cleanly to a schema you’d want to query for alerting logic.

    A second Worker fetches and normalizes both feeds every four hours. It extracts indicators — IPs, domains, hashes, URLs — maps them to a unified schema with source, confidence score, indicator type, and first-seen timestamp, then deduplicates against D1 before writing. Deduplication runs against a composite key of indicator value plus source, because the same IP might appear in both feeds with different confidence scores — and that delta is itself a signal worth preserving.

    Rate-limit state is tracked in KV, not D1. KV’s per-key TTL makes it the right store for ephemeral rate-window tracking — writing a counter to D1 for something that expires in 15 minutes is the wrong tool. This distinction between KV for transient state and D1 for durable records runs through every part of the architecture.

    RDAP Domain Monitoring: The WHOIS Replacement That Actually Has an API

    WHOIS is largely broken for programmatic use — rate limiting, inconsistent formatting, and increasingly redacted registrant data make it unreliable. RDAP, the Registration Data Access Protocol, is the IANA-standardized replacement. It returns structured JSON, it’s queried via HTTPS, and it works inside a Workers environment without requiring a separate proxy layer.

    The domain monitoring Worker queries RDAP endpoints for newly registered domains matching typosquat patterns — character substitutions, homoglyph variants, hyphenated versions, TLD variations — and stores registration records in D1. In the first month of running this against a client’s brand scope, the system identified three domains that had been registered with clear impersonation intent: two used homoglyph substitutions in the brand name, one combined the brand name with a plausible-sounding financial suffix. None had been caught by any other monitoring the client had in place.

    The Alerting Pipeline: From Detection to Discord in Under 90 Seconds

    All three monitors — CT logs, threat feeds, and RDAP — write findings to a shared D1 table with a processed flag. A fourth Worker, AlertDispatch, runs on a two-minute Cron Trigger, queries for unprocessed records, enriches each finding with contextual data from KV, formats a Discord embed with severity color coding and direct links to supporting sources, and POSTs to a configured webhook. End-to-end alert latency from detection event to Discord notification runs under 90 seconds in normal operation.

    The system currently processes approximately 47,000 CT log entries daily across active client scope. The false positive rate — certificates that match patterns but aren’t actual threats — runs at about 4%, which is low enough that analysts review every alert rather than implementing suppression rules. Keeping the false positive rate manageable required iterative refinement of the pattern matching logic over the first six weeks; the initial version was closer to 18%.

    What Transfers

    The architectural pattern here — scheduled edge Workers feeding a central D1 store, KV for rate control and transient state, a dispatch Worker for enrichment and notification — applies to any event-driven monitoring use case that doesn’t require persistent server processes. Price monitoring, compliance change detection, API health surveillance, social mention tracking: the structure is the same. The zero-cost operation isn’t incidental to the design. It’s the reason this architecture is worth understanding — it puts monitoring capability that used to require dedicated infrastructure into a pattern any developer can deploy and maintain solo. That’s the broader lesson from running 60+ Cloudflare Workers in production: the constraint that keeps most teams from building this kind of tooling isn’t technical complexity. It’s the assumption that it requires budget they don’t have.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Building RAG Over 10,000 Research Papers: Architecture and Hard Lessons

    I built a retrieval-augmented generation system over 10,000 academic papers in I/O psychology, organizational behavior, and AI ethics — not as a demonstration project, but because I needed a research assistant that could actually find the supporting evidence buried in dense methodology sections and tell me where it came from.

    Why Academic Paper RAG Is Harder Than Product RAG

    Most RAG tutorials assume clean, well-structured source documents — FAQs, product manuals, internal wikis. Academic PDFs are none of those things. Multi-column layouts break naive text extractors. Equations render as gibberish or disappear entirely. Citations frequently span page breaks, so the reference text you want is split across two chunks. Abstracts — the obvious candidate for a summary embedding — often don’t reflect the actual contribution of the paper. A paper titled “A Meta-Analysis of TAM Adoption Predictors” might have its most useful content buried in a limitations section that no one would retrieve based on the title alone.

    The naive approach — extract all text, split into 512-token chunks with 50-token overlap, embed, and retrieve — produces results that look plausible but miss the actual evidence. For casual Q&A over product documentation, that’s acceptable. For research work where you’re trying to accurately represent the state of evidence on a topic, it’s not.

    The Chunking Strategy That Actually Works

    The architecture I settled on uses section-aware chunking rather than a sliding window. Each paper is parsed to extract named sections — Introduction, Methods, Results, Discussion, Limitations — and each section becomes its own chunk set. Every chunk carries metadata: paper title, publication year, authors, journal, and section name. That metadata travels with the chunk through retrieval and into the prompt.

    The reason section-level metadata matters is that the provenance of a claim is different depending on whether it comes from a Results section or a Discussion section. “Participants showed a 23% improvement in task completion” means something different in Results than in Discussion. Knowing which section a retrieved chunk came from changes how you should weight it in your synthesis.

    For the embedding model, I started with text-embedding-3-small and moved to text-embedding-3-large for this corpus specifically. The quality difference for dense academic prose — methodology descriptions, statistical terminology, theoretical constructs — was meaningful enough to justify the cost difference. For retrieval over casual conversational text, small would have been fine.

    Hybrid Retrieval and the Reranker You Cannot Skip

    Academic text has a property that makes pure vector retrieval underperform: exact terminology matters enormously. When a research question involves “Cohen’s kappa” or “confirmatory factor analysis” or “heteroscedasticity,” semantic similarity search will retrieve conceptually adjacent material that doesn’t actually contain the relevant methodology. BM25 keyword retrieval catches exact matches that vector search misses.

    The retrieval pipeline runs both — vector search weighted at 0.6, BM25 at 0.4 — then merges the candidate pools. The weight split was empirically tuned on a held-out evaluation set of 200 research questions where I had manually verified the correct source chunks. Getting the balance wrong in either direction cost meaningful retrieval quality.

    The reranker is not optional. Without a cross-encoder reranker applied to the merged candidate pool, the top-ranked results consistently look relevant — they match on topic — but miss the actual supporting evidence. The cross-encoder evaluates query-chunk relevance jointly rather than independently, which costs more compute but catches the cases where a chunk that seems topically relevant doesn’t actually answer the question. I tested the pipeline with and without reranking on the same 200-question evaluation set, and the difference in retrieval precision was large enough that I would not ship academic RAG without it.

    Three Hard Lessons

    The first hard lesson was the reranker, already described. The second was metadata filtering by publication year. A 2018 paper’s recommendations for model evaluation benchmarks, fairness metrics, or dataset standards may be functionally obsolete. Without a year filter built into the retrieval interface, the system would confidently surface outdated methodology as current best practice. Year-range filtering is now a first-class parameter in every query I run against this corpus.

    The third lesson was citation chains. The paper that directly answers your question is frequently not the paper that contains the data supporting its claim — it cites another paper, which may cite a third. A retrieval system that only returns the top-ranked chunks and stops there will miss this structure entirely. I added a citation-following step that, when a retrieved chunk contains a reference to a specific study, fetches that study’s chunks and includes them as secondary context. This added complexity, but it’s the step that makes the difference between a system that finds what you asked for and a system that finds the actual evidence.

    What Transfers

    The chunking-with-metadata principle applies to any long-form document RAG — legal contracts, engineering manuals, case files, regulatory filings. The insight is the same: your chunks should carry enough provenance information that downstream reasoning can assess not just what a chunk says, but where it sits in the document’s argumentative structure and how current it is. Sliding-window chunking that strips that context produces retrieval that feels right more often than it is right — which is a failure mode that’s harder to catch than one that fails loudly. If you’re working on similar document retrieval problems, the broader lessons from enterprise AI deployment apply here as well: the architecture decisions that seem like premature optimization at 100 documents become the only thing standing between you and retrieval collapse at 10,000.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • What I/O Psychologists Know About AI Adoption That Engineers Miss

    My M.S. in Industrial/Organizational Psychology has proven more useful for shipping AI systems than most engineers expect, and I say that having spent 19 years at Lockheed Martin watching technically excellent tools fail because their designers never studied how human beings actually adopt new behaviors inside organizations.

    The Technology Acceptance Model Is Right and Also Insufficient

    Every I/O psychology graduate student learns the Technology Acceptance Model — Davis’s framework from 1989 that predicts adoption from two variables: perceived usefulness and perceived ease of use. TAM holds up remarkably well across four decades of research. If people don’t believe a tool will make their work better, or if they find it confusing to operate, they won’t use it. That part is correct.

    What TAM underweights — and what I watched play out repeatedly across 17 AI and automation projects — is the social and identity layer. Perceived usefulness is not calculated in isolation. It’s calculated in comparison: useful compared to what I do now, useful according to people I respect, threatening to what I’m known for being good at. The research on social influence in technology adoption has expanded substantially since Davis wrote his original paper, and the organizational behavior literature is unambiguous: your colleagues’ opinions about a tool predict your adoption of it more reliably than the documentation does.

    Two Adoption Outcomes from the Same Organization

    At Lockheed, I watched two AI-adjacent tools launch into similar finance analyst populations within the same 18-month window. The first was technically superior — better accuracy, cleaner interface, faster outputs. Adoption landed below 20% after 90 days and never recovered. The tool was positioned, implicitly through its design and explicitly through its rollout messaging, as a replacement for the judgment the analysts had spent years developing. When a senior analyst felt the tool was saying her expertise was no longer necessary, she didn’t complain loudly. She just stopped opening it.

    The second tool, which I helped design and position, reached 94% adoption within 60 days. The difference was not the algorithm. The difference was a deliberate framing decision made before a single line of code was written: the AI surfaces patterns and flags anomalies, and the analyst decides what they mean. Every interface element, every training session, every executive communication reinforced that frame. The analysts’ judgment was the point. The tool was the assistant.

    The Champion Model: Influence Nodes Before Broad Rollout

    Organizational psychology has a well-documented framework for change diffusion — Rogers’s Diffusion of Innovations — and it tells you something specific about early adopters: they are not just first users. They are influence nodes. Their opinion ripples through the social network of the team in ways that documentation and training cannot replicate.

    My practice across enterprise AI deployments is to identify three to five people who carry social credibility in the target group — not necessarily the most senior people, but the ones others ask for opinions — and get genuine buy-in from them before the general rollout. This is not manipulation. It’s recognizing that organizational adoption is a social process, not a rational calculation performed independently by each individual. If you skip this step and go straight to broad deployment, you’re hoping the math works in your favor. It often doesn’t.

    Resistance, when it comes, deserves the same interpretive generosity. When an analyst says a tool doesn’t work for her use case, she is frequently correct. The edge case she’s describing is real. Engineers who treat resistance as obstruction miss the signal — they get slower adoption and worse systems. Engineers who treat resistance as data get faster adoption and better systems.

    What Transfers

    The parallel that I find most clarifying comes from my training as a Licensed Marriage and Family Therapist. In clinical practice, the research on therapeutic outcomes is consistent: client buy-in to the treatment model must be established before behavior change begins. A technically correct intervention delivered to an ambivalent client produces worse outcomes than a slightly less refined intervention delivered to a client who understands and endorses the approach. The therapeutic alliance predicts outcomes more strongly than the specific technique.

    AI adoption follows the same structure. The quality of your model matters. The quality of your deployment relationship matters more. Teams that build organizational buy-in before broad rollout — that treat adoption as a human behavior problem rather than a communication problem — consistently outperform teams that don’t, regardless of the underlying technical quality of the system. I’ve written about the intersection of clinical training and AI systems design in more depth elsewhere, but the short version is this: the behavioral science was never separate from the engineering work. It was always the harder part.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.