Grounded Intelligences

Tag: data-science

Data science practice, tools, and lessons from enterprise and independent engineering projects.

  • The AI Roles Worth Targeting in 2026: A Field Guide for Unconventional Candidates

    I spent 19 years in defense finance and digital transformation before becoming someone companies want for AI work, and here is what I see in 2026: the most interesting AI roles are not being filled by candidates submitting the standard machine learning pipeline resume. They are being filled by people who understand both the technology and the domain it’s being applied in. The pure-ML resume is now common enough to be a commodity. The combination is rare enough to be a differentiator.

    Where Domain Expertise Beats Pure ML Background

    Several roles reward domain depth over model depth. The AI Solutions Architect needs system design thinking and domain credibility — the job is fitting AI into a real organization with real constraints, and knowledge of model internals alone doesn’t get you there. The AI Product Manager needs to understand user behavior and organizational dynamics; candidates from a pure ML background often struggle here precisely because the hard part isn’t the model. The AI Ethicist or Responsible AI Lead draws on philosophy, law, social science, and clinical backgrounds — none of which appear on a standard ML résumé. And the ML Evaluator or RLHF Specialist needs cognitive science, linguistics, and domain expertise far more than systems engineering. In each case the rare skill is the human and organizational understanding, not the technical baseline.

    A Case Study in Recombination

    My own path is the argument. A Licensed Marriage and Family Therapist license, a master’s in Industrial/Organizational Psychology, 19 years in defense finance at Lockheed Martin, and a track record building AI systems that produced quantified outcomes — $360K in annual cost avoidance on a major defense aircraft program Withholds automation, 415,000 tax scenarios processed in 4 hours through the internal tax scenario classifier. None of those pieces gets you the job on its own. A therapist isn’t an AI engineer; an I/O psychologist isn’t a finance lead; a defense finance background isn’t responsible-AI expertise. Together, though, they represent something genuinely uncommon: a person who can build the system and understand the human, organizational, and regulatory context it has to live inside.

    What Companies Want but Can’t Articulate

    What organizations are actually hiring for — even when they can’t put it in a job description — is someone who will catch the problems that emerge at the intersection of the model and the organization. The model that is technically correct but organizationally wrong. The system that is accurate but not trusted by the people meant to use it. The tool that performs beautifully in evaluation and fails in deployment because nobody accounted for how humans actually behave around it. Those failures don’t show up in a model card. They show up six months after launch, and the person who can foresee them is worth more than the person who can squeeze another point of accuracy out of a training run.

    How to Position as an Unconventional Candidate

    The move that works is building in public. Every project you document is evidence — not a claim about what you could do, but a record of what you did. This site is the argument: posts about building real systems, quantified outcomes, and the behavioral science lens most AI engineers don’t carry. A working job search dashboard at jobs.groundedintelligences.com tells a story no resume bullet can, because it’s a live system someone can actually use and inspect. When the standard candidate hands over a list of skills, the unconventional candidate hands over a working thing and says: I built that, here’s what it does, here’s what it cost, here’s what it produced.

    The Most Important Credential in 2026

    The credential that matters most now is a portfolio of working AI systems with documented outcomes. Not a certification, not a bootcamp completion, not a GitHub repository full of Jupyter notebooks that ran once on a laptop — working systems with real usage and quantified results. The bar has risen because building has gotten easier, which means the proof of competence is no longer knowing how but having done. A deployed system with users and numbers settles the question that a credential only gestures at. I wrote about the discipline behind shipping those systems in lessons from 17 enterprise projects.

    What Transfers

    These positioning principles apply in any field where the standard path is well-worn and the non-standard path creates genuine differentiation. When everyone arrives with the same resume, the credential stops distinguishing anyone — and the person who combines an unusual background with demonstrable work becomes legible in a way the conventional candidate can’t. Build the thing, document the outcome, and let the combination of skills nobody else has stacked become the reason you’re the obvious hire. The unconventional path is only a liability until you make it the point.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Building My Career Intelligence Dashboard: A Personal AI Job Search Engine

    I built a fully functional career intelligence platform running at jobs.groundedintelligences.com at zero hosting cost, doing AI-powered job matching and processing hundreds of job postings every day. It is not a demo or a prototype — it ingests live job data, scores it against my profile, and surfaces the best matches every morning, all on Cloudflare’s free tiers. Here is how it’s put together and the one design decision that mattered most.

    The Stack

    The whole thing sits on Cloudflare primitives. A Worker acts as the API server. D1 holds the job database. KV handles rate limiting and session caching. R2 stores resume versions and company logos. Workers AI generates the embeddings that drive job-to-resume matching. That’s five services, each on its free tier, composing into a real application — no virtual machines, no container bills, no separate database host. The same architecture approach underlies my personal finance PWA on Cloudflare Workers, which runs on the same foundation.

    Data Sources

    The data comes from three kinds of sources. Job board APIs — Greenhouse, Lever, and Ashby all expose free public endpoints — provide the bulk of structured listings. Curated RSS feeds from the careers pages of target companies fill in the roles those APIs miss. And a custom scraper Worker handles sites that have no API at all, pulling listings on a schedule and depositing them into D1 alongside the structured data. Together these feed a steady stream of new postings without any paid data subscription.

    The ICP Scoring System

    At the center is an Ideal Candidate Profile stored in D1 that weights the factors I care about — industry, company stage, required-background match, title level, and location preference. Every ingested job gets an ICP score from 0 to 100, calculated as the cosine similarity between the job’s embedding and the profile’s embedding, both produced by Workers AI with bge-base. The embedding approach means the system understands a role described in unfamiliar wording, not just one that happens to share keywords with my profile. A job that never says “applied AI” but describes exactly that work still scores high.

    The Matching Interface and Feedback Loop

    The front end is a PWA served by a Worker, showing jobs ranked by ICP score — each with a three-sentence AI summary explaining why the role matches my profile, generated by Workers AI running LLaMA 3 8B over the job description and the ICP. When I rate a job with a thumbs up or down, that signal feeds back into the ICP score through a simple update, so the system gets sharper the more I use it. The feedback loop is deliberately lightweight; it doesn’t retrain anything, it just nudges the profile toward what I’ve actually shown interest in versus what only looked good on paper.

    The Real Numbers

    The system ingests 200 to 400 new jobs daily across all sources. Every morning it surfaces the top 12 matches through the PWA — a curated shortlist instead of a firehose. The processing cost is $0 per month, because every component lives inside a free tier and the daily volume never approaches the limits. A dozen well-ranked roles each morning, drawn from several hundred raw postings, is a fundamentally different experience than scrolling a job board for an hour, and it costs nothing to run.

    The Decision That Mattered Most

    The single best design decision was not trying to scrape LinkedIn. It would have been a constant fight against anti-scraping measures, and it would have pulled in noise from every direction. The job boards with open APIs — Greenhouse, Lever, Ashby — are used by the kinds of companies that build things with APIs. That is exactly the target company profile for my own search. The constraint became a filter: by limiting intake to companies that expose clean public endpoints, I automatically biased the whole pipeline toward technically serious organizations. The thing I chose not to build improved the results more than another feature would have.

    What Transfers

    The personal intelligence tool architecture generalizes to any domain where you’re sorting through high-volume, heterogeneous inputs to find relevant signals. Scheduled ingestion from multiple sources, embedding-based matching against a profile of what you care about, and a lightweight feedback loop that sharpens the profile over time — that pattern works for jobs, for monitoring news, for tracking research, for anything where the problem is too much input and not enough relevance. Build the profile, embed the inputs, rank by similarity, and let your own feedback tune it.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Fine-Tuning LLMs on Proprietary Data: The Annotation Quality Problem

    A LoRA fine-tune of Mistral 7B on 10,000 examples cost me under $50 on RunPod in 2026 — the compute is cheap and getting cheaper. The bottleneck moved years ago, and most teams haven’t noticed. It is no longer the training run. It is the quality of the data you feed into it, and annotation quality is where fine-tuning projects quietly fail.

    What Annotation Quality Actually Means

    Annotation quality is not just “did the annotators agree with each other.” It is three harder questions. Did the annotation rubric actually capture the task you care about, or a convenient proxy for it? Were annotators calibrated against a shared set of examples before they started labeling, or did they each invent their own interpretation? And were the edge cases documented as they came up, or silently resolved and forgotten? Agreement on a bad rubric just gives you consistent noise. The quality lives in the rubric and the calibration, not in the raw agreement number alone.

    Inter-Rater Reliability as the Starting Gate

    Cohen’s kappa is the first gate, and I treat it as a gate rather than a vanity metric. A kappa below 0.70 means your annotators are not reliably applying the same rubric — which means your fine-tuning data carries the disagreement as noise straight into the weights. A kappa below 0.60 tells you something more serious: the rubric itself probably needs redesign, not just more annotator calibration. The distinction matters because the fix is different. Below 0.70 you retrain the people. Below 0.60 you rewrite the instructions.

    The internal tax scenario classifier Annotation Workflow

    Building the internal tax scenario classifier meant creating more than 10,000 training examples for tax scenario classification, and the workflow reflected everything above. Three annotators reviewed each batch. Every batch opened with a pre-calibration session using 50 canonical examples, so everyone was anchored to the same interpretation before touching new data. A resolution protocol handled disagreements — an adjudicator reviewed any annotator pair that fell below a kappa of 0.75. The first 3,000 examples took six weeks. The annotating itself was fast; what was slow was resolving rubric ambiguities, the genuinely hard classification calls where reasonable experts disagreed and the instructions had to be sharpened before we could proceed. That slowness was the work paying off, not the work failing.

    The Failure Mode

    The common failure is predictable. A team rushes annotation, skips the pre-calibration session, skips the inter-rater checks, and ships a large pile of labeled examples on a deadline. Then they fine-tune and discover the model is worse than the base model on their own task. They blame the model, the framework, the learning rate — anything but the data. The model learned exactly what it was given, which was noise dressed up as signal. There is no hyperparameter that recovers from a corrupted training set.

    The Quality Floor

    Here is the number that should reshape how teams budget annotation effort: 1,000 high-quality annotated examples with a kappa above 0.80 beats 20,000 noisy ones for fine-tuning. This is consistent across the fine-tuning literature and consistent with my own production experience. The instinct to gather more data is usually the wrong instinct. The right instinct is to make the data you already have cleaner — tighten the rubric, recalibrate the annotators, and accept a smaller, sharper training set over a large, blurry one.

    Documenting Edge Cases as Your Test Set

    For every annotation batch, I require documenting the 20 to 30 cases that were hardest to classify and why they were hard. Those documented edge cases become the test set for evaluating the fine-tuned model — deliberately, not random samples pulled from the annotation pool. A random test set tells you how the model does on average. The hard-case test set tells you how it does exactly where it’s most likely to break, which is the only place where evaluation earns its keep. The same rigor shows up across regulated deployments, which I wrote about in lessons from 17 enterprise projects.

    What Transfers

    This is clinical assessment methodology applied to machine learning annotation. Structured clinical interviews, rubric-based scoring, inter-rater reliability checks — all of these exist in clinical psychology, and they exist there for exactly the same reasons they matter in annotation. When you need multiple humans to apply the same judgment consistently to ambiguous material, the discipline is identical whether the material is a patient presentation or a tax scenario. The field that has spent decades measuring human agreement has a great deal to teach the field that just discovered it needs to.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Cloudflare Workers AI in 2026: Running Free Inference at the Edge

    Running LLM inference at 300 locations worldwide for free sounds like marketing copy — but it’s actually the production architecture I use for lightweight classification and embedding tasks across my fleet of 60-plus Cloudflare Workers, every one of them sitting at $0 per month. I want to walk through what Workers AI actually delivers in 2026, what the free tier buys in real terms, and where this approach stops making sense.

    The 2026 Model Catalog

    The Workers AI catalog in 2026 covers most of what an edge application needs without reaching for an external API. For text generation you have LLaMA 3 in both 8B and 70B variants, Mistral 7B for faster lighter work, and Gemma 2B when you need the smallest possible footprint. For embeddings, BAAI/bge-base-en-v1.5 produces 768-dimension vectors that I store directly in D1. Whisper-large-v2 handles transcription. The point is that a single Worker can generate text, embed documents, and transcribe audio without a single outbound fetch to a third-party service — the model runs on Cloudflare’s GPUs at the edge node closest to the request.

    What the Free Tier Actually Buys

    Workers AI bills in “neurons” — a normalized compute unit that abstracts away the differences between a tiny embedding call and a heavy 70B generation. The free tier gives you 10,000 neurons per day. That abstraction is confusing until you map it to real work, so here are the numbers from my own usage logs. Ten thousand neurons buys roughly 5,000 LLaMA 3 8B inference calls in a day, or about 50,000 embedding generations with bge-base, or close to 300 minutes of Whisper transcription. For a personal tool or a low-traffic production service, that ceiling is rarely the constraint — I have Workers that have never come close to exhausting a day’s allotment, which is how the whole fleet stays at zero cost.

    Production Use Cases From the Fleet

    The clearest example is daniel-job-search, my career intelligence PWA. It uses Workers AI with bge-base to generate embeddings for every incoming job description and for my Ideal Candidate Profile, then runs cosine similarity in D1 to score matches. The entire matching pipeline runs inside a Worker — no external API calls, no per-token billing, no network hop to a model provider. Digital Sentinel, my OSINT monitoring system, uses Workers AI text classification to triage alerts before routing the important ones to Discord, which means the noise filtering happens at the edge instead of clogging a downstream queue. I wrote more about that build in my post on running OSINT monitoring on the edge with Cloudflare Workers.

    Latency Benchmarks Measured in Production

    Numbers people quote in blog posts are usually synthetic. These are not — I pulled them from production traffic in June 2026. Embedding generation with bge-base runs at 35 to 45 milliseconds at the P50, fast enough that I embed inline during a request rather than queuing it. LLaMA 3 8B generation lands at 280 to 340 milliseconds for a 100-token output, which is comfortable for a summary that renders after a user action but too slow for anything you want to feel instantaneous. Knowing those two ranges shapes every design decision — if a task needs sub-50ms, it gets an embedding; if it can absorb a third of a second, it gets a generation; if it needs anything heavier, it leaves the edge entirely.

    When Not to Use Workers AI

    Edge inference is not a universal answer, and pretending otherwise leads to bad systems. Complex reasoning — multi-step analysis, nuanced judgment, anything where a wrong answer is expensive — belongs on Claude or GPT-4 through an external API, not on an 8B model at the edge. Tasks that reliably need more than a 4K context window are a poor fit, because you’ll spend more effort chunking than you save in latency. And if your task demands a fine-tuned model, Workers AI is the wrong tool — you fine-tune elsewhere and serve the result elsewhere. The free tier is generous precisely because it’s scoped to inference on stock models for bounded tasks.

    What Transfers

    The lesson generalizes well beyond Cloudflare. Run cheap models close to the user for latency-sensitive, low-stakes tasks, and route complex reasoning to centralized APIs that have the full context and the bigger models. The edge inference paradigm is about matching the model’s location and size to the task’s actual requirements — not defaulting to the largest model for everything, and not forcing every workload through one pipe. Cost discipline and latency discipline turn out to be the same discipline: put the right computation in the right place.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Applying Lean Six Sigma to AI Projects: A Green Belt’s Framework

    As a Lean Six Sigma Green Belt who has applied DMAIC to AI and ML projects across 19 years in defense finance, the most consistent failure pattern I’ve seen is AI teams who skip the Define and Measure phases entirely and go straight to building a model. They have a dataset, they have a use case, they have a timeline — and they have no baseline, no measurable problem statement, and no control plan. The model gets deployed. Three months later, nobody can prove it’s better than what it replaced.

    DMAIC — Define, Measure, Analyze, Improve, Control — is a 60-year-old quality framework that was not designed for AI. It maps onto AI project structure almost perfectly anyway, because the failure modes it was designed to prevent are the same failure modes that kill AI deployments.

    Define and Measure: The Phases That Prevent Failure

    The Define phase requires a problem statement with a measurable Y. Not “improve efficiency in the tax processing workflow” — that’s a direction, not a target. “Reduce purchase order error rate from 31% to below 10% in the internal tax scenario classifier pipeline, measured over a 90-day post-deployment window” is a problem statement. The Y is specific, the current state is quantified, the target is defined, and the measurement window is explicit. You cannot claim success without meeting that specification. You also cannot accidentally redefine success after the fact when results are disappointing, which is the most common way AI projects survive their pilots despite not working.

    The Measure phase is where most AI projects actually fail, because if you don’t have a baseline, you can’t prove improvement — and yet most teams skip it because it delays the part that feels like progress. The protocol is to baseline the current process for 4 to 6 weeks before any automation or model is deployed, using the exact same metrics you’ll use post-deployment. The 2,132 hours per year figure from a major defense aircraft program withholds automation — which justified $360,000 in annual cost avoidance — started as a manual time study. Analysts logged their time against specific task categories for 3 weeks before any automation existed. That baseline wasn’t overhead; it was the evidence that made the business case and gave us a threshold to beat.

    Analyze and Improve: Choosing the Right Intervention

    The Analyze phase is root cause work: understanding why the problem exists before deciding what to do about it. Is the error rate high because the data is bad? Because the process is poorly designed? Because humans are making decisions under time pressure with incomplete information? Because the system produces ambiguous outputs that require judgment calls? Each of those root causes has a different correct intervention — and sometimes that intervention is a process redesign that doesn’t involve AI at all.

    This is the discipline that prevents solution-chasing. When you start with a model and work backward to the problem it solves, you’ll find a problem that matches the model’s capabilities — whether or not it’s the actual root cause of the performance gap you’re trying to close. Starting with root cause analysis in Alteryx or Python to understand the data quality and process patterns before touching a model architecture is slower in the short term and dramatically more reliable in the long term.

    The Improve phase is where the AI model enters — framed as a process change, not as a technology implementation. Design it, run a controlled pilot with a scope limited enough to measure cleanly, compare against the baseline using the metrics defined in the Measure phase, and iterate. Don’t skip the pilot. A model that performs well on a holdout test set and a model that performs well in production are not the same thing, and the difference is almost always in the human and process factors that a test set doesn’t capture.

    Control: The Phase That Makes It a Production System

    The Control phase is what separates an AI project from an AI system. A project ends at deployment. A system has ongoing monitoring, drift detection, retraining protocols, and documented ownership. Without those, you have a model that was accurate when it launched and an unknown amount of time before it’s quietly wrong in ways nobody has instrumented to detect.

    The Control plan for an AI system needs to specify: what metrics indicate the model is performing within acceptable bounds, what threshold triggers a retraining evaluation, who owns the retraining decision, and what the rollback procedure is if a retrained model performs worse than its predecessor. These aren’t optional governance niceties — they’re the difference between a system that survives its first year and one that gets quietly retired when the person who built it moves on.

    The sigma equivalent framing is useful for calibrating expectations here. A 3-sigma AI system — 99.73% accuracy — sounds impressive. It produces 2,700 errors per million decisions. If those decisions are purchase order classifications affecting a multi-hundred-million-dollar contract financial position, 2,700 errors is not an acceptable defect rate. The sigma target has to be set against the consequence of the error, not against an abstract accuracy benchmark.

    What Transfers

    Six Sigma rigor applied to AI is not methodology overhead imposed by people who don’t understand how AI development works. It’s the framework that answers the question every executive eventually asks: “How do we know this is working?” The answer to that question has to be built into the project from the Define phase — you can’t construct it after deployment from whatever data happens to be available. The 17 AI and automation projects I led at Lockheed Martin that produced defensible, sustained business value all had one thing in common: the measurement infrastructure was designed before the model was built, not after.

    The teams that resist the DMAIC discipline usually frame it as a speed concern — moving fast, not getting bogged down in process. In practice, the projects that skip Define and Measure don’t move faster. They move quickly toward a result they can’t prove, then spend months in a political argument about whether the system is working. The upfront rigor is the shortcut.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • The Non-Linear Career Path: How Psychology + Finance Made Me a Better AI Engineer

    The question I get most often from people who look at my background is some version of: “How does a therapist end up doing AI work for defense contractors?” The answer is that the path wasn’t non-linear — it was compound. Each layer added something the previous layer couldn’t provide alone, and the combination produces a profile that’s genuinely unusual in AI deployment work.

    What the Therapy Training Actually Provided

    The Licensed Marriage and Family Therapist licensure isn’t a credential I hold as a curiosity. It came from graduate training in systems theory — understanding how behavior in one part of a system affects behavior in other parts — and from supervised clinical practice that required learning to hold complexity, ambiguity, and conflicting stakeholder interests simultaneously. Those skills transfer directly to AI deployment work, where the hardest problems are rarely technical. The hard problems are: why are the people this system is supposed to help resistant to using it, how do you design human-AI interaction so that people engage appropriately rather than over-trusting or under-trusting the outputs, and what happens to team dynamics when a model starts doing work that used to define someone’s role. A pure ML background doesn’t prepare you to navigate those questions. Clinical training does.

    What I/O Psychology Added

    Industrial-Organizational Psychology is the discipline that applies behavioral science to organizational contexts — selection, performance, motivation, team dynamics, organizational change. The master’s-level training in I/O psychology gave me a rigorous framework for thinking about how AI tools change work, not just how they perform tasks. When I was deploying automation in defense finance, the questions that mattered weren’t only “does the model work?” They were: how do we measure performance in a way that’s fair to the people whose work is changing, how do we design the transition so that the analysts whose time is being recovered feel like the beneficiaries rather than the displaced, and how do we structure human-AI collaboration so that human judgment is applied where it actually adds value rather than where it just feels necessary. I/O psychology has decades of research on exactly those questions.

    What Defense Finance Contributed

    Nineteen years in defense finance at Lockheed Martin provided two things that can’t be replicated by reading about them. The first is operational credibility: when I build an AI system for finance, I understand the workflows, the error consequences, the audit requirements, and the organizational dynamics of getting a finance team to trust a model. The second is a track record of consequential deployments: the major defense aircraft program withholds automation that produced $360K in annual cost avoidance and 2,132 recovered analyst hours per year, the internal tax scenario classifier that processed 415,000 tax scenarios in 4 hours and reduced purchase order error rates by 31%. Those outcomes exist because the technical work was grounded in deep domain knowledge.

    Why the Combination Is Rare

    Most AI engineers don’t have clinical training. Most clinicians don’t have 19 years in defense finance. Most finance professionals don’t have the technical depth to build production AI systems. The combination is rare not because it’s strategically constructed but because each step seemed like the obvious next thing given where I was. The therapy training came first and was about understanding people. The finance career built on that. The AI work emerged from the automation problems the finance work surfaced. The I/O psychology formalized frameworks I was already applying intuitively. In retrospect the path looks deliberate; at each step it was just following what was interesting and useful.

    What Transfers

    The argument this background makes isn’t that every AI engineer needs clinical training or that domain expertise always compounds into something useful. The argument is that AI deployment problems are fundamentally human problems — adoption, trust, organizational change, consequence management — and the people best positioned to solve them are those who have spent time on both the technical and the human sides. A pure-ML background is increasingly common. The combination is not.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • LLM-as-Judge: When to Trust Your AI Evaluator (And When to Override It)

    LLM-as-judge is seductive because it solves an expensive problem cheaply: instead of paying human annotators to evaluate model outputs, you have a larger or more capable model do the grading. The cost savings are real. The risks are also real, and teams that adopt LLM-as-judge without understanding the failure modes end up with evaluation infrastructure that gives them confident-sounding metrics while quietly measuring the wrong thing.

    When It Works

    LLM-as-judge works well when the evaluation criteria are explicit and the judge model can be given precise instructions that human annotators would also receive. Factual accuracy, format compliance, completeness against a checklist, tone matching against a defined style guide — these are tasks where a capable judge model can achieve inter-rater reliability with human annotation that makes it a reasonable substitute. The test for whether your LLM-as-judge setup is working is to run it in parallel with human annotation on a sample and calculate Cohen’s kappa between the two. If kappa is above 0.7, your automated evaluation is reliable enough to trust at scale. Below that threshold, you’re measuring something that doesn’t sufficiently correlate with human judgment to be useful.

    When It Fails

    LLM-as-judge fails predictably on tasks that require domain expertise the judge model doesn’t have, tasks where the rubric is genuinely ambiguous, and tasks where the judge model has systematic biases that don’t match the target population. On domain expertise: a general-purpose model grading the accuracy of specialized tax scenario classification outputs is grading against its own training data, not against ground truth. The grade it gives correlates with how confident the output sounds, not whether it’s correct. On rubric ambiguity: if the evaluation criteria aren’t precise enough for two human annotators to agree consistently, the judge model will also be inconsistent — but it will be inconsistent in a way that looks consistent because it applies the same internal model repeatedly. On bias: LLMs have documented preferences for certain output styles, lengths, and formats that don’t always correlate with quality. A judge model that prefers longer, more hedged responses will systematically grade those outputs higher regardless of their actual correctness.

    Calibration Against Human Annotation

    The calibration protocol that worked in the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours — applied the same methodology to LLM-as-judge validation. Before deploying automated evaluation at scale, we sampled 200 outputs and had both human annotators and the judge model evaluate them independently. The kappa between human annotators came in at 0.81 on unambiguous cases and 0.54 on boundary cases. The kappa between human annotation and LLM-as-judge came in at 0.78 on unambiguous cases and 0.41 on boundary cases. That calibration told us exactly where to trust the automated evaluation and where to require human review: for clear-cut cases at scale, the judge model was reliable; for boundary cases, it was not, and routing those to human annotators was the correct design.

    The Consistency Trap

    The most dangerous property of LLM-as-judge is that it can be consistently wrong. A human annotation process with poor inter-rater reliability at least has the property that its errors are somewhat random — with enough annotators, the noise averages out. A judge model applies the same systematic bias repeatedly, which means the errors don’t average out; they compound. If your judge model systematically misclassifies a category of outputs, every data point in that category gets the wrong grade, and your optimization process confidently drives the model you’re evaluating in the wrong direction. The consistency that makes LLM-as-judge seem reliable is also what makes its systematic errors harder to detect.

    Practical Implementation Guidelines

    Three guidelines have held up across my use of LLM-as-judge in production systems. First, calibrate before deploying: run the judge in parallel with human annotation on a representative sample and calculate kappa before trusting it at scale. Second, stratify by case type: most evaluation tasks have a subset of cases where automated grading is reliable and a subset where it isn’t — treat those subsets differently rather than applying a single threshold. Third, re-calibrate periodically: the judge model’s behavior can change across versions, and a calibration established on one model version doesn’t automatically transfer to the next.

    What Transfers

    The clinical assessment parallel holds here too. Structured clinical interviews have a validated set of questions and scoring criteria specifically because clinician judgment varies in ways that affect diagnostic reliability — the same symptoms get different interpretations from different practitioners without standardized tools. LLM-as-judge is automated judgment, and it has the same reliability problem as human judgment: it needs to be calibrated, validated, and monitored rather than trusted by default. The tools for doing that — inter-rater reliability measurement, calibration protocols, stratified validation — come from the same field that figured this out for human judgment decades ago.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Explainability for Executives: Translating Model Outputs Into Decisions

    The meeting that clarified the explainability problem for me happened when a CFO looked at a dashboard I’d helped deploy, saw a field labeled “anomaly score: 0.87,” and asked — with complete seriousness, not as a challenge — “What does 0.87 mean? Is that good?” The room went quiet. The team had spent months on the model. They had spent approximately zero time on what the model’s outputs should mean to the person who was supposed to act on them.

    Three Levels of Explainability That Actually Matter

    Enterprise AI systems serve at least three distinct audiences, and each audience needs a different form of explanation. Conflating them — building one explanation interface and pointing everyone at it — produces a tool that serves none of them well.

    The operational level is for the analyst who runs the system daily. She needs to understand what the model did on this specific transaction or record — which inputs drove the output, whether the result is consistent with similar cases she’s reviewed, and what she should check if she wants to verify or override it. Technical detail is appropriate here. She uses the system 40 hours a week and can develop fluency with model-specific language.

    The managerial level is for the supervisor or team lead who reviews AI-flagged items and decides how to allocate analyst time. He needs to know what action to take — which flags require immediate investigation, which can wait, and how to triage a backlog of 200 alerts that all show scores above 0.7. He does not need to know how the model produces a score. He needs to know what the score implies about urgency and risk in terms he already uses to manage his team.

    The executive level is for the CFO, program director, or board member who needs to understand what the AI system means for the business — not what it does technically, but what decisions it supports, what it can’t tell you, and what it would take for you to rely on it for a consequential call. Most AI teams never build this level of explanation at all. They present the model’s performance metrics — accuracy, F1 score, precision-recall curve — to audiences who have no frame for evaluating what those numbers mean in operational terms.

    Why SHAP Values Don’t Belong in a Leadership Meeting

    SHAP values are genuinely useful for the operational level — an analyst who wants to understand why the model flagged a particular record can look at feature attributions and build intuition about the system’s behavior over time. That is a legitimate use of the tool.

    Presenting SHAP values to an executive is a category error. It’s not that executives are unsophisticated — it’s that SHAP values answer a question they’re not asking. They’re asking whether to trust the system’s recommendation on a $2M procurement decision. A bar chart showing that “vendor tenure” contributed 0.34 to the anomaly score does not answer that question. It transfers the cognitive work of interpretation to someone who shouldn’t have to do it.

    What works at the executive level is natural language that maps model outputs to business concepts they already use. “This purchase order is $142K above the 3-year average for this vendor category” is an explanation. “Anomaly score: 0.87” is an output that requires explanation. Counterfactuals work well too — “If the forecast had used Q3 actuals instead of Q2 estimates, the variance flag would not have fired” gives an executive the conditional reasoning she needs to evaluate whether the flag is meaningful or an artifact of a data lag.

    The Dashboard Redesign That Moved Adoption by 44 Points

    At Lockheed, a finance AI dashboard that had been running at 34% active adoption for three months went through an explainability redesign. The change was not to the model — the underlying system was unchanged. The change was entirely in the output layer: probability scores were replaced with natural language summaries written in the vocabulary of the finance analysts who used it. “Probability: 0.73” became “This cost pattern occurs in fewer than 8% of similar transactions — the last time this pattern appeared in this cost center, the variance was caused by a billing timing mismatch that took 11 days to resolve.”

    Two months after the redesign, active adoption was at 78%. The model hadn’t changed. The explainability layer had. Analysts who had been ignoring the dashboard because they couldn’t tell whether 0.73 warranted action were now using it because the output spoke in terms of their existing workflow and experience.

    What Transfers

    In clinical training, one of the first things you learn is to speak the client’s language rather than the DSM’s. A client describing their experience as “I get really wound up and can’t settle down” is telling you something useful. Responding with “that sounds like hyperarousal consistent with an anxiety spectrum presentation” communicates nothing to them — it’s accurate and useless. The skill is translation: taking precise clinical language and rendering it in terms the person in front of you can act on.

    AI explainability design is the same skill applied to a different context. The one-sentence test I apply before any model goes to a leadership meeting: can I explain this system’s decision in one sentence to a non-technical executive who will use that explanation to make a consequential call? If the answer is no, the model isn’t ready for that meeting — not because the model is bad, but because the explanation layer isn’t built yet. Technical completeness and operational readiness are not the same thing, and confusing them is how teams ship AI systems that sit unused at 34% adoption when they had the capacity to reach 78%.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • How to Quantify AI ROI Without Misleading Your Stakeholders

    The $3M in savings slide is almost always wrong — not because the savings don’t exist, but because the methodology didn’t hold up to scrutiny when anyone actually looked.

    I’ve sat in enough enterprise finance reviews — 19 years of them at Lockheed Martin, many directly tied to AI and automation projects I led — to recognize the pattern. A team builds something that genuinely improves a process, they attach a number to the improvement, and then that number gets challenged in a budget review and can’t be defended. The project survives or doesn’t based on whether the finance organization decides to extend credibility rather than because the ROI case was actually sound.

    Two Failure Modes Before You Calculate Anything

    Underselling is the failure mode that gets less attention. ROI exists — real hours are recovered, real error rates drop — but the team has no rigorous way to measure it, so the project gets framed as qualitative improvement. Qualitative improvements don’t survive budget compression. They get cut when finance needs to find savings and the project has nothing auditable to defend itself with.

    Overselling is more visible and more damaging to the broader AI program. A team attributes all observed efficiency gains to the AI system when the gains were actually produced by a combination of the tool, a concurrent process redesign, an analyst who changed how she organized her work, and a quarter with lower transaction volume than the prior year. If you claim credit for all of it and a skeptical CFO asks how you isolated the AI contribution, you don’t have an answer. That moment — one moment — kills credibility for the next five AI proposals your organization brings forward.

    The Costs That Disappear From ROI Slides

    Every ROI methodology I’ve seen that didn’t survive scrutiny had the same structural problem: it counted only the benefits of the AI and the initial build cost, then stopped. The cost categories that routinely disappear from the slide are annotation and labeling cost for supervised systems, the retraining cadence — a model that needs quarterly retraining has a recurring engineering cost that should appear in the denominator, human review of AI outputs in any high-stakes workflow, and the cost of errors the system makes rather than only the errors it prevents.

    That last one is particularly common in automation ROI. A system that processes 10,000 transactions per month with a 0.5% error rate introduces 50 errors per month that someone has to catch and correct. If those errors are in finance or compliance workflows, the correction cost — loaded labor rate × correction time, plus any downstream rework — can meaningfully offset the efficiency gains. An honest ROI methodology puts that number in the model. A credibility-optimized ROI methodology hopes no one asks about it.

    How a Defense Finance Automation ROI Number Was Built

    The $360K annual cost avoidance figure from a major defense aircraft program withholds automation project held up to finance leadership scrutiny because it was constructed to be auditable, not to be impressive. The methodology: loaded labor rate for the analyst population multiplied by 2,132 recovered hours per year — hours that were timed, not estimated, against a documented pre-automation baseline — minus annual system maintenance cost, minus periodic retraining and validation cost for the Alteryx and Python workflows involved, minus an allocation for edge cases that still required manual intervention.

    The methodology was presented visibly, not buried in appendices. Finance leadership could see exactly which assumptions the number depended on and could stress-test any of them. When the program office asked what the number looked like if loaded labor rate was adjusted down by 15%, we had that sensitivity analysis ready. The figure that survived scrutiny was lower than the first draft — and more credible precisely because of that.

    Framing ROI by Audience

    A single ROI figure presented identically to every audience is a communication failure before it’s a methodology failure. CFOs want payback period and net present value — they’re evaluating capital allocation and need to compare this project against others competing for the same budget. Operational leaders want hours recovered per analyst per month because they’re managing capacity and need to understand what changes for their team on Monday. CTOs want the technical debt comparison — build versus buy, and what the maintenance cost trajectory looks like in years two and three when the team that built the system has moved on.

    Each of those framings uses the same underlying data. The work is disaggregating the ROI model into components that answer the specific question each audience is actually asking. Presenting a single number to all three audiences and hoping it resonates is why most AI ROI conversations stall in the room rather than producing decisions.

    What Transfers

    Financial rigor from defense FP&A is not defense-specific. The discipline of building auditable cost and benefit models — with documented baselines, explicit attribution logic, visible cost categories, and audience-specific framings — applies to any AI project that needs to survive budget scrutiny. The aerospace context made the stakes higher and the review process more formal, but the underlying methodology is the same one any enterprise AI team should be using. The first AI project that can’t defend its ROI doesn’t just lose its own budget. It makes the case against the next five.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Building RAG Over 10,000 Research Papers: Architecture and Hard Lessons

    I built a retrieval-augmented generation system over 10,000 academic papers in I/O psychology, organizational behavior, and AI ethics — not as a demonstration project, but because I needed a research assistant that could actually find the supporting evidence buried in dense methodology sections and tell me where it came from.

    Why Academic Paper RAG Is Harder Than Product RAG

    Most RAG tutorials assume clean, well-structured source documents — FAQs, product manuals, internal wikis. Academic PDFs are none of those things. Multi-column layouts break naive text extractors. Equations render as gibberish or disappear entirely. Citations frequently span page breaks, so the reference text you want is split across two chunks. Abstracts — the obvious candidate for a summary embedding — often don’t reflect the actual contribution of the paper. A paper titled “A Meta-Analysis of TAM Adoption Predictors” might have its most useful content buried in a limitations section that no one would retrieve based on the title alone.

    The naive approach — extract all text, split into 512-token chunks with 50-token overlap, embed, and retrieve — produces results that look plausible but miss the actual evidence. For casual Q&A over product documentation, that’s acceptable. For research work where you’re trying to accurately represent the state of evidence on a topic, it’s not.

    The Chunking Strategy That Actually Works

    The architecture I settled on uses section-aware chunking rather than a sliding window. Each paper is parsed to extract named sections — Introduction, Methods, Results, Discussion, Limitations — and each section becomes its own chunk set. Every chunk carries metadata: paper title, publication year, authors, journal, and section name. That metadata travels with the chunk through retrieval and into the prompt.

    The reason section-level metadata matters is that the provenance of a claim is different depending on whether it comes from a Results section or a Discussion section. “Participants showed a 23% improvement in task completion” means something different in Results than in Discussion. Knowing which section a retrieved chunk came from changes how you should weight it in your synthesis.

    For the embedding model, I started with text-embedding-3-small and moved to text-embedding-3-large for this corpus specifically. The quality difference for dense academic prose — methodology descriptions, statistical terminology, theoretical constructs — was meaningful enough to justify the cost difference. For retrieval over casual conversational text, small would have been fine.

    Hybrid Retrieval and the Reranker You Cannot Skip

    Academic text has a property that makes pure vector retrieval underperform: exact terminology matters enormously. When a research question involves “Cohen’s kappa” or “confirmatory factor analysis” or “heteroscedasticity,” semantic similarity search will retrieve conceptually adjacent material that doesn’t actually contain the relevant methodology. BM25 keyword retrieval catches exact matches that vector search misses.

    The retrieval pipeline runs both — vector search weighted at 0.6, BM25 at 0.4 — then merges the candidate pools. The weight split was empirically tuned on a held-out evaluation set of 200 research questions where I had manually verified the correct source chunks. Getting the balance wrong in either direction cost meaningful retrieval quality.

    The reranker is not optional. Without a cross-encoder reranker applied to the merged candidate pool, the top-ranked results consistently look relevant — they match on topic — but miss the actual supporting evidence. The cross-encoder evaluates query-chunk relevance jointly rather than independently, which costs more compute but catches the cases where a chunk that seems topically relevant doesn’t actually answer the question. I tested the pipeline with and without reranking on the same 200-question evaluation set, and the difference in retrieval precision was large enough that I would not ship academic RAG without it.

    Three Hard Lessons

    The first hard lesson was the reranker, already described. The second was metadata filtering by publication year. A 2018 paper’s recommendations for model evaluation benchmarks, fairness metrics, or dataset standards may be functionally obsolete. Without a year filter built into the retrieval interface, the system would confidently surface outdated methodology as current best practice. Year-range filtering is now a first-class parameter in every query I run against this corpus.

    The third lesson was citation chains. The paper that directly answers your question is frequently not the paper that contains the data supporting its claim — it cites another paper, which may cite a third. A retrieval system that only returns the top-ranked chunks and stops there will miss this structure entirely. I added a citation-following step that, when a retrieved chunk contains a reference to a specific study, fetches that study’s chunks and includes them as secondary context. This added complexity, but it’s the step that makes the difference between a system that finds what you asked for and a system that finds the actual evidence.

    What Transfers

    The chunking-with-metadata principle applies to any long-form document RAG — legal contracts, engineering manuals, case files, regulatory filings. The insight is the same: your chunks should carry enough provenance information that downstream reasoning can assess not just what a chunk says, but where it sits in the document’s argumentative structure and how current it is. Sliding-window chunking that strips that context produces retrieval that feels right more often than it is right — which is a failure mode that’s harder to catch than one that fails loudly. If you’re working on similar document retrieval problems, the broader lessons from enterprise AI deployment apply here as well: the architecture decisions that seem like premature optimization at 100 documents become the only thing standing between you and retrieval collapse at 10,000.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.