Grounded Intelligences

Tag: data-science

Data science practice, tools, and lessons from enterprise and independent engineering projects.

  • The Non-Linear Career Path: How Psychology + Finance Made Me a Better AI Engineer

    The question I get most often from people who look at my background is some version of: “How does a therapist end up doing AI work for defense contractors?” The answer is that the path wasn’t non-linear — it was compound. Each layer added something the previous layer couldn’t provide alone, and the combination produces a profile that’s genuinely unusual in AI deployment work.

    What the Therapy Training Actually Provided

    The Licensed Marriage and Family Therapist licensure isn’t a credential I hold as a curiosity. It came from graduate training in systems theory — understanding how behavior in one part of a system affects behavior in other parts — and from supervised clinical practice that required learning to hold complexity, ambiguity, and conflicting stakeholder interests simultaneously. Those skills transfer directly to AI deployment work, where the hardest problems are rarely technical. The hard problems are: why are the people this system is supposed to help resistant to using it, how do you design human-AI interaction so that people engage appropriately rather than over-trusting or under-trusting the outputs, and what happens to team dynamics when a model starts doing work that used to define someone’s role. A pure ML background doesn’t prepare you to navigate those questions. Clinical training does.

    What I/O Psychology Added

    Industrial-Organizational Psychology is the discipline that applies behavioral science to organizational contexts — selection, performance, motivation, team dynamics, organizational change. The master’s-level training in I/O psychology gave me a rigorous framework for thinking about how AI tools change work, not just how they perform tasks. When I was deploying automation in defense finance, the questions that mattered weren’t only “does the model work?” They were: how do we measure performance in a way that’s fair to the people whose work is changing, how do we design the transition so that the analysts whose time is being recovered feel like the beneficiaries rather than the displaced, and how do we structure human-AI collaboration so that human judgment is applied where it actually adds value rather than where it just feels necessary. I/O psychology has decades of research on exactly those questions.

    What Defense Finance Contributed

    Nineteen years in defense finance at Lockheed Martin provided two things that can’t be replicated by reading about them. The first is operational credibility: when I build an AI system for finance, I understand the workflows, the error consequences, the audit requirements, and the organizational dynamics of getting a finance team to trust a model. The second is a track record of consequential deployments: the major defense aircraft program withholds automation that produced $360K in annual cost avoidance and 2,132 recovered analyst hours per year, the internal tax scenario classifier that processed 415,000 tax scenarios in 4 hours and reduced purchase order error rates by 31%. Those outcomes exist because the technical work was grounded in deep domain knowledge.

    Why the Combination Is Rare

    Most AI engineers don’t have clinical training. Most clinicians don’t have 19 years in defense finance. Most finance professionals don’t have the technical depth to build production AI systems. The combination is rare not because it’s strategically constructed but because each step seemed like the obvious next thing given where I was. The therapy training came first and was about understanding people. The finance career built on that. The AI work emerged from the automation problems the finance work surfaced. The I/O psychology formalized frameworks I was already applying intuitively. In retrospect the path looks deliberate; at each step it was just following what was interesting and useful.

    What Transfers

    The argument this background makes isn’t that every AI engineer needs clinical training or that domain expertise always compounds into something useful. The argument is that AI deployment problems are fundamentally human problems — adoption, trust, organizational change, consequence management — and the people best positioned to solve them are those who have spent time on both the technical and the human sides. A pure-ML background is increasingly common. The combination is not.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • LLM-as-Judge: When to Trust Your AI Evaluator (And When to Override It)

    LLM-as-judge is seductive because it solves an expensive problem cheaply: instead of paying human annotators to evaluate model outputs, you have a larger or more capable model do the grading. The cost savings are real. The risks are also real, and teams that adopt LLM-as-judge without understanding the failure modes end up with evaluation infrastructure that gives them confident-sounding metrics while quietly measuring the wrong thing.

    When It Works

    LLM-as-judge works well when the evaluation criteria are explicit and the judge model can be given precise instructions that human annotators would also receive. Factual accuracy, format compliance, completeness against a checklist, tone matching against a defined style guide — these are tasks where a capable judge model can achieve inter-rater reliability with human annotation that makes it a reasonable substitute. The test for whether your LLM-as-judge setup is working is to run it in parallel with human annotation on a sample and calculate Cohen’s kappa between the two. If kappa is above 0.7, your automated evaluation is reliable enough to trust at scale. Below that threshold, you’re measuring something that doesn’t sufficiently correlate with human judgment to be useful.

    When It Fails

    LLM-as-judge fails predictably on tasks that require domain expertise the judge model doesn’t have, tasks where the rubric is genuinely ambiguous, and tasks where the judge model has systematic biases that don’t match the target population. On domain expertise: a general-purpose model grading the accuracy of specialized tax scenario classification outputs is grading against its own training data, not against ground truth. The grade it gives correlates with how confident the output sounds, not whether it’s correct. On rubric ambiguity: if the evaluation criteria aren’t precise enough for two human annotators to agree consistently, the judge model will also be inconsistent — but it will be inconsistent in a way that looks consistent because it applies the same internal model repeatedly. On bias: LLMs have documented preferences for certain output styles, lengths, and formats that don’t always correlate with quality. A judge model that prefers longer, more hedged responses will systematically grade those outputs higher regardless of their actual correctness.

    Calibration Against Human Annotation

    The calibration protocol that worked in the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours — applied the same methodology to LLM-as-judge validation. Before deploying automated evaluation at scale, we sampled 200 outputs and had both human annotators and the judge model evaluate them independently. The kappa between human annotators came in at 0.81 on unambiguous cases and 0.54 on boundary cases. The kappa between human annotation and LLM-as-judge came in at 0.78 on unambiguous cases and 0.41 on boundary cases. That calibration told us exactly where to trust the automated evaluation and where to require human review: for clear-cut cases at scale, the judge model was reliable; for boundary cases, it was not, and routing those to human annotators was the correct design.

    The Consistency Trap

    The most dangerous property of LLM-as-judge is that it can be consistently wrong. A human annotation process with poor inter-rater reliability at least has the property that its errors are somewhat random — with enough annotators, the noise averages out. A judge model applies the same systematic bias repeatedly, which means the errors don’t average out; they compound. If your judge model systematically misclassifies a category of outputs, every data point in that category gets the wrong grade, and your optimization process confidently drives the model you’re evaluating in the wrong direction. The consistency that makes LLM-as-judge seem reliable is also what makes its systematic errors harder to detect.

    Practical Implementation Guidelines

    Three guidelines have held up across my use of LLM-as-judge in production systems. First, calibrate before deploying: run the judge in parallel with human annotation on a representative sample and calculate kappa before trusting it at scale. Second, stratify by case type: most evaluation tasks have a subset of cases where automated grading is reliable and a subset where it isn’t — treat those subsets differently rather than applying a single threshold. Third, re-calibrate periodically: the judge model’s behavior can change across versions, and a calibration established on one model version doesn’t automatically transfer to the next.

    What Transfers

    The clinical assessment parallel holds here too. Structured clinical interviews have a validated set of questions and scoring criteria specifically because clinician judgment varies in ways that affect diagnostic reliability — the same symptoms get different interpretations from different practitioners without standardized tools. LLM-as-judge is automated judgment, and it has the same reliability problem as human judgment: it needs to be calibrated, validated, and monitored rather than trusted by default. The tools for doing that — inter-rater reliability measurement, calibration protocols, stratified validation — come from the same field that figured this out for human judgment decades ago.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Explainability for Executives: Translating Model Outputs Into Decisions

    The meeting that clarified the explainability problem for me happened when a CFO looked at a dashboard I’d helped deploy, saw a field labeled “anomaly score: 0.87,” and asked — with complete seriousness, not as a challenge — “What does 0.87 mean? Is that good?” The room went quiet. The team had spent months on the model. They had spent approximately zero time on what the model’s outputs should mean to the person who was supposed to act on them.

    Three Levels of Explainability That Actually Matter

    Enterprise AI systems serve at least three distinct audiences, and each audience needs a different form of explanation. Conflating them — building one explanation interface and pointing everyone at it — produces a tool that serves none of them well.

    The operational level is for the analyst who runs the system daily. She needs to understand what the model did on this specific transaction or record — which inputs drove the output, whether the result is consistent with similar cases she’s reviewed, and what she should check if she wants to verify or override it. Technical detail is appropriate here. She uses the system 40 hours a week and can develop fluency with model-specific language.

    The managerial level is for the supervisor or team lead who reviews AI-flagged items and decides how to allocate analyst time. He needs to know what action to take — which flags require immediate investigation, which can wait, and how to triage a backlog of 200 alerts that all show scores above 0.7. He does not need to know how the model produces a score. He needs to know what the score implies about urgency and risk in terms he already uses to manage his team.

    The executive level is for the CFO, program director, or board member who needs to understand what the AI system means for the business — not what it does technically, but what decisions it supports, what it can’t tell you, and what it would take for you to rely on it for a consequential call. Most AI teams never build this level of explanation at all. They present the model’s performance metrics — accuracy, F1 score, precision-recall curve — to audiences who have no frame for evaluating what those numbers mean in operational terms.

    Why SHAP Values Don’t Belong in a Leadership Meeting

    SHAP values are genuinely useful for the operational level — an analyst who wants to understand why the model flagged a particular record can look at feature attributions and build intuition about the system’s behavior over time. That is a legitimate use of the tool.

    Presenting SHAP values to an executive is a category error. It’s not that executives are unsophisticated — it’s that SHAP values answer a question they’re not asking. They’re asking whether to trust the system’s recommendation on a $2M procurement decision. A bar chart showing that “vendor tenure” contributed 0.34 to the anomaly score does not answer that question. It transfers the cognitive work of interpretation to someone who shouldn’t have to do it.

    What works at the executive level is natural language that maps model outputs to business concepts they already use. “This purchase order is $142K above the 3-year average for this vendor category” is an explanation. “Anomaly score: 0.87” is an output that requires explanation. Counterfactuals work well too — “If the forecast had used Q3 actuals instead of Q2 estimates, the variance flag would not have fired” gives an executive the conditional reasoning she needs to evaluate whether the flag is meaningful or an artifact of a data lag.

    The Dashboard Redesign That Moved Adoption by 44 Points

    At Lockheed, a finance AI dashboard that had been running at 34% active adoption for three months went through an explainability redesign. The change was not to the model — the underlying system was unchanged. The change was entirely in the output layer: probability scores were replaced with natural language summaries written in the vocabulary of the finance analysts who used it. “Probability: 0.73” became “This cost pattern occurs in fewer than 8% of similar transactions — the last time this pattern appeared in this cost center, the variance was caused by a billing timing mismatch that took 11 days to resolve.”

    Two months after the redesign, active adoption was at 78%. The model hadn’t changed. The explainability layer had. Analysts who had been ignoring the dashboard because they couldn’t tell whether 0.73 warranted action were now using it because the output spoke in terms of their existing workflow and experience.

    What Transfers

    In clinical training, one of the first things you learn is to speak the client’s language rather than the DSM’s. A client describing their experience as “I get really wound up and can’t settle down” is telling you something useful. Responding with “that sounds like hyperarousal consistent with an anxiety spectrum presentation” communicates nothing to them — it’s accurate and useless. The skill is translation: taking precise clinical language and rendering it in terms the person in front of you can act on.

    AI explainability design is the same skill applied to a different context. The one-sentence test I apply before any model goes to a leadership meeting: can I explain this system’s decision in one sentence to a non-technical executive who will use that explanation to make a consequential call? If the answer is no, the model isn’t ready for that meeting — not because the model is bad, but because the explanation layer isn’t built yet. Technical completeness and operational readiness are not the same thing, and confusing them is how teams ship AI systems that sit unused at 34% adoption when they had the capacity to reach 78%.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • How to Quantify AI ROI Without Misleading Your Stakeholders

    The $3M in savings slide is almost always wrong — not because the savings don’t exist, but because the methodology didn’t hold up to scrutiny when anyone actually looked.

    I’ve sat in enough enterprise finance reviews — 19 years of them at Lockheed Martin, many directly tied to AI and automation projects I led — to recognize the pattern. A team builds something that genuinely improves a process, they attach a number to the improvement, and then that number gets challenged in a budget review and can’t be defended. The project survives or doesn’t based on whether the finance organization decides to extend credibility rather than because the ROI case was actually sound.

    Two Failure Modes Before You Calculate Anything

    Underselling is the failure mode that gets less attention. ROI exists — real hours are recovered, real error rates drop — but the team has no rigorous way to measure it, so the project gets framed as qualitative improvement. Qualitative improvements don’t survive budget compression. They get cut when finance needs to find savings and the project has nothing auditable to defend itself with.

    Overselling is more visible and more damaging to the broader AI program. A team attributes all observed efficiency gains to the AI system when the gains were actually produced by a combination of the tool, a concurrent process redesign, an analyst who changed how she organized her work, and a quarter with lower transaction volume than the prior year. If you claim credit for all of it and a skeptical CFO asks how you isolated the AI contribution, you don’t have an answer. That moment — one moment — kills credibility for the next five AI proposals your organization brings forward.

    The Costs That Disappear From ROI Slides

    Every ROI methodology I’ve seen that didn’t survive scrutiny had the same structural problem: it counted only the benefits of the AI and the initial build cost, then stopped. The cost categories that routinely disappear from the slide are annotation and labeling cost for supervised systems, the retraining cadence — a model that needs quarterly retraining has a recurring engineering cost that should appear in the denominator, human review of AI outputs in any high-stakes workflow, and the cost of errors the system makes rather than only the errors it prevents.

    That last one is particularly common in automation ROI. A system that processes 10,000 transactions per month with a 0.5% error rate introduces 50 errors per month that someone has to catch and correct. If those errors are in finance or compliance workflows, the correction cost — loaded labor rate × correction time, plus any downstream rework — can meaningfully offset the efficiency gains. An honest ROI methodology puts that number in the model. A credibility-optimized ROI methodology hopes no one asks about it.

    How a Defense Finance Automation ROI Number Was Built

    The $360K annual cost avoidance figure from a major defense aircraft program withholds automation project held up to finance leadership scrutiny because it was constructed to be auditable, not to be impressive. The methodology: loaded labor rate for the analyst population multiplied by 2,132 recovered hours per year — hours that were timed, not estimated, against a documented pre-automation baseline — minus annual system maintenance cost, minus periodic retraining and validation cost for the Alteryx and Python workflows involved, minus an allocation for edge cases that still required manual intervention.

    The methodology was presented visibly, not buried in appendices. Finance leadership could see exactly which assumptions the number depended on and could stress-test any of them. When the program office asked what the number looked like if loaded labor rate was adjusted down by 15%, we had that sensitivity analysis ready. The figure that survived scrutiny was lower than the first draft — and more credible precisely because of that.

    Framing ROI by Audience

    A single ROI figure presented identically to every audience is a communication failure before it’s a methodology failure. CFOs want payback period and net present value — they’re evaluating capital allocation and need to compare this project against others competing for the same budget. Operational leaders want hours recovered per analyst per month because they’re managing capacity and need to understand what changes for their team on Monday. CTOs want the technical debt comparison — build versus buy, and what the maintenance cost trajectory looks like in years two and three when the team that built the system has moved on.

    Each of those framings uses the same underlying data. The work is disaggregating the ROI model into components that answer the specific question each audience is actually asking. Presenting a single number to all three audiences and hoping it resonates is why most AI ROI conversations stall in the room rather than producing decisions.

    What Transfers

    Financial rigor from defense FP&A is not defense-specific. The discipline of building auditable cost and benefit models — with documented baselines, explicit attribution logic, visible cost categories, and audience-specific framings — applies to any AI project that needs to survive budget scrutiny. The aerospace context made the stakes higher and the review process more formal, but the underlying methodology is the same one any enterprise AI team should be using. The first AI project that can’t defend its ROI doesn’t just lose its own budget. It makes the case against the next five.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Building RAG Over 10,000 Research Papers: Architecture and Hard Lessons

    I built a retrieval-augmented generation system over 10,000 academic papers in I/O psychology, organizational behavior, and AI ethics — not as a demonstration project, but because I needed a research assistant that could actually find the supporting evidence buried in dense methodology sections and tell me where it came from.

    Why Academic Paper RAG Is Harder Than Product RAG

    Most RAG tutorials assume clean, well-structured source documents — FAQs, product manuals, internal wikis. Academic PDFs are none of those things. Multi-column layouts break naive text extractors. Equations render as gibberish or disappear entirely. Citations frequently span page breaks, so the reference text you want is split across two chunks. Abstracts — the obvious candidate for a summary embedding — often don’t reflect the actual contribution of the paper. A paper titled “A Meta-Analysis of TAM Adoption Predictors” might have its most useful content buried in a limitations section that no one would retrieve based on the title alone.

    The naive approach — extract all text, split into 512-token chunks with 50-token overlap, embed, and retrieve — produces results that look plausible but miss the actual evidence. For casual Q&A over product documentation, that’s acceptable. For research work where you’re trying to accurately represent the state of evidence on a topic, it’s not.

    The Chunking Strategy That Actually Works

    The architecture I settled on uses section-aware chunking rather than a sliding window. Each paper is parsed to extract named sections — Introduction, Methods, Results, Discussion, Limitations — and each section becomes its own chunk set. Every chunk carries metadata: paper title, publication year, authors, journal, and section name. That metadata travels with the chunk through retrieval and into the prompt.

    The reason section-level metadata matters is that the provenance of a claim is different depending on whether it comes from a Results section or a Discussion section. “Participants showed a 23% improvement in task completion” means something different in Results than in Discussion. Knowing which section a retrieved chunk came from changes how you should weight it in your synthesis.

    For the embedding model, I started with text-embedding-3-small and moved to text-embedding-3-large for this corpus specifically. The quality difference for dense academic prose — methodology descriptions, statistical terminology, theoretical constructs — was meaningful enough to justify the cost difference. For retrieval over casual conversational text, small would have been fine.

    Hybrid Retrieval and the Reranker You Cannot Skip

    Academic text has a property that makes pure vector retrieval underperform: exact terminology matters enormously. When a research question involves “Cohen’s kappa” or “confirmatory factor analysis” or “heteroscedasticity,” semantic similarity search will retrieve conceptually adjacent material that doesn’t actually contain the relevant methodology. BM25 keyword retrieval catches exact matches that vector search misses.

    The retrieval pipeline runs both — vector search weighted at 0.6, BM25 at 0.4 — then merges the candidate pools. The weight split was empirically tuned on a held-out evaluation set of 200 research questions where I had manually verified the correct source chunks. Getting the balance wrong in either direction cost meaningful retrieval quality.

    The reranker is not optional. Without a cross-encoder reranker applied to the merged candidate pool, the top-ranked results consistently look relevant — they match on topic — but miss the actual supporting evidence. The cross-encoder evaluates query-chunk relevance jointly rather than independently, which costs more compute but catches the cases where a chunk that seems topically relevant doesn’t actually answer the question. I tested the pipeline with and without reranking on the same 200-question evaluation set, and the difference in retrieval precision was large enough that I would not ship academic RAG without it.

    Three Hard Lessons

    The first hard lesson was the reranker, already described. The second was metadata filtering by publication year. A 2018 paper’s recommendations for model evaluation benchmarks, fairness metrics, or dataset standards may be functionally obsolete. Without a year filter built into the retrieval interface, the system would confidently surface outdated methodology as current best practice. Year-range filtering is now a first-class parameter in every query I run against this corpus.

    The third lesson was citation chains. The paper that directly answers your question is frequently not the paper that contains the data supporting its claim — it cites another paper, which may cite a third. A retrieval system that only returns the top-ranked chunks and stops there will miss this structure entirely. I added a citation-following step that, when a retrieved chunk contains a reference to a specific study, fetches that study’s chunks and includes them as secondary context. This added complexity, but it’s the step that makes the difference between a system that finds what you asked for and a system that finds the actual evidence.

    What Transfers

    The chunking-with-metadata principle applies to any long-form document RAG — legal contracts, engineering manuals, case files, regulatory filings. The insight is the same: your chunks should carry enough provenance information that downstream reasoning can assess not just what a chunk says, but where it sits in the document’s argumentative structure and how current it is. Sliding-window chunking that strips that context produces retrieval that feels right more often than it is right — which is a failure mode that’s harder to catch than one that fails loudly. If you’re working on similar document retrieval problems, the broader lessons from enterprise AI deployment apply here as well: the architecture decisions that seem like premature optimization at 100 documents become the only thing standing between you and retrieval collapse at 10,000.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • What I/O Psychologists Know About AI Adoption That Engineers Miss

    My M.S. in Industrial/Organizational Psychology has proven more useful for shipping AI systems than most engineers expect, and I say that having spent 19 years at Lockheed Martin watching technically excellent tools fail because their designers never studied how human beings actually adopt new behaviors inside organizations.

    The Technology Acceptance Model Is Right and Also Insufficient

    Every I/O psychology graduate student learns the Technology Acceptance Model — Davis’s framework from 1989 that predicts adoption from two variables: perceived usefulness and perceived ease of use. TAM holds up remarkably well across four decades of research. If people don’t believe a tool will make their work better, or if they find it confusing to operate, they won’t use it. That part is correct.

    What TAM underweights — and what I watched play out repeatedly across 17 AI and automation projects — is the social and identity layer. Perceived usefulness is not calculated in isolation. It’s calculated in comparison: useful compared to what I do now, useful according to people I respect, threatening to what I’m known for being good at. The research on social influence in technology adoption has expanded substantially since Davis wrote his original paper, and the organizational behavior literature is unambiguous: your colleagues’ opinions about a tool predict your adoption of it more reliably than the documentation does.

    Two Adoption Outcomes from the Same Organization

    At Lockheed, I watched two AI-adjacent tools launch into similar finance analyst populations within the same 18-month window. The first was technically superior — better accuracy, cleaner interface, faster outputs. Adoption landed below 20% after 90 days and never recovered. The tool was positioned, implicitly through its design and explicitly through its rollout messaging, as a replacement for the judgment the analysts had spent years developing. When a senior analyst felt the tool was saying her expertise was no longer necessary, she didn’t complain loudly. She just stopped opening it.

    The second tool, which I helped design and position, reached 94% adoption within 60 days. The difference was not the algorithm. The difference was a deliberate framing decision made before a single line of code was written: the AI surfaces patterns and flags anomalies, and the analyst decides what they mean. Every interface element, every training session, every executive communication reinforced that frame. The analysts’ judgment was the point. The tool was the assistant.

    The Champion Model: Influence Nodes Before Broad Rollout

    Organizational psychology has a well-documented framework for change diffusion — Rogers’s Diffusion of Innovations — and it tells you something specific about early adopters: they are not just first users. They are influence nodes. Their opinion ripples through the social network of the team in ways that documentation and training cannot replicate.

    My practice across enterprise AI deployments is to identify three to five people who carry social credibility in the target group — not necessarily the most senior people, but the ones others ask for opinions — and get genuine buy-in from them before the general rollout. This is not manipulation. It’s recognizing that organizational adoption is a social process, not a rational calculation performed independently by each individual. If you skip this step and go straight to broad deployment, you’re hoping the math works in your favor. It often doesn’t.

    Resistance, when it comes, deserves the same interpretive generosity. When an analyst says a tool doesn’t work for her use case, she is frequently correct. The edge case she’s describing is real. Engineers who treat resistance as obstruction miss the signal — they get slower adoption and worse systems. Engineers who treat resistance as data get faster adoption and better systems.

    What Transfers

    The parallel that I find most clarifying comes from my training as a Licensed Marriage and Family Therapist. In clinical practice, the research on therapeutic outcomes is consistent: client buy-in to the treatment model must be established before behavior change begins. A technically correct intervention delivered to an ambivalent client produces worse outcomes than a slightly less refined intervention delivered to a client who understands and endorses the approach. The therapeutic alliance predicts outcomes more strongly than the specific technique.

    AI adoption follows the same structure. The quality of your model matters. The quality of your deployment relationship matters more. Teams that build organizational buy-in before broad rollout — that treat adoption as a human behavior problem rather than a communication problem — consistently outperform teams that don’t, regardless of the underlying technical quality of the system. I’ve written about the intersection of clinical training and AI systems design in more depth elsewhere, but the short version is this: the behavioral science was never separate from the engineering work. It was always the harder part.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • When AI Meets FP&A: Lessons from Automating Defense Finance

    The 2,132 hours per year figure is not an estimate. It came from a three-week manual time study conducted before any automation existed, where analysts logged their work against specific task categories while the baseline process ran unchanged. That number — and the $360,000+ in annual cost avoidance it represents — survived finance leadership scrutiny because the methodology was designed to be auditable before the first line of code was written.

    What the Problem Actually Was

    Defense program finance involves tracking a multi-hundred-million-dollar contract financial position tied to a major defense aircraft program — “withholds” being the portion of payments held back pending milestone, quality, or compliance conditions being met. Managing that position manually meant analysts reconciling data across multiple systems, building the same variance tables repeatedly, and spending a disproportionate share of their time on data assembly rather than analysis. The work wasn’t intellectually complex; it was procedurally complicated and time-consuming. That combination — low cognitive load, high time cost, clear rules — is exactly where automation performs well. The error rate in manual reconciliation was also meaningful: human error in a data assembly task with dozens of interdependent fields is not a character flaw; it’s a system design problem. Automating the assembly eliminates a category of error rather than reducing it incrementally.

    The Build: What Went In and What Didn’t

    The automation stack was Alteryx for data pipeline orchestration and Python for the more complex analytical components. Alteryx handled the extraction, transformation, and loading work: pulling from multiple source systems, applying business rules for categorization, and producing reconciled outputs. Python handled the variance detection logic — specifically, flagging variances that exceeded threshold values and generating the explanatory text that would previously require an analyst to write from scratch each cycle. What didn’t go into the automation: judgment calls. Variances above a certain dollar threshold still required an analyst to review the flag, understand the context, and decide whether escalation was warranted. Automating the detection is appropriate. Automating the decision is not, in a financial position with audit implications. The system was designed to produce recommendations, not approvals.

    What Went Wrong First

    The first version of the data pipeline produced reconciliation outputs that didn’t match the manual outputs analysts had been generating. Not because the logic was wrong — the logic was correct — but because the source data had inconsistencies that the manual process had been silently correcting through analyst judgment. Analysts knew that certain fields needed to be interpreted in context, not taken at face value, and they applied that interpretation automatically without documenting it. The automation exposed those undocumented corrections and made them explicit. The fix required going back to the analysts, documenting every correction they were making, and encoding that logic into the pipeline. This added three weeks to the build timeline and was entirely predictable if we had done more thorough process mapping upfront. The lesson isn’t that analysts were hiding things; it’s that tacit knowledge embedded in manual processes doesn’t surface until you try to replace the process with rules.

    Measuring the Outcome

    The 2,132 hours per year was calculated by comparing post-automation task logs against the pre-automation baseline, controlling for volume changes in the underlying program activity. The $360,000 figure used fully loaded labor rates — not salary alone — applied to the recovered hours, minus annual system maintenance costs and a periodic retraining allocation. The number presented to finance leadership was lower than the first draft, which made it more credible precisely because it showed the methodology rather than optimizing for the largest possible figure. Sensitivity analysis was prepared in advance: if the loaded labor rate assumption was adjusted down by 15%, the cost avoidance figure fell to $305,000. Finance could stress-test the assumption and the number still held.

    What Transfers

    Every element of this project — the time study methodology, the explicit encoding of tacit process knowledge, the threshold-based human review design, the auditable ROI methodology — transfers directly to enterprise AI work in any domain with similar characteristics: high-volume, rule-governed data tasks with meaningful error costs and financial accountability requirements. The aerospace context made the stakes and the review process more formal, but the discipline is domain-agnostic.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • LLM Evaluation at Scale: Building a Golden Dataset That Actually Works

    I have watched teams ship LLM features with no structured evaluation whatsoever — no golden dataset, no inter-rater agreement calculation, no adversarial examples — and then express genuine surprise three weeks later when production breaks in ways that dev never predicted. This is not a tooling problem or a research problem; it’s a discipline problem, and the fix is unglamorous: you need a curated, annotated evaluation set built before you ship, not assembled in a panic after your first production incident.

    What a Golden Dataset Actually Is

    A golden dataset is a curated collection of (input, expected_output) pairs that represent the real distribution of tasks your model will encounter in production — including the hard ones, the edge cases, and the inputs specifically designed to surface failure modes. It is not a sample of the easy inputs where the model already performs well. It is not a set of examples you grabbed from a tutorial. It is a structured artifact, maintained like code, that tells you whether your model is getting better or worse as you iterate. Three ingredients separate a real golden dataset from a list of test cases someone threw together on a Friday afternoon. First, it needs at least 500 examples — below that threshold, your accuracy measurements have confidence intervals wide enough to be meaningless. Second, it needs adversarial examples mixed with the easy ones in a ratio that reflects your actual production failure rate — if 15% of your production inputs are edge cases, your golden set should be at least 15% adversarial. Third, every example needs to be annotated by at least two independent annotators, with inter-rater reliability calculated using Cohen’s kappa — and your target kappa should be 0.7 or higher before you trust the labels.

    How the internal tax scenario classifier Forced the Discipline

    When I built the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours and achieved a 31% reduction in purchase order error rate — the evaluation problem was not optional. Tax classification is high-stakes and adversarially complex: the same economic transaction can have materially different tax treatments depending on jurisdiction, contract structure, and a half-dozen other variables. Getting the model wrong didn’t produce a slightly inconvenient user experience; it produced incorrect tax filings with audit exposure attached. We built an annotation corpus of 8,000+ labeled tax scenarios before we ran a single evaluation pass. The annotation workflow ran in three phases: a senior tax specialist assigned the ground-truth label for each scenario, a second specialist reviewed independently, and disagreements went to a structured resolution process where both annotators discussed the case against a written decision criteria document before a label was assigned. The kappa score in the first annotation round was 0.61 — below the 0.7 threshold — which told us the labeling criteria were ambiguous in about 20% of the cases. We spent two weeks tightening the criteria document and re-annotating the contested examples. The second pass came in at 0.74. That kappa score meant something: when the model scored 88% accuracy against that dataset, we had high confidence the 88% was real and not an artifact of easy examples and loose labels.

    The Distribution Trap

    The most common mistake teams make after building a golden dataset is constructing one that is too easy — heavy on the clear-cut cases, light on the ambiguous ones — and then celebrating when the model hits 94% accuracy. That number is meaningless if your production inputs include the 6% of scenarios where the model catastrophically fails, because that 6% might represent the highest-stakes decisions in the entire workflow. The adversarial examples you add in week three of evaluation should not be random stress tests — they should be the actual failure modes you observed when you let the model run on unlabeled production data in shadow mode. Shadow mode evaluation — running the model in parallel with the existing process without using its outputs, just observing — is how you discover what your golden set is missing. The failure modes you find in shadow mode become your adversarial examples, which makes your golden set a living artifact that improves as you learn more about where the model breaks.

    What Transfers

    Clinical assessment methodology has required structured reliability measurement for decades — the DSM field trials that established diagnostic criteria used inter-rater kappa as a primary validity metric, and a diagnostic tool with kappa below 0.8 was considered insufficiently reliable for clinical use. That standard didn’t emerge because psychologists are unusually rigorous; it emerged because the cost of diagnostic error — misdiagnosis, missed treatment, wrong intervention — made reliability non-negotiable. LLM evaluation should be held to the same logic. If the model’s output touches a consequential decision, the annotation process that produces your ground truth needs enough rigor that two independent experts agree on the right answer at least 70% of the time after accounting for chance agreement. Below that threshold, you’re not measuring model performance — you’re measuring label noise. The investment in evaluation infrastructure is front-loaded and genuinely tedious, but it’s the only thing that gives you honest signal about whether your model is improving or whether you’re just iterating on vibes.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • How I Built and Run a 60-Worker Cloudflare Fleet on $0/Month

    I run 60 production Cloudflare Workers for $0 per month — the only thing I pay is roughly $12 a year for a domain name. That’s not a development environment or a hobby experiment; that’s a fleet of real applications handling authentication, SQL queries, OSINT monitoring, AI inference, and job-matching logic, all running at the edge with no servers to manage and no cloud bill showing up at the end of the month.

    What the Free Tier Actually Gives You

    Cloudflare’s free tier is generous in ways that most people don’t fully appreciate until they map their workloads against the actual limits. Each Worker gets 100,000 requests per day — not per account, per worker — before you hit any throttle. D1, Cloudflare’s SQLite-at-the-edge product, gives you 5GB of storage and 25 million reads per day at no cost. KV gives you 100,000 reads per day. R2 gives you 10GB of object storage, 1 million Class A operations, and 10 million Class B operations monthly. Workers AI gives you 10,000 neurons per day on the free plan — enough to run real inference tasks at modest volume. Stack those together and you have a full-stack compute and storage platform that would run $200 to $400 a month on AWS if you tried to replicate it with Lambda, RDS, ElastiCache, and S3.

    The Architecture That Makes It Work

    The design principle behind the fleet is stateless compute paired with durable storage — the Worker itself holds no state between requests, which means cold starts are measured in microseconds rather than seconds, and horizontal scaling is automatic. Workers handle request routing, business logic, authentication, and API composition. D1 handles anything that needs relational structure and SQL queries. KV handles read-heavy caching where eventual consistency is acceptable — configuration values, session tokens, rate-limit counters. R2 holds files, exports, and anything that would otherwise require a file system. Workers AI sits at the inference layer, running models like LLaMA 3 without me spinning up a GPU instance. The whole architecture collapses the traditional web stack into a single deploy surface with one CLI command.

    Three Workers That Show the Range

    FinRec is my personal finance PWA — it runs entirely on Workers and D1, with a React front end served from Workers’ static asset hosting and a SQL backend running on D1. Every transaction, category, and account record lives in a D1 database. I wrote about the full build process in detail in this post on building FinRec. The application handles multi-user data with row-level scoping in SQL, processes hundreds of transactions without latency issues, and costs nothing beyond the domain. Digital Sentinel is my OSINT monitoring system — it uses Cron Triggers to run scheduled jobs every 15 minutes, polling certificate transparency logs and threat intelligence feeds, then routing alerts to a Discord webhook when it finds something worth flagging. There’s no always-on server; the Cron Trigger fires the Worker on schedule, the Worker runs its checks, and then it’s gone until the next cycle. I covered the full architecture in my post on building the OSINT monitoring system. The third is daniel-job-search, the career intelligence dashboard running at jobs.groundedintelligences.com — it uses Workers AI to run job-matching inference against my criteria, scoring inbound postings and surfacing the highest-signal opportunities. The AI layer runs on LLaMA 3 via Workers AI, the results are stored in D1, and the front end is a PWA served from the same Worker.

    The Deployment Workflow

    Every worker in the fleet deploys from a wrangler.toml file that specifies the worker name, the account ID, any KV namespace bindings, D1 database bindings, and R2 bucket bindings. Wrangler CLI handles the rest — one command, wrangler deploy, pushes the compiled Worker to Cloudflare’s edge network and it’s live globally within about 15 seconds. There’s no container to build, no cluster to update, no load balancer to reconfigure. The development loop is tight: wrangler dev runs a local emulator that mirrors the production environment closely enough that production surprises are rare. When I need to rotate a secret, wrangler secret put writes it to Cloudflare’s secret store without it ever appearing in my codebase or environment files. Across 60 workers, this workflow stays manageable because each worker is a small, focused piece of logic — nothing in the fleet exceeds about 400 lines of TypeScript.

    What Transfers

    The architecture principles here are not Cloudflare-specific — they’re the underlying logic of serverless-first design applied to any provider. Stateless compute paired with durable storage is a pattern that works on AWS Lambda with DynamoDB, on Fastly Compute with Upstash, or on any platform that separates the execution layer from the persistence layer. The key insight is that infrastructure ownership is the most expensive part of running software — not in dollar terms, but in cognitive overhead, operational burden, and the time you spend keeping things running instead of building. When you design systems where you own the code and the platform owns the infrastructure, you get back the mental bandwidth to actually work on the problem. The $0/month bill is a side effect of that design choice, not the goal. The goal is a system that does real work, stays maintainable, and doesn’t require a 2 AM pager rotation to keep alive.

    Where the $0 fleet stops working: media

    The one workload this architecture will not absorb is media at volume. Cloudflare explicitly prohibits streaming video on Free, Pro and Business plans — you are expected to move to Cloudflare Stream or Enterprise — and once you are storing tens of gigabytes and serving them repeatedly, the free tier stops being the right answer regardless of how the compute is structured.

    I recently split a media pipeline along exactly this line: the control plane — D1 index, Workers AI tagging, the ranking API — stayed on Cloudflare, and the data plane — 41 GB of originals, rendered variants and video — moved to bunny.net. Storage runs about $0.01–$0.02/GB and bandwidth $0.005/GB in North America and Europe, with no free tier to age out of and no per-plan content restrictions to trip over. The interesting part is that the split is architecturally cleaner than the monolith was: metadata and logic in one place, bytes in another, and the boundary is a signed URL.

    The bunny.net link above is an affiliate link — I earn a commission if you sign up, at no additional cost to you. I pay for the service myself and would recommend it regardless.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Deploying AI in Regulated Environments: Lessons from 17+ Enterprise Projects

    Deploying AI in Regulated Environments: Lessons from 17+ Enterprise Projects

    I’ve spent 19 years at Lockheed Martin. For the last four of those, I’ve led a 17+ project portfolio of automation, AI/ML, and dashboarding solutions in Aeronautics’ Finance & Business Operations Digital Transformation organization. Here are the patterns I’ve learned about deploying AI in environments where getting it wrong has real consequences.

    The Regulated-Environment Tax

    In defense, you can’t ship a model and iterate based on user complaints. The compliance surface is enormous — DCMA, DFAS, EVMS, FAR/DFAR, SOX-adjacent controls. Every automation I build has to survive not just technical review but audit scrutiny. This means the evaluation methodology, the guardrails, and the audit-logging are as important as the model itself.

    When I deployed an enterprise RAG pipeline for financial data classification, half the engineering effort went into the transparency artifacts — making sure every classification decision was traceable, every confidence score was logged, and every edge case had a human-review fallback.

    The Production Withholds Story

    The project that best illustrates this is a production withholds automation for a major defense program. The challenge: leadership needed visibility into a nine-figure withhold balance, but the source data was paragraph-form free text — inconsistent formatting, typos, abbreviations, no structured fields. The solution was a 356-tool Alteryx workflow combining ML/NLP text mining with structured field extraction, feeding a Tableau executive dashboard.

    The result: 2,132 labor hours saved per year, low-six-figure losses prevented, and the dashboard is now embedded in the program’s normal business rhythm. But the real lesson was that the NLP component had to be evaluated specifically for extraction accuracy against ground truth — not just general text classification metrics. In regulated environments, the eval framework is the product.

    AI at Scale

    More recently, I co-led an AI-driven tax-classification initiative — a cross-business-area AI chatbot evaluated against 415,000 tax scenarios on an internal enterprise AI platform. The target is a 31% reduction in purchase-order error rates. What made this different from a typical chatbot project was the evaluation rigor: structured behavior tests, accuracy validation against ground-truth tax determinations, and benchmark datasets that stress-test edge cases.

    What Transfers

    The patterns that work in defense AI transfer directly to any domain where trust matters: financial services, healthcare, legal tech, and especially responsible AI governance. Build the eval framework first. Log everything. Design for auditability from day one. And never ship a model without a human-review fallback for the cases you know it will get wrong.

    I’m currently exploring senior data scientist and AI/ML engineering roles where this kind of production rigor is valued. Get in touch.