# Joshua Durey (Josh Durey) - Applied AI Enablement > Applied AI enablement. I audit workflows for AI readiness, build agentic systems, and run the LLM quality control that gets teams to adoption they can trust. ## Entity & Hiring Summary - **Names:** Joshua Durey / Josh Durey (Founder, Human Creative) - **Primary Lane:** Applied AI Enablement Lead, AI Operations Strategist, Agentic Workflow Architect, Principal AI Learning Systems Lead - **Credentials:** M.A. Applied Linguistics (University of Birmingham); 15+ years leading learning & operations systems; governed 52,167 multi-agent sessions with 30+ Python hook gates; 31,500+ Udemy learners. - **Work Authorization & Availability:** US Citizen, English-only, 100% remote worldwide (US W-2 EOR via Deel/Remote, US LLC B2B contract, fractional leadership, or fixed-scope 2-week workflow sprints). - **Full Markdown Corpus for RAG & Agents:** [https://jdurey.github.io/llms-full.txt](https://jdurey.github.io/llms-full.txt) ## Live Agentic Endpoints & Interactive Proof Surfaces - [Hire / Work With Joshua Durey](https://jdurey.github.io/hire/): Full-time remote roles, fractional leadership, and fixed-scope 2-week workflow sprints (https://jdurey.github.io/hire.md). - [Live 3-Model Workflow & JD Diagnostic](https://jdurey.github.io/#diagnostic): Interactive multi-vendor workflow readiness & rubric diagnostic backed by Cloudflare Workers AI. - [Commission a Free Custom Day-1 Prototype](https://jdurey.github.io/#commission-prototype): Automated 4-stage pipeline builds a custom runnable proof at `https://jdurey.github.io/for//`. - [Live 3-Minute Interactive AI Video Screen](https://jdurey.github.io/interview/): Conversational technical & leadership screen escalating to https://cal.com/joshua-durey-wqgvgp/20-min-scope-call. - [B2B Corporate L&D SOP-to-LMS Compiler](https://jdurey.github.io/sop-to-lms/): Compress enterprise course production from 6 weeks to 24 hours with live SCORM 1.2 telemetry, 4-vendor blind-solver verification, and sample SCORM 1.2 download (https://jdurey.github.io/sop-to-lms/sample-incident-sop-scorm12.zip). - [Local-vs-Cloud Model Routing Tribunal](https://jdurey.github.io/m5-tribunal/): 72 SHA-256-verified runs comparing tiered open-weight vs frontier cloud models with TCO router calculator. - [Remote Streamable HTTP MCP Server (io.github.jdurey/human-creative)](https://josh-employer-funnel.josh-durey.workers.dev/mcp): Live MCP server (https://jdurey.github.io/.well-known/mcp.json | https://github.com/jdurey/human-creative-mcp) exposing workflow diagnostics, prototype commissioning, and SOP-to-SCORM verification tools. - [AgentCard (A2A Protocol)](https://jdurey.github.io/.well-known/agent.json): Machine-readable capabilities, endpoints, and catalog metadata (also mirrored at https://jdurey.github.io/.well-known/agent-card.json and https://jdurey.github.io/api/catalog.json). ## Commercial Implementation Playbooks (8 Guides) - [Hiring an Applied AI Enablement Lead: A 30-60-90 Day Playbook for Team Adoption](https://jdurey.github.io/guides/applied-ai-enablement-lead/) ([.md](https://jdurey.github.io/guides/applied-ai-enablement-lead.md)): What an Applied AI Enablement Lead actually ships in the first 90 days: workflow audits, self-checking agent pipelines, cross-family quality gates, and staff playbooks that turn shelfware seats into daily output. - [Enterprise AI Workflow Audit Checklist: Ranking Tasks by Payoff, Risk & Checkpoints](https://jdurey.github.io/guides/ai-workflow-audit-checklist/) ([.md](https://jdurey.github.io/guides/ai-workflow-audit-checklist.md)): A practical four-step checklist for auditing team operations before writing automation code: mapping real inputs, pricing silent errors, placing human checkpoints, and killing low-payoff pilots early. - [Self-Reviewing Agentic Workflows: Designing Pipelines Non-Technical Teams Trust](https://jdurey.github.io/guides/self-reviewing-agentic-workflows/) ([.md](https://jdurey.github.io/guides/self-reviewing-agentic-workflows.md)): How to replace fragile single-shot prompts with four-stage self-checking workflows: deterministic Python pre-gates, narrow drafting nodes, cross-family critics, and SHA-verified audit receipts. - [LLM-as-a-Judge Calibration: Stopping Self-Bias, Verdict Flips & Silent Truncations](https://jdurey.github.io/guides/llm-judge-calibration-and-qc/) ([.md](https://jdurey.github.io/guides/llm-judge-calibration-and-qc.md)): How to calibrate LLM-as-a-Judge panels against human expert baselines: binary decomposed rubrics, cross-vendor panels, golden-set agreement checks, and catching silent HTTP 200 truncations. - [SOP-to-SCORM AI Pipeline: Compiling Policy Docs into Verified LMS & QTI Modules](https://jdurey.github.io/guides/sop-to-scorm-ai-pipeline/) ([.md](https://jdurey.github.io/guides/sop-to-scorm-ai-pipeline.md)): How corporate L&D and compliance teams turn static SOPs and runbooks into ADL SCORM 1.2 packages, QTI 1.2 item banks, 300-DPI guides, and narrated 1080p lessons with 0% policy drift. - [6-Rung LLM Cost & Latency Routing: Cutting Enterprise Token Spend Without Losing Quality](https://jdurey.github.io/guides/llm-cost-and-latency-routing/) ([.md](https://jdurey.github.io/guides/llm-cost-and-latency-routing.md)): Why a tiny fraction of agent calls burn 80% of your API bill, and how a 6-rung policy router shifts routine checks to zero-token Python hooks and fast mid-tier models. - [AI Curriculum & QTI Assessment Quality Control: Blind-Solver Checks & Zero Answer Leaks](https://jdurey.github.io/guides/ai-curriculum-and-qti-assessment-qc/) ([.md](https://jdurey.github.io/guides/ai-curriculum-and-qti-assessment-qc.md)): Engineering-grade quality control for EdTech and enterprise assessment pipelines: catching ambiguous distractors, standards drift, hint answer leaks, and broken math rendering before students see them. - [Two-Week AI Workflow & QC Sprints: Moving Stalled Pilots into Daily Team Use](https://jdurey.github.io/guides/fixed-scope-ai-workflow-sprint/) ([.md](https://jdurey.github.io/guides/fixed-scope-ai-workflow-sprint.md)): How founders and operations leaders use a fixed-scope two-week sprint to audit one high-friction bottleneck, deploy a self-checking workflow with human checkpoints, and keep the runnable QC harness. ## Self-Serve B2B Enterprise Kits, Physical Amazon KDP Books & Training Catalog - [Corporate L&D Zero-Hallucination SOP-to-SCORM & Video Factory ($99 Instant Checkout via Whop, InStock)](https://whop.com/checkout/plan_y1ACgqNYGtul8/): Compile SOPs into blind-verified SCORM 1.2, QTI 1.2, 300-DPI PDFs & 1080p video; includes instant access to the 14-lecture (65-min) Video & Code-Lab Masterclass. - [Enterprise AI Eval, 6-Rung Cost Router & REA AgentEnv Pro Kit ($79 Instant Checkout via Whop, InStock)](https://whop.com/checkout/plan_lGsdWjQsePkvO/): Python harnesses for 4-vendor LLM judge evals, 6-rung cost routing & AgentEnv; includes instant access to the 14-lecture Production LLM Evals Masterclass & 40-question exam bank. - [AI for Corporate Training & Instructional Design: The UbD Glass-Box Playbook ($24.99 Live on Amazon, InStock Paperback, ISBN 9798142559005, ASIN B0HRQFWFPR)](https://www.amazon.com/dp/B0HRQFWFPR): Physical 8.5x11 print reference playbook for zero-hallucination SOP-to-SCORM, QTI 1.2, and video pipelines. - [Benchmark AP Precalculus Unit 3 Exam Prep Workbook ($16.99 Live on Amazon, InStock Paperback, ISBN 9798142561688, ASIN B0HRQJLPRZ)](https://www.amazon.com/dp/B0HRQJLPRZ): Companion 300-DPI physical curriculum workbook from Benchmark AP Prep (https://benchmark-ap.pages.dev). - [Udemy Instructor Hub (31,500+ Learners)](https://www.udemy.com/user/joshuadurey/): Instructor profile for Joshua Durey; new AI Corporate Training & Production LLM Evals courses are completing Udemy's standard 2-3 day instructor identity verification (instant access available today via Whop). - [Done-For-You B2B Async Pilot & Workflow Sprints ($2.5k-$10k Fixed-Scope)](https://cal.com/joshua-durey-wqgvgp/20-min-scope-call): Book a 20-minute scope call or request a free custom prototype at https://jdurey.github.io/#commission-prototype. ## Featured Greenfield Harnesses & Case Studies (55 Published) - [Model evaluation in machine learning](https://jdurey.github.io/work/model-evaluation-in-machine-learning/) ([.md](https://jdurey.github.io/work/model-evaluation-in-machine-learning.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. - [Llm judge metrics](https://jdurey.github.io/work/llm-judge-metrics/) ([.md](https://jdurey.github.io/work/llm-judge-metrics.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Llm as a judge framework](https://jdurey.github.io/work/llm-as-a-judge-framework/) ([.md](https://jdurey.github.io/work/llm-as-a-judge-framework.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Llm as a judge paper](https://jdurey.github.io/work/llm-as-a-judge-paper/) ([.md](https://jdurey.github.io/work/llm-as-a-judge-paper.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Llm eval harness](https://jdurey.github.io/work/llm-eval-harness/) ([.md](https://jdurey.github.io/work/llm-eval-harness.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to How to audit an LLM judge before you trust its verdicts. - [Red teaming llm applications](https://jdurey.github.io/work/red-teaming-llm-applications/) ([.md](https://jdurey.github.io/work/red-teaming-llm-applications.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Red-teaming my own publish gate. - [Llm judge](https://jdurey.github.io/work/llm-judge/) ([.md](https://jdurey.github.io/work/llm-judge.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Llm as a judge](https://jdurey.github.io/work/llm-as-a-judge/) ([.md](https://jdurey.github.io/work/llm-as-a-judge.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Llm judge model](https://jdurey.github.io/work/llm-judge-model/) ([.md](https://jdurey.github.io/work/llm-judge-model.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [What is model evaluation in machine learning](https://jdurey.github.io/work/what-is-model-evaluation-in-machine-learning/) ([.md](https://jdurey.github.io/work/what-is-model-evaluation-in-machine-learning.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. - [Why use multiple AI models instead of sampling one model many times](https://jdurey.github.io/work/multi-model-panels-vs-resampling/) ([.md](https://jdurey.github.io/work/multi-model-panels-vs-resampling.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Author with one model, attack with another. - [How to run AI content pipelines unattended overnight safely](https://jdurey.github.io/work/unattended-ai-pipeline-failure-budget/) ([.md](https://jdurey.github.io/work/unattended-ai-pipeline-failure-budget.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [How to make AI QC verdicts auditable and replayable](https://jdurey.github.io/work/replayable-ai-qc-verdicts/) ([.md](https://jdurey.github.io/work/replayable-ai-qc-verdicts.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Same answer, different grade: measuring when an AI grader can't hold a verdict. - [How to audit an AI pipeline when you cannot read its data](https://jdurey.github.io/work/audit-ai-pipeline-without-reading-data/) ([.md](https://jdurey.github.io/work/audit-ai-pipeline-without-reading-data.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [How to choose an LLM provider when many serve the same model](https://jdurey.github.io/work/choose-llm-provider-by-latency-not-price/) ([.md](https://jdurey.github.io/work/choose-llm-provider-by-latency-not-price.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [Why a few LLM calls carry most of your API bill](https://jdurey.github.io/work/llm-api-costs-concentrate-in-few-calls/) ([.md](https://jdurey.github.io/work/llm-api-costs-concentrate-in-few-calls.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [How much do reasoning tokens add to LLM API costs](https://jdurey.github.io/work/how-much-do-reasoning-tokens-cost/) ([.md](https://jdurey.github.io/work/how-much-do-reasoning-tokens-cost.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [Why does my LLM output get cut off and how do I detect it](https://jdurey.github.io/work/why-llm-output-gets-cut-off/) ([.md](https://jdurey.github.io/work/why-llm-output-gets-cut-off.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. - [What is llm judge](https://jdurey.github.io/work/what-is-llm-judge/) ([.md](https://jdurey.github.io/work/what-is-llm-judge.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails](https://jdurey.github.io/work/telemetry-gates/) ([.md](https://jdurey.github.io/work/telemetry-gates.md)): I aggregated a month of my own API telemetry, 88,000 calls, and found three failure classes that raise no exception anywhere. One model silently truncated 5.7% of its responses. Eight models spent 83 to 95 percent of their completion tokens on reasoning nobody reads. Providers serving comparable models ranged from 1.1s to 20s median latency, with one tail hitting 89 seconds. So I built three deterministic gates that run on telemetry alone, and tested them against a synthetic fleet with defects planted by construction. The gate is correct when it finds every planted defect and nothing else. - [How to evaluate llm output](https://jdurey.github.io/work/how-to-evaluate-llm-output/) ([.md](https://jdurey.github.io/work/how-to-evaluate-llm-output.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [The agent-boundary harness: measuring whether an AI agent respects a data fence, and whether the fence holds](https://jdurey.github.io/work/agent-boundary/) ([.md](https://jdurey.github.io/work/agent-boundary.md)): An agent with file access can leak in two ways. The guard code fails, or the model decides to cross. I built a two-layer instrument that measures both, scored by whether a sealed canary token ever reads out. A naive path guard passed 75% and leaked twice through known EscapeRoute-class bypasses. The hardened guard passed clean. Then I tempted real models with tasks where crossing the boundary is the easy path, and counted the crossings. Everything runs on a fictional workspace, so the whole machine is public. - [What is llm as a judge](https://jdurey.github.io/work/what-is-llm-as-a-judge/) ([.md](https://jdurey.github.io/work/what-is-llm-as-a-judge.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. - [How to red-team AI-built tools and content before they reach a classroom](https://jdurey.github.io/work/red-team-ai-built-tools-before-classroom-use/) ([.md](https://jdurey.github.io/work/red-team-ai-built-tools-before-classroom-use.md)): A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. --- # Full Playbook Reference & Buyer FAQ Corpus # Hiring an Applied AI Enablement Lead: A 30-60-90 Day Playbook for Team Adoption - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/applied-ai-enablement-lead/ - **Published:** 2026-11-15 - **Summary:** What an Applied AI Enablement Lead actually ships in the first 90 days: workflow audits, self-checking agent pipelines, cross-family quality gates, and staff playbooks that turn shelfware seats into daily output. Most companies buy AI licenses in a single afternoon and then spend six months wondering why output didn't move. People try a chat window twice, hit one confident wrong answer on a real customer ticket, and go right back to their old spreadsheet. An Applied AI Enablement Lead fixes that gap. The role sits between operations, engineering, and staff training. Slide-deck workshops alone rarely change daily habits, and hiring a research scientist to fine-tune weights won't fix a messy intake queue. You hire this seat to pick the workflows worth automating, build multi-step pipelines with clear human checkpoints, prove the outputs hold up, and coach non-technical staff until the new process sticks. ## Days 1 to 30: Audit the work and lock a failure baseline In the first month, I shadow how tickets, documents, and approvals move across desks. Teams usually ask to automate their hardest creative task first. That instinct is understandable, and it almost always stalls out. So we score recurring tasks by weekly volume, error cost, and how fast a person can check a finished draft. Before writing any agent loop, we pull 25 to 50 real historical examples, including the ugly edge cases that broke past attempts. Those real tickets form our locked test set, and we ship one narrow workflow into daily use before Day 30 so the team sees a working win. ## Days 31 to 60: Wire self-checking pipelines and cut token waste Single-prompt chat habits break as soon as policy rules stack up. In month two, we split fragile mega-prompts into small stages. Python code checks JSON fields, required links, and math rules first. Separate model families grade tone and policy rules before a human reviewer ever opens the queue. Across 52,167 multi-agent sessions I governed on contract for an AI-education company, code hooks and tiered routing did most of the heavy lifting. You don't need a frontier reasoning call to check whether a date is missing. Routing simple checks to code and lightweight models kept one 23,740-call run down to $17 total spend with zero false passes. ## Days 61 to 90: Train leads and hand over regression checks A workflow isn't finished while I'm the only person who can run or debug it. My training is an MA in Applied Linguistics paired with fifteen years leading learning systems, so month three focuses on plain playbooks, clear escalation rules, and live practice labs. Every team lead gets a short checklist showing when to trust the queue, when to override it, and how to log a bad output. When a vendor swaps a model checkpoint next quarter, the test suite runs overnight and flags drift before morning standup. ## What to test before you hire for this seat Skip generic brainteasers. Hand the candidate one messy process from your backlog and ask which step stays human, how they catch a model grading its own mistakes, and how they'll teach an ops manager to run it on Monday morning. You can test your open role description right now in the [live 3-model JD diagnostic](/#diagnostic) or [commission a free custom team prototype](/#commission-prototype). ## Frequently Asked Questions ### What does an Applied AI Enablement Lead do day to day? An Applied AI Enablement Lead audits team workflows for AI readiness, builds multi-step agentic pipelines with human checkpoints, sets up cross-family LLM quality gates, and trains non-technical staff to run those tools safely in daily production. ### How is Applied AI Enablement different from a traditional L&D trainer or ML engineer? Traditional trainers teach generic prompting workshops without changing broken workflows, while ML engineers focus on model infrastructure. Applied AI Enablement bridges both: shipping working automation harnesses with code-level quality checks and training the staff who own the outcome. ### How can hiring managers evaluate Joshua Durey for a remote AI Enablement or AI Ops role? Hiring teams can paste their job description into the live 3-model diagnostic at jdurey.github.io/#diagnostic, interview his 3-minute interactive AI video twin at jdurey.github.io/interview/, commission a free custom Day-1 prototype for their company, or book a 20-minute scope call. ## Runnable Harnesses & Proofs - [How I Run an AI Workflow Audit](https://jdurey.github.io/work/ai-workflow-audit/) - [Self-Reviewing Agentic Workflow Demo](https://jdurey.github.io/work/agentic-workflow-demo/) - [4-Vendor Judge-Trust Calibration](https://jdurey.github.io/work/judge-trust/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # Enterprise AI Workflow Audit Checklist: Ranking Tasks by Payoff, Risk & Checkpoints - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/ai-workflow-audit-checklist/ - **Published:** 2026-11-15 - **Summary:** A practical four-step checklist for auditing team operations before writing automation code: mapping real inputs, pricing silent errors, placing human checkpoints, and killing low-payoff pilots early. When an operations pilot stalls, the team almost always blames the model. Then you look at the workflow and realize nobody agreed on what a good output looked like before they wired the API. A real AI workflow audit takes two or three days. Six-week slide decks just delay the real test. I run four plain checks to separate the tasks that pay back right away from the ones that will burn senior review hours. ## Step 1: Pull actual desk inputs from last Tuesday Ask a team how a process works and they'll describe the clean version from their onboarding doc. Ask to see ten real tickets from Tuesday afternoon and you'll find missing fields, half-pasted screenshots, and three Slack overrides. Pull twenty real inputs before touching a prompt. When half the tickets lack a customer ID or policy version, a model will guess to fill the blank. Fix the intake form or add a Python pre-check first. ## Step 2: Score tasks on check speed and error cost I score every candidate task on two simple numbers: how long it takes a person to check the draft, and what breaks if a wrong detail slips out the door. High-volume tasks that take three hours to write and thirty seconds to verify against a source table pay off right away. Rare tasks that take twenty minutes to fact-check line by line save zero time once you count rework. Leave those manual and move on. ## Step 3: Build the human checkpoint as a decision packet Saying a human sits in the loop means very little if the reviewer stares at a 900-word AI draft after lunch, because reviewer fatigue sets in by item six and people start clicking approve on autopilot. Give the reviewer a compact decision packet instead of a wall of prose. Show the exact source quote, list the two policy checks that passed in code, and highlight the single gray-area clause that needs a human call. Reviewers stay sharp when you ask for one clear call. ## Step 4: Set a fail-closed rule before launch Every step needs a hard rule for what happens when a model cuts off mid-sentence, breaks JSON, or disagrees with a second checker. In my [88,000-call telemetry audit](/work/telemetry-gates/), 324 responses hit unflagged length limits while returning normal HTTP 200 codes. Without code gates checking `finish_reason` and schema fields, broken work looks identical to finished work. Want to score one of your own team bottlenecks right now? Paste it into the [live workflow diagnostic](/#diagnostic) on the home page. ## Frequently Asked Questions ### How long should an enterprise AI workflow audit take? A focused AI workflow audit for one department takes two to five days: pulling 20 to 50 real historical inputs, scoring tasks by verification speed and error cost, defining human decision packets, and running a small baseline harness. ### Why do human-in-the-loop AI workflows still ship mistakes? Reviewers rubber-stamp long AI drafts when everything looks fluent. Effective human checkpoints present a structured decision packet that highlights source evidence and isolates the exact judgment call required. ### What makes a workflow a bad fit for AI automation? Tasks that happen rarely, lack verifiable source ground truth, or take a senior staff member just as long to fact-check as to write from scratch rarely pay back automation effort. ## Runnable Harnesses & Proofs - [AI Workflow Audit Harness & Receipts](https://jdurey.github.io/work/ai-workflow-audit/) - [Telemetry Gates Across 88,000 Calls](https://jdurey.github.io/work/telemetry-gates/) - [Unattended Pipeline Failure Budgets](https://jdurey.github.io/work/unattended-ai-pipeline-failure-budget/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # Self-Reviewing Agentic Workflows: Designing Pipelines Non-Technical Teams Trust - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/self-reviewing-agentic-workflows/ - **Published:** 2026-11-15 - **Summary:** How to replace fragile single-shot prompts with four-stage self-checking workflows: deterministic Python pre-gates, narrow drafting nodes, cross-family critics, and SHA-verified audit receipts. Give one model a twelve-rule prompt and ask it to research, write, format, and self-grade in a single pass. It will follow nine rules, drop three, and tell you it nailed all twelve. Self-reviewing workflows fix that by separating the builder from the inspector, using four tight stages and a hard retry cap instead of five chat agents arguing in an open loop. ## Stage 1: Clean intake and pinned source IDs Python code checks the input file first. Before any model runs, the script strips extra noise and tags every source paragraph with a fixed ID, so later code checks can verify in one millisecond whether a cited paragraph actually exists. ## Stage 2: Narrow drafting steps Each step does one job. One node pulls dates and dollar amounts into JSON, while the next writes the customer note from that verified record. Narrow steps let you run fast, cheap models on routine work and save heavy models for hard calls. ## Stage 3: Code hooks first, second-family critic next Never pay an LLM to count words or check JSON keys. Fast Python hooks run first, and in my own pipelines 30+ hook gates catch structural bugs right away at zero token cost. Once code checks pass, a critic model from a separate vendor grades the draft against a yes/no checklist. Say Claude wrote the draft. Gemini or Llama checks it cold. When the critic spots a miss, it sends back the exact broken rule for one repair pass, and a second miss halts the item for human review. ## Stage 4: Replayable SHA-256 receipts Ops leaders trust what they can inspect. Every run logs the input hash, model version, hook results, critic scores, and final diff into a single `.jsonl` receipt, so when someone asks why an item shipped last Thursday, you open that file and replay every step in two seconds. You can read runnable code in my [self-reviewing workflow demo](/work/agentic-workflow-demo/) or [request a free custom 4-stage prototype](/#commission-prototype) built around your stack. ## Frequently Asked Questions ### Why shouldn't the same LLM review its own output? Models share blindspots with their own weights and training style. When a model grades its own draft, it routinely overlooks hallucinations and rubric misses that a separate model family catches right away. ### How many retry loops should a self-reviewing workflow allow? Cap automated repair loops at one or two retries with structured defect feedback. Beyond two retries, models often game the rubric or degrade good sections, so unresolved items should fail closed to a human queue. ### How do non-technical teams monitor multi-agent workflows? Plain status dashboards backed by JSONL audit receipts show pass rates, blocked items, token cost per item, and clean side-by-side diffs whenever human approval is needed. ## Runnable Harnesses & Proofs - [Self-Reviewing Agentic Workflow Demo](https://jdurey.github.io/work/agentic-workflow-demo/) - [Verdict Integrity Under Retry Loops](https://jdurey.github.io/work/verdict-integrity/) - [Agent Boundary & Hook Governance](https://jdurey.github.io/work/agent-boundary/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # LLM-as-a-Judge Calibration: Stopping Self-Bias, Verdict Flips & Silent Truncations - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/llm-judge-calibration-and-qc/ - **Published:** 2026-11-15 - **Summary:** How to calibrate LLM-as-a-Judge panels against human expert baselines: binary decomposed rubrics, cross-vendor panels, golden-set agreement checks, and catching silent HTTP 200 truncations. Teams love LLM-as-a-Judge because human grading is slow. So someone wires a 1-to-5 star prompt onto the same model that wrote the answer. The dashboard shows a 4.7 average on Monday, and customers still complain on Wednesday. An uncalibrated judge is worse than having no metric at all. It gives broken releases a green badge. Fixing judge trust comes down to four plain habits. ## Swap 1-to-5 star ratings for yes/no facts Ask a model to rate 'clarity' from 1 to 5 and it will pick 4 almost every time. Worse, swap the order of two bullets in the prompt and the score flips. Break vague ratings into yes/no facts instead. Did the reply cite the policy number? Was any refund tier invented outside the source table? Does the hint give away the answer? Yes/no checks match human experts far more often. ## Split the writer model from the judge panel Asking the same model five times only proves that it agrees with its own blindspots. In my [multi-model panel tests](/work/multi-model-panels-vs-resampling/) and [judge-trust benchmark](/work/judge-trust/), pairing separate model vendors cut false passes from 5% down to 0.0% on seeded bugs. When two vendors disagree, you know right where to look. ## Check API cutoffs before you grade text Across 88,000 API calls I audited across twelve models, 324 replies hit `finish_reason = length` or returned empty thinking blocks while reporting HTTP 200 OK. Check stop reasons in Python first so you never pay a judge to grade half a sentence. ## Lock a 50-item human golden set Before trusting any automated rubric, grade 50 real items yourself across clean passes, borderline calls, and real bugs. Track exact pass/fail match and false-pass rate on that folder. Every time you edit the rubric prompt, re-run those 50 items first to prove the change helped. You can pull the runnable Python judge and routing harnesses today in the [$79 Enterprise AI Eval & Router Kit](https://whop.com/checkout/plan_lGsdWjQsePkvO/). ## Frequently Asked Questions ### Why do 1-to-5 LLM judge scores fail in production? Subjective 1-to-5 scales cluster around 4 and shift when prompt formatting changes. Decomposing rubrics into specific binary pass/fail checks makes verdicts reproducible and auditable. ### How big should a golden calibration set be for an LLM judge? Starting with 30 to 50 real examples labeled by a domain expert, balanced across clear passes, hard edge cases, and realistic defects, is enough to catch high false-pass rates early. ### What is false-pass rate in LLM quality control? False-pass rate is the share of defective outputs that an automated judge mistakenly marks as acceptable. In customer-facing or instructional systems, minimizing false passes matters more than raw average score. ## Runnable Harnesses & Proofs - [Judge-Trust Cross-Family Benchmark](https://jdurey.github.io/work/judge-trust/) - [Building an LLM QC Rubric](https://jdurey.github.io/work/llm-qc-rubric/) - [Why LLM Output Gets Cut Off Silently](https://jdurey.github.io/work/why-llm-output-gets-cut-off/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # SOP-to-SCORM AI Pipeline: Compiling Policy Docs into Verified LMS & QTI Modules - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/sop-to-scorm-ai-pipeline/ - **Published:** 2026-11-15 - **Summary:** How corporate L&D and compliance teams turn static SOPs and runbooks into ADL SCORM 1.2 packages, QTI 1.2 item banks, 300-DPI guides, and narrated 1080p lessons with 0% policy drift. Corporate training teams sit on stacks of updated SOPs, security runbooks, and product docs. Turning one policy PDF into an LMS module, a scenario quiz, and a narrated lesson usually takes four to six weeks by hand. Often the software screen changes before the course even ships. An SOP-to-SCORM compiler treats course builds like a clean code pipeline. You feed in the approved policy doc and get verified training files out the other side in a single day. ## Step 1: Start from real mistakes on the job Copying policy text onto slides and adding five trivia questions at the end never changes how staff work. Using backward design, the pipeline pulls out the exact mistakes people make under time pressure first. Every scenario question and worked example maps back to a named section in your source SOP. ## Step 2: Run four blind solvers on every quiz item AI quizzes break in two common ways: either two choices are both right, or an explanation argues against your own policy. Before any question ships, four separate models take the quiz blind using only the source SOP and the question stem. All four blind solvers have to pick the same key. Split votes drop the item right away, and a fresh question takes its place. ## Step 3: Package clean SCORM 1.2, QTI 1.2 & 1080p video Once every question passes, Python builds the `imsmanifest.xml` file, wires the SCORM 1.2 score and completion hooks, formats QTI 1.2 quiz banks, lays out 300-DPI print guides, and renders 1080p narrated video with captions. Test the live SCORM 1.2 sandbox and download a sample `.zip` module at [/sop-to-lms/](/sop-to-lms/). You can also read the [print playbook on Amazon ($24.99)](https://www.amazon.com/dp/B0HRQFWFPR) or download the [$99 Self-Serve Compiler Kit & 65-Minute Video Masterclass](https://whop.com/checkout/plan_y1ACgqNYGtul8/) right away while our companion Udemy courses finish standard 2-3 day instructor identity review. ## Frequently Asked Questions ### Can AI compile SOP documents directly into SCORM 1.2 packages for corporate LMS platforms? Yes. By pairing structured extraction and blind-solver verification with deterministic XML and JavaScript packagers, an SOP-to-SCORM pipeline emits standards-compliant ADL SCORM 1.2 .zip modules ready for Cornerstone, Docebo, Workday Learning, Moodle, or Canvas. ### How do you prevent hallucinations in compliance and security training? Every claim and quiz rationale is pinned to an explicit source section in the approved SOP, and four independent blind-solver checks verify that only the keyed answer holds up under policy rules. ### Where can L&D teams try Joshua Durey's SOP-to-SCORM pipeline today? Teams can test the live SCORM 1.2 runtime demo and download a sample package at jdurey.github.io/sop-to-lms/, order the print playbook on Amazon (ASIN B0HRQFWFPR), or get instant source-kit and 65-minute video masterclass access via Whop. ## Runnable Harnesses & Proofs - [Interactive SOP-to-LMS Compiler & Sample SCORM Download](https://jdurey.github.io/sop-to-lms/) - [Assessment Authoring as Software Engineering](https://jdurey.github.io/work/assessment-authoring-as-engineering/) - [Catching Feedback That Leaks the Answer](https://jdurey.github.io/work/catch-ai-feedback-that-reveals-the-answer/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # 6-Rung LLM Cost & Latency Routing: Cutting Enterprise Token Spend Without Losing Quality - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/llm-cost-and-latency-routing/ - **Published:** 2026-11-15 - **Summary:** Why a tiny fraction of agent calls burn 80% of your API bill, and how a 6-rung policy router shifts routine checks to zero-token Python hooks and fast mid-tier models. When finance flags a runaway AI bill, teams usually try swapping the whole app onto a tiny model. Quality tanks the same week, product rolls the change back, and everyone assumes high token bills are the cost of doing business. In real logs, token spend is lopsided. A few giant context loops and unbridled thinking calls burn 80% of the budget, while most traffic just parses fields, routes tags, or checks formatting. ## How the 6-rung router splits the work Instead of picking one model for the whole product, send each step to the lowest rung that clears your golden test set: 1. **Rung 0 (Zero-token Python hooks):** Regex, AST, JSON schema, URL, word-count, and banned-phrase gates. Instant and free. 2. **Rung 1 (Exact & prompt cache):** Reuse static policy tables and rubric instructions. 3. **Rung 2 (Fast small models):** Field parsing, tag routing, and table cleanup. 4. **Rung 3 (Mid-tier workhorse models):** First-pass drafting and structured summaries. 5. **Rung 4 (Two-vendor critic pair):** Separate mid-tier models grading yes/no rubric checks. 6. **Rung 5 (Frontier reasoning escalation):** Saved strictly for hard synthesis or tie-breaking when Rung 4 critics split. ## What happens to the bill in production On a 23,740-call run through my open-source routing harness, pushing structural checks to Rung 0 and saving frontier calls for real tie-breakers brought total API spend down to $17 while holding a 0.0% false-pass rate. Compare live routing cost curves on the [Model Cost & Latency Tribunal](/m5-tribunal/) or grab the runnable router source inside the [$79 Enterprise AI Eval & Router Vault](https://whop.com/checkout/plan_lGsdWjQsePkvO/). ## Frequently Asked Questions ### How do you cut enterprise LLM API costs without hurting accuracy? Move structural checks into zero-token Python hooks first, route routine extraction to fast low-cost models, and reserve expensive frontier reasoning models strictly for hard synthesis or split-verdict tie-breakers. ### Why do reasoning models blow up production budgets unexpectedly? Reasoning models emit thousands of hidden thinking tokens even on simple formatting or yes/no checks unless max-token caps and tiered routing rules restrict where reasoning mode is invoked. ### How do you verify a cheaper model rung is safe to deploy? Replay your locked 50-item golden test set across both the frontier baseline and the candidate rung, checking that false-pass rate stays flat and p95 latency improves before switching live traffic. ## Runnable Harnesses & Proofs - [Why LLM API Costs Concentrate in Few Calls](https://jdurey.github.io/work/llm-api-costs-concentrate-in-few-calls/) - [Choosing Providers by Tail Latency](https://jdurey.github.io/work/choose-llm-provider-by-latency-not-price/) - [How Much Do Reasoning Tokens Really Cost?](https://jdurey.github.io/work/how-much-do-reasoning-tokens-cost/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # AI Curriculum & QTI Assessment Quality Control: Blind-Solver Checks & Zero Answer Leaks - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/ai-curriculum-and-qti-assessment-qc/ - **Published:** 2026-11-15 - **Summary:** Engineering-grade quality control for EdTech and enterprise assessment pipelines: catching ambiguous distractors, standards drift, hint answer leaks, and broken math rendering before students see them. Shipping a weak blog sentence is annoying. Shipping a test question with two right answers or a tutor hint that gives away Choice B wrecks learner trust in thirty seconds. I spent fifteen years in classrooms and curriculum leadership before building automated instructional pipelines. Later, I governed 52,000+ multi-agent runs on contract for an AI-education company. Fluent prose hides bad teaching unless you test every question like production software. ## Bug 1: Two defensible keys or throwaway wrong choices Let one model write a multiple-choice item and its own rationale, and it will defend a broken key every time. Blind-solver checks stop that cold. Strip out the key, hand the stem and choices to three separate model vendors, and require all three to pick the exact same answer from scratch. Good wrong choices aren't random jokes either. Each distractor needs to map to a real mistake learners make on that skill. ## Bug 2: Hints that give away the answer Practice tutors often show a hint after a wrong first try. Because models aim to please, their hints often repeat the exact words sitting inside Choice C. Run a simple leak test to catch it: show a separate model *only* the hint and the four choices, hiding the question stem. Whenever that blind guesser still nails the right letter, the hint leaked too much. ## Bug 3: Silent drift away from the target standard Ask a model for a tenth-grade question on a specific skill and it will often test side trivia or write at a graduate reading level. Python readability checks, stimulus rules, and binary skill gates block off-target items before human editors waste hours fixing them. If your team is scaling curriculum or QTI item banks, test the [live SCORM & QTI verification demo](/sop-to-lms/) or [book a 20-minute scope call](/hire/). ## Frequently Asked Questions ### What is blind-solver verification for AI-generated assessments? Blind-solver verification strips the intended answer key from a generated question and asks independent models to solve it from scratch. If solvers split across two choices or fail to match the key, the item is flagged as ambiguous and blocked. ### How do you stop AI tutoring hints from giving away the answer? Run an answer-leak test where a separate checker sees only the hint and the answer options without the question stem. If the checker can identify the right option from the hint alone, the hint reveals too much and gets rewritten. ### Can QTI XML and LMS packaging be validated automatically in CI? Yes. Deterministic schema validators check QTI XML structure, unique identifiers, math notation balance, and SCORM manifest bindings before any human editorial review begins. ## Runnable Harnesses & Proofs - [Do AI Assessment Items Measure What They Claim?](https://jdurey.github.io/work/do-ai-assessment-items-measure-what-they-claim/) - [QC AI-Generated Curriculum to Standards](https://jdurey.github.io/work/qc-ai-generated-curriculum-to-standards/) - [Catching Feedback That Reveals the Answer](https://jdurey.github.io/work/catch-ai-feedback-that-reveals-the-answer/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call # Two-Week AI Workflow & QC Sprints: Moving Stalled Pilots into Daily Team Use - **Author:** Joshua Durey (Josh Durey) - Applied AI Enablement Lead (https://jdurey.github.io/) - **Canonical URL:** https://jdurey.github.io/guides/fixed-scope-ai-workflow-sprint/ - **Published:** 2026-11-15 - **Summary:** How founders and operations leaders use a fixed-scope two-week sprint to audit one high-friction bottleneck, deploy a self-checking workflow with human checkpoints, and keep the runnable QC harness. Not every company needs a full-time AI hire on Day 1. And nobody wants an open-ended hourly retainer that hands over a slide deck in week six. Often you have one painful bottleneck in support ops, compliance training, or content QA, and you want it working cleanly in two weeks. So alongside full-time remote and fractional leadership roles, I run fixed-scope two-week workflow sprints. ## Before we even talk: See a working Day-1 prototype You don't have to guess how I build. Drop your bottleneck or job link into the [prototype form](/#commission-prototype). My automated pipeline runs a custom four-stage workflow and three-vendor check for your company, then publishes it at `jdurey.github.io/for//` for free. ## Week 1: Real desk inputs, golden set, and stage wiring In days one through five, we pull 25 to 50 real past tickets or docs from your team. We lock plain yes/no pass rules with your domain lead and wire the multi-step pipeline inside your own cloud accounts. Python hooks check structure before any model runs, so bad inputs fail fast and cheap. ## Week 2: Cross-vendor judge gate, staff playbook, and handoff In days six through ten, we tune a two-vendor quality gate against your golden set, build the human review card for gray-area cases, and run a live handoff lab with your staff. You keep 100% of the clean Python code, prompts, test cases, and `.jsonl` logs. Zero lock-in. Want to scope a bottleneck or talk through an open remote role? [Grab 20 minutes on Cal.com](https://cal.com/joshua-durey-wqgvgp/20-min-scope-call) or [interview my 3-minute AI video twin](/interview/) first. ## Frequently Asked Questions ### What deliverables come out of a two-week AI workflow sprint? You receive a ranked workflow audit, a working multi-stage pipeline with human checkpoints running in your stack, a calibrated cross-family QC harness on 25 to 50 real test cases, and a plain-language operating playbook for your team. ### How is pricing structured for a workflow sprint? Sprints are scoped and priced as a flat fixed fee ($2.5k to $10k depending on integration depth) agreed on a 20-minute scope call, with zero open-ended hourly billing. ### Can we test how the workflow architecture applies to our company before booking a call? Yes. Submit your workflow or open role at jdurey.github.io/#commission-prototype to receive a free custom runnable prototype built for your organization. ## Runnable Harnesses & Proofs - [Work With Josh: Full-Time, Fractional, or Sprint](https://jdurey.github.io/hire/) - [AI Workflow Audit Method](https://jdurey.github.io/work/ai-workflow-audit/) - [Replayable AI QC Verdicts](https://jdurey.github.io/work/replayable-ai-qc-verdicts/) ## Next Steps for Hiring Teams & Founders - Run a live 6-second JD or workflow diagnostic: https://jdurey.github.io/#diagnostic - Commission a free custom Day-1 prototype for your org: https://jdurey.github.io/#commission-prototype - Interview the 3-minute AI video twin: https://jdurey.github.io/interview/ - Book a 20-minute scope call: https://cal.com/joshua-durey-wqgvgp/20-min-scope-call --- # Complete Case Study & Open-Source Harness Index (55 Studies) ### Model evaluation in machine learning - URL: https://jdurey.github.io/work/model-evaluation-in-machine-learning/ | Markdown: https://jdurey.github.io/work/model-evaluation-in-machine-learning.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. ### Llm judge metrics - URL: https://jdurey.github.io/work/llm-judge-metrics/ | Markdown: https://jdurey.github.io/work/llm-judge-metrics.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Llm as a judge framework - URL: https://jdurey.github.io/work/llm-as-a-judge-framework/ | Markdown: https://jdurey.github.io/work/llm-as-a-judge-framework.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Llm as a judge paper - URL: https://jdurey.github.io/work/llm-as-a-judge-paper/ | Markdown: https://jdurey.github.io/work/llm-as-a-judge-paper.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Llm eval harness - URL: https://jdurey.github.io/work/llm-eval-harness/ | Markdown: https://jdurey.github.io/work/llm-eval-harness.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to How to audit an LLM judge before you trust its verdicts. ### Red teaming llm applications - URL: https://jdurey.github.io/work/red-teaming-llm-applications/ | Markdown: https://jdurey.github.io/work/red-teaming-llm-applications.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Red-teaming my own publish gate. ### Llm judge - URL: https://jdurey.github.io/work/llm-judge/ | Markdown: https://jdurey.github.io/work/llm-judge.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Llm as a judge - URL: https://jdurey.github.io/work/llm-as-a-judge/ | Markdown: https://jdurey.github.io/work/llm-as-a-judge.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Llm judge model - URL: https://jdurey.github.io/work/llm-judge-model/ | Markdown: https://jdurey.github.io/work/llm-judge-model.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### What is model evaluation in machine learning - URL: https://jdurey.github.io/work/what-is-model-evaluation-in-machine-learning/ | Markdown: https://jdurey.github.io/work/what-is-model-evaluation-in-machine-learning.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. ### Why use multiple AI models instead of sampling one model many times - URL: https://jdurey.github.io/work/multi-model-panels-vs-resampling/ | Markdown: https://jdurey.github.io/work/multi-model-panels-vs-resampling.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Author with one model, attack with another. ### How to run AI content pipelines unattended overnight safely - URL: https://jdurey.github.io/work/unattended-ai-pipeline-failure-budget/ | Markdown: https://jdurey.github.io/work/unattended-ai-pipeline-failure-budget.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### How to make AI QC verdicts auditable and replayable - URL: https://jdurey.github.io/work/replayable-ai-qc-verdicts/ | Markdown: https://jdurey.github.io/work/replayable-ai-qc-verdicts.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Same answer, different grade: measuring when an AI grader can't hold a verdict. ### How to audit an AI pipeline when you cannot read its data - URL: https://jdurey.github.io/work/audit-ai-pipeline-without-reading-data/ | Markdown: https://jdurey.github.io/work/audit-ai-pipeline-without-reading-data.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### How to choose an LLM provider when many serve the same model - URL: https://jdurey.github.io/work/choose-llm-provider-by-latency-not-price/ | Markdown: https://jdurey.github.io/work/choose-llm-provider-by-latency-not-price.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### Why a few LLM calls carry most of your API bill - URL: https://jdurey.github.io/work/llm-api-costs-concentrate-in-few-calls/ | Markdown: https://jdurey.github.io/work/llm-api-costs-concentrate-in-few-calls.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### How much do reasoning tokens add to LLM API costs - URL: https://jdurey.github.io/work/how-much-do-reasoning-tokens-cost/ | Markdown: https://jdurey.github.io/work/how-much-do-reasoning-tokens-cost.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### Why does my LLM output get cut off and how do I detect it - URL: https://jdurey.github.io/work/why-llm-output-gets-cut-off/ | Markdown: https://jdurey.github.io/work/why-llm-output-gets-cut-off.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails. ### What is llm judge - URL: https://jdurey.github.io/work/what-is-llm-judge/ | Markdown: https://jdurey.github.io/work/what-is-llm-judge.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails - URL: https://jdurey.github.io/work/telemetry-gates/ | Markdown: https://jdurey.github.io/work/telemetry-gates.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/telemetry-gates - Summary: I aggregated a month of my own API telemetry, 88,000 calls, and found three failure classes that raise no exception anywhere. One model silently truncated 5.7% of its responses. Eight models spent 83 to 95 percent of their completion tokens on reasoning nobody reads. Providers serving comparable models ranged from 1.1s to 20s median latency, with one tail hitting 89 seconds. So I built three deterministic gates that run on telemetry alone, and tested them against a synthetic fleet with defects planted by construction. The gate is correct when it finds every planted defect and nothing else. ### How to evaluate llm output - URL: https://jdurey.github.io/work/how-to-evaluate-llm-output/ | Markdown: https://jdurey.github.io/work/how-to-evaluate-llm-output.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### The agent-boundary harness: measuring whether an AI agent respects a data fence, and whether the fence holds - URL: https://jdurey.github.io/work/agent-boundary/ | Markdown: https://jdurey.github.io/work/agent-boundary.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/agent-boundary - Summary: An agent with file access can leak in two ways. The guard code fails, or the model decides to cross. I built a two-layer instrument that measures both, scored by whether a sealed canary token ever reads out. A naive path guard passed 75% and leaked twice through known EscapeRoute-class bypasses. The hardened guard passed clean. Then I tempted real models with tasks where crossing the boundary is the easy path, and counted the crossings. Everything runs on a fictional workspace, so the whole machine is public. ### What is llm as a judge - URL: https://jdurey.github.io/work/what-is-llm-as-a-judge/ | Markdown: https://jdurey.github.io/work/what-is-llm-as-a-judge.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### How to red-team AI-built tools and content before they reach a classroom - URL: https://jdurey.github.io/work/red-team-ai-built-tools-before-classroom-use/ | Markdown: https://jdurey.github.io/work/red-team-ai-built-tools-before-classroom-use.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Why an AI grader checking its own output can't be trusted, and what to do - URL: https://jdurey.github.io/work/why-an-ai-grader-checking-itself-fails/ | Markdown: https://jdurey.github.io/work/why-an-ai-grader-checking-itself-fails.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### What to check before trusting AI grading in a classroom - URL: https://jdurey.github.io/work/what-to-check-before-trusting-ai-grading-in-class/ | Markdown: https://jdurey.github.io/work/what-to-check-before-trusting-ai-grading-in-class.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### How an educator evaluates edtech AI differently from an engineer - URL: https://jdurey.github.io/work/how-an-educator-evaluates-edtech-ai/ | Markdown: https://jdurey.github.io/work/how-an-educator-evaluates-edtech-ai.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. ### How to tell if an AI tutor matches an expert teacher's judgment - URL: https://jdurey.github.io/work/does-ai-match-an-expert-teachers-judgment/ | Markdown: https://jdurey.github.io/work/does-ai-match-an-expert-teachers-judgment.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. ### How to QC multiple-choice question feedback at scale for schools - URL: https://jdurey.github.io/work/qc-multiple-choice-feedback-at-scale/ | Markdown: https://jdurey.github.io/work/qc-multiple-choice-feedback-at-scale.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### How to catch AI feedback that reveals the answer instead of teaching it - URL: https://jdurey.github.io/work/catch-ai-feedback-that-reveals-the-answer/ | Markdown: https://jdurey.github.io/work/catch-ai-feedback-that-reveals-the-answer.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### How to check whether AI-generated assessment items measure what they claim - URL: https://jdurey.github.io/work/do-ai-assessment-items-measure-what-they-claim/ | Markdown: https://jdurey.github.io/work/do-ai-assessment-items-measure-what-they-claim.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Calibrating an eval-design gate against 130 candidates. ### Why AI-generated lesson content drifts from the standard it claims to teach - URL: https://jdurey.github.io/work/ai-lesson-content-drifts-from-standard/ | Markdown: https://jdurey.github.io/work/ai-lesson-content-drifts-from-standard.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### How to evaluate an AI-education product before a school pilot - URL: https://jdurey.github.io/work/evaluate-edtech-ai-before-a-pilot/ | Markdown: https://jdurey.github.io/work/evaluate-edtech-ai-before-a-pilot.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### How to know if an AI grader is fair and consistent for students - URL: https://jdurey.github.io/work/is-your-ai-grader-fair-to-students/ | Markdown: https://jdurey.github.io/work/is-your-ai-grader-fair-to-students.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. ### How to check whether AI feedback on student work is pedagogically sound - URL: https://jdurey.github.io/work/is-ai-feedback-pedagogically-sound/ | Markdown: https://jdurey.github.io/work/is-ai-feedback-pedagogically-sound.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### What Vestmark's CTO + Chief AI Officer actually needs - and the proof I've already shipped - URL: https://jdurey.github.io/work/showcase-vestmark-2026-07-02/ | Markdown: https://jdurey.github.io/work/showcase-vestmark-2026-07-02.md - Summary: A CTO + Chief AI Officer role at Vestmark maps cleanly onto evaluation work I've already published. This page links the role's core skills to runnable, public harnesses - starting with Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Assessment authoring as engineering: build the items in code, prove them with a round-trip - URL: https://jdurey.github.io/work/assessment-authoring-as-engineering/ | Markdown: https://jdurey.github.io/work/assessment-authoring-as-engineering.md - Summary: Most people build test items by hand, in a web UI, one click at a time. That's the step where items get dropped silently, and a structural check that passes tells you nothing about whether you built the right test. So I treat authoring as software engineering. I build the items in code, drive the platform's own API, and prove the result with a round-trip check against the source, with the source bank as the oracle. Every rule in the method traces to a named principle: Boehm on verification versus validation, Weyuker on oracles, QuickCheck on property-based testing, Shingo on mistake-proofing, Bainbridge on automation, Saltzer and Schroeder on least privilege. ### How to QC AI-generated curriculum for alignment to standards (CCSS / NGSS / PYP) - URL: https://jdurey.github.io/work/qc-ai-generated-curriculum-to-standards/ | Markdown: https://jdurey.github.io/work/qc-ai-generated-curriculum-to-standards.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### A practical QC framework for AI-generated curriculum and knowledge work - URL: https://jdurey.github.io/work/llm-qc-rubric/ | Markdown: https://jdurey.github.io/work/llm-qc-rubric.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/judge-trust - Summary: Most AI pilots fail because nobody defined what good output means before they shipped it. Used means nothing if you never set the bar. So I use a six-dimension rubric to QC anything a model writes, whether it's a lesson, a policy draft, or a customer reply. Each dimension has a concrete failing example, a pass condition, and a rule for when a human has to look. The last dimension is the one most rubrics skip: reviewer confidence, the honest signal for whether a person actually checked or just clicked approve. ### How I audit a team's workflow for AI readiness - URL: https://jdurey.github.io/work/ai-workflow-audit/ | Markdown: https://jdurey.github.io/work/ai-workflow-audit.md - Summary: Most AI pilots fail before a model is ever chosen, because nobody mapped where the work actually slows down. So I run a fixed audit. I trace one workflow end to end, find the single step where the queue backs up, decide which part a model should draft and which part a human keeps, and name the one metric that proves it worked without quality sliding. Here it is on a worked example: a twelve-person customer-operations team where the real bottleneck wasn't writing replies, it was deciding which reply to write. ### From raw input to reviewed output: an AI workflow that checks itself before a human sees it - URL: https://jdurey.github.io/work/agentic-workflow-demo/ | Markdown: https://jdurey.github.io/work/agentic-workflow-demo.md - Summary: A model that writes a draft is the easy part. The hard part is the wrap-around: taking messy input, producing a draft, scoring that draft against a rubric, flagging what's risky, and routing it to the right human instead of dumping everything on one reviewer. So I built that pipeline. Five stages, fail-closed, and the design choice that matters is that the model never decides whether its own work is good enough to ship, the rubric and the routing rules do. ### How to tell if an AI tutor is teaching or just giving away the answer - URL: https://jdurey.github.io/work/does-this-ai-tutor-actually-teach/ | Markdown: https://jdurey.github.io/work/does-this-ai-tutor-actually-teach.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer. ### How to tell if your AI grader is passing bad work - URL: https://jdurey.github.io/work/is-your-ai-grader-passing-bad-work/ | Markdown: https://jdurey.github.io/work/is-your-ai-grader-passing-bad-work.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### How to audit an LLM judge before you trust its verdicts - URL: https://jdurey.github.io/work/how-to-audit-an-llm-judge/ | Markdown: https://jdurey.github.io/work/how-to-audit-an-llm-judge.md - Summary: A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work. ### Same answer, different grade: measuring when an AI grader can't hold a verdict - URL: https://jdurey.github.io/work/verdict-integrity/ | Markdown: https://jdurey.github.io/work/verdict-integrity.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/verdict-integrity - Summary: Most teams treat an AI grader like a function: same answer in, same grade out. I built a tiny instrument that checks, by submitting one fixed answer to a grader many times and counting how often the verdict flips. On a fictional item, a correct answer whose service was even on the accepted-examples list got graded pass 19 times and fail 11 on byte-identical input. The obvious-right and obvious-wrong controls stayed flat the whole time. An answer that named no service at all passed almost half the time. Single-shot QC can't see this, because the instability only shows up when you repeat the call. ### Can you trust the model that grades your content? Measuring when an AI judge waves through broken work - URL: https://jdurey.github.io/work/judge-trust/ | Markdown: https://jdurey.github.io/work/judge-trust.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/judge-trust - Summary: Teams now use an LLM to QC the content another LLM wrote. The quiet risk is that the judge passes work that's actually broken, and it gets more lenient when it grades its own output. So I built a fully-automated instrument that measures exactly that. It breaks items in known ways, with no model anywhere near the labels, then asks four different vendors' models to grade them blind. A single judge waved through 5 percent of the items I broke on purpose, and the rate ran higher when a model judged its own work. Making four independent vendors agree before a pass drove the miss rate to zero. ### Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer - URL: https://jdurey.github.io/work/feedback-integrity/ | Markdown: https://jdurey.github.io/work/feedback-integrity.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/feedback-integrity - Summary: When an AI writes explanatory feedback for a wrong answer, it tends to fail in three quiet ways. It teaches the right answer instead of diagnosing the error, it invents a concept not present in the original content, or it attaches to the wrong option entirely. A top-to-bottom read misses all three. So I built a measurement instrument that catches them, with a blind-solver at the center that turns 'is this feedback bad?' into 'can a reader exploit it?' Then I gave it a golden set and a graduation test, so the human running it can measure when it's safe to stop checking by hand. ### Structural simultaneity: designing cross-model evals that can't be faked - URL: https://jdurey.github.io/work/structural-simultaneity/ | Markdown: https://jdurey.github.io/work/structural-simultaneity.md - Summary: Why most multi-model comparisons measure nothing, and the one design constraint that makes a cross-model eval actually carry weight. ### Red-teaming my own publish gate - URL: https://jdurey.github.io/work/red-teaming-my-publish-gate/ | Markdown: https://jdurey.github.io/work/red-teaming-my-publish-gate.md | Code: https://github.com/jdurey/jdurey.github.io/blob/main/tests/firewall-scan.test.mjs - Summary: This site publishes itself unattended, so its clean-room firewall is the only thing between a leak and the open internet. I had two other model families try to break it. They found 14 ways past it in one pass. ### Instruction-hierarchy collapse: a 5-model, 28-scenario audit - URL: https://jdurey.github.io/work/instruction-hierarchy-audit/ | Markdown: https://jdurey.github.io/work/instruction-hierarchy-audit.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/instruction-hierarchy-audit - Summary: Does an instruction hidden in the content a model reads override the task it was given? I ran 28 injection scenarios across five models on identical inputs. The frontier agents held. A popular self-hosted model collapsed one time in five, and the clever reframes are what broke it. ### Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it? - URL: https://jdurey.github.io/work/expert-parity-harness/ | Markdown: https://jdurey.github.io/work/expert-parity-harness.md | Code: https://github.com/jdurey/jdurey.github.io/tree/main/harnesses/expert-parity - Summary: What clients actually want to know is whether a model can do an expert's judgment work, and whether you can defend the answer. I built the harness that measures it: domain distillation, a reference examiner, a blind A/B against the model, and κ with confidence intervals. Then I ran it on a synthetic insurance-adjudication domain so the whole machine is public while the real one stays under NDA. Three frontier models hit examiner parity, a small local model wasn't close, and the interesting failures were all in the precedence rules. ### Author with one model, attack with another - URL: https://jdurey.github.io/work/author-one-model-attack-another/ | Markdown: https://jdurey.github.io/work/author-one-model-attack-another.md | Code: https://github.com/jdurey/skills/tree/main/adversarial-harden - Summary: A rule I follow on any code where a quiet failure is expensive: don't trust your own review, and don't trust a same-family review either. One model writes it, a different family tries to break it, empirically. ### A voice assistant that runs offline by default - URL: https://jdurey.github.io/work/local-voice-assistant/ | Markdown: https://jdurey.github.io/work/local-voice-assistant.md - Summary: An always-listening voice OS where the whole loop, from wake to transcribe to reason to speak, runs locally. So sensitive conversations never have to leave the machine. ### Calibrating an eval-design gate against 130 candidates - URL: https://jdurey.github.io/work/golden-set-calibration/ | Markdown: https://jdurey.github.io/work/golden-set-calibration.md - Summary: Building a self-hardening gate to separate load-bearing eval designs from decorative ones, and the surprise that the gate out-discriminated the thing generating the candidates. ### An operating system for my AI agents - URL: https://jdurey.github.io/work/agent-context-federation/ | Markdown: https://jdurey.github.io/work/agent-context-federation.md - Summary: A canonical router, a confidentiality firewall, and a cross-agent message bus, so every AI tool I use shares one context layer. No merging, no API keys.