Applied AI · Agentic Workflows · LLM QC
I get AI working inside the teams that adopt it: the workflows worth automating, the agentic systems that run them, and the quality control that keeps the output trustworthy.
Most AI pilots stall in the same spot. The demo works, then nobody can say whether the output holds up or whether the workflow was worth automating at all. I close that gap end to end. I find the work worth handing to AI, build the system that does it, and QC the result until a team can put it in front of real users. The instinct for where things break before anyone notices comes from fifteen years measuring learning when the stakes were real.
Selected work
All work →A practical QC framework for AI-generated curriculum and knowledge work
Most AI pilots fail because nobody defined what good output means before they shipped it. Used means nothing if you never set the bar. So I use a six-dimension rubric to QC anything a model writes, whether it's a lesson, a policy draft, or a customer reply. Each dimension has a concrete failing example, a pass condition, and a rule for when a human has to look. The last dimension is the one most rubrics skip: reviewer confidence, the honest signal for whether a person actually checked or just clicked approve.
How I audit a team's workflow for AI readiness
Most AI pilots fail before a model is ever chosen, because nobody mapped where the work actually slows down. So I run a fixed audit. I trace one workflow end to end, find the single step where the queue backs up, decide which part a model should draft and which part a human keeps, and name the one metric that proves it worked without quality sliding. Here it is on a worked example: a twelve-person customer-operations team where the real bottleneck wasn't writing replies, it was deciding which reply to write.
From raw input to reviewed output: an AI workflow that checks itself before a human sees it
A model that writes a draft is the easy part. The hard part is the wrap-around: taking messy input, producing a draft, scoring that draft against a rubric, flagging what's risky, and routing it to the right human instead of dumping everything on one reviewer. So I built that pipeline. Five stages, fail-closed, and the design choice that matters is that the model never decides whether its own work is good enough to ship, the rubric and the routing rules do.
Can you trust the model that grades your content? Measuring when an AI judge waves through broken work
Teams now use an LLM to QC the content another LLM wrote. The quiet risk is that the judge passes work that's actually broken, and it gets more lenient when it grades its own output. So I built a fully-automated instrument that measures exactly that. It breaks items in known ways, with no model anywhere near the labels, then asks four different vendors' models to grade them blind. A single judge waved through 5 percent of the items I broke on purpose, and the rate ran higher when a model judged its own work. Making four independent vendors agree before a pass drove the miss rate to zero.
Changelog
What I shipped recently. This page regenerates every night from the lab.
- Red teaming llm applications guide
- Llm judge guide
- Llm as a judge guide
- Llm judge model guide
- What is model evaluation in machine learning guide
- Why use multiple AI models instead of sampling one model many times guide
- How to run AI content pipelines unattended overnight safely guide
- How to make AI QC verdicts auditable and replayable guide