Work
Runnable proof of AI work that holds up: workflow audits, agentic systems, and the LLM quality control behind them. Every case study links to code and a result you can act on.
AI Enablement & Workflow
A practical QC framework for AI-generated curriculum and knowledge work
Most AI pilots fail because nobody defined what good output means before they shipped it. Used means nothing if you never set the bar. So I use a six-dimension rubric to QC anything a model writes, whether it's a lesson, a policy draft, or a customer reply. Each dimension has a concrete failing example, a pass condition, and a rule for when a human has to look. The last dimension is the one most rubrics skip: reviewer confidence, the honest signal for whether a person actually checked or just clicked approve.
How I audit a team's workflow for AI readiness
Most AI pilots fail before a model is ever chosen, because nobody mapped where the work actually slows down. So I run a fixed audit. I trace one workflow end to end, find the single step where the queue backs up, decide which part a model should draft and which part a human keeps, and name the one metric that proves it worked without quality sliding. Here it is on a worked example: a twelve-person customer-operations team where the real bottleneck wasn't writing replies, it was deciding which reply to write.
From raw input to reviewed output: an AI workflow that checks itself before a human sees it
A model that writes a draft is the easy part. The hard part is the wrap-around: taking messy input, producing a draft, scoring that draft against a rubric, flagging what's risky, and routing it to the right human instead of dumping everything on one reviewer. So I built that pipeline. Five stages, fail-closed, and the design choice that matters is that the model never decides whether its own work is good enough to ship, the rubric and the routing rules do.
LLM QC
Three failures your LLM pipeline never logs: gates for truncation, reasoning tax, and latency tails
I aggregated a month of my own API telemetry, 88,000 calls, and found three failure classes that raise no exception anywhere. One model silently truncated 5.7% of its responses. Eight models spent 83 to 95 percent of their completion tokens on reasoning nobody reads. Providers serving comparable models ranged from 1.1s to 20s median latency, with one tail hitting 89 seconds. So I built three deterministic gates that run on telemetry alone, and tested them against a synthetic fleet with defects planted by construction. The gate is correct when it finds every planted defect and nothing else.
The agent-boundary harness: measuring whether an AI agent respects a data fence, and whether the fence holds
An agent with file access can leak in two ways. The guard code fails, or the model decides to cross. I built a two-layer instrument that measures both, scored by whether a sealed canary token ever reads out. A naive path guard passed 75% and leaked twice through known EscapeRoute-class bypasses. The hardened guard passed clean. Then I tempted real models with tasks where crossing the boundary is the easy path, and counted the crossings. Everything runs on a fictional workspace, so the whole machine is public.
Can you trust the model that grades your content? Measuring when an AI judge waves through broken work
Teams now use an LLM to QC the content another LLM wrote. The quiet risk is that the judge passes work that's actually broken, and it gets more lenient when it grades its own output. So I built a fully-automated instrument that measures exactly that. It breaks items in known ways, with no model anywhere near the labels, then asks four different vendors' models to grade them blind. A single judge waved through 5 percent of the items I broke on purpose, and the rate ran higher when a model judged its own work. Making four independent vendors agree before a pass drove the miss rate to zero.
Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer
When an AI writes explanatory feedback for a wrong answer, it tends to fail in three quiet ways. It teaches the right answer instead of diagnosing the error, it invents a concept not present in the original content, or it attaches to the wrong option entirely. A top-to-bottom read misses all three. So I built a measurement instrument that catches them, with a blind-solver at the center that turns 'is this feedback bad?' into 'can a reader exploit it?' Then I gave it a golden set and a graduation test, so the human running it can measure when it's safe to stop checking by hand.
Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?
What clients actually want to know is whether a model can do an expert's judgment work, and whether you can defend the answer. I built the harness that measures it: domain distillation, a reference examiner, a blind A/B against the model, and κ with confidence intervals. Then I ran it on a synthetic insurance-adjudication domain so the whole machine is public while the real one stays under NDA. Three frontier models hit examiner parity, a small local model wasn't close, and the interesting failures were all in the precedence rules.
Calibrating an eval-design gate against 130 candidates
Building a self-hardening gate to separate load-bearing eval designs from decorative ones, and the surprise that the gate out-discriminated the thing generating the candidates.
Stress-Testing
Red-teaming my own publish gate
This site publishes itself unattended, so its clean-room firewall is the only thing between a leak and the open internet. I had two other model families try to break it. They found 14 ways past it in one pass.
Instruction-hierarchy collapse: a 5-model, 28-scenario audit
Does an instruction hidden in the content a model reads override the task it was given? I ran 28 injection scenarios across five models on identical inputs. The frontier agents held. A popular self-hosted model collapsed one time in five, and the clever reframes are what broke it.
Systems
A voice assistant that runs offline by default
An always-listening voice OS where the whole loop, from wake to transcribe to reason to speak, runs locally. So sensitive conversations never have to leave the machine.
An operating system for my AI agents
A canonical router, a confidentiality firewall, and a cross-agent message bus, so every AI tool I use shares one context layer. No merging, no API keys.
Method
What Vestmark's CTO + Chief AI Officer actually needs — and the proof I've already shipped
A CTO + Chief AI Officer role at Vestmark maps cleanly onto evaluation work I've already published. This page links the role's core skills to runnable, public harnesses — starting with Can you trust the model that grades your content? Measuring when an AI judge waves through broken work.
Assessment authoring as engineering: build the items in code, prove them with a round-trip
Most people build test items by hand, in a web UI, one click at a time. That's the step where items get dropped silently, and a structural check that passes tells you nothing about whether you built the right test. So I treat authoring as software engineering. I build the items in code, drive the platform's own API, and prove the result with a round-trip check against the source, with the source bank as the oracle. Every rule in the method traces to a named principle: Boehm on verification versus validation, Weyuker on oracles, QuickCheck on property-based testing, Shingo on mistake-proofing, Bainbridge on automation, Saltzer and Schroeder on least privilege.
Structural simultaneity: designing cross-model evals that can't be faked
Why most multi-model comparisons measure nothing, and the one design constraint that makes a cross-model eval actually carry weight.
Author with one model, attack with another
A rule I follow on any code where a quiet failure is expensive: don't trust your own review, and don't trust a same-family review either. One model writes it, a different family tries to break it, empirically.