How to audit an LLM judge before you trust its verdicts
A practical answer grounded in a runnable, published harness rather than opinion. Links to Can you trust the model that grades your content? Measuring when an AI judge waves through broken work.
How to audit an LLM judge before you trust its verdicts. The short answer: you test for it, you do not trust it. I build and publish the harnesses that do this. This page covers the method and links to a runnable one you can point at your own system.
This is the method behind Can you trust the model that grades your content? Measuring when an AI judge waves through broken work, which you can read and run.
1. Make the failure observable.
Pick the specific way this can go wrong and build the smallest input that triggers it. If you cannot make it fail on purpose, you cannot prove it works. A worked example: Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer.
2. Measure against a baseline, not a vibe.
Compare to a neutral control so the number means something. A score with nothing to compare it to is theater. A worked example: Same answer, different grade: measuring when an AI grader can’t hold a verdict.
3. Check it a second, independent way.
Re-run with a different model family or a held-out set. Agreement across independent checks is the only verdict worth trusting. A worked example: Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer.
Every claim here maps to a public, reproducible harness, not a slide. If you want to run this against your own system, the harnesses are open and the method applies directly.