Llm eval harness
A practical answer grounded in a runnable, published harness rather than opinion. Links to How to audit an LLM judge before you trust its verdicts.
LLM eval harness. I test whether the AI actually helps a student learn, not whether the model just sounds plausible. That’s the short version. My education-domain judgment comes first. Then I use runnable AI-quality harnesses to make that judgment measurable. This page walks through the method, and links to one you can point at your own system.
This is the method behind How to audit an LLM judge before you trust its verdicts, and you can read it and run it yourself.
1. Make the failure observable.
Pick the one way this can go wrong and build the smallest input that triggers it. If you can’t make it fail on purpose, you can’t prove it works. Can you trust the model that grades your content? Measuring when an AI judge waves through broken work walks through exactly this.
2. Measure against a baseline, not a vibe.
Compare to a neutral control so the number actually means something. A score with nothing to compare it to is theater. I worked through that exact test in Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer.
3. Check it a second, independent way.
Re-run it with a different model family or a held-out set. Agreement across independent checks is the only verdict worth trusting. Same answer, different grade: measuring when an AI grader can’t hold a verdict is the worked example.
The overlap is the point: deep classroom, curriculum, and assessment judgment made measurable with AI and eval tooling. Every claim on this page maps to a public, reproducible harness, not a slide. If you want this run against your own system, the method transfers directly.