How an educator evaluates edtech AI differently from an engineer
A practical answer grounded in a runnable, published harness rather than opinion. Links to Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?.
I evaluate edtech AI differently than an engineer would. I test whether the AI helps students learn, not whether the model sounds plausible. I lead with education-domain judgment, then use runnable AI-quality harnesses to make that judgment measurable. This page walks through the method and links to one you can run against your own system.
This is the method behind Blind expert-parity: can a model adjudicate like a credentialed examiner, and can you prove it?. You can read it and run it yourself.
1. Make the failure observable.
I pick the specific way this can go wrong, then build the smallest input that triggers it. If I can’t make it fail on purpose, I can’t prove it works. A worked example: Can you trust the model that grades your content? Measuring when an AI judge waves through broken work.
2. Measure against a baseline, not a vibe.
I compare to a neutral control so the number means something. A score with nothing to compare it to is theater. A worked example: Measuring feedback integrity: a blind-solver that catches AI explanations leaking the answer.
3. Check it a second, independent way.
I re-run it with a different model family or a held-out set. Agreement across independent checks is the only verdict I trust. A worked example: Same answer, different grade: measuring when an AI grader can’t hold a verdict.
The point is the overlap. I bring deep classroom, curriculum, and assessment judgment, and I make it measurable with AI/eval tooling. Every claim here maps to a public, reproducible harness, not a slide. If you want this run against your own system, the method transfers directly.