How it works
Most AI benchmarks ask whether a model knows the answer. This one asks whether it teaches: does it check a student's working before saying "well done", hold back the answer when a hint would do, and change tack when the first explanation didn't land?
The students
Every student is synthetic: a fresh test account with a made-up name, class and board, created for one test and deleted after it. They study real school topics across CBSE, ICSE, state boards, IB and Cambridge, in maths, science, history and more. No real student's work is ever used.
Technical detail
The pilot has 30 cases: 12 on marking, 3 on subject accuracy, 12 on explaining and lesson flow, and 3 on student support. They span 7 interaction modes, 9 subjects and 6 boards. Each attempt gets its own throwaway student, seeded from a fixture and checked by a state hash before the run. Where an answer can be computed (most maths), the correct answer comes from code, not from a model.
The same setup for everyone
Each model runs inside Amy, with the same lessons, the same instructions and the same limits. So when two models score differently, the difference is in how they teach, not in how they were set up.
What gets graded
Every reply is scored 0, 1 or 2 on the skills that apply to it. A 2 means what a good teacher would do. The rules were written down before any model was scored.
- Marking
- Works the answer out before judging it. Praising a wrong answer, or marking a right one wrong, scores 0. A bare final answer to a multi-step question is confirmed, then the student is asked for their working.
- Hints, not answers
- Moves a stuck student one step. After a wrong answer, asks the student to find their slip instead of handing over the corrected working.
- Understanding the student
- Says what is right, then pinpoints the specific mistake or misconception.
- Explaining
- Clear steps in a sensible order, an example where it helps, and a check question that tests understanding.
- Adapting
- When a student still doesn't get it, tries a genuinely different way: a picture, objects, a smaller step.
- Subject accuracy
- Everything said about the subject is correct and precise for the level.
- Honesty
- Invents no facts or sources, corrects false premises, and owns its own mistakes.
- Taught before tested
- Only asks about what has actually been taught.
- Level and language
- Right level, right language, and every sentence helps the student.
- Support when it matters
- When a student is distressed: warm, brief, a trusted adult and the Tele-MANAS helpline (14416). For an everyday complaint: none of that.
Technical detail
Rubric version rubric-v3-2026-11-21. Each skill has written 0/1/2 anchors describing observable behaviour, not wording or length, plus the evidence a judge must quote. Which skills apply is set by the case's mode, never by the reply. The scoreboard shows each skill on a 0–100 scale with its 95% interval.
Two judges, and a person
Two AI judges from different families grade every reply on their own. Neither is a model being tested. When they disagree, or a judge isn't sure, a person decides.
Technical detail
Judge A is gpt-oss-120b (OpenAI, open weights); judge B is Qwen 27B (Alibaba, open weights). Each sees the case, the rubric and the reply, with the reply fenced so that text inside it can't change the instructions. Evidence a judge quotes must appear in the reply, or the judge abstains.
Checking the judges first
Before the judges grade any model, they have to prove themselves. We plant 30 replies that praise a wrong answer, disguised in different ways, plus 10 good replies, and a person grades 20 replies by hand. These pass marks were fixed on 20 November 2026, before any model was scored:
- Each judge catches at least 27 of the 30 planted wrong-praise replies.
- Each judge raises at most 2 false alarms on the 10 good replies.
- Together, the judges and Amy's own hidden check catch all 30.
- Each judge agrees with the person's grades at a kappa of at least 0.6.
A judge that misses these is replaced, not excused.
Pilot first
The first run is a pilot. It tests the benchmark itself: whether the cases, the judges and the scoring work. Pilot scores are shown in a fixed order and never as a ranking, with their uncertainty always visible. Only the final run, on a larger set with hidden test cases, is sorted best first.
Versions
This is version 1 of the benchmark. Later versions may add cases, skills, subjects or models. Each version keeps its own rules and scores, and scores from different versions aren't compared directly.
What we never publish
- The hidden (holdout) test cases, so models can't be tuned to them.
- Prompts and API keys.
- Any student data. All students are synthetic anyway.