Cost and Speed in Evals

An eval is a program that calls a model again and again, and every call is paid for by the token. A real test set has hundreds of cases and runs on every change, so you measure what a run uses. Every reply tells you: reply.usage.input_tokens is what you sent, and reply.usage.output_tokens is what the model wrote.

Prices are given in dollars per million tokens, with one price for input and a higher one for output. Say a call used 2,000 input tokens at $3 per million and 500 output tokens at $15 per million:

(2000 × 3 + 500 × 15) / 1000000  =  $0.0135

An eval with a judge pays twice: for the assistant's work and for the judge that grades it. The number of calls matters too: it is the eval's speed. The prices in this lesson are example prices. Real ones change, so look them up before you rely on them.

After the report, your program must print what each part used, and how often a cheaper judge gives the same verdict:

assistant on claude-haiku-5-5: 34 calls, 2503 input + 200 output tokens
  example cost: $0.003503
judge on claude-sonnet-5-5: 8 calls, 879 input + 151 output tokens
  example cost: $0.004902
judge on claude-haiku-5-5: 8 calls, 879 input + 151 output tokens
  example cost: $0.001634
The judges agree on 8 of 8 answers

Your task: the eval from the last lesson is already there, with JUDGE_MODEL, PRICES and a function track(who, model, reply) that only counts calls. Make track count tokens too and call it after every model call. Print the calls, tokens and example cost of each entry, the cost rounded to 6 decimals. Then send every answer to a second, cheaper judge and print how many verdicts agree.

Assignment
Follow the Coding Tutorial to measure the tokens and example cost of one eval run, split by assistant and judge, and to compare the judge with a cheaper one.

Hint
The prices are per million tokens: add the two products first, then divide by 1000000. The assistant's calls are tracked with MODEL, the judge's calls with the model parameter of judge.