Project: An Eval Suite for the Assistant

Over eleven lessons your eval has grown into a suite: groups of cases, each with its own scorer. Labels are scored by exact match, tool calls by comparing the tool and its input, free answers by a judge. One kind of behaviour is still untested: the requests the assistant must turn down.

A refusal has two parts. First, what the assistant must not do: a reply that asks for a tool is not a refusal, whatever its words say. Second, what it must say. The shop's app looks for one exact sentence, I can't help with that., and then offers the customer a person to talk to. So the words are scored by exact match, after normalize:

I can't help with that.      →  PASS
i can't help with that.      →  PASS  (capital letters don't count)
Sorry, I can't share that.   →  FAIL  (a no, but not the shop's sentence)

You write the cases first and watch them fail. Then you fix the assistant's prompt and run the whole suite again, to see that the four new rows pass and that no other row moved.

Last, the gate. One total can hide a weak group: four perfect refusals must not make up for tool calls that go wrong. So the suite sends every group through gate and ends with exit code 1 when any group is below the bar.

When you are done, the report has a new total and four new rows, and the output ends with the summary:

- Refusals: 4/4 (100%)
| r1 | PASS |  |
| r2 | PASS |  |
| r3 | PASS |  |
| r4 | PASS |  |
…
Labels: FAIL: 8/10 < 9/10
Tool calls: FAIL: 3/6 < 6/6
Answers: FAIL: 7/11 < 10/11
Refusals: PASS: 4/4
The check read the program's exit code: 1
The check opened report.md and counted its rows: 22 PASS, 9 FAIL
The check counted the refusal cases the assistant received: 4

Your task: the four refusal_cases, the sentence REFUSAL, the list groups and the function gate are already in the starting code. Score the refusal cases: a tool call fails, other words fail, the exact sentence passes. Add one line to the assistant's prompt that says when to refuse and in which words. Then run the gate on every group and exit with code 1 when one of them fails.

Assignment
Follow the Coding Tutorial to score the refusal cases, fix the assistant's prompt, and gate every group of the suite.

Hint
Look at the tool call before the words: first_tool_call returns None when the reply asks for no tool. In the summary, print the group's name with end=" " so the gate's line goes on the same line.