A Judge Model

The Corner Shop's assistant also answers questions in its own words. A good answer has a thousand right wordings, so == can't score it, and no normalizing helps. Telling whether two sentences say the same thing takes a reader.

For four hundred cases the reader is a model: a judge, a second model call whose only job is to grade the first one's answer. Its instructions are a rubric: what passes, and the shape of the verdict. Its message holds the question, the expected text and the answer, each between tags.

The judge replies with a JSON object, and json.loads turns that text into a dictionary. A small example with another verdict:

text = '{"pass": false, "reason": "It gives the wrong price."}'
verdict = json.loads(text)
verdict["pass"]     # False
verdict["reason"]   # "It gives the wrong price."

Your program must print each answer with the judge's verdict below it, then the score:

a1: Yes, you can! Send the mug back within 14 days of delivery.
  PASS: It says yes and gives the 14 days.
a2: Of course! Send the lamp back and we'll refund you.
  FAIL: It accepts a return after a month, but returns end after 14 days.
a3: I'm afraid not. We deliver only to the EU and the UK.
  PASS: It says no and names the EU and the UK.
a4: Delivery takes 3 to 5 days, and it's free this week!
  FAIL: It gives the 3 to 5 days, but it also promises free delivery.
Answers: 2/4 (50%)

Your task: the RUBRIC and the four answer_cases are already there. Ask the assistant each question and print its answer. Write judge(case, answer), which sends the question, the expected text and the answer to the model with the rubric and returns the parsed verdict. Then print PASS or FAIL with the reason for every case, and the score.

Assignment
Follow the Coding Tutorial to grade the assistant's free-text answers with a judge model.

Hint
The judge is a second call to client.messages.create, with system=RUBRIC and no tools. verdict["pass"] is already True or False, so the if needs no ==.