A Test Set

In the last lesson you had to read every answer to know whether it was right. Write the right answers down once, and a program can do the reading. A test case is a dictionary with three keys: an "id" that names it, the "input" you send, and the answer you "expected". A list of cases is a test set.

A small example, for an assistant that names the language of a word:

case = {"id": "t1", "input": "Bonjour", "expected": "french"}
case["input"]     # "Bonjour": what you send to the model
case["expected"]  # "french": what should come back

Only the input goes to the model. What comes back is what you got. Expected and got are the two words of every test: what should come out, and what did.

A test set outlives the code around it, so it gets a file of its own, cases.json. This lesson's program saves ten cases there and runs the assistant on each one with prompt "v2". It must print two lines per case:

c1: My lamp arrived broken. I want my money back.
  expected: refund | got: refund
c2: Where is my package? I ordered ten days ago.
  expected: shipping | got: shipping
c3: How much does shipping to Canada cost?
  expected: shipping | got: refund
…
c10: Can you deliver to a parcel locker?
  expected: shipping | got: shipping

Your task: nine cases, the prompts and the function classify are already there. Add case "c1" at the top of the list, save the list to cases.json with json.dump and indent=2, then loop over the cases: run the assistant on each input and print the id, the input, the expected answer and what you got.

Assignment
Follow the Coding Tutorial to complete the test set, save it to cases.json and print expected next to got for every case.

Hint
A case is one dictionary on one line, with a comma after the closing brace. Read its values with case["id"], case["input"] and case["expected"]. The string " expected:" starts with two spaces.