Catch a Regression

You change a prompt and the score goes up. Is the new prompt better? Not always. A regression is a case that passed before the change and fails after it. A total says how many cases pass. It does not say which.

Compare case by case: run the same test set with the old version and with the new one, and keep what each run got wrong in a dictionary: the id of each failed case and the wrong answer it got. Say a spelling checker is tested on five words:

old_fails = {"w1": "teh", "w2": "recieve"}   # 3/5 pass
new_fails = {"w5": "definately"}             # 4/5 pass

The total went up, and still w5 is a regression: it is in new_fails and not in old_fails. w1 and w2 are the other way round, so the new version fixed them. To test it, "w5" in new_fails gives True: in on a dictionary looks at its keys.

In this lesson the test set is the ten customer messages, and the two versions are the prompts "v1" and "v2". Same cases, same scoring, and only the prompt differs: that is what makes two runs comparable. Your program must print these lines first:

v1: 7/10 (70%)
v2: 8/10 (80%)
FIXED c1
REGRESSION c3: expected shipping, got refund
REGRESSION c4: expected product, got refund
FIXED c6
FIXED c9

v2 has the better total, and it broke two messages that used to be sorted right. A better total can hide a regression.

Your task: the label loop now lives in a function, run_labels(version), and the starting code has its first lines. Make it collect the failed cases in a dictionary and return it. Run it with "v1" and with "v2", print the two totals, then print a REGRESSION line for every case v2 broke and a FIXED line for every case it fixed.

Assignment
Follow the Coding Tutorial to run the test set with both prompts and print which cases the new prompt broke and which it fixed.

Hint
A run passed every case that is not in its fails, so the number it passed is len(cases) - len(v1_fails). A regression is in v2_fails and not in v1_fails; a fixed case is the other way round.