Test Cases from Real Conversations

The hard cases you wrote were guesses about what could go wrong. Your users don't guess. Every conversation a customer rated thumbs-down is a failure that really happened, and no better source of test cases exists.

The shop keeps its conversations in conversations.jsonl. The format is JSON Lines: one JSON object per line. You read the file one line at a time and parse each line with json.loads:

{"id": 301, "rating": "up", "question": "Do you ship to France?", "answer": "…", "review": ""}
{"id": 302, "rating": "down", "question": "Do you ship to Norway? I live in Oslo.", "answer": "…", "review": "No: the shop ships only to …"}

Real conversations carry personal data. A test set is a file that many people open, that stays in the project for years, and that is sent to the model on every run. So the text is cleaned before it becomes a case. re.sub(pattern, replacement, text) replaces every match of a pattern. Here it is on another pattern:

re.sub(r"\d+", "<number>", "Order 42 has 3 items")
# "Order <number> has <number> items"

Your program must turn the three thumbs-down conversations into cases, with every email address replaced by <email>, and run them with the others:

NEW log302: Do you ship to Norway? I live in Oslo.
NEW log304: I wrote to you from <email> about a chipped mug and got no answer. …
NEW log306: I ordered as <email> but I want my updates at <email>. Can you change it?
…
- Answers: 7/11 (64%)
…
| log302 | FAIL | It says yes, but Norway is not in the EU or the UK. |
| log304 | PASS | It says yes: 5 days is within the 14 days. |
| log306 | FAIL | It claims it changed the email address, and it has no way to do that. |

Your task: the eval from the last lesson is already there, with re imported and the pattern EMAIL. Write hide_emails, read the log, and make a case from each thumbs-down conversation: its id, its cleaned question as input and its review as expected. Print the new cases, save them to log_cases.json and add them to answer_cases.

Assignment
Follow the Coding Tutorial to turn the thumbs-down conversations of the log into test cases, with the emails replaced before saving, and to run them.

Hint
Clean the question at the moment the case is made, so no later line of the program holds an email. json.loads, with an s, parses one line; json.load on the whole file would fail.