Start learning free

LESSON 79 OF 129 · AI ENGINEERING: EVALS

Hard Cases on Purpose

Practice

Look at the cases so far: polite questions, in English, each with one clear answer. Real customers are not that tidy. An eval made only of easy cases keeps saying that all is well until the first difficult customer shows up. So you write hard cases on purpose: messages picked because they might go wrong.

Think of a calculator tested only with 2 + 2 and 3 * 4: it passes 2/2. Add 1 / 0 and an empty input, and the score may fall to 2/4. The calculator did not get worse. The tests got more honest, and they found the two problems before a user did.

The starting code has four hard cases in hard_cases: h1 tries a prompt injection, h2 asks about an order without giving its number, h3 is rude, and h4 is written in Spanish. Their "expected" says what a good answer does, for the judge to read. You fix them in the assistant's prompt, which is made of string pieces between parentheses. Python joins pieces that follow each other:

greeting = (
    "Hello. "
    "Welcome to the shop. "
)
# greeting is "Hello. Welcome to the shop. "

With the hard cases in the set, the report's line for answers falls from 2/4 (50%) to 2/8 (25%). After your four fixes, the report must show:

- Answers: 6/8 (75%)
…
| h1 | PASS | It refuses and gives no code. |
| h2 | PASS | It asks for the order number and promises nothing. |
| h3 | PASS | It says sorry and offers the return within 14 days. |
| h4 | PASS | It answers in Spanish: yes, in 3 to 5 days. |

Every other row stays as it was. That matters: each line you add to a prompt can break a case that worked, and the eval runs everything again each time, so you check instead of hoping.

Your task: add the four hard cases to answer_cases and run the eval to see them fail. Then add four rules to ASSISTANT, one at a time, and run the eval after each one: no discount codes, ask for a missing order number, stay polite, answer in the customer's language.


Assignment
Follow the Coding Tutorial to add the hard cases to the test set and fix the assistant's prompt until all four pass.


Hint
+ between two lists makes one list with the items of both. Each new prompt piece is a string on a line of its own, with no comma and no +. Keep the space before its closing quote, or the next piece will stick to it.


STEP 1 OF 5
    Ctrl + Enter to check