Start learning free

LESSON 75 OF 129 · AI ENGINEERING: EVALS

Score Tool Calls

Practice

The Corner Shop's assistant can now act on orders with two tools: lookup_order and cancel_order. For a message about an order, the answer that matters is not a text. It is which tool the model asks for and with what input.

So a tool case expects a dictionary with a "tool" and an "input", and your scorer turns the model's reply into the same shape before it compares. The eval never runs the tool. It only reads the request, so no test cancels an order.

A tool call can be wrong in three ways. Say you expect a lookup of order 9:

expected  {'tool': 'lookup_order', 'input': {'order_id': 9}}

got       None                                                →  no tool call
got       {'tool': 'cancel_order', 'input': {'order_id': 9}}  →  wrong tool
got       {'tool': 'lookup_order', 'input': {'order_id': 1}}  →  wrong input

Each kind needs a different repair, so the scorer names it. For the six tool cases, your program must print these lines after the score of the labels:

FAIL t3: wrong input, expected {'order_id': 6}, got {'order_id': 5}
FAIL t4: wrong tool, expected lookup_order, got cancel_order
FAIL t5: no tool call
Tool calls: 3/6 (50%)

Your task: finish first_tool_call so it returns the first tool request of a reply as a dictionary with "tool" and "input". Then run the six tool_cases, print one FAIL line that names the kind of each failure, count the cases that pass, and print the score with score_text.


Assignment
Follow the Coding Tutorial to score the assistant's tool calls and name each kind of failure.


Hint
Test in this order: no tool call, then the tool, then the input. Comparing inputs only makes sense once the tool is the right one. The inputs are dictionaries, so wrap them in str() before you glue them to text with +.


STEP 1 OF 11
    Ctrl + Enter to check