Run the Tests

An agent that edits code needs a way to find out whether the code works. Reading the code is a guess. Running the tests is proof. In this lesson you give your agent a run_tests tool.

Python starts another program with subprocess.run and waits until it ends. Here it runs the command echo hello, collects what the program prints, and gives up after 5 seconds:

result = subprocess.run(["echo", "hello"], capture_output=True, text=True, timeout=5)
print(result.stdout)

Never run a command without a timeout: one test that loops forever would hang your agent forever. And a test report can be very long, while the part that matters is at the end: the verdict, OK or FAILED, and above it the failure. So the tool returns only the last 12 lines.

The project has one real bug, so one test fails. Your program must print the run, and the lesson's check then shows the exact text the model received:

Model: I'll run the tests.
Tool: run_tests {}
Tests: FAILED (failures=1)
Answer: One of the 6 tests fails: test_exactly_fifty_dollars_ships_free. …
Model calls: 2
Times python3 -m unittest ran: 1
Folder it ran in: workspace
Timeout in seconds: 10
The tool result the model saw (12 lines):
FAIL: test_exactly_fifty_dollars_ships_free (test_cart.CartTests…)
…
    self.assertEqual(shipping(5000), 0)
…
AssertionError: 500 != 0
…
Ran 6 tests in <time>

FAILED (failures=1)
The check fixes cart.py and asks run_tests again:
Tests: OK
…
OK

Your task: the description run_tests_tool is already there. Write the function run_tests: it runs python3 -m unittest inside workspace with a 10 second timeout, prints the report's last line after Tests:, and returns the last 12 lines. Then offer the tool to the model.

Assignment
Follow the Coding Tutorial to build the run_tests tool and offer it to the model.

Hint
unittest writes its report to stderr, so join done.stdout and done.stderr. In a list of lines, lines[-1] is the last line and lines[-12:] are the last 12.