Start learning free

LESSON 119 OF 129 · AI ENGINEERING: CAPSTONE

The Assistant End to End, 5: Shipping It in Python

Practice

The assistant passed its gate. To ship it, you put it behind a web address: the shop's website sends a question to POST /chat and gets the answer back as JSON. The function answer and everything it needs are already in the program.

A shipped app has to live through a busy model. A retry tries the same call again after a short wait; the SDK's client does that by itself, and client.with_options(max_retries=...) sets how many times. When a model fails on every try, a fallback asks another model instead.

A cache keeps answers that were already worked out, so the same question doesn't cost time and money twice. Here is the idea on another value, a slow lookup of delivery prices:

prices = {}

def price_for(country):
    if country in prices:
        return prices[country]
    prices[country] = slow_lookup(country)
    return prices[country]

Cache only what stays true. The shipping policy is the same tomorrow; an order's status is not. So the assistant stores only the answers that used no tools.

These are the requests the tests send, and what your app must answer:

POST /chat {"question": "Do you ship to Canada?"}  →  200  {"answer": "Yes, we ship to Canada. …", "sources": ["shipping.md"], "tools": []}
POST /chat {"question": "do you ship to canada"}  →  200  the same answer, from the cache
POST /chat {"question": "Can you look up order 5 for me?"}  →  200  {"answer": "… It is paid and has not shipped yet.", "sources": [], "tools": ["lookup_order"]}
POST /check/ship-order-5  →  200  {"order": 5, "status": "shipped"}
POST /chat {"question": "Can you look up order 5 for me?"}  →  200  {"answer": "Order 5 was placed on 2026-05-07 and it has shipped.", …}
POST /chat {"question": "Can I get my money back?"}  →  200  {"answer": "Yes. You can return any item within 14 days of delivery, …", "sources": ["returns.md"], "tools": []}
GET /check/model  →  200  13 model calls: 12 to claude-haiku-5-5, then 1 to claude-sonnet-5-5
GET /check/log  →  200  every call with its example cost, one cache hit, one warning

In these tests the model is busy on purpose: its first three calls answer 429 (too many requests), and the small model is overloaded (529) every time it is asked about money back.

Your task: write the chat function and give it the address POST /chat. Then let the client retry 3 times, add the fallback model in ask_model, add the cache to chat (only for answers that used no tools), and log the cost of every model call and every cache hit. Log each model call in ask_model with logging.info, in this form: claude-haiku-5-5 answered, example cost $0.000335 (the reply's model, then the cost from cost_of with six decimals). On a cache hit, log exactly Cache hit: no model call, no cost. Make the cache key with normalize on the masked question.


Assignment
Follow the Coding Tutorial to serve the assistant at POST /chat with retries, a fallback model, a cache and cost logging.


Hint
The decorator goes on the line right above def chat():. Inside chat the order matters: read the question, make the key, look in the cache, call answer, store the result if its "tools" list is empty, return. Run the tests after every step and read /check/model and /check/log: they show what really reached the model.


STEP 1 OF 10
    Ctrl + Enter to check

    Ai Capstone Ship in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free