Start learning free

LESSON 127 OF 129 · AI ENGINEERING: INTERVIEW QUESTIONS

Cache Model Answers in Python

Practice

Interview question: "Your support bot gets the same questions all day, and every model call costs money and a second or two. Add a cache."

A response cache is a dict from a request to its answer. The key must hold everything that changes the answer: the model, the system prompt and the messages. Turn them into one text with json.dumps(..., sort_keys=True), so equal requests always give equal text. Then hash that text with hashlib.sha256 to get a short key of fixed length. Look the key up before calling the model; on a miss, call it and store the answer.

The same idea on other data: a translation helper could key on the language and the sentence. hashlib.sha256(json.dumps(["fr", "Good morning"]).encode()).hexdigest() gives the same 64 characters on every run, and changing either part gives a new key.

This lesson's script asks "Do you ship to Canada?" four times. The second ask repeats the first, the third adds " Be friendly." to the system prompt, and the fourth uses another model. Only the repeat is a hit:

Yes, we ship to Canada in 7 to 10 days.
Yes, we ship to Canada in 7 to 10 days.
Yes, we'd love to ship to Canada! Your parcel arrives in 7 to 10 days.
Yes: delivery to Canada takes 7 to 10 business days and costs $15.
Model calls: 3
Cache entries: 3

Your task: write cache_key(model, system, messages) and use it in ask: compute the key, store each new answer in cache, and return the stored answer on a hit, before any call to the model. Your script prints only the four answers. The check runs after your program and prints the last two lines, Model calls and Cache entries, by itself. It counts the model calls on its own and reads the size of cache. So do not add a call counter or print those two lines.


Assignment
Follow the Coding Tutorial to build the cache key and the lookup that skips repeated calls.


Hint
sort_keys=True writes every dict's keys in one order, so two equal messages give the same key. A hash needs bytes: call .encode() on the text first.


STEP 1 OF 7
    Ctrl + Enter to check

    Aiq Response Cache in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free