Stream to the Browser

The model writes its answer piece by piece. On a website, your Flask server stands between the browser and the model. If the route waits for the whole answer, the customer looks at an empty chat for seconds. So the server must pass each piece on the moment it arrives.

A generator: a function with yield hands out one value, pauses, and goes on when the next value is wanted. Wrap a generator in a Response and Flask sends every value as soon as it has it. Here is one on another route:

def ticks():
    yield "data: one\n\n"
    yield "data: two\n\n"

@app.get("/ticks")
def show_ticks():
    return Response(ticks(), mimetype="text/event-stream")

Each value is a server-sent event, a format browsers understand: a line that starts with data: , then a blank line that ends the event. The page reads the events with EventSource, which can only send GET, so the question travels in the URL as ?q=…. json.dumps puts a piece in quotes and writes a line break inside it as \n, so a piece always stays on one line.

These are the requests your code must answer. The model in this course writes 12 characters at a time:

GET /chat?q=When+are+you+open  →  200  text/event-stream
data: "We are open "

data: "from 8:00 to"

data: " 20:00, ever"

data: "y day."

data: [DONE]

GET /chat?q=What+do+you+have+for+tea+lovers  →  200  data: "For tea love" … data: [DONE]
GET /chat               →  400  {"error": "A question is required"}
GET /chat?q=%20%20      →  400  {"error": "A question is required"}
GET /check/model-calls  →  200  {"model_calls": 2, "streamed": 2}

Your task: the app, the client, MODEL and SYSTEM are already created for you. Write the generator events(question): it yields each piece of the model's stream as an event, and one last event, data: [DONE], so the page knows the answer is complete. Then write the route GET /chat: it answers 400 when there is no question, and otherwise returns the stream.

Assignment
Follow the Coding Tutorial to build the events generator and the GET /chat route that streams it.

Hint
Refuse an empty question before the stream starts: the status code is the first thing a response sends, so after the first piece it is too late for a 400. Every event ends with two \n: one ends the line, one makes the blank line.