Show It a Picture

A model can look at pictures too. But it sees only what is in your request: a photo in a file on your computer has to travel inside the message.

A request is JSON, and JSON carries text, not raw bytes. Base64 is the usual way around that. It rewrites any bytes using only letters, digits and a few signs, and the other side turns them back into the same bytes:

import base64

raw = b"Hi"                              # two bytes
text = base64.b64encode(raw).decode()    # an ordinary string
print(text)                              # SGk=

The picture travels as an image block. So far a message's "content" was a string. It can also be a list of blocks: an image block, then a text block with your question. Inside the image block, a source says how the data is written (base64), what kind of file it is (the media type, like image/jpeg for a JPEG photo) and the data itself.

The shop's photo is in the file mug.png. Your program must print the model's description and the input tokens of the call. The last line is printed by the course's own check that the photo arrived whole:

A glossy blue ceramic mug with a round handle, standing on a white table.
Input tokens: 1011
The model received mug.png whole: yes

Images cost tokens too. The course model counts 1000 tokens for every image. With a real model the number grows with the size of the picture, so send photos no larger than the job needs.

Your task: the client, MODEL and the text block question are already written. Read mug.png as bytes, turn the bytes into a base64 string, build the image block, and send one user message whose content is the image and then the question. Print the reply's text, then "Input tokens:" and the call's input tokens.

Assignment
Follow the Coding Tutorial to send the photo in mug.png to the model as an image block and print its description.

Hint
Open the photo with the mode "rb": read bytes. base64.b64encode returns bytes, so add .decode() to get a string. The media type of a PNG file is image/png, and the content list holds the image first, then the question.