Generate: Temperature

Your trained TinyGPT does one thing: given up to 8 characters, it scores the character that comes next. To generate a name, start from the newline (in names.txt one comes right before every name), ask for the next character, pick one, add it to the ids and ask again, until the model picks the newline that ends the name. ChatGPT writes its answers the same way, one token at a time.

Each turn of the loop does four things. ids[-block_size:] keeps the last 8 ids at most, because the model only has positions 0 to 7; wrapped in a list and passed to torch.tensor, it is a batch of one window. model(window)[0, -1] takes the scores of that window's last position: 19 numbers, one per character. torch.softmax(logits, dim=-1) turns them into probabilities that add up to 1. torch.multinomial(probs, 1, generator=g) samples: it gives back a tensor holding one id, picked at random with those probabilities, and .item() turns it into a plain int.

Temperature: before the softmax, divide the scores by a number. Below 1 it pulls the scores apart, so the likely characters win even more often; above 1 it pushes them together, so rare ones get picked too. On three made-up scores:

scores = torch.tensor([2.0, 1.0, 0.0])
torch.softmax(scores / 1.0, dim=-1)   # about 0.665, 0.245, 0.090
torch.softmax(scores / 0.5, dim=-1)   # about 0.867, 0.117, 0.016: surer
torch.softmax(scores / 2.0, dim=-1)   # about 0.506, 0.307, 0.186: wilder

The code trains the model, then samples 8 names at temperatures 0.5, 1.0 and 2.0. A check then samples 30 more at each, counts the ones copied from names.txt, and tries 0.01. The run prints:

step 0 loss: 2.986
step 100 loss: 1.446
step 200 loss: 1.431
step 300 loss: 1.392
temperature 0.5 ['rosu', 'kari', 'noka', 'bafi', 'daka', 'roni', 'nilo', 'rosu']
temperature 1.0 ['nirotu', 'basa', 'sada', 'ninoda', 'basulo', 'veza', 'motaba', 'mikaza']
temperature 2.0 ['taleba', 'daka', 'tasu', 'rimopukfi', 'ketvetumo', 'ssapesuno', 'tulesu', 'lesututa']
temperature 0.5 | 30 names: 27 different, 22 copied from names.txt, the longest 4 letters
temperature 1.0 | 30 names: 28 different, 15 copied from names.txt, the longest 8 letters
temperature 2.0 | 30 names: 30 different, 6 copied from names.txt, the longest 12 letters
temperature 0.01: ['kasu', 'kasu', 'kasu', 'kasu']

Low is safe and repetitive: 22 of 30 names are copies, and at 0.01 it is the same name every time. High is wild: rimopukfi, ssapesuno.

Your task: fill in the loop of generate(temperature): build the window, take the last position's scores, turn them into probabilities, sample next_id, stop with break when it is the newline, and last, divide the scores by temperature.

Assignment
Follow the Coding Tutorial to write one turn of the generation loop, then add the temperature.

Hint
The temperature line goes right under logits = model(window)[0, -1], before the softmax. Dividing the probabilities instead would change nothing: every one would shrink by the same factor. To spot the end of a name, compare chars[next_id] with "\n".