Embeddings

An id like 8 is only a name tag for l: the model can't learn anything from the number 8 itself. So each character gets an embedding, a row of 32 numbers. They start random, and training changes them until they hold what the model has learned about that character. PyTorch treats them like any layer's weights: it works out their gradients for you, and the optimizer updates them.

nn.Embedding(rows, size) makes such a table: rows rows of size learnable numbers each. Calling the table on a tensor of ids gives back each id's row, so ids of shape [32, 8] become numbers of shape [32, 8, 32]. The same id always gives the same row.

table = nn.Embedding(5, 3)            # 5 rows of 3 numbers
out = table(torch.tensor([4, 1, 4]))
print(list(out.shape))                # [3, 3]: out[0] and out[2] are both row 4

A character's row says what it is, not where it stands. A second table, with one row for each position 0 to 7, adds that. torch.arange(n) gives back the positions 0, 1, …, n - 1 to look up. Adding a character's row and its position's row gives one vector that carries both; the 8 position rows have shape [8, 32], and PyTorch adds them to each of the 32 windows of the batch.

The names use 19 characters (18 letters and the newline). In the first window, 'kelolo\nr', positions 2 and 4 both hold l: the same token row, but a different sum. Once your code works, the check prints:

token table: [19, 32] position table: [8, 32]
x: [32, 8] tok: [32, 8, 32] pos: [8, 32] h: [32, 8, 32]
x[0]: 'kelolo\nr'
positions 2 and 4 hold 'l' and 'l'
tok[0, 2] is row l of the table: True
tok, first 4 numbers: position 2 [-0.039, -0.801, -0.496, -0.362] position 4 [-0.039, -0.801, -0.496, -0.362]
h, first 4 numbers:   position 2 [0.915, -0.068, -1.951, -2.407] position 4 [-0.138, -0.316, 0.212, -0.318]
h[0, 2] is tok[0, 2] + pos[2]: True
numbers the model can learn: 864 | tok learns: True

Your task: the vocabulary, data and get_batch from the earlier lessons are in the starting code, with n_embd = 32. Make a token table token_emb with a row for each of the vocab_size characters, and a position table pos_emb with a row for each of the block_size positions. Look up the ids of x in tok and the positions in pos, and store their sum in h.

Assignment
Follow the Coding Tutorial to give every character in the batch its embedding and its position.

Hint
Call a table like a function on a tensor of ids. The positions to look up are torch.arange(block_size).