How Tokenizers Learn: One Merge

One id per character works, but it makes long lists: a name is as many tokens as it has letters. Real tokenizers, like the one behind ChatGPT, also give ids to common pieces of words. They learn those pieces from text with byte-pair encoding (BPE): count every pair of tokens that sit side by side, merge the most common pair into one new token, and repeat. In this lesson you do one merge.

Two Python tools do the work. counts.get(key, 0) gives back the value stored under key, or 0 when the key isn't in the dictionary yet. max(counts, key=counts.get) goes over the keys of counts, compares them by their values, and gives back the key with the biggest value. A tuple like ("k", "a") can be a dictionary key. On the word banana:

tokens:       ['b', 'a', 'n', 'a', 'n', 'a']
pairs:        ('b', 'a') 1   ('a', 'n') 2   ('n', 'a') 2
most common:  ('a', 'n')   (a tie: max keeps the first one it met)
after merge:  ['b', 'an', 'an', 'a']

The starting code splits "kamika mika kaka roka" into 21 single characters, spaces included. It also gives you merge(tokens, pair), which gives back a new list where each pair[0] followed by pair[1] became one token. When your code is done, the checks print:

different pairs: 11
most common pair: ('k', 'a') seen 6 times
after one merge: 15 ['ka', 'm', 'i', 'ka', ' ', 'm', 'i', 'ka', ' ', 'ka', 'ka', ' ', 'r', 'o', 'ka']
the next merge would be: ('ka', ' ')
8 merges learn the new tokens: ['a\n', 'e\n', 'i\n', 'o\n', 'su', 'tu', 'za', 'lo']

The last line runs your code 8 times on names.txt: the first pieces it learns are a vowel followed by the newline, the way most names end. Real tokenizers do tens of thousands of merges on huge amounts of text.

Your task: finish count_pairs(tokens) so it gives back a dictionary with how many times each pair of neighbours appears, store the most common pair in best, and merge it into tokens.

Assignment
Follow the Coding Tutorial to count the pairs, pick the most common one and do one merge.

Hint
The last token has no neighbour after it, so the loop stops one position early: range(len(tokens) - 1). Each pair is the tuple of the token at i and the one at i + 1.