Softmax

A common ML interview question: write softmax from scratch with numpy. A classifier gives one raw score per class (interviewers call them logits). Softmax turns those scores into probabilities: each one between 0 and 1, all of them adding up to 1, and a bigger score always gets a bigger share.

The idea in one sentence: raise e to each score, then divide each result by the sum of them all. np.exp(a) gives back a new array with e (about 2.718) raised to every number in a: always positive, and much bigger for bigger numbers. a.sum() gives back the total of the array, a.max() gives back its biggest number, and dividing an array by a number divides every element. On other numbers:

a = np.array([1.0, 1.0, 2.0])
print(a.sum())                       # 4.0
print(a / a.sum())                   # 0.25, 0.25 and 0.5: shares that add up to 1
print(np.exp(np.array([0.0, 1.0])))  # 1 and about 2.718

The trap: np.exp(1000.0) is too big for a float, so numpy gives back inf (infinity) with a warning, and inf / inf is nan (not a number). So first subtract the biggest score from every score. The biggest shifted score is 0, so no exp is bigger than 1. The probabilities stay the same, because the shift divides every exp and their sum by the same number.

The starting code makes scores and big_scores and prints what your function gives back for them:

Scores: [2.  1.  0.1]
Probabilities: [0.659 0.242 0.099]
Sum: 1.0
For the big scores: [0.09  0.245 0.665]

Below that, the check prints six more lines. It shows the naive formula's [nan nan nan] for the big scores, then runs your function on very negative scores, four equal scores, one score and every score plus 100, and compares it with scipy's softmax on 20 random score lists.

Your task: write the function softmax(z). It subtracts the biggest score from every score, raises e to each shifted score, and returns each exp divided by the sum of the exps.

Assignment
Follow the Coding Tutorial to write softmax(z) in four lines: the def, the shift, the exps and the return.

Hint
Subtract z.max() before you call np.exp, not after: the shift is what keeps np.exp away from huge numbers. Then divide by the sum of the exps, not the sum of the scores.