Learn: Gradient Descent

Last lesson you measured the gradient of the loss: the slope for each weight. A negative slope means the loss falls when the weight grows, so the weight should go up. A positive slope means it should go down. Gradient descent does this over and over: each round, every number takes a small step against its own slope. It becomes itself minus learning_rate times the slope.

Here is the rule on one number, with the loss (w - 3) ** 2 and its slope 2 * (w - 3). Each round moves w closer to 3, where the loss is 0:

w = 0.0
for step in range(3):
    slope = 2 * (w - 3)
    w = w - 0.1 * slope
    print(round(w, 3))     # 0.6, then 1.08, then 1.464

The steps get shorter by themselves near the bottom, because the slope gets smaller there. learning_rate sets the size of each step: too small and learning crawls, too big and the number jumps over the bottom.

The starting code holds the AND data, the neuron's w and b, learning_rate = 0.5, neuron_loss(w, b), which gives back the loss as one number, and gradients(w, b), which gives back two things: an array with the slope of each weight, and one number, the slope of the bias. % gives the remainder of a division, so i % 500 is 0 on rounds 0, 500, 1000 and so on. Your program must print (the last line comes from the check):

round 0 loss: 0.2743
round 500 loss: 0.0252
round 1000 loss: 0.0122
round 1500 loss: 0.0078
round 2000 loss: 0.0057
round 2500 loss: 0.0044
round 3000 loss: 0.0036
w: [5.142 5.142] b: -7.808
out: [0.    0.065 0.065 0.922]
rounded: [0. 0. 0. 1.] y: [0 0 0 1]

Your task: above the last two prints, write a loop of 3001 rounds. In each round, print the round and the loss when the round is a multiple of 500, get the two slopes from gradients(w, b), and move w and b one step against their slopes.

Assignment
Follow the Coding Tutorial to train the neuron on AND with gradient descent and print its loss every 500 rounds.

Hint
The print sits inside the if, eight spaces in; the slopes and the two updates sit in the loop, four spaces in. Update b too: if only w learns, the loss gets stuck at 0.25.