Project: A Network Learns XOR

XOR answers 1 when exactly one of its two inputs is 1: for the rows 0 0, 0 1, 1 0 and 1 1 it gives 0, 1, 1, 0. One neuron can't learn it. A neuron answers above 0.5 on one side of a single straight line, and XOR's two 1s sit on opposite corners, with the 0s on the other two. The check trains one neuron on XOR for 5000 rounds: it ends at 0.5 for every row, a loss of 0.25.

A network with a hidden layer can learn it, and you have written all its parts. forward(X) gives back hidden and out; backward(X, y, hidden, out) gives back the four gradients grad_W1, grad_b1, grad_W2 and grad_b2; gradient descent moves every number a small step against its slope. The starting loop already runs forward and prints the loss every 1000 rounds, but nothing learns yet, so it prints 0.3276 every time.

To read the answers, take the one column of out. out[:, 0] means every row, column 0, and gives back a flat array of 4 numbers. np.round with no number of decimals rounds to whole numbers:

p = np.array([[0.2], [0.9], [0.7]])
print(p[:, 0])            # [0.2 0.9 0.7]
print(np.round(p[:, 0]))  # [0. 1. 1.]

Your program must print (the last line comes from the check):

round 0 loss: 0.3276
round 1000 loss: 0.1292
round 2000 loss: 0.0045
round 3000 loss: 0.0017
round 4000 loss: 0.001
round 5000 loss: 0.0007
out: [0.014 0.972 0.975 0.034]
rounded: [0. 1. 1. 0.] y: [0 1 1 0]
one neuron on XOR after 5000 rounds: loss 0.25 out [0.5 0.5 0.5 0.5]

Your task: at the end of the loop, get the four gradients from backward and update W1, b1, W2 and b2 with learning_rate. After the loop, print "rounded:", the rounded answers, "y:" and the targets y[:, 0].

Assignment
Follow the Coding Tutorial to train the 2-3-1 network on XOR and print its rounded answers next to the targets.

Hint
All four updates follow the same rule: the array minus learning_rate times its own gradient. If only the hidden layer learns, the loss stalls near 0.17: the output layer has to learn too.