Could It Be Luck? A Permutation Test

The starting code builds the same A/B test as the last lesson, with the same seed. Group a is 2,000 visitors on the old page (210 bought, rate 0.105). Group b is 2,000 visitors on the new page (236 bought, rate 0.118). The code prints the real difference, observed: 0.013. Could luck alone make a gap that big?

Imagine the new page makes no difference. Then each visitor would have bought, or not, in either group, so the labels A and B are just names. Pool all 4,000 visitors, shuffle them, and deal them into two new groups of 2,000. Their gap is one gap made by luck alone. Do it 5,000 times and you see how big luck's gaps get. This is a permutation test.

Two new functions: np.concatenate([x, y]) gives back one array, x followed by y. rng.permutation(x) gives back a shuffled copy of x: the same values in a random order.

x = np.concatenate([np.array([1, 1]), np.array([0, 0, 0])])
print(x)                    # [1 1 0 0 0]
print(rng.permutation(x))   # the same five values, in a random order

The p-value is the share of shuffled gaps at least as big as the real one, either way: a gap of -0.014 counts as bigger than 0.013. np.abs(...) gives back values without their signs. np.mean of True/False values gives back the share of True, so np.mean([True, False, False, False]) is 0.25. np.round(values, 3) gives back an array rounded to 3 decimals. A small p-value means luck rarely makes a gap like yours. A big one means it happens often. Your code must print:

Real difference: 0.013
Five shuffles: [-0.011  0.003 -0.014 -0.012  0.003]
p-value: 0.2062

Your task: pool a and b into both. Shuffle it 5,000 times, and each time append to fakes the rate of the last 2,000 minus the rate of the first 2,000. Print the first five fakes rounded to 3 decimals, then the p-value p: the share of fakes at least as big as observed, either way.

Assignment
Follow the Coding Tutorial to shuffle the group labels 5,000 times and turn the shuffled gaps into a p-value.

Hint
In the shuffled array, shuffled[:2000] plays group A and shuffled[2000:] plays group B. Compare sizes without their signs, so gaps below 0 count too.