What a p-value Does Not Say

A p-value answers one narrow question: if the two pages were the same, how often would luck alone give a gap at least as big as the one you saw? Below 0.05 is called 'significant'. A p-value is not the chance that the new page is better, and a small one is not proof: even identical pages sometimes give one.

You can see this by running tests where you know the truth. rng.random(n) gives back n random numbers between 0 and 1, so rng.random(n) < 0.10 gives back n True/False values, True for about 10% of them: n visitors who each buy with a 10% chance. The same trick flips 1,000 coins:

heads = rng.random(1000) < 0.5
print(heads.sum())   # close to 500, rarely exactly 500

The starting code gives you p_value(a, b), the permutation test from the last lesson in a faster form: rng.hypergeometric deals the buyers out to the two groups at random 2,000 times, and the function gives back the share of those shuffles with a gap at least as big as the real one.

Your program runs 200 A/B tests. In each one, groups a and b have 2,000 visitors who each buy with a 10% chance, so the pages are truly the same. It keeps the 200 p-values in a numpy array (np.array), prints the first ten (p_values[:10]) and counts the ones below 0.05: p_values < 0.05 gives back True for each 'significant' test, and .sum() counts the Trues. It must print:

First ten: [0.8435 0.696  0.223  0.6905 0.8825 0.8035 0.352  0.3695 1.     0.0075]
Below 0.05: 8 of 200

Your task: below p_value, run 200 tests on two identical groups, collect their p-values and count how many look significant. About 5% will: that is what the 0.05 threshold costs.

Assignment
Follow the Coding Tutorial to run 200 A/B tests on two identical pages and count the false winners.

Hint
Build a and b inside the loop, so every test gets new visitors. Turn the list into a numpy array before you compare it with 0.05: a plain list can't be compared with a number.