How Many Visitors Do You Need?

A test can miss a lift that is really there. When the new page truly is better, the share of experiments in which the test finds it (p below 0.05) is the power of the test. Power grows with the number of visitors: in small groups luck is bigger than the lift and hides it.

You measure power by simulation: build groups where you know the truth, test them many times and count the finds. Counting works like any loop with a counter. Here rng.random() gives back one random number between 0 and 1, so the if is true about half the time:

heads = 0
for i in range(100):
    if rng.random() < 0.5:
        heads += 1
print(heads / 100)   # close to 0.5

The starting code has p_value(a, b) from the last lessons and a loop over the group sizes 200, 1,000, 2,000 and 5,000 that sets found = 0 and prints the power, found / 300. Inside it you run 300 experiments. rng.random(size) gives back size random numbers between 0 and 1, so rng.random(size) < 0.10 gives back True for each visitor who buys: the old page (a) sells with a 10% chance, the new page (b) with 12%, a real lift of 2 points. Each time p_value(a, b) is below 0.05, the test found the lift. The program must print:

200 visitors per group: found the lift 24 of 300 times, power 0.08
1000 visitors per group: found the lift 77 of 300 times, power 0.26
2000 visitors per group: found the lift 153 of 300 times, power 0.51
5000 visitors per group: found the lift 268 of 300 times, power 0.89

Your task: inside the size loop, run 300 experiments with a real 2-point lift and count how many the test finds. A common goal is power 0.8: of these sizes, only 5,000 visitors per group gets there.

Assignment
Follow the Coding Tutorial to simulate 300 experiments for each group size and count how often the test finds the lift.

Hint
Watch the indents: the loop of 300 sits inside the size loop, the groups and the if inside that, and found += 1 inside the if. Use size for the number of visitors, not a fixed number.