Capstone: Test It Honestly

Since the split, the 75 test rows have waited untouched: cross-validation only used the 225 training rows. Now the model gets its one honest check, like an exam it has never seen. First fit trains it once, on all the training rows. Then model.score(X, y) gives back the share of rows it gets right, and model.predict(X) gives back its 0 or 1 for each row.

Accuracy hides which mistakes the model makes. confusion_matrix(real, predicted) gives back a 2×2 table: rows are the real answers (0, then 1), columns the predictions. precision_score(real, predicted) gives back the share of predicted 1s that really are 1, and recall_score(real, predicted) the share of real 1s that were predicted 1. On five made-up customers:

real = [1, 0, 1, 1, 0]
said = [1, 1, 0, 0, 0]
print(confusion_matrix(real, said))        # [[1 1]
                                           #  [2 1]]
print(precision_score(real, said))         # 0.5: 1 of the 2 predicted 1s is right
print(round(recall_score(real, said), 3))  # 0.333: 1 of the 3 real 1s is found

model.predict_proba(X) gives back two columns per row: the chance of 0 and the chance of 1; [:, 1] keeps column 1, the chance of buying again. The shop sends a coupon to customers it expects back, and each coupon costs money, so the rule is: send one when the chance is at least a cut. chances >= cut gives back True or False for each customer, and .sum() counts the Trues. A higher cut sends fewer coupons: usually more of them reach customers who come back (precision), but fewer of those customers get one (recall).

Your script must print this (the last line comes from the lesson's check, once coupon_cut is set):

Baseline: 0.596
Model: [0.733 0.822 0.778 0.711 0.778] mean 0.764
Test accuracy: 0.813
[[21  9]
 [ 5 40]]
cut coupons precision recall
0.5 49 0.816 0.889
0.6 43 0.814 0.778
0.7 37 0.838 0.689
0.8 28 0.929 0.578
Coupons at 0.8 go to 28 test customers; 26 of them came back

Your task: below the cross-validation, fit model on the training rows, print the test accuracy and the confusion matrix, and keep in chances each test customer's chance of buying again. Then, below the header print, loop over the cuts 0.5, 0.6, 0.7 and 0.8 and print each cut, its number of coupons, its precision and its recall. Last, keep in coupon_cut the lowest cut whose precision is at least 0.9.

Assignment
Follow the Coding Tutorial to give the model its one check on the test rows and pick the coupon cut.

Hint
Both confusion_matrix and the two scores take the real answers first: y_test, then the predictions or coupons. In the printed matrix, the top-right number counts customers predicted to come back who didn't.