PCA Inside a Pipeline

PCA learns from data, just like a scaler or a model. So when you cross-validate, it belongs inside the pipeline: in each round the scaler, the PCA and the model all learn from that round's training rows only, and the rows used for scoring stay unseen.

make_pipeline runs its steps in order, each one feeding the next. Put a PCA after the scaler and before the model, and the model learns from the PCA columns instead of the original ones. A pipeline can be indexed like a list: pipe[-1] gives back its last step, and after fitting, that step's n_features_in_ gives back how many columns it learned from.

from sklearn.neighbors import KNeighborsClassifier

knn = make_pipeline(StandardScaler(), PCA(n_components=2), KNeighborsClassifier())
knn.fit(X, y)
print(knn[-1].n_features_in_)   # 2: the neighbours are found in the 2 PCA columns

The starting code cross-validates plain, the scaler and the logistic regression on all 5 columns. With your lines, the script prints:

All 5 columns: [0.733 0.783 0.867 0.8   0.817] mean 0.8
3 PCA columns: [0.717 0.767 0.8   0.833 0.8  ] mean 0.783
Columns the model sees: 3
Steps: ['standardscaler', 'pca', 'logisticregression']

The last line comes from the check. With 3 columns instead of 5 the mean drops only 0.017, less than the scores move from one round to the next.

Your task: below the blank line, build with_pca: the pipeline of plain with PCA(n_components=3) between the scaler and the model. Cross-validate it on X and y with cv=5 into pca_scores and print them like the line above. Then fit it on all of X and y and print the n_features_in_ of its last step.

Assignment
Follow the Coding Tutorial to put PCA inside the pipeline, cross-validate it and compare it with the plain one.

Hint
cross_val_score trains copies of the pipeline and leaves with_pca itself untrained, so fit it before you read with_pca[-1].n_features_in_.