Start learning free

LESSON 75 OF 165 · MACHINE LEARNING: ML INTERVIEW QUESTIONS

Write train_test_split

Practice

Interviewers ask: write train_test_split yourself, with numpy only. A model is scored on a test part of the rows that it never trained on, so the score shows how it does on new customers. The test rows are picked at random, so a sorted file can't put, say, only the newest customers in the test part.

The trick is to shuffle the row numbers, not the rows. np.random.default_rng(seed) gives back a random generator; the same seed gives the same random numbers on every run. Its method rng.permutation(n) gives back the numbers 0 to n - 1, each once, in a shuffled order. int(x) drops the part after the point, so int(4.5) is 4.

Slices cut an array in two: [:k] keeps the first k items and [k:] keeps the rest. An array of row numbers in square brackets picks those rows, in that order:

names = np.array(["Ana", "Bo", "Cy", "Dan"])
print(names[np.array([3, 0])])   # ['Dan' 'Ana']
print(names[:1], names[1:])      # ['Ana'] ['Bo' 'Cy' 'Dan']

Pick X and y with the same row numbers, so every label stays with its row. With the lesson's 10 rows, test_size 0.3 and seed 0, your code and the check print:

Train rows: 7 - test rows: 3
X_test:
[[  7 210]
 [  9 330]
 [  8 300]]
y_test: [0 1 1]
y_train: [0 1 0 0 1 1 0]
20 rows, test_size 0.25: 15 train, 5 test
Every row used once: True
Labels stay with their rows: True
Same seed, same split: True
Another seed, another split: True

Your task: inside train_test_split, below rng, shuffle the row numbers into order, work out n_test, cut order into test_rows and train_rows, and return X_train, X_test, y_train, y_test.


Assignment
Follow the Coding Tutorial to shuffle the row numbers, cut them into a test part and a training part, and return the four arrays.


Hint
Cut order at n_test: the first part holds the test row numbers, the rest the training ones. Index X and y with the same row numbers, in the order the four names are unpacked.


STEP 1 OF 5
    Ctrl + Enter to check