Replace One Day's Data

Pipelines get run again all the time: a retry after a crash, a re-run after a bug fix, a teammate starting the job twice. A load is idempotent when running it twice leaves the same data as running it once. If it isn't, every re-run quietly doubles a day's revenue in each report built on the warehouse.

The starting load adds a date column and writes with df.to_parquet("warehouse/orders", partition_cols=["date"], index=False). partition_cols makes one folder per value of that column, such as date=2026-05-02, and writes a file with a new random name into it. Run the same day again and a second file lands next to the first: nothing is replaced.

The fix is to replace the partition: delete the day's folder, then write it. shutil.rmtree(path) deletes a folder and everything in it. With ignore_errors=True, a missing folder is not an error, which is what the first run of a day needs:

import shutil
shutil.rmtree("exports/2026-04", ignore_errors=True)   # the folder and its files are gone
shutil.rmtree("exports/2026-04", ignore_errors=True)   # already gone: no error, nothing happens

Only the run date's folder is deleted, never warehouse/orders itself, so the other days stay as they are. Your code runs 05-01 once and 05-02 twice; the check counts the files in each day's folder, then the rows and the distinct orders per day. The starting code leaves 2 files in date=2026-05-02: 24 rows for 12 orders. Yours must print:

date=2026-05-01: 1 file
date=2026-05-02: 1 file
date | rows | orders
2026-05-01 | 10 | 10
2026-05-02 | 12 | 12
(2 rows)

Your task: in load, set folder to the run date's folder and delete it with shutil.rmtree, above the lines that write.

Assignment
Follow the Coding Tutorial to make load replace the day's folder instead of adding to it.

Hint
Delete first, then write: the other way round deletes the file you just wrote. The folder is warehouse/orders/date= followed by the run date, the one partition_cols writes into.