The Run Date Is a Parameter

A daily pipeline loads one day of data per run. That day is the run date: a parameter you pass in, like run("2026-05-01"), not the date on the server's clock. Every path the pipeline touches is built from it: the file it reads and the folder it writes.

Why a data engineer cares: a job that builds its paths from date.today(), the clock's date, can only ever load "now". When it crashes at night and you re-run it the next morning, it reads the wrong day's file and the failed day is never loaded. With the date as a parameter, the same function loads yesterday, last month or any day again.

Building a path from a date is plain string joining with +:

def report_path(run_date):
    return "reports/sales_" + run_date + ".txt"

report_path("2026-04-30")   # 'reports/sales_2026-04-30.txt'

The target has one folder per day, such as warehouse/orders/date=2026-05-01/orders.parquet. A folder named column=value is a partition: with hive_partitioning = true, DuckDB's read_parquet reads the folder name back as a date column. extract, transform and load are mostly written; transform drops repeated rows and rows whose quantity isn't a number.

Your code calls run("2026-05-01") and run("2026-05-02"). The check lists the folders and counts the orders in each, then calls run("2026-05-04") itself, with no change to your code, and looks again:

['date=2026-05-01', 'date=2026-05-02']
date | orders | first_order | last_order
2026-05-01 | 10 | 5001 | 5010
2026-05-02 | 12 | 5011 | 5022
(2 rows)
The check calls run("2026-05-04"):
['date=2026-05-01', 'date=2026-05-02', 'date=2026-05-04']
date | orders | first_order | last_order
2026-05-01 | 10 | 5001 | 5010
2026-05-02 | 12 | 5011 | 5022
2026-05-04 | 9 | 5034 | 5042
(3 rows)

Your task: build the file path in extract and the folder in load from run_date, then write run(run_date), which extracts, transforms and loads that one day.

Assignment
Follow the Coding Tutorial to build every path from the run date and to write the run function.

Hint
run passes run_date on to the two functions that build a path, extract and load. transform doesn't need it: a DataFrame in, a DataFrame out.