Tasks and What They Wait For

A data pipeline is a chain of small jobs called tasks: fetch the orders, clean them, build a report, email it. Some tasks must wait for others: the report can't be built before the orders are clean. Tools called orchestrators (Airflow, Dagster, Prefect) run such pipelines for you every day. In this chapter you build a small one in plain Python.

An orchestrator first needs the pipeline written down as data. A dict does it: each key is a task's name, and its value is the list of the tasks it waits for. An empty list means the task waits for nothing, so it can start first. A for loop over a dict gives you its keys, in the order you wrote them. Here is a two-task pipeline from another team:

steps = {
    "download_rates": [],
    "convert_prices": ["download_rates"],
}
for name in steps:
    print(name, len(steps[name]))
# download_rates 0
# convert_prices 1

Why write it down? Without it, someone runs the scripts by hand in the order they remember, and one morning the report is built before the cleaning has finished and the email goes out with half the orders.

The shop's pipeline has five tasks: extract_orders and extract_customers wait for nothing, clean_orders waits for extract_orders, build_report waits for clean_orders and extract_customers, and send_email waits for build_report. Your program prints:

extract_orders waits for []
extract_customers waits for []
clean_orders waits for ['extract_orders']
build_report waits for ['clean_orders', 'extract_customers']
send_email waits for ['build_report']
Can start first: ['extract_orders', 'extract_customers']

Your task: the first three tasks are already in deps. Add build_report and send_email with the lists they wait for. Then loop over the task names: print what each task waits for, and add to ready every task whose list is empty.

Assignment
Follow the Coding Tutorial to finish the dict deps, print what each task waits for and find the tasks that can start first.

Hint
Each entry is a name, a colon, then a list of names: "task": ["other_task"],. A list is empty when its len() is 0.