Start learning free

LESSON 59 OF 107 · DATA ENGINEERING: ORCHESTRATION

Run It: A Status per Task

Practice

You have the order; now something must run the tasks. That part of an orchestrator is the runner: it walks the run order, calls each task, and writes down how the task went. That note is the task's status. Airflow shows the same table on its web page after every run, one colored box per task, green for success. When the morning report is missing, the statuses are the first thing a data engineer looks at, to see which task ran and which didn't.

The runner gets task names from static_order(), so it needs a way from a name to the code to run. Keep the functions in a dict, name to function. A function's name without brackets is the function itself, a value you can store and call later with ():

def say_hi():
    print("hi")

def say_bye():
    print("bye")

actions = {"hi": say_hi, "bye": say_bye}
actions["bye"]()   # prints: bye

The runner records "success" on the line after the call. If a task raises an error, that line never runs, so a crashed task is never marked as done. You'll handle failures in the next lesson; here every task works.

Your program prints each task's line as the runner calls it, then the status table:

extract_orders done
extract_customers done
clean_orders done
build_report done
send_email done
extract_orders: success
extract_customers: success
clean_orders: success
build_report: success
send_email: success

Your task: the five task functions, deps and the lines that print the table are already there. Add send_email to the dict tasks, then write run_pipeline(tasks, deps): start an empty dict status, loop over TopologicalSorter(deps).static_order(), call each task's function from tasks, record "success" for it, and return status.


Assignment
Follow the Coding Tutorial to write the runner, which calls every task in order and returns a status per task.


Hint
tasks[name] is a function and tasks[name]() calls it. Write return status at the indent of the for, so it runs once, after the loop.


STEP 1 OF 7
    Ctrl + Enter to check