Lazy Steps and Actions

In pandas, every line does its work right away. Spark is lazy. Methods like where and select are transformations: each one returns a new DataFrame at once, but that DataFrame is only a plan, the plan it started from plus one more step. No row is read.

An action asks for a result Python can hold: count() (a number), show() (rows to print), collect() (the rows as a Python list), or writing a file. Only an action makes Spark run the plan, from reading the file to the last step, and it runs it again for every action. Data engineers rely on this: because Spark sees the whole chain before it starts, it does all the steps in one pass over the data, where building a full table after every line would cost hours on terabytes.

The new pieces: F.col("status") names a column (F is pyspark.sql.functions, imported by the starting code); where(condition) keeps the rows where the condition is true, SQL's WHERE; select("a", "b") keeps only those columns, SQL's SELECT a, b. For example, on the customers file:

regulars = customers.where(F.col("visits") >= 10)  # returns at once: a plan
ids = regulars.select("customer_id", "city")       # returns at once: a bigger plan
ids.show(3)                                        # the action: Spark runs both steps

In this lesson's program, the read line goes through orders.csv once to guess the types. The lines that make paid, bulk and report return at once, and printing type(report) only looks at the Python object. count() is the action: Spark reads the file, keeps the paid lines with a quantity of 3 or more and counts them. The program prints this (the last line comes from the check, which prints report.columns; Spark knows the columns from the plan, so that is no action):

<class 'pyspark.sql.classic.dataframe.DataFrame'>
409
['order_id', 'product', 'quantity']

A Check takes about 10 seconds here, because Spark starts a Java program before your code runs.

Your task: below the read, build paid (the paid lines of orders), bulk (the lines of paid with a quantity of 3 or more) and report (order_id, product and quantity of bulk). Then print the type of report, and last print its count().

Assignment
Follow the Coding Tutorial to build a chain of three transformations and run it with one action.

Hint
Each new DataFrame starts from the one above it: paid from orders, bulk from paid, report from bulk. Conditions use == and >=; count() returns a number, so wrap it in print().