Parquet In and Out

A CSV file is only text. Every reader has to guess the types again, and one bad value such as "two" turns a whole number column into text. Parquet is the file format data teams keep tables in: it stores each column's name and type with the data and keeps each column's values together, compressed. A reader gets the right types at once and reads only the columns it needs, which is why pipelines hand data to each other as Parquet, not as CSV.

Write and read: df.write.mode("overwrite").parquet("folder") writes a DataFrame. Spark makes a folder, not one file: one part-….parquet file per partition and, last, an empty _SUCCESS file that says the whole write worked. mode("overwrite") replaces the folder an earlier run left; without it, the second run stops with an error. The write is an action: it runs the whole chain. spark.read.parquet("folder") reads the folder back with the stored types, so it needs no header and no inferSchema.

One folder per value: .partitionBy("column"), between mode(…) and .parquet(…), writes one sub-folder per value of that column and keeps the value in the folder's name. A job that needs one value opens only its folder instead of reading every row. On the customers file, this writes the sub-folders newsletter=no and newsletter=yes inside customers_pq:

customers = spark.read.csv("customers.csv", header=True, inferSchema=True)
customers.write.mode("overwrite").partitionBy("newsletter").parquet("customers_pq")

Python's glob.glob("folder/*") lists what is inside a folder; wrap it in sorted(…) so the order is the same on every run. Your program prints the schema it read back and what is inside paid_by_category, then the check counts the part files in paid_orders:

root
 |-- order_id: integer (nullable = true)
 |-- customer_id: integer (nullable = true)
 |-- date: date (nullable = true)
 |-- product: string (nullable = true)
 |-- category: string (nullable = true)
 |-- quantity: integer (nullable = true)
 |-- unit_price: integer (nullable = true)
 |-- status: string (nullable = true)

paid_by_category/_SUCCESS
paid_by_category/category=gifts
paid_by_category/category=home
paid_by_category/category=kitchen
paid_by_category/category=office
part files in paid_orders: 1

Your task: the starting code imports glob, reads orders.csv and keeps the paid lines in paid. Write paid as Parquet into paid_orders, read the folder back into back and print its schema, write paid again into paid_by_category partitioned by category, and print every path inside that folder. A Check takes about 10 seconds here, because Spark first starts a Java program.

Assignment
Follow the Coding Tutorial to write paid as Parquet, read it back with its types, and write it again with one folder per category.

Hint
The two writes differ only by .partitionBy("category") between mode("overwrite") and .parquet(…). The loop's body, print(path), is indented under the for line.