A Table Is Files Plus a Log

A Delta table is a folder of Parquet files plus a log. The function write_deltalake(folder, df) from the deltalake package writes a pandas DataFrame into the folder as Parquet files and adds one JSON file to its subfolder _delta_log. Each write is a new version of the table, numbered from 0, and its log file is named by that number with 20 digits: 00000000000000000000.json, then 00000000000000000001.json.

Each line of a log file is one action: a JSON object whose only key is the action's name. add says a Parquet file is now part of the table, remove says a file no longer is, and commitInfo records what the write was. A write with mode="overwrite" replaces the table's rows but deletes no file: it writes a new Parquet file and a log file that removes the old one. Readers such as DeltaTable("orders_delta") read the log first and skip removed files.

Why it matters: a plain Parquet folder has no log, so a reader can't tell old files from current ones. After a rewrite it reads both and counts the orders twice.

write_deltalake("visits_delta", visits)   # version 0: a Parquet file + _delta_log/00000000000000000000.json

action = json.loads('{"remove": {"path": "old.parquet"}}')
print(list(action)[0])   # remove (list() of a dict gives its keys)

Your program writes two versions of orders_delta and prints how many Parquet files the folder holds, the log's file names and the actions in the newest log file. A hidden check then compares the folder with the table. The output must be:

Parquet files in the folder: 2
Log files: ['00000000000000000000.json', '00000000000000000001.json']
commitInfo
remove
add
Parquet files in the table: 1
Rows in all the folder's Parquet files: 22
Rows in the table: 12

Your task: the starting code reads 05-01's and 05-02's exports into day1 and day2 and prints how many Parquet files the folder holds, with glob. Above that print, write day1 as the table orders_delta and overwrite it with day2. Below it, put the sorted names in orders_delta/_delta_log in a list log and print it, then open the newest log file, log[-1], and print the name of the action on each of its lines.

Assignment
Follow the Coding Tutorial to write the table twice and then read its log yourself.

Hint
os.listdir gives a folder's names in no fixed order: sort them so log[-1] is the newest version. json.loads(line) turns one line into a dict with one key; list() of the dict gives its keys, and [0] the first one.