Small Files and Compaction

Every write to a Delta table adds at least one new Parquet file. A table fed by a daily or hourly job ends up as many small files, and every read has to open all of them: each open costs time, even for a file of ten rows. This is the small files problem, and at work it's why a table of thousands of tiny files makes every query and dashboard slow.

Compaction fixes it. dt.optimize.compact() reads the table's small files and writes their rows into fewer, bigger files, as a new version (its history entry says OPTIMIZE). The rows stay the same; only the files change. It returns a dict of counts, among them numFilesAdded and numFilesRemoved. To see how many files a read opens, count dt.file_uris(), the list of the current version's data files.

The old files leave the table, not the disk: the new version's log entry marks them removed, but older versions still list them, so time travel keeps working. glob.glob("orders_delta/*.parquet"), from Python's glob module, lists every file in the folder whose name ends in .parquet, whichever version uses it. Delta's vacuum deletes such files later, once you no longer need the old versions.

Here a customers table is compacted, and an older version still reads its own files:

dt = DeltaTable("customers_delta")
dt.optimize.compact()
print(dt.history()[0]["operation"])   # OPTIMIZE
old = DeltaTable("customers_delta", version=2)
print(len(old.file_uris()), "files in version 2")

Your code prints four lines. The check then prints the table's version and rows, its two newest history entries, and what version 4 still reads:

Files before: 5
Added: 1 - removed: 5
Files after: 1
Parquet files in the folder: 6
Version: 5 - 56 rows
5 OPTIMIZE
4 WRITE
Version 4 still reads 5 files, 56 rows

Your task: the starting code appends five daily files (05-01, 05-02, 05-04, 05-05 and 05-07: 56 rows) to orders_delta, one write each, so the table is at version 4, and opens it as dt. Print how many data files the table has, compact it and keep the counts as result, print the files added and removed, print the table's files again, and count the .parquet files in the folder.

Assignment
Follow the Coding Tutorial to count the table's files, compact them, and compare what the table lists with what the folder holds.

Hint
len(dt.file_uris()) counts the files of the table's current version; len(glob.glob(...)) counts every Parquet file on disk. After the compaction they differ: the five small files left the table but not the folder.