Time Travel

Every write to a Delta table makes a new version, and the old versions stay. An append never changes the Parquet files already there. It adds a new file, and the log in _delta_log records which files make up each version. So you can read the table as it was at any earlier version. This is called time travel: DeltaTable("orders_delta", version=1) opens version 1 for reading and changes nothing.

Here is an example with a different version. Version 0 is the first write, 05-01's 10 rows:

first = DeltaTable("orders_delta", version=0)
print(first.version(), len(first.to_pandas()))   # prints: 0 10

Restore goes one step further. dt.restore(1) makes version 1's rows the current table again, and it deletes nothing. It writes a new version (here 3) with version 1's rows. The history lists it as RESTORE, and the bad version can still be read.

Why a data engineer needs this: at work, a job sometimes runs twice or a bug writes wrong rows. Without versions, you would rebuild the table from every source file. With versions, you read the last good one, check it, and restore it in one line.

The starting code writes 05-01 (10 rows), appends 05-02 (12 rows), then appends 05-02 again: the job ran twice. Its report(name, table) prints a table's version, its number of rows and its number of different order ids. Your program must print:

Now - version 2 - 34 rows, 22 order ids
Before the mistake - version 1 - 22 rows, 22 order ids
After restore - version 3 - 22 rows, 22 order ids

Then the check prints the history (3 RESTORE, 2 WRITE, 1 WRITE, 0 WRITE) and reads version 2 again: it still has 34 rows.

Your task: below report("Now", dt), open version 1 of the table as old and report it as "Before the mistake". Then restore version 1 on dt and report dt as "After restore".

Assignment
Follow the Coding Tutorial to read the table as it was before the mistake, then restore it as a new version.

Hint
DeltaTable takes the folder's name, then version= with a number. restore is a method of dt and takes the number of the version to bring back. After it runs, dt is already the new version, so you report dt itself.