Offsets: Where Was I?

Every event in a log has an offset: its position, 0, 1, 2 and so on. A consumer that crashes in the middle loses all its variables, so the next consumer has to know where to go on. If it starts again at offset 0, the first events are handled twice. If it guesses a later place, events are lost. A data engineer saves offsets so that a restart neither loses an event nor counts one twice.

The fix: after handling each event, the consumer saves that event's offset somewhere that survives a crash, usually a file (here a dict saved stands in for it). A new consumer reads the saved offset and starts right after it, at saved offset + 1. Before any event is handled, the saved offset is -1, so the first consumer starts at 0. The same idea with a bookmark:

pages = ["p0", "p1", "p2", "p3"]
bookmark = {"page": 1}              # p1 was the last page read
for page in pages[bookmark["page"] + 1:]:
    print(page)                     # p2, then p3

Save after handling, never before. If a crash falls between the two, the next consumer handles that one event again (the dedupe from The Same Event Twice absorbs it), but it never skips one. Kafka does this bookkeeping for you: each consumer group stores its offset per partition in Kafka itself, and a restarted consumer resumes from it.

The starting code holds 10 events, offsets 0 to 9, and run_consumer(name, crash_at), which stops when it reaches the offset crash_at. Consumer 1 crashes at offset 6. Your program must print the first five lines below. The check then starts a consumer 3, which finds nothing left, and lists the events lost or handled twice:

consumer 1 starts at offset 0
consumer 1 crashes at offset 6
consumer 2 starts at offset 6
consumer 2 is done | saved offset: 9
handled: ['e1', 'e2', 'e3', 'e4', 'e5', 'e6', 'e7', 'e8', 'e9', 'e10']
consumer 3 starts at offset 10
consumer 3 is done | saved offset: 9
lost: [] | handled twice: []

Your task: make the dict saved with offset -1, start each consumer at the saved offset + 1, run a second consumer after the crash, and save each event's offset right after handling it.

Assignment
Follow the Coding Tutorial to save the consumer's place after each event and let a new consumer go on from it.

Hint
The start is the saved offset plus 1. Save inside the loop, on the line below handled.append(...), so an offset is written only once its event is handled.