Start learning free

LESSON 71 OF 107 · DATA ENGINEERING: STREAMING BASICS

The Same Answer as the Batch Job

Practice

A stream counts events as they arrive. A batch job waits and computes the same numbers over the full list, in one pass. Running both and checking that they agree is a reconciliation. A data engineer runs one so that a stream that drops, doubles or misses events is caught before its numbers reach a dashboard.

The batch job has it easy, because it sees every event at once. To drop repeats, it stores each event in a dict under its event_id: a dict holds each key once, so a second copy just replaces the first. unique.values() then gives the stored events, one per id, and the batch adds each amount to its minute, event_time[:16], as the stream does. Arrival order doesn't change a sum. Two dicts are equal with == when they hold the same keys with the same values, in any order:

views = {}
views["v1"] = "/home"
views["v1"] = "/home"      # the same key: replaced, not added
print(len(views))           # 1
print({"a": 1, "b": 2} == {"b": 2, "a": 1})   # True

The starting code holds 11 events (e5 arrives twice, e6 out of order, and e10, from 10:00:50, arrives last), the finished stream_revenue, and a batch_revenue that returns its revenue still empty. Your program must print the first six lines below. The check adds two more: the stream as it stood before e10 arrived doesn't match the batch, and the batch over the events in reverse order still does.

2026-05-01 10:00 | stream: 92 | batch: 92
2026-05-01 10:01 | stream: 60 | batch: 60
2026-05-01 10:02 | stream: 72 | batch: 72
2026-05-01 10:03 | stream: 55 | batch: 55
2026-05-01 10:04 | stream: 30 | batch: 30
same answer: True
stream before the late e10 arrived | same answer: False
batch over the events in reverse order | same answer: True

When the two differ at work, look for duplicates (the stream is too high) or late events (the stream is too low). Kafka keeps its log for days, so a batch job can re-read the very same events from offset 0 and recount them.

Your task: fill in batch_revenue: keep one copy of each event in a dict unique, add each unique event's amount to its minute's revenue, and print whether stream equals batch.


Assignment
Follow the Coding Tutorial to build the batch job's revenue per minute and compare it with the stream's.


Hint
Storing each event under its event_id keeps one copy per id. The two lines inside the second loop are the same as in stream_revenue, and stream == batch gives True or False.


STEP 1 OF 6
    Ctrl + Enter to check