Stream processing

A stream processor reads a log forever, keeps state, and emits an updated answer. Flink, Spark Structured Streaming, Kafka Streams, and a few cloud cousins do this. A pool of workers pulling jobs does not, even if the jobs happen to come from Kafka.

Why it exists

You can write a consumer that increments a counter. It will work until a restart replays an hour of events and the counter doubles, or a batch of late events falls into the next minute, or you need the count per ad per city and the state no longer fits in a hash map you manage yourself. A stream processor checkpoints state and offsets together, so a restart continues rather than redoing the side effect blindly. It also understands time.

Time

Processing time is when your job saw the event. Event time is when it happened, carried in the payload. Windows (tumbling five minutes, sliding one hour, a session that ends after inactivity) should usually be event-time windows for anything a human will call a metric.

A watermark is the job’s hypothesis that events earlier than T have already arrived. When the watermark passes the end of a window, the window closes and emits. Events later than the watermark are late. You either drop them, update the result (a retractable sink), or put them in a side output. Say which. A source that stops advancing watermarks freezes every window downstream, which is a production incident mode worth naming.

State and sinks

Keyed state (the count for this ad, the set of users in this window) lives with the key’s partition and is checkpointed. The sink that receives the aggregate should be idempotent, because some sinks sit outside the checkpoint transaction. Writing “hour=11, ad=9, clicks=1042” as an upsert is safe. Blindly incrementing a row is not, unless the checkpoint and the write are tied together.

When to skip it

If the interview is “send a notification when an order is placed,” draw a queue and a worker. If the interview is “top ads in the last five minutes” or “dedupe clicks across a day,” draw a log and a stream job, and spend your time on the window, the late-event rule, and the hot key.