Data Engineering/Processing
Batch vs Streaming Processing
Two fundamentally different ways to move and process data — running on a schedule versus reacting to events as they happen — and how to decide which one a problem needs.
Data processing splits into two broad models: batch, where data is collected and processed in chunks on a schedule, and streaming, where data is processed continuously as individual events arrive. Almost every pipeline design decision downstream — orchestration, storage, latency, cost — flows from this choice.
Batch processing
Batch jobs run periodically (hourly, daily) over a bounded set of data — for example, 'yesterday's orders.' Tools like Spark, dbt, and plain SQL scripts running on a schedule are classic batch tools. Batch is simpler to reason about, easier to reprocess (just re-run over the same data), and cheaper per unit of data, at the cost of latency: results are only as fresh as the last run.
Streaming processing
Streaming systems (Kafka, Kinesis, Flink, Spark Structured Streaming) process an unbounded sequence of events as they arrive, often within milliseconds to seconds. This enables use cases batch can't: fraud detection, real-time personalization, live operational dashboards. The cost is complexity — you now have to reason about out-of-order events, exactly-once processing, and state that lives across a continuously running job instead of a single bounded run.
- Batch: bounded data, scheduled runs, higher latency, simpler failure recovery
- Streaming: unbounded data, continuous processing, low latency, more operational complexity
- Micro-batching (e.g. short, frequent batch runs) is a common middle ground
Watch out — Streaming is a cost, not a default
Teams sometimes reach for streaming because it feels more modern, even when the business need is a report refreshed once a day. Streaming infrastructure carries real operational overhead — only take it on when latency requirements actually demand it.
A practical decision rule
Ask: does a person or system need this data within seconds, or is 'as of this morning' good enough? Most analytics and reporting use cases are genuinely fine with batch, refreshed hourly or daily. Reach for streaming when the value of the data decays within minutes — operational alerting, live pricing, real-time fraud checks.