A regular consumer reads messages and handles each one on its own. But some tasks don't fit that model: count orders over the last five minutes, join a stream of payments with a stream of orders, keep a running balance per account. That is stream processing — and Kafka Streams turns it into an ordinary Java library: no separate cluster, right inside your service.

What it is

Kafka Streams is a library shipped with Apache Kafka: add the dependency, describe a topology (which topics to read, how to transform, where to write) — and the application runs as a normal Spring Boot service. Compare that with Flink or Spark Streaming, which need their own managed cluster. Scaling works exactly like consumer groups: start a second instance and the partitions get split between them.

StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders", Consumed.with(Serdes.String(), orderSerde))
    .filter((key, order) -> order.amount().compareTo(BigDecimal.ZERO) > 0)
    .groupByKey()
    .count()
    .toStream()
    .to("order-counts");

KStream and KTable: event versus state

The central pair of concepts:

  • KStream — a stream of events: every message is an independent fact ("order created", "payment received"). A new message cancels nothing.
  • KTable — a table of state: messages with the same key overwrite each other, and the table holds the latest value ("current account balance", "up-to-date profile").

The same data can be read either way — it's a question of semantics: a history of changes or the current state. A compacted topic is the natural companion of a KTable.

Stateful operations: windows, joins, state stores

The real power of Kafka Streams is stateful processing:

  • Windowed aggregations: "orders per 5 minutes" — the window slides over time, counters live in a state store (embedded RocksDB on the instance's disk).
  • Stream joins: match a payment with its order by key within a window — the classic "waiting for the second event of a pair" problem.
  • Fault-tolerant state: every state store is mirrored to a changelog topic in Kafka; if an instance dies, a new one rebuilds the state from the topic and carries on.

This is exactly what is long and error-prone to hand-build on top of a consumer: your own storage, your own recovery logic, your own window deadlines.

Exactly-once

Kafka Streams supports processing.guarantee=exactly_once_v2: reading, processing, writing results and committing offsets are wrapped into a Kafka transaction. For topic-to-topic pipelines this is honest exactly-once without hand-rolled idempotency. Know the boundary: the guarantee holds inside the Kafka world — an external REST call or a database write is not covered by the transaction; there, idempotency remains your job.

When to use it — and when not

Take it when: time-windowed aggregations, stream joins, maintaining derived state, materializing a "current view" out of events — and the data is already in Kafka.

Skip it when:

  • processing is "read a message, call a service, write to a database": a plain consumer is enough, Streams adds complexity without payoff;
  • you need heavy computation on a large cluster, SQL over streams, ML pipelines — that's Flink territory;
  • the data isn't in Kafka: Streams reads and writes Kafka topics only.

In short

  • Kafka Streams is a stream-processing library: the topology lives inside your service, no separate cluster; scaling follows partitions like consumer groups.
  • KStream is a flow of independent events, KTable is "latest value per key" state; the choice is about data semantics.
  • Stateful operations (windows, joins, aggregations) keep state in local RocksDB, recovered from a changelog topic.
  • exactly_once_v2 gives true exactly-once within Kafka; external calls and databases still need idempotency.
  • Simple "message → action" processing — plain consumer; heavy analytics — Flink; derived state and windows over Kafka data — Streams.
  • Kafka fundamentals — partitions, offsets, consumer groups: the foundation Streams stands on.
  • Kafka in production — idempotency, rebalances, monitoring.
  • Sync or async — whether you need events at all.