Kafka Streams vs Flink vs Plain Consumer: Choosing the Right Layer

Every Kafka-based pipeline eventually faces the same fork: should I use a plain KafkaConsumer, Kafka Streams, or a dedicated engine like Flink? The honest answer is that each is a different product for a different class of problem.

Plain consumer

A thread pool of KafkaConsumer instances. You control every byte. Good when:

• The work is stateless or the state lives in an external store.
• You want a single language (Java/Go/Python) and zero new infrastructure.
• Latency budget is tight and you don't want the overhead of a DSL.

You'll be the one implementing retries, DLQs, graceful shutdown, and windowing. That's fine if those rarely change.

Kafka Streams

A library that runs inside your JVM application. State is local (RocksDB) with a changelog topic as the backup. The DSL gives you joins, windowed aggregations, and exactly-once semantics for free.

StreamsBuilder b = new StreamsBuilder();
b.stream("payments", Consumed.with(Serdes.String(), paymentSerde))
 .filter((k, v) -> v.amount() > 0)
 .groupByKey()
 .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
 .aggregate(Summary::empty, Summary::fold, Materialized.as("summary-store"))
 .toStream()
 .to("payment-summary");

Best when your app is already JVM and your state fits comfortably on a single node (RocksDB scales per instance, not across).

Flink

A cluster of its own. Savepoints, event-time processing with watermarks, SQL, and real distributed state. Pick Flink when:

• State is large enough that you want it sharded and checkpointed to object storage.
• You have multiple sources and sinks beyond Kafka.
• Event-time correctness (late data, watermarks) is load-bearing to the business.

A quick heuristic

1. Stateless or externally-stateful → plain consumer.
2. Stateful, JVM, single-team → Kafka Streams.
3. Stateful, multi-source, event-time correctness → Flink.

Don't pick the fanciest one by default. Every one of these engines is someone's full-time job to operate.