SQLStreams vs. Kafka

Kafka is a magnificent piece of engineering for a problem most teams don’t have. The pattern we see constantly: a team adopts Kafka for semantics (retained messages, fan-out, ordering) and then pays every month for scale they never use — in brokers, partition math, consumer-group rebalances, and a dedicated platform engineer who is “the Kafka person.”

SQLStreams’s claim: you can have those semantics on Postgres, and you probably should — until the day your numbers say otherwise.

Feature by feature

Scored on shipped behavior; proposed capabilities are labeled, never checkmarked.

	Kafka	SQLStreams
Retained, append-only log	yes	yes
Independent consumer groups	yes	yes
New consumer reads history	yes	yes — a new group’s default
Per-key ordering	yes — partitions (count fixed up front)	per-key order, or exclusivity with latest-wins (opt-in per message)
Per-message retry/backoff	no — retry streams, hand-built	yes — native
Dead-lettering	DIY stream + tooling	native, per group, queryable
Routing by key bindings	no — consumers filter client-side	yes — native
Producing in the same transaction as your DB writes	structurally impossible	the headline feature
Ops footprint	brokers + KRaft/ZK + partitions + rebalancing	the Postgres you already run
Inspect state	CLI tools, JMX, vendor UI	psql
Throughput ceiling	effectively unbounded	bounded by your Postgres — published numbers wait on the benchmark pipeline

The two pains Kafka can’t shed

The operational tax. A Kafka deployment is a distributed system with its own failure modes: partition leadership, ISR shrinkage, rebalance storms, consumer-group offset mysteries. Managed Kafka (MSK, Confluent) moves the tax from ops time to invoice, and the semantic complexity — partition counts, rebalancing, retry-topic plumbing — stays in your code.

The transaction gap. Kafka lives outside your database, so every DB-write-plus-send is a dual write. The standard fix is an outbox table in Postgres plus Debezium relaying to Kafka — at which point you are running Postgres, Kafka, Kafka Connect, and Debezium to deliver a message that started and could have ended as a row.

When you genuinely need Kafka

We’d rather be the platform that told you the truth than the one you ripped out angrily. Reach for Kafka (or Redpanda) when:

• Sustained firehose throughput at a scale your measurements show Postgres can’t hold — log-structured storage and sequential disk I/O genuinely beat a B-tree database at the top end. (Where SQLStreams’s own ceiling sits is exactly what the benchmark pipeline exists to publish; until then we won’t name a number in either direction.)
• Massive retention — months or years of full-firehose history that shouldn’t live inside your OLTP database.
• The ecosystem is the point — Kafka Connect, Flink, ksqlDB; if your architecture is stream processing, Kafka is its native soil.
• Cross-org pipelines where “Kafka topic” is the contract.

The decision in one question

Are you buying Kafka for its semantics, or for its scale?

If it’s semantics — retention, fan-out, per-message state, durability — SQLStreams gives you those, plus the failure lifecycle and the transaction guarantee Kafka can’t offer, with zero new infrastructure. If it’s scale, and the numbers are real: use Kafka, with our blessing.