Queues, Logs & the Fusion
Edit this page
Every messaging system you’ve ever used — Kafka, RabbitMQ, SQS, NATS, Pulsar — is built from two primitives:
- A queue treats messages as work: something claims each message, processes it, and it’s gone. The interesting state is per-message — claimed by whom, attempted how many times, failed why.
- A log treats messages as facts: append-only, retained, re-readable. Nothing is consumed; consumers just remember how far they’ve read — a single cursor.
Each shape is great at what the other can’t do:
| Queue (SQS, RabbitMQ) | Log (Kafka, Kinesis) | |
|---|---|---|
| Retries, backoff, dead-lettering | per-message state, yes | no — a cursor can’t say “5 failed but 6–8 are done” |
| Read history again | no — consumed means gone | yes — read from an older position |
| Many independent consumers | no — one consumer wins | yes — cursor per consumer |
| ”Handle this one bad message” | yes | no — skip it or stall the partition |
This is why you end up running both — Kafka and SQS, or Kafka and RabbitMQ — and bolting on workarounds where each falls short (Kafka’s “retry streams” are a queue impersonation; SQS’s 14-day retention is a log impersonation).
The fusion
SQLStreams’s design move is to stop choosing. Each stream keeps the two concerns in separate tables:
- The message log (
message_log_<stream id>) — immutable, append-only facts, ordered by id. Retention, fan-out, and routing live here. - Per-group delivery state (
exception_queue_<stream id>, one row per failed message per group) — claims, attempts, backoff, dead-lettering live here. - Per-group cursors (
consumer_group_cursor_<stream id>) — how far each group has read and resolved.
And the load-bearing detail: every consumer group gets both, with the split falling exactly where the cost is. The cursor sweeps the happy path — a message that succeeds never gets its own row, so a million clean messages cost one advancing integer, not a million state rows. A message that fails materializes its own delivery row and enters the full state machine — retries, backoff, dead-lettering, per group. You don’t pick “queue semantics or log semantics” per stream; the design gives you log economics for success and queue lifecycle for failure, simultaneously.
Why Postgres specifically
Three properties make Postgres an unusually good substrate for this fusion:
FOR UPDATE SKIP LOCKED— competing consumers in one SQL clause. Two instances run the same claim at the same instant and get different rows. A crashed instance needs zero recovery code; transaction rollback plus an expiring lease is the recovery.- Transactions that reach your data. The log lives next to your business tables, so producing a message in the same transaction as a business write is just… a transaction. No broker can offer this. The dual-write problem →
- It’s already there. Already deployed, already backed up, already monitored, already trusted with your most important data.
Where each system fits in this model
Once you have the queue/log lens, the whole landscape snaps into focus:
- SQS / RabbitMQ — queues. Lifecycle-rich, history-poor.
- Kafka — a log. History-rich, lifecycle-poor (a committed offset is the only per-consumer state).
- Pulsar — the closest existing fusion (per-subscription cursors plus individual message state), at the cost of running BookKeeper + ZooKeeper + brokers.
- SQLStreams — the same fusion as tables, in the database you already run.