Architecture

SQLStreams runs inside your Go processes against Postgres. Producers write messages, consumers claim and handle them, and a system manager runs maintenance. Consumers run that manager by default; a deployment that only produces needs a standalone manager.

The flow of one message

For orders.created (stream id 1), the email-receipts group (id 7) handles message 101:

1. Produce: the application inserts the message into sqlstreams.message_log_1. With ProduceFunc, the business write runs in the same transaction. Commit makes both visible together.
2. Claim: a consumer advances group 7’s claimed cursor over a range containing 101 and takes a range lease. The transaction commits before the handler starts. Version, routing, and compaction rules determine which messages reach the handler.
3. Resolve: a first delivery that succeeds normally writes no exception row. An error, requested delay, or key wait records state in sqlstreams.exception_queue_1. The cursor advancer moves committed past resolved ranges; exception delivery continues independently.
4. Recover: if the process crashes before resolving its range, another consumer can reclaim the range after lease expiry. Message 101 may run again even if its handler already succeeded.

Message Lifecycle describes the states and Transactional Produce shows the atomic write.

Two kinds of table

Shared tables describe resources and maintenance. Every stream also owns physical tables named with its id, so stream 1 uses message_log_1, exception_queue_1, and the rest of that family.

Scope	Tables	Purpose
Shared	system_config, stream_config, consumer_group_config	resource declarations
Shared	worker_config, worker_instance, schedule_config, schedule_cursor	maintenance and schedule state
Stream	message_log_1	retained payloads, versions, routing keys, and message keys
Stream	consumer_group_cursor_1, claim_lease_1	group progress and range claims
Stream	exception_queue_1, delivery_log_1	retries, delays, key waits, and attempt history
Stream	binding_config_1	group routing patterns
Stream	message_key_lease_1, compaction_head_1, idempotency_key_1	per-key concurrency, current compacted values, and produce deduplication

Table Design lists the full family, declaration history, and relationships. A per-stream table carries no stream_id column because its name supplies the scope.

The first stream registration creates the shared tables if absent, then the stream’s family. It validates the stream name and config before creating resources. A later database failure can still leave partial registration progress. System().Register explicitly declares system configuration; sqlstreams system register creates the shared baseline from the CLI.

Separate table families isolate retention decisions: a lagging group on one stream cannot prevent a different stream’s partitions from expiring. They also separate table and index churn, while still sharing database CPU, memory, and I/O. Destroying a stream drops its family of tables.

The maintenance fleet

Workers are registered in worker_config and claimed by running managers. Replicas coordinate through expiring worker claims. A live consumer’s manager can maintain the deployment, and ClientConfig.DisableManager lets a process opt out when another process carries that responsibility.

Worker	Maintains
stream_janitor	expired partitions, idempotency keys, and message-key leases
stream_vacuum	optional vacuum and analyze of the stream’s idempotency-key table; disabled by default
cursor_advancer	committed cursors behind resolved ranges
consumer_group_janitor	superseded binding declarations
schedule_producer	requests for due schedules
metrics_collector	snapshots of cursor, exception, and lease state
alert.* checks	operator warnings on __system.alerts

Stream maintenance

Each stream declares a janitor and a vacuum worker. The janitor starts active; vacuum starts suspended. StreamConfig.Janitor and StreamConfig.Vacuum set their timing and limits. The stream’s maintenance handles suspend and unsuspend them. Registration updates settings while preserving these operational choices.

For example, unsuspending vacuum on orders permits a manager to claim one instance. Registering orders again leaves vacuum active. Suspending it prevents new claims; its next successful heartbeat cancels the running request, and the instance releases its claim after stopping.

Vacuum runs VACUUM (ANALYZE) on the stream’s idempotency-key table using one connection from its client’s pool. The janitor deletes expired rows independently. Keep PostgreSQL autovacuum enabled; scheduled vacuum adds maintenance I/O and makes deleted space reusable within the table.

The first vacuum request starts immediately after the instance is claimed. Subsequent requests wait PollRate, with 10% jitter, after completion. Failures use the existing worker backoff. Requests do not accumulate while a previous request is running.

Costs and timing

Claims poll with ClaimPollRate (default 500ms). There is no LISTEN/NOTIFY wake-up path. A shorter interval increases idle database queries; a due retry also waits for a poll and any key-ordering checks.

Successful first deliveries share the cost of range claims and cursor advancement. Exceptions add per-message inserts, updates, and audit writes. Enabling success audit logging also adds per-message writes. Consumer Tuning covers queue, batch, and concurrency tradeoffs.

Retention uses id-range partitions (PartitionSize, default 1_000_000): whole expired partitions can be dropped without deleting each row individually. A slow group can delay that cleanup, as described in retention.

Database capacity remains the throughput limit. This page makes no throughput claim; recorded benchmark work is tracked on the roadmap.