Background managers and workers

SQLStreams requires background processes to keep things running smoothly. Retention policies need to cull data. Metrics have to be calculated. And schedules need to run on time.

To do this, SQLStreams runs background workers embedded directly in every running consumer instance.

A worker is designed to run a particular background task. We have janitors that clean up data. And managers that start, monitor, and stop other workers.

Suppose a payment application runs consumer instance A in the charge-cards group on the payments.requested stream. The same process also runs that stream’s janitor.

Messages: 1 · 2 · 3 · 4

Claims: 1–4

Cleans expired stream data

Svelte Flow

Message processing and maintenance run in the same application process.

Janitor

A stream’s janitor is the worker responsible for background cleanup. It periodically checks for eligible data and attempts to remove it.

See message retention for how message age and group progress determine which messages it can remove.

Manager

SQLStreams uses managers at two scopes:

The consumer’s manager runs the consumer group’s processing and progress tracking, along with maintenance for its stream.
The system manager looks across the installation. It can run stream maintenance, collect metrics, produce scheduled messages, and evaluate alerts, including for streams with no active consumer instance.

By default, Consume runs both.

Leader election

You might reasonably think, at this point, that each running consumer has a lot of extra process bloat and inefficiencies. While this remains true in some respects, SQLStreams uses claims and leases to elect leaders and cut down on excess running workers.

For example, say consumer instances A and B run in separate processes, both reading the payments.requested stream:

Both processes look for work to run. Their managers discover the stream’s janitor, which permits one active instance.
A’s process claims the janitor first. Postgres records a time-limited lease, and the process starts running cleanup alongside its consumer.
B’s process cannot claim the same janitor while that lease is valid. It continues processing messages without starting another janitor. Its manager periodically checks whether it can claim the work.
A’s janitor keeps its lease alive. It periodically renews the lease through heartbeats, so B’s process continues leaving cleanup to A’s.
If A’s process crashes, cleanup can move to B’s. The heartbeats stop. Once A’s lease expires, B’s manager can claim the janitor and start cleanup in its own process.

Adding application replicas therefore does not create an independent janitor for every replica. Each worker type controls how many can be active at a given time.

Maintenance needs a running process

This is the most important line here:

SQLStreams’ maintenance requires a running process.

The idea is that, for most use cases, running a producer and consumer will automagically keep SQLStreams up and running in the expected manner.

However, if you have special circumstances where running a consumer doesn’t make sense, you can also just run client.Manager().Run(ctx) in a separate process. This maintains SQLStreams even when no consumers are running.

Run the system manager — Run maintenance independently of consumer sessions, or change the automatic manager behavior.
Control stream maintenance — Inspect, suspend, or resume a stream’s janitor and vacuum worker.
Worker is not running — Investigate maintenance that should be active but is not making progress.