0506 — Reconciliation refunds systemic failures on CLOSE, run by the probe winner; recovery is automatic

Context. During an outage, messages that fail for the systemic reason get parked, dead-lettered, and accumulate attempts and reclaims for a failure that was never theirs. Someone has to repair that damage, and the repair must be safe to run automatically.

Decision. On CLOSE, rows whose error_class is systemic are un-marked — dead becomes ready, attempts and reclaims refunded; rows with unrelated errors stay dead. Refund-on-CLOSE (not on-trip) is what makes automatic safe, three ways: repaired rows flow straight into a dependency that just passed the probe; a wrong global trip that never closes never mutates anything; and the zombie loop (misclassified poison refunded, dies, refunded forever) self-limits, since each refund cycle requires a successful probe first. Leadership reuses what exists: on global OPEN one prober is elected via a session-level advisory lock (self-releases if the prober crashes), and the prober who observes the successful probe is the closer — prober = closer = reconciler, one leader, one lock, no new election. Persisted breaker state is just CLOSED/OPEN — “half-open” is OPEN plus currently holding the probe lock, not a stored state. Global close does not force local closes: each instance re-verifies itself, so a bad node is not “healed” by the group’s recovery. Real traffic is the probe; no user health-check hook in v1. Reconciliation must NOT run on a lone local trip — a bad-node instance records its local faults as systemic (DNS failure is indistinguishable from a dead dependency from inside), and refunding on its word would be wrong; the quorum plus the successful probe make the classification trustworthy.

Consequences. Net contract: systemic failures are retroactively not failures. Pre-trip dead rows sit visibly dead for the outage’s duration and un-die on close — dead-letter alerting should say “breaker open, N dead rows pending reconciliation” rather than paging as if they were real. For the bad-deploy case, probing fails until a human ships the fix — correct: the breaker’s job is holding, not healing. An open breaker also gates both the claim that fills the future buffered-claim pool and the dequeue-into-attempt — buffered rows are not run against a known-dead dependency, and not resolved by letting leases expire. Rejected: cursor rewind — committed is a contiguity frontier; moving it back redelivers successes. Rejected: a manual pause API — a maintenance window is “scale the group to zero”, infra’s job (accepted gap: an embedded consumer in a process doing other work cannot scale down alone). Rejected: operator-invoked repair as breaker surface — a generic requeue-dead-rows escape hatch belongs with the audit-API ideas.