myxce/.kiro/specs/ha-convergence-recovery/requirements.md

Requirements Document

Sub-spec 4 of 4 implementing the ha-consistency-invariants umbrella contract. The umbrella documents (.kiro/specs/ha-consistency-invariants/requirements.md, design.md, research.md) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.4, 2.5, 2.7, 6.1, 6.2, 6.3, 6.4, 7.2, 7.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Umbrella criteria 6.1, 6.3, 6.4, and 7.3 are verification-of-existing-behavior requirements: they pin behavior that already exists with regression tests and introduce no new production code.

Project Description (Input)

Five verified convergence gaps undermine the umbrella's single-outcome guarantee (research deep-dive §C): a demoted ex-primary never truncates its divergent WAL suffix; phantom provisional in-memory state is never evicted; restart replays the uncommitted suffix (committed_lsn is volatile, correctly per Raft, but nothing reconciles at rejoin); standby apply has no conflict detection; and uncommitted jobs leak to the scheduler during the commit window (orphan-dispatch risk). Cluster operators need cluster-wide single-outcome convergence and commit-gated egress: demotion truncates above committed_lsn and self-restarts through the existing boot recovery (provisional state discarded); a rejoining standby whose WAL tip exceeds the primary's committed boundary truncates and restarts once; the scheduler export path never sees a job whose submit entry is uncommitted; catch-up streaming continues on WAL acknowledgment without waiting for heartbeats; and quorum-loss transitions are operator-visible in logs.

Introduction

This spec delivers the convergence-and-recovery slice of the umbrella contract. After this spec, an operation that timed out at the commit barrier converges to a single cluster-wide outcome: a demoted primary discards its provisional state by truncating its uncommitted WAL suffix and restarting through boot recovery; a rejoining node whose log runs ahead of the cluster's committed boundary reconciles by truncation and a one-time restart; and the scheduler — the consumer whose actions cannot be revoked — never observes a job whose submit entry is uncommitted. Catch-up of lagging standbys becomes acknowledgment-driven rather than heartbeat-polled, and entry into and exit from the quorum-lost degraded state is recorded in the log for operators. Existing behaviors the umbrella relies on — full fan-out replication, snapshot fallback for far-behind standbys, re-inclusion of caught-up standbys, and degraded-mode status service — are pinned with regression tests.

Boundary Context

• In scope: Demotion convergence on the primary (disable persistence hooks, truncate above committed LSN, log, process self-restart, with a test-mode restart seam); rejoin reconciliation for a standby whose WAL tip exceeds the primary's committed boundary; commit-watermark egress filtering of scheduler-bound job exports (job submit-LSN tagging); acknowledgment-driven catch-up streaming continuation; quorum-loss/restore transition logging; regression tests pinning full fan-out, snapshot fallback, re-inclusion, and degraded-mode status service; the 3-node E2E convergence test extending ha-wal-replication task 5.4.
• Out of scope: Commit-index advancement and majority evaluation (ha-commit-engine — this spec consumes committed_lsn, never modifies commit.c); operation classification and reply ordering (ha-commit-barrier-rollout); promotion no-op, own-term gate consumers, and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role); entity-bearing snapshots (roadmap backlog — would later upgrade the eviction strategy); read-consistency filtering of status/API queries (documented deviation, future work); primary step-down on quorum loss (roadmap); worker rejoin job-state resync (roadmap backstop for facts lost across the restart this spec introduces); election/vote correctness (raft-library); replication wire format (ha-wal-replication).
• Adjacent expectations: Upstream, this spec relies on ha-commit-engine's committed_lsn semantics (per-LSN majority, monotonic, contiguous — complete) and on ha-wal-replication's ACK machinery and catch-up streaming. The full 3-node E2E additionally requires ha-single-leader-role to have landed (direct promotion, no transitional state); E2E tasks are explicitly marked with that dependency. Downstream, entity-bearing snapshots may upgrade demotion eviction from restart to snapshot-based rebuild, and worker-rejoin-resync recovers worker-reported facts across the restart this spec introduces.

Requirements

Requirement 1: Demotion Convergence (Truncate and Self-Restart)

Objective: As a cluster operator, I want a demoted ex-primary to discard its uncommitted, potentially divergent state and rejoin from its committed state only, so that a failed-over cluster converges to a single outcome and a demoted node never keeps serving phantom state.

Acceptance Criteria

1. When a primary observes a higher term and is demoted, the controller shall stop persisting new state changes before continuing demotion, so that no further provisional entries are produced. (umbrella 2.5)
2. When a primary is demoted, the controller shall remove from its log all entries above its committed LSN. (umbrella 2.5)
3. When demotion truncation completes, the controller shall restart its own process so that in-memory state is rebuilt through boot recovery from the persisted committed state only, discarding all provisional in-memory state. (umbrella 2.5, 2.7)
4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. (umbrella 2.5; locked constraint) — SUPERSEDED (2026-07-10): "terminate fatally" (abort()) violated the project-wide never-self-terminate invariant (CLAUDE.md "Process Lifecycle") on an operational condition (execv() can fail on ENOMEM/EMFILE/a permissions change mid-upgrade — none of these are programming-invariant violations). The safety property this criterion actually protects — a divergent ex-primary never serves authoritatively — is preserved without dying: on execv() failure the controller now sets a sticky, never-live-cleared ha->self_restart_pending flag (checked unconditionally by raft_cb_become_leader / promotion_fence_timeout_fn alongside apply_divergent, ha.c) and retries the re-exec periodically via delaymgr (ha_self_restart_enter_fence_and_retry, ha_convergence.c). The node stays alive, permanently non-promotable, and self-heals the moment a retry succeeds — never requiring manual admin recovery. See ha-async-io.md's own SUPERSEDES (2026-06-25) note for the precedent of revising a locked decision when a stronger mechanism (loud-fence-and-stay-alive) was established after the original spec closed.
5. When a demotion occurs, the controller shall log the event with the truncation boundary (committed LSN and prior log tip) before restarting, so that operators can observe the discarded range. (umbrella 2.5)
6. Where test mode is enabled, the controller shall invoke a substitutable restart seam in place of process re-execution. (locked constraint — testability)
7. The controller shall be able to reconstruct its in-memory state from its persisted committed state, discarding any provisional state not present in the committed log. (umbrella 2.7)

Requirement 2: Rejoin Reconciliation and Single-Outcome Convergence

Objective: As a cluster operator, I want a rejoining node whose log runs ahead of the cluster's committed boundary to reconcile by truncation and a one-time restart, so that every member's state converges to the same outcome even after a crash that skipped demotion convergence.

Acceptance Criteria

1. When a standby learns the primary's committed boundary at (re)join and its local log tip exceeds that boundary, the standby shall truncate its log above that boundary and restart its process once. (umbrella 2.5)
2. The standby shall not remove log entries at or below its own locally known committed LSN during rejoin reconciliation. (umbrella 2.5 — safety clamp; commit index never regresses)
3. While a standby's local log tip does not exceed the primary's committed boundary at rejoin, the standby shall join without truncation or restart. (umbrella 2.5 — no spurious restarts)
4. If an operation was reported as failed due to commit timeout, the cluster shall converge on a single outcome for its log entry — committed on all members or discarded on all members — the primary shall never retroactively report that operation as successful to the original requester, and each member's in-memory state shall converge to that same outcome. (umbrella 2.5 — verified end-to-end)

Requirement 3: Commit-Gated Scheduler Egress

Objective: As a cluster operator, I want the scheduler to never observe a job whose submit entry is uncommitted, so that no job can be dispatched whose existence the post-failover cluster may never have heard of (orphan-dispatch prevention).

Acceptance Criteria

1. While a job's submit entry is not yet committed, the controller shall exclude that job from data exports that drive scheduling and dispatch decisions. (umbrella 2.4)
2. When a job's submit entry becomes committed, the controller shall include that job in subsequent scheduler-bound exports. (umbrella 2.4)
3. When a job is created or restored from the persisted log, the controller shall associate the job with the log position of its submit entry so its commit status is decidable at export time. (umbrella 2.4)
4. While a job's submit entry is not yet committed, the controller shall continue to include the job in status and API query responses; read paths are intentionally not commit-filtered. (umbrella 2.4 boundary — documented deviation; read consistency is out of scope)

Requirement 4: Full Fan-Out Replication (Verification of Existing Behavior)

Objective: As a cluster operator, I want the primary to replicate to every reachable standby, so that the cluster trends toward full consistency — pinned by regression test, since the umbrella relies on this existing behavior.

Acceptance Criteria

1. While holding leadership, the primary shall replicate new log entries to all reachable standbys, not only to a majority subset. (umbrella 6.1 — existing behavior, regression test only)

Requirement 5: Acknowledgment-Driven Catch-Up Continuation

Objective: As a cluster operator, I want a lagging standby brought current as fast as it can acknowledge, so that catch-up progress is not throttled to the heartbeat cadence.

Acceptance Criteria

1. When the primary receives a WAL acknowledgment from a standby that remains behind the replication target, the primary shall continue streaming the missing entries to that standby without waiting for the next heartbeat exchange. (umbrella 6.2)
2. While a catch-up continuation for a standby is already in flight, the primary shall not start a duplicate continuation for that standby. (umbrella 6.2 — bounded continuation, no streaming storms)
3. The heartbeat-driven replication evaluation shall remain in place as the fallback initiator of catch-up and as the initiator of snapshot-transfer decisions. (umbrella 6.2, 6.3 — locked design boundary)

Requirement 6: Snapshot Fallback (Verification of Existing Behavior)

Objective: As a cluster operator, I want a standby that has fallen behind the oldest retained log entry brought current via snapshot, so that log retention limits never strand a member — pinned by regression test.

Acceptance Criteria

1. If a standby is behind the oldest retained log entry, the primary shall bring it current by transferring a snapshot followed by subsequent log entries. (umbrella 6.3 — existing behavior, regression test only)

Requirement 7: Catch-Up Re-Inclusion (Verification of Existing Behavior)

Objective: As a cluster operator, I want a standby that has caught up re-included in live replication, so that it counts toward future commits — pinned by regression test.

Acceptance Criteria

1. When a lagging standby catches up to the primary's replication boundary, the primary shall re-include it in live replication. (umbrella 6.4 — existing re-inclusion mechanism, regression test; the acknowledgment-driven continuation of Requirement 5 provides the production trigger at parity)

Requirement 8: Quorum Transition Observability

Objective: As a cluster operator, I want entry into and exit from the quorum-lost degraded state recorded in the log, so that I can observe and correlate degraded periods without instrumentation.

Acceptance Criteria

1. When the primary loses write quorum, the controller shall record the transition in its log, including the connected and required member counts. (umbrella 7.2)
2. When write quorum is restored, the controller shall record the transition in its log. (umbrella 7.2)
3. While quorum remains lost, the controller shall not repeatedly log the loss condition for every rejected operation; the transition shall be logged once per state change. (umbrella 7.2 + umbrella error strategy — no log storms)

Requirement 9: Degraded-Mode Service Continuity (Verification of Existing Behavior)

Objective: As a cluster operator, I want a quorum-lost primary to keep answering status queries and replication traffic, so that the degraded cluster remains observable and can heal itself — pinned by regression test.

Acceptance Criteria

1. While quorum is lost, the primary shall continue to respond to status queries. (umbrella 7.3 — existing dispatch-gate allowlist, regression test only)
2. While quorum is lost, the primary shall continue to process replication-related traffic from returning members. (umbrella 7.3 — existing dispatch-gate allowlist, regression test only)

Umbrella Traceability

This spec	Umbrella ID	Notes
1.1–1.7	2.5, 2.7	Demotion convergence: hook disable, truncate above committed_lsn, self-restart via boot recovery; FATAL on restart failure; XCE_TESTING seam (locked decisions)
2.1–2.3	2.5	Rejoin reconciliation — generalizes the existing RDHUP truncation pattern to the rejoin/hello path; restart at most once
2.4	2.5	Single-outcome convergence incl. in-memory state; never retroactive success — E2E-verified
3.1–3.3	2.4	Provisional-state egress gate: watermark filter on the submit entry's LSN, scheduler export only (locked decision)
3.4	2.4 boundary	Status/API queries intentionally unfiltered — documented deviation (read consistency out of scope)
4.1	6.1	Verification of existing full fan-out broadcast — regression test, no new code
5.1–5.3	6.2	ACK-driven catch-up continuation; heartbeat path remains fallback + snapshot initiator
6.1	6.3	Verification of existing snapshot fallback — regression test, no new code
7.1	6.4	Verification of existing re-inclusion — regression test; Requirement 5 supplies the parity trigger
8.1–8.3	7.2	Quorum-loss/restore WARN logging, once per transition
9.1, 9.2	7.3	Verification of existing degraded-mode allowlist — regression test, no new code