fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort

Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.

.kiro/specs/ha-commit-durability/design.md

@@ -162,12 +162,13 @@ graph TD
     Floor --> Eligible[eligible to lead]
     Truncated --> CanCatchUp{healthy peer reachable}
     CanCatchUp -->|yes| Fence[fence from promotion until caught up]
-    CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
+    CanCatchUp -->|no| LoudFence[LOUD PERMANENT FENCE: non-promotable, STAYS ALIVE, keeps retrying catch-up — NEVER self-exit]
+    LoudFence --> Caught
     Fence --> Caught{caught up past last committed boundary}
     Caught -->|yes| Eligible
 ```
 
-Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
+Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state, the controller logs a LOUD operator-visible alert and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up (Req 2.4) — it refuses to serve divergent state by refusing leadership, NOT by terminating. **A controller MUST NEVER self-exit** (`exit`/`abort`) as a designed response to an operational condition (user invariant, 2026-06-18): a self-exited controller needs manual admin recovery and degrades quorum meanwhile, whereas one that stays alive self-heals when a peer returns (Req 2.6). A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
 
 ## Requirements Traceability
 
@@ -182,7 +183,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
 | 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
 | 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
 | 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
-| 2.4 | fail-loud when unrecoverable | RecoveryIntegrity | FATAL policy | Recovery fence |
+| 2.4 | loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | RecoveryIntegrity | loud-fence policy + sticky `recovery_unrecoverable` flag | Recovery fence |
 | 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
 | 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
 | 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
@@ -194,7 +195,7 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
 | Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
 |-----------|-------|--------|--------------|--------------------------|-----------|
 | DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
-| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; FATAL policy | 2.1, 2.4 | replay_wal (P0) | State |
+| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; loud permanent non-promotable fence policy (stays alive, NEVER self-exit) | 2.1, 2.4 | replay_wal (P0) | State |
 | PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
 | FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
 | DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
@@ -235,14 +236,14 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
 
 | Field | Detail |
 |-------|--------|
-| Intent | Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable |
+| Intent | Distinguish clean log end from corruption; re-derive durable floor; loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) |
 | Requirements | 2.1, 2.4 |
 
 **Responsibilities & Constraints**
 - `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
 - On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
 - The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
-- FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.
+- Loud-fence policy (NEVER self-exit, user invariant 2026-06-18): if a recovery-truncated / behind member cannot reach a healthy peer to catch up, the controller logs a LOUD operator-visible alert (truncation boundary vs cluster HWM) via `persist_recover_loud_fence_if_unrecoverable`, sets a sticky `recovery_unrecoverable` flag, and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up. It refuses to serve divergent state by refusing leadership — it does NOT `exit`/`abort`. The flag clears (and the fence releases → eligible, Req 2.6) automatically when a peer returns and catch-up reaches a committed-consistent state.
 
 **Contracts**: State [x]
 
@@ -254,15 +255,15 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
 **Implementation Notes**
 - Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
 - Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
-- Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).
+- Risks: must not loud-fence on a legitimately empty first-boot WAL (no entries ≠ corruption).
 
 ### Boot Policy Decision Table (task 3.2)
 
-> **Status:** DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task **3.3** (the FATAL branch) and task **4.3** (the promotion fence + release) implement. The skeleton tests in `test_commit_durability.c` (one per row, `TEST_IGNORE`-skipped pending 3.3/4.3) encode each row's expected outcome.
+> **Status:** DECISION recorded (task 3.2); **terminal action corrected by the never-self-exit rework (2026-06-18).** The original table's `FATAL` outcome (a self-`abort`) was REPLACED: the unrecoverable terminal is now a **LOUD PERMANENT FENCE** — non-promotable, **stays alive**, keeps retrying catch-up. This table is the contract that task **3.3** (the loud-fence terminal) and task **4.3** (the promotion fence + release) implement.
 
-**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates **loudly (FATAL)** only on a **bounded** failure to reach that state — i.e. it cannot recover locally **and** cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state). 
+**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. If it cannot reach that state (no healthy peer reachable / cannot reach a committed-consistent state), it enters a **LOUD PERMANENT FENCE**: it logs a loud, operator-visible alert (truncation boundary vs cluster HWM), holds itself **permanently non-promotable**, and **stays alive while continuing to retry catch-up** — it **NEVER self-terminates**. The fence releases (→ ELIGIBLE) automatically once catch-up reaches a committed-consistent state.
 
-**Invariant (authoritative):** the fence is **never a silent indefinite wait**. A truncated/behind member always resolves to exactly one of two terminal outcomes: it **catches up → ELIGIBLE**, or it **FATALs (fail-loud)** on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.
+**Invariant (authoritative):** A truncated/behind member always resolves to exactly one of two outcomes: it **catches up → ELIGIBLE**, or it enters a **LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit)**. It is never silently divergent (it refuses to lead), and the process never self-`exit`s/`abort`s as a designed response to this operational condition (user invariant, 2026-06-18) — a self-exited controller needs manual admin recovery and degrades quorum, whereas one that stays alive self-heals when a peer returns (Req 2.6). Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never the unrecoverable terminal on a legitimate sentinel.
 
 **Inputs (evaluated at boot / promotion-eligibility):**
 - **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
@@ -273,21 +274,21 @@ Key decisions: corruption is never silently accepted (Req 2.1). A truncated memb
 **Outcomes:**
 - **ELIGIBLE** — may promote (no fence).
 - **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
-- **FATAL** — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).
+- **LOUD PERMANENT FENCE** — log a loud operator-visible alert (truncation/boundary), set the sticky `recovery_unrecoverable` flag, and hold the member permanently non-promotable while it **stays alive** and keeps retrying catch-up — **NEVER self-exit**; releases → ELIGIBLE on catch-up (Req 2.4, 2.6).
 
 | # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
 |---|---------|----------------|--------------|-----------------|---------|-----------------|
 | 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
 | 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
-| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6) |
-| 4 | TRUNCATED | below | no | fails (bounded window elapses / no committed-consistent state reachable) | **FATAL** | Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4) |
-| 4b | CLEAN | below | no | fails (bounded window elapses, no peer to advance from) | **FATAL** | Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2) |
-| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never FATAL** on a legitimate sentinel. (Req 2.1, 2.2) |
+| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never the unrecoverable terminal when a peer is reachable. (Req 2.2, 2.6) |
+| 4 | TRUNCATED | below | no | fails (no committed-consistent state reachable) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Cannot recover locally AND no peer to catch up from ⇒ log loudly + hold non-promotable + alive; releases → ELIGIBLE when a peer returns and it catches up. NEVER `exit`/`abort`. (Req 2.4, 2.6) |
+| 4b | CLEAN | below | no | fails (no peer to advance from) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Behind the HWM and no healthy peer to catch up from — same non-terminating loud-fence as row 4; NEVER self-exit. (Req 2.4, 2.2, 2.6) |
+| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never the unrecoverable terminal** on a legitimate sentinel. (Req 2.1, 2.2) |
 | 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
 
 **Notes on the dimensions:**
 - A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
-- "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be **finite** so the fence cannot become a silent indefinite wait.
+- the catch-up retry is **indefinite** (the member stays alive and keeps trying) and **loud** (a periodic operator-visible re-alert, controlled by `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`); it is never silently divergent because the member refuses leadership the entire time. The previous "bounded window before FATAL" deadline was REMOVED by the never-self-exit rework — there is no deadline-driven termination.
 - The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
 
 ### Election — PromotionFence + FreshnessFloor
@@ -458,7 +459,7 @@ Walking the relevant items under the pinned formula:
 
 ### Error Strategy
 - **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
-- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
+- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; when catch-up is impossible (no peer) the node enters a LOUD PERMANENT FENCE — non-promotable, **stays alive**, keeps retrying catch-up, NEVER self-exits (Req 2.4, 2.6).
 - **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
 
 ### Monitoring
@@ -489,7 +490,7 @@ Walking the relevant items under the pinned formula:
 
 ## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
 - **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
-- **FATAL-vs-fence decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). **RESOLVED (task 3.2):** see the **Boot Policy Decision Table (task 3.2)** above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
+- **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it.
 - **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
-- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
+- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
 - **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).

.kiro/specs/ha-commit-durability/requirements.md

@@ -0,0 +1,61 @@
+# Requirements Document
+
+## Project Description (Input)
+
+**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover.
+
+**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority.
+
+**What should change (chosen scope — durable commit + promotion fence):**
+1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted.
+2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations.
+
+The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision.
+
+## Introduction
+
+This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing.
+
+## Boundary Context
+
+- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage.
+- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`).
+- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation.
+
+## Requirements
+
+### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence
+
+**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover.
+
+#### Acceptance Criteria
+
+1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members.
+2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers.
+3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment.
+4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester.
+5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster.
+6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover.
+
+### Requirement 2: Recovery Integrity and Leadership Fence
+
+**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations.
+
+#### Acceptance Criteria
+
+1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete.
+2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member.
+3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations.
+4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.)
+5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded.
+6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership.
+
+### Requirement 3: Deterministic Durability Verification
+
+**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing.
+
+#### Acceptance Criteria
+
+1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing.
+2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member.
+3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract.

.kiro/specs/ha-commit-durability/tasks.md

@@ -143,3 +143,4 @@
 - 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
 - 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
 - 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).
+- REWORK (2026-06-18, user invariant — a controller MUST NEVER self-terminate): tasks 3.3/4.3/4.4 had implemented Req 2.4 "fail loud" as a SELF-ABORT (`persist_recover_fatal_if_unrecoverable → tlog_exit(); abort()` + `persist_fatal_hook`). That violated the invariant (a self-exited controller needs MANUAL admin recovery + leaves quorum degraded; a live retrying node self-heals when a peer returns) and was the true root cause of the 5.2 soak quorum collapse. CORRECTED: removed the abort/exit terminal entirely. New: `persist_recover_loud_fence_if_unrecoverable(persist_t*)` (NO abort) logs a loud operator-visible ERROR (truncation boundary vs cluster HWM) + sets sticky `_Atomic bool recovery_unrecoverable` (accessors `persist_recovery_unrecoverable`/`persist_clear_recovery_unrecoverable`); the member STAYS ALIVE, permanently non-promotable/fenced (`in_catch_up` + the sticky flag both consulted by the promotion gate), and KEEPS RETRYING catch-up. Decision enum `RECOVER_DECISION_FATAL` → `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). 4.4 timer no longer aborts — it re-emits the loud operator alert + re-arms (operator visibility), never exits. Flag + fence CLEAR automatically on catch-up (not truncated AND durable_floor ≥ HWM) → re-eligible (Req 2.6), fully automatic. `persist_fatal_hook` removed. Verified: grep/nm show NO abort/exit/tlog_exit on the recovery path; ASAN suite :FAIL:==0; reworked tests (row4/4b → loud_fence, unrecoverable-stays-alive-loud, released-on-catchup, cancel-stops-realert) PASS. Spec updated: requirements.md Req 2.4 + design.md (decision table rows 4/4b, RecoveryIntegrity, traceability, FATAL-vs-fence open question) reworded "FATAL/terminate" → "LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — NEVER self-exit)". PROJECT-WIDE GUIDANCE added: `src/controller/CLAUDE.md` §"Process Lifecycle (DESIGN INVARIANT)" + `.kiro/steering/ha-async-io.md` — controllers never self-terminate. (`ha_self_restart` execv-restart in convergence/demotion is a SEPARATE primitive, untouched.)