fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort

Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.

src/controller/CLAUDE.md

@@ -24,6 +24,15 @@ Links against `libxce` from `src/lib/`.
 - Component-specific headers in `include/`
 - Shared types defined in `src/include/xce/` (libxce)
 
+## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)
+
+**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.
+
+- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
+- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
+- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
+- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)
+
 ## HA Protocol
 
 See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
@@ -67,6 +76,10 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
 
 **Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.
 
+**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).
+
+**Commit Durability** (acknowledged ⇒ durable on a majority — TARGET RULE): when the controller replies success to a client, the operation's entry must be `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **KNOWN GAP — bug-661, spec `ha-commit-durability` (in progress, 2026-06-18):** the commit majority currently counts the primary's *appended* (page-cache) tip via `majority_candidate_locked`, and primary fsync is async (`fsync_worker` timer), so "committed" does NOT yet imply durable-on-a-majority — an acknowledged entry can be durable on a single member and lost across failover. Until that spec lands, do not assume an acked operation survives a crash of the acking minority.
+
 ### Post-Election Safe Commit Point (Critical)
 
 After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:
@@ -89,6 +102,10 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
 - [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
 - [ ] WAL truncation on promotion only removes entries above the safe commit point
 - [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
+- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
+- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
+- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
+- [ ] (post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
 
 ## HA Design Constraints