fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.

.kiro/specs/config-upgrade-snapshot/requirements.md

@@ -15,7 +15,7 @@ This feature makes a configuration upgrade establish a **full, entity-bearing st
   - The Raft protocol (term/log/commit) — owned by the `Raft TLA+ Conformance` library series.
   - Controller voting-set / membership changes — `ha-config-bootstrap`.
 - **Adjacent expectations**:
-  - **HARD dependency on `entity-bearing-snapshots`**: today snapshots are a 16-byte LSN sentinel carrying no entity state; the COW machinery (`xzarray_freeze`/`thaw`) is built but dormant. That work must land first.
+  - **Dependency on entity-bearing snapshots — SATISFIED (2026-07-09) by `ha-entity-snapshots` (21/21 tasks, fully approved).** That spec landed `write_snapshot`/`load_snapshot` (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time, restoring before advancing position — not the COW `xzarray_freeze`/`thaw` mechanism this line originally anticipated (that machinery remains dormant; `ha-entity-snapshots` took a different, already-shipped implementation path). This spec no longer needs to wait; verify `write_snapshot`'s actual on-disk entity format (persist.c, `ha-entity-snapshots` design.md) before writing this spec's own design so it builds on the real interface, not the COW one originally assumed here.
   - Anchors on the `(term, lsn)` configuration-change marker from `ha-term-safety`.
   - Reuses the existing `snapshot_transfer_s` snapshot-install transport (chunked, `(term, lsn)`-anchored, hash-verified, install-before-activate).
 

.kiro/specs/config-upgrade-snapshot/spec.json

@@ -19,5 +19,5 @@
     }
   },
   "ready_for_implementation": false,
-  "blocked_on": ["entity-bearing-snapshots"]
+  "blocked_on": []
 }

.kiro/specs/ha-convergence-recovery/requirements.md

@@ -28,7 +28,7 @@ This spec delivers the convergence-and-recovery slice of the umbrella contract.
 1. When a primary observes a higher term and is demoted, the controller shall stop persisting new state changes before continuing demotion, so that no further provisional entries are produced. *(umbrella 2.5)*
 2. When a primary is demoted, the controller shall remove from its log all entries above its committed LSN. *(umbrella 2.5)*
 3. When demotion truncation completes, the controller shall restart its own process so that in-memory state is rebuilt through boot recovery from the persisted committed state only, discarding all provisional in-memory state. *(umbrella 2.5, 2.7)*
-4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)*
+4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)* — **SUPERSEDED (2026-07-10):** "terminate fatally" (`abort()`) violated the project-wide never-self-terminate invariant (CLAUDE.md "Process Lifecycle") on an operational condition (`execv()` can fail on `ENOMEM`/`EMFILE`/a permissions change mid-upgrade — none of these are programming-invariant violations). The safety property this criterion actually protects — a divergent ex-primary never serves authoritatively — is preserved without dying: on `execv()` failure the controller now sets a sticky, never-live-cleared `ha->self_restart_pending` flag (checked unconditionally by `raft_cb_become_leader` / `promotion_fence_timeout_fn` alongside `apply_divergent`, `ha.c`) and retries the re-exec periodically via delaymgr (`ha_self_restart_enter_fence_and_retry`, `ha_convergence.c`). The node stays alive, permanently non-promotable, and self-heals the moment a retry succeeds — never requiring manual admin recovery. See `ha-async-io.md`'s own `SUPERSEDES (2026-06-25)` note for the precedent of revising a locked decision when a stronger mechanism (loud-fence-and-stay-alive) was established after the original spec closed.
 5. When a demotion occurs, the controller shall log the event with the truncation boundary (committed LSN and prior log tip) before restarting, so that operators can observe the discarded range. *(umbrella 2.5)*
 6. Where test mode is enabled, the controller shall invoke a substitutable restart seam in place of process re-execution. *(locked constraint — testability)*
 7. The controller shall be able to reconstruct its in-memory state from its persisted committed state, discarding any provisional state not present in the committed log. *(umbrella 2.7)*

.kiro/specs/raft-apply-decoupling/requirements.md

@@ -32,6 +32,7 @@ The authoritative subject for the acceptance criteria below is **the XCE control
   - Moving snapshot send, configuration-push waiting, or peer connect off the iomgr/delaymgr threads — owned by `raft-drive-async` (P2 transport/timer drive hardening); this feature owns only the commit-wait drive conversion (P1).
   - Changes to the worker join/rejoin protocol itself — owned by `worker-join-refactor` / `worker-rejoin-resync`.
   - Asynchronous DNS resolution for peer connections.
+  - Extract-from-hook / apply-after-commit ordering for queue-control and worker-control commands: `queue.c`'s `AFTER_EVENT` path and `persist_queue_hook_fun`/`persist_worker_hook_fun` still mutate in-memory state before firing the WAL append, the same double-append-class ordering this feature fixes for job submission and facts — un-fixed here (2026-07-09 validate-impl finding). Tracked as a follow-up, same shape as this feature's Requirement 1/2 but for the queue/worker command family.
 - **Adjacent expectations**:
   - This feature relies on the majority-commit and idempotency primitives already delivered by `ha-commit-engine` and `ha-commit-barrier-rollout`; it does not re-implement them.
   - This feature touches the same submit/commit path that `ha-term-safety` is concurrently wiring into the raft library adapter; the two must be sequenced/coordinated so neither reintroduces a term-blind or apply-before-commit regression.

.kiro/specs/raft-apply-decoupling/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "raft-apply-decoupling",
   "created_at": "2026-07-03T03:32:11Z",
-  "updated_at": "2026-07-03T04:35:00Z",
+  "updated_at": "2026-07-09T21:00:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
   "approvals": {
     "requirements": {
       "generated": true,

.kiro/specs/raft-ha-core-safety/design.md

@@ -51,6 +51,13 @@ The design therefore: (1) collapses every entry-ingest path into the one reconci
 - Any change to the snapshot install boundary contract → revalidate `ha-convergence-recovery` rejoin path.
 - Closing this gate (green soak) is itself the revalidation trigger that **unblocks the `raft-drive-async` production flag-flip**.
 
+### Retroactive Boundary Crossings (2026-07-09 validate-impl finding)
+
+Two tasks materially crossed the "Out of Boundary" lines above. Both are self-labeled in tasks.md, non-conflicting with the specs that inherited them, and confirmed by a later cross-task validation pass — documented here so a future validate-impl doesn't re-flag them as new findings:
+
+- **Task 5.9** shipped a commit-durability fix (`persist_submit_wal_and_commit` extraction, hook-rc-clobber job-loss) — territory line 19/36 assigns to `ha-commit-durability`. Landed here as a "down payment" (tasks.md's own words) because `raft-apply-decoupling` had no tasks.md yet and this spec's Req 8 soak needed zero job-loss to close. `raft-apply-decoupling`'s own task 3.1/3.2 later built the canonical version on the same code path — confirmed non-duplicating successor, not a competing fix.
+- **Task 5.8** shipped an endpoint-refcount/quiescence mechanism — territory line 20 explicitly excludes ("`raft-drive-async`... endpoint-quiescence — out of scope and must not change"). In practice it became the shared foundation `raft-drive-async`'s own task 3.2/3.4a cite as the reason their UAF hazards are "closed for real" — both specs' authors acknowledged the dependency in their own tasks.md, making this real, coherent, bidirectional coupling rather than silent scope creep.
+
 ## Architecture
 
 ### Existing Architecture Analysis

.kiro/specs/raft-ha-core-safety/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "raft-ha-core-safety",
   "created_at": "2026-06-27T03:15:23Z",
-  "updated_at": "2026-06-27T04:30:00Z",
+  "updated_at": "2026-07-09T21:00:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
   "approvals": {
     "requirements": {
       "generated": true,

.kiro/specs/xce-test-e2e-primitives/tasks.md

@@ -3,7 +3,7 @@
 > TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor).
 > Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code.
 
-- [ ] 1. Foundation: verdict, loop wrapper, ergonomics
+- [x] 1. Foundation: verdict, loop wrapper, ergonomics
 - [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling
   - Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle.
   - Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored.
@@ -16,7 +16,7 @@
   - _Boundary: VerdictLoopWrapper_
   - _Depends: 1.1_
 
-- [ ] 2. Shell-escape
+- [x] 2. Shell-escape
 - [x] 2.1 `!<cmd>` runs a shell command and feeds `$?`
   - In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
   - Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle.
@@ -24,7 +24,7 @@
   - _Boundary: ShellEscape_
   - _Depends: 1.1_
 
-- [ ] 3. Auto-variable substitution
+- [x] 3. Auto-variable substitution
 - [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount`
   - Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized.
   - Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value.
@@ -32,7 +32,7 @@
   - _Boundary: AutoVars_
   - _Depends: 1.1_
 
-- [ ] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
+- [x] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
 - [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId
   - Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`).
   - Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId.
@@ -40,7 +40,7 @@
   - _Boundary: JobsubFix_
   - _Depends: 1.1, 3.1_
 
-- [ ] 5. Job-count capture and assertions
+- [x] 5. Job-count capture and assertions
 - [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper
   - Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`.
   - Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions.
@@ -60,7 +60,7 @@
   - _Boundary: Assertions_
   - _Depends: 5.1, 3.1, 4.1_
 
-- [ ] 6. Wait for elected primary
+- [x] 6. Wait for elected primary
 - [x] 6.1 `wait-primary [timeout_ms]`
   - Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout.
   - Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
@@ -68,7 +68,7 @@
   - _Boundary: WaitPrimary_
   - _Depends: 1.1_
 
-- [ ] 7. Reference E2E driver
+- [x] 7. Reference E2E driver
 - [x] 7.1 Convert `test-commit-durability.sh` to use the primitives
   - Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step.
   - Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.

.kiro/steering/build-deployment.md

@@ -11,18 +11,22 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
 ```bash
 /xce-build              # all components + all tests
 /xce-build lib          # libxce + lib tests
-/xce-build controller   # lib + controller + controller tests
+/xce-build controller   # lib + controller + controller tests + xce_test driver
 /xce-build scheduler    # lib + scheduler + scheduler tests
 /xce-build worker       # lib + worker (daemon only, no unit tests)
 /xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
 /xce-build xcecli       # xcecli + Go tests
 ```
 
+`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
+
 Each invocation:
 1. `nix develop` into the hermetic shell
-2. `meson setup build --buildtype=debug -Db_sanitize=address` (fresh clean build)
-3. `ninja -C build` (compile all + test executables)
-4. Run registered meson tests
+2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
+3. `ninja -C <dir>` (compile all + test executables)
+4. Run registered meson tests (skipped for non-`asan` profiles — see below)
+
+**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
 
 ## ASAN Integration
 

.kiro/steering/ha-async-io.md

@@ -42,12 +42,14 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send
 
 The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.
 
+**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.
+
 - **RequestVote + RequestVoteReply** over Cap'n Proto
 - **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
 - **Votes persisted atomically** with term increment (raft_cb_persist)
 - **Majority wins**: quorum = ceil(n_peers / 2)
 - **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
-- **WAL truncation rule (load-bearing — bug-828)**: the **ONLY** truncates are (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, and (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile — both run *before* the new leader commits anything. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
+- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
 
 **Key safety invariants**:
 - `currentTerm` is monotonic (never decreases)
@@ -59,7 +61,8 @@ The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed v
 Bidirectional heartbeat with Raft-style AppendEntries semantics:
 
 - **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
-- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
+- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
+- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
 - **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
 
 ## Replication Consistency (Two-Tier)

.kiro/steering/persistence.md

@@ -58,6 +58,12 @@ Result: a truncated/behind node cannot win an election or pass the promotion gat
 - **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
 - **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.
 
+## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability)
+
+A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419).
+
+**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed).
+
 ## Fail-Loud, Never Self-Exit
 
 When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".