fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.

.kiro/specs/config-upgrade-snapshot/requirements.md

@@ -15,7 +15,7 @@ This feature makes a configuration upgrade establish a **full, entity-bearing st
   - The Raft protocol (term/log/commit) — owned by the `Raft TLA+ Conformance` library series.
   - Controller voting-set / membership changes — `ha-config-bootstrap`.
 - **Adjacent expectations**:
-  - **HARD dependency on `entity-bearing-snapshots`**: today snapshots are a 16-byte LSN sentinel carrying no entity state; the COW machinery (`xzarray_freeze`/`thaw`) is built but dormant. That work must land first.
+  - **Dependency on entity-bearing snapshots — SATISFIED (2026-07-09) by `ha-entity-snapshots` (21/21 tasks, fully approved).** That spec landed `write_snapshot`/`load_snapshot` (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time, restoring before advancing position — not the COW `xzarray_freeze`/`thaw` mechanism this line originally anticipated (that machinery remains dormant; `ha-entity-snapshots` took a different, already-shipped implementation path). This spec no longer needs to wait; verify `write_snapshot`'s actual on-disk entity format (persist.c, `ha-entity-snapshots` design.md) before writing this spec's own design so it builds on the real interface, not the COW one originally assumed here.
   - Anchors on the `(term, lsn)` configuration-change marker from `ha-term-safety`.
   - Reuses the existing `snapshot_transfer_s` snapshot-install transport (chunked, `(term, lsn)`-anchored, hash-verified, install-before-activate).
 

.kiro/specs/config-upgrade-snapshot/spec.json

@@ -19,5 +19,5 @@
     }
   },
   "ready_for_implementation": false,
-  "blocked_on": ["entity-bearing-snapshots"]
+  "blocked_on": []
 }

.kiro/specs/ha-convergence-recovery/requirements.md

@@ -28,7 +28,7 @@ This spec delivers the convergence-and-recovery slice of the umbrella contract.
 1. When a primary observes a higher term and is demoted, the controller shall stop persisting new state changes before continuing demotion, so that no further provisional entries are produced. *(umbrella 2.5)*
 2. When a primary is demoted, the controller shall remove from its log all entries above its committed LSN. *(umbrella 2.5)*
 3. When demotion truncation completes, the controller shall restart its own process so that in-memory state is rebuilt through boot recovery from the persisted committed state only, discarding all provisional in-memory state. *(umbrella 2.5, 2.7)*
-4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)*
+4. If the process self-restart cannot be performed, the controller shall terminate fatally rather than continue running with potentially divergent state. *(umbrella 2.5; locked constraint)* — **SUPERSEDED (2026-07-10):** "terminate fatally" (`abort()`) violated the project-wide never-self-terminate invariant (CLAUDE.md "Process Lifecycle") on an operational condition (`execv()` can fail on `ENOMEM`/`EMFILE`/a permissions change mid-upgrade — none of these are programming-invariant violations). The safety property this criterion actually protects — a divergent ex-primary never serves authoritatively — is preserved without dying: on `execv()` failure the controller now sets a sticky, never-live-cleared `ha->self_restart_pending` flag (checked unconditionally by `raft_cb_become_leader` / `promotion_fence_timeout_fn` alongside `apply_divergent`, `ha.c`) and retries the re-exec periodically via delaymgr (`ha_self_restart_enter_fence_and_retry`, `ha_convergence.c`). The node stays alive, permanently non-promotable, and self-heals the moment a retry succeeds — never requiring manual admin recovery. See `ha-async-io.md`'s own `SUPERSEDES (2026-06-25)` note for the precedent of revising a locked decision when a stronger mechanism (loud-fence-and-stay-alive) was established after the original spec closed.
 5. When a demotion occurs, the controller shall log the event with the truncation boundary (committed LSN and prior log tip) before restarting, so that operators can observe the discarded range. *(umbrella 2.5)*
 6. Where test mode is enabled, the controller shall invoke a substitutable restart seam in place of process re-execution. *(locked constraint — testability)*
 7. The controller shall be able to reconstruct its in-memory state from its persisted committed state, discarding any provisional state not present in the committed log. *(umbrella 2.7)*

.kiro/specs/raft-apply-decoupling/requirements.md

@@ -32,6 +32,7 @@ The authoritative subject for the acceptance criteria below is **the XCE control
   - Moving snapshot send, configuration-push waiting, or peer connect off the iomgr/delaymgr threads — owned by `raft-drive-async` (P2 transport/timer drive hardening); this feature owns only the commit-wait drive conversion (P1).
   - Changes to the worker join/rejoin protocol itself — owned by `worker-join-refactor` / `worker-rejoin-resync`.
   - Asynchronous DNS resolution for peer connections.
+  - Extract-from-hook / apply-after-commit ordering for queue-control and worker-control commands: `queue.c`'s `AFTER_EVENT` path and `persist_queue_hook_fun`/`persist_worker_hook_fun` still mutate in-memory state before firing the WAL append, the same double-append-class ordering this feature fixes for job submission and facts — un-fixed here (2026-07-09 validate-impl finding). Tracked as a follow-up, same shape as this feature's Requirement 1/2 but for the queue/worker command family.
 - **Adjacent expectations**:
   - This feature relies on the majority-commit and idempotency primitives already delivered by `ha-commit-engine` and `ha-commit-barrier-rollout`; it does not re-implement them.
   - This feature touches the same submit/commit path that `ha-term-safety` is concurrently wiring into the raft library adapter; the two must be sequenced/coordinated so neither reintroduces a term-blind or apply-before-commit regression.

.kiro/specs/raft-apply-decoupling/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "raft-apply-decoupling",
   "created_at": "2026-07-03T03:32:11Z",
-  "updated_at": "2026-07-03T04:35:00Z",
+  "updated_at": "2026-07-09T21:00:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
   "approvals": {
     "requirements": {
       "generated": true,

.kiro/specs/raft-ha-core-safety/design.md

@@ -51,6 +51,13 @@ The design therefore: (1) collapses every entry-ingest path into the one reconci
 - Any change to the snapshot install boundary contract → revalidate `ha-convergence-recovery` rejoin path.
 - Closing this gate (green soak) is itself the revalidation trigger that **unblocks the `raft-drive-async` production flag-flip**.
 
+### Retroactive Boundary Crossings (2026-07-09 validate-impl finding)
+
+Two tasks materially crossed the "Out of Boundary" lines above. Both are self-labeled in tasks.md, non-conflicting with the specs that inherited them, and confirmed by a later cross-task validation pass — documented here so a future validate-impl doesn't re-flag them as new findings:
+
+- **Task 5.9** shipped a commit-durability fix (`persist_submit_wal_and_commit` extraction, hook-rc-clobber job-loss) — territory line 19/36 assigns to `ha-commit-durability`. Landed here as a "down payment" (tasks.md's own words) because `raft-apply-decoupling` had no tasks.md yet and this spec's Req 8 soak needed zero job-loss to close. `raft-apply-decoupling`'s own task 3.1/3.2 later built the canonical version on the same code path — confirmed non-duplicating successor, not a competing fix.
+- **Task 5.8** shipped an endpoint-refcount/quiescence mechanism — territory line 20 explicitly excludes ("`raft-drive-async`... endpoint-quiescence — out of scope and must not change"). In practice it became the shared foundation `raft-drive-async`'s own task 3.2/3.4a cite as the reason their UAF hazards are "closed for real" — both specs' authors acknowledged the dependency in their own tasks.md, making this real, coherent, bidirectional coupling rather than silent scope creep.
+
 ## Architecture
 
 ### Existing Architecture Analysis

.kiro/specs/raft-ha-core-safety/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "raft-ha-core-safety",
   "created_at": "2026-06-27T03:15:23Z",
-  "updated_at": "2026-06-27T04:30:00Z",
+  "updated_at": "2026-07-09T21:00:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
   "approvals": {
     "requirements": {
       "generated": true,

.kiro/specs/xce-test-e2e-primitives/tasks.md

@@ -3,7 +3,7 @@
 > TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor).
 > Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code.
 
-- [ ] 1. Foundation: verdict, loop wrapper, ergonomics
+- [x] 1. Foundation: verdict, loop wrapper, ergonomics
 - [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling
   - Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle.
   - Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored.
@@ -16,7 +16,7 @@
   - _Boundary: VerdictLoopWrapper_
   - _Depends: 1.1_
 
-- [ ] 2. Shell-escape
+- [x] 2. Shell-escape
 - [x] 2.1 `!<cmd>` runs a shell command and feeds `$?`
   - In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
   - Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle.
@@ -24,7 +24,7 @@
   - _Boundary: ShellEscape_
   - _Depends: 1.1_
 
-- [ ] 3. Auto-variable substitution
+- [x] 3. Auto-variable substitution
 - [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount`
   - Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized.
   - Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value.
@@ -32,7 +32,7 @@
   - _Boundary: AutoVars_
   - _Depends: 1.1_
 
-- [ ] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
+- [x] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
 - [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId
   - Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`).
   - Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId.
@@ -40,7 +40,7 @@
   - _Boundary: JobsubFix_
   - _Depends: 1.1, 3.1_
 
-- [ ] 5. Job-count capture and assertions
+- [x] 5. Job-count capture and assertions
 - [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper
   - Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`.
   - Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions.
@@ -60,7 +60,7 @@
   - _Boundary: Assertions_
   - _Depends: 5.1, 3.1, 4.1_
 
-- [ ] 6. Wait for elected primary
+- [x] 6. Wait for elected primary
 - [x] 6.1 `wait-primary [timeout_ms]`
   - Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout.
   - Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
@@ -68,7 +68,7 @@
   - _Boundary: WaitPrimary_
   - _Depends: 1.1_
 
-- [ ] 7. Reference E2E driver
+- [x] 7. Reference E2E driver
 - [x] 7.1 Convert `test-commit-durability.sh` to use the primitives
   - Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step.
   - Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.