docs(config-upgrade-snapshot): re-block task 8.6 on bug-1477 (HA bootstrap retry gap)

bug-1472 (the prior blocker, worker-control replication) is fixed and
verified (7547a8f1). Rebuilding and re-running --config-upgrade on top of
that fix surfaced a new, unrelated defect: a node that takes the
skip_election fast path in myapp_run can permanently lose commit_engine_init
if its one cluster_init_phase2() attempt races config availability, since
none of ha.c's four retry call sites re-attempt for a node that never
contests an election. That member then never fires the marker-triggered
checkpoint (task 5), leaving it permanently unable to catch up past that
LSN (WAL_CHECKPOINT is local-only, non-replicated by design) -- a distinct,
permanent wedge from bug-1475's already-fixed transient race.

Root-caused via a dedicated debug pass (log/grep/git-blame evidence, logged
as bug-1477). Out of config-upgrade-snapshot's boundary -- same pattern as
bug-1471/1472/1474 -- so task 8.6 stays blocked until bug-1477 is fixed
upstream in HA bootstrap/election.

.kiro/specs/config-upgrade-snapshot/tasks.md

@@ -127,7 +127,7 @@
   - Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
   - _Requirements: 4.1, 4.3, 6.1, 6.4_
   - _Depends: 7, 8.1, 8.2, 8.3, 8.4_
-  - _Blocked: bug-1472 — pre-existing, out-of-spec-boundary HA worker-control replication defect (`worker_from_capnp`, `worker.c:827-856`, introduced 2026-05-08, commit `9b6a91a8`, ~2.5 months before this spec). `WAL_WORKER_CTRL` apply treats "-EEXIST" (target worker already exists) as a benign no-op for ALL hook details — correct for JOIN replay, but wrong for UPDATE/DEAD, where the target already existing is the normal case. Consequence: a host admin-status change (`hostctrl -c`/`-o`) never propagates past the member that directly processed the RPC — completely general, unrelated to config upgrades or checkpoints. Live soak infra + task 8.6's own bug-1468/bug-1469 fixes are committed (959b0cd) and independently reviewed; only the Requirement 6.4 host-admin-status-survival assertion is blocked, on a bug this spec does not own. See `.wolf/buglog.json` bug-1471 (byte-precise diagnosis) and bug-1472 (root cause) for full forensic detail. Unblocks when bug-1472 is fixed by whoever owns worker/HA replication — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
+  - _Blocked: bug-1477 — pre-existing, out-of-spec-boundary HA bootstrap/election defect (`myapp.c`'s `skip_election` fast path, ~343-425, plus `ha.c`'s unconditional `phase2_done=1` latch across all 4 retry call sites — `raft_cb_become_follower` 2000-2012, `raft_cb_become_leader` 1772-1783, `ha_elect_evaluate_and_signal` 5609-5629/5699-5710; introduced commits `20e59d0b`/`8f0d2d55`, ~1 month before this spec). bug-1472 (the PRIOR blocker on this task, worker-control replication) is now fixed and verified (commit `7547a8f1`); rebuilding and re-running `--config-upgrade` on top of that fix surfaced a NEW, unrelated defect: a node that finds "leader already known" at boot (`skip_election`) makes exactly one `cluster_init_phase2()` attempt, which can lose the race against config becoming available and return `-EINVAL` — and since none of the 4 retry call sites ever fire for a node that never itself contests an election, that node's `commit_engine_init` (and therefore `apply_committed_entry`/`persist_checkpoint_marker_triggered`, config-upgrade-snapshot's own task-5 mechanism) never runs, for the rest of that process's life. The member's config still "activates" cosmetically via the independent `handle_repl_config_push` RPC path, masking the defect until a checkpoint-identity check catches it: that member permanently lacks the marker-triggered checkpoint every other member takes at the same LSN, and since `WAL_CHECKPOINT` is local-only/non-replicated by design, it can never reconstruct one — a permanent `ae_reject_logmismatch` catch-up wedge, distinct from bug-1475's (already-fixed) transient race. Completely general, unrelated to config upgrades specifically — any HA bootstrap timing that lands a node on `skip_election` before its config is available is exposed the same way. See `.wolf/buglog.json` bug-1477 for full forensic detail (log/grep/git-blame evidence, no unit-level repro built — the multi-node startup-stagger timing isn't cleanly unit-testable in isolation). Unblocks when bug-1477 is fixed by whoever owns HA bootstrap/election (add a retry path for a `skip_election` node whose first `cluster_init_phase2` attempt fails, and gate the `phase2_done=1` latch on that call's return value) — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
 
 - [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
   - Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart

.wolf/cerebrum.md

@@ -108,6 +108,7 @@
 - (2026-07-10) A chaotic-HA-soak "COMMITTED JOB LOSS" verdict with `committed_divergence=0` and all jobs present at final settlement is a VISIBILITY-LATENCY gap, not a durability violation: the job was legitimately committed by the old primary before it died, but the new leader's post-election `safe_committed_lsn` (computed only from THIS election's granted-voter sample, which can be conservative when a second voter is mid-restart) is lower until fresh quorum reconfirmation — correct Raft Figure-8 safety, not a bug by itself. Per this project's test philosophy (skill xce-test-ha: "a committed job an elected primary cannot list IS a real loss regardless of eventual reappearance") this is still a hard soak FAIL and must be fixed — but ONLY by shrinking the reconfirmation window (e.g. sending the first post-promotion heartbeat immediately), never by unsafely exposing unconfirmed old-term entries early just to make the check pass.
 - (2026-07-10, bug-1433, controller-invariant-audit) `peer_ctrl_classify` (ha.c ~5935) hardcodes `VTASK_LIVENESS` for EVERY inbound peer-control packet (vote replies, PreVote replies, WAL/config ACKs) — a task class documented as "never admission-gated," i.e. never expected to block. `raft_cb_become_leader` (the raft library's `.on_become_leader` callback) is invoked synchronously from THIS dispatch path whenever a real multi-node winning vote reply arrives — NOT from the separate `VTASK_LIVE_BLOCK`-dispatched election-*start* path (`ha_dispatch_election_start`, which only covers RDHUP-failover/single-node self-grant). Any blocking call added inside `raft_cb_become_leader` (or anything it calls) is reachable on a worker class the design guarantees never blocks — same bug class as bug-1198 (closed by moving `persist_checkpoint`'s snapshot-send fdatasync off VTASK_LIVENESS entirely). Before adding ANY blocking call to the promotion path, trace peer_ctrl_classify's actual dispatch, don't trust an existing nearby comment claiming "already on VTASK_LIVE_BLOCK" — one was WRONG (see bug-1433 in buglog.json for the full trace). `persist_apply_pump_run_sync` still has this residual (bounded `pread`, lower risk than `fdatasync`, not fixed this session — see buglog.json bug-1433 for why it can't simply be removed).
 - (2026-07-10, bug-1434, controller-invariant-audit) `persist_promotion_advance_committed` and `ha_standby_advance_committed_and_apply` both did an unlocked `load; if(target<=cur) return; store` on `p->committed_lsn` — safe ONLY if truly single-writer, but `raft_cb_become_leader` calls the former BEFORE setting `role = CTRL_PRIMARY` (~170 lines later), so a still-in-flight WAL catch-up entry on another thread can call the latter concurrently during that window — classic lost-update race that can regress `committed_lsn` (monotonicity violation). General lesson: when two independent call sites do "monotonic-max store to a shared atomic, no lock" and you can't prove they're strictly serialized by role/state gating (gating checked AT CALL TIME can still race with a WINDOW where the gate value itself is about to change), use a CAS retry loop (`persist_committed_lsn_advance_atomic`), not a bare load-then-store — it's safe under ANY interleaving with zero lock-ordering risk.
+- (2026-07-28, bug-1477) `cluster_init_phase2()` (cluster.c) is the SOLE place that runs `commit_engine_init()` -- which is itself the ONLY place that registers `persist_t`'s `on_durable_advance` wake hook (persist.c:796-799 says so explicitly: "starts unregistered ... the commit engine registers it"). Without that hook, `persist_apply_pump_run`/`apply_committed_entry` NEVER runs on that node, for ANY committed LSN, for its whole process lifetime -- a node can still fully participate in WAL replication/ACK (that path is independent, in ha.c) while never materializing ANY committed state. `cluster_init_phase2` short-circuits with `-EINVAL` (cluster.c:1349-1358) whenever `c->myconf->cluster_config == NULL` (config not loaded yet) -- normal during cold boot -- and its THREE raft-driven callers (`raft_cb_become_follower` ha.c:2000-2012, `raft_cb_become_leader` ha.c:1772-1783, `ha_elect_evaluate_and_signal`'s two branches ha.c:5609-5629/5699-5710) all set `ha->phase2_done=1` UNCONDITIONALLY after calling it, with NO check of its return value -- unlike `myapp.c`'s own `skip_election` fast-path caller (~line 415-425, bug-849 lineage) which correctly gates the latch on success. Worse: a low-priority node (highest `my_priority`, longest startup stagger) that finds "leader already known" on its very first check takes `goto skip_election` and NEVER runs `ha_start_election_async` at all -- so it never reaches ANY of the three raft-callback retry opportunities, ever, for its whole life, if it never later contests its own election. If its one-shot `myapp.c` attempt races ahead of config load, `cluster_init_phase2`/`commit_engine_init` never runs again, PERIOD -- a real, permanent HA-bootstrap gap, predates config-upgrade-snapshot by ~1 month (myapp.c skip_election = commit 20e59d0b 2026-06-21; ha.c raft-callback structure = commit 8f0d2d55 task 6.2), NOT introduced by it. It stayed invisible until config-upgrade-snapshot's task 5 (`persist_checkpoint_marker_triggered`) created a NEW cross-member invariant (every member independently writes a local checkpoint at a synchronized marker-boundary LSN) that a node with a never-wired apply pump silently violates -- surfacing as a PERMANENT (not transient/racy) `ae_reject_logmismatch` catch-up wedge that superficially resembles bug-1475 but is a categorically different, unrecoverable failure (the missing entry is genuinely, permanently absent -- not merely stale bookkeeping) since a WAL_CHECKPOINT entry is local-only/non-replicated by design and can never arrive over the wire to fill the gap. Diagnostic tell: grep the suspect node's log for "commit engine wired" -- if it NEVER appears (vs. multiple times on healthy peers, once per boot/restart/election), the commit engine/apply pump never wired on that node at all.
 
 ## Do-Not-Repeat