docs(config-upgrade-snapshot): re-block task 8.6 on bug-1477 (HA bootstrap retry gap)

bug-1472 (the prior blocker, worker-control replication) is fixed and
verified (7547a8f1). Rebuilding and re-running --config-upgrade on top of
that fix surfaced a new, unrelated defect: a node that takes the
skip_election fast path in myapp_run can permanently lose commit_engine_init
if its one cluster_init_phase2() attempt races config availability, since
none of ha.c's four retry call sites re-attempt for a node that never
contests an election. That member then never fires the marker-triggered
checkpoint (task 5), leaving it permanently unable to catch up past that
LSN (WAL_CHECKPOINT is local-only, non-replicated by design) -- a distinct,
permanent wedge from bug-1475's already-fixed transient race.

Root-caused via a dedicated debug pass (log/grep/git-blame evidence, logged
as bug-1477). Out of config-upgrade-snapshot's boundary -- same pattern as
bug-1471/1472/1474 -- so task 8.6 stays blocked until bug-1477 is fixed
upstream in HA bootstrap/election.
This commit is contained in:
Rongsong Shen 2026-07-28 08:47:55 +00:00
parent b08e392eeb
commit 71223ec712
3 changed files with 37 additions and 1 deletions

View file

@ -127,7 +127,7 @@
- Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
- _Requirements: 4.1, 4.3, 6.1, 6.4_
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
- _Blocked: bug-1472 — pre-existing, out-of-spec-boundary HA worker-control replication defect (`worker_from_capnp`, `worker.c:827-856`, introduced 2026-05-08, commit `9b6a91a8`, ~2.5 months before this spec). `WAL_WORKER_CTRL` apply treats "-EEXIST" (target worker already exists) as a benign no-op for ALL hook details — correct for JOIN replay, but wrong for UPDATE/DEAD, where the target already existing is the normal case. Consequence: a host admin-status change (`hostctrl -c`/`-o`) never propagates past the member that directly processed the RPC — completely general, unrelated to config upgrades or checkpoints. Live soak infra + task 8.6's own bug-1468/bug-1469 fixes are committed (959b0cd) and independently reviewed; only the Requirement 6.4 host-admin-status-survival assertion is blocked, on a bug this spec does not own. See `.wolf/buglog.json` bug-1471 (byte-precise diagnosis) and bug-1472 (root cause) for full forensic detail. Unblocks when bug-1472 is fixed by whoever owns worker/HA replication — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
- _Blocked: bug-1477 — pre-existing, out-of-spec-boundary HA bootstrap/election defect (`myapp.c`'s `skip_election` fast path, ~343-425, plus `ha.c`'s unconditional `phase2_done=1` latch across all 4 retry call sites — `raft_cb_become_follower` 2000-2012, `raft_cb_become_leader` 1772-1783, `ha_elect_evaluate_and_signal` 5609-5629/5699-5710; introduced commits `20e59d0b`/`8f0d2d55`, ~1 month before this spec). bug-1472 (the PRIOR blocker on this task, worker-control replication) is now fixed and verified (commit `7547a8f1`); rebuilding and re-running `--config-upgrade` on top of that fix surfaced a NEW, unrelated defect: a node that finds "leader already known" at boot (`skip_election`) makes exactly one `cluster_init_phase2()` attempt, which can lose the race against config becoming available and return `-EINVAL` — and since none of the 4 retry call sites ever fire for a node that never itself contests an election, that node's `commit_engine_init` (and therefore `apply_committed_entry`/`persist_checkpoint_marker_triggered`, config-upgrade-snapshot's own task-5 mechanism) never runs, for the rest of that process's life. The member's config still "activates" cosmetically via the independent `handle_repl_config_push` RPC path, masking the defect until a checkpoint-identity check catches it: that member permanently lacks the marker-triggered checkpoint every other member takes at the same LSN, and since `WAL_CHECKPOINT` is local-only/non-replicated by design, it can never reconstruct one — a permanent `ae_reject_logmismatch` catch-up wedge, distinct from bug-1475's (already-fixed) transient race. Completely general, unrelated to config upgrades specifically — any HA bootstrap timing that lands a node on `skip_election` before its config is available is exposed the same way. See `.wolf/buglog.json` bug-1477 for full forensic detail (log/grep/git-blame evidence, no unit-level repro built — the multi-node startup-stagger timing isn't cleanly unit-testable in isolation). Unblocks when bug-1477 is fixed by whoever owns HA bootstrap/election (add a retry path for a `skip_election` node whose first `cluster_init_phase2` attempt fails, and gate the `phase2_done=1` latch on that call's return value) — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
- [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
- Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart