myxce/.kiro/specs/config-upgrade-snapshot/tasks.md

Implementation Plan

• 1. Define HostAdminState and QueueAdminState Cap'n Proto structs and bump the snapshot format version

  • Add HostAdminState { name; status } and QueueAdminState { name; status } messages to the controller schema, independently extensible for future admin-state fields
  • Bump the on-disk checkpoint format version so any binary reading a checkpoint under the old queue-frame layout hard-rejects it rather than misinterpreting the bytes
  • Observable: schema compiles, generates the new message types, and a checkpoint file written under the new version is rejected by code still expecting the old version
  • Requirements: 1.9, 5.7

• 2. Core: snapshot entity and configuration payload — collection and write side

• 2.1 Collect host admin-state frames into the checkpoint

  • Walk the live host table the same way existing job/queue frame collection does, encoding only administratively-set status bits (open/closed/locked/tagged) per host as a HostAdminState frame
  • Exclude every worker-reported, dynamically-changing field (resource values, liveness timestamps, non-admin status bits) from the encoded frame
  • Observable: a checkpoint taken while a host is administratively closed contains exactly one HostAdminState frame for that host with only the admin bits set, and none of its dynamic fields
  • Requirements: 1.6, 1.7, 1.9

• 2.2 Migrate queue admin-state frames to the new wire format

  • Replace the queue frame's fixed raw byte layout with QueueAdminState encoding, preserving the same name+status content
  • Observable: a checkpoint's queue section decodes via the new message type, and a checkpoint written under the prior raw layout is rejected fail-loud rather than silently reinterpreted
  • Requirements: 1.9, 5.7

• 2.3 Embed the active configuration bytes into every checkpoint

  • Read the locally-held active configuration bytes directly from where they are already durably saved on disk, and append them as a new checkpoint section after the host frames
  • Ensure this section is populated on every checkpoint (not only a marker-triggered one), so a member joining or catching up after the marker-triggered snapshot has rotated away still receives configuration and state together
  • Keep configuration bytes out of WAL entry payloads entirely — only the checkpoint file carries them
  • Observable: any checkpoint file, marker-triggered or periodic, contains a configuration-bytes section whose content hash matches the currently active configuration; two independent materializations at the same marker boundary and configuration produce byte-identical checkpoint files
  • Requirements: 2.1, 2.3, 2.4, 5.1

• 3. (P) Restore a host admin-state frame safely regardless of whether the host has ever joined this member

  • Look up the host by name in the live host table; if present, OR the restored admin-status bits into its current status
  • If absent, create a minimal placeholder entry (unjoined, dead-until-connected) carrying only the restored admin-status bits, using the same host-creation mechanism the existing synthetic sentinel host already uses without requiring a live join
  • Ensure the placeholder is valid enough that the host's real, later join is treated as an ordinary reconnect rather than a first-time join, so its existing admin-bit-preserving behavior applies unchanged
  • Guard against a retried restore attempt re-creating or conflicting with a placeholder an earlier, partially-failed attempt already created — find first, create only if truly missing
  • Observable: restoring a host admin-state frame for a host with no local record creates exactly one placeholder entry carrying the correct admin bits; restoring it again (simulating a retry) does not create a duplicate or error; when that host later joins for real, its admin status (e.g. closed) is still set afterward
  • Requirements: 1.10, 1.11, 5.8
  • Boundary: Host Restore (worker.c)

• 4. Decode host and queue admin-state frames on checkpoint load

  • Add host-frame decoding to the checkpoint load path, dispatching each decoded frame to the host restore capability
  • Update queue-frame decoding to read the new QueueAdminState encoding instead of the old raw layout, without changing the existing queue-state-restore behavior itself
  • Preserve the existing restore discipline: every frame in the checkpoint restores successfully before any position is advanced; any restore failure leaves prior state untouched
  • Observable: loading a checkpoint containing host and queue admin-state frames restores every host and queue's admin status correctly regardless of restore order between entity types
  • Requirements: 1.10
  • Depends: 2.1, 2.2, 3

• 5. Materialize the upgrade snapshot when a configuration-change marker is applied

  • Add explicit handling for the configuration-change marker in the single, unified entity-apply dispatch point (the same seam every member's apply already routes through — primary, standby, and restart replay alike), replacing its current no-op treatment
  • Trigger a forced checkpoint at that exact point, so materialization happens immediately and deterministically at the marker's boundary rather than waiting for a periodic trigger
  • Ensure the forced checkpoint shares the existing single-in-flight checkpoint guard, so a concurrent periodic checkpoint and this one can never both proceed at once
  • Treat a "checkpoint already in progress" outcome (e.g. a second configuration marker committing while an earlier one's checkpoint is still running) as an expected, self-resolving condition, not a failure to retry specially
  • Ensure this trigger only materializes state — it must not itself change how entities are interpreted or re-homed under the new configuration
  • Observable: applying a configuration-change marker produces a checkpoint at that marker's exact position, on both the primary and a standby, without requiring a controller restart; a second marker committing while the first's checkpoint is still running does not corrupt or lose either checkpoint
  • Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
  • Depends: 2.1, 2.2, 2.3

• 6. Core: activation fence gating serving and promotion on materialization

• 6.1 (P) Add a materialization-and-hash-verification fence independent of the existing configuration-byte fence

  • Introduce a new fence condition distinct from the existing configuration-byte-mismatch fence, since "bytes don't match" and "materialization for a matching configuration isn't done yet" are different conditions that must be distinguishable
  • Verify by reading the current committed configuration-change marker and this member's own materialized-and-verified state for it, always re-reading the live, currently-active configuration bytes on disk on every verification attempt rather than trusting a previously-embedded or cached copy
  • Set the fence and log the actionable mismatch loudly on failure; clear it on success; never terminate the process on this condition
  • Observable: a member whose materialization for the current marker is incomplete or mismatched shows the fence set and a loud log entry naming the marker; once materialization completes and bytes match, the fence clears without any process restart
  • Requirements: 1.8, 3.2, 3.4
  • Boundary: Activation Fence (ha.c/ha.h)

• 6.2 Gate primary request-serving and promotion on the new fence

  • Extend the primary request-dispatch gate with this new fence check, composed alongside the existing configuration-byte fence rather than merged into it, admitting replication and read-only operations while fenced and refusing state-changing client requests
  • Extend the promotion gate the same way, so a member whose materialization is incomplete cannot become primary and immediately violate the serving gate the moment it wins an election
  • Introduce no new promotion-eligibility mechanism for WAL/log completeness — this fence is about materialization completeness only, layered on top of the existing Raft log-freshness election gate, not replacing or duplicating it
  • Wire a non-blocking, self-triggering retry so the fence re-verifies and clears automatically once conditions allow, without any operator action
  • Observable: a fenced primary continues answering read-only and status requests but rejects a state-changing client request with a not-ready response; a fenced member that wins an election does not begin serving under the new configuration until the fence clears on its own
  • Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5
  • Depends: 6.1

• 7. Remove the per-node restart/replay fallback and replace it with the marker-triggered materialization

  • Delete the configuration-upgrade handler's restart-request step now that materialization is driven entirely by the marker-apply trigger, decoupled from that request-handling code path
  • Add a deterministic test seam that can force a leadership change to land exactly between a configuration-change marker's commit and every member completing its snapshot install, for later regression use
  • Observable: pushing a configuration upgrade no longer restarts any controller process, and every member reaches the new configuration through the marker-apply trigger alone
  • Requirements: 1.2, 1.3, 1.4, 2.2, 5.4, 6.3
  • Depends: 5

• 8. Validation: regression coverage for every new mechanism

• 8.1 (P) Unit-test the marker-apply trigger and its interaction with WAL retention

  • Assert the configuration-marker case fires a checkpoint exactly once per apply and is safe under a forced retry
  • Assert the trigger does not itself re-home entities
  • Add a deterministic fault-injection seam at the marker-apply-to-checkpoint-durable boundary, mirroring the existing fault-hook pattern already used elsewhere in this codebase, for later chaotic-scenario use
  • Assert a snapshot encoded in an unrecognized format version (simulating a rolling binary upgrade or the prior raw queue-frame layout) is rejected fail-loud, never partially installed
  • Assert that when a standby is genuinely mid-catch-up (has not yet acknowledged the marker's position) at the moment the marker-triggered checkpoint's WAL trim runs, that standby's still-needed WAL segments are not deleted
  • Observable: the new test suite passes, demonstrating exactly-once checkpoint firing, safe retry, format-mismatch rejection, and trim safety under the new trigger path
  • Requirements: 1.1, 1.5, 1.8, 5.2, 5.3, 5.6, 5.7, 6.3, 7.3, 7.4
  • Depends: 5
  • Boundary: Marker Apply Trigger (test_upgrade_snapshot_apply.c)

• 8.2 (P) Unit-test host placeholder restore and rejoin preservation

  • Assert restoring a host admin-state frame for a never-joined host creates a placeholder carrying the correct admin bits
  • Assert a retried restore after a simulated partial failure finds and reuses the existing placeholder rather than conflicting with it
  • Assert that a host's real first join after a placeholder was created preserves the admin status through the ordinary reconnect path
  • Observable: the new test suite passes, demonstrating create-if-missing correctness, idempotent retry, and admin-status survival through a real join
  • Requirements: 1.10, 1.11, 5.8, 7.1
  • Depends: 3
  • Boundary: Host Restore (test_upgrade_snapshot_restore.c)

• 8.3 (P) Unit-test the activation fence

  • Assert the fence sets on incomplete materialization or a configuration-hash mismatch and clears on successful verification
  • Assert the request-dispatch gate's allowlist behavior while fenced, and that promotion is refused while fenced
  • Observable: the new test suite passes, demonstrating fence set/clear transitions and correct gating of both serving and promotion
  • Requirements: 3.2, 3.3, 3.4, 4.3, 5.5
  • Depends: 6.2
  • Boundary: Activation Fence (test_upgrade_activation_fence.c)

• 8.4 (P) Build a cross-member active-configuration comparator and extend checkpoint verification tooling

  • Build a new offline, file-based tool that compares each member's active configuration content hash, recomputed fresh from each member's own on-disk configuration file, mirroring the existing checkpoint-comparison tool's shape
  • Extend the existing checkpoint-structure verification tool to decode and assert the embedded configuration reference's tag and hash (not merely report their presence), and to report host-frame count and configuration-bytes-section presence
  • Observable: the comparator correctly reports two members with matching configuration as identical and two with differing configuration as divergent; the extended verification tool reports a concrete tag/hash value and section counts for a real checkpoint file
  • Requirements: 4.2
  • Depends: 2.3
  • Boundary: Regression Tooling (configcompare, snapverify)

• 8.5 Reproduce bug-857 on the pre-change behavior and confirm it no longer occurs

  • Run the original materialized-state-divergence repro scenario against the pre-change code path and confirm the divergence occurs
  • Run the same scenario against the post-change behavior and confirm every member's materialized state converges identically
  • Observable: the same repro scenario demonstrably fails on the old behavior and passes on the new one
  • Requirements: 6.2
  • Depends: 7, 8.1, 8.2, 8.3, 8.4

• 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover

  • Drive a configuration upgrade while repeatedly killing and restarting the primary, and assert every member's materialized state and active configuration are identical afterward
  • Assert a host's administratively-set status specifically survives the same scenario, not merely inferred from the aggregate state check
  • Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
  • Requirements: 4.1, 4.3, 6.1, 6.4
  • Depends: 7, 8.1, 8.2, 8.3, 8.4
  • Unblocked 2026-07-29 — bug-1479 fully fixed and live-verified. run-ha-soak.sh --config-upgrade now reports a clean PASS through all 3 force_elect_node rounds (see Implementation Notes bullet "Task 8.6 — bug-1479 fully fixed 2026-07-29" below for the full account). Prior blockers bug-1472/bug-1477/bug-1479 all fixed and verified in sequence; no further blocker remains.

• 8.7 Inject a crash and a leadership change at the two critical upgrade windows

  • Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart
  • Force a leadership change between a configuration-change marker's commit and every member completing its snapshot install, and assert the new leader continues driving materialization to completion without any special resume logic
  • Observable: both fault-injection scenarios complete with the post-upgrade consistency invariant and the crash-safety invariant intact
  • Requirements: 5.2, 5.4, 5.5, 6.3
  • Depends: 7, 8.1, 8.2, 8.3, 8.4

• 8.8 Exercise a second configuration upgrade committing before the first's installation completes everywhere

  • Commit a second configuration-change marker while the first upgrade's snapshot is still installing on at least one member, and assert the earlier upgrade's in-flight state is cleanly superseded rather than left partially applied or conflated with the newer one
  • Observable: after both markers settle, every member reflects only the newer configuration and state, with no trace of a partially-applied intermediate state
  • Requirements: 4.4, 7.2
  • Depends: 7, 8.1, 8.2, 8.3, 8.4

• 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions

  • Repeat the mid-catch-up WAL-trim assertion from the unit-level fixture, but under the full chaotic multi-kill soak rather than an isolated fixture
  • Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up
  • Requirements: 7.4
  • Depends: 7, 8.1, 8.2, 8.3, 8.4

Implementation Notes

• Task 8.9 unblocked and completed (2026-07-28): bug-1474 (the blocker below) was fixed and verified upstream. Re-running run-ha-soak.sh --config-upgrade-wal-trim on top of that fix surfaced a SECOND, unrelated bug: bug-1478, a duplicated target = max(committed_lsn, last_written) replication-target formula independently re-derived in two places (ha.c's heartbeat/stateReport dispatch and config_repl.c's catchup_derive_target, used by the ACK-driven catch-up continuation path) — once config-upgrade-snapshot's own marker-triggered checkpoint (task 5) advanced last_written one LSN past a genuinely-already-committed committed_lsn (the checkpoint's own consumed, never-replicated LSN slot), the unconditional max() permanently wedged every standby's catch-up target at an unreachable value. Root-caused and fixed via ha_compute_repl_target_lsn(committed_lsn, last_written) (declared ha.h, defined ha.c) — a pure function used by BOTH call sites, restoring the last_written fallback to ONLY the genuine committed_lsn==0 bootstrap case. Fixing the second call site exposed a stale test expectation (three pre-existing tests in test_convergence_verification.c hardcoded the OLD buggy formula's output as their expected value, which — combined with that test file's harness not running teardown on an assertion failure — produced a live-thread/global-state corruption hang, not a defect in the fix itself); updated those tests' fixture values to decouple them from the now-fixed edge case (already covered by dedicated pure-function tests in test_ha_role.c). Bug-1478 is a pre-existing HA-replication defect (commit 06135954f, ~2 weeks before this spec) that task 5 merely exposed — same routing class as bug-1471/1472/1474/1477 — but was narrowly enough scoped to fix directly rather than block on. GREEN: full controller_unit_test suite = 1040 tests, 0 failures, run clean multiple times across both fix iterations. Live-verified: run-ha-soak.sh --config-upgrade-wal-trim reports a clean PASS (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3 using the preserved floor segment for real incremental catch-up). See .wolf/buglog.json bug-1474 and bug-1478 for full forensic detail.

• Task 8.9 (live soak, genuinely requires chaos — unlike 8.5/8.7/8.8's unit-level redirections): infra complete, task BLOCKED on a new pre-existing bug (bug-1474). Unlike every other 8.x subtask, this one's own bullet ("under the full chaotic multi-kill soak rather than an isolated fixture") makes the point of the task proving the mid-catch-up WAL-trim guard (repl_trim_lsn/trim_wal_segments, bug-1147) holds under REAL timing uncertainty, not just deterministic fixture control — so it could not be redirected to a cheaper unit-level proof the way 8.5/8.7/8.8 were. Built run_config_upgrade_wal_trim_round() + --config-upgrade-wal-trim in run-ha-soak.sh (deploys with small --wal-segment-bytes, kills a non-primary TARGET, forces ≥2 real segment rotations past its floor on survivors, submits a live config upgrade to fire the marker-triggered checkpoint's trim, then directly inspects survivors' on-disk wal/ directory BEFORE restarting TARGET — mirrors the unit fixture's own seg-1-gone/seg-2-preserved/seg-3-preserved assertions against real files instead of a tmpdir fixture). Ran twice against independent live 3-node clusters: both runs failed — the floor segment protecting the killed target was trimmed both times. Root-caused (not guessed) to bug-1474, a general HA connection-management race entirely unrelated to config-marker triggering: killing a standby frees its replicator slot and sets the retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an already-connected peer as "missing" (comparing indices under two different conventions — relative votes_idx vs. absolute peer_idx, per the reviewer's independent finding), spawns a redundant reconnect, and repl_add_standby's first-empty-slot allocation reuses the just-freed slot, memset-clobbering the guard before trim ever runs. Reviewer independently re-derived this from repl.c:74-133/ha.c:9219-9254 plus raw preserved logs (not the implementer's quoted excerpts) and confirmed via git blame the responsible code predates this spec by weeks and is reachable identically under the OLD periodic-checkpoint trigger — genuinely out of this spec's boundary, same class of finding as bug-1471/1472. Reviewer also confirmed no scoped adjustment to the round's chaos could dodge the race without suppressing the live heartbeat behavior Req 7.4 requires testing under, so BLOCKED (not a redesign) is the correct call. Spec-wide status after this task: every implementable task (1 through 8.5, 8.7, 8.8) is [x]; the only two remaining open items (8.6, 8.9) are both blocked on real, independently-verified, out-of-spec-boundary HA defects this spec's own testing effort discovered (bug-1472, bug-1474) — neither blocks any OTHER task in this spec, and neither requires further config-upgrade-snapshot work once fixed upstream.

• Task 8.8 (unit-level, no live soak needed — deliberately excludes 8.6): proves Req 4.4/7.2 (a second marker committing before the first's install completes must cleanly supersede it, never conflate) via two independently-built t42/t43-family fixtures in test_config_repl.c: c_control commits+applies+checkpoints marker A then marker B normally; c_raced holds the shared checkpoint_in_progress CAS across both markers' commit+apply (inlining t43_commit_marker's 4-step body rather than changing its signature), so both markers' forced-checkpoint attempts hit -EBUSY and skip — genuinely "still installing." Mid-race assertions (captured before releasing the CAS) prove mc->cluster_config already reflects marker B while cfgref_present is still false and the fence is set; post-resolve assertions (after releasing the CAS, calling persist_checkpoint(), then an explicit upgrade_materialization_verify()) prove the settled state reflects only B. Discovered mid-implementation that whole-file snapdiff_paths byte-identity between the two fixtures' final checkpoints does NOT hold — c_control's marker-A checkpoint physically consumes one extra WAL LSN (persist_write_checkpoint_marker, persist.c:1684) before marker B is appended, so B lands at a different absolute LSN in each fixture, and write_snapshot's sentinel header embeds checkpoint_lsn at byte offset 8. Adapted to a field-aware comparison (full memcmp from byte 16 onward) plus snapdiff_paths diagnostic corroboration that offset 8 is the only divergence — a more precise proof than blind byte-identity, reviewer independently re-derived and confirmed this against write_snapshot's actual layout rather than trusting the claim. One RED→GREEN cycle (bug-1473, test-construction gap not a production bug): the bare persist_checkpoint() resolve call doesn't itself trigger upgrade_materialization_verify() in a delaymgr-less unit fixture — production's self-healing (schedule_upgrade_materialization_retry, fired from ha.c:2541-2542 on every commit-advance) only runs on a live cluster; the fixture needs the manual call, mirroring the existing 6.1-c fence tests' pattern. Reviewer's only findings were non-blocking style notes (could have used the public upgrade_materialization_retry_fn wrapper instead of calling upgrade_materialization_verify directly — functionally identical here since the fence was already independently confirmed set). Suite: 1027 → 1028 tests, 0 failures.

• Task 8.7 (both scenarios, unit-level, no live soak needed): design.md's Testing Strategy was explicit this task should use deterministic fault injection rather than chaotic-soak timing probability, and both hooks needed already existed from earlier tasks (persist_marker_checkpoint_fault_hook, task 8.1; xce_upgrade_marker_committed_hook, task 7) — task 8.7 was purely additive test work, zero production code touched. Scenario 1 (test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization, test_upgrade_snapshot_apply.c) builds on the existing seam test (which only proved the fault hook fires and gates the checkpoint call) by clearing the fault, tearing down without deleting the on-disk WAL, and booting a genuinely fresh persist_t/cluster_t via persist_recover() — a real second-process boot-replay simulation, deliberately not a same-process re-call (which the test's own comment argues would be a weaker, trivially-passing proof). Scenario 2 (test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain, test_config_repl.c) proves Req 5.4 by building a promoted member and a non-promoted control member from byte-identical committed-but-unapplied markers, draining each via the respective real production path (persist_apply_pump_run_sync, which production promotion — raft_cb_become_leader, ha.c — calls unconditionally right after the role flip; vs. plain persist_apply_pump_run for the control), and asserting the resulting checkpoints are byte-identical via snapdiff_paths (task 8.4's tool). Reviewer independently verified (not trusted) both design claims by reading ha.c directly (upgrade_materialization_fenced only refuses an already-fenced member; the sync-drain call is unconditional and unguarded by role-specific logic) and specifically interrogated whether this was a tautology (calling the same underlying function twice) — concluded it genuinely is not, since the real variable under test is role state at drain time plus the production entry point, and the test separately asserts promotion alone has zero effect on the fence. Also confirmed via direct comparison against the existing 6.2/8.3 fence tests that this is new continuation coverage (fence unset → promotion succeeds → drain completes → config activates), not a duplicate of the existing gating coverage (fence set → promotion refused, never reaching the drain).

• Task 8.6 progress (infrastructure + two real production bugs found and fixed; task NOT YET complete — one more bug blocks a clean live-soak PASS): building this task's live 3-node chaotic-upgrade soak required substantial new infrastructure that never existed anywhere in this codebase: a client-side send path for OP_upgradeConfig (xce_upgradeconfig_req_raw/xce_upgradeconfig/xcecli_upgradeconfig — the auto-generated capnp encoder for this request type is a literal no-op for its Data fields, worked around via a hand-rolled bypass mirroring the existing xce_replconfigpush_req_raw idiom), a cmd_hostctrl/cmd_upgradeconfig CLI pair in test.c, and a new --config-upgrade mode in run-ha-soak.sh (+ SKILL.md docs). Building and running this surfaced two real, independently-confirmed production defects:

  • bug-1468 (cluster.c): myreq_handler's OPCode_upgradeConfig dispatch guard was inverted by an unrelated commit (fd93997f, 2026-05-31) — it rejected the only role (primary) that could ever legitimately reach it, making config upgrades unreachable from any network client at all. Fixed: restored correct polarity.
  • bug-1469 (persist.c): apply_committed_entry's WAL_CONFIG_MARKER case never called config_activate_committed() — so the node that originates a config upgrade never refreshed its own in-memory config/queue state as part of committing it (only receiving standbys did, via a separate out-of-band push path). If that node was killed before some later, independent event happened to refresh it, its checkpoint froze with stale config content — a genuine, live-reproduced instance of bug-857's originally-hypothesized failure mode (task 8.5 could only prove a different, unit-level mechanism; this is the first real empirical case). Fixed: added config_activate_committed(c) inside the marker-apply case, positioned before the marker-triggered checkpoint write (checkpoint content is built from live runtime tables that only reflect the reload after activation runs, not from myconf directly) and guarded against c->myconf == NULL (a real crash risk hit in an existing test fixture). Fixing this also exposed and required fixing a genuine race in a pre-existing test (test_commit_advance_retry_clears_push_before_commit_fence) that the new activation side-effect made observable for the first time — narrowed the race window, did not weaken any assertion. Reviewer independently re-derived the lock-safety claim from scratch (traced every mutex site in config_repl.c, confirmed none reachable from config_activate_committed) and the thread-class claim (apply_committed_entry runs on a VTASK_LIVE_BLOCK pool worker, confirmed via persist_apply_pump_kick) rather than trusting the implementer's account — both held.
  • Still open, blocking a clean PASS — bug-1471: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately not triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a new queue to an already-running member's runtime qmgr without a restart (config_activate_committed/load_primary_config refresh myconf->cluster_config but nothing syncs that into the live qmgr) — the soak's upgrade-target config change was deliberately scoped to modifying an existing queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in .wolf/buglog.json for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS.
  • Process note: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live git status/diff before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree.

• Task 8.6 re-run 2026-07-29 (bug-1477 fix reconfirmed; NEW blocker bug-1479 reached, still open): rebuilt xce_controller/xce_test/controller_unit_test fresh against the working tree's bug-1479 fix attempt (ha_raft_on_repl_nack/repl_get_catchup_join_lsn in ha.c/ha.h/repl.c/repl.h, 1048/1048 controller_unit_test green) and re-ran run-ha-soak.sh --config-upgrade end to end (deploy preserved, not deleted, at ~/tmp/xce_ha_bug1479_verify). The chaotic 3-round primary rotation, the live config-upgrade commit, and the checkpoint-identity/config-identity/host-admin-status-survival checks ALL PASSED cleanly a second time (confirms bug-1477 stays fixed). The run then failed in this task's own dedicated end-of-round force_elect_node loop (the freshness-fence maneuver bullet 3 of task 8.6's Requirement 6.4 verification): force-elect ctrl-1 no-op'd correctly, but force-elect ctrl-2 hard-failed — after killing spoiler ctrl-3, submit_jobs against {ctrl-1,ctrl-2} got 0/3 acked across all 5 retry attempts. Root-caused (log-correlated across both nodes, not guessed): this is bug-1479 still not fully fixed — the fix's new primary-side NACK-routing code is correct but unreachable, because the standby's own AppendEntries-reject path (ha_apply_wal_catchup_entry/ha_apply_term_entry_via_library, ha.c ~3230-3232/3320-3321/3373-3374) sends no reply at all on reject, and ha_send_wal_ack_reply (the only WalAckReply sender in the controller) hardcodes success=1 — there is no production path that ever sends a NACK. See .wolf/buglog.json bug-1479 (occurrence 2, updated 2026-07-29) for the full live-log evidence trail. This is a pre-existing HA-replication defect, same family as bug-1466/1471/1472/1474/1475/1477/1478, out of THIS spec's boundary (task 8.6 only exercises the force_elect_node mechanism that exposes it, same relationship the task already had to bug-1477/bug-1472) — task 8.6's checkbox stays unchecked until it is fixed and a fresh run-ha-soak.sh --config-upgrade reports a clean PASS through all 3 force_elect_node rounds. New follow-up spec created to track the fix plus a broader systematic check for siblings of this same gap: .kiro/specs/raft-adapter-conformance/ (a raft.tla action-by-action conformance pass over the controller's Raft adapter — ha.c/repl.c/config_repl.c — distinct in scope from the existing raft-sim-conformance spec, which validates only src/lib/raft/raft_sim.c, the in-process library harness, and explicitly excludes this controller-adapter wire code).

• Task 8.6 — bug-1479 fully fixed 2026-07-29 (task now [x], clean live PASS): closing the gap identified in the re-run above needed TWO more compounding fixes beyond the primary-side NACK routing, each verified with its own dedicated live soak re-run because the first two looked complete after a clean unit-test pass yet still failed the identical live scenario for a NEW reason each time. Fix A (standby-side NACK sending): ha_apply_term_entry_via_library's logOk-reject path (the one live-reachable reject site; the other two in ha_apply_wal_catchup_entry are documented dormant/unreachable with real senders) now calls a new ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success) with success=0 instead of silently dropping the reject — ha_send_wal_ack_reply had hardcoded success=1 unconditionally, the ONLY WalAckReply sender anywhere in the controller, which made the primary's ha_raft_on_repl_nack unreachable dead code no matter how correct it was. New ha_wal_ack_send_hook (ha.h, XCE_TESTING) lets a unit test observe the NACK without a live endpoint (test_bug1479_ae_reject_sends_nack_not_silence, test_config_repl.c). Live re-run (~/tmp/xce_ha_bug1479_verify_v2) confirmed NACKs now genuinely reach the primary (ctrl-1.log: "WAL NACK from standby ... success=0 — marking catch-up") — but the SAME force-elect ctrl-2 scenario still failed, because of gap B. Fix B (catch-up stream ceiling): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) — a self-referential deadlock invisible from either side's code in isolation. New pure function ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn) (ha.c/ha.h), wired into ha_dispatch_repl_action_for_standby, widens the STREAM_WAL ceiling (and the ha_evaluate_repl_action convergence check gating it) to the leader's own log tail instead of just committed_lsn — matching raft.tla's real AppendEntries semantics (leaderCommit is advisory to the follower, never a send-cap) — while re-using bug-1478's exact guard (never widen onto a trailing local-only checkpoint's own unstreamable LSN slot) so this fix did not reopen that one. 4 new pure-function tests in test_ha_role.c (test_compute_repl_stream_ceiling_lsn_*). Both fixes RED-confirmed individually (each reverted to its pre-fix behavior in turn, re-run, confirmed the predicted failure, restored) before being combined. Final verification (~/tmp/xce_ha_bug1479_verify_v3, not deleted): controller_unit_test 1053/1053 PASS twice clean; live run-ha-soak.sh --config-upgrade PASSED end-to-end — all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7, never previously reached), final verdict "PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members." .kiro/specs/raft-adapter-conformance/'s concrete seed defect (missing NACK reply) is now fixed here; that spec's broader raft.tla action-by-action conformance audit remains valid, separate future work (fix B's own committed_lsn-vs-log-tail conflation is exactly the kind of second sibling gap that audit is meant to catch systematically).

• Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection): the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to lost_and_found on some members but not others, based on per-node restart timing — turned out to be aspirational, never-implemented code. Exhaustive grep across job.c/queue.c/worker.c/config_repl.c at both the pre-fix baseline commit (e0c5e8b7) and current HEAD confirmed: create_lost_and_found only ever creates the sentinel queue/host objects (and its entire body is #ifndef XCE_TESTING-gated); restore_job/restore_job_array never consult qmgr; restore_queue_state explicitly skips rather than re-homes when a queue is missing. .wolf/buglog.json's bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: schedule_checkpoint's pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker (term,lsn) boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within controller_unit_test, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as test_bug857_regression.c (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see git blame/review transcript for the full verification chain (schedule_checkpoint predates this spec via git merge-base --is-ancestor, zero lines of its body touched by tasks 1-7, persist_apply_pump_run's strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. Heads-up for tasks 8.6-8.9: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of test.c/src/frontend/src/tools) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live OP_upgradeConfig RPC against a running cluster.

• Task 8.4 round-1 rejection (real gap, caught by review): the first pass built configcompare_paths() (correct, well-tested against persist.c/config_repl.c's real workdir/name/conf/current+conf/cluster.bin.<hex> resolution convention) and linked configcompare.c into both xce_test_src and controller_unit_test_src in meson, but never registered it as a cmd_configcompare subcommand in testsuite/test.c — unlike its siblings walcompare/snapverify/snapdiff, each of which has a cmd_* wrapper in BOTH top_commands[] and client_commands[]. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke configcompare directly against each node's data directory. Fixed with a 25-line, test.c-only remediation mirroring cmd_snapdiff's exact shape; verified end-to-end via echo 'configcompare <dir1> <dir2>' | xce_test (checked $? for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family: linking the .c file into the meson source lists is NOT sufficient — it must also get a cmd_* wrapper + top_commands[]/client_commands[] entries in test.c, or it silently has no way to be invoked outside the unit test binary.

• Task 8.3: no new tests needed — the 10 tests already in test_config_repl.c from tasks 6.1/6.2 fully satisfy both bullets and Requirements 3.2/3.3/3.4/4.3/5.5 (verified independently by implementer and reviewer; zero-diff task). The two "fence sets" tests genuinely toggle different halves of upgrade_materialization_verify's condition (materialized vs. bytes_verified), not duplicates. Retry-trampoline automatic-firing (delaymgr actually running) is untested, matching the exact precedent depth of config_activate_retry_fn/config_schedule_activate_retry (also untested at that depth anywhere in this codebase) — judged out of scope, not a gap.

• Task 8.2: no new tests needed — task 3's test_upgrade_snapshot_restore.c already satisfies all 3 bullets and Requirements 1.10/1.11/5.8/7.1 (verified independently by both implementer and reviewer; zero-diff task). Requirement 7.1's full scenario is compositionally covered across two files (test_upgrade_snapshot_restore.c's real-rejoin test + test_persist_snapshot_entities.c's test_snapent_load_snapshot_restores_hosts_and_queues_end_to_end's real checkpoint-decode test), not by one continuous test — judged adequate since restore_host_state is a single invocation-site-independent function. design.md's Requirements Traceability table (row 7.1) under-credits this and should eventually be corrected to cite both files, not just test_upgrade_snapshot_restore.c — docs-only, not a testing gap.

• Task 6.2 round-1 rejection (real bug, caught by review): the first implementation wired the fence into both gates + a retry trampoline correctly, but upgrade_materialization_fence had no production path that could ever SET it from its zero-initialized default — every setter call site was gated on the fence already being set (closed loop). Fixed with two unconditional call sites: persist.c's apply_committed_entry WAL_CONFIG_MARKER case now unconditionally calls upgrade_materialization_verify(c) right after persist_checkpoint_marker_triggered returns (regardless of its rc), and myapp.c's boot sequence unconditionally calls it right after config_activate_committed (Req 5.2, crash/restart resume). New cross-module direction: persist.c now externs an ha.c function (mirrors the existing persist.c → worker.c extern pattern, no circular include). Lock-reentrancy re-verified safe (neither call chain touches apply_pump_lock).

• Task 6.1 process note: the Claude account's monthly API spend limit was hit mid-task (implementer subagent cut off after landing ha.c/ha.h but before tests/build). The orchestrator finished this task directly (no subagent implementer/reviewer pair) rather than stopping: verified the partial diff, wrote 4 tests into test_config_repl.c (reusing existing t42_/t43_ fixtures rather than creating test_upgrade_activation_fence.c — that file is task 8.3's own boundary per design.md, not 6.1's), built, ran the full suite (1001 Tests 0 Failures 0 Ignored), and self-reviewed the diff in lieu of an independent reviewer subagent. Flagging this so a future session/reviewer knows this task's diff did not go through the normal two-subagent review gate.

• Task 5 correction to design.md: design.md's "Marker Apply Trigger" section says "Call persist_checkpoint() directly" — this is unsafe as written. persist_apply_pump_run holds apply_pump_lock across the whole dispatch loop (including the apply_committed_entry call), and persist_checkpoint() itself re-acquires that same non-recursive mutex — a real self-deadlock on the live pump path (primary AND standby online-apply). Separately, last_applied_lsn/replay position only advance AFTER apply_committed_entry returns at both call sites, so pinning checkpoint_lsn from last_applied_lsn inside the marker case would mislabel the checkpoint with the previous entry's LSN, not the marker's own. Fixed via persist_checkpoint_write_locked(p, checkpoint_lsn) (no self-lock, explicit LSN param) + persist_checkpoint_marker_triggered(p, lsn), called from apply_committed_entry's new WAL_CONFIG_MARKER case with the marker's own lsn parameter. design.md should be updated to reflect this; not done as part of task 5 (docs-only, deferred).

• Task 5 → ha-term-safety/raft-log-replication finding (bug-1466, NOT this spec's to fix): persist_write_checkpoint_marker advances p->current_lsn directly, bypassing ha_primary_log_append/raft_log_append, breaking ha_primary_log_append's own documented p->current_lsn == s->log_len + 1 invariant (ha.c:4054-4060) once any checkpoint interleaves with primary appends. Pre-existing (periodic checkpoint already did this); task 5 just made the interleaving common instead of rare, surfacing it as real test flakiness for the first time. ha->raft.log_len feeds real RequestVote freshness comparisons (ha.c:258-261, 401-406) — NOT advisory-only as bug-1466's original text claimed; severity corrected in .wolf/buglog.json. Out of this spec's boundary (Raft log/term safety is ha-term-safety/raft-log-replication's territory) — flagging for whoever next owns that area.

• Task 3 → task 4 heads-up: restore_host_state(cluster_t*, const char*, uint32_t) (worker.c) is standalone and NOT yet wired into load_snapshot. Task 4 dispatches decoded HostAdminState frames to it. restore_host_state runs on the live vtask pool (called from ha_handle_snapshot_chunk → persist_load_snapshot), not a quiesced phase — a concurrent real first-time join for the same host is architecturally possible; verified safe by convergent key_find/makeWorker semantics (see worker.c comment above restore_host_state's key_find call), not by absence of concurrency. Task 4 should confirm this contract still holds once wired in.

• Task 2.3 → task 6.1 heads-up: read_active_config_bytes (persist.c) is the fully-inlined (no config_repl.c call) pattern for reading live config bytes off disk fresh each time — task 6.1's activation-fence verify function needs the identical "always re-read live bytes, never trust a cached/embedded copy" discipline (design.md is explicit about this). Reuse/mirror read_active_config_bytes rather than re-deriving the config_repl.c-avoidance reasoning from scratch.

• Task 2.2 → task 4 MUST re-enable — DONE (task 4): test_snapent_load_snapshot_restores_entities_before_advancing_position is un-ignored and passes for real now that load_snapshot's queue decode reads QueueAdminState capnp.

• test_snapent_checkpoint_queue_status_roundtrip's comments (~line 1019) are stale (still reference "format_version=1" / "task 3.2 not yet implemented" from an earlier spec pass) — harmless, not a task 2.2 defect, but worth cleaning up whenever that test is next touched.

• Task 2.1 → task 3 heads-up: host-frame collection is unconditional (every host in c->workermgr gets a frame, status masked to admin bits, may be 0) — confirmed correct against requirements.md/design.md by both implementer and reviewer. Consequence for task 3 (restore_host_state): a checkpoint may contain many zero-admin-status host frames on large clusters. Not required to special-case this, but consider whether restore_host_state should skip placeholder creation when admin_status == 0 on a miss, purely as a cost optimization — current design.md contract doesn't require it either way.

• Build environment (bug-918 recurrence 4, fixed in this worktree): nix develop was hard-broken — xbuild's capnpc package pins its src to the MUTABLE main ref of shen390s/c-capnproto with a hardcoded sha256, so it hash-mismatches every time that branch moves upstream (recurred 3x before in this same worktree per .wolf/buglog.json). Fixed durably by overriding capnpc's src in src/nixSupport/xce.nix to a pkgs.fetchFromGitHub pinned at an immutable commit (rev+sha256) instead of tracking main. Any future task in this spec that needs a fresh nix develop should not hit this again; if it does, the fix pattern is the same (re-pin to the then-current upstream commit).

• Test-runner gotcha: running controller_unit_test with default ASan settings hangs indefinitely in this sandbox — LeakSanitizer's stop-the-world thread scan never completes (confirmed: 50+ min wall clock, ~5s CPU, futex_do_wait). Always run with ASAN_OPTIONS=detect_leaks=0 (matches the project's own meson test registration). Do not wrap the binary in stdbuf or anything else that adds its own LD_PRELOAD — conflicts with ASan's preload requirement and the binary refuses to start.

• After any SNAPSHOT_FORMAT_VERSION bump: grep the whole testsuite/ tree for other hardcoded copies of the old format-version literal before considering the bump done — test_persist_snapshot_entities.c (a different, already-shipped spec's test file) had its own independent SNAPENT_TEST_FORMAT_VERSION constant that silently went stale after task 1's bump and broke 3 unrelated tests until caught.