feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply

Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:

  persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
  core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
  an explicit LSN and never acquiring apply_pump_lock itself
  persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
  checkpoint_in_progress CAS guard, calls the above with the marker's
  own lsn

This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).

New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.

Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.

Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3

.wolf/buglog.json

@@ -7756,18 +7756,20 @@
     "error_message": "nix develop fails: hash mismatch for c-capnproto fixed-output derivation (expected sha256-9dzSc3w+yxYzsF/rZe3qIarKOpjt2zvIyjuS5FXngIM=, got sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=)",
     "file": "src/flake.nix",
     "root_cause": "flake pins c-capnproto to the MUTABLE github archive/main.tar.gz; upstream main branch advanced so the tarball content/hash drifted, breaking the fixed-output derivation. Only pre-cached nix stores still build.",
-    "fix": "RESOLVED: user refreshed src/flake.lock; fresh nix develop builds (capnproto 1.2.0). A clean controller rebuild made the full controller test suite green (Ok:1 Fail:0, 12.7s) \u2014 the earlier wal_catchup -EAGAIN/decode failures were a STALE incremental-build artifact (schema compiled against the drifted toolchain), not fsync or a source regression.",
+    "fix": "RESOLVED: user refreshed src/flake.lock; fresh nix develop builds (capnproto 1.2.0). A clean controller rebuild made the full controller test suite green (Ok:1 Fail:0, 12.7s) \u2014 the earlier wal_catchup -EAGAIN/decode failures were a STALE incremental-build artifact (schema compiled against the drifted toolchain), not fsync or a source regression. RECURRENCE 2026-07-23 (config worktree, /xce-build controller build+report task for config-upgrade-snapshot Task 1 RED-phase capture): identical hash mismatch reproduced verbatim (same specified sha256-9dzSc3w+yxYzsF/rZe3qIarKOpjt2zvIyjuS5FXngIM= vs got sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk= for capnpc-1.0.1.drv source.drv) on FIRST invocation and again on an identical immediate RETRY (deterministic, not transient -- unlike the unrelated ninja-race bugs, retrying does not help here). Build never reached compilation; no xce_controller, controller_unit_test, or xce_test binaries produced; controller.capnp.c/h were not regenerated. This worktree (config, at .../worktrees/config) still has the pre-fix flake.lock/pin state -- the 2026-06-22 fix (refreshing src/flake.lock) was apparently not carried into this worktree, or upstream c-capnproto main.tar.gz drifted again since. NOT fixed this run (build+report only, no source/flake edits authorized by the task). Next step: refresh src/flake.lock in this worktree (nix flake update / lock the xbuild input) the same way the 2026-06-22 fix did, then retry the controller build. RECURRENCE 3 (2026-07-23, config worktree, config-upgrade-snapshot Task 1 implementer session): identical hash mismatch reproduced again AFTER a prior subagent had already logged occurrence 2 (build+report task, no fix attempted). This session went further: (1) confirmed `nix flake lock --update-input xbuild` in src/ is a NO-OP -- xbuild is already pinned to the latest available shen390s/my-nix-channel@master commit (073f527e), so re-locking flake.lock does NOT fix this recurrence (the previous 2026-06-22 fix's mechanism -- refresh flake.lock -- no longer applies once xbuild is already current); (2) confirmed via direct `nix develop` in the CANONICAL non-worktree repo (/home/rshen/projects/xce/xce/src) that it fails IDENTICALLY (same two sha256 hashes, same capnpc-1.0.1.drv path) -- this is a live, machine-wide, upstream c-capnproto main.tar.gz drift, NOT a worktree-isolation problem, and is not fixable from within this repo at all (the broken pin lives inside the external shen390s/my-nix-channel flake, at whatever commit is currently latest); (3) WORKAROUND discovered and used successfully: this machine's /nix/store already has REALIZED (fully built, not just .drv) output paths for capnpc-1.0.1 and capnproto-1.2.0 from a build that completed before the drift -- `find /nix/store -maxdepth 1 -iname '*capnp*'` surfaces them; invoking their bin/capnp and bin/capnpc-c binaries DIRECTLY (bypassing `nix develop`'s full devShell evaluation entirely) still works and successfully compiled a controller.capnp schema change end-to-end, generating correct bindings. This unblocks schema-level capnp verification even while the full `nix develop` shell is broken, but does NOT unblock a full project build/link -- controller_unit_test additionally needs `librtti` (fetched via a private SSH git flake input), which has NEVER been built/cached on this machine at all (no .drv or realized store path exists for it), so a full build+test run remains genuinely infeasible until either upstream fixes the c-capnproto pin or a machine with a pre-cached librtti+full toolchain is used.",
     "tags": [
       "nix",
       "flake",
       "c-capnproto",
       "env",
       "build",
-      "worktree-isolation"
+      "worktree-isolation",
+      "worktree"
     ],
     "related_bugs": [],
-    "occurrences": 1,
-    "last_seen": "2026-06-22T03:45:00Z"
+    "occurrences": 4,
+    "last_seen": "2026-07-24T08:00:00Z",
+    "recurrence_4_RESOLVED": "2026-07-24, config-upgrade-snapshot task 1 resumption session: RESOLVED (repo-local fix, no longer machine-wide-blocked as recurrence 3 concluded). Added a local override in src/nixSupport/xce.nix's capnpc derivation -- `src = pkgs.fetchFromGitHub { owner = \"shen390s\"; repo = \"c-capnproto\"; rev = \"e45ff682c4756ac5067cbac6d0f282a8ea5f0155\"; sha256 = \"sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=\"; }` -- pinning capnpc's source to an IMMUTABLE commit (the exact content the drifted `main` tarball happened to resolve to, so the hash matches what was already realized in the local /nix/store from recurrence 3's workaround) instead of upstream xbuild's package.nix pinning the MUTABLE `main` ref. This makes the fix independent of upstream's moving target permanently, not just until the next drift. Verified end-to-end: `nix develop --command meson setup xce.build --buildtype=debug -Db_sanitize=address --reconfigure && ninja -C xce.build controller/controller_unit_test` built clean (librtti and all other deps resolved fine -- recurrence 3's librtti-never-cached concern did not reproduce, this machine already had it), and the full `controller_unit_test` binary ran to completion: 978 Tests 0 Failures 0 Ignored, OK. Do-not-repeat: when a fixed-output-derivation hash mismatch traces to an upstream flake pinning a MUTABLE branch/ref (not a tag/commit), the durable fix is a local override.overrideAttrs pinning `src` to an explicit immutable commit + matching sha256 in the CONSUMING repo's own nix file -- do not wait on upstream or keep re-locking flake.lock (recurrence 3 already proved that path is a dead end once xbuild is already at its latest available commit)."
   },
   {
     "id": "bug-920",
@@ -13775,5 +13777,68 @@
     ],
     "occurrences": 1,
     "last_seen": "2026-07-23T06:58:42.000Z"
+  },
+  {
+    "id": "bug-1464",
+    "timestamp": "2026-07-23T09:50:00Z",
+    "error_message": "deploy-ha.sh FAIL: Error: xceadmin binary not found -- run 'xce-build.sh xceadmin' or 'nix build .#xcecore' first (after controller had just been rebuilt and xceadmin had been built successfully earlier in the same session)",
+    "file": "src/tools/xce-build.sh",
+    "root_cause": "setup_build() unconditionally does rm -rf \"$BUILD_DIR\" before every single-component build (controller, xceadmin, etc.), and all components share the same default BUILD_DIR=xce.build for the asan profile. Building controller after xceadmin (or vice versa) wipes the previous component entirely out of the shared build dir -- only the most-recently-built component survives. Only COMPONENT=all does one setup_build() + a single unfiltered meson compile, so it is the only invocation that leaves multiple components coexisting.",
+    "fix": "No code change; workflow fix -- when a downstream task (HA soak deploy-ha.sh) needs both xce_controller/xce_test AND the xceadmin venv present simultaneously in xce.build, run '/xce-build all' (or '/xce-build all asan') once instead of separate '/xce-build controller' + '/xce-build xceadmin' invocations, since the second call's setup_build() rm -rf wipes what the first call produced.",
+    "tags": [
+      "build",
+      "xce-build.sh",
+      "meson",
+      "xceadmin",
+      "controller",
+      "ha-soak",
+      "workflow"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-23T09:50:00Z"
+  },
+  {
+    "id": "bug-1465",
+    "timestamp": "2026-07-24T08:00:00Z",
+    "error_message": "test_snapent_checkpoint_job_content_roundtrip / test_snapent_checkpoint_concurrent_mutation_not_torn / test_snapent_checkpoint_queue_status_roundtrip: FAIL: checkpoint file must parse as a well-formed entity-bearing format_version=1 payload",
+    "file": "src/controller/testsuite/test_persist_snapshot_entities.c",
+    "root_cause": "config-upgrade-snapshot task 1 bumped persist.c's SNAPSHOT_FORMAT_VERSION from 1u to 2u (reserving the version for the HostAdminState/QueueAdminState capnp migration). test_persist_snapshot_entities.c (ha-entity-snapshots spec) defines its OWN independent SNAPENT_TEST_FORMAT_VERSION=1u constant to structurally validate checkpoint bytes it reads directly, bypassing load_snapshot by design (testing the write-side/task 2.1-2.2 boundary, not the restore boundary). persist_checkpoint() now always writes format_version=2, so the test's stale hardcoded 1u check correctly (if inconveniently) rejected every checkpoint as unrecognized -- this is the same fail-loud format-mismatch behavior Requirement 5.7 mandates, just tripping up a test written before the bump existed.",
+    "fix": "Updated SNAPENT_TEST_FORMAT_VERSION from 1u to 2u in test_persist_snapshot_entities.c (one #define site; both use sites at the job-frame and queue-frame readers already referenced the constant, not a separate hardcoded literal). Mirrors the same fix already applied to test_config_repl.c's three fake_fmt sites for the same version bump. No other code in that file changed -- it belongs to a different, already-shipped spec.",
+    "tags": [
+      "controller",
+      "persist.c",
+      "snapshot",
+      "format-version",
+      "test-regression",
+      "config-upgrade-snapshot",
+      "ha-entity-snapshots"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-24T08:00:00Z"
+  },
+  {
+    "id": "bug-1466",
+    "timestamp": "2026-07-24T15:36:56.329888Z",
+    "error_message": "test_config_repl.c: test_committed_marker_survives_checkpoint_trim and test_commit_advance_retry_clears_push_before_commit_fence flaked/failed (persist_checkpoint returned -EBUSY; new_mlsn <= committed_at_old) after wiring config-upgrade-snapshot task 5's WAL_CONFIG_MARKER apply case",
+    "file": "src/controller/testsuite/test_config_repl.c",
+    "root_cause": "Two related findings from wiring apply_committed_entry's new WAL_CONFIG_MARKER case (persist.c, config-upgrade-snapshot task 5), which for the first time makes applying a config marker synchronously trigger a real checkpoint: (1) persist_apply_pump_kick's dispatch is a CAS race between the calling thread (inside commit_submit's synchronous commit_resolve_pending) and persist_start's independent fsync_worker thread (via commit_on_durable_advance) -- when the fsync thread wins, commit_submit's WAIT mode returns having waited only for COMMIT, not for the marker's now-real APPLY+CHECKPOINT side effect on that other thread, so a fixture's own subsequent persist_checkpoint()/LSN bookkeeping can race the still-in-flight background checkpoint. This race was always latent (applying a marker used to be a no-op) but was invisible until task 5 gave it a real, observable side effect. (2) DEEPER pre-existing architectural finding: persist_write_checkpoint_marker (called by any checkpoint, periodic or marker-triggered) appends its local-only, never-replicated WAL_CHECKPOINT entry by incrementing persist_t's current_lsn directly, WITHOUT going through ha_primary_log_append/raft_log_append -- this violates ha_primary_log_append's own documented co-tracking invariant 'p->current_lsn == s->log_len + 1 always holds' (ha.c:4057). Consequence: config_append_marker's returned lsn (sourced from raft's own s->log_len, which under-counts once a checkpoint has interleaved) becomes unreliable as an exact physical-position identifier for any marker appended after an earlier marker's checkpoint has run -- exactly the scenario Requirement 4.4 (overlapping markers) requires. Verified this does NOT corrupt on-disk data or commit/durability correctness: commit_advance_locked's majority_candidate_locked (commit.c) is driven entirely by persist_t's own durable_lsn/current_lsn, never by raft's s->log_len (that library value is documented as advisory-only for commit-ceiling purposes, ha-term-safety commit-ceiling-optA). The bug is confined to callers trusting the RETURNED lsn from config_append_marker/ha_primary_log_append for precise identity/ordering once a checkpoint has occurred earlier in the WAL. This is pre-existing (the periodic WAL-threshold checkpoint trigger has always had the same bypass) but was never observable before because periodic checkpoints require >=100 WAL records to fire, essentially never happening between two closely-spaced marker appends in a unit test; task 5's marker-triggered checkpoint fires on EVERY marker commit, making it the common case. SEVERITY CORRECTION (2026-07-24, kiro-review of task 5): the original \"advisory-only... does NOT affect correctness\" framing UNDERSTATES real risk. ha->raft.log_len (the same value as s->log_len) is a direct input to Raft RequestVote freshness comparisons (ha.c:258-261, my_last_log_index; ha.c:401-406, my_last_log_term via persist_entry_term(c, log_len)) -- NOT debug/advisory-only. ha_primary_log_append itself documents (ha.c:4054-4060) that its own LSN-assignment correctness RELIES on p->current_lsn == s->log_len + 1 holding; once a checkpoint marker breaks that invariant, the next primary append receives a library-assigned index that may no longer match its true physical WAL position, so persist_entry_term() lookups keyed by that index can resolve to the wrong WAL entry. Confirmed via kiro-review independent trace, not yet reproduced as a live failure. Pre-existing (not introduced by task 5 -- the periodic WAL-threshold checkpoint trigger already had this bypass; task 5 only made the interleaving common instead of rare). Still correctly out of config-upgrade-snapshot task 5's boundary to fix -- flagging with corrected severity for whoever next owns ha-term-safety/raft-log-replication so it is not deprioritized based on the original advisory-only framing.",
+    "fix": "(1) Added t42_wait_applied() (mirrors test_apply_pump.c's tap_wait_applied) and called it from t43_commit_marker and directly in test_committed_marker_survives_checkpoint_trim, polling persist_last_applied_lsn() before any LSN-sensitive step following a marker commit, closing the fsync-thread race. (2) Changed t43_append_marker_uncommitted to return the TRUE physical lsn read directly from persist_t (c->persist.current_lsn - 1) instead of trusting config_append_marker's returned (raft-library-derived, now-unreliable) lsn -- matching how production code that needs precise positioning (persist.c's apply seam, which receives lsn from the WAL entry's own on-disk header) already effectively does it. Did NOT attempt to fix the deeper co-tracking invariant gap in ha.c/raft.c (out of config-upgrade-snapshot task 5's scope -- would require raft-log-replication/ha-term-safety spec owners to add a log_len resync after any local-only checkpoint-marker append on a raft leader); flagged as a follow-up finding in the task 5 status report instead.",
+    "tags": [
+      "checkpoint",
+      "needs-owner",
+      "severity-corrected",
+      "persist.c",
+      "vote-freshness",
+      "config-upgrade-snapshot",
+      "raft-log-index",
+      "architectural-finding",
+      "race-condition",
+      "test-flake"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-24T15:36:56.329908Z"
   }
-]
+]