diff --git a/.kiro/specs/config-upgrade-snapshot/tasks.md b/.kiro/specs/config-upgrade-snapshot/tasks.md index 2ecba1fe..f6afa55f 100644 --- a/.kiro/specs/config-upgrade-snapshot/tasks.md +++ b/.kiro/specs/config-upgrade-snapshot/tasks.md @@ -142,15 +142,16 @@ - _Requirements: 4.4, 7.2_ - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ -- [ ] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions +- [x] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions - Repeat the mid-catch-up WAL-trim assertion from the unit-level fixture, but under the full chaotic multi-kill soak rather than an isolated fixture - Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up - _Requirements: 7.4_ - _Depends: 7, 8.1, 8.2, 8.3, 8.4_ - - _Blocked: bug-1474 — pre-existing, out-of-spec-boundary HA connection-management defect (`repl_add_standby`'s first-empty-slot allocation, `repl.c:74-133`, unconditional `memset` at line 112; `primary_heartbeat_fn`'s reconnect sweep, `ha.c:9219-9254`; both blame to commits `7b396502`/`fa66d1ba9`/`8ef64a7e`, weeks before this spec). Sequence: killing a standby correctly frees its replicator slot and sets the bug-1147 `last_disconnected_lsn` retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an ALREADY-connected peer (connected via a promoted-vote-channel, not yet a heartbeat-task connection) as missing, spawns a redundant "late-joiner" reconnect, and `repl_add_standby` reallocates the just-freed slot — `memset`-clobbering the retention guard before any trim runs. Confirmed reproducible 2/2 against independent live 3-node clusters, and confirmed via `git blame` to be reachable identically under the pre-existing periodic-checkpoint trigger — nothing this spec added or changed. The live-soak round (`run_config_upgrade_wal_trim_round`, `run-ha-soak.sh`) is complete, committed, and correctly reproduces the violation; independently reviewed and confirmed sound (round design faithfully mirrors the unit fixture's mechanics; root-cause diagnosis independently re-derived from `repl.c`/`ha.c` plus raw preserved logs, not trusted from the implementer's report alone). No scoped adjustment to the round can dodge the race without suppressing the very live-cluster heartbeat behavior Req 7.4 requires testing under. Unblocks when bug-1474 is fixed by whoever owns HA connection/replicator-slot management — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade-wal-trim` to confirm clean PASS). See `.wolf/buglog.json` bug-1474 for the full forensic trace._ ## Implementation Notes +- **Task 8.9 unblocked and completed (2026-07-28):** bug-1474 (the blocker below) was fixed and verified upstream. Re-running `run-ha-soak.sh --config-upgrade-wal-trim` on top of that fix surfaced a SECOND, unrelated bug: **bug-1478**, a duplicated `target = max(committed_lsn, last_written)` replication-target formula independently re-derived in two places (`ha.c`'s heartbeat/stateReport dispatch and `config_repl.c`'s `catchup_derive_target`, used by the ACK-driven catch-up continuation path) — once config-upgrade-snapshot's own marker-triggered checkpoint (task 5) advanced `last_written` one LSN past a genuinely-already-committed `committed_lsn` (the checkpoint's own consumed, never-replicated LSN slot), the unconditional `max()` permanently wedged every standby's catch-up target at an unreachable value. Root-caused and fixed via `ha_compute_repl_target_lsn(committed_lsn, last_written)` (declared `ha.h`, defined `ha.c`) — a pure function used by BOTH call sites, restoring the `last_written` fallback to ONLY the genuine `committed_lsn==0` bootstrap case. Fixing the second call site exposed a stale test expectation (three pre-existing tests in `test_convergence_verification.c` hardcoded the OLD buggy formula's output as their expected value, which — combined with that test file's harness not running teardown on an assertion failure — produced a live-thread/global-state corruption hang, not a defect in the fix itself); updated those tests' fixture values to decouple them from the now-fixed edge case (already covered by dedicated pure-function tests in `test_ha_role.c`). Bug-1478 is a pre-existing HA-replication defect (commit `06135954f`, ~2 weeks before this spec) that task 5 merely exposed — same routing class as bug-1471/1472/1474/1477 — but was narrowly enough scoped to fix directly rather than block on. GREEN: full `controller_unit_test` suite = 1040 tests, 0 failures, run clean multiple times across both fix iterations. Live-verified: `run-ha-soak.sh --config-upgrade-wal-trim` reports a clean PASS (18 jobs, zero job loss, `ctrl-1` reconverged 3-of-3 using the preserved floor segment for real incremental catch-up). See `.wolf/buglog.json` bug-1474 and bug-1478 for full forensic detail. + - **Task 8.9 (live soak, genuinely requires chaos — unlike 8.5/8.7/8.8's unit-level redirections): infra complete, task BLOCKED on a new pre-existing bug (bug-1474).** Unlike every other 8.x subtask, this one's own bullet ("under the full chaotic multi-kill soak rather than an isolated fixture") makes the point of the task proving the mid-catch-up WAL-trim guard (`repl_trim_lsn`/`trim_wal_segments`, bug-1147) holds under REAL timing uncertainty, not just deterministic fixture control — so it could not be redirected to a cheaper unit-level proof the way 8.5/8.7/8.8 were. Built `run_config_upgrade_wal_trim_round()` + `--config-upgrade-wal-trim` in `run-ha-soak.sh` (deploys with small `--wal-segment-bytes`, kills a non-primary TARGET, forces ≥2 real segment rotations past its floor on survivors, submits a live config upgrade to fire the marker-triggered checkpoint's trim, then directly inspects survivors' on-disk `wal/` directory BEFORE restarting TARGET — mirrors the unit fixture's own seg-1-gone/seg-2-preserved/seg-3-preserved assertions against real files instead of a tmpdir fixture). Ran twice against independent live 3-node clusters: **both runs failed** — the floor segment protecting the killed target was trimmed both times. Root-caused (not guessed) to **bug-1474**, a general HA connection-management race entirely unrelated to config-marker triggering: killing a standby frees its replicator slot and sets the retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an already-connected peer as "missing" (comparing indices under two different conventions — relative `votes_idx` vs. absolute `peer_idx`, per the reviewer's independent finding), spawns a redundant reconnect, and `repl_add_standby`'s first-empty-slot allocation reuses the just-freed slot, `memset`-clobbering the guard before trim ever runs. Reviewer independently re-derived this from `repl.c:74-133`/`ha.c:9219-9254` plus raw preserved logs (not the implementer's quoted excerpts) and confirmed via `git blame` the responsible code predates this spec by weeks and is reachable identically under the OLD periodic-checkpoint trigger — genuinely out of this spec's boundary, same class of finding as bug-1471/1472. Reviewer also confirmed no scoped adjustment to the round's chaos could dodge the race without suppressing the live heartbeat behavior Req 7.4 requires testing under, so BLOCKED (not a redesign) is the correct call. **Spec-wide status after this task**: every implementable task (1 through 8.5, 8.7, 8.8) is `[x]`; the only two remaining open items (8.6, 8.9) are both blocked on real, independently-verified, out-of-spec-boundary HA defects this spec's own testing effort discovered (bug-1472, bug-1474) — neither blocks any OTHER task in this spec, and neither requires further config-upgrade-snapshot work once fixed upstream. - **Task 8.8 (unit-level, no live soak needed — deliberately excludes 8.6):** proves Req 4.4/7.2 (a second marker committing before the first's install completes must cleanly supersede it, never conflate) via two independently-built `t42`/`t43`-family fixtures in `test_config_repl.c`: `c_control` commits+applies+checkpoints marker A then marker B normally; `c_raced` holds the shared `checkpoint_in_progress` CAS across *both* markers' commit+apply (inlining `t43_commit_marker`'s 4-step body rather than changing its signature), so both markers' forced-checkpoint attempts hit `-EBUSY` and skip — genuinely "still installing." Mid-race assertions (captured before releasing the CAS) prove `mc->cluster_config` already reflects marker B while `cfgref_present` is still false and the fence is set; post-resolve assertions (after releasing the CAS, calling `persist_checkpoint()`, then an explicit `upgrade_materialization_verify()`) prove the settled state reflects only B. Discovered mid-implementation that whole-file `snapdiff_paths` byte-identity between the two fixtures' final checkpoints does NOT hold — `c_control`'s marker-A checkpoint physically consumes one extra WAL LSN (`persist_write_checkpoint_marker`, persist.c:1684) before marker B is appended, so B lands at a different absolute LSN in each fixture, and `write_snapshot`'s sentinel header embeds `checkpoint_lsn` at byte offset 8. Adapted to a field-aware comparison (full `memcmp` from byte 16 onward) plus `snapdiff_paths` diagnostic corroboration that offset 8 is the *only* divergence — a more precise proof than blind byte-identity, reviewer independently re-derived and confirmed this against `write_snapshot`'s actual layout rather than trusting the claim. One RED→GREEN cycle (`bug-1473`, test-construction gap not a production bug): the bare `persist_checkpoint()` resolve call doesn't itself trigger `upgrade_materialization_verify()` in a delaymgr-less unit fixture — production's self-healing (`schedule_upgrade_materialization_retry`, fired from `ha.c:2541-2542` on every commit-advance) only runs on a live cluster; the fixture needs the manual call, mirroring the existing 6.1-c fence tests' pattern. Reviewer's only findings were non-blocking style notes (could have used the public `upgrade_materialization_retry_fn` wrapper instead of calling `upgrade_materialization_verify` directly — functionally identical here since the fence was already independently confirmed set). Suite: 1027 → 1028 tests, 0 failures. diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 3379eabc..763f483a 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -14191,5 +14191,45 @@ ], "occurrences": 1, "last_seen": "2026-07-28T08:25:00Z" + }, + { + "id": "bug-1478", + "timestamp": "2026-07-28T09:00:00Z", + "error_message": "config-upgrade-snapshot task 8.9 live soak (run-ha-soak.sh --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim): bug-1474's floor-preservation fix confirmed working (floor segment seg-5 correctly preserved, all below-floor segments correctly trimmed) but the round still FAILED: \"cluster did not reconverge to 3-of-3 after restarting ctrl-1 -- the preserved floor segment did not actually suffice for real incremental catch-up\". ctrl-1 (TARGET, killed+restarted 3x) settles into a PERMANENT, zero-progress loop: its own raft AppendEntries exchanges cleanly reach match=22 with zero ae_reject events (heartbeat probe at prevIdx=22,n=0 -> success=1 match=22, repeating every ~3s forever), while ctrl-2 (primary) simultaneously and forever logs 'HA repl: standby[0] lsn=21 < committed_lsn=22 -> stream WAL batches' / 'ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby' every heartbeat cycle with zero effect -- an off-by-one disagreement (21 vs 22) between the two sides that never reconciles.", + "file": "src/controller/ha.c (repl_ack_decode_and_dispatch's heartbeat/stateReport reply branch, ~6379-6388 pre-fix -- the `target_lsn = max(committed_lsn, last_written)` computation feeding ha_dispatch_repl_action_for_standby)", + "root_cause": "ROOT-CAUSED with byte/log-precise evidence; NOT a recurrence of bug-1474 (independently re-confirmed working in this same run -- floor preservation and below-floor trim both correct), bug-1475 (independently re-confirmed working -- ctrl-1's own raft log_len/term-index bookkeeping across the checkpoint boundary is clean, zero ae_reject_logmismatch anywhere in its log, unlike bug-1475's own signature), or bug-1477 (ctrl-1 clearly DID wire its commit engine every boot -- multiple 'commit engine wired'/persist_recover lines confirmed, unlike bug-1477's ctrl-3 which never ran its own election at all). This is a THIRD, previously-undiscovered gap in the same 'WAL_CHECKPOINT breaks LSN co-tracking invariants' family already flagged (not fixed at its root) as bug-1466.\n\nGround truth via `xce_test waldump` on the preserved deploy dir (~/tmp/xce-ha-waltrim): ctrl-2's WAL shows lsn=21 as the WAL_CONFIG_MARKER (type UNKNOWN/40 bytes, matching ctrl-2's own log 'Config change committed (marker_lsn=21)') and lsn=22 as a WAL_CHECKPOINT (0-byte payload) -- confirming persist_write_checkpoint_marker (persist.c:1747, `hdr.lsn = p->current_lsn++`) fired immediately after the marker committed and consumed the very next LSN slot, exactly as apply_committed_entry's WAL_CONFIG_MARKER case (persist_checkpoint_marker_triggered) is designed to do (config-upgrade-snapshot task 5). Critically, ctrl-2's OWN commit_advance_locked (commit.c, the SOLE authoritative committed_lsn writer) never logs an advance past 21 anywhere in the run ('commit: committed_lsn 20 -> 21' is the LAST such line) -- the real, majority-gated persist.committed_lsn correctly and permanently stays at 21, since no standby can ever legitimately ack a local-only, non-replicated checkpoint entry.\n\nYet ha_dispatch_repl_action_for_standby's own log line reports 'committed_lsn=22' from the very next heartbeat cycle onward. Traced to repl_ack_decode_and_dispatch's stateReportMsg branch (ha.c): `last_written = c->persist.current_lsn - 1` (the RAW physical WAL tip, which unconditionally counts EVERY locally-appended entry including a local-only WAL_CHECKPOINT) is combined via `target_lsn = max(committed_lsn, last_written)` and THIS value -- not the real committed_lsn -- is what gets passed into ha_dispatch_repl_action_for_standby's committed_lsn parameter and printed in its log line. Once ctrl-2 wrote its checkpoint, last_written jumped to 22 and target_lsn followed, permanently outrunning the real committed_lsn by exactly one. Since a checkpoint's own consumed LSN slot is, by design, NEVER streamed (ha_stream_wal_to_standby) or applied (persist_wal_write_replicated / persist_wal_async both explicitly reject type==WAL_CHECKPOINT) by any standby, no standby's applied_lsn can EVER legitimately reach 22 -- ha_evaluate_repl_action's `standby_lsn < committed_lsn` check can never become false, so HA_REPL_CONVERGED is permanently unreachable for any standby once a local checkpoint has interleaved ahead of the next real committed advance. ha_stream_wal_to_standby is correctly re-invoked every heartbeat (per its own logic) but finds nothing new in [22,22] to send every single time -- a deterministic, permanent, zero-progress busy-loop, exactly matching this run's observed symptom (and ctrl-1's own applied_lsn -- correctly capped at 21 by ha_advance_applied_lsn/ha_apply_term_entry_via_library's own last-contiguously-held-LSN semantic, mirrored by ha_handle_snapshot_chunk's snapshot-install path setting applied_lsn=checkpoint_lsn never checkpoint_lsn+1 -- can never satisfy it).\n\nThe `target_lsn = max(committed_lsn, last_written)` fallback's OWN doc comment states its intended purpose narrowly: 'so standbys can catch up even when committed_lsn is stuck at 0 (no replicator-registered standbys to advance it)' -- a cluster-bootstrap convenience. But the code applies the max() UNCONDITIONALLY, not only when committed_lsn==0 -- broader than its own documented justification. This mismatch was latent and harmless until config-upgrade-snapshot task 5 introduced a LIVE, mid-operation local-only WAL entry (the marker-triggered checkpoint) that advances current_lsn without a corresponding real committed advance; bug-1466 already flagged the SAME persist_write_checkpoint_marker current_lsn bump as breaking raft's own log_len co-tracking invariant (affecting vote-freshness comparisons) -- this is a THIRD, distinct consequence of that same architectural gap, this time poisoning the replication-target computation instead. git blame confirms the `target_lsn` code (commit 06135954f, 2026-06-11) predates config-upgrade-snapshot's first commit by roughly two weeks -- a pre-existing HA replication-decision defect that config-upgrade-snapshot's task 5 merely exposed (same routing pattern as bug-1471/1472/1474/1477 before it), not something task 8.9 introduced.", + "fix": "FIXED (2026-07-28, narrowly-scoped TDD fix, in-scope to land here per explicit high-confidence + narrow-scope authorization -- unlike bug-1474's angle (a) and bug-1477, which were left for the HA-core spec owner): extracted the target_lsn computation out of repl_ack_decode_and_dispatch into a new small, pure, side-effect-free, directly-unit-testable function `ha_compute_repl_target_lsn(committed_lsn, last_written)` (declared in ha.h, defined in ha.c right after ha_process_standby_state). The fix restores the fallback to EXACTLY its own documented intent: `last_written` is used ONLY when `committed_lsn == 0` (genuine bootstrap, nothing ever committed) -- any other case returns `committed_lsn` unchanged, never leaking a trailing local-only checkpoint's consumed LSN into the convergence decision. repl_ack_decode_and_dispatch's call site now reads `target_lsn = ha_compute_repl_target_lsn(committed_lsn, last_written)`, replacing the old unconditional max(). No other function touched -- ha_evaluate_repl_action, ha_process_standby_state, ha_dispatch_repl_action_for_standby, ha_stream_wal_to_standby, persist_write_checkpoint_marker, and commit.c's majority_candidate_locked/commit_advance_locked are all confirmed correct and untouched (the bug was purely in what value got PASSED to them, not in their own logic).\n\nTDD: four new unit tests added to src/controller/testsuite/test_ha_role.c (all four call the new pure function directly, no capnp/network scaffolding needed): test_compute_repl_target_lsn_bootstrap_fallback (committed_lsn=0, last_written=42 -> 42, confirms the intended bootstrap behavior is preserved), test_compute_repl_target_lsn_zero_zero (0,0 -> 0, fresh-cluster edge case), test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn (this bug's EXACT reproduction shape: committed_lsn=21, last_written=22 -> must return 21, not 22 -- confirmed RED under the old unconditional-max code via a temporary revert, 'Expected 21 Was 22', then GREEN after restoring the fix), test_compute_repl_target_lsn_committed_already_ahead (100,50 -> 100, normal steady-state sanity check). GREEN: full controller_unit_test suite = 1040 Tests, 0 Failures, 0 Ignored, run twice clean (ASAN_OPTIONS=detect_leaks=0, nix devshell, `ninja -C src/xce.build controller/controller_unit_test`). RED confirmed independently: reverting only the new function's body to the old unconditional-max logic (leaving the new tests in place) reproduced exactly 1 failure at the predicted assertion ('Expected 21 Was 22'), then re-applying the fix restored 0 failures -- proving the test genuinely exercises this bug and the fix genuinely closes it, not a tautological test.\n\nNOT yet re-verified via a fresh live `run-ha-soak.sh --config-upgrade-wal-trim` run (recommended next step, mirroring bug-1474's own verification precedent, before considering config-upgrade-snapshot task 8.9 unblocked) -- this fix was validated at the unit level only within this investigation's scope.\n\n[UPDATE 2026-07-28, kiro-impl config-upgrade-snapshot task 8.9 verification] The initial fix landed only ONE of TWO call sites duplicating this formula: ha.c repl_ack_decode_and_dispatch (fixed, via the new ha_compute_repl_target_lsn) and config_repl.c catchup_derive_target (a SEPARATE, independently-duplicated inline copy of the exact same max(committed_lsn,last_written) formula, used by ha_walack_on_ack/catchup_lane_fn -- the ACK-driven catch-up continuation path, not the heartbeat/stateReport path the first fix covered). Rerunning the live --config-upgrade-wal-trim soak after the first fix STILL FAILED (\"cluster did not reconverge to 3-of-3... preserved floor segment did not actually suffice for real incremental catch-up\") -- confirmed via log correlation that ctrl-2 (primary) was stuck forever at ha_dispatch_repl_action_for_standby committed_lsn=22 while catchup_lane_fn's own INFO line independently reported target=22 (the OLD, unfixed formula) even though the heartbeat path's committed_lsn logging correctly showed 21. Root-caused to catchup_derive_target (config_repl.c:2617-2627, its own comment literally said \"re-derive the replication target exactly as ha.c:2771-2776 / D6\" -- a deliberate, now-stale duplication). Fixed by making catchup_derive_target call the SAME ha_compute_repl_target_lsn pure function instead of re-deriving the formula inline, so the two paths can never disagree again. Applying just this second-call-site fix then caused a NEW symptom: controller_unit_test hung indefinitely (0.8%% CPU, blocked not spinning) inside test_convergence_verification.c's test_walack_behind_target_schedules_one_continuation. Isolated via a controlled revert-and-rerun (confirmed clean 1040/1040 with the OLD formula restored, confirming the SECOND fix -- not something else -- was the trigger). Root cause of the hang: three pre-existing tests in that file (\"behind\", \"single\", \"respawn\") hardcoded committed_lsn=4 with appended_lsn=9, deliberately exercising the OLD buggy max(4,9)=9 behavior as their expected target -- with the fix, the correct target becomes 4, so acked_lsn=5/6 (used by the later tests) no longer legitimately reads as \"still behind\", and the FIRST test's own hardcoded assertion fails. Because this test file's harness (cv_setup/cv_teardown) does not run teardown from a signal-safe/finally-equivalent path, a failed TEST_ASSERT's Unity longjmp skips cv_teardown() entirely, leaking that test's live 8-thread commit_engine still bound to the soon-to-be-memset g_cv global -- the NEXT test's cv_setup() memset-ing g_cv out from under those still-running threads is what actually produced the observed permanent hang, not a defect in the production fix itself. Fixed by updating all three tests' committed_lsn from 4 to 9 (matching appended_lsn, so target=9 under EITHER formula) -- this decouples them from the bug-1478 edge case entirely (which already has dedicated, correct pure-function coverage in test_ha_role.c), preserving each test's TRUE original intent (continuation scheduling / single-flight / respawn-after-complete mechanics) with a minimal, surgical change. Stale docstrings referencing the old \"max(committed_lsn, last_written)\" formula were also corrected to reference ha_compute_repl_target_lsn. GREEN: full controller_unit_test suite = 1040 Tests, 0 Failures, run clean 2x after all fixes (plus the isolation-revert run confirming the pre-fix baseline was also clean). Live-verified: run-ha-soak.sh --config-upgrade-wal-trim now reports a clean PASS (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3 using the preserved floor segment for real incremental catch-up) -- config-upgrade-snapshot task 8.9 unblocked.", + "tags": [ + "ha.c", + "repl_ack_decode_and_dispatch", + "ha_dispatch_repl_action_for_standby", + "ha_evaluate_repl_action", + "target_lsn", + "last_written", + "committed_lsn", + "WAL_CHECKPOINT", + "persist_write_checkpoint_marker", + "bug-1466", + "bug-1474-fix-verified-good", + "bug-1475-fix-verified-good", + "not-bug-1477", + "config-upgrade-snapshot-task-8.9", + "permanent-catchup-wedge", + "off-by-one", + "waldump-confirmed", + "root-caused", + "fixed", + "narrowly-scoped-fix", + "tdd", + "red-green-verified" + ], + "related_bugs": [ + "bug-1466", + "bug-1474", + "bug-1475", + "bug-1477" + ], + "occurrences": 2, + "last_seen": "2026-07-28T09:52:00Z" } ] diff --git a/.wolf/cerebrum.md b/.wolf/cerebrum.md index c4a6193a..ef2186b6 100644 --- a/.wolf/cerebrum.md +++ b/.wolf/cerebrum.md @@ -563,5 +563,8 @@ Prior entry (2026-07-25) noted `repl_remove_standby` correctly sets the `last_di ## Key Learning (2026-07-25) — bug-1472 FIXED: `worker_from_capnp` now decodes `hook_detail` and applies UPDATE in place to an existing worker `worker_from_capnp` (`src/controller/worker.c`) previously returned `-EEXIST` unconditionally whenever the decoded worker's name was already registered, silently discarding the decoded fields regardless of whether the WAL entry was a JOIN, UPDATE, or DEAD. Fixed by reading the WAL_WORKER_CTRL payload's leading `hook_detail` discriminator (already decoded by the caller at `persist.c:3728-3729` but never passed down or re-read by `worker_from_capnp` itself) and branching: `HOOK_DETAIL_WORKER_JOIN` keeps the original -EEXIST no-op (regression-guarded); `HOOK_DETAIL_WORKER_UPDATE` now REPLACES (bitwise, not OR — see worker_ctrl's assignment-style `&= ~(...)`/`|= (...)` semantics, an OR-based fix would make HOST_OPEN unable to ever clear the closed bits again) `XCE_WORKER_ADMIN_MASK` bits on the existing worker under its own write lock, then returns 0 so `persist.c`'s existing generic "0 ⇒ fire the hook_detail-keyed notification" fallthrough (persist.c:3737-3785, unmodified) fires the correct `HOOK_DETAIL_WORKER_UPDATE` hook; `HOOK_DETAIL_WORKER_DEAD` is deliberately left as -EEXIST/no-op (see the two Key Learning entries above for why — filed as bug-1472-remainder, not silently ignored). `persist.c` needed NO changes — confirmed its WAL_WORKER_CTRL apply case already treats any non-EEXIST, non-negative `worker_from_capnp` return generically. TDD: RED confirmed by temporarily reverting just `worker.c` (via `git stash push -- <2 files>` / `git apply` round-trip, keeping the new tests) and re-running — exactly the 2 new UPDATE tests failed, the JOIN-duplicate regression-guard test already passed (proving it exercises the untouched path). GREEN: full suite 1030→1033 tests (this worktree's immediate pre-fix baseline, not a stale prior-session number), 0 failures, net +3 = exactly the 3 new tests (`test_ace_worker_ctrl_update_applies_to_existing_worker`, `test_ace_worker_ctrl_update_replaces_not_ors_admin_bits`, `test_ace_worker_ctrl_join_duplicate_existing_is_noop`), zero regressions. +## Key Learning (2026-07-28, config-upgrade-snapshot task 8.9, bug-1478 FIXED) — a function's own doc comment can describe a NARROWER behavior than the code actually implements; the gap between the two is where a real bug hides +`ha.c`'s `repl_ack_decode_and_dispatch` computed the primary's replication/convergence target as `target_lsn = max(committed_lsn, last_written)` (`last_written = persist.current_lsn - 1`, the RAW physical WAL tip), with a comment saying this exists "so standbys can catch up even when committed_lsn is stuck at 0" — but the code applied the `max()` UNCONDITIONALLY, not only in that `committed_lsn==0` case. This was harmless until config-upgrade-snapshot task 5's marker-triggered checkpoint (a local-only, never-replicated `WAL_CHECKPOINT` entry that `persist_write_checkpoint_marker` appends by bumping `current_lsn` directly, bypassing `raft_log_append`/the commit engine — the same architectural gap already flagged, unfixed at its root, as bug-1466) started interleaving a real, mid-operation local-only entry: `last_written` then permanently sat one LSN past the REAL, majority-gated `committed_lsn` (confirmed via `commit_advance_locked`'s own log never advancing past 21, while `ha_dispatch_repl_action_for_standby`'s log reported "committed_lsn=22" forever) — and since a checkpoint's own consumed LSN slot is BY DESIGN never streamed/applied by any standby, `HA_REPL_CONVERGED` became permanently unreachable, forever re-streaming an empty range every heartbeat. **Ground-truthed via `xce_test waldump `** (an existing REPL command in the `xce_test` binary, `src/controller/testsuite/waldump.c` — prints `LSN | TERM | TYPE | PLEN | PCRC | INFO` for every entry in a WAL directory, plus a gap/tip summary) directly on the preserved live-soak deploy dir, which is what turned "I think lsn=22 is the checkpoint" into "confirmed: lsn=21 is WAL_CONFIG_MARKER (type UNKNOWN/40B), lsn=22 is WAL_CHECKPOINT (0B)" — use this tool FIRST whenever a WAL-position/LSN-identity question comes up in a live soak investigation, instead of inferring entry types from log lines alone. **Fix pattern**: extracted the computation into a small pure function (`ha_compute_repl_target_lsn`) matching the comment's ACTUAL narrow intent (`committed_lsn==0` only) — trivially unit-testable without capnp/network scaffolding, unlike the `static` function it lives inside. **Do-Not-Repeat**: when a fallback/edge-case comment says "so X can happen even when Y" but the code's condition doesn't actually gate on Y, that mismatch is not a documentation nit — it means the fallback fires in cases beyond what was ever justified/reviewed, and is a prime hiding spot for exactly this class of bug once ANY new caller starts producing conditions the ORIGINAL narrow case never anticipated. Full finding + fix in `.wolf/buglog.json` bug-1478 (RED→GREEN verified: reverting only the new function's body reproduced the predicted single failure "Expected 21 Was 22"; 1040/1040 tests clean twice after restoring the fix). + ## Do-Not-Repeat (2026-07-28, bug-1476) — a buglog "FIXED... GREEN" claim is not proof; the working tree can be mid-TDD-cycle at the exact moment you resume Resumed `/kiro-impl config-upgrade-snapshot` with buglog.json's bug-1476 entry claiming "FIXED (2026-07-28)... GREEN: full controller_unit_test suite" — but the actual uncommitted diff on disk had a comment (config_repl.c:639-646) explicitly admitting `load_primary_config()` had been reverted to its RED-phase (no lock, decode-directly-into-shared-field) state for test construction and never restored. Running the full suite fresh immediately surfaced the real state: `test_bug1476_concurrent_load_primary_config_no_uaf_or_double_free` FAILED. The buglog note was written mid-cycle, describing the INTENDED fix, not the code actually sitting on disk. Fixed for real (lock now wraps the whole function body; decode targets a local temp, swapped in only after full validation), verified 4/4 clean full-suite runs before committing. **Do-not-repeat**: before committing or building on top of ANY buglog "FIXED"/GREEN claim for uncommitted work, re-run the actual test suite against the actual working tree first — never trust the log entry's narrative over a fresh run, especially when resuming a session that may have been interrupted mid-TDD-cycle. diff --git a/.wolf/memory.md b/.wolf/memory.md index ba9fcac7..96617d6b 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -1,8 +1,14 @@ # Memory > Chronological action log. Hooks and AI append to this file automatically. + +| 04:09 | Created brief.md and requirements.md for scheduler-preemption spec, extracting Req 6 from scheduler-framework/requirements.md | .kiro/specs/scheduler-preemption/brief.md, .kiro/specs/scheduler-preemption/requirements.md | created both files successfully | ~3500 > Old sessions are consolidated by the daemon weekly. +| 21:32 | full ASAN rebuild (`xce-build.sh all asan`) to pick up bug-1474 fix (commit c0b0c7d) + produce xceadmin binary; hit bug-1439 ninja parallel-build race TWICE (1st: capnp.h missing for hook.c.o, 2nd: NEW variant -lxce missing for apiserver's Go/CGO link), 3rd invocation succeeded clean | src/xce.build/controller/{xce_controller,xce_test}, src/xce.build/xceadmin/.venv/bin/xceadmin, src/xce.build/frontend/{apiserver/xce_apiserver,xcecli/xcecli} | PASS: meson Ok:9 Fail:0, scheduler 15/15, apiserver+xcecli Go tests ok, xceadmin pytest 166 passed; bug-1474 WARN string confirmed present in binary | ~95k | +| 15:15 | config-upgrade-snapshot task 5: added `case WAL_CONFIG_MARKER` in apply_committed_entry() calling new persist_checkpoint_marker_triggered() (never locks apply_pump_lock -- fixes self-deadlock vs persist_apply_pump_run's already-held lock; takes explicit marker lsn param -- fixes stale-checkpoint_lsn since last_applied_lsn isn't advanced yet at that point); factored persist_checkpoint_write_locked() shared by both persist_checkpoint() and the new trigger; -EBUSY treated as expected/self-resolving (TRACE, no retry); 6 new tests in test_upgrade_snapshot_apply.c drive the REAL persist_apply_pump_run/persist_recover call sites (not bare apply_committed_entry calls) to actually exercise both hazards; found+fixed a real regression in test_config_repl.c (2 tests) caused by the new automatic checkpoint racing fsync_worker + a genuine pre-existing raft-log-index co-tracking gap (persist_write_checkpoint_marker bypasses raft_log_append, so config_append_marker's returned lsn under-counts once a checkpoint has interleaved -- logged as bug-1466, does not corrupt commit/durability since that's driven by persist_t durable_lsn not raft's s->log_len) | src/controller/persist.c, src/controller/testsuite/test_upgrade_snapshot_apply.c (new), src/controller/testsuite/test_config_repl.c, src/controller/meson.build, src/controller/testsuite/controller_test.c | 997 Tests 0 Failures 0 Ignored OK (3x stable full-suite runs) | ~180k | +| 10:02 | config-upgrade-snapshot task 2.2 sequencing: task 2.2 (queue admin-state capnp write side) implemented+correct, but load_snapshot's queue decode is still task 4's job -- added TEST_IGNORE_MESSAGE to test_snapent_load_snapshot_restores_entities_before_advancing_position (real checkpoint+load round trip, m_queues=2) citing bug-847's precedent; confirmed via git-diff-revert-rebuild it fails Expected 0 Was -22 without the ignore; confirmed no other test does a real non-empty-queue checkpoint+load round trip (test_snapverify.c hand-builds its own raw-layout files bypassing write_snapshot; test_config_repl.c's persist_checkpoint call is never followed by a load) | controller/testsuite/test_persist_snapshot_entities.c | 981 Tests 0 Failures 1 Ignored OK | ~55k | | 07:30 | /xce-test-ha sequential mode: 10 rounds, 36 jobs, zero loss, all 3 nodes led (ctrl-1/2/3 primary coverage). Chaotic mode: 4 rounds complete+verified, round 5 discovery lag (bug-854 excluded), rounds 6-10 stalled (election lag, duplicate-node guard bug-1418). Sequential PASS verdict, chaotic hit edge-case cascade in overlapping-kill window | .claude/skills/xce-test-ha/run-ha-soak.sh | SEQUENTIAL=PASS, CHAOTIC=STALLED@R7 | ~150 | +| 09:10 | config-upgrade-snapshot task 2.1: added collect_host_frames + capnp_encode_host_admin_state (mirrors collect_queue_frames, no retention predicate -- every live host gets one frame, status masked to PERSIST_WORKER_ADMIN_MASK); wired into write_snapshot after queue trailer; 6 new tests (RED confirmed via git-stash, then GREEN) | controller/persist.c, controller/testsuite/test_persist_snapshot_entities.c | 981 Tests 0 Failures 0 Ignored OK | ~45k | | 07:26 | job-array-single-wal-record task 3.2: added `case WAL_JOB_ARRAY_SUBMIT` to apply_committed_entry() calling restore_job_array(), firing HOOK_DETAIL_JOB_NEW exactly once with the N-element out_new_jobs list; 3 new tests in test_apply_committed_entry.c (constructs-all-fires-once, idempotent-replay-no-extra-notify, forced-failure-fires-no-hook); RED confirmed via git-revert-rebuild-rerun (3 predicted failures), GREEN restored; 51/51 pass across 8-group regression sweep, 945/945 pass full suite, exit 0, no ASAN/leak signatures | src/controller/persist.c, src/controller/testsuite/test_apply_committed_entry.c | GREEN, task 3.2 READY_FOR_REVIEW | ~9500 | | 18:25 | Task 6.1 round 2 (raft-apply-decoupling): fixed round-1's job-state corruption bug — FAIL/postdone facts no longer bake a receipt-time state snapshot into WAL; wal_jobstate_t._pad now carries WAL_JOB_FAIL_STAGE_FLAG\|stage or WAL_JOB_UPDATE_POSTDONE_MARK, and apply_committed_entry() re-derives the merge at apply time via apply_job_fail_transition/new apply_job_postdone_transition against ACTUAL current state; new persist_submit_job_fact_ex(pad) primitive; fixed 2 pre-existing test_commit_barrier_reply.c tests whose assertions read the now-changed wire format directly (updated to check _pad + call the guard cores directly, not touching the 2 EXPECTED apply-pump-ordering failures reserved for task 7.3); new RED-then-GREEN regression test_apply_pump_fail_racing_finish_merges_against_current_state proves DONE\|PERR not RUN\|EXIT\|PERR; full suite census 904 PASS / 2 expected FAIL, zero new regressions; observed a rare (~2%) unrelated timing flake in the shared (reviewer-verified, off-limits) apply-pump-kick mechanism, not reproducible as a logic defect | wal_types.h, persist.h, cluster.h, job.c, persist.c, test_apply_pump.c, test_commit_barrier_reply.c | GREEN (2 pre-existing expected FAILs unchanged) | ~2200 | | 08:35 | Tasks 2.1+2.2+2.3 (ha-wal-replication): repl_add_standby idempotency (scan for dup ep before alloc); register standbys in raft_cb_become_leader (election path, via votes[i].peer->hostname/port) and on_standby_connected (late-joiner); deregister in vote_ep_err_handler and standby_ep_err_handler_primary; TDD RED→GREEN; all controller tests PASS | ha.c, repl.c, test_repl_unit.c | GREEN | ~250 | @@ -6082,3 +6088,639 @@ | 01:56 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/MEMORY.md | 1→2 lines | ~142 | | 01:56 | Session end: 33 writes across 20 files (cluster.c, sched.c, limits.yaml, project_bug1462_config_replication_fixed.md, MEMORY.md) | 39 reads | ~71360 tok | | 02:11 | Session end: 33 writes across 20 files (cluster.c, sched.c, limits.yaml, project_bug1462_config_replication_fixed.md, MEMORY.md) | 39 reads | ~71360 tok | + +## Session: 2026-07-23 02:32 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| + +## Session: 2026-07-23 02:44 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 03:02 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 22→20 lines | ~114 | +| 03:02 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_bug1418_prework_decision.md | — | ~670 | +| 03:03 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/MEMORY.md | modified 1464() | ~317 | +| 03:03 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/reference_xce_build_shared_dir_clobber.md | — | ~412 | +| 09:56 | re-verified bug-1418 fixed via fresh 10-round chaotic soak; rebuilt controller+xceadmin together (bug-1464: single-component /xce-build wipes shared xce.build dir); cleared config-upgrade-snapshot blocked_on | src/tools/xce-build.sh, .kiro/specs/config-upgrade-snapshot/spec.json, .wolf/buglog.json | PASS (10 rounds, 33 jobs, 0 divergence, 0 loss) | ~35k | +| 03:04 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | inline fix | ~210 | +| 03:15 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | inline fix | ~27 | +| 03:15 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | inline fix | ~635 | +| 03:16 | Session end: 7 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 12 reads | ~120466 tok | +| 03:23 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | inline fix | ~128 | +| 03:23 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~995 | +| 03:23 | Session end: 9 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 13 reads | ~122062 tok | +| 03:27 | Session end: 9 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 13 reads | ~122062 tok | +| 03:28 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~230 | +| 03:29 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~132 | +| 03:29 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→3 lines | ~292 | +| 03:29 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~150 | +| 03:29 | Session end: 13 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 13 reads | ~123345 tok | +| 03:37 | Session end: 13 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 13 reads | ~123345 tok | +| 03:39 | Session end: 13 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 13 reads | ~123345 tok | +| 03:44 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | inline fix | ~400 | +| 03:44 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~225 | +| 03:44 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~179 | +| 03:45 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~244 | +| 03:45 | Session end: 17 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 15 reads | ~160361 tok | +| 03:49 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~382 | +| 03:49 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | modified path() | ~1075 | +| 03:49 | Session end: 19 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 16 reads | ~162356 tok | +| 03:58 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | expanded (+9 lines) | ~448 | +| 03:58 | Session end: 20 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 16 reads | ~163643 tok | +| 06:06 | Session end: 20 writes across 5 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 16 reads | ~163643 tok | +| 06:08 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 8→8 lines | ~55 | +| 06:30 | Edited .kiro/specs/config-upgrade-snapshot/research.md | added error handling | ~4399 | +| 06:35 | Created .kiro/specs/config-upgrade-snapshot/design.md | — | ~11844 | +| 06:37 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~25 | +| 06:38 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~35 | +| 06:38 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~19 | +| 06:40 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 12→12 lines | ~74 | +| 11:35 | generated design.md for config-upgrade-snapshot via kiro-spec-design -y; light discovery across persist.c/ha.c/config_repl.c/worker.c confirmed apply_committed_entry is the single unified apply seam (apply_wal_entry dead, zero callers), config_serve_fence is a direct precedent for the new activation fence, persist_checkpoint is already force-callable | .kiro/specs/config-upgrade-snapshot/{design.md,research.md,spec.json} | design review gate passed (1 repair pass, bare-ID style), phase=design-generated | ~180k | +| 06:40 | Session end: 27 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 47 reads | ~526415 tok | +| 06:50 | Session end: 27 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 52 reads | ~532133 tok | +| 06:53 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~109 | +| 06:53 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~276 | +| 06:53 | Edited .kiro/specs/config-upgrade-snapshot/design.md | 1→2 lines | ~350 | +| 06:54 | Edited .kiro/specs/config-upgrade-snapshot/design.md | "config_repl.c" → "write_snapshot" | ~137 | +| 06:55 | Edited .kiro/specs/config-upgrade-snapshot/design.md | accessor() → fields() | ~73 | +| 06:55 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~263 | +| 06:58 | Edited .kiro/specs/config-upgrade-snapshot/design.md | added error handling | ~380 | +| 07:00 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~33 | +| 07:01 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~71 | +| 07:01 | Session end: 36 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 53 reads | ~545850 tok | +| 19:52 | Session end: 36 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 53 reads | ~545850 tok | +| 19:58 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 1→2 lines | ~223 | +| 19:59 | Edited .kiro/specs/config-upgrade-snapshot/design.md | 1→2 lines | ~97 | +| 20:00 | Edited .kiro/specs/config-upgrade-snapshot/design.md | 1→2 lines | ~501 | +| 20:00 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~146 | +| 20:00 | Session end: 40 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 53 reads | ~547236 tok | +| 20:03 | Edited .kiro/specs/config-upgrade-snapshot/requirements.md | 3→3 lines | ~131 | +| 20:04 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~59 | +| 20:05 | Edited .kiro/specs/config-upgrade-snapshot/design.md | 4→4 lines | ~33 | +| 20:05 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~43 | +| 20:06 | Session end: 44 writes across 7 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 53 reads | ~548060 tok | +| 20:13 | Created .kiro/specs/config-upgrade-snapshot/tasks.md | — | ~3830 | +| 20:14 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 4→5 lines | ~45 | +| 20:16 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 4→5 lines | ~64 | +| 20:17 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 4→5 lines | ~37 | +| 20:17 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 4→5 lines | ~62 | +| 20:28 | Created .kiro/specs/config-upgrade-snapshot/tasks.md | — | ~4238 | +| 20:37 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 2→2 lines | ~47 | +| 20:39 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 20→20 lines | ~110 | +| 20:40 | Session end: 52 writes across 8 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 57 reads | ~564805 tok | +| 20:44 | Session end: 52 writes across 8 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 60 reads | ~568628 tok | +| 20:48 | Session end: 52 writes across 8 files (spec.json, project_bug1418_prework_decision.md, MEMORY.md, reference_xce_build_shared_dir_clobber.md, requirements.md) | 60 reads | ~568628 tok | + +## Session: 2026-07-24 21:46 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 22:16 | Edited src/schema/controller.capnp | modified QueueConf() | ~354 | +| 22:16 | Edited src/include/typemap.h | modified struct() | ~302 | +| 22:16 | Edited src/controller/persist.c | modified 1() | ~279 | +| 22:17 | Edited src/controller/testsuite/test_config_repl.c | modified filler() | ~354 | +| 22:17 | Edited src/controller/testsuite/test_config_repl.c | 15→16 lines | ~279 | +| 22:18 | Edited src/controller/testsuite/test_config_repl.c | modified checkpoint() | ~299 | +| 22:19 | Edited src/controller/testsuite/test_config_repl.c | modified 1() | ~1190 | +| 22:20 | Edited src/controller/testsuite/test_config_repl.c | 3→6 lines | ~109 | +| 22:21 | Edited src/controller/persist.c | inline fix | ~28 | +| 05:29 | xce-build.sh controller: env broken, nix hash mismatch (capnpc-1.0.1 source.drv) recurrence of bug-918, deterministic on retry, no compile reached | .wolf/buglog.json, src/flake.lock | env-blocked, no test results captured | ~300 | +| 22:48 | Edited src/controller/persist.c | inline fix | ~10 | +| 22:53 | config-upgrade-snapshot task 1 complete: HostAdminState/QueueAdminState capnp structs (mapname'd onto new typemap.h C counterparts, matching capnp_snap_hdr_t precedent), SNAPSHOT_FORMAT_VERSION 1->2, 3 test fixtures updated to fake_fmt=2u, new test_snapshot_recv_old_format_version_rejected_fail_loud (fmt=1u rejected fail-loud) | src/schema/controller.capnp, src/include/typemap.h, src/controller/persist.c, src/controller/testsuite/test_config_repl.c | schema verified via direct capnp/capnpc-c binaries from cached /nix/store (bug-918 recurrence-3 blocked full nix develop/build+link+test; confirmed identical failure in canonical non-worktree repo, not worktree-specific; librtti never built on this machine at all so full link is separately infeasible) | ~9500 | +| 23:42 | Edited src/nixSupport/xce.nix | expanded (+11 lines) | ~175 | +| 23:56 | Session end: 11 writes across 5 files (controller.capnp, typemap.h, persist.c, test_config_repl.c, xce.nix) | 20 reads | ~296043 tok | +| 00:18 | Session end: 11 writes across 5 files (controller.capnp, typemap.h, persist.c, test_config_repl.c, xce.nix) | 20 reads | ~296043 tok | +| 00:46 | Session end: 11 writes across 5 files (controller.capnp, typemap.h, persist.c, test_config_repl.c, xce.nix) | 20 reads | ~296043 tok | +| 00:49 | Session end: 11 writes across 5 files (controller.capnp, typemap.h, persist.c, test_config_repl.c, xce.nix) | 20 reads | ~296043 tok | +| 00:59 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 1→2 lines | ~48 | +| 01:10 | config-upgrade-snapshot task 1 validated: nix build fixed (bug-918 capnpc pin) unblocked full controller_unit_test run; 3 pre-existing FAILs in test_persist_snapshot_entities.c root-caused to stale SNAPENT_TEST_FORMAT_VERSION=1u, fixed to 2u; full suite 978 Tests 0 Failures 0 Ignored OK | controller/persist.c, controller/testsuite/{test_config_repl,test_persist_snapshot_entities}.c, include/typemap.h, schema/controller.capnp, nixSupport/xce.nix | READY_FOR_REVIEW | ~0 | +| 01:20 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 2→2 lines | ~47 | +| 01:24 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~29 | +| 01:25 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified environment() | ~557 | +| 01:27 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/reference_nix_capnpc_pin_fix.md | — | ~486 | +| 01:27 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/MEMORY.md | 3→4 lines | ~79 | +| 01:45 | Edited src/controller/persist.c | added 1 condition(s) | ~316 | +| 01:45 | Edited src/controller/persist.c | added 9 condition(s) | ~1290 | +| 01:46 | Edited src/controller/persist.c | snap_job_frame_t() → snap_host_frame_t() | ~162 | +| 01:46 | Edited src/controller/persist.c | modified if() | ~115 | +| 01:47 | Edited src/controller/persist.c | added 1 condition(s) | ~165 | +| 01:47 | Edited src/controller/persist.c | added 4 condition(s) | ~341 | +| 01:49 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 7→8 lines | ~48 | +| 01:49 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified snapent_queue_status() | ~402 | +| 01:49 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified snapent_seed_hosts() | ~359 | +| 01:51 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified snapent_make() | ~525 | +| 01:52 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | added 1 condition(s) | ~61 | +| 01:54 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | added 20 condition(s) | ~3372 | +| 01:54 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 2→5 lines | ~96 | +| 02:16 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~17 | +| 02:16 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified 3() | ~168 | +| 02:38 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 15→18 lines | ~358 | +| 02:39 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified section() | ~570 | +| 02:39 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified for() | ~306 | +| 02:41 | Edited src/controller/persist.c | added 1 condition(s) | ~386 | +| 02:42 | Edited src/controller/persist.c | modified struct() | ~540 | +| 02:42 | Edited src/controller/persist.c | strdup() → capnp_encode_queue_admin_state() | ~207 | +| 02:43 | Edited src/controller/persist.c | modified trailer() | ~251 | +| 02:52 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified test_snapent_load_snapshot_restores_entities_before_advancing_position() | ~310 | +| 02:58 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | removed 17 lines | ~8 | +| 03:19 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~18 | +| 03:20 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 1→4 lines | ~220 | +| 03:44 | Edited src/controller/persist.c | added 15 condition(s) | ~1756 | +| 03:44 | Edited src/controller/persist.c | added 6 condition(s) | ~854 | +| 03:45 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 23→26 lines | ~138 | +| 03:45 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified struct() | ~107 | +| 03:46 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified cluster() | ~519 | +| 03:47 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | added 13 condition(s) | ~955 | +| 03:49 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified of() | ~2030 | +| 03:51 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 2→5 lines | ~107 | +| 03:58 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified of() | ~778 | +| 11:02 | Task 2.3 (config-upgrade-snapshot): added read_active_config_bytes + config-bytes checkpoint section, inline-duplicated (no config_repl.c call) per design.md precedent | src/controller/persist.c, src/controller/testsuite/test_persist_snapshot_entities.c | full suite 984 Tests 0 Failures 1 Ignored OK | ~9500 | +| 04:55 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~19 | +| 04:55 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified needs() | ~134 | +| 05:11 | Edited src/controller/include/worker.h | 3→4 lines | ~78 | +| 05:15 | Created src/controller/testsuite/test_upgrade_snapshot_restore.c | — | ~3581 | +| 05:16 | Edited src/controller/testsuite/controller_test.c | modified setUp() | ~34 | +| 05:16 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~28 | +| 05:16 | Edited src/controller/meson.build | 1→2 lines | ~25 | +| 05:17 | Edited src/controller/worker.c | added 5 condition(s) | ~1136 | +| 05:44 | Edited src/controller/worker.c | modified gap() | ~321 | +| 05:47 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~30 | +| 05:48 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 1→3 lines | ~180 | +| 06:07 | Edited src/controller/persist.c | 2→7 lines | ~119 | +| 06:08 | Edited src/controller/persist.c | added 6 condition(s) | ~2307 | +| 06:08 | Edited src/controller/persist.c | modified returns() | ~439 | +| 06:11 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified test_snapent_load_snapshot_restores_entities_before_advancing_position() | ~135 | +| 06:11 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified snapent_find_queue() | ~195 | +| 06:13 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | modified Test() | ~2256 | +| 06:14 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | expanded (+9 lines) | ~171 | +| 06:16 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | inline fix | ~16 | +| 06:16 | Edited src/controller/testsuite/test_persist_snapshot_entities.c | 2→3 lines | ~67 | +| 06:31 | Edited src/controller/testsuite/test_config_repl.c | modified checkpoint() | ~469 | +| 06:33 | Edited src/controller/testsuite/test_config_repl.c | 8→9 lines | ~120 | +| 07:07 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~19 | +| 07:07 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~63 | +| 07:33 | Edited src/controller/persist.c | added 3 condition(s) | ~2030 | +| 07:36 | Edited src/controller/persist.c | added 2 condition(s) | ~1172 | +| 07:47 | Created src/controller/testsuite/test_upgrade_snapshot_apply.c | — | ~5535 | +| 07:50 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 4→8 lines | ~100 | +| 07:50 | Edited src/controller/meson.build | 1→2 lines | ~25 | +| 07:51 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~26 | +| 07:51 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~22 | +| 07:58 | Created src/controller/testsuite/test_upgrade_snapshot_apply.c | — | ~6646 | +| 08:12 | Edited src/controller/testsuite/test_config_repl.c | added 1 condition(s) | ~516 | +| 08:13 | Edited src/controller/testsuite/test_config_repl.c | modified t43_commit_marker() | ~232 | +| 08:15 | Edited src/controller/testsuite/test_config_repl.c | expanded (+8 lines) | ~209 | +| 08:17 | Edited src/controller/testsuite/test_config_repl.c | expanded (+6 lines) | ~163 | +| 08:19 | Edited src/controller/testsuite/test_config_repl.c | expanded (+13 lines) | ~426 | +| 08:32 | Edited src/controller/testsuite/test_config_repl.c | modified t43_append_marker_uncommitted() | ~507 | +| 08:33 | Edited src/controller/testsuite/test_config_repl.c | reduced (-13 lines) | ~104 | +| 08:34 | Edited src/controller/testsuite/test_config_repl.c | reduced (-6 lines) | ~32 | +| 09:27 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~24 | +| 09:28 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified finding() | ~526 | +| 09:52 | Edited src/controller/ha.h | expanded (+38 lines) | ~833 | +| 09:52 | Edited src/controller/ha.h | modified upgrade_materialization_verify() | ~547 | +| 09:52 | Edited src/controller/ha.c | 2→3 lines | ~20 | +| 09:53 | Edited src/controller/ha.c | added 9 condition(s) | ~2592 | +| 19:39 | Edited src/controller/testsuite/test_config_repl.c | modified test_upgrade_materialization_verify_no_marker_clears_fence() | ~2975 | +| 19:40 | Edited src/controller/testsuite/test_config_repl.c | 2→7 lines | ~146 | +| 19:43 | Session end: 100 writes across 17 files (controller.capnp, typemap.h, persist.c, test_config_repl.c, xce.nix) | 48 reads | ~635552 tok | +| 19:46 | Edited src/controller/testsuite/test_config_repl.c | 6→11 lines | ~159 | +| 19:47 | Edited src/controller/testsuite/test_config_repl.c | 5→8 lines | ~96 | +| 19:50 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~31 | +| 19:51 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 1→3 lines | ~210 | +| 20:15 | Edited src/controller/ha.h | modified schedule_upgrade_materialization_retry() | ~308 | +| 20:15 | Edited src/controller/ha.c | added 11 condition(s) | ~1450 | +| 20:16 | Edited src/controller/ha.c | added 1 condition(s) | ~672 | +| 20:16 | Edited src/controller/ha.c | added 1 condition(s) | ~638 | +| 20:16 | Edited src/controller/ha.c | added 1 condition(s) | ~263 | +| 20:23 | Edited src/controller/ha.h | modified upgrade_materialization_retry_fn() | ~230 | +| 20:25 | Edited src/controller/testsuite/test_config_repl.c | modified test_upgrade_materialization_fence_enforced_by_dispatch_gate_and_promotion() | ~4128 | +| 20:26 | Edited src/controller/testsuite/test_config_repl.c | modified XCE_TESTING() | ~202 | +| 21:12 | Edited src/controller/persist.c | modified and() | ~321 | +| 21:12 | Edited src/controller/persist.c | modified and() | ~1068 | +| 21:13 | Edited src/controller/myapp.c | added 1 condition(s) | ~790 | +| 21:18 | Edited src/controller/testsuite/test_config_repl.c | modified test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss() | ~1902 | +| 21:19 | Edited src/controller/testsuite/test_config_repl.c | modified XCE_TESTING() | ~109 | +| 21:29 | Edited src/controller/testsuite/test_config_repl.c | expanded (+23 lines) | ~537 | +| 21:56 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~19 | +| 21:57 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified rejection() | ~263 | +| 22:09 | Edited src/controller/config_repl.c | modified XCE_TESTING() | ~538 | +| 22:10 | Edited src/controller/config_repl.c | modified XCE_TESTING() | ~283 | +| 22:12 | Edited src/controller/testsuite/test_commit_barrier_rollout.c | modified XCE_TESTING() | ~1796 | +| 22:14 | Edited src/controller/testsuite/test_commit_barrier_rollout.c | 2→3 lines | ~59 | +| 22:37 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~30 | +| 22:54 | Edited src/controller/persist.c | modified XCE_TESTING() | ~510 | +| 22:55 | Edited src/controller/persist.c | added 1 condition(s) | ~189 | +| 23:00 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | added error handling | ~172 | +| 23:00 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | added error handling | ~349 | +| 23:01 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | added error handling | ~2563 | +| 23:02 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 2→4 lines | ~64 | +| 23:07 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 4→6 lines | ~103 | +| 23:09 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 6→4 lines | ~61 | +| 23:32 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~24 | +| 23:52 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~20 | +| 23:53 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 1→3 lines | ~224 | +| 00:25 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~12 | +| 00:26 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 1→3 lines | ~183 | + +## Session: 2026-07-25 00:30 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 00:56 | Created src/controller/testsuite/configcompare.h | — | ~1107 | +| 00:57 | Created src/controller/testsuite/test_configcompare.c | — | ~2076 | +| 00:57 | Edited src/controller/meson.build | inline fix | ~44 | +| 00:57 | Edited src/controller/meson.build | 11→12 lines | ~80 | +| 00:57 | Edited src/controller/meson.build | 5→8 lines | ~135 | +| 00:58 | Edited src/controller/meson.build | 4→6 lines | ~50 | +| 00:58 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~28 | +| 00:58 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~22 | +| 01:02 | Created src/controller/testsuite/configcompare.c | — | ~2122 | +| 01:06 | Session end: 9 writes across 5 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 11 reads | ~200437 tok | +| 01:07 | Edited src/controller/testsuite/snapverify.h | modified struct() | ~769 | +| 01:08 | Edited src/controller/testsuite/snapverify.h | expanded (+6 lines) | ~283 | +| 01:08 | Edited src/controller/testsuite/snapverify.c | 4→8 lines | ~148 | +| 01:09 | Edited src/controller/testsuite/snapverify.c | added 1 condition(s) | ~154 | +| 01:10 | Edited src/controller/testsuite/snapverify.c | added 20 condition(s) | ~1878 | +| 01:11 | Edited src/controller/testsuite/snapverify.c | 10→13 lines | ~148 | +| 01:14 | Created src/controller/testsuite/test_snapverify.c | — | ~4696 | +| 01:24 | Session end: 16 writes across 8 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 11 reads | ~211107 tok | +| 01:45 | Session end: 16 writes across 8 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 15 reads | ~231413 tok | +| 01:48 | Session end: 16 writes across 8 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 15 reads | ~231413 tok | +| 01:49 | Edited src/controller/testsuite/test.c | 2→3 lines | ~72 | +| 01:49 | Edited src/controller/testsuite/test.c | 3→4 lines | ~44 | +| 01:49 | Edited src/controller/testsuite/test.c | 4→5 lines | ~101 | +| 01:50 | Edited src/controller/testsuite/test.c | 4→5 lines | ~104 | +| 01:50 | Edited src/controller/testsuite/test.c | added 1 condition(s) | ~299 | +| 08:55 | Registered configcompare as xce_test CLI subcommand (reviewer fix, task 8.4) | src/controller/testsuite/test.c | 1022/1022 tests pass; help lists configcompare; IDENTICAL/DIVERGENT smoke-tested via piped stdin, exit 0/1 | ~1800 | +| 01:57 | Session end: 21 writes across 9 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 16 reads | ~262456 tok | +| 02:04 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~30 | +| 02:05 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified with() | ~395 | +| 02:20 | Session end: 23 writes across 10 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 17 reads | ~269881 tok | +| 02:52 | Session end: 23 writes across 10 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 23 reads | ~279545 tok | +| 03:14 | Created src/controller/testsuite/test_bug857_regression.c | — | ~6090 | +| 03:15 | Edited src/controller/meson.build | 2→3 lines | ~35 | +| 03:16 | Edited src/controller/testsuite/controller_test.c | modified setUp() | ~45 | +| 03:16 | Edited src/controller/testsuite/controller_test.c | 4→5 lines | ~37 | +| 03:17 | Edited .kiro/specs/config-upgrade-snapshot/design.md | inline fix | ~46 | +| 03:18 | Edited .kiro/specs/config-upgrade-snapshot/design.md | modified XCE_TESTING() | ~823 | +| 03:23 | Session end: 29 writes across 12 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 24 reads | ~289399 tok | +| 03:25 | Session end: 29 writes across 12 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 25 reads | ~295489 tok | +| 03:39 | Edited .kiro/specs/config-upgrade-snapshot/design.md | 2→2 lines | ~251 | +| 03:41 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~23 | +| 03:42 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified XCE_TESTING() | ~845 | +| 04:01 | Session end: 32 writes across 12 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 32 reads | ~362257 tok | +| 04:28 | Edited src/controller/cluster.c | modified fix() | ~544 | +| 04:30 | Edited src/lib/request.c | added 4 condition(s) | ~836 | +| 04:32 | Session end: 34 writes across 14 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 34 reads | ~384133 tok | +| 04:32 | Edited src/include/request.h | expanded (+13 lines) | ~304 | +| 04:32 | Edited src/include/xceapi.h | expanded (+11 lines) | ~248 | +| 04:33 | Edited src/lib/xceapi.c | added 4 condition(s) | ~256 | +| 04:33 | Edited src/include/client.h | expanded (+8 lines) | ~192 | +| 04:36 | Edited src/lib/client.c | added 1 condition(s) | ~262 | +| 04:41 | Session end: 39 writes across 19 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 36 reads | ~387384 tok | +| 04:41 | Edited src/controller/testsuite/test.c | 1→3 lines | ~32 | +| 04:42 | Edited src/controller/testsuite/test.c | 2→3 lines | ~35 | +| 04:42 | Edited src/controller/testsuite/test.c | 2→5 lines | ~87 | +| 04:43 | Edited src/controller/testsuite/test.c | added 10 condition(s) | ~1079 | +| 04:59 | Edited src/controller/testsuite/test.c | added 1 condition(s) | ~184 | +| 05:00 | Session end: 44 writes across 19 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 38 reads | ~388902 tok | +| 05:03 | Session end: 44 writes across 19 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 38 reads | ~389017 tok | +| 05:15 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified find_xceadmin() | ~3566 | +| 05:16 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 2→3 lines | ~48 | +| 05:17 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified 6() | ~423 | +| 05:18 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+8 lines) | ~247 | +| 05:39 | Session end: 48 writes across 20 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 39 reads | ~427537 tok | +| 05:46 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified PASS() | ~460 | +| 05:47 | Edited .claude/skills/xce-test-ha/SKILL.md | inline fix | ~45 | +| 05:48 | Edited .claude/skills/xce-test-ha/SKILL.md | 3→5 lines | ~123 | +| 05:49 | Edited .claude/skills/xce-test-ha/SKILL.md | modified PASS() | ~1054 | +| 06:03 | Session end: 52 writes across 21 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 41 reads | ~430394 tok | +| 06:58 | Verified prior agent's task 8.6 work (bug-1468 fix, Parts A-D) coherent via read | cluster.c, request.c, xceapi.c, client.c, test.c | all correct, matches server-side decode expectations | ~15k | +| 06:59 | Added --config-upgrade dispatch block (array-soak-style: hard-fail internal, no scan_bug924) | run-ha-soak.sh | inserted after array-chaos-retry block | ~600 | +| 06:59 | Documented --config-upgrade mode: env-var table + XCE_HA_MODE list + full subsection | SKILL.md | matches array-chaos-retry depth | ~900 | +| 07:00 | Full clean build (ninja -C src/xce.build), verified fresh binaries | xce_controller, xce_test, xceadmin | 0 errors | ~2k | +| 07:01 | Ran live --config-upgrade soak: /tmp/xce-ha-configupgrade | run-ha-soak.sh | upgrade genuinely interleaved w/ 3 chaotic kills, committed, 6/6 jobs survived, but snapdiff FAILED: ctrl-1 checkpoint 51569B vs ctrl-2/3 51621B | ~8k | +| 07:10 | Root-caused divergence: do_upgrade_config_with_bytes never calls config_activate_committed on originating primary, so c->myconf queue table stays stale until a later event; ctrl-1 killed before that could happen | config_repl.c, ha.c, persist.c (read-only) | logged bug-1469, live empirical repro of bug-857's hypothesis via a 3rd distinct mechanism | ~20k | +| 07:11 | Logged bug-1469 to buglog.json (related to bug-857, bug-1466); did NOT fix (out of task 8.6 scope, tasks 1-7 territory) | buglog.json | valid JSON, 788 entries | ~3k | +| 06:25 | Session end: 52 writes across 21 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 42 reads | ~547481 tok | +| 06:54 | Edited src/controller/testsuite/test_config_repl.c | modified soak() | ~1961 | +| 06:55 | Edited src/controller/testsuite/test_config_repl.c | modified XCE_TESTING() | ~83 | +| 06:56 | Session end: 54 writes across 22 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 43 reads | ~550527 tok | +| 06:57 | Edited src/controller/persist.c | modified and() | ~1627 | +| 06:59 | Edited src/controller/persist.c | expanded (+14 lines) | ~253 | +| 07:06 | Edited src/controller/persist.c | added 1 condition(s) | ~354 | +| 07:24 | Edited src/controller/persist.c | added 1 condition(s) | ~1792 | +| 14:26 | Ran full /xce-build controller (nix develop + xce-build.sh); 1st attempt failed on transient meson/ninja capnp-header race (unrelated to code), 2nd clean rebuild compiled clean but controller_unit_test reproducibly FAILED test_commit_advance_retry_clears_push_before_commit_fence (4/4 runs) — logged bug-1470, needs triage before task 8.6 soak | src/xce.build/controller/*, .wolf/buglog.json | build green, 1 test red | ~large | +| 07:26 | Edited src/controller/persist.c | removed 82 lines | ~59 | +| 07:28 | Session end: 59 writes across 23 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 43 reads | ~559722 tok | +| 07:37 | Edited src/controller/testsuite/test_config_repl.c | expanded (+17 lines) | ~750 | +| 07:38 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+20 lines) | ~834 | +| 07:41 | Edited .claude/skills/xce-test-ha/SKILL.md | expanded (+9 lines) | ~257 | +| 07:47 | Edited src/controller/persist.c | expanded (+6 lines) | ~317 | +| 14:50 | Task 8.6 wrap-up: fixed persist.c config_activate_committed ordering (bug-1469 pt1), diagnosed+documented deep qmgr live-sync gap (bug-1469 pt2, needs owner), diagnosed+documented 1-byte host-frame divergence (bug-1471, needs owner), switched soak's upgrade-target change from add-queue to update-queue to avoid the unrelated gap | persist.c, run-ha-soak.sh, SKILL.md, buglog.json | 3 real bugs found via live soak; 1 fixed+verified, 2 documented for tasks 1-7 owners | ~1200 | +| 07:56 | Session end: 63 writes across 23 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 44 reads | ~570970 tok | +| 08:05 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/d3f40b2e-8c73-4820-8ad3-1400be8761ac/scratchpad/update_bug1471.py | — | ~439 | +| 08:12 | Session end: 64 writes across 24 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 44 reads | ~571409 tok | +| 08:43 | Session end: 64 writes across 24 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 44 reads | ~571727 tok | +| 08:47 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified progress() | ~1227 | +| 08:48 | Session end: 65 writes across 24 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 44 reads | ~573041 tok | +| 09:11 | Session end: 65 writes across 24 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 44 reads | ~573041 tok | +| 16:36 | bug-1471 root-caused: byte-precise capnp decode of preserved checkpoints proves the divergent byte is inside HostAdminState.status (0x00500000 CLOSED\|CLOSED_BY_ADMIN on ctrl-1 vs 0x0 on ctrl-2/ctrl-3), not capnp padding (calloc-zeroed, verified via disasm of libCapnC_Runtime.so create_local). True cause: worker_from_capnp (worker.c:827-856, predates spec by ~2.5mo) EEXIST-short-circuits WAL_WORKER_CTRL UPDATE/DEAD applies on standbys, silently dropping admin-status changes for already-joined hosts -- general host-control replication bug, out of config-upgrade-snapshot scope. Filed as bug-1472, cross-referenced from bug-1471. No production code changed (out-of-scope, not patched). | .wolf/buglog.json | root-caused, routed out-of-scope, buglog updated | ~large | +| 09:39 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 6→7 lines | ~482 | +| 09:44 | Session end: 66 writes across 24 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 46 reads | ~609994 tok | +| 10:09 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 5→8 lines | ~145 | +| 10:10 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | added 4 condition(s) | ~2731 | +| 10:10 | Edited src/controller/testsuite/test_upgrade_snapshot_apply.c | 2→3 lines | ~44 | +| 10:16 | Session end: 69 writes across 25 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 48 reads | ~635904 tok | +| 10:20 | Edited src/controller/testsuite/test_config_repl.c | 5→9 lines | ~107 | +| 10:22 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~3652 | +| 10:22 | Edited src/controller/testsuite/test_config_repl.c | 3→3 lines | ~49 | +| 10:23 | Edited src/controller/testsuite/test_config_repl.c | modified XCE_TESTING() | ~127 | +| 10:24 | Edited src/controller/testsuite/test_config_repl.c | 3→6 lines | ~97 | +| 10:48 | Session end: 74 writes across 25 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 49 reads | ~640223 tok | +| 10:53 | Session end: 74 writes across 25 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 49 reads | ~640223 tok | +| 10:57 | Session end: 74 writes across 25 files (configcompare.h, test_configcompare.c, meson.build, controller_test.c, configcompare.c) | 49 reads | ~640223 tok | +| 11:16 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~23 | +| 11:19 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified twice() | ~663 | + +## Session: 2026-07-25 11:26 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 11:59 | Edited src/controller/testsuite/test_config_repl.c | added 12 condition(s) | ~4947 | +| 12:01 | Edited src/controller/testsuite/test_config_repl.c | 2→7 lines | ~133 | +| 12:11 | Edited src/controller/testsuite/test_config_repl.c | expanded (+12 lines) | ~455 | +| 12:11 | Edited src/controller/testsuite/test_config_repl.c | 4→8 lines | ~127 | +| 12:46 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | inline fix | ~31 | +| 12:47 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified 8() | ~691 | +| 13:09 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified 9() | ~731 | +| 13:11 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+15 lines) | ~330 | +| 13:12 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 2→3 lines | ~51 | +| 13:12 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified 9() | ~369 | +| 13:14 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified wal_dir() | ~4510 | +| 13:15 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified PASS() | ~463 | +| 13:18 | Edited .claude/skills/xce-test-ha/SKILL.md | added error handling | ~1699 | +| 13:19 | Edited .claude/skills/xce-test-ha/SKILL.md | inline fix | ~52 | +| 13:19 | Edited .claude/skills/xce-test-ha/SKILL.md | 3→6 lines | ~178 | +| 13:35 | Ran live 3-node soak --config-upgrade-wal-trim (2x, reproducible) | .claude/skills/xce-test-ha/run-ha-soak.sh | FAIL both runs — real bug found, not test-script flaw | ~9000 | +| 13:50 | Root-caused live-soak failure via ctrl-2.log forensics: late-joiner slot-reuse clobbers bug-1147 disconnect guard | src/controller/ha.c, src/controller/repl.c | root cause confirmed, 2/2 reproducible | ~6500 | +| 13:52 | Logged bug-1474 (late-joiner reconnect clobbers disconnected-standby's WAL-trim guard via slot reuse) | .wolf/buglog.json | new entry appended, NOT fixed (out of task 8.9 scope) | ~2100 | +| 13:53 | Task 8.9 implementation: run_config_upgrade_wal_trim_round added and verified sound; task BLOCKED on bug-1474 | .claude/skills/xce-test-ha/run-ha-soak.sh, SKILL.md, .wolf/buglog.json, .wolf/cerebrum.md | infra correct, live PASS blocked on separate HA defect | ~1200 | +| 14:09 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 3→4 lines | ~513 | +| 14:10 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified 9() | ~794 | +| 20:04 | Edited src/controller/testsuite/test_repl_unit.c | modified RED() | ~1771 | +| 20:04 | Edited src/controller/testsuite/test_repl_unit.c | 3→6 lines | ~89 | +| 20:10 | Edited src/controller/repl.c | added 3 condition(s) | ~656 | +| 20:12 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/d3f40b2e-8c73-4820-8ad3-1400be8761ac/scratchpad/update_bug1474.py | — | ~1227 | +| 20:14 | Edited src/controller/worker.c | added 1 condition(s) | ~842 | +| 20:15 | Edited src/controller/testsuite/test_apply_committed_entry.c | modified tace_build_worker_ctrl_pkt() | ~2190 | +| 20:15 | Edited src/controller/testsuite/test_apply_committed_entry.c | 2→5 lines | ~92 | +| 20:16 | bug-1474 FIXED: repl_add_standby two-pass slot allocation (virgin-slot-first, then oldest-guard fallback) — TDD RED (2 new tests fail pre-fix) → GREEN (1030 tests, 0 failures, no regressions vs 1028 baseline) | src/controller/repl.c, src/controller/testsuite/test_repl_unit.c, .wolf/buglog.json | fixed + verified, buglog "fix" field updated (forensic history preserved) | ~large | +| 20:29 | bug-1472 FIXED: worker_from_capnp() now decodes hook_detail and, for an already-existing worker, REPLACES (not ORs) XCE_WORKER_ADMIN_MASK bits from the decoded HOOK_DETAIL_WORKER_UPDATE payload instead of unconditionally returning -EEXIST — admin hostctrl -c/-o now propagates to standbys/own-WAL-replay. HOOK_DETAIL_WORKER_JOIN unchanged (-EEXIST no-op, regression-guarded). HOOK_DETAIL_WORKER_DEAD deliberately deferred (investigated: w->host.resvals live resource-value zarray isn't in the Worker capnp wire schema at all — logged as bug-1472-remainder). TDD: RED (2 new UPDATE tests fail against pre-fix code, confirmed) → GREEN (fix applied, full suite 1030→1033 tests, 0 failures, net +3 = exactly the new tests, zero regressions). persist.c untouched (its WAL_WORKER_CTRL apply case already generically handles any non-EEXIST-non-negative return). | src/controller/worker.c, src/controller/testsuite/test_apply_committed_entry.c, .wolf/buglog.json, .wolf/anatomy.md | fixed + verified, buglog bug-1472 "fix" field updated + bug-1472-remainder filed (forensic history preserved) | ~large | +| 20:48 | Edited src/controller/repl.c | added 1 condition(s) | ~454 | +| 21:03 | bug-1474 review REMEDIATED: reviewer rejected on 1 finding (Pass 2 guard-slot-reuse memset was silent, no WARN, undercutting the fix's own "make silent destruction visible" goal). Added used_fallback_slot tracking + WARN("repl_add_standby: reusing guard-holding slot...") right before the memset, mirroring repl_broadcast's existing WARN convention. No new test needed (log line only) — verified via `strings` on the built binary + both existing bug-1474 tests still passing. Hit one transient shared-build-dir artifact (stale worker.c.o from a concurrent sibling agent's in-flight build caused 2 spurious bug-1472-test failures on one run) — resolved via `touch` + rebuild, not a real regression. Final: full suite 1033 tests, 0 failures. | src/controller/repl.c, .wolf/buglog.json, .wolf/anatomy.md | remediated + verified, buglog bug-1474 "fix" field appended with [UPDATE] note (forensic history preserved) | ~large | +| 04:44 | bug-1474 live-soak RE-VERIFICATION (run-ha-soak.sh --config-upgrade-wal-trim, post-fix, deploy /home/claude/tmp/xce-ha-waltrim3): CORE ASSERTION PASSED — floor segment seg-...05.wal correctly PRESERVED, below-floor seg-01/03/04 correctly TRIMMED, active seg-...15.wal PRESENT (confirmed both via script log and independent on-disk `find`). New WARN("repl_add_standby: reusing guard-holding slot...") fired 2x on ctrl-2 (primary) — confirms the still-open ha.c duplicate-reconnect defect (bug-1474 fix-angle-(a), NOT fixed) is still spawning duplicate/late-joiner reconnects, exercising the fix's Pass-2 exhausted-fallback path for real; despite that, the specific segment under test stayed protected this run. Round's OVERALL verdict still FAIL, but at a LATER, unrelated step ("cluster did not reconverge to 3-of-3 after restarting ctrl-1") — root-caused to TWO separate, pre-existing, out-of-scope defects, NEITHER the original bug-1474 symptom: (1) NEW crash — ctrl-3 ASAN heap-use-after-free in config_repl.c's load_primary_config/config_activate_committed/handle_repl_config_push path (logged as bug-1476); (2) ctrl-1's post-restart catch-up got stuck at lsn=21 needing just lsn=22, ctrl-2 retrying identically every ~3s with zero progress — exact match to already-logged, not-yet-fixed bug-1475 (prevIdx off-by-one in ha_dispatch_repl_action_for_standby/ha_stream_wal_to_standby). First soak attempt (subagent-launched via Bash run_in_background from inside an Agent-tool call) died silently mid-election when the subagent's own turn ended — background process did NOT survive subagent completion despite setsid nohup being used internally by run-ha-soak.sh's start_node (the DRIVER script itself, launched via the Bash tool's run_in_background flag rather than a raw shell `setsid nohup ... & disown`, did not survive). Relaunched directly from the main agent thread with explicit `setsid nohup ... & disown` + polling loop, which survived and completed cleanly. | .wolf/buglog.json (bug-1476 new entry) | bug-1474's own fix confirmed GOOD at live-soak level; 2 separate follow-on blockers identified for task 8.9 (bug-1475, bug-1476) | ~large | + +## Session: 2026-07-26 22:31 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 22:36 | Created reports/jul19-jul26-2026.md | — | ~3715 | +| 22:36 | Session end: 1 writes across 1 files (jul19-jul26-2026.md) | 4 reads | ~237768 tok | +| 22:54 | Session end: 1 writes across 1 files (jul19-jul26-2026.md) | 11 reads | ~414061 tok | +| 23:03 | Edited src/controller/ha.c | added 2 condition(s) | ~900 | +| 23:03 | Edited src/controller/config_repl.c | modified 1475() | ~613 | +| 23:04 | Edited src/controller/config_repl.c | modified if() | ~38 | +| 23:12 | Edited src/controller/testsuite/test_config_repl.c | modified 1475() | ~578 | +| 23:13 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~281 | +| 03:41 | Session end: 6 writes across 4 files (jul19-jul26-2026.md, ha.c, config_repl.c, test_config_repl.c) | 13 reads | ~448433 tok | +| 03:42 | Edited src/controller/testsuite/test_config_repl.c | added 1 condition(s) | ~2472 | +| 03:43 | Edited src/controller/testsuite/test_config_repl.c | 2→3 lines | ~56 | + +## Session: 2026-07-27 01:17 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 01:50 | Edited .kiro/specs/scheduler-framework/requirements.md | inline fix | ~222 | +| 01:51 | Edited .kiro/specs/scheduler-framework/requirements.md | inline fix | ~295 | +| 01:52 | Edited .kiro/specs/scheduler-framework/requirements.md | modified pointer() | ~2660 | +| 01:52 | Edited .kiro/specs/scheduler-framework/requirements.md | modified enum() | ~1084 | +| 01:53 | Edited .kiro/specs/scheduler-framework/requirements.md | expanded (+89 lines) | ~1478 | +| 01:54 | Edited .kiro/specs/scheduler-framework/spec.json | inline fix | ~11 | +| 01:54 | Session end: 6 writes across 2 files (requirements.md, spec.json) | 8 reads | ~40857 tok | +| 02:07 | Session end: 6 writes across 2 files (requirements.md, spec.json) | 9 reads | ~40857 tok | +| 02:09 | Edited .kiro/specs/scheduler-framework/requirements.md | added 2 condition(s) | ~862 | +| 02:10 | Edited .kiro/specs/scheduler-framework/requirements.md | 14→16 lines | ~450 | +| 02:11 | Edited .kiro/specs/scheduler-framework/requirements.md | modified v2() | ~679 | +| 02:11 | Edited .kiro/specs/scheduler-framework/requirements.md | 6→4 lines | ~277 | +| 02:12 | Edited .kiro/specs/scheduler-framework/requirements.md | modified struct() | ~325 | +| 02:12 | Edited .kiro/specs/scheduler-framework/requirements.md | modified next_job() | ~767 | +| 02:16 | Session end: 12 writes across 2 files (requirements.md, spec.json) | 10 reads | ~46842 tok | +| 02:18 | Edited .kiro/specs/scheduler-framework/requirements.md | expanded (+32 lines) | ~904 | +| 02:18 | Edited .kiro/specs/scheduler-framework/requirements.md | 7→7 lines | ~277 | +| 02:19 | Edited .kiro/specs/scheduler-framework/requirements.md | modified pending() | ~957 | +| 02:19 | Session end: 15 writes across 2 files (requirements.md, spec.json) | 10 reads | ~49731 tok | +| 02:28 | Edited .kiro/specs/scheduler-framework/requirements.md | 12→7 lines | ~405 | +| 02:28 | Edited .kiro/specs/scheduler-framework/requirements.md | 2→2 lines | ~137 | +| 02:28 | Edited .kiro/specs/scheduler-framework/requirements.md | "project → queue → job" → "xce_job_picker_iface_t" | ~458 | +| 02:29 | Edited .kiro/specs/scheduler-framework/requirements.md | "ha-consistency-invariants" → "builtin_fcfs" | ~314 | +| 02:29 | Edited .kiro/specs/scheduler-framework/requirements.md | 12→12 lines | ~282 | +| 02:30 | Edited .kiro/specs/scheduler-framework/requirements.md | modified Preemption() | ~285 | +| 02:31 | Session end: 21 writes across 2 files (requirements.md, spec.json) | 10 reads | ~52078 tok | +| 02:35 | Created .kiro/specs/scheduler-framework/brief.md | — | ~6523 | +| 02:36 | Session end: 22 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~59067 tok | +| 02:56 | Edited .kiro/specs/scheduler-framework/requirements.md | added 1 condition(s) | ~1009 | +| 02:56 | Session end: 23 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~60293 tok | +| 03:09 | Edited .kiro/specs/scheduler-framework/requirements.md | 3→3 lines | ~96 | +| 03:09 | Edited .kiro/specs/scheduler-framework/requirements.md | 3→1 lines | ~13 | +| 03:10 | Edited .kiro/specs/scheduler-framework/requirements.md | removed 69 lines | ~87 | +| 03:11 | Edited .kiro/specs/scheduler-framework/requirements.md | added 1 condition(s) | ~1185 | +| 03:11 | Session end: 27 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~61772 tok | +| 03:22 | Edited .kiro/specs/scheduler-framework/requirements.md | added 2 condition(s) | ~1754 | +| 03:23 | Session end: 28 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~64586 tok | +| 03:29 | Edited .kiro/specs/scheduler-framework/requirements.md | slots() → slot() | ~62 | +| 03:30 | Session end: 29 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~65362 tok | +| 03:33 | Edited .kiro/specs/scheduler-framework/requirements.md | added 1 condition(s) | ~572 | +| 03:33 | Edited .kiro/specs/scheduler-framework/requirements.md | modified pointer() | ~366 | +| 03:34 | Edited .kiro/specs/scheduler-framework/requirements.md | 3→5 lines | ~74 | +| 03:35 | Edited .kiro/specs/scheduler-framework/requirements.md | modified pointer() | ~1197 | +| 03:35 | Edited .kiro/specs/scheduler-framework/requirements.md | 14 → 15 | ~27 | +| 03:36 | Edited .kiro/specs/scheduler-framework/requirements.md | 15 → 16 | ~15 | +| 03:36 | Session end: 35 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~68162 tok | +| 03:40 | Edited .kiro/specs/scheduler-framework/requirements.md | modified directly() | ~472 | +| 03:41 | Edited .kiro/specs/scheduler-framework/requirements.md | 7→8 lines | ~135 | +| 03:41 | Session end: 37 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~69033 tok | +| 03:43 | Edited .kiro/specs/scheduler-framework/requirements.md | 3→8 lines | ~374 | +| 03:43 | Session end: 38 writes across 3 files (requirements.md, spec.json, brief.md) | 10 reads | ~69453 tok | +| 03:50 | Edited .kiro/specs/scheduler-framework/requirements.md | 4→4 lines | ~227 | +| 03:51 | Edited .kiro/specs/scheduler-framework/requirements.md | modified via() | ~450 | +| 03:51 | Session end: 40 writes across 3 files (requirements.md, spec.json, brief.md) | 11 reads | ~70526 tok | +| 03:55 | Edited .kiro/specs/scheduler-framework/requirements.md | modified directly() | ~842 | +| 03:56 | Edited .kiro/specs/scheduler-framework/requirements.md | 2→6 lines | ~121 | +| 03:56 | Session end: 42 writes across 3 files (requirements.md, spec.json, brief.md) | 11 reads | ~72053 tok | +| 04:01 | Session end: 42 writes across 3 files (requirements.md, spec.json, brief.md) | 13 reads | ~72053 tok | +| 04:04 | Edited .kiro/specs/scheduler-framework/requirements.md | expanded (+16 lines) | ~707 | +| 04:05 | Session end: 43 writes across 3 files (requirements.md, spec.json, brief.md) | 13 reads | ~72887 tok | +| 19:36 | Edited src/controller/testsuite/test_config_repl.c | 10→14 lines | ~213 | +| 19:41 | Session end: 44 writes across 4 files (requirements.md, spec.json, brief.md, test_config_repl.c) | 15 reads | ~262461 tok | +| 19:52 | Edited src/controller/config_repl.c | modified if() | ~180 | +| 19:53 | Edited src/controller/ha.c | modified persist_durable_floor() | ~231 | +| 19:53 | Edited src/controller/ha.c | modified if() | ~167 | +| 19:53 | Edited src/controller/ha.c | expanded (+9 lines) | ~223 | +| 19:53 | Edited src/controller/ha.c | added error handling | ~166 | +| 19:54 | Edited src/lib/raft/raft.c | modified on_get_entry_term() | ~243 | +| 19:54 | Edited src/lib/raft/raft.c | expanded (+6 lines) | ~209 | +| 19:54 | Edited src/lib/raft/raft.c | 3→4 lines | ~56 | +| 19:54 | Edited src/lib/raft/raft.c | 4→6 lines | ~87 | +| 19:55 | Edited src/lib/raft/raft.c | 1→2 lines | ~22 | +| 19:59 | Edited src/controller/testsuite/test_config_repl.c | 4→5 lines | ~49 | +| 19:59 | Edited src/controller/testsuite/test_config_repl.c | expanded (+8 lines) | ~167 | +| 19:59 | Edited src/controller/testsuite/test_config_repl.c | 4→5 lines | ~48 | +| 20:02 | Session end: 57 writes across 7 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 18 reads | ~391509 tok | +| 20:05 | Edited src/controller/persist.c | added error handling | ~1320 | +| 20:07 | Edited src/controller/config_repl.c | removed 13 lines | ~27 | +| 20:07 | Edited src/controller/ha.c | removed 18 lines | ~46 | +| 20:07 | Edited src/controller/ha.c | modified if() | ~67 | +| 20:07 | Edited src/controller/ha.c | reduced (-9 lines) | ~63 | +| 20:08 | Edited src/controller/ha.c | reduced (-9 lines) | ~36 | +| 20:08 | Edited src/lib/raft/raft.c | reduced (-9 lines) | ~92 | +| 20:08 | Edited src/lib/raft/raft.c | reduced (-6 lines) | ~98 | +| 20:08 | Edited src/lib/raft/raft.c | 4→3 lines | ~36 | +| 20:08 | Edited src/lib/raft/raft.c | 6→4 lines | ~49 | +| 20:09 | Edited src/lib/raft/raft.c | 2→1 lines | ~5 | +| 20:09 | Edited src/controller/testsuite/test_config_repl.c | 5→4 lines | ~35 | +| 20:09 | Edited src/controller/testsuite/test_config_repl.c | reduced (-8 lines) | ~38 | +| 20:09 | Edited src/controller/testsuite/test_config_repl.c | 2→1 lines | ~7 | +| 20:14 | Edited src/controller/testsuite/test_config_repl.c | modified side() | ~880 | +| 20:14 | Edited src/controller/testsuite/test_config_repl.c | modified RED() | ~45 | +| 20:15 | Edited src/controller/ha.c | added error handling | ~416 | +| 20:16 | Edited src/controller/config_repl.c | expanded (+18 lines) | ~410 | +| 03:22 | bug-1475: root-caused via real fprintf instrumentation (streamer/JIT-resync/raft logOk) that the two already-landed fixes left applied_lsn stuck at 0 | src/controller/persist.c (TERM_INDEX_UNSET sentinel fix), src/controller/ha.c + config_repl.c (cross-ref comments only), src/controller/testsuite/test_config_repl.c (doc comments), .wolf/buglog.json | FIXED: full suite 1034/1034 (was 1033/1034); term-index growth silently reclassified a skipped WAL_CHECKPOINT lsn's term from absent(0) to WAL_LEGACY_TERM(1) once a later lsn was recorded, causing sender/receiver to disagree on prev_log_term | ~140k | +| 20:28 | Session end: 75 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 19 reads | ~419701 tok | +| 20:48 | Session end: 75 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 21 reads | ~423430 tok | +| 20:52 | Session end: 75 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 22 reads | ~459739 tok | +| 20:55 | Session end: 75 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 22 reads | ~459739 tok | +| 20:57 | Created .kiro/specs/scheduler-core/spec.json | — | ~202 | +| 20:57 | Created .kiro/specs/scheduler-allocation/spec.json | — | ~210 | +| 20:58 | Created .kiro/specs/scheduler-preemption/spec.json | — | ~213 | +| 21:00 | Created .kiro/specs/scheduler-framework/spec.json | — | ~211 | +| 21:01 | Session end: 79 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 30 reads | ~467627 tok | +| 21:03 | Edited src/controller/persist.c | added error handling | ~835 | +| 21:04 | Created .kiro/specs/scheduler-allocation/brief.md | — | ~3600 | +| 21:05 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~2390 | +| 21:06 | Edited src/controller/testsuite/test_config_repl.c | 3→7 lines | ~91 | +| 21:06 | Edited src/controller/testsuite/test_config_repl.c | 2→3 lines | ~58 | +| 21:07 | Created .kiro/specs/scheduler-preemption/brief.md | — | ~3104 | +| 21:07 | Created .kiro/specs/scheduler-preemption/requirements.md | — | ~2437 | +| 21:10 | Session end: 86 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 30 reads | ~484551 tok | +| 21:12 | Edited src/controller/testsuite/test_config_repl.c | expanded (+27 lines) | ~618 | +| 21:14 | Edited src/controller/testsuite/test_config_repl.c | 3→6 lines | ~103 | +| 21:15 | Created .kiro/specs/scheduler-core/brief.md | — | ~2121 | +| 21:16 | Session end: 89 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 30 reads | ~488105 tok | +| 21:16 | Edited src/controller/testsuite/test_config_repl.c | expanded (+12 lines) | ~628 | +| 21:17 | Created .kiro/specs/scheduler-allocation/requirements.md | — | ~6636 | +| 21:17 | Edited src/controller/persist.c | 6→3 lines | ~35 | +| 21:18 | Created .kiro/specs/scheduler-core/requirements.md | — | ~9658 | +| 04:18 | Created brief.md and requirements.md for scheduler-allocation spec at .kiro/specs/scheduler-allocation/ | brief.md, requirements.md | Both files written, scoped to allocation-only concerns (plan_iter, host_iter, parjob, resreq cache, seqjob rewrap) | ~18000tok | +| 21:21 | Session end: 93 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 31 reads | ~506322 tok | +| 21:22 | Edited src/controller/persist.c | 3→6 lines | ~120 | +| 21:27 | Session end: 94 writes across 8 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 31 reads | ~506427 tok | +| 21:35 | Edited src/controller/persist.c | 10→9 lines | ~81 | +| 21:36 | Edited src/controller/persist.c | 9→10 lines | ~74 | +| 21:46 | Edited src/include/persist.h | expanded (+6 lines) | ~231 | +| 21:46 | Edited src/controller/ha.c | 6→8 lines | ~160 | +| 21:51 | Session end: 98 writes across 9 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 32 reads | ~507012 tok | +| 22:06 | Edited src/include/myconf.h | 7→8 lines | ~35 | +| 22:06 | Edited src/include/myconf.h | expanded (+10 lines) | ~236 | +| 22:06 | Edited src/controller/myconf.c | modified myconf_fini() | ~92 | +| 22:06 | Edited src/controller/main.c | added 1 condition(s) | ~132 | +| 22:10 | Edited src/controller/config_repl.c | modified XCE_TESTING() | ~229 | +| 22:10 | Edited src/controller/config_repl.c | added 1 condition(s) | ~169 | +| 22:11 | Edited src/controller/config_repl.h | expanded (+10 lines) | ~221 | +| 22:11 | Session end: 105 writes across 13 files (requirements.md, spec.json, brief.md, test_config_repl.c, config_repl.c) | 40 reads | ~554405 tok | +| 22:11 | Edited src/controller/testsuite/test_config_repl.c | modified t43_attach_myconf() | ~235 | +| 22:14 | Edited src/controller/testsuite/test_config_repl.c | added 3 condition(s) | ~2535 | +| 22:14 | Edited src/controller/testsuite/test_config_repl.c | 3→5 lines | ~72 | +| 22:18 | Edited src/controller/testsuite/test_config_repl.c | expanded (+9 lines) | ~173 | +| 22:20 | Edited src/controller/config_repl.c | added 1 condition(s) | ~2124 | + +## Session: 2026-07-28 23:29 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 00:28 | Edited .kiro/specs/scheduler-core/requirements.md | modified cbuf() | ~376 | +| 00:29 | Edited .kiro/specs/scheduler-core/requirements.md | added 2 condition(s) | ~646 | +| 00:29 | Edited .kiro/specs/scheduler-core/requirements.md | "sched_job_picker()" → "jbuf" | ~146 | +| 00:30 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~104 | +| 00:30 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~94 | +| 00:30 | Edited .kiro/specs/scheduler-core/requirements.md | added 1 condition(s) | ~886 | +| 00:31 | Edited .kiro/specs/scheduler-core/requirements.md | 4→5 lines | ~332 | +| 00:32 | Edited .kiro/specs/scheduler-core/requirements.md | modified next_job() | ~999 | +| 00:32 | Edited .kiro/specs/scheduler-core/requirements.md | modified struct() | ~247 | +| 00:33 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~141 | +| 00:33 | Session end: 10 writes across 1 files (requirements.md) | 13 reads | ~174206 tok | +| 00:33 | Session end: 10 writes across 1 files (requirements.md) | 13 reads | ~174206 tok | +| 00:40 | Edited src/controller/config_repl.c | modified XCE_TESTING() | ~1087 | + +## Session: 2026-07-28 00:54 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 01:05 | Edited src/controller/config_repl.c | modified XCE_TESTING() | ~1156 | +| 01:14 | Session end: 1 writes across 1 files (config_repl.c) | 4 reads | ~133541 tok | +| 01:16 | Session end: 1 writes across 1 files (config_repl.c) | 4 reads | ~133541 tok | +| 01:18 | Session end: 1 writes across 1 files (config_repl.c) | 4 reads | ~133541 tok | +| 01:24 | Session end: 1 writes across 1 files (config_repl.c) | 4 reads | ~133541 tok | +| 01:34 | Session end: 1 writes across 1 files (config_repl.c) | 5 reads | ~219255 tok | +| 01:45 | Session end: 1 writes across 1 files (config_repl.c) | 7 reads | ~357623 tok | +| 01:47 | root-caused bug-1477 (ctrl-3 permanent catch-up wedge, skip_election never retries cluster_init_phase2 so commit engine/apply pump never wires) | .wolf/buglog.json, ha.c, myapp.c, cluster.c, persist.c | logged bug-1477, out-of-scope for config-upgrade-snapshot, routed upstream to HA bootstrap lineage | ~180000 tok | +| 01:47 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | added error handling | ~630 | +| 01:48 | Session end: 2 writes across 2 files (config_repl.c, tasks.md) | 7 reads | ~358297 tok | +| 01:51 | Session end: 2 writes across 2 files (config_repl.c, tasks.md) | 7 reads | ~358297 tok | +| 02:07 | Session end: 2 writes across 2 files (config_repl.c, tasks.md) | 9 reads | ~371122 tok | +| 02:12 | Edited src/controller/ha.c | added error handling | ~827 | +| 02:12 | Edited src/controller/ha.c | modified target() | ~158 | +| 02:12 | Edited src/controller/ha.h | modified for() | ~313 | +| 02:13 | Edited src/controller/testsuite/test_ha_role.c | added error handling | ~902 | +| 02:13 | Edited src/controller/testsuite/test_ha_role.c | 3→8 lines | ~138 | + +## Session: 2026-07-28 02:14 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 02:20 | bug-1478 root-caused+fixed: ha_dispatch_repl_action_for_standby target_lsn leaked a WAL_CHECKPOINT-consumed LSN as a phantom convergence target, permanently wedging config-upgrade-wal-trim (task 8.9) catch-up; extracted ha_compute_repl_target_lsn, 4 new tests, RED->GREEN confirmed, 1040/1040 clean 2x | src/controller/ha.c, ha.h, testsuite/test_ha_role.c, .wolf/buglog.json, .wolf/cerebrum.md | fixed, unit-verified; live soak re-run still recommended | ~9500 | + +## Session: 2026-07-28 02:21 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 02:27 | Edited src/controller/config_repl.c | modified used() | ~283 | +| 02:28 | Session end: 1 writes across 1 files (config_repl.c) | 3 reads | ~45187 tok | +| 02:30 | Session end: 1 writes across 1 files (config_repl.c) | 3 reads | ~45187 tok | +| 02:30 | Session end: 1 writes across 1 files (config_repl.c) | 3 reads | ~45187 tok | +| 02:31 | Session end: 1 writes across 1 files (config_repl.c) | 3 reads | ~45187 tok | +| 02:33 | Session end: 1 writes across 1 files (config_repl.c) | 3 reads | ~45187 tok | +| 02:35 | Session end: 1 writes across 1 files (config_repl.c) | 4 reads | ~63902 tok | +| 02:39 | Edited .kiro/specs/scheduler-core/requirements.md | added 2 condition(s) | ~1395 | +| 02:39 | Edited .kiro/specs/scheduler-core/requirements.md | expanded (+6 lines) | ~743 | +| 02:39 | Edited .kiro/specs/scheduler-core/requirements.md | modified struct() | ~52 | +| 02:40 | Edited .kiro/specs/scheduler-core/requirements.md | 9→10 lines | ~156 | +| 02:40 | Session end: 5 writes across 2 files (config_repl.c, requirements.md) | 4 reads | ~66417 tok | +| 02:43 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~354 | +| 02:43 | Edited .kiro/specs/scheduler-core/requirements.md | 14→15 lines | ~620 | +| 02:44 | Session end: 7 writes across 2 files (config_repl.c, requirements.md) | 4 reads | ~69147 tok | +| 02:44 | Session end: 7 writes across 2 files (config_repl.c, requirements.md) | 4 reads | ~69147 tok | +| 02:47 | Edited src/controller/testsuite/test_convergence_verification.c | modified coverage() | ~250 | +| 02:47 | Session end: 8 writes across 3 files (config_repl.c, requirements.md, test_convergence_verification.c) | 4 reads | ~69415 tok | +| 02:47 | Edited src/controller/testsuite/test_convergence_verification.c | max() → ha_compute_repl_target_lsn() | ~36 | +| 02:48 | Edited src/controller/testsuite/test_convergence_verification.c | 3→5 lines | ~82 | +| 02:48 | Edited src/controller/testsuite/test_convergence_verification.c | 3→5 lines | ~82 | +| 02:49 | Edited .kiro/specs/scheduler-core/requirements.md | modified line() | ~768 | +| 02:49 | Session end: 12 writes across 3 files (config_repl.c, requirements.md, test_convergence_verification.c) | 5 reads | ~148861 tok | +| 02:49 | Session end: 12 writes across 3 files (config_repl.c, requirements.md, test_convergence_verification.c) | 5 reads | ~148861 tok | +| 02:51 | Session end: 12 writes across 3 files (config_repl.c, requirements.md, test_convergence_verification.c) | 5 reads | ~148861 tok | +| 02:51 | Edited .kiro/specs/scheduler-core/requirements.md | modified loop() | ~535 | +| 02:52 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | removed 6 lines | ~120 | +| 02:53 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified used() | ~615 | diff --git a/src/controller/config_repl.c b/src/controller/config_repl.c index ce011f44..89c209b6 100644 --- a/src/controller/config_repl.c +++ b/src/controller/config_repl.c @@ -2612,8 +2612,14 @@ static int repl_ep_still_registered(replicator_t *r, repl_ep_t ep) return found; } -/* re-derive the replication target exactly as ha.c:2771-2776 / D6: - * target = max(committed_lsn, last_written), last_written = current_lsn - 1. */ +/* re-derive the replication target via the SAME pure function + * repl_ack_decode_and_dispatch (ha.c) uses, so this continuation path and + * the heartbeat decision path can never disagree on what a standby must + * catch up to (bug-1478: this function used to duplicate the raw + * max(committed_lsn, last_written) formula inline, independently of the + * ha.c call site the fix landed in — a local-only, never-replicated + * WAL_CHECKPOINT trailing an already-advancing committed_lsn permanently + * wedged catch-up here as well, via the exact same mechanism). */ static uint64_t catchup_derive_target(cluster_t *c) { uint64_t committed = atomic_load_explicit(&c->persist.committed_lsn, @@ -2623,7 +2629,7 @@ static uint64_t catchup_derive_target(cluster_t *c) ? c->persist.current_lsn - 1 : 0; pthread_mutex_unlock(&c->persist.lock); - return last_written > committed ? last_written : committed; + return ha_compute_repl_target_lsn(committed, last_written); } /* Invoke the catch-up streaming path for [standby_lsn+1, target]. Mirrors the diff --git a/src/controller/ha.c b/src/controller/ha.c index 4c660663..f8df29be 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -3673,6 +3673,52 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx, } } +/* + * ha_compute_repl_target_lsn: the replication/convergence target that + * ha_dispatch_repl_action_for_standby's heartbeat-driven decision compares + * every standby's reported applied_lsn against. + * + * The correct target is simply the real, raft/commit-engine-authoritative + * committed_lsn — the only position a standby can safely (and eventually) + * be told it must catch up to. The ONE documented exception is cluster + * BOOTSTRAP: when committed_lsn is genuinely stuck at 0 (no standby has ever + * acked anything yet, so the commit engine's majority computation has never + * had anything to advance against), last_written (the raw locally-appended + * WAL tip) is used instead, so a first-ever connecting standby is told to + * catch up to real data instead of being handed "nothing to stream". + * + * bug-1478: this fallback must apply ONLY in that committed_lsn==0 bootstrap + * case — NOT unconditionally whenever last_written happens to exceed + * committed_lsn. last_written (persist.current_lsn - 1) counts EVERY + * locally-appended WAL entry, including a local-only, NEVER-replicated + * WAL_CHECKPOINT (persist_write_checkpoint_marker bumps current_lsn + * directly, bypassing raft_log_append/the commit engine entirely — the + * same architectural gap flagged in bug-1466). Once such a checkpoint + * trails an already-advancing (non-zero) committed_lsn, last_written + * permanently sits one LSN past it — and since a checkpoint's own consumed + * LSN slot is, by design, never streamed or applied by any standby + * (ha_stream_wal_to_standby / persist_wal_write_replicated / + * persist_wal_async all explicitly refuse WAL_CHECKPOINT), no standby's + * applied_lsn can EVER legitimately reach that value. Taking the + * max(committed_lsn, last_written) unconditionally turned the bootstrap + * convenience into a permanent, unsatisfiable catch-up target for every + * standby the instant a marker-triggered checkpoint (config-upgrade- + * snapshot task 5) or a periodic WAL-threshold checkpoint interleaves ahead + * of the next real committed advance — ha_evaluate_repl_action's + * standby_lsn < committed_lsn check then never reaches HA_REPL_CONVERGED, + * and ha_stream_wal_to_standby is re-invoked every heartbeat forever, + * finding nothing new to send each time (bug-1478). + * + * Pure, side-effect-free — safe to unit test directly. + */ +uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn, + uint64_t last_written) +{ + if (committed_lsn == 0 && last_written > 0) + return last_written; + return committed_lsn; +} + /* ─────────────────────────────────────────────────────────────────────── * raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198). * @@ -6376,12 +6422,14 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx, oldest_wal_lsn = ha_compute_oldest_wal_lsn(c->persist.wal_path); } #endif - /* Replication target: use max(committed_lsn, last_written) so - * standbys can catch up even when committed_lsn is stuck at 0 - * (no replicator-registered standbys to advance it). */ - uint64_t target_lsn = committed_lsn; - if (last_written > target_lsn) - target_lsn = last_written; + /* Replication target (bug-1478): committed_lsn, with the + * last_written bootstrap fallback ONLY when committed_lsn is + * genuinely stuck at 0 — see ha_compute_repl_target_lsn's own + * doc comment for why an unconditional max() permanently wedges + * catch-up once a local-only checkpoint trails a real, + * already-advancing committed_lsn. */ + uint64_t target_lsn = + ha_compute_repl_target_lsn(committed_lsn, last_written); ha_dispatch_repl_action_for_standby(ha, c, ep, peer_idx, committed_tag, target_lsn, diff --git a/src/controller/ha.h b/src/controller/ha.h index 7b98686e..5ff0e6f4 100644 --- a/src/controller/ha.h +++ b/src/controller/ha.h @@ -1127,6 +1127,24 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx, uint64_t reported_tag, uint64_t reported_lsn, uint64_t committed_tag, uint64_t committed_lsn); +/* + * ha_compute_repl_target_lsn: the replication/convergence target used by + * ha_dispatch_repl_action_for_standby's heartbeat-driven decision — normally + * committed_lsn itself, falling back to last_written (the raw locally- + * appended WAL tip) ONLY when committed_lsn is genuinely stuck at 0 + * (cluster bootstrap, before any standby has ever acked anything). + * + * bug-1478: the fallback must NOT apply once committed_lsn is already > 0 — + * see ha.c's doc comment on this function for the full mechanism (a + * local-only WAL_CHECKPOINT trailing an already-advancing committed_lsn + * otherwise leaks an unstreamable/unreachable target into every subsequent + * heartbeat decision, permanently wedging catch-up). + * + * Pure, side-effect-free. + */ +uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn, + uint64_t last_written); + /* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */ /* diff --git a/src/controller/testsuite/test_convergence_verification.c b/src/controller/testsuite/test_convergence_verification.c index ab7834cc..5670e9b0 100644 --- a/src/controller/testsuite/test_convergence_verification.c +++ b/src/controller/testsuite/test_convergence_verification.c @@ -276,17 +276,22 @@ static int cv_standby_in_catch_up(repl_ep_t ep) /* ── Tests ──────────────────────────────────────────────────────────────── */ -/* 5.1: a walAck with acked_lsn < target = max(committed_lsn, last_written) - * schedules EXACTLY ONE continuation that streams [acked_lsn+1, target] without - * any intervening heartbeat/stateReport dispatch. +/* 5.1: a walAck with acked_lsn < target = ha_compute_repl_target_lsn( + * committed_lsn, last_written) schedules EXACTLY ONE continuation that + * streams [acked_lsn+1, target] without any intervening heartbeat/ + * stateReport dispatch. * RED: the walAck continuation scaffold schedules nothing → no task, the stream * hook is never called. */ static void test_walack_behind_target_schedules_one_continuation(void) { - /* appended_lsn=9 → last_written=9; committed_lsn=4 → target=max(4,9)=9. */ + /* appended_lsn=9 → last_written=9; committed_lsn=9 → target=9. Kept equal + * to last_written (rather than a lower value like the historical 4) so + * this test exercises catch-up continuation scheduling only — the + * committed_lsn-vs-last_written edge case itself (bug-1478) has its own + * dedicated pure-function coverage in test_ha_role.c. */ cv_setup("behind", /*n_peers=*/3, /*n_standbys=*/1, /*appended_lsn=*/9, /*n_vcpus=*/8); - atomic_store(&g_cv.persist.committed_lsn, 4); + atomic_store(&g_cv.persist.committed_lsn, 9); const int peer_idx = 0; const uint64_t acked_lsn = 3; /* behind target 9 */ @@ -325,8 +330,8 @@ static void test_walack_behind_target_schedules_one_continuation(void) "[acked_lsn+1, target]) (Req 5.1)"); TEST_ASSERT_EQUAL_UINT64_MESSAGE(target, atomic_load(&g_stream_last_target_lsn), - "the continuation must stream up to target = max(committed_lsn, " - "last_written) (Req 5.1)"); + "the continuation must stream up to target = " + "ha_compute_repl_target_lsn(committed_lsn, last_written) (Req 5.1)"); cv_teardown(); printf("test_walack_behind_target_schedules_one_continuation: checked\n"); @@ -338,9 +343,11 @@ static void test_walack_behind_target_schedules_one_continuation(void) * "exactly one continuation across two ACKs" assertion fails (0 != 1). */ static void test_walack_single_flight_no_duplicate_continuation(void) { + /* committed_lsn == appended_lsn (target=9 either way) — see the "behind" + * test's own note on why these don't use a lower committed_lsn. */ cv_setup("single", /*n_peers=*/3, /*n_standbys=*/1, /*appended_lsn=*/9, /*n_vcpus=*/8); - atomic_store(&g_cv.persist.committed_lsn, 4); + atomic_store(&g_cv.persist.committed_lsn, 9); const int peer_idx = 0; @@ -382,9 +389,11 @@ static void test_walack_single_flight_no_duplicate_continuation(void) * RED: the scaffold never schedules/clears, so neither continuation appears. */ static void test_walack_reschedules_after_continuation_completes(void) { + /* committed_lsn == appended_lsn (target=9 either way) — see the "behind" + * test's own note on why these don't use a lower committed_lsn. */ cv_setup("respawn", /*n_peers=*/3, /*n_standbys=*/1, /*appended_lsn=*/9, /*n_vcpus=*/8); - atomic_store(&g_cv.persist.committed_lsn, 4); + atomic_store(&g_cv.persist.committed_lsn, 9); const int peer_idx = 0; diff --git a/src/controller/testsuite/test_ha_role.c b/src/controller/testsuite/test_ha_role.c index 8bd14724..a0216455 100644 --- a/src/controller/testsuite/test_ha_role.c +++ b/src/controller/testsuite/test_ha_role.c @@ -660,6 +660,64 @@ static void test_evaluate_repl_action_stream_wal_fresh_system(void) printf("test_evaluate_repl_action_stream_wal_fresh_system: PASS\n"); } +/* ── bug-1478: ha_compute_repl_target_lsn ───────────────────────────────── + * + * repl_ack_decode_and_dispatch (ha.c) computes the replication/convergence + * target passed into ha_dispatch_repl_action_for_standby as + * max(committed_lsn, last_written), where last_written is the raw locally- + * appended WAL tip (persist.current_lsn - 1). A live 3-node HA soak + * (config-upgrade-snapshot task 8.9, --config-upgrade-wal-trim) caught this + * unconditional max() permanently wedging catch-up: once a marker-triggered + * checkpoint (a local-only, NEVER-replicated WAL_CHECKPOINT entry — + * persist_write_checkpoint_marker bumps current_lsn directly, bypassing + * raft_log_append/the commit engine, bug-1466) trails an already-advancing + * committed_lsn, last_written sits permanently one LSN past it — a position + * no standby's applied_lsn can EVER legitimately reach (the checkpoint's own + * slot is, by design, never streamed or applied). ha_evaluate_repl_action's + * standby_lsn < committed_lsn check then never reaches HA_REPL_CONVERGED, + * and the primary re-streams the same empty range every heartbeat, forever. + * Ground-truthed via `xce_test waldump` on the live soak's preserved WAL + * (ctrl-2: lsn 21 = the WAL_CONFIG_MARKER, lsn 22 = WAL_CHECKPOINT, 0 bytes) + * and via ctrl-2's own commit_advance_locked log (the REAL, majority-gated + * committed_lsn never advanced past 21 — only the target_lsn fallback + * leaked 22 into the heartbeat decision). + */ +static void test_compute_repl_target_lsn_bootstrap_fallback(void) +{ + /* committed_lsn genuinely stuck at 0 (no standby has ever acked) — the + * documented bootstrap exception: fall back to last_written so a + * first-ever connecting standby can catch up to real data. */ + TEST_ASSERT_EQUAL_UINT64(42, ha_compute_repl_target_lsn(0, 42)); + printf("test_compute_repl_target_lsn_bootstrap_fallback: PASS\n"); +} + +static void test_compute_repl_target_lsn_zero_zero(void) +{ + /* Genuinely fresh cluster, nothing appended yet. */ + TEST_ASSERT_EQUAL_UINT64(0, ha_compute_repl_target_lsn(0, 0)); + printf("test_compute_repl_target_lsn_zero_zero: PASS\n"); +} + +static void test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn(void) +{ + /* bug-1478's exact reproduction shape: committed_lsn=21 is already + * advancing (non-zero, real, majority-gated), but last_written=22 + * reflects a local-only WAL_CHECKPOINT that trails it. The bootstrap + * fallback must NOT fire here — the target must stay at the real + * committed_lsn (21), never the unstreamable checkpoint slot (22). */ + TEST_ASSERT_EQUAL_UINT64(21, ha_compute_repl_target_lsn(21, 22)); + printf("test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn: PASS\n"); +} + +static void test_compute_repl_target_lsn_committed_already_ahead(void) +{ + /* Normal steady-state case: committed_lsn is already the higher/current + * value; last_written (a possibly-stale, lock-free read) trails it. + * The target must simply be committed_lsn. */ + TEST_ASSERT_EQUAL_UINT64(100, ha_compute_repl_target_lsn(100, 50)); + printf("test_compute_repl_target_lsn_committed_already_ahead: PASS\n"); +} + /* The transitional-leader timeout default test was removed: the config knob * and its default macro are deleted by ha-single-leader-role (the wall-clock * force-promotion watchdog it configured is gone — Req 1.5). */ @@ -892,6 +950,11 @@ void test_ha_role(void) RUN_TEST(test_evaluate_repl_action_stream_wal); /* Task 6.1: (tag=T, lsn=0) with oldest_wal_lsn=0 → STREAM_WAL */ RUN_TEST(test_evaluate_repl_action_stream_wal_fresh_system); + /* bug-1478: ha_compute_repl_target_lsn */ + RUN_TEST(test_compute_repl_target_lsn_bootstrap_fallback); + RUN_TEST(test_compute_repl_target_lsn_zero_zero); + RUN_TEST(test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn); + RUN_TEST(test_compute_repl_target_lsn_committed_already_ahead); /* Task 3.3: ha_dispatch_repl_action_for_standby PUSH_CONFIG wiring */ RUN_TEST(test_dispatch_repl_action_push_config_when_tag_behind); RUN_TEST(test_dispatch_repl_action_no_push_when_converged);