fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)

Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.

Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.

Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.

GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.

.kiro/specs/config-upgrade-snapshot/tasks.md

@@ -142,15 +142,16 @@
   - _Requirements: 4.4, 7.2_
   - _Depends: 7, 8.1, 8.2, 8.3, 8.4_
 
-- [ ] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions
+- [x] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions
   - Repeat the mid-catch-up WAL-trim assertion from the unit-level fixture, but under the full chaotic multi-kill soak rather than an isolated fixture
   - Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up
   - _Requirements: 7.4_
   - _Depends: 7, 8.1, 8.2, 8.3, 8.4_
-  - _Blocked: bug-1474 — pre-existing, out-of-spec-boundary HA connection-management defect (`repl_add_standby`'s first-empty-slot allocation, `repl.c:74-133`, unconditional `memset` at line 112; `primary_heartbeat_fn`'s reconnect sweep, `ha.c:9219-9254`; both blame to commits `7b396502`/`fa66d1ba9`/`8ef64a7e`, weeks before this spec). Sequence: killing a standby correctly frees its replicator slot and sets the bug-1147 `last_disconnected_lsn` retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an ALREADY-connected peer (connected via a promoted-vote-channel, not yet a heartbeat-task connection) as missing, spawns a redundant "late-joiner" reconnect, and `repl_add_standby` reallocates the just-freed slot — `memset`-clobbering the retention guard before any trim runs. Confirmed reproducible 2/2 against independent live 3-node clusters, and confirmed via `git blame` to be reachable identically under the pre-existing periodic-checkpoint trigger — nothing this spec added or changed. The live-soak round (`run_config_upgrade_wal_trim_round`, `run-ha-soak.sh`) is complete, committed, and correctly reproduces the violation; independently reviewed and confirmed sound (round design faithfully mirrors the unit fixture's mechanics; root-cause diagnosis independently re-derived from `repl.c`/`ha.c` plus raw preserved logs, not trusted from the implementer's report alone). No scoped adjustment to the round can dodge the race without suppressing the very live-cluster heartbeat behavior Req 7.4 requires testing under. Unblocks when bug-1474 is fixed by whoever owns HA connection/replicator-slot management — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade-wal-trim` to confirm clean PASS). See `.wolf/buglog.json` bug-1474 for the full forensic trace._
 
 ## Implementation Notes
 
+- **Task 8.9 unblocked and completed (2026-07-28):** bug-1474 (the blocker below) was fixed and verified upstream. Re-running `run-ha-soak.sh --config-upgrade-wal-trim` on top of that fix surfaced a SECOND, unrelated bug: **bug-1478**, a duplicated `target = max(committed_lsn, last_written)` replication-target formula independently re-derived in two places (`ha.c`'s heartbeat/stateReport dispatch and `config_repl.c`'s `catchup_derive_target`, used by the ACK-driven catch-up continuation path) — once config-upgrade-snapshot's own marker-triggered checkpoint (task 5) advanced `last_written` one LSN past a genuinely-already-committed `committed_lsn` (the checkpoint's own consumed, never-replicated LSN slot), the unconditional `max()` permanently wedged every standby's catch-up target at an unreachable value. Root-caused and fixed via `ha_compute_repl_target_lsn(committed_lsn, last_written)` (declared `ha.h`, defined `ha.c`) — a pure function used by BOTH call sites, restoring the `last_written` fallback to ONLY the genuine `committed_lsn==0` bootstrap case. Fixing the second call site exposed a stale test expectation (three pre-existing tests in `test_convergence_verification.c` hardcoded the OLD buggy formula's output as their expected value, which — combined with that test file's harness not running teardown on an assertion failure — produced a live-thread/global-state corruption hang, not a defect in the fix itself); updated those tests' fixture values to decouple them from the now-fixed edge case (already covered by dedicated pure-function tests in `test_ha_role.c`). Bug-1478 is a pre-existing HA-replication defect (commit `06135954f`, ~2 weeks before this spec) that task 5 merely exposed — same routing class as bug-1471/1472/1474/1477 — but was narrowly enough scoped to fix directly rather than block on. GREEN: full `controller_unit_test` suite = 1040 tests, 0 failures, run clean multiple times across both fix iterations. Live-verified: `run-ha-soak.sh --config-upgrade-wal-trim` reports a clean PASS (18 jobs, zero job loss, `ctrl-1` reconverged 3-of-3 using the preserved floor segment for real incremental catch-up). See `.wolf/buglog.json` bug-1474 and bug-1478 for full forensic detail.
+
 - **Task 8.9 (live soak, genuinely requires chaos — unlike 8.5/8.7/8.8's unit-level redirections): infra complete, task BLOCKED on a new pre-existing bug (bug-1474).** Unlike every other 8.x subtask, this one's own bullet ("under the full chaotic multi-kill soak rather than an isolated fixture") makes the point of the task proving the mid-catch-up WAL-trim guard (`repl_trim_lsn`/`trim_wal_segments`, bug-1147) holds under REAL timing uncertainty, not just deterministic fixture control — so it could not be redirected to a cheaper unit-level proof the way 8.5/8.7/8.8 were. Built `run_config_upgrade_wal_trim_round()` + `--config-upgrade-wal-trim` in `run-ha-soak.sh` (deploys with small `--wal-segment-bytes`, kills a non-primary TARGET, forces ≥2 real segment rotations past its floor on survivors, submits a live config upgrade to fire the marker-triggered checkpoint's trim, then directly inspects survivors' on-disk `wal/` directory BEFORE restarting TARGET — mirrors the unit fixture's own seg-1-gone/seg-2-preserved/seg-3-preserved assertions against real files instead of a tmpdir fixture). Ran twice against independent live 3-node clusters: **both runs failed** — the floor segment protecting the killed target was trimmed both times. Root-caused (not guessed) to **bug-1474**, a general HA connection-management race entirely unrelated to config-marker triggering: killing a standby frees its replicator slot and sets the retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an already-connected peer as "missing" (comparing indices under two different conventions — relative `votes_idx` vs. absolute `peer_idx`, per the reviewer's independent finding), spawns a redundant reconnect, and `repl_add_standby`'s first-empty-slot allocation reuses the just-freed slot, `memset`-clobbering the guard before trim ever runs. Reviewer independently re-derived this from `repl.c:74-133`/`ha.c:9219-9254` plus raw preserved logs (not the implementer's quoted excerpts) and confirmed via `git blame` the responsible code predates this spec by weeks and is reachable identically under the OLD periodic-checkpoint trigger — genuinely out of this spec's boundary, same class of finding as bug-1471/1472. Reviewer also confirmed no scoped adjustment to the round's chaos could dodge the race without suppressing the live heartbeat behavior Req 7.4 requires testing under, so BLOCKED (not a redesign) is the correct call. **Spec-wide status after this task**: every implementable task (1 through 8.5, 8.7, 8.8) is `[x]`; the only two remaining open items (8.6, 8.9) are both blocked on real, independently-verified, out-of-spec-boundary HA defects this spec's own testing effort discovered (bug-1472, bug-1474) — neither blocks any OTHER task in this spec, and neither requires further config-upgrade-snapshot work once fixed upstream.
 
 - **Task 8.8 (unit-level, no live soak needed — deliberately excludes 8.6):** proves Req 4.4/7.2 (a second marker committing before the first's install completes must cleanly supersede it, never conflate) via two independently-built `t42`/`t43`-family fixtures in `test_config_repl.c`: `c_control` commits+applies+checkpoints marker A then marker B normally; `c_raced` holds the shared `checkpoint_in_progress` CAS across *both* markers' commit+apply (inlining `t43_commit_marker`'s 4-step body rather than changing its signature), so both markers' forced-checkpoint attempts hit `-EBUSY` and skip — genuinely "still installing." Mid-race assertions (captured before releasing the CAS) prove `mc->cluster_config` already reflects marker B while `cfgref_present` is still false and the fence is set; post-resolve assertions (after releasing the CAS, calling `persist_checkpoint()`, then an explicit `upgrade_materialization_verify()`) prove the settled state reflects only B. Discovered mid-implementation that whole-file `snapdiff_paths` byte-identity between the two fixtures' final checkpoints does NOT hold — `c_control`'s marker-A checkpoint physically consumes one extra WAL LSN (`persist_write_checkpoint_marker`, persist.c:1684) before marker B is appended, so B lands at a different absolute LSN in each fixture, and `write_snapshot`'s sentinel header embeds `checkpoint_lsn` at byte offset 8. Adapted to a field-aware comparison (full `memcmp` from byte 16 onward) plus `snapdiff_paths` diagnostic corroboration that offset 8 is the *only* divergence — a more precise proof than blind byte-identity, reviewer independently re-derived and confirmed this against `write_snapshot`'s actual layout rather than trusting the claim. One RED→GREEN cycle (`bug-1473`, test-construction gap not a production bug): the bare `persist_checkpoint()` resolve call doesn't itself trigger `upgrade_materialization_verify()` in a delaymgr-less unit fixture — production's self-healing (`schedule_upgrade_materialization_retry`, fired from `ha.c:2541-2542` on every commit-advance) only runs on a live cluster; the fixture needs the manual call, mirroring the existing 6.1-c fence tests' pattern. Reviewer's only findings were non-blocking style notes (could have used the public `upgrade_materialization_retry_fn` wrapper instead of calling `upgrade_materialization_verify` directly — functionally identical here since the fence was already independently confirmed set). Suite: 1027 → 1028 tests, 0 failures.