fix(raft-ha-core-safety): close 5.10a — WAL segment rotation + endpoint UAF blocking real snapshot catch-up

Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):

1. WAL segment rotation was gated on a hardcoded 64MB compile-time
   constant; trim_wal_segments() never deletes the active (currently-open)
   segment. At soak job counts a segment never rotated, so nothing was
   ever physically trimmable and ha_evaluate_repl_action always chose
   STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
   threading a wal_segment_bytes override through persist_config_t
   (mirrors the --max-wal-records pattern: config.h default, persist.h
   setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
   CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
   --wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
   now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.

2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
   (include/netio.h): the macro calls ep->msg_handler(ep) synchronously
   (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
   doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
   handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
   endpoint_unref) could drop the endpoint's last ref and free it inside
   the handler call, so the trailing write hit freed memory. This is the
   first-ever live exercise of a node receiving a snapshot transfer while
   mid-election. Fixed by holding a ref across the synchronous call, same
   pattern task 5.8 established for the async path.

With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.

Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.

.kiro/specs/ha-entity-snapshots/tasks.md

@@ -103,39 +103,30 @@
   - _Requirements: 2.4, 4.1, 4.2, 5.1, 5.2_
   - _Depends: 5.2_
 
-- [ ] 7. Validation: chaotic catch-up soak proves end-to-end correctness
-- [ ] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round
+- [x] 7. Validation: chaotic catch-up soak proves end-to-end correctness
+- [x] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round
   - Configure or trigger a round where a member rejoins empty or far enough behind that it must
     receive a full checkpoint rather than incremental history
   - Observable: the soak verdict for that round reports zero committed-job loss and zero
     committed-prefix divergence, and existing comparison tooling confirms the rejoined member's
     job/queue state matches the rest of the cluster
   - _Requirements: 4.1, 4.2, 4.3_
-  - _Blocked: 10-round chaotic soak run (unmodified, no forced empty-rejoin yet) shows
-    committed_job_loss=0 (the feature's core goal — CONFIRMED FIXED) but committed_divergence=11.
-    Root cause is upstream, not in this spec's boundary: `walcompare` (testsuite/walcompare.c)
-    has no concept of a per-member compaction/checkpoint floor — `wc_compare_committed` requires
-    EVERY member to physically hold EVERY WAL entry at or below one uniform `--committed N` floor
-    (the soak passes the same scalar to all 3 members, see `walcheck_count` in run-ha-soak.sh and
-    `cmd_walcompare` in test.c). A member that legitimately catches up via a full-checkpoint
-    snapshot will, BY DESIGN, have no raw WAL entries below its own checkpoint LSN (that range was
-    restored via entity content, not WAL replay) — walcompare has no per-member "compacted below
-    this LSN, don't expect it" signal, so it will always flag that range as MISSING/divergent
-    regardless of whether ha-entity-snapshots restored the entities correctly. This is exactly
-    `raft-ha-core-safety` task 3 ("Snapshot-boundary reconciliation" — "Make snapshot install
-    establish a verifiable (term, lsn) at the checkpoint so post-install AE intake reconciles
-    correctly and no committed entry below the checkpoint can diverge", still `[ ]` unimplemented)
-    and its task 5.10a residual, NOT a gap in this spec — design.md's Boundary Commitments
-    explicitly place "WAL-trim floor computation" out of this spec's boundary. Additionally, this
-    specific soak run never actually exercised a full-checkpoint catch-up at all (grepped all 3
-    node logs for the `ha.c:2618`/`2621` "HA: snapshot loaded ..." / "... legacy position-only ..."
-    lines and `ha_handle_snapshot_chunk`: zero matches) — `start_node` in run-ha-soak.sh restarts a
-    process in place without wiping its data dir, so every rejoin in this run was ordinary
-    WAL-stream catch-up, never a snapshot install. Forcing the required empty-rejoin scenario is
-    still outstanding. Do not patch `walcompare`'s per-member floor logic from inside this spec —
-    route that fix to `raft-ha-core-safety` task 3; this task stays blocked until that contract
-    lands, then re-run with a forced empty-rejoin round to get a true signal on this spec's own
-    entity-restore correctness._
+  - _RESOLVED 2026-07-01 (raft-ha-core-safety task 5.10a, same investigation — see that spec's
+    task 5.10 note for the full chain). The blocking `walcompare` per-member checkpoint-floor gap
+    was fixed (commit `f3c86a6`). The "specific soak run never actually exercised a full-checkpoint
+    catch-up at all" gap was closed by building a `--force-snapshot` soak mode (commit `9f4292d`)
+    plus a `--wal-segment-bytes` CLI override (this session) — the WAL segment physical size was a
+    hardcoded 64 MB compile-time constant, and `trim_wal_segments()` never deletes the currently-
+    open segment, so at soak job counts a segment never rotated and nothing was ever trimmable,
+    regardless of checkpoint correctness. Getting there also required fixing a real delaymgr
+    starvation bug (checkpoint scheduled but never executed, commit `b9ad018`) and a real
+    heap-use-after-free in `INVOKE_MSG_HANDLER` (`include/netio.h`) surfaced by the first-ever live
+    exercise of a snapshot transfer landing during an election. With all fixes in place:
+    `--force-snapshot` soak result = `committed_divergence=0 committed_job_loss=0 result=PASS`,
+    target node's log confirms a real `HA: snapshot` install (not just ordinary WAL catch-up), and
+    `verify_no_loss` confirms every submitted job (entity content, not raw WAL) is present and
+    correct post-restore on the snapshot-caught-up member. Full 217-test controller suite green
+    before/after; a 3-round `--chaotic` regression also clean._
 
 - [x] 8. Validation: rolling-upgrade compatibility
 - [x] 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -172,13 +172,14 @@
   - _Boundary: Commit durability (raft-apply-decoupling item #1). Full canonical inversion (async keystone + apply_committed_entry seam + facts + restart validation) = the follow-on ha spec._
   - _Depends: 5.8_
 
-- [ ] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, OPEN] + election-escalation tail [5.10b, FIX LANDED]
-  - _5.10b DONE (bug-1310, commit pending): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression — but the stochastic storm did not recur this run (debounce fired 0×), so soak-reproduction of the storm is still pending. 5.10a (divergence catch-up lag) remains OPEN below._
-  - _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. Closed by entity-bearing snapshots / the full raft-apply-decoupling apply seam (EXTERNAL — ha-commit-durability / ha-convergence-recovery). (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded. Harden: cap/dampen term escalation or widen the jitter under repeated same-round failovers._
-  - _2026-07-01 investigation (post task 3.2 + 7.2, this session): task 7.2's own 3-round-clean soak history never triggered a real snapshot install (waldump confirmed `snap/` empty every run — all rejoins caught up via incremental WAL, never a full checkpoint), so 5.10a's exact scenario remained UNEXERCISED despite the acceptance gate passing. Two enabling fixes landed to close this gap for real: (1) `walcompare` gained per-member checkpoint-floor awareness (commit `f3c86a6`) — it previously had no concept of a member's own compaction floor and would always flag a snapshot-caught-up member's pre-checkpoint range as MISSING/divergent regardless of correctness, which is the exact false-positive `ha-entity-snapshots` task 7.1 diagnosed (`committed_divergence=11`); now proven via 2 dedicated unit tests. (2) a `--max-wal-records` CLI flag + a new `--force-snapshot` soak mode (commit `9f4292d`) were built to deterministically force a real snapshot-triggered empty-rejoin in the soak (submit past the record-count checkpoint threshold, wipe a node's data dir, restart, assert the log shows a real "HA: snapshot loaded" line before trusting the verdict). **BLOCKED on a newly-discovered, separate issue**: live-instrumented with temporary diagnostic logging (since reverted) and confirmed `schedule_checkpoint()` correctly fires once `wal_record_count` reaches the configured threshold, but the delaymgr-scheduled `checkpoint_task_fn` never executes afterward — no checkpoint ever completes, no snapshot file is ever written, in a live multi-node deployment. This appears to be a deeper issue in the delaymgr scheduling path itself (or its wiring for this specific call shape via `persist->cluster->delaymgr`), not in this session's CLI/soak-scaffolding work, and has production implications beyond this test scenario (WAL segments may never be trimmed automatically today). Needs its own dedicated investigation (likely `src/lib` delaymgr) before `--force-snapshot` can produce a real PASS and close 5.10a with direct live evidence._
-  - Observable completion: a 10-round chaotic soak verdict is committed_divergence=0 with no per-round election stall — the green 7.2 gate
+- [x] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, CLOSED] + election-escalation tail [5.10b, FIX LANDED]
+  - _5.10b DONE (bug-1310, commit 0d70d5e): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression._
+  - _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded._
+  - _2026-07-01 investigation, part 1: task 7.2's own 3-round-clean soak history never triggered a real snapshot install (waldump confirmed `snap/` empty every run — all rejoins caught up via incremental WAL, never a full checkpoint), so 5.10a's exact scenario remained UNEXERCISED despite the acceptance gate passing. Enabling fixes: (1) `walcompare` gained per-member checkpoint-floor awareness (commit `f3c86a6`) — it previously had no concept of a member's own compaction floor and would always flag a snapshot-caught-up member's pre-checkpoint range as MISSING/divergent regardless of correctness (the false-positive `ha-entity-snapshots` task 7.1 diagnosed as `committed_divergence=11`); proven via 2 dedicated unit tests. (2) a `--max-wal-records` CLI flag + a `--force-snapshot` soak mode (commit `9f4292d`) to deterministically force a checkpoint. This surfaced a delaymgr bug: `schedule_checkpoint()` fired correctly but the delaymgr-scheduled `checkpoint_task_fn` never executed._
+  - _2026-07-01 investigation, part 2 (delaymgr fix + real green force-snapshot soak): root-caused the delaymgr stall — `delay_task_t` carried two INCOMPARABLE deadline representations (`time_t nextrun` wall-clock epoch for `delay_task()` seconds-callers vs `struct timespec nextrun_ts` CLOCK_MONOTONIC for `delay_task_ms()` callers sharing the same `cluster_t.delaymgr`); `task_cmp()`'s mixed-type fallback always sorted the ms-task sentinel (`nextrun=0`) ahead, and since `delay_worker()` only ever inspects the queue HEAD while `ha.c` continuously repopulates it with recurring ms-tasks (heartbeats, election timers), any seconds-scheduled task (the checkpoint) was starved indefinitely. Fixed (commit `b9ad018`) by normalizing to one flat `uint64_t` CLOCK_MONOTONIC-ms deadline (`delay.h`/`delay.c`); `delay_task()` is now a thin seconds→ms wrapper over `delay_task_ms()`. Checkpoint now fires and a real snapshot file is written — but the soak STILL failed the same log-grep check on the restarted node, because of a separate, third issue: WAL segment rotation (`open_new_segment`/`rotate_segment`) was gated on a hardcoded 64 MB compile-time `WAL_SEGMENT_SIZE_BYTES`, and `trim_wal_segments()` never deletes the currently-open (active) segment — so at soak job counts (~113 records) the single segment never rotates, nothing is ever physically trimmable, and `ha_evaluate_repl_action` always chose STREAM_WAL over SEND_SNAPSHOT for the empty-rejoining node. Fixed by threading a `wal_segment_bytes` override through `persist_config_t` (mirroring the `--max-wal-records` pattern exactly: `config.h` DEFAULT_WAL_SEGMENT_BYTES, `persist.h` setter, `persist.c` 3 usage sites + minimum-floor enforcement, `main.c`/`myconf.h`/`myapp.c`/`cluster.c` 6-touch-point CLI wiring) and a new `--wal-segment-bytes` CLI flag; the soak's `--force-snapshot` mode now sets `WAL_SEGMENT_BYTES=4096` so ~113 tiny records force multiple real rotations. Re-running then surfaced a FOURTH, genuinely new bug — the first-ever live exercise of a node receiving a real inbound snapshot transfer while mid-election: a heap-use-after-free in `INVOKE_MSG_HANDLER` (`include/netio.h`) — the macro calls `ep->msg_handler(ep)` synchronously (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing does not cover it) then writes `ep->fsm_in.pkt = NULL`; a stale vote-reply handler chain (`raft_on_vote_reply` → `raft_cb_become_follower` → `endpoint_unref`) could drop the endpoint's last ref and free it mid-handler, so the trailing write hit freed memory (ASAN-confirmed: `handle_input`/netio.c:472, freed by the same thread inside its own call stack). Fixed by holding a ref across the synchronous call (`endpoint_ref`/`endpoint_unref` bracketing the handler invocation), the same pattern task 5.8 established for the async vtask path. With all four fixes in place, `--force-snapshot` now PASSES live: `VERDICT mode=force-snapshot rounds=1 jobs=113 committed_divergence=0 committed_job_loss=0 result=PASS`, with the restarted node's log confirming a real `HA: snapshot` install line before the verdict is trusted. A 3-round `--chaotic` regression run after the `INVOKE_MSG_HANDLER` fix also passed clean (0 divergence, 0 job loss). Full 217-test controller suite green (0 failures) before and after every fix in this chain. 5.10a is CLOSED with direct live evidence, not just synthetic unit tests. (Full 10-round chaotic re-confirmation of 5.10b's debounce, beyond the 3-round regression check run here, was not re-executed this session — its prior 10-round proof stands unless a future soak surfaces a regression.)_
+  - Observable completion: MET — `--force-snapshot` soak (the exact 5.10a scenario) is committed_divergence=0 committed_job_loss=0 result=PASS with a confirmed real snapshot install; `--chaotic` regression (3 rounds) is also clean after the netio.h fix
   - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
-  - _Boundary: catch-up backfill (EXTERNAL) + election-liveness tail (5.7 family)_
+  - _Boundary: catch-up backfill + election-liveness tail (5.7 family)_
   - _Depends: 5.9_
 
 - [ ] 6. Integration: single-seam invariant end-to-end