fix(raft-ha-core-safety): close 5.10a — WAL segment rotation + endpoint UAF blocking real snapshot catch-up

Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):

1. WAL segment rotation was gated on a hardcoded 64MB compile-time
   constant; trim_wal_segments() never deletes the active (currently-open)
   segment. At soak job counts a segment never rotated, so nothing was
   ever physically trimmable and ha_evaluate_repl_action always chose
   STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
   threading a wal_segment_bytes override through persist_config_t
   (mirrors the --max-wal-records pattern: config.h default, persist.h
   setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
   CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
   --wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
   now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.

2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
   (include/netio.h): the macro calls ep->msg_handler(ep) synchronously
   (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
   doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
   handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
   endpoint_unref) could drop the endpoint's last ref and free it inside
   the handler call, so the trailing write hit freed memory. This is the
   first-ever live exercise of a node receiving a snapshot transfer while
   mid-election. Fixed by holding a ref across the synchronous call, same
   pattern task 5.8 established for the async path.

With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.

Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
This commit is contained in:
Rongsong Shen 2026-07-02 04:11:20 +00:00
parent b9ad01878c
commit 1439e2655e
11 changed files with 115 additions and 40 deletions

View file

@ -44,6 +44,12 @@
# XCE_HA_WAITMS wait-primary budget (default 30000)
# MAX_WAL_RECORDS --max-wal-records passed to every node (default: unset
# for sequential/chaotic; 100 for --force-snapshot)
# WAL_SEGMENT_BYTES --wal-segment-bytes passed to every node (default: unset
# for sequential/chaotic; 4096 for --force-snapshot — a
# segment is only eligible for trim/delete once it rotates
# out, so the 64 MB production default never rotates at
# soak job counts and force-snapshot could never observe
# a real snapshot-triggered catch-up without this)
set -u
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
@ -113,8 +119,10 @@ WAITMS="${XCE_HA_WAITMS:-30000}"
# leaves it unset (empty ⇒ start_node passes no override, unchanged behavior).
if [ "$MODE" = "force-snapshot" ]; then
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-100}"
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
else
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-}"
fi
NODES=(ctrl-1 ctrl-2 ctrl-3)
@ -175,8 +183,11 @@ start_node() {
# libc/ASAN crash message) leaves a forensic trace. Appended (>>) so a
# restarted node's later crash does not clobber the earlier instance's output.
mkdir -p "$TOP/logs"
if [ -n "$MAX_WAL_RECORDS" ]; then
bash "$TOP/scripts/start.sh" "$1" -vvv --max-wal-records "$MAX_WAL_RECORDS" \
local extra_args=()
[ -n "$MAX_WAL_RECORDS" ] && extra_args+=(--max-wal-records "$MAX_WAL_RECORDS")
[ -n "$WAL_SEGMENT_BYTES" ] && extra_args+=(--wal-segment-bytes "$WAL_SEGMENT_BYTES")
if [ ${#extra_args[@]} -gt 0 ]; then
bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
>>"$TOP/logs/$1.stderr" 2>&1 &
else
bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
@ -852,7 +863,7 @@ walcheck "${#ALL_JOBIDS[@]}"
# run_force_snapshot_round's own header comment. Single round, not a loop —
# emits a verdict and exits, like chaotic mode.
if [ "$MODE" = "force-snapshot" ]; then
log "MODE=force-snapshot — forcing one full-checkpoint catch-up round (max-wal-records=$MAX_WAL_RECORDS)"
log "MODE=force-snapshot — forcing one full-checkpoint catch-up round (max-wal-records=$MAX_WAL_RECORDS wal-segment-bytes=$WAL_SEGMENT_BYTES)"
run_force_snapshot_round
scan_bug924
verdict_emit "force-snapshot" 1 "${#ALL_JOBIDS[@]}" \