fix(raft-ha-core-safety): close 5.10a — WAL segment rotation + endpoint UAF blocking real snapshot catch-up
Chain of fixes needed to get --force-snapshot soak to a real live PASS (raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1): 1. WAL segment rotation was gated on a hardcoded 64MB compile-time constant; trim_wal_segments() never deletes the active (currently-open) segment. At soak job counts a segment never rotated, so nothing was ever physically trimmable and ha_evaluate_repl_action always chose STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by threading a wal_segment_bytes override through persist_config_t (mirrors the --max-wal-records pattern: config.h default, persist.h setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new --wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale. 2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER (include/netio.h): the macro calls ep->msg_handler(ep) synchronously (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply handler chain (raft_on_vote_reply -> raft_cb_become_follower -> endpoint_unref) could drop the endpoint's last ref and free it inside the handler call, so the trailing write hit freed memory. This is the first-ever live exercise of a node receiving a snapshot transfer while mid-election. Fixed by holding a ref across the synchronous call, same pattern task 5.8 established for the async path. With both fixes: --force-snapshot soak now PASSES live (committed divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the restarted node) — first genuine live proof of this scenario. 3-round --chaotic regression clean. Full 217-test controller suite green before/after every fix in the chain. Builds on the delaymgr starvation fix (b9ad018) and walcompare checkpoint-floor fix (f3c86a6) from earlier in this investigation.
This commit is contained in:
parent
b9ad01878c
commit
1439e2655e
11 changed files with 115 additions and 40 deletions
|
|
@ -44,6 +44,12 @@
|
|||
# XCE_HA_WAITMS wait-primary budget (default 30000)
|
||||
# MAX_WAL_RECORDS --max-wal-records passed to every node (default: unset
|
||||
# for sequential/chaotic; 100 for --force-snapshot)
|
||||
# WAL_SEGMENT_BYTES --wal-segment-bytes passed to every node (default: unset
|
||||
# for sequential/chaotic; 4096 for --force-snapshot — a
|
||||
# segment is only eligible for trim/delete once it rotates
|
||||
# out, so the 64 MB production default never rotates at
|
||||
# soak job counts and force-snapshot could never observe
|
||||
# a real snapshot-triggered catch-up without this)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -113,8 +119,10 @@ WAITMS="${XCE_HA_WAITMS:-30000}"
|
|||
# leaves it unset (empty ⇒ start_node passes no override, unchanged behavior).
|
||||
if [ "$MODE" = "force-snapshot" ]; then
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-100}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
|
||||
else
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-}"
|
||||
fi
|
||||
|
||||
NODES=(ctrl-1 ctrl-2 ctrl-3)
|
||||
|
|
@ -175,8 +183,11 @@ start_node() {
|
|||
# libc/ASAN crash message) leaves a forensic trace. Appended (>>) so a
|
||||
# restarted node's later crash does not clobber the earlier instance's output.
|
||||
mkdir -p "$TOP/logs"
|
||||
if [ -n "$MAX_WAL_RECORDS" ]; then
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv --max-wal-records "$MAX_WAL_RECORDS" \
|
||||
local extra_args=()
|
||||
[ -n "$MAX_WAL_RECORDS" ] && extra_args+=(--max-wal-records "$MAX_WAL_RECORDS")
|
||||
[ -n "$WAL_SEGMENT_BYTES" ] && extra_args+=(--wal-segment-bytes "$WAL_SEGMENT_BYTES")
|
||||
if [ ${#extra_args[@]} -gt 0 ]; then
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
|
||||
>>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
else
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
|
|
@ -852,7 +863,7 @@ walcheck "${#ALL_JOBIDS[@]}"
|
|||
# run_force_snapshot_round's own header comment. Single round, not a loop —
|
||||
# emits a verdict and exits, like chaotic mode.
|
||||
if [ "$MODE" = "force-snapshot" ]; then
|
||||
log "MODE=force-snapshot — forcing one full-checkpoint catch-up round (max-wal-records=$MAX_WAL_RECORDS)"
|
||||
log "MODE=force-snapshot — forcing one full-checkpoint catch-up round (max-wal-records=$MAX_WAL_RECORDS wal-segment-bytes=$WAL_SEGMENT_BYTES)"
|
||||
run_force_snapshot_round
|
||||
scan_bug924
|
||||
verdict_emit "force-snapshot" 1 "${#ALL_JOBIDS[@]}" \
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue