lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): close 5.10a — WAL segment rotation + endpoint UAF blocking real snapshot catch-up

Browse source 
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):

1. WAL segment rotation was gated on a hardcoded 64MB compile-time
   constant; trim_wal_segments() never deletes the active (currently-open)
   segment. At soak job counts a segment never rotated, so nothing was
   ever physically trimmable and ha_evaluate_repl_action always chose
   STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
   threading a wal_segment_bytes override through persist_config_t
   (mirrors the --max-wal-records pattern: config.h default, persist.h
   setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
   CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
   --wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
   now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.

2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
   (include/netio.h): the macro calls ep->msg_handler(ep) synchronously
   (not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
   doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
   handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
   endpoint_unref) could drop the endpoint's last ref and free it inside
   the handler call, so the trailing write hit freed memory. This is the
   first-ever live exercise of a node receiving a snapshot transfer while
   mid-election. Fixed by holding a ref across the synchronous call, same
   pattern task 5.8 established for the async path.

With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.

Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
This commit is contained in:
Rongsong Shen 2026-07-02 04:11:20 +00:00
parent b9ad01878c
commit 1439e2655e
11 changed files with 115 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/include/persist.h
View file

@ -100,6 +100,7 @@ typedef struct {
uint32_t worker_retain_s; /* retain unavailable workers N seconds (default 86400) */
uint32_t sync_timeout_ms; /* sync replication ACK timeout ms (default 50) */
uint32_t max_wal_records; /* checkpoint trigger: records since last snap (default 10000) */
uint32_t wal_segment_bytes; /* physical WAL segment file size (default 64 MB) */
} persist_config_t;
static inline persist_config_t persist_config_defaults(void)
@ -114,6 +115,7 @@ static inline persist_config_t persist_config_defaults(void)
c.worker_retain_s = DEFAULT_WORKER_RETAIN_S;
c.sync_timeout_ms = DEFAULT_SYNC_TIMEOUT_MS;
c.max_wal_records = DEFAULT_MAX_WAL_RECORDS;
c.wal_segment_bytes = DEFAULT_WAL_SEGMENT_BYTES;
return c;
}
@ -154,6 +156,25 @@ static inline void persist_config_set_max_wal_records(persist_config_t *c,
c->max_wal_records = override_n;
}
/*
* persist_config_set_wal_segment_bytes — apply the --wal-segment-bytes
* override (raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1:
* the physical WAL segment size, not the record-count checkpoint threshold,
* gates when trim_wal_segments() has anything to delete — a segment is
* never rotated out until it fills, so forcing a real snapshot-triggered
* empty-rejoin in the soak requires a small deterministic segment size
* rather than waiting on the 64 MB production default).
*
* Same convention as persist_config_set_max_wal_records: override == 0
* means "flag not passed", leaves the default segment size untouched.
*/
static inline void persist_config_set_wal_segment_bytes(persist_config_t *c,
uint32_t override_bytes)
{
if (override_bytes > 0)
c->wal_segment_bytes = override_bytes;
}
/* ============================================================
* Key→index mapping for checkpoint serialization (Part 3.8)
* ============================================================ */