Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single
source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied
last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved
reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs
loudly, never self-terminates. 8 unit cases, mutation-verified.
Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.
Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
persist pcfg (override applied before persist_init); 0 = use default, positive
overrides — so operators can tune per environment.
TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
