Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.
Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().
Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.
New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.
Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.
g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
The chaotic soak's walcheck()/walcheck_count() previously forced a
static `1 + 2*njobs` boundary onto walcompare, which doesn't account
for per-election NOOP WAL entries and overshoots the real tip --
producing spurious committed_divergence counts on an otherwise-clean
cluster (verified via waldump: byte-identical, gap-free WALs across
all 3 nodes despite a reported divergence=7).
Replaces the static estimate with the LIVE, authoritative committedLsn
(via the existing probe_primary client probe), applied as an explicit
--committed boundary only when the cluster is genuinely converged
(cluster_committed_converged) -- falling back to walcompare's own
auto-derive (min CRC-intact tip) when not, so a legitimately-lagging
just-restarted standby never gets a false "MISSING" verdict mid-chaos.
Pure auto-derive alone can't catch a member that permanently lost a
committed entry (its own low tip just becomes the shared boundary),
so this restores exact-boundary detection at every genuinely-settled
instant while keeping the auto-derive fallback for in-flight kills.
Also fixes a live-caught edge case in the same mechanism: a transient
ctrlstatus race can report committedLsn=0 for the primary right after
reconvergence; live_committed_boundary() now rejects a bare "0" as
untrustworthy and falls back to auto-derive, instead of handing
walcompare a vacuous --committed 0 boundary that would check nothing.
Selftest 17/17. Three independent fresh 6-round chaotic soaks (own
run + reviewer's reproduction + post-guard-fix re-run) all PASS:
committed_divergence=0, committed_job_loss=0, 6/6 elections, 21/21
jobs present throughout. No production C code touched.
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.
Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
after being restarted into the churning cluster (recovery->failover/PreVote),
dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
debug+core+gdb env.
Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Release build block-buffers --logfile (flush on signal only), so log-grep
primary/convergence detection false-aborted. Replace all log-grep detection
with client probes (xce_test ctrlstatus): probe_primary/node_probe/
cluster_committed_converged. Both modes use shared helpers; 1.2 verdict
self-test 17/17 intact; bug-854 isolation preserved. Live release-binary
runs reach the verdict (chaotic re-confirms committed_divergence=3 / job
loss 0; sequential PASS). No production code touched. Phase 1 complete.
Verdict: raft-drive-async introduces NO new regression.
- Unit: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites green;
dispatch/snapshot/configpush flags default OFF, inert).
- Live kill-based failover soak on a real fs (~/tmp), release + ASAN builds:
zero committed-JOB loss every completed round (jobs 3->12 monotonic).
- The 6-round CHAOTIC soak does not run clean to round 6 (fails ~round 4), but a
baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4
IDENTICALLY (election stall / reconverge). The round-4 committed-prefix
divergence at LSN 9 is the documented pre-existing bug-851/795 family
(chaotic-failover silent divergence), present before Phase 3 and out of scope
for this spec. Logged as bug-1225 (baseline-confirmed).
Also: make the xce-test-ha soak driver's controller/xce_test paths
env-overridable (XCE_HA_BIN / XCE_HA_XT, backward-compatible defaults) — used to
run the baseline binaries through the same driver for the regression comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Add an optional [dir] argument to the /xce-test-ha skill so the 3-node
soak cluster can be deployed to a caller-chosen path.
- run-ha-soak.sh: deploy dir resolves positional $1 > $XCE_HA_TOP > default;
relative paths absolutized in pure bash (this host's busybox realpath
lacks -m and would mangle nonexistent/relative paths).
- Fail-loud rm -rf guard: writes a .xce-ha-deploy sentinel at deploy time and
refuses to wipe a non-empty dir lacking it (protects a typo'd home/source
path); empty/nonexistent dirs stay safe to (re)deploy into.
- SKILL.md: document /xce-test-ha [rounds] [dir] and the safety semantics.
Verified: guard refuses an unmarked non-empty dir (dir survives, exit 1);
full 5-round soak into a custom dir PASS (18 jobs, zero job loss), cluster
landed in the named dir while the default dir was untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
cleanup.sh is generated by deploy-ha.sh and already does the right thing:
keeps ctrl-1/conf/* and all certs, wipes ctrl-{2..N}/conf, all data/, and logs.
Update skill and CLAUDE.md to use it as the primary clean-restart command
instead of manual rm loops.
- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a
local 3-node XCE HA controller cluster. Covers build prerequisites,
deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace,
health verification checks, clean restart procedure, and a
troubleshooting table for common failure modes.
- src/controller/CLAUDE.md: add HA Cluster Deployment section with
quick-deploy commands, health check one-liners, and reference to the
xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
- create_lost_and_found(): provide hostattrs_t so makeWorker succeeds
- collect_queues/collect_workers (dataex.c): skip lost_and_found sentinel
so it is never pushed to apiserver or scheduler
- mark_worker_unavail(): reset hosttype/hostmodel to "unknown" and zero
all numeric attrs, then update resvals and slots — prevents stale
capacity from being visible while the host is down
- restore_worker(): apply the same attr reset on WAL recovery so
restored-but-unavailable workers advertise no capacity until reconnect;
also remove dead calloc() that was immediately overwritten by makeWorker
- Extract reset_hostattrs_unknown() helper to eliminate copy-paste
- Define XCE_LOST_AND_FOUND_NAME constant in config_repl.h; replace all
six raw "lost_and_found" string literals across config_repl.c and dataex.c
