lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(xce-test-ha): guard start_node against duplicate launches (bug-1418)

Browse source 
Chaotic mode's run_chaotic_rounds restarts "the previously-downed node"
every round unconditionally via start_node(\$DOWN), but a round that skips
(no primary to kill) never updates \$DOWN -- so a stale node name reaches
start_node again while its own earlier restart is still alive, stacking a
second xce_controller on the same --myname/port without ever killing the
first. Six stacked ctrl-2 processes were observed by the end of a 10-round
run.

Fixed inside start_node itself with a pgrep guard, rather than reworking
run_chaotic_rounds's DOWN bookkeeping, so every call site is covered:
refuse to launch a duplicate if "xce_controller --myname \$1" is already
running.

Verified: re-ran the identical 10-round --chaotic soak that surfaced this.
The guard fired on every skip-round; exactly 3 xce_controller processes
alive throughout the entire run.

Note: the same soak run still hits a genuine, separate election-liveness
stall (a surviving node fenced non-promotable by a real apply-correctness
divergence, traced to non-identical duplicate WAL submits for the same
jobid) -- filed as bug-1419, not fixed here.
This commit is contained in:
Rongsong Shen 2026-07-07 13:53:54 +00:00
parent d5122752fe
commit 9b173c9a5e
2 changed files with 70 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

12
.claude/skills/xce-test-ha/run-ha-soak.sh
View file

@ -214,6 +214,18 @@ trap cleanup EXIT INT TERM
xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; }
start_node() {
# bug-1418: chaotic mode's DOWN bookkeeping restarts "the previously-downed
# node" every round unconditionally — but a round that skips (no primary to
# kill) never updates DOWN, so a stale node name can reach here again while
# its OWN earlier restart is still alive, stacking a second xce_controller
# on the same --myname/port without ever killing the first. Guard here
# (not by "fixing" DOWN's bookkeeping) so every call site is covered:
# refuse to launch a duplicate if one is already running.
if [ -n "$(pgrep -f -- "xce_controller --myname $1 ")" ]; then
log "start_node: $1 already running — skipping duplicate launch (bug-1418)"
return 0
fi
# Capture each controller's stdout+stderr to a per-node file so a SILENT crash
# (segfault/abort to stderr — the --logfile only holds tlog INFO/DEBUG, not a
# libc/ASAN crash message) leaves a forensic trace. Appended (>>) so a