Chaotic mode's run_chaotic_rounds restarts "the previously-downed node"
every round unconditionally via start_node(\$DOWN), but a round that skips
(no primary to kill) never updates \$DOWN -- so a stale node name reaches
start_node again while its own earlier restart is still alive, stacking a
second xce_controller on the same --myname/port without ever killing the
first. Six stacked ctrl-2 processes were observed by the end of a 10-round
run.
Fixed inside start_node itself with a pgrep guard, rather than reworking
run_chaotic_rounds's DOWN bookkeeping, so every call site is covered:
refuse to launch a duplicate if "xce_controller --myname \$1" is already
running.
Verified: re-ran the identical 10-round --chaotic soak that surfaced this.
The guard fired on every skip-round; exactly 3 xce_controller processes
alive throughout the entire run.
Note: the same soak run still hits a genuine, separate election-liveness
stall (a surviving node fenced non-promotable by a real apply-correctness
divergence, traced to non-identical duplicate WAL submits for the same
jobid) -- filed as bug-1419, not fixed here.
Rongsong Shen
9b173c9a5e