A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.
Two structural gaps let that one typo cascade this badly:
1. deploy-ha.sh had no pre-flight check that target ports were free --
it would happily start a second Raft group on ports an existing one
already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
(reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
pidfiles unconditionally as part of state cleanup, with no check
that the process each one pointed to was actually dead first --
silently erasing the only signal that guard depends on.
Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:
- Added a port pre-flight check before any deploy state is created:
refuses with the exact blocking PID/process if a target port is
already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
canonical, typo-proof way to stop everything a specific deploy
started, since it reads that deploy's own tracked PIDs instead of
matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
didn't wait for worker/scheduler/apiserver to write their pidfiles
before returning "done", so cleanup.sh/stop-all.sh run immediately
after could orphan those two specifically -- the same zombie-process
class this whole fix targets, via a narrower window instead of an
operator typo. Added a short pidfile-appearance poll to close it.
SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.
bug-1816.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Rongsong Shen
7f3237aee1