fix(xce-deploy-ha): stop a stale deploy's processes surviving a hand-typed cleanup mistake
A live 3-node HA cluster went into a repeated PreVote-failover storm (term climbing, RDHUP loops, "scheduler not connected") even though system load was low. Root cause: a hand-typed cleanup command had a transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path) and silently killed nothing, leaving a full controller cluster alive. A fresh deploy then bound a second, independent cluster to the exact same ports, and the two clusters' vote traffic genuinely cross-talked -- producing symptoms that looked like a real HA bug but weren't. Two structural gaps let that one typo cascade this badly: 1. deploy-ha.sh had no pre-flight check that target ports were free -- it would happily start a second Raft group on ports an existing one already held, instead of failing cleanly. 2. start.sh/start-all.sh already had a correct "already running" guard (reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those pidfiles unconditionally as part of state cleanup, with no check that the process each one pointed to was actually dead first -- silently erasing the only signal that guard depends on. Fixes, all live-verified end to end against real controller/worker/ scheduler/apiserver binaries: - Added a port pre-flight check before any deploy state is created: refuses with the exact blocking PID/process if a target port is already listening. - cleanup.sh now stops every live PID in $TOP/run/*.pid before removing any pidfile or state. - Added stop-all.sh (generated alongside start.sh/cleanup.sh): the canonical, typo-proof way to stop everything a specific deploy started, since it reads that deploy's own tracked PIDs instead of matching process names/paths by hand. - Found and fixed a related race while testing the above: start-all.sh didn't wait for worker/scheduler/apiserver to write their pidfiles before returning "done", so cleanup.sh/stop-all.sh run immediately after could orphan those two specifically -- the same zombie-process class this whole fix targets, via a narrower window instead of an operator typo. Added a short pidfile-appearance poll to close it. SKILL.md updated to prefer a known deploy's own stop-all.sh over a hand-typed kill command, document cleanup.sh's new stop-then-wipe behavior, and add two Troubleshooting rows matching this incident directly. bug-1816. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
dfabb67447
commit
7f3237aee1
2 changed files with 177 additions and 15 deletions
|
|
@ -186,6 +186,46 @@ fi
|
|||
IFS=',' read -ra PEER_ARRAY <<< "$PEERS"
|
||||
N=${#PEER_ARRAY[@]}
|
||||
|
||||
# Pre-flight: refuse to deploy on top of ports a DIFFERENT still-running
|
||||
# deployment already holds. Silently starting a second cluster bound to the
|
||||
# same ports doesn't fail cleanly -- it causes real cross-talk (mixed vote
|
||||
# traffic, corrupted term/leader state) between two otherwise-unrelated Raft
|
||||
# groups, which looks exactly like a genuine HA bug. Found live, 2026-09-15:
|
||||
# two full 3-node clusters ended up sharing ports 9901-9903 after a stale
|
||||
# deploy's processes were never actually killed (a hand-typed cleanup
|
||||
# command had a typo), producing a real failover storm from otherwise-
|
||||
# correct code. start.sh/start-all.sh's own "already running" guard further
|
||||
# below only catches THIS $TOP's own pidfiles -- it can't see a different,
|
||||
# unrelated deploy (or this same $TOP after cleanup.sh wiped its pidfiles
|
||||
# out from under a still-alive process, the exact gap that let this
|
||||
# happen -- see cleanup.sh's own fix below), so this check is independent,
|
||||
# at the network level, before any state for this deploy is touched.
|
||||
if command -v ss >/dev/null 2>&1; then
|
||||
PORT_CONFLICT=0
|
||||
for peer in "${PEER_ARRAY[@]}"; do
|
||||
peer_port="${peer##*:}"
|
||||
if ss -tln 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p {f=1} END{exit !f}'; then
|
||||
echo "Error: port $peer_port is already in use by a running process:"
|
||||
ss -tlnp 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p' || true
|
||||
PORT_CONFLICT=1
|
||||
fi
|
||||
done
|
||||
if [ "$PORT_CONFLICT" -eq 1 ]; then
|
||||
echo ""
|
||||
echo "Refusing to deploy on top of an already-occupied port -- this usually"
|
||||
echo "means an earlier deployment (possibly to a DIFFERENT top-dir) was never"
|
||||
echo "fully stopped. If it has its own \$TOP/scripts/stop-all.sh, run that;"
|
||||
echo "otherwise stop it by process name, then re-run this deploy:"
|
||||
echo " kill -9 \$(pgrep xce_controller) 2>/dev/null || true"
|
||||
echo " kill -9 \$(pgrep -f xcecd_virtual) 2>/dev/null || true"
|
||||
echo " kill -9 \$(pgrep xce_scheduler) 2>/dev/null || true"
|
||||
echo " sudo kill -9 \$(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "Warning: 'ss' not found -- skipping the port-conflict pre-flight check. If this deploy collides with an already-running one, expect confusing HA behavior rather than a clean failure."
|
||||
fi
|
||||
|
||||
# Create directory structure
|
||||
mkdir -p "$TOP/certs"
|
||||
mkdir -p "$TOP/logs"
|
||||
|
|
@ -269,10 +309,29 @@ exec $BINARY --myname \$NAME --peers "$PEERS_STR" --workdir $TOP/controller --ce
|
|||
EOF
|
||||
chmod +x "$TOP/scripts/start.sh"
|
||||
|
||||
# Generate cleanup script
|
||||
# Generate cleanup script. Stops anything still running for this deploy
|
||||
# BEFORE wiping its pidfile -- removing a pidfile while its process is still
|
||||
# alive would silently defeat start.sh/start-all.sh's own "already running"
|
||||
# guard on the next deploy, letting a stale controller/worker/scheduler
|
||||
# survive undetected and collide with a freshly-started one on the same
|
||||
# ports (this is exactly how a real HA failover storm happened live,
|
||||
# 2026-09-15 -- see the port pre-flight check above for the full story).
|
||||
cat > "$TOP/scripts/cleanup.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
for PF in $TOP/run/*.pid; do
|
||||
[ -f "\$PF" ] || continue
|
||||
PID=\$(cat "\$PF" 2>/dev/null || true)
|
||||
NAME=\$(basename "\$PF" .pid)
|
||||
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "cleanup.sh: stopping \$NAME (PID \$PID)"
|
||||
kill -9 "\$PID" 2>/dev/null || true
|
||||
sleep 0.2
|
||||
if kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
rm -f $TOP/logs/*.log
|
||||
rm -f $TOP/run/*.pid
|
||||
for i in \$(seq 1 $N); do
|
||||
|
|
@ -284,6 +343,37 @@ done
|
|||
EOF
|
||||
chmod +x "$TOP/scripts/cleanup.sh"
|
||||
|
||||
# Generate stop-all script -- the canonical, typo-proof way to stop every
|
||||
# component THIS deploy started. Reads $TOP/run/*.pid (scoped exactly to
|
||||
# what this deploy is tracking) instead of matching process names/cmdlines
|
||||
# by hand, so there's no path/name string to get wrong. Unlike cleanup.sh,
|
||||
# leaves all state (data/conf/logs) in place -- just stops processes.
|
||||
cat > "$TOP/scripts/stop-all.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
STOPPED=0
|
||||
for PF in $TOP/run/*.pid; do
|
||||
[ -f "\$PF" ] || continue
|
||||
PID=\$(cat "\$PF" 2>/dev/null || true)
|
||||
NAME=\$(basename "\$PF" .pid)
|
||||
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Stopping \$NAME (PID \$PID)"
|
||||
kill -9 "\$PID" 2>/dev/null || true
|
||||
sleep 0.2
|
||||
if kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
|
||||
else
|
||||
STOPPED=1
|
||||
fi
|
||||
fi
|
||||
rm -f "\$PF"
|
||||
done
|
||||
if [ "\$STOPPED" -eq 0 ]; then
|
||||
echo "Nothing was running for this deploy ($TOP)."
|
||||
fi
|
||||
EOF
|
||||
chmod +x "$TOP/scripts/stop-all.sh"
|
||||
|
||||
# Generate worker start script (component-failover-adoption's --peers path):
|
||||
# same $PEERS_STR the controllers themselves were just given, so a worker
|
||||
# started via this script fails over across the identical candidate list.
|
||||
|
|
@ -623,22 +713,44 @@ else
|
|||
echo "Controller cluster healthy (primary elected)."
|
||||
fi
|
||||
|
||||
# wait_for_pidfile: poll up to 5s for a just-launched background process to
|
||||
# reach its own "echo \$\$ > \$PIDFILE" line. Without this, start-all.sh
|
||||
# could return "done" before a pidfile exists at all -- and cleanup.sh/
|
||||
# stop-all.sh's own pidfile-based stop logic (see those scripts' own
|
||||
# comments) would then find nothing to kill AND still wipe run/*.pid on its
|
||||
# final pass, orphaning the process it never got a chance to see. This is
|
||||
# the same failure class (a live process with no pidfile pointing at it)
|
||||
# that caused a real HA failover storm live, 2026-09-15 -- just via a
|
||||
# different path (a race here, versus a hand-typed kill typo there).
|
||||
wait_for_pidfile() {
|
||||
local pf="\$1" label="\$2" n
|
||||
for n in \$(seq 1 25); do
|
||||
[ -f "\$pf" ] && return 0
|
||||
sleep 0.2
|
||||
done
|
||||
echo "Warning: \$label did not write a pidfile at \$pf within 5s -- it may have failed to start, or started slower than expected. Check $TOP/logs/ for details before assuming it's running."
|
||||
return 1
|
||||
}
|
||||
|
||||
if [ -x "$TOP/scripts/start-worker.sh" ]; then
|
||||
echo "=== start-all.sh: starting worker ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-worker.sh" worker-1 -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/worker-1.pid" "worker-1"
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-scheduler.sh" ]; then
|
||||
echo "=== start-all.sh: starting scheduler (-vvv) ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-scheduler.sh" -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/scheduler.pid" "scheduler"
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-apiserver.sh" ]; then
|
||||
echo "=== start-all.sh: starting apiserver ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-apiserver.sh" </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/apiserver.pid" "apiserver"
|
||||
fi
|
||||
|
||||
echo "=== start-all.sh: done -- cluster left running ==="
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue