fix(xce-deploy-ha): stop a stale deploy's processes surviving a hand-typed cleanup mistake
A live 3-node HA cluster went into a repeated PreVote-failover storm (term climbing, RDHUP loops, "scheduler not connected") even though system load was low. Root cause: a hand-typed cleanup command had a transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path) and silently killed nothing, leaving a full controller cluster alive. A fresh deploy then bound a second, independent cluster to the exact same ports, and the two clusters' vote traffic genuinely cross-talked -- producing symptoms that looked like a real HA bug but weren't. Two structural gaps let that one typo cascade this badly: 1. deploy-ha.sh had no pre-flight check that target ports were free -- it would happily start a second Raft group on ports an existing one already held, instead of failing cleanly. 2. start.sh/start-all.sh already had a correct "already running" guard (reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those pidfiles unconditionally as part of state cleanup, with no check that the process each one pointed to was actually dead first -- silently erasing the only signal that guard depends on. Fixes, all live-verified end to end against real controller/worker/ scheduler/apiserver binaries: - Added a port pre-flight check before any deploy state is created: refuses with the exact blocking PID/process if a target port is already listening. - cleanup.sh now stops every live PID in $TOP/run/*.pid before removing any pidfile or state. - Added stop-all.sh (generated alongside start.sh/cleanup.sh): the canonical, typo-proof way to stop everything a specific deploy started, since it reads that deploy's own tracked PIDs instead of matching process names/paths by hand. - Found and fixed a related race while testing the above: start-all.sh didn't wait for worker/scheduler/apiserver to write their pidfiles before returning "done", so cleanup.sh/stop-all.sh run immediately after could orphan those two specifically -- the same zombie-process class this whole fix targets, via a narrower window instead of an operator typo. Added a short pidfile-appearance poll to close it. SKILL.md updated to prefer a known deploy's own stop-all.sh over a hand-typed kill command, document cleanup.sh's new stop-then-wipe behavior, and add two Troubleshooting rows matching this incident directly. bug-1816. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
dfabb67447
commit
7f3237aee1
2 changed files with 177 additions and 15 deletions
|
|
@ -115,6 +115,20 @@ to the intended interpreter).
|
|||
|
||||
## Step 2 — Stop any running instances
|
||||
|
||||
**If you know the `$TOP` of a previous deploy, prefer its own generated
|
||||
`stop-all.sh`** — it reads that deploy's own `$TOP/run/*.pid` files (exactly
|
||||
what it started, nothing else) instead of matching process names/paths by
|
||||
hand, so there's no string to typo:
|
||||
|
||||
```bash
|
||||
"$BASH_BIN" "$TOP/scripts/stop-all.sh" # only if $TOP already exists from a prior deploy
|
||||
```
|
||||
|
||||
Otherwise (first deploy in this session, or you don't know/remember the
|
||||
prior `$TOP`), fall back to the blanket by-process-name kill — this is what
|
||||
actually needs to run before a **fresh** deploy to a `$TOP` you've never
|
||||
used before, or when a prior deploy's own scripts are gone/inaccessible:
|
||||
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
||||
|
|
@ -122,6 +136,22 @@ kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
|||
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root — plain kill can't touch it
|
||||
```
|
||||
|
||||
**Don't skip this and rely on Step 3 alone.** `deploy-ha.sh` refuses to
|
||||
start a *second* cluster on top of ports an existing one already holds
|
||||
(see Step 3's port pre-flight check below) — but that only catches a
|
||||
**redeploy attempt**. If you instead just run `start-all.sh` (or `start.sh`)
|
||||
again without redeploying, and an old, no-longer-pidfile-tracked process is
|
||||
still alive on those ports from an earlier session, you can still get two
|
||||
clusters cross-talking. This is exactly what happened live, 2026-09-15: an
|
||||
operator's own hand-typed `ps axwww | grep xce-ha-deploy | ... | xargs kill`
|
||||
had a typo (`xce-ha-deploy` vs. the real `xce-deploy-ha` in the workdir
|
||||
path) and silently killed nothing, leaving a full 3-node cluster running
|
||||
that a fresh deploy to the same ports then collided with — producing a
|
||||
genuine HA failover storm (mixed vote traffic between the two unrelated
|
||||
clusters) that looked like a code bug but wasn't. Prefer a generated
|
||||
`stop-all.sh` over any hand-typed process-matching command whenever one is
|
||||
available, precisely because it can't have that class of typo.
|
||||
|
||||
## Step 3 — Deploy cluster
|
||||
|
||||
Join peers as a comma-separated string and call the deploy script — always
|
||||
|
|
@ -146,7 +176,8 @@ a rebuild, nothing is ever reused across deploys:
|
|||
real security property since TLS verification is `SSL_VERIFY_NONE`
|
||||
everywhere in this codebase). Errors if `openssl` isn't on `PATH`.
|
||||
- The bash baked into the shebang of every script `deploy-ha.sh` generates
|
||||
(`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is
|
||||
(`start.sh`, `cleanup.sh`, `stop-all.sh`, `start-worker.sh`,
|
||||
`start-scheduler.sh`) is
|
||||
whatever `command -v bash` resolves to *inside deploy-ha.sh's own
|
||||
invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that
|
||||
interpreter under `nix develop`. This skill always still invokes each
|
||||
|
|
@ -160,7 +191,7 @@ This:
|
|||
- Generates `xce.crt`/`xce.key` directly into `$TOP/certs/`
|
||||
- Places `cluster.bin.<tag>` + `conf/current` in `controller/ctrl-1/conf/` only
|
||||
(standbys receive config from primary via `OPCode_replConfigPush`)
|
||||
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
|
||||
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`), `cleanup.sh`, and `stop-all.sh` (stop everything for this deploy, no state wipe — reads `$TOP/run/*.pid`, the canonical alternative to hand-typing a process-name kill)
|
||||
- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh`
|
||||
**if** the relevant binary exists alongside `$BINARY` (same build dir) —
|
||||
soft-skipped with a warning, not a hard error, if missing (so
|
||||
|
|
@ -405,38 +436,55 @@ RAFT[p0] term=2 event=became_leader
|
|||
Stop all instances, clean up state, and restart without redeploying:
|
||||
|
||||
```bash
|
||||
# 1. Stop running controllers (and worker/scheduler, if you started them)
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
||||
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
||||
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root
|
||||
|
||||
# 2. Run the generated cleanup script
|
||||
# 1. Run the generated cleanup script -- this now ALSO stops every live
|
||||
# process it's about to orphan (reads $TOP/run/*.pid) before wiping
|
||||
# state, precisely so a still-running component never survives with its
|
||||
# pidfile gone out from under it (see below for why that matters).
|
||||
# Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
|
||||
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
|
||||
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log, run/*.pid
|
||||
"$BASH_BIN" "$TOP/scripts/cleanup.sh"
|
||||
|
||||
# 3. Restart all nodes
|
||||
# 2. Restart all nodes
|
||||
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
|
||||
sleep 0.5
|
||||
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
|
||||
sleep 0.5
|
||||
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
|
||||
|
||||
# 4. Restart worker/scheduler if you had them running (Step 4b) — the
|
||||
# 3. Restart worker/scheduler if you had them running (Step 4b) — the
|
||||
# generated scripts still carry the same --peers, nothing to redo there
|
||||
"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &
|
||||
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &
|
||||
```
|
||||
|
||||
To only stop everything (no state wipe, e.g. before redeploying fresh peers
|
||||
or a new binary), run `"$BASH_BIN" "$TOP/scripts/stop-all.sh"` instead —
|
||||
same pidfile-based stop logic as `cleanup.sh`, but leaves all state in place.
|
||||
|
||||
**What cleanup.sh (and stop-all.sh) stop first, before touching any state:**
|
||||
- Every PID recorded in `$TOP/run/*.pid` that's still alive — `ctrl-{1..N}`,
|
||||
`worker-1`, `scheduler`, `apiserver`, whichever were actually started.
|
||||
**Why this matters:** the pidfile is the *only* thing `start.sh`/
|
||||
`start-all.sh`'s own "already running" guard checks before starting a new
|
||||
process on the same ports. Deleting a pidfile without first confirming its
|
||||
process is dead (what the OLD `cleanup.sh` did) silently defeats that
|
||||
guard on the next deploy/restart, letting a stale process survive
|
||||
undetected and collide with a freshly-started one on the same ports. This
|
||||
is exactly what caused a real HA failover storm live, 2026-09-15 (see
|
||||
Step 2's own note above) — the underlying gap this fixes, independent of
|
||||
the hand-typed-kill-command mistake that triggered it that day.
|
||||
- The real (root-run) worker may not stop from a plain `kill -9` if you're
|
||||
not running as root yourself — `cleanup.sh`/`stop-all.sh` will print a
|
||||
warning naming the PID; follow up with `sudo kill -9 <pid>`.
|
||||
|
||||
**What cleanup.sh preserves:**
|
||||
- `$TOP/certs/` — shared TLS cert and key
|
||||
- `controller/ctrl-1/conf/cluster.bin.*` + `conf/current` — primary config (standbys re-receive via push)
|
||||
|
||||
**What cleanup.sh removes:**
|
||||
**What cleanup.sh removes (after stopping the processes above):**
|
||||
- All `controller/ctrl-{1..N}/data/` dirs (WAL, snapshots, `ha-term.bin`) — forces fresh election next start
|
||||
- All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush`
|
||||
- All `logs/*.log` — clean log slate
|
||||
- All `logs/*.log` and `run/*.pid` — clean log/pid slate
|
||||
|
||||
**Not touched by cleanup.sh, if you ran the real worker:** `/var/run/xcecd.db`
|
||||
and `/var/run/.xcecd.sock` live outside `$TOP` entirely (hardcoded, not
|
||||
|
|
@ -474,3 +522,5 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
|
|||
| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template |
|
||||
| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first |
|
||||
| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists |
|
||||
| `deploy-ha.sh` refuses with `Error: port <N> is already in use by a running process` (before creating any state) | A different, still-running deployment (this `$TOP` or a different one entirely) already holds one of the ports you're deploying to — the pre-flight check (Step 3) catches this before it can cause cross-talk between two clusters | The error prints the PID/process holding the port. If it's a known prior deploy, run its own `$TOP/scripts/stop-all.sh`; otherwise use Step 2's blanket kill-by-name, then re-run the deploy |
|
||||
| Controller HA is inexplicably failing over (repeated PreVote elections, `primary_ctrl_fd RDHUP`) despite low system load, and log files show more than one PID for the same `ctrl-N` role | Two separate deployments ended up bound to the same ports (see the port-conflict row above) — their vote traffic genuinely cross-talks, producing symptoms that look like a real HA bug but aren't | `ps axwww \| grep -E "xce_controller\|xce_scheduler\|xcecd"` — if more than one process exists per role/port, kill the stale set (or everything, then redeploy clean); double-check any hand-typed kill/grep pattern actually matches these processes' real command lines before trusting it did anything |
|
||||
|
|
|
|||
|
|
@ -186,6 +186,46 @@ fi
|
|||
IFS=',' read -ra PEER_ARRAY <<< "$PEERS"
|
||||
N=${#PEER_ARRAY[@]}
|
||||
|
||||
# Pre-flight: refuse to deploy on top of ports a DIFFERENT still-running
|
||||
# deployment already holds. Silently starting a second cluster bound to the
|
||||
# same ports doesn't fail cleanly -- it causes real cross-talk (mixed vote
|
||||
# traffic, corrupted term/leader state) between two otherwise-unrelated Raft
|
||||
# groups, which looks exactly like a genuine HA bug. Found live, 2026-09-15:
|
||||
# two full 3-node clusters ended up sharing ports 9901-9903 after a stale
|
||||
# deploy's processes were never actually killed (a hand-typed cleanup
|
||||
# command had a typo), producing a real failover storm from otherwise-
|
||||
# correct code. start.sh/start-all.sh's own "already running" guard further
|
||||
# below only catches THIS $TOP's own pidfiles -- it can't see a different,
|
||||
# unrelated deploy (or this same $TOP after cleanup.sh wiped its pidfiles
|
||||
# out from under a still-alive process, the exact gap that let this
|
||||
# happen -- see cleanup.sh's own fix below), so this check is independent,
|
||||
# at the network level, before any state for this deploy is touched.
|
||||
if command -v ss >/dev/null 2>&1; then
|
||||
PORT_CONFLICT=0
|
||||
for peer in "${PEER_ARRAY[@]}"; do
|
||||
peer_port="${peer##*:}"
|
||||
if ss -tln 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p {f=1} END{exit !f}'; then
|
||||
echo "Error: port $peer_port is already in use by a running process:"
|
||||
ss -tlnp 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p' || true
|
||||
PORT_CONFLICT=1
|
||||
fi
|
||||
done
|
||||
if [ "$PORT_CONFLICT" -eq 1 ]; then
|
||||
echo ""
|
||||
echo "Refusing to deploy on top of an already-occupied port -- this usually"
|
||||
echo "means an earlier deployment (possibly to a DIFFERENT top-dir) was never"
|
||||
echo "fully stopped. If it has its own \$TOP/scripts/stop-all.sh, run that;"
|
||||
echo "otherwise stop it by process name, then re-run this deploy:"
|
||||
echo " kill -9 \$(pgrep xce_controller) 2>/dev/null || true"
|
||||
echo " kill -9 \$(pgrep -f xcecd_virtual) 2>/dev/null || true"
|
||||
echo " kill -9 \$(pgrep xce_scheduler) 2>/dev/null || true"
|
||||
echo " sudo kill -9 \$(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "Warning: 'ss' not found -- skipping the port-conflict pre-flight check. If this deploy collides with an already-running one, expect confusing HA behavior rather than a clean failure."
|
||||
fi
|
||||
|
||||
# Create directory structure
|
||||
mkdir -p "$TOP/certs"
|
||||
mkdir -p "$TOP/logs"
|
||||
|
|
@ -269,10 +309,29 @@ exec $BINARY --myname \$NAME --peers "$PEERS_STR" --workdir $TOP/controller --ce
|
|||
EOF
|
||||
chmod +x "$TOP/scripts/start.sh"
|
||||
|
||||
# Generate cleanup script
|
||||
# Generate cleanup script. Stops anything still running for this deploy
|
||||
# BEFORE wiping its pidfile -- removing a pidfile while its process is still
|
||||
# alive would silently defeat start.sh/start-all.sh's own "already running"
|
||||
# guard on the next deploy, letting a stale controller/worker/scheduler
|
||||
# survive undetected and collide with a freshly-started one on the same
|
||||
# ports (this is exactly how a real HA failover storm happened live,
|
||||
# 2026-09-15 -- see the port pre-flight check above for the full story).
|
||||
cat > "$TOP/scripts/cleanup.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
for PF in $TOP/run/*.pid; do
|
||||
[ -f "\$PF" ] || continue
|
||||
PID=\$(cat "\$PF" 2>/dev/null || true)
|
||||
NAME=\$(basename "\$PF" .pid)
|
||||
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "cleanup.sh: stopping \$NAME (PID \$PID)"
|
||||
kill -9 "\$PID" 2>/dev/null || true
|
||||
sleep 0.2
|
||||
if kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
rm -f $TOP/logs/*.log
|
||||
rm -f $TOP/run/*.pid
|
||||
for i in \$(seq 1 $N); do
|
||||
|
|
@ -284,6 +343,37 @@ done
|
|||
EOF
|
||||
chmod +x "$TOP/scripts/cleanup.sh"
|
||||
|
||||
# Generate stop-all script -- the canonical, typo-proof way to stop every
|
||||
# component THIS deploy started. Reads $TOP/run/*.pid (scoped exactly to
|
||||
# what this deploy is tracking) instead of matching process names/cmdlines
|
||||
# by hand, so there's no path/name string to get wrong. Unlike cleanup.sh,
|
||||
# leaves all state (data/conf/logs) in place -- just stops processes.
|
||||
cat > "$TOP/scripts/stop-all.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
STOPPED=0
|
||||
for PF in $TOP/run/*.pid; do
|
||||
[ -f "\$PF" ] || continue
|
||||
PID=\$(cat "\$PF" 2>/dev/null || true)
|
||||
NAME=\$(basename "\$PF" .pid)
|
||||
if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Stopping \$NAME (PID \$PID)"
|
||||
kill -9 "\$PID" 2>/dev/null || true
|
||||
sleep 0.2
|
||||
if kill -0 "\$PID" 2>/dev/null; then
|
||||
echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID"
|
||||
else
|
||||
STOPPED=1
|
||||
fi
|
||||
fi
|
||||
rm -f "\$PF"
|
||||
done
|
||||
if [ "\$STOPPED" -eq 0 ]; then
|
||||
echo "Nothing was running for this deploy ($TOP)."
|
||||
fi
|
||||
EOF
|
||||
chmod +x "$TOP/scripts/stop-all.sh"
|
||||
|
||||
# Generate worker start script (component-failover-adoption's --peers path):
|
||||
# same $PEERS_STR the controllers themselves were just given, so a worker
|
||||
# started via this script fails over across the identical candidate list.
|
||||
|
|
@ -623,22 +713,44 @@ else
|
|||
echo "Controller cluster healthy (primary elected)."
|
||||
fi
|
||||
|
||||
# wait_for_pidfile: poll up to 5s for a just-launched background process to
|
||||
# reach its own "echo \$\$ > \$PIDFILE" line. Without this, start-all.sh
|
||||
# could return "done" before a pidfile exists at all -- and cleanup.sh/
|
||||
# stop-all.sh's own pidfile-based stop logic (see those scripts' own
|
||||
# comments) would then find nothing to kill AND still wipe run/*.pid on its
|
||||
# final pass, orphaning the process it never got a chance to see. This is
|
||||
# the same failure class (a live process with no pidfile pointing at it)
|
||||
# that caused a real HA failover storm live, 2026-09-15 -- just via a
|
||||
# different path (a race here, versus a hand-typed kill typo there).
|
||||
wait_for_pidfile() {
|
||||
local pf="\$1" label="\$2" n
|
||||
for n in \$(seq 1 25); do
|
||||
[ -f "\$pf" ] && return 0
|
||||
sleep 0.2
|
||||
done
|
||||
echo "Warning: \$label did not write a pidfile at \$pf within 5s -- it may have failed to start, or started slower than expected. Check $TOP/logs/ for details before assuming it's running."
|
||||
return 1
|
||||
}
|
||||
|
||||
if [ -x "$TOP/scripts/start-worker.sh" ]; then
|
||||
echo "=== start-all.sh: starting worker ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-worker.sh" worker-1 -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/worker-1.pid" "worker-1"
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-scheduler.sh" ]; then
|
||||
echo "=== start-all.sh: starting scheduler (-vvv) ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-scheduler.sh" -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/scheduler.pid" "scheduler"
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-apiserver.sh" ]; then
|
||||
echo "=== start-all.sh: starting apiserver ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-apiserver.sh" </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
wait_for_pidfile "$TOP/run/apiserver.pid" "apiserver"
|
||||
fi
|
||||
|
||||
echo "=== start-all.sh: done -- cluster left running ==="
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue