diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 004c5c5a..2cdd1be4 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -115,6 +115,20 @@ to the intended interpreter). ## Step 2 — Stop any running instances +**If you know the `$TOP` of a previous deploy, prefer its own generated +`stop-all.sh`** — it reads that deploy's own `$TOP/run/*.pid` files (exactly +what it started, nothing else) instead of matching process names/paths by +hand, so there's no string to typo: + +```bash +"$BASH_BIN" "$TOP/scripts/stop-all.sh" # only if $TOP already exists from a prior deploy +``` + +Otherwise (first deploy in this session, or you don't know/remember the +prior `$TOP`), fall back to the blanket by-process-name kill — this is what +actually needs to run before a **fresh** deploy to a `$TOP` you've never +used before, or when a prior deploy's own scripts are gone/inaccessible: + ```bash kill -9 $(pgrep xce_controller) 2>/dev/null || true kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true @@ -122,6 +136,22 @@ kill -9 $(pgrep xce_scheduler) 2>/dev/null || true sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root — plain kill can't touch it ``` +**Don't skip this and rely on Step 3 alone.** `deploy-ha.sh` refuses to +start a *second* cluster on top of ports an existing one already holds +(see Step 3's port pre-flight check below) — but that only catches a +**redeploy attempt**. If you instead just run `start-all.sh` (or `start.sh`) +again without redeploying, and an old, no-longer-pidfile-tracked process is +still alive on those ports from an earlier session, you can still get two +clusters cross-talking. This is exactly what happened live, 2026-09-15: an +operator's own hand-typed `ps axwww | grep xce-ha-deploy | ... | xargs kill` +had a typo (`xce-ha-deploy` vs. the real `xce-deploy-ha` in the workdir +path) and silently killed nothing, leaving a full 3-node cluster running +that a fresh deploy to the same ports then collided with — producing a +genuine HA failover storm (mixed vote traffic between the two unrelated +clusters) that looked like a code bug but wasn't. Prefer a generated +`stop-all.sh` over any hand-typed process-matching command whenever one is +available, precisely because it can't have that class of typo. + ## Step 3 — Deploy cluster Join peers as a comma-separated string and call the deploy script — always @@ -146,7 +176,8 @@ a rebuild, nothing is ever reused across deploys: real security property since TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase). Errors if `openssl` isn't on `PATH`. - The bash baked into the shebang of every script `deploy-ha.sh` generates - (`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is + (`start.sh`, `cleanup.sh`, `stop-all.sh`, `start-worker.sh`, + `start-scheduler.sh`) is whatever `command -v bash` resolves to *inside deploy-ha.sh's own invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that interpreter under `nix develop`. This skill always still invokes each @@ -160,7 +191,7 @@ This: - Generates `xce.crt`/`xce.key` directly into `$TOP/certs/` - Places `cluster.bin.` + `conf/current` in `controller/ctrl-1/conf/` only (standbys receive config from primary via `OPCode_replConfigPush`) -- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh` +- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`), `cleanup.sh`, and `stop-all.sh` (stop everything for this deploy, no state wipe — reads `$TOP/run/*.pid`, the canonical alternative to hand-typing a process-name kill) - Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh` **if** the relevant binary exists alongside `$BINARY` (same build dir) — soft-skipped with a warning, not a hard error, if missing (so @@ -405,38 +436,55 @@ RAFT[p0] term=2 event=became_leader Stop all instances, clean up state, and restart without redeploying: ```bash -# 1. Stop running controllers (and worker/scheduler, if you started them) -kill -9 $(pgrep xce_controller) 2>/dev/null || true -kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true -kill -9 $(pgrep xce_scheduler) 2>/dev/null || true -sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root - -# 2. Run the generated cleanup script +# 1. Run the generated cleanup script -- this now ALSO stops every live +# process it's about to orphan (reads $TOP/run/*.pid) before wiping +# state, precisely so a still-running component never survives with its +# pidfile gone out from under it (see below for why that matters). # Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/ -# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log +# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log, run/*.pid "$BASH_BIN" "$TOP/scripts/cleanup.sh" -# 3. Restart all nodes +# 2. Restart all nodes "$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv & sleep 0.5 "$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv & sleep 0.5 "$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv & -# 4. Restart worker/scheduler if you had them running (Step 4b) — the +# 3. Restart worker/scheduler if you had them running (Step 4b) — the # generated scripts still carry the same --peers, nothing to redo there "$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv & "$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & ``` +To only stop everything (no state wipe, e.g. before redeploying fresh peers +or a new binary), run `"$BASH_BIN" "$TOP/scripts/stop-all.sh"` instead — +same pidfile-based stop logic as `cleanup.sh`, but leaves all state in place. + +**What cleanup.sh (and stop-all.sh) stop first, before touching any state:** +- Every PID recorded in `$TOP/run/*.pid` that's still alive — `ctrl-{1..N}`, + `worker-1`, `scheduler`, `apiserver`, whichever were actually started. + **Why this matters:** the pidfile is the *only* thing `start.sh`/ + `start-all.sh`'s own "already running" guard checks before starting a new + process on the same ports. Deleting a pidfile without first confirming its + process is dead (what the OLD `cleanup.sh` did) silently defeats that + guard on the next deploy/restart, letting a stale process survive + undetected and collide with a freshly-started one on the same ports. This + is exactly what caused a real HA failover storm live, 2026-09-15 (see + Step 2's own note above) — the underlying gap this fixes, independent of + the hand-typed-kill-command mistake that triggered it that day. +- The real (root-run) worker may not stop from a plain `kill -9` if you're + not running as root yourself — `cleanup.sh`/`stop-all.sh` will print a + warning naming the PID; follow up with `sudo kill -9 `. + **What cleanup.sh preserves:** - `$TOP/certs/` — shared TLS cert and key - `controller/ctrl-1/conf/cluster.bin.*` + `conf/current` — primary config (standbys re-receive via push) -**What cleanup.sh removes:** +**What cleanup.sh removes (after stopping the processes above):** - All `controller/ctrl-{1..N}/data/` dirs (WAL, snapshots, `ha-term.bin`) — forces fresh election next start - All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush` -- All `logs/*.log` — clean log slate +- All `logs/*.log` and `run/*.pid` — clean log/pid slate **Not touched by cleanup.sh, if you ran the real worker:** `/var/run/xcecd.db` and `/var/run/.xcecd.sock` live outside `$TOP` entirely (hardcoded, not @@ -474,3 +522,5 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template | | Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first | | `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists | +| `deploy-ha.sh` refuses with `Error: port is already in use by a running process` (before creating any state) | A different, still-running deployment (this `$TOP` or a different one entirely) already holds one of the ports you're deploying to — the pre-flight check (Step 3) catches this before it can cause cross-talk between two clusters | The error prints the PID/process holding the port. If it's a known prior deploy, run its own `$TOP/scripts/stop-all.sh`; otherwise use Step 2's blanket kill-by-name, then re-run the deploy | +| Controller HA is inexplicably failing over (repeated PreVote elections, `primary_ctrl_fd RDHUP`) despite low system load, and log files show more than one PID for the same `ctrl-N` role | Two separate deployments ended up bound to the same ports (see the port-conflict row above) — their vote traffic genuinely cross-talks, producing symptoms that look like a real HA bug but aren't | `ps axwww \| grep -E "xce_controller\|xce_scheduler\|xcecd"` — if more than one process exists per role/port, kill the stale set (or everything, then redeploy clean); double-check any hand-typed kill/grep pattern actually matches these processes' real command lines before trusting it did anything | diff --git a/src/tools/deploy-ha.sh b/src/tools/deploy-ha.sh index 5550c945..24038047 100755 --- a/src/tools/deploy-ha.sh +++ b/src/tools/deploy-ha.sh @@ -186,6 +186,46 @@ fi IFS=',' read -ra PEER_ARRAY <<< "$PEERS" N=${#PEER_ARRAY[@]} +# Pre-flight: refuse to deploy on top of ports a DIFFERENT still-running +# deployment already holds. Silently starting a second cluster bound to the +# same ports doesn't fail cleanly -- it causes real cross-talk (mixed vote +# traffic, corrupted term/leader state) between two otherwise-unrelated Raft +# groups, which looks exactly like a genuine HA bug. Found live, 2026-09-15: +# two full 3-node clusters ended up sharing ports 9901-9903 after a stale +# deploy's processes were never actually killed (a hand-typed cleanup +# command had a typo), producing a real failover storm from otherwise- +# correct code. start.sh/start-all.sh's own "already running" guard further +# below only catches THIS $TOP's own pidfiles -- it can't see a different, +# unrelated deploy (or this same $TOP after cleanup.sh wiped its pidfiles +# out from under a still-alive process, the exact gap that let this +# happen -- see cleanup.sh's own fix below), so this check is independent, +# at the network level, before any state for this deploy is touched. +if command -v ss >/dev/null 2>&1; then + PORT_CONFLICT=0 + for peer in "${PEER_ARRAY[@]}"; do + peer_port="${peer##*:}" + if ss -tln 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p {f=1} END{exit !f}'; then + echo "Error: port $peer_port is already in use by a running process:" + ss -tlnp 2>/dev/null | awk -v p=":$peer_port\$" '$4 ~ p' || true + PORT_CONFLICT=1 + fi + done + if [ "$PORT_CONFLICT" -eq 1 ]; then + echo "" + echo "Refusing to deploy on top of an already-occupied port -- this usually" + echo "means an earlier deployment (possibly to a DIFFERENT top-dir) was never" + echo "fully stopped. If it has its own \$TOP/scripts/stop-all.sh, run that;" + echo "otherwise stop it by process name, then re-run this deploy:" + echo " kill -9 \$(pgrep xce_controller) 2>/dev/null || true" + echo " kill -9 \$(pgrep -f xcecd_virtual) 2>/dev/null || true" + echo " kill -9 \$(pgrep xce_scheduler) 2>/dev/null || true" + echo " sudo kill -9 \$(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root" + exit 1 + fi +else + echo "Warning: 'ss' not found -- skipping the port-conflict pre-flight check. If this deploy collides with an already-running one, expect confusing HA behavior rather than a clean failure." +fi + # Create directory structure mkdir -p "$TOP/certs" mkdir -p "$TOP/logs" @@ -269,10 +309,29 @@ exec $BINARY --myname \$NAME --peers "$PEERS_STR" --workdir $TOP/controller --ce EOF chmod +x "$TOP/scripts/start.sh" -# Generate cleanup script +# Generate cleanup script. Stops anything still running for this deploy +# BEFORE wiping its pidfile -- removing a pidfile while its process is still +# alive would silently defeat start.sh/start-all.sh's own "already running" +# guard on the next deploy, letting a stale controller/worker/scheduler +# survive undetected and collide with a freshly-started one on the same +# ports (this is exactly how a real HA failover storm happened live, +# 2026-09-15 -- see the port pre-flight check above for the full story). cat > "$TOP/scripts/cleanup.sh" << EOF #!$GEN_BASH set -e +for PF in $TOP/run/*.pid; do + [ -f "\$PF" ] || continue + PID=\$(cat "\$PF" 2>/dev/null || true) + NAME=\$(basename "\$PF" .pid) + if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then + echo "cleanup.sh: stopping \$NAME (PID \$PID)" + kill -9 "\$PID" 2>/dev/null || true + sleep 0.2 + if kill -0 "\$PID" 2>/dev/null; then + echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID" + fi + fi +done rm -f $TOP/logs/*.log rm -f $TOP/run/*.pid for i in \$(seq 1 $N); do @@ -284,6 +343,37 @@ done EOF chmod +x "$TOP/scripts/cleanup.sh" +# Generate stop-all script -- the canonical, typo-proof way to stop every +# component THIS deploy started. Reads $TOP/run/*.pid (scoped exactly to +# what this deploy is tracking) instead of matching process names/cmdlines +# by hand, so there's no path/name string to get wrong. Unlike cleanup.sh, +# leaves all state (data/conf/logs) in place -- just stops processes. +cat > "$TOP/scripts/stop-all.sh" << EOF +#!$GEN_BASH +set -e +STOPPED=0 +for PF in $TOP/run/*.pid; do + [ -f "\$PF" ] || continue + PID=\$(cat "\$PF" 2>/dev/null || true) + NAME=\$(basename "\$PF" .pid) + if [ -n "\$PID" ] && kill -0 "\$PID" 2>/dev/null; then + echo "Stopping \$NAME (PID \$PID)" + kill -9 "\$PID" 2>/dev/null || true + sleep 0.2 + if kill -0 "\$PID" 2>/dev/null; then + echo "Warning: \$NAME (PID \$PID) still alive after kill -9 -- likely the real worker running as root; try: sudo kill -9 \$PID" + else + STOPPED=1 + fi + fi + rm -f "\$PF" +done +if [ "\$STOPPED" -eq 0 ]; then + echo "Nothing was running for this deploy ($TOP)." +fi +EOF +chmod +x "$TOP/scripts/stop-all.sh" + # Generate worker start script (component-failover-adoption's --peers path): # same $PEERS_STR the controllers themselves were just given, so a worker # started via this script fails over across the identical candidate list. @@ -623,22 +713,44 @@ else echo "Controller cluster healthy (primary elected)." fi +# wait_for_pidfile: poll up to 5s for a just-launched background process to +# reach its own "echo \$\$ > \$PIDFILE" line. Without this, start-all.sh +# could return "done" before a pidfile exists at all -- and cleanup.sh/ +# stop-all.sh's own pidfile-based stop logic (see those scripts' own +# comments) would then find nothing to kill AND still wipe run/*.pid on its +# final pass, orphaning the process it never got a chance to see. This is +# the same failure class (a live process with no pidfile pointing at it) +# that caused a real HA failover storm live, 2026-09-15 -- just via a +# different path (a race here, versus a hand-typed kill typo there). +wait_for_pidfile() { + local pf="\$1" label="\$2" n + for n in \$(seq 1 25); do + [ -f "\$pf" ] && return 0 + sleep 0.2 + done + echo "Warning: \$label did not write a pidfile at \$pf within 5s -- it may have failed to start, or started slower than expected. Check $TOP/logs/ for details before assuming it's running." + return 1 +} + if [ -x "$TOP/scripts/start-worker.sh" ]; then echo "=== start-all.sh: starting worker ===" setsid nohup "$GEN_BASH" "$TOP/scripts/start-worker.sh" worker-1 -vvv /dev/null 2>&1 & disown + wait_for_pidfile "$TOP/run/worker-1.pid" "worker-1" fi if [ -x "$TOP/scripts/start-scheduler.sh" ]; then echo "=== start-all.sh: starting scheduler (-vvv) ===" setsid nohup "$GEN_BASH" "$TOP/scripts/start-scheduler.sh" -vvv /dev/null 2>&1 & disown + wait_for_pidfile "$TOP/run/scheduler.pid" "scheduler" fi if [ -x "$TOP/scripts/start-apiserver.sh" ]; then echo "=== start-all.sh: starting apiserver ===" setsid nohup "$GEN_BASH" "$TOP/scripts/start-apiserver.sh" /dev/null 2>&1 & disown + wait_for_pidfile "$TOP/run/apiserver.pid" "apiserver" fi echo "=== start-all.sh: done -- cluster left running ==="