lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(xce-deploy-ha): stop a stale deploy's processes surviving a hand-typed cleanup mistake

Browse source 
A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.

Two structural gaps let that one typo cascade this badly:

1. deploy-ha.sh had no pre-flight check that target ports were free --
   it would happily start a second Raft group on ports an existing one
   already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
   (reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
   pidfiles unconditionally as part of state cleanup, with no check
   that the process each one pointed to was actually dead first --
   silently erasing the only signal that guard depends on.

Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:

- Added a port pre-flight check before any deploy state is created:
  refuses with the exact blocking PID/process if a target port is
  already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
  removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
  canonical, typo-proof way to stop everything a specific deploy
  started, since it reads that deploy's own tracked PIDs instead of
  matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
  didn't wait for worker/scheduler/apiserver to write their pidfiles
  before returning "done", so cleanup.sh/stop-all.sh run immediately
  after could orphan those two specifically -- the same zombie-process
  class this whole fix targets, via a narrower window instead of an
  operator typo. Added a short pidfile-appearance poll to close it.

SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.

bug-1816.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 14:04:16 +00:00
parent dfabb67447
commit 7f3237aee1
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
2 changed files with 177 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

78
.claude/skills/xce-deploy-ha/SKILL.md
View file

@ -115,6 +115,20 @@ to the intended interpreter).
## Step 2 — Stop any running instances
**If you know the `$TOP` of a previous deploy, prefer its own generated
`stop-all.sh`** — it reads that deploy's own `$TOP/run/*.pid` files (exactly
what it started, nothing else) instead of matching process names/paths by
hand, so there's no string to typo:
```bash
"$BASH_BIN" "$TOP/scripts/stop-all.sh" # only if $TOP already exists from a prior deploy
```
Otherwise (first deploy in this session, or you don't know/remember the
prior `$TOP`), fall back to the blanket by-process-name kill — this is what
actually needs to run before a **fresh** deploy to a `$TOP` you've never
used before, or when a prior deploy's own scripts are gone/inaccessible:
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
@ -122,6 +136,22 @@ kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root — plain kill can't touch it
```
**Don't skip this and rely on Step 3 alone.** `deploy-ha.sh` refuses to
start a *second* cluster on top of ports an existing one already holds
(see Step 3's port pre-flight check below) — but that only catches a
**redeploy attempt**. If you instead just run `start-all.sh` (or `start.sh`)
again without redeploying, and an old, no-longer-pidfile-tracked process is
still alive on those ports from an earlier session, you can still get two
clusters cross-talking. This is exactly what happened live, 2026-09-15: an
operator's own hand-typed `ps axwww | grep xce-ha-deploy | ... | xargs kill`
had a typo (`xce-ha-deploy` vs. the real `xce-deploy-ha` in the workdir
path) and silently killed nothing, leaving a full 3-node cluster running
that a fresh deploy to the same ports then collided with — producing a
genuine HA failover storm (mixed vote traffic between the two unrelated
clusters) that looked like a code bug but wasn't. Prefer a generated
`stop-all.sh` over any hand-typed process-matching command whenever one is
available, precisely because it can't have that class of typo.
## Step 3 — Deploy cluster
Join peers as a comma-separated string and call the deploy script — always
@ -146,7 +176,8 @@ a rebuild, nothing is ever reused across deploys:
real security property since TLS verification is `SSL_VERIFY_NONE`
everywhere in this codebase). Errors if `openssl` isn't on `PATH`.
- The bash baked into the shebang of every script `deploy-ha.sh` generates
(`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is
(`start.sh`, `cleanup.sh`, `stop-all.sh`, `start-worker.sh`,
`start-scheduler.sh`) is
whatever `command -v bash` resolves to *inside deploy-ha.sh's own
invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that
interpreter under `nix develop`. This skill always still invokes each
@ -160,7 +191,7 @@ This:
- Generates `xce.crt`/`xce.key` directly into `$TOP/certs/`
- Places `cluster.bin.<tag>` + `conf/current` in `controller/ctrl-1/conf/` only
(standbys receive config from primary via `OPCode_replConfigPush`)
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`), `cleanup.sh`, and `stop-all.sh` (stop everything for this deploy, no state wipe — reads `$TOP/run/*.pid`, the canonical alternative to hand-typing a process-name kill)
- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh`
**if** the relevant binary exists alongside `$BINARY` (same build dir) —
soft-skipped with a warning, not a hard error, if missing (so
@ -405,38 +436,55 @@ RAFT[p0] term=2 event=became_leader
Stop all instances, clean up state, and restart without redeploying:
```bash
# 1. Stop running controllers (and worker/scheduler, if you started them)
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root
# 2. Run the generated cleanup script
# 1. Run the generated cleanup script -- this now ALSO stops every live
# process it's about to orphan (reads $TOP/run/*.pid) before wiping
# state, precisely so a still-running component never survives with its
# pidfile gone out from under it (see below for why that matters).
# Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log, run/*.pid
"$BASH_BIN" "$TOP/scripts/cleanup.sh"
# 3. Restart all nodes
# 2. Restart all nodes
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
# 4. Restart worker/scheduler if you had them running (Step 4b) — the
# 3. Restart worker/scheduler if you had them running (Step 4b) — the
# generated scripts still carry the same --peers, nothing to redo there
"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &
```
To only stop everything (no state wipe, e.g. before redeploying fresh peers
or a new binary), run `"$BASH_BIN" "$TOP/scripts/stop-all.sh"` instead —
same pidfile-based stop logic as `cleanup.sh`, but leaves all state in place.
**What cleanup.sh (and stop-all.sh) stop first, before touching any state:**
- Every PID recorded in `$TOP/run/*.pid` that's still alive — `ctrl-{1..N}`,
`worker-1`, `scheduler`, `apiserver`, whichever were actually started.
**Why this matters:** the pidfile is the *only* thing `start.sh`/
`start-all.sh`'s own "already running" guard checks before starting a new
process on the same ports. Deleting a pidfile without first confirming its
process is dead (what the OLD `cleanup.sh` did) silently defeats that
guard on the next deploy/restart, letting a stale process survive
undetected and collide with a freshly-started one on the same ports. This
is exactly what caused a real HA failover storm live, 2026-09-15 (see
Step 2's own note above) — the underlying gap this fixes, independent of
the hand-typed-kill-command mistake that triggered it that day.
- The real (root-run) worker may not stop from a plain `kill -9` if you're
not running as root yourself — `cleanup.sh`/`stop-all.sh` will print a
warning naming the PID; follow up with `sudo kill -9 <pid>`.
**What cleanup.sh preserves:**
- `$TOP/certs/` — shared TLS cert and key
- `controller/ctrl-1/conf/cluster.bin.*` + `conf/current` — primary config (standbys re-receive via push)
**What cleanup.sh removes:**
**What cleanup.sh removes (after stopping the processes above):**
- All `controller/ctrl-{1..N}/data/` dirs (WAL, snapshots, `ha-term.bin`) — forces fresh election next start
- All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush`
- All `logs/*.log` — clean log slate
- All `logs/*.log` and `run/*.pid` — clean log/pid slate
**Not touched by cleanup.sh, if you ran the real worker:** `/var/run/xcecd.db`
and `/var/run/.xcecd.sock` live outside `$TOP` entirely (hardcoded, not
@ -474,3 +522,5 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template |
| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first |
| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists |
| `deploy-ha.sh` refuses with `Error: port <N> is already in use by a running process` (before creating any state) | A different, still-running deployment (this `$TOP` or a different one entirely) already holds one of the ports you're deploying to — the pre-flight check (Step 3) catches this before it can cause cross-talk between two clusters | The error prints the PID/process holding the port. If it's a known prior deploy, run its own `$TOP/scripts/stop-all.sh`; otherwise use Step 2's blanket kill-by-name, then re-run the deploy |
| Controller HA is inexplicably failing over (repeated PreVote elections, `primary_ctrl_fd RDHUP`) despite low system load, and log files show more than one PID for the same `ctrl-N` role | Two separate deployments ended up bound to the same ports (see the port-conflict row above) — their vote traffic genuinely cross-talks, producing symptoms that look like a real HA bug but aren't | `ps axwww \| grep -E "xce_controller\|xce_scheduler\|xcecd"` — if more than one process exists per role/port, kill the stale set (or everything, then redeploy clean); double-check any hand-typed kill/grep pattern actually matches these processes' real command lines before trusting it did anything |