A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.
Two structural gaps let that one typo cascade this badly:
1. deploy-ha.sh had no pre-flight check that target ports were free --
it would happily start a second Raft group on ports an existing one
already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
(reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
pidfiles unconditionally as part of state cleanup, with no check
that the process each one pointed to was actually dead first --
silently erasing the only signal that guard depends on.
Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:
- Added a port pre-flight check before any deploy state is created:
refuses with the exact blocking PID/process if a target port is
already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
canonical, typo-proof way to stop everything a specific deploy
started, since it reads that deploy's own tracked PIDs instead of
matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
didn't wait for worker/scheduler/apiserver to write their pidfiles
before returning "done", so cleanup.sh/stop-all.sh run immediately
after could orphan those two specifically -- the same zombie-process
class this whole fix targets, via a narrower window instead of an
operator typo. Added a short pidfile-appearance poll to close it.
SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.
bug-1816.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Extend SKILL.md's existing worker/scheduler/xceadmin soft-skip
documentation to cover harness/xce_test: a Prerequisites bullet and a
Troubleshooting row, matching the existing style exactly.
Confirmed documentation-only: deploy-ha.sh (653 lines, read in full)
has zero reference to xce_test/harness -- it only builds/wraps
worker/scheduler/apiserver/xceadmin, never touches xce_test, which is
a manually-invoked tool rather than something deploy-ha.sh generates
a wrapper for. No script change needed or made.
Task 3.3 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds auto-detection of real vs virtual worker based on systemd
availability (/run/systemd/system), plus an XCE_REAL_WORKER override.
Real xcecd requires root (systemd D-Bus transient units, hardcoded
/var/run paths) and is used opportunistically now that passwordless sudo
is available in the sandbox; defaults to xcecd_virtual when systemd/D-Bus
isn't available.
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh no longer takes a <data-dir> argument or --build-config
flag. Every deploy now generates a fresh self-signed TLS cert/key
(openssl, RSA 2048, 10y validity) directly into the deploy dir's
certs/, and builds cluster.bin directly into controller/ctrl-1/conf/
via xceadmin -- no intermediate staging directory, nothing preserved
across runs. run-ha-soak.sh (/xce-test-ha) updated to match: dropped
its own XCE_HA_DATADIR-based precondition entirely. Updated 4 callers
(run-ha-soak.sh, test-heartbeat-tolerance.sh, test-commit-durability.sh,
both xce-deploy-ha/xce-test-ha SKILL.md docs).
Separately: xcecommon.wire.connect() was loading a client-side TLS
cert, diverging from the real reference client -- netio.c's
create_ssl_context() presents no certificate at all when acting as a
client (cert_file/key_file both NULL, bare TLS_client_method()).
Fixed connect() to match, and since nothing else consumed cert_dir
(verify_mode=CERT_NONE throughout, no CA check either), removed it
end-to-end: discovery.follow_redirects(), push.push_config()/
probe_config(), and the --cert-dir CLI flag.
Server-side cert loading (mock TLS test servers) is unaffected --
moved into a shared session-scoped tls_certs fixture (new
tests/conftest.py, openssl-generated) replacing 3 duplicated
REPO_HA_DATA_DIR/skipif blocks. Net result: those tests no longer skip
under a hermetic nix build sandbox (which never had ha/data/) --
verified 165/165 passed, 0 skipped, via both meson compile and a full
nix build --option sandbox false .#xcecore_debug rebuild (previously
128 passed / 37 skipped there).
ha/data/{xce.crt,xce.key} removed from git tracking. Requirements
2.3/2.4 and 8.3/8.4/8.5 marked superseded in requirements.md with
dated correction notes.
deploy-ha.sh's --build-config path (task 4.2) pointed at
src/xceadmin/.venv/bin/xceadmin, broken by today's meson/nix rewiring
(xceadmin no longer builds a venv there). Fixed to search the meson
build dir (xce.build/build.release/build.dbg), a nix-installed
result/bin, then PATH.
/xce-deploy-ha's SKILL.md never mentioned xceadmin or --build-config
(predates task 4.2) — documented both.
/xce-test-ha's run-ha-soak.sh hard-required a pre-existing cluster.bin
under ha/data/, exit 2 if missing, with no xceadmin path at all. Now
always builds cluster.bin fresh from src/configs/*.yaml via xceadmin
before every soak run, same resolution order as deploy-ha.sh.
ha/data/cluster.bin is no longer a checked-in fixture: removed from
git tracking and gitignored. It was stale relative to xceadmin's
output anyway (different hash from a different, older generation
path) -- the xceadmin-built one matches the known-good golden-fixture
hash from admin-config-tool task 5.4.
- xce-build.sh: `controller` component now always builds `controller/xce_test`
alongside `xce_controller`, in every profile — xce-test-ha needs both
binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
subagent never receives a usable task and silently does nothing. Documented
that this skill should instead always be dispatched via the plain Agent
tool with a fully-resolved, self-contained prompt (not `context: fork`,
which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
invoke via subagent" warning with the precise finding — `context: fork`
(Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
SIGHUPs the whole controller process group mid-soak/mid-deploy), but
Agent-tool subagent dispatch is a different mechanism and is safe
(empirically verified: a `setsid nohup ... & disown` process launched from
inside an Agent subagent survives that subagent returning, reparented to
init). Documented both the foreground-blocking and fire-and-forget
subagent patterns for each skill; xce-deploy-ha additionally flagged that
its `start.sh &` should gain `setsid` to match the soak driver's stronger
detach pattern before relying on unattended subagent dispatch.
cleanup.sh is generated by deploy-ha.sh and already does the right thing:
keeps ctrl-1/conf/* and all certs, wipes ctrl-{2..N}/conf, all data/, and logs.
Update skill and CLAUDE.md to use it as the primary clean-restart command
instead of manual rm loops.
- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a
local 3-node XCE HA controller cluster. Covers build prerequisites,
deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace,
health verification checks, clean restart procedure, and a
troubleshooting table for common failure modes.
- src/controller/CLAUDE.md: add HA Cluster Deployment section with
quick-deploy commands, health check one-liners, and reference to the
xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
