docs: sync skills and steering docs with bug-1562..1566 findings

- .claude/skills/xce-build/SKILL.md: --incremental flag, xceadmin wheel building, corrected the standalone-install guidance (the earlier --system-site-packages venv approach was wrong -- nix devShell packages are $PYTHONPATH entries, not merged into site-packages). - .claude/skills/xce-deploy-ha/SKILL.md: $BASH_BIN resolution, worker/scheduler startup, xceadmin usage against a deployment, troubleshooting rows for the newly-fixed bugs. - .kiro/steering/ha-async-io.md: bug-1562's standby liveness watchdog re-arm invariant. - .kiro/steering/persistence.md: bug-1563 (fsync backoff) and bug-1564 (persist_flush durable-advance callback). - .kiro/steering/build-deployment.md: xceadmin wheel docs, bug-1565's --incremental flag, deploy dir tree updates. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 10:21:25 +00:00 · 2026-08-17 10:21:25 +00:00 · 9f539583d3
commit 9f539583d3
parent 0c8e417bda
5 changed files with 225 additions and 33 deletions
--- a/.claude/skills/xce-build/SKILL.md
+++ b/.claude/skills/xce-build/SKILL.md
@ -1,33 +1,90 @@
 ---
 name: xce-build
-description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
+description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
 disable-model-invocation: false
 allowed-tools: Bash(*)
 ---

 ## build xce source code

-Args: `[component] [profile|build_dir] [build_dir]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.
+Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. `--incremental` can appear anywhere in the words (it's stripped out before the rest are parsed positionally — see below). Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.

 - **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt.
 - **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`, `xceadmin`
 - **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir).
 - Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked).
 - **`controller` always builds `xce_test` too** (`meson compile … xce_controller controller_unit_test controller/xce_test`), in every profile — not just `asan`. `xce_test` is the driver `/xce-test-ha` needs; you never have to build it as a separate step after `/xce-build controller`, and the two binaries in a build dir are always the same commit/profile in lockstep.
- **`xceadmin` is Python, not meson, and ignores the ASAN/debug/release sanitizer semantics entirely** — it is dispatched by a plain shell case branch in `xce-build.sh`, deliberately not wired into `src/meson.build`/`meson.options` (see `.kiro/specs/admin-config-tool/design.md`). The `profile` arg still matters for it, but only to pick the pip install mode: default/`asan`/`debug` → `pip install -e .` (editable, for dev); `release` → `pip install .` (pinned, non-editable, for production). Every invocation — either install mode — syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, then runs `pytest` against whichever install just happened. Only the `release` run actually exercises the packaged (non-editable) schema resolution path — an editable-only test run can't catch a `package_data` regression there. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`).
+- **`xceadmin` is Python, meson-driven (`src/xceadmin/meson.build`'s `custom_target` wraps `meson-build.sh`), and ignores the ASAN/debug/release sanitizer semantics entirely** — `xce-build.sh`'s `xceadmin` case just runs `meson compile -C "$BUILD_DIR" xceadmin`, the same target `nix build .#xcecore` and every other component use; the `profile` arg is accepted but has no effect on it (every invocation does the identical pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release` — unlike the old editable-install-by-default behavior, this always exercises real packaged schema resolution, not just a `release` run). For fast iterative dev without a full meson reconfigure, `xceadmin` can still be built standalone: `cd src/xceadmin && pip install -e .` (see `meson-build.sh` for the exact out-of-tree steps the meson target itself runs). Every invocation syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, installs pinned into a fresh build-dir venv, runs `pytest` against that install, and — only after tests pass — builds an installable wheel from that same schema-synced tree into `$BUILD_DIR/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (`pip wheel … --no-deps`, so it's xceadmin's own wheel only, not a vendored bundle of `click`/`pycapnp`/etc). `xce-build.sh` prints the wheel's path and a ready-to-run `pip install "<path>"` line after the build. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`).
+- **`--incremental` (bug-1565 escape hatch, opt-in, NOT the default)**: every invocation normally does `rm -rf "$BUILD_DIR"` + a full fresh `meson setup`, unconditionally — correct in isolation, but since the actual compile step afterward only rebuilds the ONE requested component, a plain single-component build silently deletes any OTHER already-built binary sharing that dir (e.g. `/xce-build xceadmin` after `/xce-build controller` wipes `xce_controller`). Pass `--incremental` to reconfigure the existing dir in place instead (`meson setup --reconfigure`), preserving whatever else was already built there — use it specifically when you want a sibling component alongside an already-built one, e.g. `/xce-build controller` then `/xce-build xceadmin --incremental` to end up with both in the same dir at once (needed for `/xce-deploy-ha`, which wants `xce_controller` + `xcecd_virtual` + `xce_scheduler` + the xceadmin wheel all together). Deliberately not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json — normally papered over as a loud, retry-fixable failure on a clean build; skipping the wipe could instead let the same gap cause silent staleness). If anything looks wrong after an incremental build, rebuild without the flag.

-Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (editable install + tests), `/xce-build xceadmin release` (pinned install + tests).
+Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (build + test + wheel, any profile arg is accepted and ignored), `/xce-build xceadmin --incremental` (same, but keeps other components already built in the same dir).

-Run (substituting the parsed component/profile/build_dir; omit trailing empty ones):
+Run (substituting the parsed component/profile/build_dir; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested):

 ```bash
-nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir>
+nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> [--incremental]
 ```

-e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`.
+e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. For `/xce-build xceadmin --incremental`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --incremental`.

 ## Always dispatch the build via the Agent tool

 Do not run the `nix develop … xce-build.sh` command inline in the current session's Bash tool. Instead, on every invocation of this skill, launch a subagent with the Agent tool, giving it a fully self-contained prompt: repo working directory, the exact resolved command (component/profile/build_dir already substituted — do not make the subagent re-parse ARGUMENTS itself), what the command does, and what to report back (build success/failure, compiler errors, the meson test Ok/Fail summary line, and confirmation the expected binaries were freshly rebuilt with fresh mtimes). Wait for it (do not background it — the build is synchronous and finishes in well under this session's turn budget) and relay its report.

 This is deliberately a plain Agent-tool dispatch, not `context: fork` — that frontmatter field reproducibly failed to thread a non-empty ARGUMENTS payload into the forked subagent (see above), whereas an explicit, self-contained Agent prompt has no such dependency and is proven to work.
+
+## Installing the xceadmin wheel elsewhere
+
+`/xce-build xceadmin` always leaves a fresh, tested wheel at
+`<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (e.g.
+`xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl`). It's a normal
+wheel — `pip install` it anywhere a compatible Python is available, no nix
+or meson needed at install time:
+
+```bash
+pip install /path/to/xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl
+```
+
+The wheel bundles its own `schema_files/` (the same `controller.capnp`,
+`c.capnp`, and 7 JSON schemas synced in above) so `xceadmin config build`
+works fully standalone against any `src/configs/`-shaped directory — it
+does not need to run from inside this repo or the nix dev shell. Its 4 real
+runtime imports (`click`, `pycapnp`, `PyYAML`, `jsonschema` —
+`pyproject.toml`'s `dependencies`, verified against every actual `import`
+in `src/xceadmin`/`src/xcecommon`) are declared, so a plain `pip install`
+resolves them from PyPI on its own; the one caveat is `pycapnp`, whose
+prebuilt PyPI wheels are usually fine outside the nix shell but aren't the
+same guaranteed-consistent link against the capnproto C++ runtime the nix
+shell's own `python3Packages.pycapnp` gets (see `pyproject.toml`'s
+`dependencies` comment).
+
+**For that guarantee, do NOT use a `--system-site-packages` venv** — tried
+and confirmed broken: this nix devShell injects `click`/`pycapnp`/`PyYAML`/
+`jsonschema` via `$PYTHONPATH`, not into the base python3 interpreter's own
+site-packages (it's the bare interpreter derivation; each package is a
+separate `/nix/store/...` path added to `$PYTHONPATH` by the shell hook —
+confirmed via `nix develop … --command sh -c 'echo $PYTHONPATH'`). A
+`--system-site-packages` venv only inherits the base interpreter's own
+(empty) site-packages, so it silently can't see any of them —
+`ModuleNotFoundError: No module named 'click'` the moment you actually run
+it, even though the venv's own `pip install` step succeeds. Instead extract
+the wheel directly and bake the resolved `$PYTHONPATH` into a wrapper at
+install time (no venv/pip needed at all) — this is exactly what
+`/xce-deploy-ha`'s `deploy-ha.sh` does for `scripts/xceadmin`, and it's
+proven to work with zero nix/network dependency at runtime:
+```bash
+mkdir -p /path/to/xceadmin-site
+python3 -m zipfile -e /path/to/xceadmin-0.1.0-py3-none-any.whl /path/to/xceadmin-site
+nix develop ./src --command bash -c '
+  cat > /path/to/xceadmin << EOF
+#!'"$(nix develop ./src --command which bash)"'
+export PYTHONPATH="/path/to/xceadmin-site:$PYTHONPATH"
+exec '"$(nix develop ./src --command which python3)"' -m xceadmin.cli "\$@"
+EOF
+  chmod +x /path/to/xceadmin
+'
+```
+(`$PYTHONPATH` must be expanded **inside** the `nix develop --command` shell,
+not the outer one, so the resolved nix store paths get baked into the
+wrapper literally — see `deploy-ha.sh`'s own xceadmin-install block for the
+exact working pattern.)
--- a/.claude/skills/xce-deploy-ha/SKILL.md
+++ b/.claude/skills/xce-deploy-ha/SKILL.md
@ -53,7 +53,7 @@ Arguments are space-separated `host:port` pairs, one per node, in priority order

 If no arguments are given, default to `localhost:9991 localhost:9992 localhost:9993`.

-## Step 1 — Resolve paths
+## Step 1 — Resolve paths and the deploy environment's bash

 ```bash
 BINARY="src/xce.build/controller/xce_controller"
@ -67,19 +67,45 @@ ls -lh "$BINARY"

 If missing, tell the user to run `/xce-build controller` first.

+Also resolve the bash that actually lives in the deploy environment (the nix
+dev shell) instead of assuming a bare `bash` on the invoker's own `PATH`
+means the same thing:
+
+```bash
+BASH_BIN=$(nix develop ./src --command which bash)
+```
+
+**Why not just type `bash`:** `deploy-ha.sh` and every script it generates
+are bash (arrays, `+=`, `<<<`) even though `deploy-ha.sh` itself carries a
+`#!/bin/sh` shebang in the repo today — a real incident, not a hypothetical:
+a working-tree edit once flipped that shebang while the script still used
+`${BASH_SOURCE[0]}` and bash-only array syntax, which only kept working here
+because this environment's `/bin/sh` happens to be symlinked to bash. On a
+system where `/bin/sh` is dash (common on Debian/Ubuntu), that same file
+would fail immediately. Resolving `$BASH_BIN` once via `which bash` *inside*
+the nix dev shell — the same shell that provides `openssl` and the correct
+runtime libs for the binaries — and using it explicitly for every script
+invocation below sidesteps trusting any file's shebang line at all, in
+either direction (a stale one, or an ambient `bash` that might not resolve
+to the intended interpreter).
+
 ## Step 2 — Stop any running instances

 ```bash
 kill -9 $(pgrep xce_controller) 2>/dev/null || true
+kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
+kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
 ```

 ## Step 3 — Deploy cluster

-Join peers as a comma-separated string and call the deploy script:
+Join peers as a comma-separated string and call the deploy script — always
+via `nix develop` (needed for `openssl`) and the resolved `$BASH_BIN` (never
+a bare `bash`, never `./deploy-ha.sh` relying on its own shebang):

 ```bash
 PEERS="localhost:9991,localhost:9992,localhost:9993"  # from args
-bash src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS"
+nix develop ./src --command "$BASH_BIN" src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS"
 ```

 Every invocation is a fresh, self-contained deploy — no flag needed to force
@ -94,6 +120,15 @@ a rebuild, nothing is ever reused across deploys:
  RSA 2048, 10y validity — long-lived since it's per-deploy anyway; loses no
  real security property since TLS verification is `SSL_VERIFY_NONE`
  everywhere in this codebase). Errors if `openssl` isn't on `PATH`.
+- The bash baked into the shebang of every script `deploy-ha.sh` generates
+  (`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is
+  whatever `command -v bash` resolves to *inside deploy-ha.sh's own
+  invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that
+  interpreter under `nix develop`. This skill always still invokes each
+  generated script explicitly as `"$BASH_BIN" script.sh …` below (the
+  shebang is belt-and-suspenders for a human running one directly later, not
+  load-bearing for this skill or `/xce-test-ha`, which calls the same
+  `deploy-ha.sh`).

 This:
 - Creates `$TOP/certs/`, `$TOP/controller/ctrl-{1..N}/conf`, `$TOP/controller/ctrl-{1..N}/data`, `$TOP/logs/`
@ -101,22 +136,87 @@ This:
 - Places `cluster.bin.<tag>` + `conf/current` in `controller/ctrl-1/conf/` only
  (standbys receive config from primary via `OPCode_replConfigPush`)
 - Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
+- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh`
+  **if** `xcecd_virtual`/`xce_scheduler` exist alongside `$BINARY` (same
+  build dir) — soft-skipped with a warning, not a hard error, if either is
+  missing (so `/xce-test-ha`, which never builds worker/scheduler and calls
+  this same script, is unaffected). See Step 4b.
+- Also extracts a standalone xceadmin into `$TOP/xceadmin-site` and generates
+  `$TOP/scripts/xceadmin` **if** a wheel exists at
+  `<same build dir>/xceadmin/dist/*.whl` — same soft-skip-with-warning
+  convention. See Step 4c.

 ## Step 4 — Start all nodes

-Start each controller in the background. Use `-vvv` for raft trace logs:
+Start each controller in the background, via `$BASH_BIN`. Use `-vvv` for raft trace logs:

 ```bash
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
 sleep 0.5
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
 sleep 0.5
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
 ```

 The startup stagger (0.5s between nodes) allows higher-priority nodes to
 begin their election before lower-priority ones start.

+## Step 4b — Start worker and scheduler (optional)
+
+Once at least one controller is up (no need to wait for a full election —
+both fail over across the whole candidate list on their own), start a worker
+and/or the scheduler with the **identical `--peers` configuration** the
+controllers were deployed with — `deploy-ha.sh` bakes the exact same
+`ctrl-1=host:port,ctrl-2=host:port,...` string into both generated scripts,
+so there is nothing to keep in sync by hand:
+
+```bash
+"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &   # name defaults to worker-1
+"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &            # scheduler is a cluster singleton, no name
+```
+
+- **`start-worker.sh`** runs `xcecd_virtual`, not production `xcecd` — it
+  skips the root-only sqlite/db-recovery and domain-socket subsystems, so it
+  runs as an ordinary user with no `sudo` (this session/skill cannot `sudo`
+  anyway). Logs land at `$TOP/logs/xcecd-<name>.log`. Start more than one by
+  passing a different name: `start-worker.sh worker-2`. To point at a real
+  `xcecd` instead (needs root — run it yourself outside this skill), set
+  `XCE_WORKER_BIN=/path/to/xcecd` before invoking the script.
+- **`start-scheduler.sh`** runs `xce_scheduler`; its identity is fixed
+  internally to `xce_scheduler` (no name argument). Logs to
+  `$TOP/logs/scheduler.log`.
+- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`.
+- Verify the scheduler joined: `grep "sched_init: joined cluster" "$TOP/logs/scheduler.log"`.
+
+## Step 4c — Use xceadmin against this deployment (optional)
+
+`deploy-ha.sh` also installs a **standalone** `xceadmin` at `$TOP/xceadmin-site`
+(extracted from the `/xce-build xceadmin`-produced wheel — soft-skipped with
+a warning if no wheel exists in the same build dir as `$BINARY`, same as
+`start-worker.sh`/`start-scheduler.sh`) and generates a wrapper at
+`$TOP/scripts/xceadmin`:
+
+```bash
+"$BASH_BIN" "$TOP/scripts/xceadmin" config push "$TOP/controller/ctrl-1/conf/cluster.bin.<tag>"
+"$BASH_BIN" "$TOP/scripts/xceadmin" scheduler record status
+```
+
+- **Genuinely standalone**: works with zero nix/network dependency at
+  runtime (`$PYTHONPATH` is baked into the wrapper at deploy time — see
+  `deploy-ha.sh`'s own comment for why a `--system-site-packages` venv does
+  NOT work for this, a real dead end already ruled out). No need to prefix
+  with `"$BASH_BIN"`/`nix develop` to run it, only to *generate* it in Step 3.
+- **Defaults `--controller` to this deployment's own first `--peers`
+  candidate**, not xceadmin's generic bare fallback (`localhost:9991`,
+  unrelated to any real deployment's actual ports). The wrapper exports
+  `XCEADMIN_CONTROLLER="<first --peers candidate>"`, which every
+  `--controller`-accepting xceadmin subcommand (`config push`, `scheduler
+  record start`/`stop`/`status`) reads via `envvar=` fallback — an explicit
+  `--controller host:port` on the command line still overrides it. Before
+  this (bug-1566), a bare `scripts/xceadmin scheduler record status` with no
+  `--controller` silently tried port 9991 and failed with a connection-
+  refused that had nothing to do with the actual deployment.
+
 ## Step 5 — Verify cluster health

 Wait ~15 seconds for election and config replication, then check:
@ -154,20 +254,27 @@ RAFT[p0] term=2 event=became_leader
 Stop all instances, clean up state, and restart without redeploying:

 ```bash
-# 1. Stop running controllers
+# 1. Stop running controllers (and worker/scheduler, if you started them)
 kill -9 $(pgrep xce_controller) 2>/dev/null || true
+kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
+kill -9 $(pgrep xce_scheduler) 2>/dev/null || true

 # 2. Run the generated cleanup script
 #    Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
 #    Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
-bash "$TOP/scripts/cleanup.sh"
+"$BASH_BIN" "$TOP/scripts/cleanup.sh"

 # 3. Restart all nodes
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
 sleep 0.5
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
 sleep 0.5
-VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
+"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
+
+# 4. Restart worker/scheduler if you had them running (Step 4b) — the
+#    generated scripts still carry the same --peers, nothing to redo there
+"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &
+"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &
 ```

 **What cleanup.sh preserves:**
@ -190,3 +297,9 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
 | `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary |
 | `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports |
 | `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above |
+| `SSL_do_handshake: Broken pipe` flooding from a worker/scheduler | A typo'd hostname in the `--peers` you passed (e.g. `local` instead of `localhost`) — the connect reaches SSL before failing, so it doesn't error out cleanly | Diff the failing `--peers` string against `$TOP/scripts/start.sh`'s (they must be byte-identical); or just use `start-worker.sh`/`start-scheduler.sh`, which always carry the exact string the controllers were deployed with |
+| `start-worker.sh`/`start-scheduler.sh` not generated | `xcecd_virtual`/`xce_scheduler` don't exist in `$BINARY`'s build dir (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) |
+| `xcecd should be run as root` | You substituted real `xcecd` via `XCE_WORKER_BIN` | Either run it yourself with `sudo` outside this skill, or drop back to the default `xcecd_virtual` (no `sudo` needed, skips root-only subsystems) |
+| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
+| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
+| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |