lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: sync skills and steering docs with bug-1562..1566 findings

Browse source 
- .claude/skills/xce-build/SKILL.md: --incremental flag,
  xceadmin wheel building, corrected the standalone-install
  guidance (the earlier --system-site-packages venv approach was
  wrong -- nix devShell packages are $PYTHONPATH entries, not
  merged into site-packages).
- .claude/skills/xce-deploy-ha/SKILL.md: $BASH_BIN resolution,
  worker/scheduler startup, xceadmin usage against a deployment,
  troubleshooting rows for the newly-fixed bugs.
- .kiro/steering/ha-async-io.md: bug-1562's standby liveness
  watchdog re-arm invariant.
- .kiro/steering/persistence.md: bug-1563 (fsync backoff) and
  bug-1564 (persist_flush durable-advance callback).
- .kiro/steering/build-deployment.md: xceadmin wheel docs,
  bug-1565's --incremental flag, deploy dir tree updates.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 10:21:25 +00:00
parent 0c8e417bda
commit 9f539583d3
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 225 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

71
.claude/skills/xce-build/SKILL.md
View file

@ -1,33 +1,90 @@
---
name: xce-build
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
disable-model-invocation: false
allowed-tools: Bash(*)
---
## build xce source code
Args: `[component] [profile|build_dir] [build_dir]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.
Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. `--incremental` can appear anywhere in the words (it's stripped out before the rest are parsed positionally — see below). Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.
- **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt.
- **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`, `xceadmin`
- **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir).
- Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked).
- **`controller` always builds `xce_test` too** (`meson compile … xce_controller controller_unit_test controller/xce_test`), in every profile — not just `asan`. `xce_test` is the driver `/xce-test-ha` needs; you never have to build it as a separate step after `/xce-build controller`, and the two binaries in a build dir are always the same commit/profile in lockstep.
- **`xceadmin` is Python, not meson, and ignores the ASAN/debug/release sanitizer semantics entirely** — it is dispatched by a plain shell case branch in `xce-build.sh`, deliberately not wired into `src/meson.build`/`meson.options` (see `.kiro/specs/admin-config-tool/design.md`). The `profile` arg still matters for it, but only to pick the pip install mode: default/`asan`/`debug` → `pip install -e .` (editable, for dev); `release` → `pip install .` (pinned, non-editable, for production). Every invocation — either install mode — syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, then runs `pytest` against whichever install just happened. Only the `release` run actually exercises the packaged (non-editable) schema resolution path — an editable-only test run can't catch a `package_data` regression there. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`).
- **`xceadmin` is Python, meson-driven (`src/xceadmin/meson.build`'s `custom_target` wraps `meson-build.sh`), and ignores the ASAN/debug/release sanitizer semantics entirely** — `xce-build.sh`'s `xceadmin` case just runs `meson compile -C "$BUILD_DIR" xceadmin`, the same target `nix build .#xcecore` and every other component use; the `profile` arg is accepted but has no effect on it (every invocation does the identical pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release` — unlike the old editable-install-by-default behavior, this always exercises real packaged schema resolution, not just a `release` run). For fast iterative dev without a full meson reconfigure, `xceadmin` can still be built standalone: `cd src/xceadmin && pip install -e .` (see `meson-build.sh` for the exact out-of-tree steps the meson target itself runs). Every invocation syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, installs pinned into a fresh build-dir venv, runs `pytest` against that install, and — only after tests pass — builds an installable wheel from that same schema-synced tree into `$BUILD_DIR/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (`pip wheel … --no-deps`, so it's xceadmin's own wheel only, not a vendored bundle of `click`/`pycapnp`/etc). `xce-build.sh` prints the wheel's path and a ready-to-run `pip install "<path>"` line after the build. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`).
- **`--incremental` (bug-1565 escape hatch, opt-in, NOT the default)**: every invocation normally does `rm -rf "$BUILD_DIR"` + a full fresh `meson setup`, unconditionally — correct in isolation, but since the actual compile step afterward only rebuilds the ONE requested component, a plain single-component build silently deletes any OTHER already-built binary sharing that dir (e.g. `/xce-build xceadmin` after `/xce-build controller` wipes `xce_controller`). Pass `--incremental` to reconfigure the existing dir in place instead (`meson setup --reconfigure`), preserving whatever else was already built there — use it specifically when you want a sibling component alongside an already-built one, e.g. `/xce-build controller` then `/xce-build xceadmin --incremental` to end up with both in the same dir at once (needed for `/xce-deploy-ha`, which wants `xce_controller` + `xcecd_virtual` + `xce_scheduler` + the xceadmin wheel all together). Deliberately not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json — normally papered over as a loud, retry-fixable failure on a clean build; skipping the wipe could instead let the same gap cause silent staleness). If anything looks wrong after an incremental build, rebuild without the flag.
Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (editable install + tests), `/xce-build xceadmin release` (pinned install + tests).
Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (build + test + wheel, any profile arg is accepted and ignored), `/xce-build xceadmin --incremental` (same, but keeps other components already built in the same dir).
Run (substituting the parsed component/profile/build_dir; omit trailing empty ones):
Run (substituting the parsed component/profile/build_dir; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested):
```bash
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir>
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> [--incremental]
```
e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`.
e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. For `/xce-build xceadmin --incremental`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --incremental`.
## Always dispatch the build via the Agent tool
Do not run the `nix develop … xce-build.sh` command inline in the current session's Bash tool. Instead, on every invocation of this skill, launch a subagent with the Agent tool, giving it a fully self-contained prompt: repo working directory, the exact resolved command (component/profile/build_dir already substituted — do not make the subagent re-parse ARGUMENTS itself), what the command does, and what to report back (build success/failure, compiler errors, the meson test Ok/Fail summary line, and confirmation the expected binaries were freshly rebuilt with fresh mtimes). Wait for it (do not background it — the build is synchronous and finishes in well under this session's turn budget) and relay its report.
This is deliberately a plain Agent-tool dispatch, not `context: fork` — that frontmatter field reproducibly failed to thread a non-empty ARGUMENTS payload into the forked subagent (see above), whereas an explicit, self-contained Agent prompt has no such dependency and is proven to work.
## Installing the xceadmin wheel elsewhere
`/xce-build xceadmin` always leaves a fresh, tested wheel at
`<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (e.g.
`xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl`). It's a normal
wheel — `pip install` it anywhere a compatible Python is available, no nix
or meson needed at install time:
```bash
pip install /path/to/xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl
```
The wheel bundles its own `schema_files/` (the same `controller.capnp`,
`c.capnp`, and 7 JSON schemas synced in above) so `xceadmin config build`
works fully standalone against any `src/configs/`-shaped directory — it
does not need to run from inside this repo or the nix dev shell. Its 4 real
runtime imports (`click`, `pycapnp`, `PyYAML`, `jsonschema` —
`pyproject.toml`'s `dependencies`, verified against every actual `import`
in `src/xceadmin`/`src/xcecommon`) are declared, so a plain `pip install`
resolves them from PyPI on its own; the one caveat is `pycapnp`, whose
prebuilt PyPI wheels are usually fine outside the nix shell but aren't the
same guaranteed-consistent link against the capnproto C++ runtime the nix
shell's own `python3Packages.pycapnp` gets (see `pyproject.toml`'s
`dependencies` comment).
**For that guarantee, do NOT use a `--system-site-packages` venv** — tried
and confirmed broken: this nix devShell injects `click`/`pycapnp`/`PyYAML`/
`jsonschema` via `$PYTHONPATH`, not into the base python3 interpreter's own
site-packages (it's the bare interpreter derivation; each package is a
separate `/nix/store/...` path added to `$PYTHONPATH` by the shell hook —
confirmed via `nix develop … --command sh -c 'echo $PYTHONPATH'`). A
`--system-site-packages` venv only inherits the base interpreter's own
(empty) site-packages, so it silently can't see any of them —
`ModuleNotFoundError: No module named 'click'` the moment you actually run
it, even though the venv's own `pip install` step succeeds. Instead extract
the wheel directly and bake the resolved `$PYTHONPATH` into a wrapper at
install time (no venv/pip needed at all) — this is exactly what
`/xce-deploy-ha`'s `deploy-ha.sh` does for `scripts/xceadmin`, and it's
proven to work with zero nix/network dependency at runtime:
```bash
mkdir -p /path/to/xceadmin-site
python3 -m zipfile -e /path/to/xceadmin-0.1.0-py3-none-any.whl /path/to/xceadmin-site
nix develop ./src --command bash -c '
cat > /path/to/xceadmin << EOF
#!'"$(nix develop ./src --command which bash)"'
export PYTHONPATH="/path/to/xceadmin-site:$PYTHONPATH"
exec '"$(nix develop ./src --command which python3)"' -m xceadmin.cli "\$@"
EOF
chmod +x /path/to/xceadmin
'
```
(`$PYTHONPATH` must be expanded **inside** the `nix develop --command` shell,
not the outer one, so the resolved nix store paths get baked into the
wrapper literally — see `deploy-ha.sh`'s own xceadmin-install block for the
exact working pattern.)

137
.claude/skills/xce-deploy-ha/SKILL.md
View file

@ -53,7 +53,7 @@ Arguments are space-separated `host:port` pairs, one per node, in priority order
If no arguments are given, default to `localhost:9991 localhost:9992 localhost:9993`.
## Step 1 — Resolve paths
## Step 1 — Resolve paths and the deploy environment's bash
```bash
BINARY="src/xce.build/controller/xce_controller"
@ -67,19 +67,45 @@ ls -lh "$BINARY"
If missing, tell the user to run `/xce-build controller` first.
Also resolve the bash that actually lives in the deploy environment (the nix
dev shell) instead of assuming a bare `bash` on the invoker's own `PATH`
means the same thing:
```bash
BASH_BIN=$(nix develop ./src --command which bash)
```
**Why not just type `bash`:** `deploy-ha.sh` and every script it generates
are bash (arrays, `+=`, `<<<`) even though `deploy-ha.sh` itself carries a
`#!/bin/sh` shebang in the repo today — a real incident, not a hypothetical:
a working-tree edit once flipped that shebang while the script still used
`${BASH_SOURCE[0]}` and bash-only array syntax, which only kept working here
because this environment's `/bin/sh` happens to be symlinked to bash. On a
system where `/bin/sh` is dash (common on Debian/Ubuntu), that same file
would fail immediately. Resolving `$BASH_BIN` once via `which bash` *inside*
the nix dev shell — the same shell that provides `openssl` and the correct
runtime libs for the binaries — and using it explicitly for every script
invocation below sidesteps trusting any file's shebang line at all, in
either direction (a stale one, or an ambient `bash` that might not resolve
to the intended interpreter).
## Step 2 — Stop any running instances
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
```
## Step 3 — Deploy cluster
Join peers as a comma-separated string and call the deploy script:
Join peers as a comma-separated string and call the deploy script — always
via `nix develop` (needed for `openssl`) and the resolved `$BASH_BIN` (never
a bare `bash`, never `./deploy-ha.sh` relying on its own shebang):
```bash
PEERS="localhost:9991,localhost:9992,localhost:9993" # from args
bash src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS"
nix develop ./src --command "$BASH_BIN" src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS"
```
Every invocation is a fresh, self-contained deploy — no flag needed to force
@ -94,6 +120,15 @@ a rebuild, nothing is ever reused across deploys:
RSA 2048, 10y validity — long-lived since it's per-deploy anyway; loses no
real security property since TLS verification is `SSL_VERIFY_NONE`
everywhere in this codebase). Errors if `openssl` isn't on `PATH`.
- The bash baked into the shebang of every script `deploy-ha.sh` generates
(`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is
whatever `command -v bash` resolves to *inside deploy-ha.sh's own
invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that
interpreter under `nix develop`. This skill always still invokes each
generated script explicitly as `"$BASH_BIN" script.sh …` below (the
shebang is belt-and-suspenders for a human running one directly later, not
load-bearing for this skill or `/xce-test-ha`, which calls the same
`deploy-ha.sh`).
This:
- Creates `$TOP/certs/`, `$TOP/controller/ctrl-{1..N}/conf`, `$TOP/controller/ctrl-{1..N}/data`, `$TOP/logs/`
@ -101,22 +136,87 @@ This:
- Places `cluster.bin.<tag>` + `conf/current` in `controller/ctrl-1/conf/` only
(standbys receive config from primary via `OPCode_replConfigPush`)
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh`
**if** `xcecd_virtual`/`xce_scheduler` exist alongside `$BINARY` (same
build dir) — soft-skipped with a warning, not a hard error, if either is
missing (so `/xce-test-ha`, which never builds worker/scheduler and calls
this same script, is unaffected). See Step 4b.
- Also extracts a standalone xceadmin into `$TOP/xceadmin-site` and generates
`$TOP/scripts/xceadmin` **if** a wheel exists at
`<same build dir>/xceadmin/dist/*.whl` — same soft-skip-with-warning
convention. See Step 4c.
## Step 4 — Start all nodes
Start each controller in the background. Use `-vvv` for raft trace logs:
Start each controller in the background, via `$BASH_BIN`. Use `-vvv` for raft trace logs:
```bash
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
sleep 0.5
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
sleep 0.5
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
```
The startup stagger (0.5s between nodes) allows higher-priority nodes to
begin their election before lower-priority ones start.
## Step 4b — Start worker and scheduler (optional)
Once at least one controller is up (no need to wait for a full election —
both fail over across the whole candidate list on their own), start a worker
and/or the scheduler with the **identical `--peers` configuration** the
controllers were deployed with — `deploy-ha.sh` bakes the exact same
`ctrl-1=host:port,ctrl-2=host:port,...` string into both generated scripts,
so there is nothing to keep in sync by hand:
```bash
"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv & # name defaults to worker-1
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & # scheduler is a cluster singleton, no name
```
- **`start-worker.sh`** runs `xcecd_virtual`, not production `xcecd` — it
skips the root-only sqlite/db-recovery and domain-socket subsystems, so it
runs as an ordinary user with no `sudo` (this session/skill cannot `sudo`
anyway). Logs land at `$TOP/logs/xcecd-<name>.log`. Start more than one by
passing a different name: `start-worker.sh worker-2`. To point at a real
`xcecd` instead (needs root — run it yourself outside this skill), set
`XCE_WORKER_BIN=/path/to/xcecd` before invoking the script.
- **`start-scheduler.sh`** runs `xce_scheduler`; its identity is fixed
internally to `xce_scheduler` (no name argument). Logs to
`$TOP/logs/scheduler.log`.
- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`.
- Verify the scheduler joined: `grep "sched_init: joined cluster" "$TOP/logs/scheduler.log"`.
## Step 4c — Use xceadmin against this deployment (optional)
`deploy-ha.sh` also installs a **standalone** `xceadmin` at `$TOP/xceadmin-site`
(extracted from the `/xce-build xceadmin`-produced wheel — soft-skipped with
a warning if no wheel exists in the same build dir as `$BINARY`, same as
`start-worker.sh`/`start-scheduler.sh`) and generates a wrapper at
`$TOP/scripts/xceadmin`:
```bash
"$BASH_BIN" "$TOP/scripts/xceadmin" config push "$TOP/controller/ctrl-1/conf/cluster.bin.<tag>"
"$BASH_BIN" "$TOP/scripts/xceadmin" scheduler record status
```
- **Genuinely standalone**: works with zero nix/network dependency at
runtime (`$PYTHONPATH` is baked into the wrapper at deploy time — see
`deploy-ha.sh`'s own comment for why a `--system-site-packages` venv does
NOT work for this, a real dead end already ruled out). No need to prefix
with `"$BASH_BIN"`/`nix develop` to run it, only to *generate* it in Step 3.
- **Defaults `--controller` to this deployment's own first `--peers`
candidate**, not xceadmin's generic bare fallback (`localhost:9991`,
unrelated to any real deployment's actual ports). The wrapper exports
`XCEADMIN_CONTROLLER="<first --peers candidate>"`, which every
`--controller`-accepting xceadmin subcommand (`config push`, `scheduler
record start`/`stop`/`status`) reads via `envvar=` fallback — an explicit
`--controller host:port` on the command line still overrides it. Before
this (bug-1566), a bare `scripts/xceadmin scheduler record status` with no
`--controller` silently tried port 9991 and failed with a connection-
refused that had nothing to do with the actual deployment.
## Step 5 — Verify cluster health
Wait ~15 seconds for election and config replication, then check:
@ -154,20 +254,27 @@ RAFT[p0] term=2 event=became_leader
Stop all instances, clean up state, and restart without redeploying:
```bash
# 1. Stop running controllers
# 1. Stop running controllers (and worker/scheduler, if you started them)
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
# 2. Run the generated cleanup script
# Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
# Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
bash "$TOP/scripts/cleanup.sh"
"$BASH_BIN" "$TOP/scripts/cleanup.sh"
# 3. Restart all nodes
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
sleep 0.5
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
sleep 0.5
VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 &
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &
# 4. Restart worker/scheduler if you had them running (Step 4b) — the
# generated scripts still carry the same --peers, nothing to redo there
"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &
```
**What cleanup.sh preserves:**
@ -190,3 +297,9 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
| `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary |
| `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports |
| `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above |
| `SSL_do_handshake: Broken pipe` flooding from a worker/scheduler | A typo'd hostname in the `--peers` you passed (e.g. `local` instead of `localhost`) — the connect reaches SSL before failing, so it doesn't error out cleanly | Diff the failing `--peers` string against `$TOP/scripts/start.sh`'s (they must be byte-identical); or just use `start-worker.sh`/`start-scheduler.sh`, which always carry the exact string the controllers were deployed with |
| `start-worker.sh`/`start-scheduler.sh` not generated | `xcecd_virtual`/`xce_scheduler` don't exist in `$BINARY`'s build dir (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) |
| `xcecd should be run as root` | You substituted real `xcecd` via `XCE_WORKER_BIN` | Either run it yourself with `sudo` outside this skill, or drop back to the default `xcecd_virtual` (no `sudo` needed, skips root-only subsystems) |
| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |

39
.kiro/steering/build-deployment.md
View file

@ -16,17 +16,21 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
/xce-build worker # lib + worker (daemon only, no unit tests)
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
/xce-build xcecli # xcecli + Go tests
/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests)
/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
```
`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
Each invocation:
1. `nix develop` into the hermetic shell
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
## ASAN Integration
@ -93,26 +97,42 @@ controller/
conf/ # receives config via OPCode_replConfigPush
data/
ctrl-3/ (same as ctrl-2)
logs/ # per-node log files
xceadmin-site/ # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
logs/ # per-node + worker/scheduler log files
scripts/
start.sh ctrl-N # launch a single node
start-worker.sh [name] # launch xcecd_virtual against the SAME --peers (if worker built)
start-scheduler.sh # launch xce_scheduler against the SAME --peers (if scheduler built)
xceadmin ... # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs
```
**Startup** (from deploy dir):
`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.
**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
```bash
VERBOSE=-vvv scripts/start.sh ctrl-1 & # primary first
BASH_BIN=$(nix develop ./src --command which bash) # never a bare `bash`; see below
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv & # primary first
sleep 0.5
VERBOSE=-vvv scripts/start.sh ctrl-2 & # standby (0.5s stagger for election ordering)
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv & # standby (0.5s stagger for election ordering)
sleep 0.5
VERBOSE=-vvv scripts/start.sh ctrl-3 &
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &
# Optional, once at least one controller is up:
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
"$BASH_BIN" scripts/start-scheduler.sh -v &
```
(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)
**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.
**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
bash scripts/cleanup.sh
# Then re-start all nodes
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
"$BASH_BIN" scripts/cleanup.sh
# Then re-start all nodes (and worker/scheduler, if you had them running)
```
## Go Frontend Build
@ -138,5 +158,6 @@ Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` f
| Scheduler daemon | same | `xce_scheduler` binary |
| REST API server | same | `xce_apiserver` binary |
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) |
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |

2
.kiro/steering/ha-async-io.md
View file

@ -65,7 +65,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics:
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary).
## Replication Consistency (Two-Tier)

9
.kiro/steering/persistence.md
View file

@ -25,7 +25,8 @@ Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_ca
Two correctness rules when touching this path:
- **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`.
- **Commit wake-up via callback, not a back-reference.** The flusher calls `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state.
- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix.
- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved.
`--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`).
@ -70,7 +71,7 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi
## Testing Durability Deterministically
- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync` and advances `durable_lsn`) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip).
- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one.
- **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`.
- **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures).
- Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`.
@ -82,8 +83,8 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi
| Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` |
| Durable floor (WAL tip) | `persist_durable_floor(p)` |
| Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` |
| Group-commit wake | `persist_set_durable_cb` → `on_durable_advance(ctx)` |
| Synchronous flush (tests) | `persist_flush(p)` |
| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance |
| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) |
| Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` |
| Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` |
| Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |