diff --git a/.claude/skills/xce-build/SKILL.md b/.claude/skills/xce-build/SKILL.md index ee32c1db..fba78ce5 100644 --- a/.claude/skills/xce-build/SKILL.md +++ b/.claude/skills/xce-build/SKILL.md @@ -1,33 +1,90 @@ --- name: xce-build -description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." +description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." disable-model-invocation: false allowed-tools: Bash(*) --- ## build xce source code -Args: `[component] [profile|build_dir] [build_dir]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. Parse the ARGUMENTS text yourself and run the command below with the parsed values inline. +Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. `--incremental` can appear anywhere in the words (it's stripped out before the rest are parsed positionally — see below). Parse the ARGUMENTS text yourself and run the command below with the parsed values inline. - **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt. - **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`, `xceadmin` - **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir). - Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked). - **`controller` always builds `xce_test` too** (`meson compile … xce_controller controller_unit_test controller/xce_test`), in every profile — not just `asan`. `xce_test` is the driver `/xce-test-ha` needs; you never have to build it as a separate step after `/xce-build controller`, and the two binaries in a build dir are always the same commit/profile in lockstep. -- **`xceadmin` is Python, not meson, and ignores the ASAN/debug/release sanitizer semantics entirely** — it is dispatched by a plain shell case branch in `xce-build.sh`, deliberately not wired into `src/meson.build`/`meson.options` (see `.kiro/specs/admin-config-tool/design.md`). The `profile` arg still matters for it, but only to pick the pip install mode: default/`asan`/`debug` → `pip install -e .` (editable, for dev); `release` → `pip install .` (pinned, non-editable, for production). Every invocation — either install mode — syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, then runs `pytest` against whichever install just happened. Only the `release` run actually exercises the packaged (non-editable) schema resolution path — an editable-only test run can't catch a `package_data` regression there. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`). +- **`xceadmin` is Python, meson-driven (`src/xceadmin/meson.build`'s `custom_target` wraps `meson-build.sh`), and ignores the ASAN/debug/release sanitizer semantics entirely** — `xce-build.sh`'s `xceadmin` case just runs `meson compile -C "$BUILD_DIR" xceadmin`, the same target `nix build .#xcecore` and every other component use; the `profile` arg is accepted but has no effect on it (every invocation does the identical pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release` — unlike the old editable-install-by-default behavior, this always exercises real packaged schema resolution, not just a `release` run). For fast iterative dev without a full meson reconfigure, `xceadmin` can still be built standalone: `cd src/xceadmin && pip install -e .` (see `meson-build.sh` for the exact out-of-tree steps the meson target itself runs). Every invocation syncs the bundled `controller.capnp`/`c.capnp` and the 7 JSON validation schemas from `src/schema/` and `src/configs/schema/` into `xcecommon`'s packaged `schema_files/`, installs pinned into a fresh build-dir venv, runs `pytest` against that install, and — only after tests pass — builds an installable wheel from that same schema-synced tree into `$BUILD_DIR/xceadmin/dist/xceadmin--py3-none-any.whl` (`pip wheel … --no-deps`, so it's xceadmin's own wheel only, not a vendored bundle of `click`/`pycapnp`/etc). `xce-build.sh` prints the wheel's path and a ready-to-run `pip install ""` line after the build. Requires the nix dev shell (`pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` — all provided by `src/nixSupport/xce.nix`). +- **`--incremental` (bug-1565 escape hatch, opt-in, NOT the default)**: every invocation normally does `rm -rf "$BUILD_DIR"` + a full fresh `meson setup`, unconditionally — correct in isolation, but since the actual compile step afterward only rebuilds the ONE requested component, a plain single-component build silently deletes any OTHER already-built binary sharing that dir (e.g. `/xce-build xceadmin` after `/xce-build controller` wipes `xce_controller`). Pass `--incremental` to reconfigure the existing dir in place instead (`meson setup --reconfigure`), preserving whatever else was already built there — use it specifically when you want a sibling component alongside an already-built one, e.g. `/xce-build controller` then `/xce-build xceadmin --incremental` to end up with both in the same dir at once (needed for `/xce-deploy-ha`, which wants `xce_controller` + `xcecd_virtual` + `xce_scheduler` + the xceadmin wheel all together). Deliberately not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json — normally papered over as a loud, retry-fixable failure on a clean build; skipping the wipe could instead let the same gap cause silent staleness). If anything looks wrong after an incremental build, rebuild without the flag. -Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (editable install + tests), `/xce-build xceadmin release` (pinned install + tests). +Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (build + test + wheel, any profile arg is accepted and ignored), `/xce-build xceadmin --incremental` (same, but keeps other components already built in the same dir). -Run (substituting the parsed component/profile/build_dir; omit trailing empty ones): +Run (substituting the parsed component/profile/build_dir; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested): ```bash -nix develop ./src --command bash src/tools/xce-build.sh +nix develop ./src --command bash src/tools/xce-build.sh [--incremental] ``` -e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. +e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. For `/xce-build xceadmin --incremental`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --incremental`. ## Always dispatch the build via the Agent tool Do not run the `nix develop … xce-build.sh` command inline in the current session's Bash tool. Instead, on every invocation of this skill, launch a subagent with the Agent tool, giving it a fully self-contained prompt: repo working directory, the exact resolved command (component/profile/build_dir already substituted — do not make the subagent re-parse ARGUMENTS itself), what the command does, and what to report back (build success/failure, compiler errors, the meson test Ok/Fail summary line, and confirmation the expected binaries were freshly rebuilt with fresh mtimes). Wait for it (do not background it — the build is synchronous and finishes in well under this session's turn budget) and relay its report. This is deliberately a plain Agent-tool dispatch, not `context: fork` — that frontmatter field reproducibly failed to thread a non-empty ARGUMENTS payload into the forked subagent (see above), whereas an explicit, self-contained Agent prompt has no such dependency and is proven to work. + +## Installing the xceadmin wheel elsewhere + +`/xce-build xceadmin` always leaves a fresh, tested wheel at +`/xceadmin/dist/xceadmin--py3-none-any.whl` (e.g. +`xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl`). It's a normal +wheel — `pip install` it anywhere a compatible Python is available, no nix +or meson needed at install time: + +```bash +pip install /path/to/xce.build/xceadmin/dist/xceadmin-0.1.0-py3-none-any.whl +``` + +The wheel bundles its own `schema_files/` (the same `controller.capnp`, +`c.capnp`, and 7 JSON schemas synced in above) so `xceadmin config build` +works fully standalone against any `src/configs/`-shaped directory — it +does not need to run from inside this repo or the nix dev shell. Its 4 real +runtime imports (`click`, `pycapnp`, `PyYAML`, `jsonschema` — +`pyproject.toml`'s `dependencies`, verified against every actual `import` +in `src/xceadmin`/`src/xcecommon`) are declared, so a plain `pip install` +resolves them from PyPI on its own; the one caveat is `pycapnp`, whose +prebuilt PyPI wheels are usually fine outside the nix shell but aren't the +same guaranteed-consistent link against the capnproto C++ runtime the nix +shell's own `python3Packages.pycapnp` gets (see `pyproject.toml`'s +`dependencies` comment). + +**For that guarantee, do NOT use a `--system-site-packages` venv** — tried +and confirmed broken: this nix devShell injects `click`/`pycapnp`/`PyYAML`/ +`jsonschema` via `$PYTHONPATH`, not into the base python3 interpreter's own +site-packages (it's the bare interpreter derivation; each package is a +separate `/nix/store/...` path added to `$PYTHONPATH` by the shell hook — +confirmed via `nix develop … --command sh -c 'echo $PYTHONPATH'`). A +`--system-site-packages` venv only inherits the base interpreter's own +(empty) site-packages, so it silently can't see any of them — +`ModuleNotFoundError: No module named 'click'` the moment you actually run +it, even though the venv's own `pip install` step succeeds. Instead extract +the wheel directly and bake the resolved `$PYTHONPATH` into a wrapper at +install time (no venv/pip needed at all) — this is exactly what +`/xce-deploy-ha`'s `deploy-ha.sh` does for `scripts/xceadmin`, and it's +proven to work with zero nix/network dependency at runtime: +```bash +mkdir -p /path/to/xceadmin-site +python3 -m zipfile -e /path/to/xceadmin-0.1.0-py3-none-any.whl /path/to/xceadmin-site +nix develop ./src --command bash -c ' + cat > /path/to/xceadmin << EOF +#!'"$(nix develop ./src --command which bash)"' +export PYTHONPATH="/path/to/xceadmin-site:$PYTHONPATH" +exec '"$(nix develop ./src --command which python3)"' -m xceadmin.cli "\$@" +EOF + chmod +x /path/to/xceadmin +' +``` +(`$PYTHONPATH` must be expanded **inside** the `nix develop --command` shell, +not the outer one, so the resolved nix store paths get baked into the +wrapper literally — see `deploy-ha.sh`'s own xceadmin-install block for the +exact working pattern.) diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 29cd4afc..49d2e73a 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -53,7 +53,7 @@ Arguments are space-separated `host:port` pairs, one per node, in priority order If no arguments are given, default to `localhost:9991 localhost:9992 localhost:9993`. -## Step 1 — Resolve paths +## Step 1 — Resolve paths and the deploy environment's bash ```bash BINARY="src/xce.build/controller/xce_controller" @@ -67,19 +67,45 @@ ls -lh "$BINARY" If missing, tell the user to run `/xce-build controller` first. +Also resolve the bash that actually lives in the deploy environment (the nix +dev shell) instead of assuming a bare `bash` on the invoker's own `PATH` +means the same thing: + +```bash +BASH_BIN=$(nix develop ./src --command which bash) +``` + +**Why not just type `bash`:** `deploy-ha.sh` and every script it generates +are bash (arrays, `+=`, `<<<`) even though `deploy-ha.sh` itself carries a +`#!/bin/sh` shebang in the repo today — a real incident, not a hypothetical: +a working-tree edit once flipped that shebang while the script still used +`${BASH_SOURCE[0]}` and bash-only array syntax, which only kept working here +because this environment's `/bin/sh` happens to be symlinked to bash. On a +system where `/bin/sh` is dash (common on Debian/Ubuntu), that same file +would fail immediately. Resolving `$BASH_BIN` once via `which bash` *inside* +the nix dev shell — the same shell that provides `openssl` and the correct +runtime libs for the binaries — and using it explicitly for every script +invocation below sidesteps trusting any file's shebang line at all, in +either direction (a stale one, or an ambient `bash` that might not resolve +to the intended interpreter). + ## Step 2 — Stop any running instances ```bash kill -9 $(pgrep xce_controller) 2>/dev/null || true +kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true +kill -9 $(pgrep xce_scheduler) 2>/dev/null || true ``` ## Step 3 — Deploy cluster -Join peers as a comma-separated string and call the deploy script: +Join peers as a comma-separated string and call the deploy script — always +via `nix develop` (needed for `openssl`) and the resolved `$BASH_BIN` (never +a bare `bash`, never `./deploy-ha.sh` relying on its own shebang): ```bash PEERS="localhost:9991,localhost:9992,localhost:9993" # from args -bash src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS" +nix develop ./src --command "$BASH_BIN" src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS" ``` Every invocation is a fresh, self-contained deploy — no flag needed to force @@ -94,6 +120,15 @@ a rebuild, nothing is ever reused across deploys: RSA 2048, 10y validity — long-lived since it's per-deploy anyway; loses no real security property since TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase). Errors if `openssl` isn't on `PATH`. +- The bash baked into the shebang of every script `deploy-ha.sh` generates + (`start.sh`, `cleanup.sh`, `start-worker.sh`, `start-scheduler.sh`) is + whatever `command -v bash` resolves to *inside deploy-ha.sh's own + invocation* — i.e. `$BASH_BIN` from Step 1, since you invoked it with that + interpreter under `nix develop`. This skill always still invokes each + generated script explicitly as `"$BASH_BIN" script.sh …` below (the + shebang is belt-and-suspenders for a human running one directly later, not + load-bearing for this skill or `/xce-test-ha`, which calls the same + `deploy-ha.sh`). This: - Creates `$TOP/certs/`, `$TOP/controller/ctrl-{1..N}/conf`, `$TOP/controller/ctrl-{1..N}/data`, `$TOP/logs/` @@ -101,22 +136,87 @@ This: - Places `cluster.bin.` + `conf/current` in `controller/ctrl-1/conf/` only (standbys receive config from primary via `OPCode_replConfigPush`) - Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh` +- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh` + **if** `xcecd_virtual`/`xce_scheduler` exist alongside `$BINARY` (same + build dir) — soft-skipped with a warning, not a hard error, if either is + missing (so `/xce-test-ha`, which never builds worker/scheduler and calls + this same script, is unaffected). See Step 4b. +- Also extracts a standalone xceadmin into `$TOP/xceadmin-site` and generates + `$TOP/scripts/xceadmin` **if** a wheel exists at + `/xceadmin/dist/*.whl` — same soft-skip-with-warning + convention. See Step 4c. ## Step 4 — Start all nodes -Start each controller in the background. Use `-vvv` for raft trace logs: +Start each controller in the background, via `$BASH_BIN`. Use `-vvv` for raft trace logs: ```bash -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv & sleep 0.5 -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv & sleep 0.5 -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv & ``` The startup stagger (0.5s between nodes) allows higher-priority nodes to begin their election before lower-priority ones start. +## Step 4b — Start worker and scheduler (optional) + +Once at least one controller is up (no need to wait for a full election — +both fail over across the whole candidate list on their own), start a worker +and/or the scheduler with the **identical `--peers` configuration** the +controllers were deployed with — `deploy-ha.sh` bakes the exact same +`ctrl-1=host:port,ctrl-2=host:port,...` string into both generated scripts, +so there is nothing to keep in sync by hand: + +```bash +"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv & # name defaults to worker-1 +"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & # scheduler is a cluster singleton, no name +``` + +- **`start-worker.sh`** runs `xcecd_virtual`, not production `xcecd` — it + skips the root-only sqlite/db-recovery and domain-socket subsystems, so it + runs as an ordinary user with no `sudo` (this session/skill cannot `sudo` + anyway). Logs land at `$TOP/logs/xcecd-.log`. Start more than one by + passing a different name: `start-worker.sh worker-2`. To point at a real + `xcecd` instead (needs root — run it yourself outside this skill), set + `XCE_WORKER_BIN=/path/to/xcecd` before invoking the script. +- **`start-scheduler.sh`** runs `xce_scheduler`; its identity is fixed + internally to `xce_scheduler` (no name argument). Logs to + `$TOP/logs/scheduler.log`. +- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`. +- Verify the scheduler joined: `grep "sched_init: joined cluster" "$TOP/logs/scheduler.log"`. + +## Step 4c — Use xceadmin against this deployment (optional) + +`deploy-ha.sh` also installs a **standalone** `xceadmin` at `$TOP/xceadmin-site` +(extracted from the `/xce-build xceadmin`-produced wheel — soft-skipped with +a warning if no wheel exists in the same build dir as `$BINARY`, same as +`start-worker.sh`/`start-scheduler.sh`) and generates a wrapper at +`$TOP/scripts/xceadmin`: + +```bash +"$BASH_BIN" "$TOP/scripts/xceadmin" config push "$TOP/controller/ctrl-1/conf/cluster.bin." +"$BASH_BIN" "$TOP/scripts/xceadmin" scheduler record status +``` + +- **Genuinely standalone**: works with zero nix/network dependency at + runtime (`$PYTHONPATH` is baked into the wrapper at deploy time — see + `deploy-ha.sh`'s own comment for why a `--system-site-packages` venv does + NOT work for this, a real dead end already ruled out). No need to prefix + with `"$BASH_BIN"`/`nix develop` to run it, only to *generate* it in Step 3. +- **Defaults `--controller` to this deployment's own first `--peers` + candidate**, not xceadmin's generic bare fallback (`localhost:9991`, + unrelated to any real deployment's actual ports). The wrapper exports + `XCEADMIN_CONTROLLER=""`, which every + `--controller`-accepting xceadmin subcommand (`config push`, `scheduler + record start`/`stop`/`status`) reads via `envvar=` fallback — an explicit + `--controller host:port` on the command line still overrides it. Before + this (bug-1566), a bare `scripts/xceadmin scheduler record status` with no + `--controller` silently tried port 9991 and failed with a connection- + refused that had nothing to do with the actual deployment. + ## Step 5 — Verify cluster health Wait ~15 seconds for election and config replication, then check: @@ -154,20 +254,27 @@ RAFT[p0] term=2 event=became_leader Stop all instances, clean up state, and restart without redeploying: ```bash -# 1. Stop running controllers +# 1. Stop running controllers (and worker/scheduler, if you started them) kill -9 $(pgrep xce_controller) 2>/dev/null || true +kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true +kill -9 $(pgrep xce_scheduler) 2>/dev/null || true # 2. Run the generated cleanup script # Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/ # Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log -bash "$TOP/scripts/cleanup.sh" +"$BASH_BIN" "$TOP/scripts/cleanup.sh" # 3. Restart all nodes -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-1 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv & sleep 0.5 -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-2 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv & sleep 0.5 -VERBOSE=-vvv "$TOP/scripts/start.sh" ctrl-3 & +"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv & + +# 4. Restart worker/scheduler if you had them running (Step 4b) — the +# generated scripts still carry the same --peers, nothing to redo there +"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv & +"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & ``` **What cleanup.sh preserves:** @@ -190,3 +297,9 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | `repl_ack_decode_and_dispatch: unexpected opcode` | Heartbeat NOP on promoted ep | Fixed in current build; update binary | | `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports | | `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above | +| `SSL_do_handshake: Broken pipe` flooding from a worker/scheduler | A typo'd hostname in the `--peers` you passed (e.g. `local` instead of `localhost`) — the connect reaches SSL before failing, so it doesn't error out cleanly | Diff the failing `--peers` string against `$TOP/scripts/start.sh`'s (they must be byte-identical); or just use `start-worker.sh`/`start-scheduler.sh`, which always carry the exact string the controllers were deployed with | +| `start-worker.sh`/`start-scheduler.sh` not generated | `xcecd_virtual`/`xce_scheduler` don't exist in `$BINARY`'s build dir (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) | +| `xcecd should be run as root` | You substituted real `xcecd` via `XCE_WORKER_BIN` | Either run it yourself with `sudo` outside this skill, or drop back to the default `xcecd_virtual` (no `sudo` needed, skips root-only subsystems) | +| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy | +| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` before the `exec` line | +| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) | diff --git a/.kiro/steering/build-deployment.md b/.kiro/steering/build-deployment.md index d45c81d0..beac0b90 100644 --- a/.kiro/steering/build-deployment.md +++ b/.kiro/steering/build-deployment.md @@ -16,17 +16,21 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`): /xce-build worker # lib + worker (daemon only, no unit tests) /xce-build apiserver # lib + apiserver + xcecli + Go unit tests /xce-build xcecli # xcecli + Go tests -/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests) +/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel) ``` +`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `/xceadmin/dist/xceadmin--py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets. + `controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it. Each invocation: 1. `nix develop` into the hermetic shell -2. `meson setup --buildtype= -Db_sanitize=` (fresh clean build) +2. `meson setup --buildtype= -Db_sanitize=` (fresh clean build — `rm -rf ` then reconfigure, unconditionally, for EVERY component including single-component invocations) 3. `meson compile -C [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything 4. Run registered meson tests (skipped for non-`asan` profiles — see below) +**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward. + **Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use. ## ASAN Integration @@ -93,26 +97,42 @@ controller/ conf/ # receives config via OPCode_replConfigPush data/ ctrl-3/ (same as ctrl-2) -logs/ # per-node log files +xceadmin-site/ # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below +logs/ # per-node + worker/scheduler log files scripts/ start.sh ctrl-N # launch a single node + start-worker.sh [name] # launch xcecd_virtual against the SAME --peers (if worker built) + start-scheduler.sh # launch xce_scheduler against the SAME --peers (if scheduler built) + xceadmin ... # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs ``` -**Startup** (from deploy dir): +`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment. + +**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below): ```bash -VERBOSE=-vvv scripts/start.sh ctrl-1 & # primary first +BASH_BIN=$(nix develop ./src --command which bash) # never a bare `bash`; see below +"$BASH_BIN" scripts/start.sh ctrl-1 -vvv & # primary first sleep 0.5 -VERBOSE=-vvv scripts/start.sh ctrl-2 & # standby (0.5s stagger for election ordering) +"$BASH_BIN" scripts/start.sh ctrl-2 -vvv & # standby (0.5s stagger for election ordering) sleep 0.5 -VERBOSE=-vvv scripts/start.sh ctrl-3 & +"$BASH_BIN" scripts/start.sh ctrl-3 -vvv & + +# Optional, once at least one controller is up: +"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv & +"$BASH_BIN" scripts/start-scheduler.sh -v & ``` +(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.) + +**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates. **Clean restart** (keeps certs + ctrl-1 config, wipes all data): ```bash kill -9 $(pgrep xce_controller) 2>/dev/null || true -bash scripts/cleanup.sh -# Then re-start all nodes +kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true +kill -9 $(pgrep xce_scheduler) 2>/dev/null || true +"$BASH_BIN" scripts/cleanup.sh +# Then re-start all nodes (and worker/scheduler, if you had them running) ``` ## Go Frontend Build @@ -138,5 +158,6 @@ Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` f | Scheduler daemon | same | `xce_scheduler` binary | | REST API server | same | `xce_apiserver` binary | | CLI tool | `nix build .#xcecli` | `xcecli` binary | +| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin--py3-none-any.whl` at `/xceadmin/dist/` — `pip install` it directly | | AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) | | Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries | diff --git a/.kiro/steering/ha-async-io.md b/.kiro/steering/ha-async-io.md index f61779aa..c57e5797 100644 --- a/.kiro/steering/ha-async-io.md +++ b/.kiro/steering/ha-async-io.md @@ -65,7 +65,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics: - **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY. - **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence. - **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever. -- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. +- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary). ## Replication Consistency (Two-Tier) diff --git a/.kiro/steering/persistence.md b/.kiro/steering/persistence.md index 94dac3e3..14a84fbf 100644 --- a/.kiro/steering/persistence.md +++ b/.kiro/steering/persistence.md @@ -25,7 +25,8 @@ Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_ca Two correctness rules when touching this path: - **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`. -- **Commit wake-up via callback, not a back-reference.** The flusher calls `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. +- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix. +- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved. `--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`). @@ -70,7 +71,7 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi ## Testing Durability Deterministically -- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync` and advances `durable_lsn`) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). +- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one. - **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`. - **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures). - Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`. @@ -82,8 +83,8 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi | Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` | | Durable floor (WAL tip) | `persist_durable_floor(p)` | | Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` | -| Group-commit wake | `persist_set_durable_cb` → `on_durable_advance(ctx)` | -| Synchronous flush (tests) | `persist_flush(p)` | +| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance | +| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) | | Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` | | Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` | | Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |