docs: sync skills and steering docs with bug-1562..1566 findings

- .claude/skills/xce-build/SKILL.md: --incremental flag,
  xceadmin wheel building, corrected the standalone-install
  guidance (the earlier --system-site-packages venv approach was
  wrong -- nix devShell packages are $PYTHONPATH entries, not
  merged into site-packages).
- .claude/skills/xce-deploy-ha/SKILL.md: $BASH_BIN resolution,
  worker/scheduler startup, xceadmin usage against a deployment,
  troubleshooting rows for the newly-fixed bugs.
- .kiro/steering/ha-async-io.md: bug-1562's standby liveness
  watchdog re-arm invariant.
- .kiro/steering/persistence.md: bug-1563 (fsync backoff) and
  bug-1564 (persist_flush durable-advance callback).
- .kiro/steering/build-deployment.md: xceadmin wheel docs,
  bug-1565's --incremental flag, deploy dir tree updates.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.kiro/steering/build-deployment.md

@@ -16,17 +16,21 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
 /xce-build worker       # lib + worker (daemon only, no unit tests)
 /xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
 /xce-build xcecli       # xcecli + Go tests
-/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests)
+/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
 ```
 
+`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.
+
 `controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
 
 Each invocation:
 1. `nix develop` into the hermetic shell
-2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
+2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
 3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything
 4. Run registered meson tests (skipped for non-`asan` profiles — see below)
 
+**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.
+
 **Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
 
 ## ASAN Integration
@@ -93,26 +97,42 @@ controller/
     conf/                        # receives config via OPCode_replConfigPush
     data/
   ctrl-3/ (same as ctrl-2)
-logs/                           # per-node log files
+xceadmin-site/                  # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
+logs/                           # per-node + worker/scheduler log files
 scripts/
   start.sh ctrl-N               # launch a single node
+  start-worker.sh [name]        # launch xcecd_virtual against the SAME --peers (if worker built)
+  start-scheduler.sh            # launch xce_scheduler against the SAME --peers (if scheduler built)
+  xceadmin ...                  # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
   cleanup.sh                    # wipe data/standby-conf/logs, keep ctrl-1 config + certs
 ```
 
-**Startup** (from deploy dir):
+`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.
+
+**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
 ```bash
-VERBOSE=-vvv scripts/start.sh ctrl-1 &   # primary first
+BASH_BIN=$(nix develop ./src --command which bash)   # never a bare `bash`; see below
+"$BASH_BIN" scripts/start.sh ctrl-1 -vvv &   # primary first
 sleep 0.5
-VERBOSE=-vvv scripts/start.sh ctrl-2 &   # standby (0.5s stagger for election ordering)
+"$BASH_BIN" scripts/start.sh ctrl-2 -vvv &   # standby (0.5s stagger for election ordering)
 sleep 0.5
-VERBOSE=-vvv scripts/start.sh ctrl-3 &
+"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &
+
+# Optional, once at least one controller is up:
+"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
+"$BASH_BIN" scripts/start-scheduler.sh -v &
 ```
+(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)
+
+**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.
 
 **Clean restart** (keeps certs + ctrl-1 config, wipes all data):
 ```bash
 kill -9 $(pgrep xce_controller) 2>/dev/null || true
-bash scripts/cleanup.sh
-# Then re-start all nodes
+kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
+kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
+"$BASH_BIN" scripts/cleanup.sh
+# Then re-start all nodes (and worker/scheduler, if you had them running)
 ```
 
 ## Go Frontend Build
@@ -138,5 +158,6 @@ Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` f
 | Scheduler daemon | same | `xce_scheduler` binary |
 | REST API server | same | `xce_apiserver` binary |
 | CLI tool | `nix build .#xcecli` | `xcecli` binary |
+| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
 | AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) |
 | Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |

.kiro/steering/ha-async-io.md

@@ -65,7 +65,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics:
 - **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
 - **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
 - **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
-- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
+- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary).
 
 ## Replication Consistency (Two-Tier)
 

.kiro/steering/persistence.md

@@ -25,7 +25,8 @@ Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_ca
 Two correctness rules when touching this path:
 
 - **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`.
-- **Commit wake-up via callback, not a back-reference.** The flusher calls `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state.
+- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix.
+- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved.
 
 `--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`).
 
@@ -70,7 +71,7 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi
 
 ## Testing Durability Deterministically
 
-- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync` and advances `durable_lsn`) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip).
+- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one.
 - **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`.
 - **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures).
 - Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`.
@@ -82,8 +83,8 @@ When a member cannot reach a committed-consistent state (recovery-truncated/behi
 | Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` |
 | Durable floor (WAL tip) | `persist_durable_floor(p)` |
 | Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` |
-| Group-commit wake | `persist_set_durable_cb` → `on_durable_advance(ctx)` |
-| Synchronous flush (tests) | `persist_flush(p)` |
+| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance |
+| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) |
 | Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` |
 | Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` |
 | Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |